The Setup
Tornadoes kill roughly 150 people in the United States every year. The average warning lead time is 13 minutes. Doppler weather radar sees tornadoes forming — velocity couplets appear minutes before touchdown — but no one is watching every radar, every minute, across 50+ stations.
We asked an AI agent to build a system that could.
The agent designed the architecture, built the data pipeline, trained the model, and validated the results. A human set the direction, reviewed the work, and made calls on what was good enough. The AI did the heavy lifting.
This post is about V3 — the latest iteration of that system.
V3: What It Is
V3 is a U-Net segmentation model (25.3M parameters) trained on 1,203 NEXRAD radar events across 53 radar stations, including 153 NOAA-verified tornadoes. The dataset uses a temporal holdout split: training on 2013–2022, testing on 2023–2025, with zero data leakage verified across splits.
The model takes 3-channel 256×256 radar rasters (reflectivity, velocity, absolute velocity) and produces a per-pixel probability map of rotation zones.
Architecture
V3 improves on V2 with:
Deep skip connections that carry high-frequency detail from early layers to the decoder
Attention gates that let the model focus on rotation-relevant features
Multi-scale feature fusion for both large mesocyclones and tight microbursts
Residual blocks throughout for stable training
Training
Two-phase strategy, developed after analyzing V2's failure modes:
Phase 1 (complete): Balanced subset with Dice + BCE loss, optimized for separation ratio.
Best separation: 4.02× at epoch 11
Validation loss: 0.418
Training loss: 0.420
Phase 2 (pending): Fine-tuning on the full 1,203-event dataset with moderate tornado oversampling.
Results
Moore, OK EF-5 (May 20, 2013) — Training Event
The Moore tornado is in the training set. V3 tracks it across 24 frames with 10 waypoints, maintaining high confidence throughout. Peak prediction: 0.9996.
V3 segmentation heatmap for Moore, OK EF-5 tornado. Red indicates high rotation probability. The model tracks the rotation zone across 24 consecutive radar scans.
This is a training event. The model has seen this storm. It's a demonstration of what the model learned, not a test of generalization.
KMAF, TX Tornado (May 3, 2024) — Test Event
This tornado is in the test set. The model has never seen this radar, this date, or this storm during training.
V3 segmentation heatmap for a May 2024 tornado near KMAF, TX. This event was held out from training (temporal holdout split: 2023–2025). The model correctly identifies the rotation zone despite never seeing this radar station's data during training.
This is the more interesting result. The model is seeing unfamiliar radar geometry and making a prediction it wasn't explicitly trained on.
The Hard Parts — What We're Getting Wrong
If this reads like a product page, we've failed. The model has real problems, and pretending otherwise would be a disservice to anyone who might one day rely on it.
False Positives
The heatmap precision is 50%. Half the time the model flags a rotation zone, there's no tornado. The Ozarks radar (KSGF) is the worst offender — hook-echo-like signatures in non-tornadic storms that look suspiciously like what we're hunting for. KARX and KAMA are also hotspots, with false positive rates above 40%.
This isn't a bug. It's a feature of the problem. Nature produces ambiguous signals. A model that says "maybe" when it's unsure is better than one that doesn't. But we need to get the precision higher before this is operationally useful.
Missed Tornadoes
Tornado recall sits at 78% — we're missing roughly 1 in 5 tornadic events. The model is conservative in ways we haven't fully diagnosed yet. Some tornadoes are small, fast-moving, or occur in radar blind spots. Some are simply outside the patterns the model learned from our training set.
The Dataset Isn't Complete
62.7% of our target events downloaded successfully. The Unidata NEXRAD archive has gaps — pre-2013 data is nearly absent, some files are corrupt, and compression formats change without warning. Of the 350 tornado events in our manifest, only 153 have NOAA verification. The rest are inferred from storm reports. Our labels might be wrong, and we don't always know.
V3 Phase 1 Separation Is Lower Than V2
V2 achieved an 11.6× separation ratio on ~200 events. V3 Phase 1 hit 4.02× on 1,203. The reason: V3 is trained on a much larger, more diverse dataset that includes harder negative cases. V2 was trained on a curated subset where the signal was easier to find.
This isn't a regression — it's reality. The model is seeing messier data. Phase 2 fine-tuning is designed to push the separation back up, but we don't know yet if it will reach V2's levels on this harder distribution.
The CNN Is a Separate Problem
The CNN classifier hits 100% accuracy on our 33-event validation set. That number should make you nervous, not confident. 33 events is tiny. The checkpoint's own internal validation accuracy is 83.3% — a reminder that the 100% number is fragile. The CNN needs more data, harder negatives, and an honest evaluation on unseen storms.
The Fusion Model Is Unproven
Our gradient boosting fusion model (CNN + U-Net + persistence → single probability) achieves 100% per-event accuracy on 8 events and 62.5% leave-one-out cross-validation. Eight events is not a sample size. The fusion architecture works in principle, but it needs stress-testing on hundreds of events before we can trust its output.
How We Fix It
Phase 2 fine-tuning is immediate. The full 1,203-event dataset with moderate tornado oversampling should push V3's separation ratio toward production levels.
Hard negative mining is next. We're actively searching for non-tornadic events that fool the model — hook echoes, mesovortices, boundary layer rotation — and adding them to training. A model that can't tell the difference between a tornado and a convincing fake isn't ready.
Label verification is ongoing. We're cross-referencing every tornado event with NOAA storm reports and correcting labels as discrepancies surface.
Station-specific calibration is planned. KARX, KAMA, and KSGF have known false positive biases. We're exploring per-station thresholds and radar-specific fine-tuning.
Real-world evaluation is the goal. Eventually, this model needs to run on live radar data and be judged on how it performs when the stakes are real. We're not there yet. But we're building toward it.
Why Share This Now
We're not releasing weights because the model isn't ready for public use. But we're sharing the research because:
Transparency matters. A model with 50% precision and 78% recall should be discussed honestly, not hidden behind a polished demo.
The autonomous angle is worth examining. An AI agent built this system. That's not a gimmick — it's a workflow that deserves scrutiny. Did it make good architectural choices? Did it overfit? Did it miss obvious problems? We think the answers are interesting.
The radar community should see this. Weather researchers, NWS forecasters, and anyone working on severe storm detection — this is early work, but it's real work. We'd rather share it now and get feedback than wait until it's "ready."
What's Next
Phase 2. Hard negatives. Label verification. Station calibration. And the question that keeps us up at night: can a model built by AI be trusted to warn people about tornadoes?
We don't know yet. We're not pretending we do.
What we do know is that the human is still in the loop. Setting the goals. Checking the results. Reading the failure analysis and asking "why did we miss that one?" Making the call on what's good enough — and right now, the answer is not yet.
The AI did the heavy lifting. The human keeps the compass. And the compass says: more work.
OpenNado is a research project exploring autonomous AI for severe weather detection. The code and methodology are documented in our repository. We welcome contributors, reviewers, and skeptics. If you work in meteorology, ML, or both — we'd love to hear from you.