Research group update · Preliminary results

Object-based evaluation of HRRR-forecast MCSs

Tracking mesoscale convective systems across forecast cycles, quantifying structural errors, and establishing a foundation for probabilistic hazard guidance.

01 · Scientific objective

Can tracked HRRR MCSs support useful hazard guidance?

Evaluate the full forecast chain—not only pointwise reflectivity skill.

Track MCSs independently within each HRRR initialization.
Compare forecast objects with MRMS-observed MCSs.
Separate occurrence, location, size, structure, and lead-time errors.
Preserve initialization, valid time, and forecast lead for fair comparisons.

48 h maximum forecast lead

4× daily initialization cycles

1,836 April-August 2021-2023 initializations

02 · Experimental design

Four forecasts converge on the same Day +2 12Z endpoint

D 12ZF01–F48

D 18ZF01–F42

D+1 00ZF01–F36

D+1 06ZF01–F30

HRRR fields

PyFLEXTRKR

Forecast MCS masks

MRMS matching

Lead/cycle scores

Fairness constraint: initialization-cycle rankings must use either common F01–F30 leads or identical valid times. Pooled curves alone mix forecast lead with the diurnal cycle.

03 · Object verification

Skinner-style total-interest matching, adapted for MCS scale

TI = ½[(250 − d_centroid)/250 + (100 − d_boundary)/100]

Candidate match when TI > 0.2.
Greedy highest-interest, one-to-one assignment.
Matched pair = hit; unmatched forecast = false alarm; unmatched observation = miss.
Identical hourly valid times, so temporal displacement is zero.

Open matching-scale sensitivity

250 km maximum centroid scale

100 km boundary-distance scale

POD observed objects successfully forecast

CSI hits relative to all forecast errors

04 · 2021-2023 warm-season baseline

Moderate object skill with substantial overforecast frequency

0.559POD

0.594FAR

0.308CSI

1.376frequency bias

1,836initializations

69,701valid forecast hours

73,842matched object pairs

The median centroid displacement of matched objects is 157 km; large MCS geometry makes the minimum-boundary term important.

05 · Initialization-time comparison

Cycle differences shrink when verification weather is held fixed

Cycle	Lead range	POD	FAR	CSI
12Z	F19–F48	0.561	0.636	0.283
18Z	F13–F42	0.568	0.614	0.298
00Z	F07–F36	0.599	0.588	0.323
06Z	F01–F30	0.538	0.561	0.319

10,582 identical valid times per cycle; 454 complete forecast groups.

12Z and 18Z are significantly worse than 06Z:
ΔCSI = −0.035 [−0.047, −0.024] and −0.020 [−0.032, −0.009].

Bootstrap resamples complete forecast groups, not correlated hours.
00Z and 06Z are not statistically distinguishable.
Lead bins beyond F30 contain fewer cycles and shifted valid hours.

06 · Structural error

Smaller rain areas are offset by excessive rain rates

Violin distributions of matched MCS cloud-shield area, precipitation-feature area, mean rain rate, and volumetric rain rate by initialization cycle

2.07× median forecast/observed cloud-shield area

0.64× median forecast/observed precipitation-feature area

HRRR concentrates excessive rain intensity into precipitation features that are too small, while the surrounding cold cloud shield is too broad.

Total-interest-matched objects, April-August 2021-2023; cycle medians are printed below each violin. Click to enlarge.

07 · Case study · 20 June 2021

Same valid time, different leads—and persistent oversized cloud shields

Observed MRMS and four HRRR forecast MCS outlines valid 20 June 2021 at 12 UTC

1.86×12Z F48 area ratio

2.18×18Z F42 area ratio

2.24×00Z F36 area ratio

1.75×06Z F30 area ratio

The spatial envelope is recognizable, but forecast MCS masks extend well beyond the observed objects.

Black: MRMS observed MCS; red: HRRR forecast MCS. Click to enlarge.

Open case animation · Open case time series

08 · Initialization-cycle diagnostics

CSI and frequency bias evolve differently across forecast cycles

CSI and frequency bias by forecast lead and initialization cycle for April-August 2021-2023

00Z has the strongest mid-lead CSI.
12Z degrades most clearly toward long leads.
Frequency bias generally rises above one after F06.
Only F01–F30 is sampled by all four initialization cycles.

Interpretation: the apparent long-lead decline cannot be attributed entirely to forecast aging because cycle coverage and valid hour also change beyond F30.

April-August 2021-2023. Click to enlarge.

Open lead-time performance · Open forecast-hour performance

Take-home messages

What we know so far

Object-based HRRR MCS verification is operational from tracking through matched statistics.
Cycle comparisons require matched valid times; apparent lead degradation is partly confounded by sampling and diurnal variability.
The dominant preliminary signal is structural: cloud shields are too large, while precipitation features are too small.

Discussion: how should these scale-dependent biases be handled before forecast hazard probabilities are interpreted?