Forecast
Scoreboard

Stations, not people. Days 1–5. Compared with the National Weather Service forecast from the same hour. No one pays for a rank.

Ruleset 1.1.0 · d7ae66670a5c

How a station gets a score

Forecast Scoreboard is an independent, neutral scoreboard of forecast accuracy for television stations, the National Weather Service, and weather models. These rules are the file the scorer reads. The page and the arithmetic use the same version. A score is a function of confirmed forecasts, final observations, and this file. Nothing commercial is an input.

What is ranked

The ranked unit is the television station. A station forecast is usually a team product. Putting a person’s name on it from a shift schedule, a chyron, or an uploader field would often be wrong, and a wrong public ranking of a named person does more harm than an empty leaderboard.

The database can store a meteorologist and can attach one to a forecast that has an explicit byline or a named post. Those rows are not read by the scorer. rank_individuals in this version is false.

The pilot is two markets:

Coordinates and NWS grid points were taken from api.weather.gov station and points documents on September 26, 2026. They are stored with the site so the baseline forecast is for the airport, not a downtown point chosen later.

Which forecast counts

The forecast of record is the last confirmed version captured at or before 23:00 local time. Day 1 is the next calendar day. A capture after the cutoff belongs to the next issue date.

Each stored forecast records a source type: on-air graphic, vendor widget, station-modified values, NWS point forecast, or a local fixture. The text or JSON “7-day” on a station page is often an automated vendor feed and can disagree with the graphic that aired. This version is built to score the on-air graphic. Vendor widgets can be stored later, labeled as what they are, and kept out of the rank.

Station websites are not collected. The terms of use checked for Scripps, Hearst, Gray, and Nexstar prohibit scraping. The collector interface exists, the NWS collector runs, and an example collector reads a local file. Enabling a station collector is a later decision, after counsel, not a switch this site flips on its own.

Lead day is recomputed when the page is scored, from the capture time, the valid date a reviewer confirmed, and the cutoff in this file. Forecasts already stored keep their bytes. Changing the cutoff changes which capture is the forecast of record; it does not rewrite the archive.

The climate day

High, low, and precipitation are the official climate-day values: midnight to midnight, local standard time. In summer that window is 1:00 a.m. to 1:00 a.m. daylight time. That is the day the National Weather Service climate report describes, and it is the day ACIS returns for the airport station.

Primary source is ACIS (data.rcc-acis.org, elements maxt, mint, pcpn). The NWS daily climate report on api.weather.gov is the fallback, because that product list only keeps about a week of reports. Every report we fetch is copied into the local archive. A report whose text says “VALID TODAY AS OF” is preliminary. It is stored and it is not scored. A final report, or a complete ACIS day, replaces it.

If ACIS and a final climate report both have values and they disagree by half a degree or more, or they disagree on whether the day was a precipitation event, the row is flagged. The scored values stay with ACIS.

The low temperature

Version 1 scores the low printed for a forecast date against the climate-day minimum of that same date. Television graphics usually mean the overnight low, the one that falls on the night after that day’s high and often after local-standard midnight. That overnight low can belong to the next climate day.

An overnight window, built from hourly observations, is the right sensitivity test and it is not what this version scores. The observation table has a column for it. The column is empty. Absolute low error can include that mismatch. Skill against the NWS forecast is still a comparison of like with like: the baseline low is the NWS overnight period, scored against the same climate-day minimum.

The high is the climate-day maximum of the labeled date. That lines up with what a 7-day graphic means by “high.”

Precipitation

An event is at least 0.01 inch on the climate day. A trace is not an event, which matches the usual “measurable rain” line. The Brier score uses 1 for an event and 0 otherwise.

For the NWS baseline, the daily probability is the daytime period on the gridpoint forecast. The low is the nighttime period that starts on that same local date. Station probabilities are whatever the extractor read, after review.

A blank probability on a graphic is scored as 0%, because that is what a viewer sees when the number is absent. An unreadable number is missing, not zero, and it stays out of the score until someone fills it in.

Metrics

Temperature error is forecast minus observed. Mean absolute error is the average of the absolute errors. Bias is the average of the signed errors. Positive bias means the forecast was too warm. “Within 3°F” is reported beside the high and is not part of the composite.

Precipitation uses the Brier score, the average of (probability − outcome) squared, with probability as a fraction from 0 to 1.

Skill against the NWS forecast captured for the same issue date is 1 − (this station’s mean loss / the NWS mean loss). Loss is absolute error for temperature and squared error for precipitation. Zero is a tie with the NWS. Positive means a lower loss. If both losses are zero, skill is zero. If the NWS loss is zero and the station’s is not, that component is left out rather than turned into an infinite penalty.

The composite

The headline score uses days 1 through 5. Days 6 and 7 are shown on the station page and are not in the composite. Extended forecasts have little skill and are easy to market; they should not decide a rank.

Lead weights:

Variable weights:

The composite is the weighted mean of the component skills. If a component is missing, it is dropped and the remaining weights are scaled so they still sum to 1. The station page also shows the equal-weight mean of whatever components exist.

The whole stored record is scored (all_confirmed_days_on_file). A fixed 12-month season is not computed yet. When it is, it should be a separate published number, not a silent replacement of this one.

Matched days

Stations are compared on days they all published. A station with no confirmed forecast yet is not in the comparison set, so it does not block the others and it does not get a score. Once a station has any confirmed forecast, a day it misses is dropped for every station in the market.

A day counts as matched for a lead and a variable when every station in that set, and the NWS baseline, have a value, and the final observation exists. The record length used for “provisional” and “established” is the number of issue dates with a matched day-1 high. The composite uses those issue dates. On each of them, a longer lead counts only when that lead is matched too.

The national list ranks stations by that market-relative skill. Kansas City and Oklahoma City do not have to share calendar days. The NWS error shown on a market page is separate: it uses every day the baseline forecast and a final observation both exist, not only the matched-station days.

When a rank is published

Under 90 matched days, the station is collecting. The arithmetic can be shown as a running tally. It is not a rank.

At 90 matched days the rank is provisional. At 180 it is established. An official all-season ranking is a later milestone and is not claimed here.

Rows stored as sample data are excluded from every public page. They are not a rank.

Intervals and ties

The 95% interval is a percentile bootstrap of the composite, resampling the matched day-1 issue dates 2000 times. The random seed is a hash of this ruleset and the station, so the same archive and the same file always produce the same interval.

Stations are ordered by the point estimate. If a station’s interval overlaps the station above it, the page marks them statistically tied. Overlapping intervals are not a license to treat a small sample as a settled order.

Review

An extracted graphic does not count until a person confirms it or corrects it. The original extractor output is kept. Corrected values are what the scorer reads. Rejected extractions are ignored. The NWS document is confirmed on ingest because it is a structured feed, not an OCR guess.

There is no public corrections form yet. Until there is one, the review queue is the corrections path, and it should sit behind an admin token on any shared host.

Models

A model is a forecast source with role model. The station comparison set is unchanged: stations that have started, plus the NWS baseline. Models are scored in their own set, also against that NWS baseline, with the same mean absolute error, Brier score, skill, weights, and sample thresholds. station_comparison_set_excludes_models is true.

GFS daily values come from the 0.25° NOAA Open Data files. The collector reads the index, fetches only the 2 m maximum temperature, 2 m minimum temperature, and surface precipitation messages, and keeps the nearest grid point. The daily high is the maximum of the four 6-hour TMAX buckets on the local-standard climate day. The low is the minimum of the four TMIN buckets. Precipitation is the sum of the four 6-hour accumulations, not the overlapping run-total. A day missing any bucket is dropped.

GFS does not issue a probability. At ingest, a climate-day total at or above 0.01 inch is stored as 100%, and a smaller total as 0%. The threshold used is written into the archived JSON. The “beats the raw model” line on a station page ignores that conversion and compares headline temperature error only. Lower error wins. Shared issue dates are those on which both sources have a day-1 high and low and the observation is final.

ECMWF IFS, ECMWF AIFS, Canadian GEM, GEML, ICON, and NAM have public files and no collector yet. HRRR is public and too short for days 1–5. UKMET’s global grid and Google’s operational model output are not anonymous public downloads, so they are listed and skipped. Licenses are on the model catalog, checked 2026-09-26.

Events

Hurricane track error is the great-circle distance in statute miles (Earth radius 3,958.7613 miles) at leads 24, 48, 72, 120 hours. Intensity error is the absolute difference in knots. A forecast is paired with the NHC operational b-deck fix at the same valid time. Positions are not interpolated. Skill uses the same ratio as the daily board, against the NHC official forecast (OFCL) from that issue time. Public a-deck aids ingested for scoring are GFS (AVNO), GEFS mean (AEMN), Canadian GEM (CMC), UKMET (UKX), HAFS-A, HAFS-B, HWRF, HMON, and NAVGEM. Interpolated aids and ensemble perturbations are not the primary score. ECMWF and AIFS were not in the public a-deck at the last check.

Landfall location and timing are scored only when an explicit landfall row is stored. The track is not scanned for a coastline crossing.

Snow, heavy rain, and local storm impacts may be a range. The absolute error is the distance from the observation to the midpoint. A hit means the observation is inside the inclusive range, so 6 inches hits a 4–8 inch forecast and the midpoint error is 0. A single number is a range whose ends are equal. Verification sources, when collection starts, are NWS Public Information Statements, CoCoRaHS, and ASOS for snow and rain, and SPC storm reports for severe weather. Those collectors are not running, and no sample storm is mixed into the real event list.

Left out of this version