The dashboard that was wrong for five weeks

The mess

The supplier feed stopped on a Tuesday in March. The ingestion job kept running, kept succeeding, and kept finding nothing new. The dashboard, built to show “the latest available figures”, did exactly that — it showed the latest available figures, which were five weeks old, with no indication anywhere on the page that they had stopped moving.

Two procurement decisions were made on those numbers.

The job’s own monitoring was green throughout. It had been written to alert on failure, and it never failed. It just had nothing to do.

The method

The fix is one idea: never trust the freshness of a snapshot, assert on the data’s own maximum date.

Every ingestion step now compares the newest record in the payload against what it expects for the day it runs. If the newest record is older than the tolerance for that feed, the run fails — loudly, with the feed named and both dates printed. A pipeline that stops is a visible problem. A pipeline that serves stale data is an invisible one, and invisible problems are the expensive kind.

This sounds obvious written down. It is also the single most common defect I find, and it is almost always in systems whose monitoring reports green.

The dashboard

The report now carries its own data’s date, in words, at the top — not in a footnote, not in a tooltip. When a feed is behind, the affected section is marked and says how far behind, rather than rendering a confident chart of old numbers.

The uncomfortable part of this engagement was the first conversation: going back through five weeks of decisions to work out which ones would have been made differently. That conversation is the reason the freshness check exists in every build I have shipped since.