Moonlytics.io
Engineering

Measuring data freshness

Most published freshness numbers are configuration values, not measurements. The gap between the two is where 'real-time' claims come from.

Two different questions

Ask a team how fresh their data is and you will usually get the polling interval — how often a job runs. That is a fact about the scheduler. The question a consumer is asking is different: how old is the newest value I can read?

Those diverge whenever polling and persistence run at different rates, which is extremely common. Our own market job wakes every few minutes; it persists one observation per asset per hour. Quoting the polling interval would be technically true and a lie in every way that matters, because everything downstream — every chart, every score, every join — reads persisted rows.

Measure the artefact, not the intention

The measurement is a group-by. It takes thirty seconds and it is the only version of this number worth publishing.

freshness, measured
-- Do not read the cron expression. Count the rows.
SELECT date_trunc('hour', observed_at) AS hour, count(*)
FROM observation
WHERE observed_at > now() - interval '6 hours'
GROUP BY 1 ORDER BY 1;

         hour          | count
-----------------------+-------
 2026-08-31 04:00:00Z  |  1000
 2026-08-31 05:00:00Z  |  1000
 2026-08-31 06:00:00Z  |  1000
 2026-08-31 07:00:00Z  |  1000
 2026-08-31 08:00:00Z  |  1000
 2026-08-31 09:00:00Z  |  1000

-- Exactly 1000 rows an hour. The delivered cadence is
-- hourly, no matter how often the job wakes up.

Three things fall out of this that a config file cannot tell you:

  • The real cadence — the spacing of distinct timestamps.
  • The real breadth — rows per period, which catches partial writes.
  • The real gaps — periods that are simply absent.

The three honest classes

Freshness claims collapse neatly into three, and almost all overstatement comes from putting a system in the wrong one:

  • Near-real-time. Sub-minute. If your data is not in this class, say so explicitly — silence gets read as a claim.
  • Periodic. Fixed schedule, hourly or daily. Age of the newest value is bounded by the interval.
  • Historical. Written once, never changed.

Our own table puts every category in a class and states that we have nothing in the first one. That sentence is worth more to someone building on the data than any amount of "real-time" language, because it lets them design correctly on the first attempt instead of discovering the truth in production.

The failure that measurement catches

Stale-but-well-formed is the hardest data failure to detect, and the row count is what finds it. An upstream provider returning a successful response with values it has not refreshed produces perfectly valid rows, correct cardinality, and a series that has quietly stopped moving.

Monitoring completion tells you nothing here — the job succeeded. What tells you is watching the distribution of values change, or not. A row count per period catches the partial-write case; value-level monitoring catches this one.

Never fill to make freshness look better

When a run fails, the honest artefact is a gap. The tempting fix is to write the previous value under the current timestamp, which makes every freshness metric green and turns an outage into a fabricated observation — one that is indistinguishable from a real one, forever. Same argument as gap filling, with worse consequences, because it defeats the monitoring that would have told you.

Publish it where a consumer will look

Freshness belongs in three places, and only the first is usually done:

  1. In the payload. Every value carries the time it describes.
  2. In the documentation. A measured cadence per category, with the measurement date.
  3. In an endpoint. Last write and age per category, so a consumer can check without asking.

We do the first two. The third is planned alongside the public Data API, and it is listed as planned rather than described as if it existed.

The short version

Measure what you deliver, not what you schedule. Publish the number even when it is unflattering. A documented hourly cadence is something a competent team can build on. A "real-time" claim that turns out to mean hourly costs you the credibility of everything else on the page.

Cadences on this site were measured on 2026-08-31 with the query above, against the production store.

Related