Data quality
Every data set has failure modes. The difference between a usable one and a dangerous one is whether they are documented. Here are ours, what we do about each, and what you should do about them downstream.
The governing principle
A gap is more honest than a guess. Every quality decision in this layer follows from that. When something cannot be observed, the layer records that it was not observed, rather than producing a plausible value that nothing downstream can distinguish from a real one.
The cost of this is visible: series have holes, fields are null, and a consumer has to handle both. The benefit is that no result computed on this data is quietly built on numbers that were never measured.
The eight failure modes
| Failure mode | What it looks like | How the layer handles it | Why it matters downstream |
|---|---|---|---|
| Missing data | A nullable field is absent for an asset in a snapshot. | Stored as null. Never zero, never filled. Documented as nullable on the data page. | Consumers that treat null as zero produce averages that drift toward zero as coverage worsens. |
| Missing snapshot | An entire hour has no observations. | The hour stays empty. No backfill under an earlier timestamp. | Forward-filling across the gap fabricates observations and creates false flat periods. |
| Stale data | The newest row is older than its cadence allows. | Detected by comparing the newest observedAt against the expected cadence. | A cached value served as current is worse than no value, because nothing signals the age. |
| Outliers | A value orders of magnitude away from the asset's own recent range. | Range-checked at ingest. Impossible values rejected; extreme-but-possible values stored. | Crypto genuinely produces 100x moves. Clipping real events is as damaging as accepting bad ones. |
| Duplicates | Two records for the same asset and the same window. | One observation per asset per snapshot is enforced structurally, not by post-hoc dedup. | Duplicate rows double-count in any aggregate and are invisible in a chart. |
| Inconsistent symbols | The same symbol on different assets, or a changed symbol on one asset. | Nothing joins on symbols. Resolution runs on provider identifiers only. | The single most damaging failure in this asset class. See entity resolution. |
| Provider failure | Upstream returns an error, a truncated list, or an authentication failure. | The run aborts. Nothing partial is written. | A half-written snapshot looks like assets vanishing from the market rather than like an outage. |
| Delayed updates | The provider returns successfully but with values it has not refreshed. | Written as observed at the current snapshot. Repeated identical values are visible in the series. | Hardest failure to detect, because the response is well-formed. Flat series are the tell. |
Validation at ingest
Records are checked before they are written, not after. The checks are deliberately boring, because interesting validation logic is validation logic that produces surprises:
- Required fields present. A record missing a non-nullable field is rejected outright.
- Types coerce cleanly. A monetary string that does not parse to a decimal invalidates the record rather than becoming zero.
- Ranges are sane. Negative capitalisations, dominance above 100, timestamps in the future.
- Identity resolves. No canonical asset, no write.
- Snapshot uniqueness. One observation per asset per window.
Confidence is a property of coverage, not a score
We do not attach a numeric confidence to individual values, because a confidence score computed from data whose quality is unknown is itself unknown. What is published instead is the information a reader needs to form their own judgement: measured cadence on the freshness page, explicit nullability per field, explicit coverage per category, and a stated limitations section on every data page.
Two concrete cases where that matters more than a score would:
- Sentiment on thin volume. A sentiment value computed from eleven posts is not wrong, it is imprecise. Serving it next to
socialVolume24hlets the reader see that; a single blended confidence number would hide it. - Assets near the coverage boundary. An asset that drifts in and out of the ranked set has a fragmented series. That is visible in the data itself, and no summary statistic communicates it better than the gaps do.
What a consumer should actually do
- Check
observedAt. Never assume the newest row is the current hour. - Treat null as unknown. Exclude from aggregates rather than coercing to zero.
- Do not forward-fill across gaps unless the analysis explicitly tolerates it, and say so when you do.
- Join on the canonical asset id, never on a symbol.
- Read sentiment against volume, and any ratio against its denominator.
- Expect flat stretches. Repeated identical daily values are usually a real property of the source, not a bug.
Monitoring
Ingest jobs are monitored for completion, run duration and written-row counts, which is what catches a silent partial failure — a run that succeeded but wrote a fraction of the expected rows. Service-level state is summarised on the status page. Alerting thresholds and internal dashboards are not published.
All measurements referenced on this page were taken against the production store on 2026-08-31.
Related
- Data freshness — measured cadence per category.
- Pipeline — where each check runs.
- Measuring data freshness — the engineering write-up.