Infrastructure
The part of the system nobody sees and everything depends on. Five documented stages between a provider payload and a row an application can trust.
- PipelineEnd to end: what runs, on what schedule, and where each value lands.
- Entity resolutionThree providers, three identifiers, one asset. The hardest problem in the layer.
- NormalizationDifferent schemas, units, time bases and frequencies collapsed into one model.
- EnrichmentJoining market, social, news and ecosystem context onto one asset identity.
- Data qualityMissing values, stale windows, outliers, duplicates and provider failures.
Why this is documented at all
Most data vendors describe their coverage and stop. Coverage tells you how much data there is; it tells you nothing about whether it is right. The failure modes that actually cost people money — an observation attached to the wrong asset, a gap filled with a stale value, two providers' volumes silently summed — all live in the processing, not in the coverage.
So the processing is documented. Every stage below is also a place where the layer can be wrong, and each page says how it is wrong when it is.
Design constraints we hold to
- Identity before anything else. A value with an uncertain asset is discarded, not stored optimistically.
- Observations are immutable. Nothing is edited in place; a correction is a later observation.
- Missing stays missing. No forward-fill, no interpolation, no carrying a value forward under a new timestamp.
- One clock. Everything is UTC and aligned to the snapshot grid, so series from different sources can be joined without interpolation.
- Provider cadences are not blended. An hourly series and a daily series stay separate series.
What is deliberately not on these pages
Provider contract terms, credentials, internal hostnames, queue topology, rate-limit headroom and the specific matching heuristics that make identity resolution work are not published. The public documentation describes the architecture and its behaviour, which is what someone integrating needs. Anything that would help someone circumvent a provider agreement or attack the infrastructure stays internal.