Data pipeline
Eight scheduled jobs, three storage targets and one rule: a job either writes a complete, resolved, validated record or it writes nothing.
Shape of the system
The pipeline is scheduled, not event-driven. Jobs wake on a cron-style schedule, pull from upstream providers, resolve identity, normalize, validate and append. There is no message queue between stages and no streaming ingest, because at this cadence and volume neither would earn its operational cost.
That is a real architectural choice with real consequences, and it is worth stating plainly rather than dressing up: the system is a batch pipeline on a short interval. It is well suited to producing consistent, joinable, reproducible series. It is not suited to sub-minute reaction, and it is not offered as if it were.
Scheduled jobs
| Job | Runs | Writes | Volume | Target |
|---|---|---|---|---|
| Market + social snapshot | every few minutes | one observation per asset per hour | 24,000 rows/day | observation store |
| Asset metadata refresh | daily ~11:00 UTC | reference figures and channel pointers | one record per asset | asset registry |
| Repository activity | daily ~11:00 UTC | repository metrics | 69 rows/day | ecosystem series |
| Channel reach | daily ~13:00 UTC | follower and post counts | 69 rows/day | ecosystem series |
| Contributors | daily ~15:00 UTC | contributor counts | 69 rows/day | ecosystem series |
| Community metrics | daily ~16:00 UTC | community counts | 69 rows/day | ecosystem series |
| News aggregation | hourly | deduplicated news items | rolling window | news feed |
| Narrative clustering | hourly | topic snapshot | 10 topics/hour | narrative snapshots |
Cadences measured on the running system on 2026-08-31. See freshness for what they mean for the age of a value.
Polling interval versus write interval
The market job polls more often than it writes. The poll keeps a warm view of the current market available to the serving layer; the write produces the durable hourly observation. Those are two different jobs wearing one schedule, and confusing them is the easiest way to misstate freshness.
Everything downstream reads written rows. The cache exists to keep read latency low and upstream load bounded; it is never a source of record and never introduces a value that was not written.
Storage targets
Asset registry
Canonical identities. Mutable — names, categories and channel pointers are updated in place as they change — but the identifier never moves. See asset metadata.
Observation store
The hourly grid: market, social, sentiment and score for each asset. Append-only, relational, indexed on asset and timestamp. Currently 6.7 million rows, growing by 24,000 a day.
Series and feeds
Daily ecosystem series, the news window and narrative snapshots, each keyed and timestamped on its own cadence rather than forced onto the hourly grid.
Failure handling
- Upstream error. The run aborts. No partial snapshot is written, because a snapshot missing half its assets would look like a market event rather than an outage.
- Malformed record. The individual record is rejected. Fields that are legitimately nullable stay null; fields that are required and absent invalidate the record.
- Unresolvable asset. The observation is discarded rather than attached to a guess.
- Missed window. The hour stays empty. There is no catch-up backfill that would write a late value under an earlier timestamp.
The consequence of all four is the same: gaps are possible, and they are truthful. Data quality covers how they are detected and what a consumer should do about them.
What is not published here
Hostnames, credentials, queue and container topology, provider rate-limit headroom and retry parameters are internal. This page describes what the pipeline does and how it behaves under failure — which is what you need in order to depend on it — without describing how to attack or circumvent it.