Moonlytics.io
Infrastructure / Normalization

Data normalization

Six incompatibilities stand between a provider payload and a usable row: schema, naming, type, unit, time base and frequency. Normalization is the stage that resolves all six, the same way, every time.

The goal

One consistent data model. Not a superset of every provider's fields, and not a lowest common denominator — a model designed around what an asset observation actually is, into which provider payloads are mapped.

That distinction matters. A layer that simply forwards provider fields under provider names pushes the reconciliation problem downstream to every consumer, who will each solve it slightly differently. The point of normalizing once is that nobody else has to.

What arrives

raw provider payloads
# Two providers describing the same asset in the same minute

# provider A - market and social aggregator
{ "market_cap": 1568738541274, "volume_24h": 13029310589,
  "percent_change_24h": 0.7, "social_volume_24h": 41208,
  "galaxy_score": 72.5, "alt_rank": 3 }

# provider B - metadata provider
{ "market_cap": "1568738541274", "total_volume": "13029310589",
  "price_change_percentage_24h": 0, "circulating_supply": "20077215",
  "last_updated": "2026-08-30T11:01:04.377Z" }

# different names, different types, different rounding,
# different cadence, no shared timestamp

What is stored

canonical observation
{
  "asset": { "id": 1, "symbol": "BTC", "name": "Bitcoin" },
  "observedAt": "2026-08-31T09: 00: 00Z",
  "marketCap": 1568738541274,
  "volume24h": 13029310589,
  "socialVolume24h": 41208,
  "score": 72.5,
  "altRank": 3
}

The six problems

1. Different schemas

Providers do not agree on what an asset record contains. One nests social data under the asset; another keeps it in a separate resource. One returns a flat list; another returns an envelope with metadata. Mapping is explicit and per-provider: each source field is mapped to a canonical field by a rule that exists in one place, not scattered through consuming code.

2. Different field names

volume_24h, total_volume and volume24h are three names for a quantity that is almost the same. Almost is the problem: one is aggregated across venues by one methodology, another by a different one. Canonical names are assigned deliberately, and where two providers' versions of a quantity are not interchangeable they become two fields, not one.

3. Different types

Large numbers arrive as strings, as floats, and as integers, depending on the provider and sometimes on the magnitude. Market capitalisations exceed the range where floating point is exact. Everything monetary is coerced to a fixed precision decimal type on the way in, and stored that way — not as a float that quietly loses the last digits of a trillion-dollar figure.

4. Different units and scales

Percentages arrive as 0.7 and as 70. Dominance is a fraction at one source and a percent at another. Prices come in USD and in BTC. Each canonical field has one documented unit, and conversion happens at the boundary — never in a consumer, and never twice.

5. Different time zones and timestamps

Providers stamp records in UTC, in local time, in epoch seconds and in epoch milliseconds, and some stamp when they generated the response rather than when they observed the value. Everything is converted to UTC on ingest. There is no local time anywhere in the system, and no field that means "when we happened to ask".

6. Different frequencies

This is the one that is usually handled badly. Market data arrives continuously, repository statistics change daily, follower counts drift slowly. The tempting move is to join them into one wide row per asset per hour, forward-filling the slow fields.

We do not do that

Forward-filling a daily value across twenty-four hourly rows manufactures twenty-three observations that were never made. A model trained on it will find structure that is an artefact of the fill. Series are stored at their own native cadence and joined explicitly at query time, where the person doing the joining can see exactly what they are aligning.

Snapshot alignment

Within the hourly grid, every observation for a given hour carries the same timestamp — the top of that hour — regardless of the second at which the job actually wrote it. That is what makes a cross-sectional query at a point in time meaningful: all thousand assets in a snapshot describe the same moment, not a thousand moments spread over a minute of write time.

Missing data

Absent values are stored as null and stay null. Nullable fields are documented as nullable on every data page. Null means not observed; it never means zero, and it is never replaced by a previous value. Handling of gaps is covered under data quality.

Where normalization stops

Normalization does not compute derived metrics. Ratios, z-scores, momentum measures and composite scores are the business of the layers above, and mixing them into the normalization stage would make the stored record depend on a formula that can change. What is stored is what was observed, in one shape.

Related