Saturation and registry metrics

While a transform run walks the corpus, a registry (unique keys such as lemmas) grows. Saturation metrics answer dashboard questions like:

How many new keys appeared in this checkpoint window vs keys we had already seen (n_new / n_hit)?
How large is the dictionary now (distinct_keys)?
How “concentrated” are assignments — is mass piled on a few keys or spread out (homogeneity)?

Those numbers are computed outside Prometheus. Telemetry only emits gauges from a neutral slice DTO. That keeps scrape code free of psycopg and free of domain registry internals.

Who computes what

Quantity	Who owns the math
`n_new` / `n_hit` in a checkpoint window	Domain registry builds in-RAM slices
`distinct_keys`	Generic count (often a PostgreSQL proc, or Python `count_distinct`)
Homogeneity %	One shared formula (`homogeneity_percent` / formula B) — not reinvented per step
Gauge emit	`telemetry/prometheus/saturation.py` from `SaturationSlice`

DTO: framework/contracts/saturation_slices.py.

Glue on a live run

During partition_loop, checkpoint cadence (from DRTML) triggers a poller that snapshots registry / distinct-key state and hands slices to emit. Plan/snapshot helpers live under processing/partition_loop/checkpoint_metrics/. Storage and DB transport stay out of the Prometheus package.

Multi-worker caveat

With workers > 1, each worker sees only its shard. Treating a single worker’s homogeneity gauge as the global truth is unsafe unless the driver aggregates. Prefer designs that either run registry metrics inprocess for that path or publish aggregated slices after merge.