Step01 Metrics

step01_metrics evaluates the lemma vocabulary produced by Step01. It is an offline DRTML v4 dataset-metrics step:

drtml: 4
step: step01_metrics
schema: step01_lemma_metrics_v1
run:
  type: dataset_metrics
  executor: postgres

It does not read or parse raw text. It consumes durable PostgreSQL datasets from one or more completed Step01 runs.

Why metrics are a separate step

Step01’s hot path should spend time on corpus parsing and durable evidence, not on repeatedly recomputing distribution statistics.

The split provides:

• independent retry and scheduling;
• exact metrics over completed upstream runs;
• changing milestone cadence without rerunning spaCy;
• direct PostgreSQL aggregation without PG → Python → PG row movement;
• one metric-point contract shared by CSV, PostgreSQL, Prometheus, and Grafana.

Step01 publishes only live execution observability. Vocabulary quality, shape, saturation, and growth belong here.

Inputs

Dataset	Physical table	Used by
step01_lemma_vocab_run	pipeline.drt_tbl_lemma_run_staging_v1	final
step01_lemma_document_count	pipeline.drt_tbl_lemma_document_count_v1	milestones

metrics.upstream_runs selects Step01 run ids. A comma-separated value can merge multiple runs into one evaluation.

Every selected run must use compatible lemma identity (the same tenant HMAC key and lemma/POS semantics).

Runtime selection

The manifest declares both jobs. metrics.mode selects one:

final       → exact metrics over merged run vocabulary
milestones  → exact deterministic document-progress curves

The Step01 Metrics run id is separate from every upstream Step01 run id:

• upstream ids select source evidence;
• metrics run id identifies output points, CSV artifacts, Prometheus labels, and Grafana $run_id.

Final mode

Input: step01_lemma_vocab_run.

The PostgreSQL executor uses the direct exact family procedure (pipeline.drt_proc_corpus_growth_metrics_final) because all selected methods advertise direct-final support. Source rows remain in PostgreSQL; Python transports only result points.

Exactly one point is emitted per metric with:

• corpus_fraction = 1.0;
• merged upstream source ids;
• order_policy = final_merged.

Final metrics

Metric	Method	Meaning
cardinality	count_distinct	number of unique lemma/POS ids
mass	total_mass	total lemma-token occurrences
homogeneity	assignment_homogeneity	percentage of assignments opening a new key
saturation	assignment_saturation	percentage hitting an already-known key
new_in_sample	assignment_new_count	new keys in the complete sample
singleton_ratio	singleton_ratio	share of types occurring once
repeated_ratio	repeated_ratio	share of repeated types
frequency_mean	frequency_mean	mean frequency per type
frequency_p50 / p90 / p99	matching methods	frequency percentiles
entropy_normalized	matching method	normalized frequency-distribution entropy
effective_cardinality	matching method	entropy-equivalent vocabulary size
frequency_gini	matching method	inequality of type frequencies
top_1pct_mass_share	matching method	mass held by top 1% of types
top_10pct_mass_share	matching method	mass held by top 10% of types
aggregation_ratio	matching method	average mass per type
effective_sample_size	matching method	concentration-adjusted sample size
zipf_alpha	matching method	fitted Zipf exponent
zipf_fit_r2	matching method	Zipf log-rank fit quality

Formula B definitions:

homogeneity = 100 × n_new / (n_new + n_hit)
saturation  = 100 × n_hit / (n_new + n_hit)

For non-empty samples they are complementary.

Milestones mode

Input: step01_lemma_document_count.

Each source row represents (run_id, doc_id, lemma_token_id, token_count). PostgreSQL:

1. assigns documents a deterministic stable-hash order using compute.corpus_growth.order_seed;
2. groups documents into windows of metrics.milestone_step;
3. accumulates key/mass state at each window;
4. invokes generic corpus-growth metric procedures;
5. emits points whose X axis is document progress.

No MinIO/parquet evidence rows are read for this mode.

Milestone axes

Field	Meaning
corpus_bucket_end	number of ordered documents included at the point
corpus_fraction	corpus_bucket_end / total_documents
order_policy	stable_hash_document_milestones
order_seed	configured deterministic ordering seed

The final short window is retained, so the last point reaches corpus_fraction = 1.0.

Milestone metrics

Milestones compute the same cardinality, mass, distribution, novelty, saturation, and Zipf families as final mode, plus:

Metric	Method	Meaning
new_in_sample	assignment_window_new_count	unique ids first observed since the previous milestone
heaps_beta	vocabulary_growth_beta	exponent β in vocabulary growth V ∼ N^β
heaps_r2	vocabulary_growth_fit_r2	Heaps-curve fit quality

Changing metrics.milestone_step or the order seed reruns only Step01 Metrics. It does not require spaCy extraction again.

Outputs

Both selected jobs write the same logical metric-point rows.

step01_lemma_metric_points

CSV artifact under the Step01 artifact root:

Column	Meaning
dataset	logical source label (step01_lemma_vocab_run)
metric	metric alias
value	numeric result
corpus_fraction	normalized corpus progress
corpus_bucket_end	terminal bucket/document count
group_key	group dimension (empty for this step)
accuracy	execution accuracy
order_policy / order_seed	reproducibility metadata
source_run_id	selected upstream run ids

step01_lemma_metric_points_pg

PostgreSQL output in pipeline.drt_tbl_corpus_growth_metric_point_v1.

The unique point identity is:

(run_id, dataset, metric, group_key, corpus_fraction, order_policy)

Retrying the same metrics run updates value/provenance fields instead of duplicating points.

Grafana reads these points through the whitelisted function pipeline.drt_proc_corpus_growth_curve.

Parameters

Param	Default	Meaning
metrics.upstream_runs	empty (required at run)	comma-separated Step01 run ids
metrics.mode	final	final or milestones
metrics.milestone_step	100	documents per milestone window
metrics.progress_log_every_n_rows	100000	staging/progress log cadence
compute.corpus_growth.order_seed	v1	deterministic corpus ordering seed
compute.corpus_growth.max_milestones	1000	safety bound on generated windows
runtime.metrics_enabled	true	publish computed points to Prometheus

For milestone mode:

expected windows = ceil(total documents / milestone_step)

If this exceeds max_milestones, increase the step size or the explicit safety cap.

Observability and dashboard

Computed points are published through source: corpus_growth_point. Prometheus receives the result; it does not recompute formulas.

The generated dashboard (grafana.uid: drtoller-step01-metrics) contains:

• CPU and RAM;
• final stats for vocabulary size, mass, singleton ratio, Gini, Zipf, saturation, homogeneity, entropy, concentration, and novelty;
• PostgreSQL-backed corpus-growth curves for cardinality, mass, singleton ratio, Gini, Zipf, Heaps, novelty, homogeneity, and saturation.

Metric points retain corpus_fraction and document bucket metadata for reproducible growth analysis. Select the Step01 Metrics run id, not the upstream Step01 id, in Grafana.

Running Step01 Metrics

Run Step01 first and keep its run id.

Streamlit

Open Step01 Metrics — lemma vocab curves:

1. enter one or more Step01 run ids;
2. select final or milestones;
3. for milestones, set documents per milestone;
4. optionally set the order seed;
5. choose a new metrics run id and submit.

The generated form triggers DAG drt_step01_metrics.

Airflow: final

{
  "step_id": "step01_metrics",
  "run_id": "step01_metrics_final_20260721_001",
  "metrics.upstream_runs": "step01_20260721_001",
  "metrics.mode": "final"
}

Airflow: milestones

{
  "step_id": "step01_metrics",
  "run_id": "step01_metrics_growth_20260721_001",
  "metrics.upstream_runs": "step01_20260721_001",
  "metrics.mode": "milestones",
  "metrics.milestone_step": 100,
  "compute.corpus_growth.order_seed": "v1"
}

Multiple upstream runs

{
  "step_id": "step01_metrics",
  "run_id": "step01_metrics_merged_001",
  "metrics.upstream_runs": "step01_part_a,step01_part_b",
  "metrics.mode": "final"
}

Prerequisites

• selected Step01 runs completed successfully;
• postgres_app connection is configured;
• PostgreSQL migrations and corpus-growth procedures are deployed;
• source and output table schema versions match;
• milestone runs were produced after per-document evidence support was deployed;
• selected runs share compatible tenant lemma ids.

Troubleshooting

Symptom	Cause / action
metrics.upstream_runs is required	supply one or more Step01 run ids
no source rows	wrong run id or Step01 vocabulary write failed
no per-document evidence	rerun Step01 after deploying document-count support
milestone limit exceeded	increase metrics.milestone_step or max_milestones
dashboard is empty	select the metrics run id; verify PG datasource and generated dashboard sync
inconsistent merged cardinality	upstream runs used different tenant HMAC keys/semantics
missing procedure/table	run drtoller update, then verify PostgreSQL schema_version

Related

• Why these steps are OSS
• Step01 extraction
• Step01 data contract
• Dataset-metrics runtime
• Dataset metric catalog