Dataset metrics

Declare in DRTML

datasets:
  vocab:
    kind: registry
    backend: postgres   # or parquet
    metric_defs:
      cardinality: { method: count_distinct, column: lemma_token_id }
      homogeneity: { method: assignment_homogeneity, column: lemma_token_id }
      new_in_sample: { method: assignment_new_count, column: lemma_token_id }

metrics:
  prometheus:
    - name: unique_entities
      kind: gauge
      labels: [run_id, step_id, stage]
      source: dataset_metric
      dataset: vocab
      metric: cardinality

Registry

algorithms/dataset_metric_registry.py:

@dataclass(frozen=True)
class MetricMethod:
    name: str
    python: Callable | None
    postgres_proc_metric: str | None
    qdrant_metric: str | None = None
    needs_novelty: bool = False
    # …

API: resolve_method, register_method, known_methods.

A new method = one registration (+ optional SQL fragment under platform/stack/postgres-init/).
No if method == copy-paste in emit.

Backend strategy

Backend	How it is computed
parquet / RAM	MetricMethod.python
postgres	generic proc pipeline.dataset_metric_snapshot + postgres_proc_metric
qdrant	dedicated geometry/snapshot path today; registry qdrant_metric is a capability marker, not yet a generic dispatcher

Prometheus does not know the source — it receives SaturationSlice / gauge values from the orchestrator.

Cookbook

New metric method

Execution modes and capability validation

Each MetricMethod advertises supported modes: live_exact, checkpoint_exact, checkpoint_approx, and/or final_exact. Compile rejects an unsupported cadence/accuracy combination. Registry flags also declare whether a method needs novelty samples, growth state, a threshold param, sampling knobs, or a related dataset.

Backend capability is explicit: python for in-memory/parquet evaluation, postgres_proc_metric for generic PostgreSQL snapshot procedures, and qdrant_metric as a declared vector-store capability. A method may support only one backend path. Filling qdrant_metric still requires a concrete runtime consumer.

Cross-dataset relation

metric_defs:
  retained_coverage:
    method: related_retained_mass_ratio
    column: token_id
    threshold_param: compute.embed_min_token_count
    cadence: final
    accuracy: exact
    related:
      dataset: trained_vocabulary
      group_columns: [language]
      left_endpoint_columns: [head_token_id, child_token_id]
      right_key_column: token_id
      related_column: language

The related dataset must be a declared PostgreSQL output; referenced columns are validated across both datasets.