Dataset metrics

Declare in DRTML

datasets:
  vocab:
    kind: registry
    backend: postgres   # or parquet
    metric_defs:
      cardinality: { method: count_distinct, column: lemma_token_id }
      homogeneity: { method: assignment_homogeneity, column: lemma_token_id }
      new_in_sample: { method: assignment_new_count, column: lemma_token_id }

metrics:
  prometheus:
    - name: unique_entities
      kind: gauge
      labels: [run_id, step_id, stage]
      source: dataset_metric
      dataset: vocab
      metric: cardinality

Registry

algorithms/dataset_metrics/types.py defines the method contract; registration and lookup live in algorithms/dataset_metrics/registry.py:

@dataclass(frozen=True)
class MetricMethod:
    name: str
    python: Callable | None
    postgres_proc_metric: str | None
    qdrant_metric: str | None = None
    postgres_direct_final: bool = False
    needs_novelty: bool = False
    # …

API: resolve_method, register_method, known_methods.

A new generic dataset metric requires matching Python and PostgreSQL implementations: one registry entry plus the corresponding family-procedure branch under platform/stack/postgres-init/. No if method == copy-paste in emit.

postgres_direct_final=True allows an exact final job to query a PostgreSQL source directly, without run-scoped staging. The catalog rejects a direct-final method missing either implementation.

Shared formula kernels

Dataset-metric adapters keep their existing callable contract, but reusable mathematics lives in neutral packages:

• grouped Jaccard, weighted overlap and Jensen–Shannon formulas delegate to algorithms.profiles;
• graph topology formulas use neutral graph inputs rather than domain column names;
• entropy and information-gain kernels live in algorithms.information;
• generic offline ratios live in algorithms.evaluation.bindings.

The dataset-metric layer binds rows and backend capabilities. It must not duplicate profile, graph or information-theory implementations.

Backend strategy

Backend	How it is computed
parquet / RAM	MetricMethod.python
postgres	generic proc pipeline.drt_proc_dataset_metric_snapshot + postgres_proc_metric (naming)
qdrant	dedicated geometry/snapshot path today; registry qdrant_metric is a capability marker, not yet a generic dispatcher

Prometheus does not know the source — it receives SaturationSlice / gauge values from the orchestrator.

Cookbook

New metric method

Execution modes and capability validation

Each MetricMethod advertises supported modes: live_exact, checkpoint_exact, checkpoint_approx, final_exact, and/or milestone_exact. Compile rejects an unsupported cadence/accuracy combination. Registry flags also declare whether a method needs novelty samples, growth state, a threshold param, sampling knobs, or a related dataset.

Builtins are assembled from semantic families (core_distribution, assignment_growth, graph_group, vector_cross_dataset). The public extension API remains register_method; family modules are catalog organization, not a second registry.

Backend capability is explicit: python for in-memory/parquet evaluation, postgres_proc_metric for generic PostgreSQL snapshot procedures, and qdrant_metric as a declared vector-store capability. A method may support only one backend path. Filling qdrant_metric still requires a concrete runtime consumer.

Cross-dataset relation

metric_defs:
  retained_coverage:
    method: related_retained_mass_ratio
    column: token_id
    threshold_param: compute.embed_min_token_count
    cadence: final
    accuracy: exact
    related:
      dataset: trained_vocabulary
      group_columns: [language]
      left_endpoint_columns: [head_token_id, child_token_id]
      right_key_column: token_id
      related_column: language

The related dataset must be a declared PostgreSQL output; referenced columns are validated across both datasets.