Prometheus runtime

Contract flow

fragments/metrics.drtml
  → StepManifest.metrics
  → metrics plan / RunMetrics
  → partition-loop, mapping, embed-train lifecycle events
  → Prometheus collectors
  → Grafana PromQL generated from metrics.views

Domain processors return rows and integer stats. They never import prometheus_client or choose metric names.

Lifecycle

Framework sets up metrics at run start, records progress/current/persisted/rate/elapsed state, receives dataset snapshots and processor stats, emits process CPU/RAM, and finalizes collectors at run end.

Builtins include:

• step_cpu_percent
• step_ram_used_mib
• run status/probe and elapsed metrics
• progress/rate/ETA

DRTML sources

See Metrics reference for the full source catalog. Sources bind generic lifecycle data to declared names. Dataset metrics bind (dataset, metric alias); processor stats bind returned stat keys.

Multiprocess/Ray

Workers emit deltas/snapshots to the driver. The driver is the source of truth for Prometheus. Multi-worker registry homogeneity is not inferred by independently scraping workers.

Cardinality safety

Labels are validated against a small allowlist. Never use document id, file path, raw text, UUID, or another unbounded value as a label. Put such values in datasets/logs.

Adding a metric

First decide whether it is:

• a new name using an existing source (DRTML only);
• a new dataset formula (algorithms/dataset_metric_registry);
• a new lifecycle source (processing contract + telemetry emitter);
• a dashboard-only derivation (PromQL/view).

Keep source collection separate from metric naming and dashboard layout.