Metrics and dashboard reference

In DRTML v4 the public section is observability. It lowers into the internal metrics/grafana plans used by emit and dashboard codegen.

Dataset-owned metrics can also be declared inline on outputs via measurements (+ optional publish / panel).

Full structure

observability:
  progress:
    unit: documents
    stage: run
    total:
      from: scan
      cap_param: runtime.max_docs
    timing:
      elapsed: {metric: step_elapsed_seconds}
      eta: {metric: step_eta_seconds, window: 1m}

  metrics:
    - name: step_documents_processed_total
      kind: counter
      labels: [run_id, step_id, stage]
      source: counter_tick

    - name: step_registry_cardinality
      kind: gauge
      labels: [run_id, step_id, stage, dataset]
      source: dataset_metric
      dataset: token_counts
      metric: cardinality

    - name: step_domain_seconds
      kind: histogram
      labels: [run_id, step_id, stage]
      source: domain_sec_per_doc
      unit: seconds
      buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 1.0]

  curves:
    - id: vocabulary_growth
      x: progress_current
      y: step_registry_cardinality
      title: Vocabulary growth

  dashboard:
    panels:
      - {id: throughput, type: timeseries, title: Throughput, metric: step_documents_rate}
      - {id: cpu, type: stat, title: CPU, metric: step_cpu_percent, stage: run}
      - {id: ram, type: stat, title: RAM, metric: step_ram_used_mib, stage: run}
      - {id: growth, type: curve, curve: vocabulary_growth}

  grafana:
    uid: normalize-text-v1

Prometheus metric declarations

kind is counter, gauge, or histogram. Histogram-only fields are unit (seconds, bytes, count) and buckets.

Allowed labels are bounded to avoid cardinality explosions: run_id, step_id, stage, artifact, worker, k, kind, pair_source, unit, window, dataset, phase. Document ids, paths, filenames, raw text, and UUID labels are rejected.

Metric sources

Current source catalog:

• progress: progress_current, progress_persisted, progress_rate, progress_elapsed
• registry: registry_cardinality, registry_saturation_homogeneity, registry_saturation_new
• execution: counter_tick, domain_sec_per_doc, input_shards_closed
• dataset: dataset_metric
• processor output: processor_stat, processor_stat_ratio
• mapping build: mapping_stat, mapping_stat_ratio, mapping_per_source, mapping_elapsed, mapping_counter

dataset_metric requires dataset + metric alias. processor_stat requires stat. processor_stat_ratio requires numerator_stat and denominator_stat. mapping_stat and mapping_stat_ratio use the same stat fields over mapping build stats. The remaining mapping sources emit per-source entities, elapsed time, or mapping counters from the standard mapping result.

Dataset metric definitions

Metric computation is declared on the output dataset, while Prometheus naming is declared under metrics.prometheus:

datasets:
  token_counts:
    metric_defs:
      retained:
        method: count_ge_threshold
        column: token_id
        unit_column: document_id
        threshold_param: compute.embed_min_token_count
        cadence: final
        accuracy: exact

Common fields:

• method: registered algorithm method.
• column: entity/key column.
• unit_column: progress axis (does not have to be an output column).
• sample_every_n, window_points: sampling/growth controls for compatible methods.
• cadence: live, checkpoint, or final.
• accuracy: exact or bounded_approx.
• threshold_param: merged param used by threshold methods.
• related: cross-dataset relation for coverage/consistency methods.

Bounded approximate checkpoint metrics require an appropriate metric_profile sample size.

Dashboard views

The code generator supports:

• Prometheus timeseries, stat, piechart
• XY curve panels backed by metrics.curves
• PostgreSQL timeseries, stat, and xychart panels through whitelisted query builders

Panel grid, units, mappings and legends are declared in the view item. Builtin status/elapsed panels are inserted by framework code. Every step dashboard contract must include CPU and RAM panels.

Generated dashboard behavior

generate_dashboard_from_metrics_views() creates:

• stable UID from grafana.uid (fallback <step_id>_auto);
• run_id and step_id template variables (run_id is scoped to the selected step);
• active-run PromQL wrapping;
• panel JSON for supported types;
• default refresh/time range.

PostgreSQL corpus-growth xychart panels use corpus_fraction as X. SQL template variables are formatted as ${name:sqlstring} rather than interpolated as identifiers.

Generation/sync is server-only. Agents do not run sync and generated JSON is not committed.