Grafana dashboard generation

Grafana dashboards are generated artifacts. The source of truth is each step’s metrics.prometheus, metrics.curves, metrics.views, dataset outputs, and grafana.uid.

Generation pipeline

step manifest + included metrics fragment
  → validate metric names/sources/labels/views
  → generate_dashboard_from_metrics_views
  → add builtin panels and template variables
  → build Prometheus / curve / PostgreSQL panels
  → server sync script provisions Grafana

The repository does not contain hand-maintained or generated dashboard JSON.

Panel dispatch

panel_builders.py supports:

• Prometheus: timeseries, stat, piechart
• curve: curve
• PostgreSQL datasource: timeseries, stat

Unsupported type/datasource combinations are skipped. Focused modules own series, curve, PostgreSQL, field config, grid layout and PromQL.

Note: the generator and tests support piechart, but the older v2 panel validator allowlist is narrower. Prefer panels that both generate and validate cleanly for the target DRTML version, or extend validator + generator together.

Template variables

Every generated dashboard receives run_id and step_id variables from a probe metric. The current step id is preselected. Queries are wrapped to respect active run selection rather than displaying stale series.

Example

metrics:
  curves:
    - id: growth
      x: step_documents_processed_total
      y: step_vocabulary_size
      title: Vocabulary growth

  views:
    panels:
      - id: progress
        type: timeseries
        title: Documents
        metric: step_documents_processed_total
        stage: run

      - id: result_ratio
        type: piechart
        title: Results
        metric: step_result_total
        legend: "{{kind}}"

      - id: growth_curve
        type: curve
        curve: growth

      - id: pg_registry
        type: stat
        datasource: postgres
        dataset: token_registry
        metric: cardinality

      - id: cpu_stat
        type: stat
        metric: step_cpu_percent
        stage: run

      - id: ram_stat
        type: stat
        metric: step_ram_used_mib
        stage: run

grafana:
  uid: normalize-v1

CPU and RAM panels are mandatory for each step dashboard contract. Framework may also insert builtin status/elapsed panels.

PostgreSQL panels

PostgreSQL panels do not accept arbitrary SQL from step manifests. The generator selects whitelisted query builders/stored functions using declared dataset and metric metadata. This preserves schema and security boundaries.

Server-only generation

Dashboard generation/sync runs during server deployment/update through the platform sync script. Do not run it from agents, step execution, UI, or local processors. Do not commit *_generated_dashboard.json or place JSON in static Grafana directories.

Extending panels

Add a DRTML view schema, focused builder, dispatcher entry, field/grid behavior, and JSON snapshot tests. PromQL construction stays centralized. Deployment sync remains unchanged.