Grafana dashboard sync

Pipeline Grafana dashboards are generated artifacts. Source of truth is each step manifest: metrics.prometheus, metrics.curves, metrics.views, dataset outputs, and grafana.uid.

The repository does not commit *_generated_dashboard.json. Generation runs only on the server during stack start and update.

End-to-end flow

config/steps_roots.txt
  → iter step *.drtml manifests (edge + ops roots)
  → validate metrics / views / grafana.uid
  → generate_dashboard_from_metrics_views (framework/telemetry/dashboards/)
  → write <step_id>_generated_dashboard.json
       under drtoller.edge/static/grafana/ or drtoller.ops/static/grafana/
  → Grafana file provider (pipeline-steps.yaml) picks up JSON
  → restart grafana container

Where it runs

Trigger	Script
`drtoller up`	`stack/bin/servstart.sh` → `sync-grafana-dashboards.sh`
`drtoller update`	`stack/bin/drtupdate.sh` → same sync → `docker compose restart grafana`

sync-grafana-dashboards.sh:

Ensures airflow-worker is up (has framework + repo mounts).
Runs inside worker:

drtoller.telemetry.dashboards --all

Fixes ownership on */static/grafana dirs for the deploy user.

Output locations

Roots come from DRTOLLER_STEPS_ROOTS_FILE (default: stack/config/steps_roots.txt):

/drtoller/drtoller.edge/advanced/steps  →  drtoller.edge/static/grafana/
/drtoller/drtoller.ops/steps            →  drtoller.ops/static/grafana/

Grafana mounts those trees at /drtoller.edge/static and /drtoller.ops/static and loads JSON via grafana/provisioning/dashboards/pipeline-steps.yaml (folder pipeline, refresh every 30s).

DRTML contract (summary)

Declare panels in fragments/metrics.drtml (or inline metrics.views). Every step dashboard must include CPU and RAM stat panels.

metrics:
  views:
    panels:
      - id: cpu_stat
        type: stat
        metric: step_cpu_percent
        stage: run
      - id: ram_stat
        type: stat
        metric: step_ram_used_mib
        stage: run

grafana:
  uid: my-step-v1

Panel types supported by codegen: Prometheus timeseries, stat, piechart; curve panels; Postgres timeseries / stat (whitelisted query builders only — no arbitrary SQL in DRTML).

Full panel schema and PromQL rules: Grafana codegen and Metrics in DRTML.

Manual sync (server only)

On the deployment host, from stack dir with .env loaded:

./bin/sync-grafana-dashboards.sh
docker compose restart grafana

Single root / dest (advanced):

drtoller.telemetry.dashboards \
  --steps-root /path/to/steps \
  --dest /path/to/static/grafana

What not to do

Do not run dashboard sync from step code, Streamlit, Airflow tasks, or local dev agents as part of a run.
Do not commit *_generated_dashboard.json or hand-edit generated files in static/grafana/.
Do not put pipeline JSON under platform/grafana/dashboards/ — that directory is for Ray/static assets only.

Runtime metric emit (Prometheus scrape) is separate from dashboard codegen — see Prometheus.

After changing metrics in DRTML

Deploy code (git pull / drtoller update).
Sync runs automatically in drtoller update; or run sync-grafana-dashboards.sh manually.
Open Grafana → folder pipeline → dashboard for your grafana.uid.
Template variables run_id and step_id are injected by the generator; select active run to avoid stale series.

Postgres table DDL uses the same “scan steps roots → codegen → apply” pattern via sync-postgres-ddl.sh inside migrate-postgres.sh on drtoller up / drtoller update. Physical names must be drt_tbl_* / drt_proc_* and listed in pg_object_inventory.txt — PostgreSQL naming.