Extending integrations

Integrations translate external systems into framework contracts. They do not contain domain transforms.

Generic extension checklist

1. Start from a declarative DRTML need, not a particular step id.
2. Add typed/validated configuration where practical.
3. Convert external values into RunContext, namespaced params, or a neutral contract DTO.
4. Call existing dispatch/storage/telemetry APIs.
5. Keep external SDK imports lazy and inside the integration/transport module.
6. Add tests with fake clients or fake Streamlit/Airflow context.
7. Document deployment/env requirements.

Add a Streamlit field

• Generic scalar field: extend _render_field.
• Reusable multi-field capability: add a focused renderer module and invoke it from DrtmlStepLauncher.
• Output must be a params overlay only.
• Add UI validation and a runtime consumer.
• Never add if step_id == or import a step package.

Add an Airflow connection type

1. Add a generic connection adapter in integration/airflow.
2. Match connection ids from orchestrator.connections.
3. Translate Airflow connection data into framework env/client configuration.
4. Do not expose Airflow objects to processing or domain code.
5. Add tests for absent, malformed, and valid connections.

Add a Ray transport capability

Connection/init remains in integration/ray. Submit/wait/object-ref logic belongs in the owning processing transport backend. Worker payloads must be picklable and contain manifest path + params rather than compiled sessions/clients.

If behavior is a computation algorithm, add it under algorithms/; if it is distributed dispatch, add a transport backend. Do not make steps import Ray.

Add Prometheus instrumentation

Declare the metric in DRTML, extend the emit plan/source adapter, and emit through RunMetrics. Processors may return named stats but must not call prometheus_client.

Add a Grafana panel type

1. Define the DRTML view shape.
2. Add a focused builder under telemetry/dashboards.
3. Dispatch it from panel_builders.py.
4. Keep PromQL in promql.py, grid math in panel_grid.py, and field config in field_config.py.
5. Add JSON contract tests without syncing Grafana.

Dashboard sync remains a server deployment operation.