Streamlit integration

Streamlit is a replaceable launch shell. The durable contract is ui.streamlit in DRTML plus the namespaced params overlay it produces.

End-to-end flow

step roots
  → discover and validate manifests
  → keep manifests with ui.streamlit.enabled
  → DrtmlStepLauncher
  → generated widgets from params_defaults
  → clean user overlay
  → build_trigger_conf(step_id, run_id, artifacts_root, overlay)
  → Airflow REST trigger

No step package contains Streamlit widgets or a launcher class.

Discovery and registration

integration/ui/registry.py loads manifests from configured step roots and registers generic launchers. has_streamlit_ui() accepts a non-empty block unless enabled: false.

Deployment should configure step roots explicitly. Legacy repository-specific discovery is only a development fallback.

Form generation

DrtmlStepLauncher.render_form() performs these stages:

1. render explicit fields in declaration order;
2. add IO controls when outputs exist;
3. add per-dataset flush controls when requested;
4. add runtime controls;
5. add partition parallelism controls;
6. add compute/embed-train controls;
7. return one flat params overlay.

Defaults are resolved first from params_defaults, then field/group fallback. Empty optional values are omitted so they cannot erase defaults/env fallbacks.

See Streamlit form schema for every field and control.

Trigger payload

The UI sends only reserved run identity plus the user overlay:

{
  "step_id": "normalize",
  "run_id": "normalize_20260718_001",
  "artifacts_root": "/var/lib/drtoller/runs/normalize",
  "runtime.max_docs": 1000,
  "domain.lowercase": true
}

Workers reload the manifest and merge defaults. Sending the entire defaults dict from the UI would make deployed default changes ineffective and is intentionally avoided.

Airflow and local parity

The UI normally calls trigger_dag_run() through Airflow REST. Local execution uses the same manifest/default merge and env/connection hooks through integration.local.run.

Theme and navigation

theme.py / theme_css.py provide generic styling. links.py builds links to Grafana and other services. Theme configuration must remain product-generic; step behavior belongs in DRTML.

Adding capabilities

• Scalar widget → add a generic field type to _render_field.
• Reusable block → add a focused renderer and call it from DrtmlStepLauncher.
• New launch target → consume the same params overlay and run identity.

Always add validation and prove that a runtime/domain consumer reads the emitted param. Do not add step-id branches.