Step and framework

A step is an independently installable package: *.drtml + domain Python.
The framework owns infrastructure around the workload: feed, storage, metrics, UI, Airflow, parallelism backends (for example Ray).

Canon

Receive data → transform → return data through an explicit execution contract. Domain steps are isolated from infrastructure concerns: they do not open storage, emit Prometheus, or connect Ray/Airflow. Stateful modes (such as embedding training) still keep those infrastructure concerns in the Framework.

integration (UI / Airflow / local)
        → dispatch / mode runner
        → feed WorkUnit / ProcessorBatch     ← framework supplies data
        → domain batch_processor             ← step transforms
        → ProcessResult (logical rows)
        → StorageSession.write / checkpoint / Prometheus emit   ← framework

Ownership table

Concern	Step	Framework
DRTML parse / compile / validate	no	`drtml/`
Manifest path resolve in `hooks.py`	allowed (`manifest_path_adjacent`)	—
Read / write parquet, MinIO, Postgres, Qdrant	no	`storage/`, `db/`
Checkpoint / flush / manifests	no	storage + partition_loop glue
Prometheus emit / scrape	no	`telemetry/prometheus/`
Grafana dashboard JSON	no (declare views in DRTML)	codegen on server only
Streamlit widgets	no (`ui.drtml` only)	`integration/ui/`
Airflow DAG / Ray connect	no	`integration/` + parallel backends
Embed / cluster / metric formulas	call API	`algorithms/`
Choice of K, probe, early-stop	yes (domain policy)	no
In-memory registry / counters	yes (no disk I/O)	flush/checkpoint
spaCy / other domain NLP libs	yes (transform only)	no

Forbidden in the step

Any I/O: open, read_text, write_text, mkdir, shutil, Path for data
ctx.storage.read_* / ctx.storage.write (or private session APIs)
pyarrow, boto3 / minio, psycopg, qdrant_client, raw HTTP to MinIO/DB
prometheus_client, RunMetrics, scrape endpoints
Airflow API, import ray, multiprocessing shard routing
Parsing DRTML / YAML / JSON manifests (load_manifest_path, compile, …)
Homegrown embed/cluster math (sklearn / torch / manual SGNS) — use framework.algorithms
Custom Streamlit UI in hooks or domain modules
Truncating / wire-limiting wide lists (storage owns wire format)

Allowed in the step

Pure functions and stateful in-memory registry plugins (no disk write)
ProcessResult with logical row-dicts (framework.processing.process_result)
ProcessorBatch as the wire DTO at the batch_processor boundary
framework.mapping.remap / hash helpers
framework.algorithms.* — data in → result out; policy stays in the step
Domain libraries required by the transform (e.g. spaCy)
Thin hooks.py that only resolves manifest_path and calls run_partition_loop / mapping entry

Params: who may read what

Namespace	Step may read	Framework may read
`domain.*`	yes	passthrough only — must not interpret
`runtime.`, `storage.`, `flush.`, `compute.`, `mapping.`, `reduce.`, `metrics.*`, …	declare in DRTML / UI	yes, by layer contract

Framework must not hardcode domain column names (e.g. doc_id) as defaults. Cadence and boundaries come from DRTML (checkpoint, flush.boundary_columns, feed plan).

Full table: Namespaces.

Metrics and dashboards

Metric names and source: live in DRTML (metrics.prometheus / views).
Domain may accumulate stats for ProcessResult / saturation slices; emit is framework.
Every dashboard must include CPU and RAM panels (step_cpu_percent, step_ram_used_mib).
Generated Grafana JSON is a deploy-time artifact — see Grafana sync.

Algorithms vs policy

Layer	Owns
`framework/algorithms`	Formulas: embed, cluster, dataset metric methods
Step domain	Whether / how to call them (K, probes, arbiter, early-stop)
DRTML	Method names and knobs as `params` / `compute.*` — not policy recipes

Mapping and parallelism

Mapping merge algorithm is framework (mapping/build); the mapping step is DRTML + thin hooks.
Parallel workers (runtime.parallelism.*) are chosen in DRTML / UI; the step does not pick Ray vs processes in Python.
Domain batch_processor / registry API is the same for inprocess, processes, and ray.

Checklist (step change)

Does the processor open files or call storage? → move to framework feed/write path.
New metric? → DRTML + framework emit path, not prometheus_client in the step.
New UI field? → ui.drtml + params_defaults, no custom widget.
New math (embed/cluster/metric)? → framework/algorithms + registry; step only calls API.
Framework code mentions step01 / spaCy / domain.*? → leak — fix namespaces.

CI

pytest …/test_step_domain_boundaries.py -q