Core concepts

A pipeline step is not “a script that opens parquet”. Framework always runs the same loop in spirit:

DRTML execution contract  →  feed batches of rows  →  domain transform  →  framework write

Integrations (Streamlit, Airflow, local runner) only build a run context and ask dispatch to execute the step’s manifest. The step author writes domain logic; the Framework owns I/O, checkpoints, and metrics emit. DRTML is a machine-readable execution contract — see DRTML overview and Compare.

The story in four beats

1. Contract (DRTML) — declarative workload language (*.drtml + fragments) declares inputs, outputs, params, and how the run should execute. Only framework/drtml/ parses it. Public surface is DRTML v4 (drtml: 4); the loader lowers it into internal plans.
2. Feed — for transform modes, storage turns feed plans into work units (bounded batches of logical rows, optionally with attach data). The processor never opens part files itself.
3. Transform — domain code returns a ProcessResult: logical row-dicts keyed by logical output names. outputs.*.from maps those names to physical datasets.
4. Write — StorageSession (and friends) persist rows to one backend per dataset (parquet, postgres, qdrant, …), rotate parts, flush, emit metrics.

Special modes (mapping, reduce, embed train, evaluation, …) still follow the same idea: DRTML describes the job; framework owns lifecycle; domain owns policy where the rules say so.

Names you will meet

Name	Plain meaning
RunContext	This run’s identity: run_id, step_id, merged params, artifacts root, optional metrics handle. Built by UI/Airflow/local, passed into dispatch.
run.type	Which built-in job shape to run: partition, mapping, reduce, dataset_metrics, evaluation, embed_train, custom. See Modes.
Dataset backend	Where a dataset lives. Declared via storage.stores (+ optional physical). One dataset → one sink — no dual-write.
Feed / WorkUnit	How rows are sliced for the processor (inputs.*.feed / attach → StepFeedPlan). Partition semantics are fixed; do not “simplify” them casually.
Contracts	Layer-neutral DTOs in framework/contracts/ (saturation slices, probe rows, worker metric deltas, …) so storage/processing/telemetry do not import each other the wrong way.
Param namespaces	Nested DRTML params flatten to namespace.key. domain.* is for the step only; runtime.*, storage.*, compute.*, … are read by framework layers. Details: Namespaces.

Boundaries

Who may open files, emit Prometheus, or import whom: Principles. When in doubt: the step transforms data it was given; the framework moves data and runs the platform.