Manifest reference

A DRTML manifest is a validated YAML contract. The root model is StepManifest; known fields are typed and additional version-specific sections are validated during compile.

Minimal v3 manifest

drtml_version: 3
step_id: normalize_text
execution:
  entry: acme_steps.normalize.hooks:run
params_defaults:
  runtime.metrics_enabled: true
  runtime.document_batch_size: 64

Top-level fields

Identity and loading

• drtml_version: contract version. New steps should use 3.
• step_id: stable step identifier. It must match the RunContext.step_id.
• schema_id: optional schema identifier.
• include: paths to fragments merged before model validation.

Execution

• pattern: builtin structural pattern. Currently mapping or reduce_merge.
• execution.mode: legacy/builtin mode: partition_loop, mapping, reduce_merge, ingest, or embed_train.
• execution.entry: v3 dotted callable (module:callable) for execution not covered by a builtin pattern.
• execution.recipe: optional recipe identifier retained by the execution contract.
• runner: legacy callable fallback.

Dispatch order is documented in Dispatch.

Runtime configuration

• params_defaults: flat namespace.key → value defaults.
• storage: binding definitions and resume policy.
• datasets: canonical input/output dataset declarations.
• processing: processor imports, outputs, registry wiring, mapping and feed bindings.
• registry: registry plugin entries.
• mapping_read: crosswalk indexes loaded before the processor runs.
• compute.*: flat algorithm parameters compiled into ComputePlan.

Patterns

• mapping: entry, symbolic inputs and outputs for a mapping step.
• reduce_merge: entry, inputs, outputs, grouping and sum columns.
• embed_train: stateful training input/materialization contract (compiled by embed_train mode).

Integrations and observability

• ui.streamlit: generated form schema.
• metrics: progress, Prometheus declarations, curves, dashboard views, observations.
• grafana.uid: stable dashboard UID.
• orchestrator: DAG identity, connections, artifacts root, env fallbacks, params hook and XCom outputs.
• postgres: legacy shared table declarations; new PostgreSQL datasets use inline datasets.<id>.postgres.

Includes and merge behavior

include:
  - {path: fragments/params.drtml, merge: params_defaults}
  - {path: fragments/ui.drtml, merge: ui}
  - {path: fragments/metrics.drtml, merge: metrics}

Each include entry requires both path and merge. Paths are relative to the containing file. Keep the main manifest focused on execution and datasets; put defaults, forms and metric views in fragments. Includes merge before StepManifest validation and compile. See Fragments and include.

Compile outputs

compile_step_manifest() converts the validated manifest and merged params into runtime plans:

• StoragePlan
• StepFeedPlan
• MappingPlan
• ReduceMergePlan
• ComputePlan
• mode-specific plans such as embed training

Compile produces plans only. It does not open datasets or perform runtime I/O.

Strictness

Typed blocks generally use extra="forbid", so misspelled keys fail early. Root and selected integration blocks allow extra keys for versioned extensions, but those fields have no effect until a parser/compiler/runtime consumer is added. See Extending DRTML.