Manifest reference

A DRTML v4 manifest is a validated YAML contract. The public loader lowers it into an internal StepManifest used by compile and runtime.

Minimal partition step

drtml: 4
step: normalize_text
run:
  type: partition
  processor:
    batch: acme_steps.normalize.processor:process_batch
storage:
  default: artifacts
  stores:
    artifacts: {backend: local, root: /tmp/drtoller-data}
inputs:
  documents:
    schema: {document_id: string, text: string}
    run: mapping.upstream_runs
    feed: {batch: 64}
outputs:
  normalized_documents:
    schema: {document_id: string, text: string}
    from: documents
params:
  runtime:
    metrics_enabled: true
    document_batch_size: 64

orchestration.dag_id defaults to drt_<step> when omitted.

Top-level fields

Identity and loading

• drtml: must be 4.
• step: stable step identifier. It must match RunContext.step_id.
• schema: optional schema identifier.
• include: list of fragment paths or globs. No merge: target.

Run

• run.type: partition | mapping | reduce | dataset_metrics | evaluation | embed_train | custom.
• Builtin types use a framework entry automatically.
• run.type: custom requires run.entry (module:callable).
• Partition fields: processor, optional derive, lookup_bindings, infer.
• Mapping fields: entities, registry, crosswalk, ids.
• Reduce / metrics / evaluation / embed_train: type-specific job blocks (same semantics as before, under run).

Storage and data

• storage.stores: named profiles (local / s3 / postgres / qdrant / neo4j).
• storage.default: default object store for datasets without an explicit store.
• inputs / outputs: dataset declarations. Role is implied by the section.
• Dataset fields use schema, run, key, group, aggregate, identity, physical, measurements.

Params and integrations

• params: nested namespaces (runtime, storage, domain, …). Flattened to namespace.key for runtime.
• orchestration: DAG identity, connections, artifacts root, env fallbacks, params hook, XCom.
• observability: progress, Prometheus metrics, curves, dashboard panels, grafana uid.
• ui: Streamlit form schema (bind preferred over legacy param).
• lookups: crosswalk indexes loaded before the processor runs.

Includes

include:
  - fragments/params.drtml
  - fragments/ui.drtml
  - fragments/metrics.drtml
  - fragments/datasets/*.drtml

Fragments are deep-merged in order. The entry manifest wins over included defaults. See Fragments and include.

Compile outputs

compile_step_manifest() converts the lowered manifest and merged params into runtime plans:

• StoragePlan
• StepFeedPlan
• MappingPlan
• ReduceMergePlan
• DatasetMetricsPlan
• EvaluationPlan
• ComputePlan
• mode-specific plans such as embed training

Compile produces plans only. It does not open datasets or perform runtime I/O.

Strictness

Public v4 rejects unknown root fields. Typed internal blocks generally use extra="forbid". See Extending DRTML.