Execution and orchestration

Run types

run.type	Builtin entry	Typical section
partition	partition loop	processor, optional derive / infer
mapping	mapping build	entities, registry, crosswalk, ids
reduce	reduce merge	jobs[] or inputs/outputs
dataset_metrics	offline metrics	jobs[], executor
evaluation	offline evaluation	jobs[]
embed_train	SGNS train	train_pairs, vocab_counts
custom	required run.entry	anything not covered above

Builtin types do not take a custom run.entry. Thin step hooks still exist for packaging and Airflow discovery; they usually call the framework runner.

Partition

run:
  type: partition
  processor:
    batch: pkg.processor:process_batch
    single: pkg.processor:process_one   # optional; defaults to batch
    init: pkg.processor:init            # optional
  derive:
    lemma_vocab:
      from: sentence.lemma_token_id
      operation: count_values
      count: token_count

derive replaces explicit registry_flush / registry_ingest. Omit it for plain transforms that do not maintain a registry.

Mapping

run:
  type: mapping
  entities: lemma_vocab_run
  registry: lemma_registry
  crosswalk: lemma_run_mapping
  ids:
    registry: lemma_id
    crosswalk: lemma_id

Orchestration

orchestrator:
  artifacts_root: env:DRTOLLER_ARTIFACTS_ROOT
  connections: [minio_s3, postgres_app]
  retries: 0
  retry_delay_s: 60
  xcom:
    - key: storage_manifests
      from: outputs.storage_manifests

Defaults when omitted:

• dag_id → drt_<step>
• step_id → <step>
• retries → 0, retry_delay_s → 60 (opt in to Airflow retries; default does not retry OOM inflate)

XCom from paths resolve against the run result dict.