Core concepts
Declarative step contract: YAML *.drtml plus include fragments (ui, metrics, params).
Parse / validate / compile live only in framework/drtml/. Steps must not parse YAML.
RunContext
Section titled “RunContext”One-run context: run_id, step_id, params, artifacts_root, optional metrics.
Created by integration (UI / Airflow / local), passed to dispatch.
ProcessResult
Section titled “ProcessResult”Domain processor output contract: logical row-dicts + target dataset.
I/O is done by the framework (storage/run + backend adapter).
Mode / pattern
Section titled “Mode / pattern”| Concept | Where | Examples |
|---|---|---|
pattern (v3) |
top-level DRTML | mapping, reduce_merge |
execution.mode (legacy) |
execution | partition_loop, mapping, … |
execution.entry |
v3 fallback | dotted path module:callable |
Runner registry: processing/dispatch.py.
Dataset backend
Section titled “Dataset backend”Field datasets.*.backend: parquet | postgres | qdrant | neo4j.
One dataset — one sink. Dispatch via storage/dataset/backend_registry.py.
Feed / WorkUnit
Section titled “Feed / WorkUnit”datasets.*.feed → StepFeedPlan → runtime WorkUnit in storage/feed/.
Partition strategies: file / rows / corpus_scan — semantics are fixed; do not break them.
Contracts
Section titled “Contracts”framework/contracts/ — neutral DTOs (SaturationSlice, WorkerMetricDelta, …).
Imported by storage/processing/telemetry; no reverse dependencies.
Param namespaces
Section titled “Param namespaces”Flat keys namespace.key: domain.* only in the step; runtime.* / storage.* / flush.* / compute.* / … — framework by table.
See Namespaces.
Metrics plan
Section titled “Metrics plan”fragments/metrics.drtml → emit plan. Dataset metrics: metric_defs on the dataset + source: dataset_metric in prometheus.
Methods live in algorithms/dataset_metric_registry.py.