Extending DRTML

DRTML extension is a vertical change. A YAML field is not a feature until it has a model/parser, validation, a compiled representation, a runtime consumer, tests, and documentation.

Choose the owning layer first

• Manifest syntax/model/validation → framework/drtml/.
• Runtime data I/O → framework/storage/ or framework/db/<engine>/.
• Execution lifecycle → framework/processing/<mode>/.
• Pure math → framework/algorithms/.
• UI/Airflow/Ray connection → framework/integration/.
• Metric emit/dashboard codegen → framework/telemetry/.
• Shared DTO across those layers → framework/contracts/.

Do not put runtime behavior in the parser and do not teach domain steps to interpret framework sections.

Add a field to an existing typed section

Example: a new feed.read option.

1. Add it to the smallest Pydantic model under drtml/models/<section>/.
2. Use extra="forbid" unless the block is an explicit plugin envelope.
3. Add field-level and cross-field validators.
4. Propagate it into the frozen compiled plan.
5. Consume it in the owning runtime module.
6. Add parse/validation, compile, and runtime tests.
7. Add a DRTML example and update the relevant reference page.

Never read raw manifest __pydantic_extra__ in unrelated runtime code when a typed plan is appropriate.

Add a new section

1. Create a model package under drtml/models/<section>/ (or a dedicated drtml/<section>/ parser for storage-style sections).
2. Add the root field to StepManifest if it is universally typed; use versioned validation for compatibility.
3. Add parse_<section>_plan and an immutable plan model.
4. Wire it once in compile_step_manifest.
5. Pass the plan to runtime; runtime must not reload or reparse YAML.
6. Add it to FILES.md and docs.

Do not create import-only shims such as drtml/new_section.py that only re-export models.new_section.

Add a new param namespace

1. Add the namespace to ALLOWED_PARAM_NAMESPACES.
2. Decide and document who may read it.
3. Add params_defaults to manifests that require it.
4. If exposed in UI, ensure a runtime/domain reader consumes it.
5. Never use domain.* for framework orchestration.

Add a dataset backend

1. Add the backend literal/whitelist in DRTML storage parsing and StoragePlan.
2. Add backend-specific inline block model/parser and compatibility validation.
3. Implement transport in db/<engine>/ when an external system is involved.
4. Implement DatasetWriteBackend / DatasetReadBackend adapters.
5. Register adapters in backend_builtins or plugin initialization.
6. Keep StorageSession free of if backend == branches.
7. Add fake contract tests and integration tests.

See New dataset backend.

Add a new execution pattern or mode

1. Define its typed DRTML block and plan.
2. Implement a thin orchestration runner under processing/<mode>/.
3. Put algorithms and I/O in their owning packages.
4. Register with register_pattern_runner or register_mode_runner.
5. Prefer execution.entry for product-specific execution that does not deserve a framework-wide pattern.

Do not add another branch to run_from_manifest.

Add a metric source or method

A metric method computes a dataset value: add MetricMethod through register_method, plus backend capabilities/SQL when needed.

A metric source maps runtime state to Prometheus: extend the metrics model source catalog, emit plan, source emitter, tests, and dashboard behavior if applicable.

Keep computation in algorithms/DB, orchestration in processing, and Prometheus emission in telemetry.

Compatibility policy

• Additive optional fields are safe within a DRTML version.
• Renames/removals need explicit validation errors and migration text (not silent aliases forever).
• Defaults that change run behavior belong in each step’s DRTML, not hidden in generic framework code.
• A parser accepting a field that runtime ignores is a bug.