Add an execution mode or pattern

A step uses an existing mode (partition, mapping, …). This cookbook is for extending the framework with a new execution path.

Prefer, in order:

Reuse partition / mapping / reduce / … if the lifecycle fits.
run.type: custom + run.entry for product-specific one-offs (no new builtin type).
New framework-wide run.type / pattern only when many steps will share the lifecycle.

Mental model

flowchart TD
  YAML["Public DRTML v4 run.type"] --> LOW["source_v4.lower_run"]
  LOW --> INT["Internal manifest: pattern and/or entry"]
  INT --> VAL["manifest_validate"]
  VAL --> DISP["run_from_manifest"]
  DISP --> REG{"pattern registered?"}
  REG -->|yes| PR["pattern runner"]
  REG -->|no| ENT["import_symbol execution.entry"]
  PR --> ORCH["processing mode: thin orchestration"]
  ENT --> ORCH
  ORCH --> OWN["storage / algorithms / mapping"]

Pattern vs entry

Kind	When	Dispatch
Pattern	Framework owns a typed DRTML block (`mapping`, `reduce_merge`, …) and a shared runner	`register_pattern_runner`
Entry-only	Lifecycle is special but invoked like partition/embed_train	builtin `_BUILTIN_ENTRIES[run_type]` → `execution.entry`
Custom	One product step, no shared type	`run.type: custom` + `run.entry` — no framework type change

Do not grow if branches inside run_from_manifest.

1. Decide the public surface

Example: a new builtin type run.type: distill.

drtml: 4
step: my_distill
run:
  type: distill
  # type-specific fields…

If it is not framework-wide, stop here and use:

run:
  type: custom
  entry: my_pkg.distill.run:run_distill

2. Lowering (`source_v4`)

Add the type to _RUN_TYPES in drtml/source_v4/common.py.
Add _BUILTIN_ENTRIES["distill"] = "drtoller.framework.processing.distill.run:run_distill" (or pattern entry).
Extend lower_run in drtml/source_v4/run.py:
- either set out["pattern"] = "distill" + typed section;
- or only set execution.entry / mode (like embed_train / partition).

Public YAML must not require authors to write internal pattern: / execution.entry by hand for builtins.

3. Models + validation

Pydantic model for the typed section under drtml/models/ (or models/patterns.py if it is a pattern block).
Cross-field checks in manifest_validate/ (datasets refs, required columns, forbidden flush options, …).
Compile to an immutable *Plan consumed by the runner — runtime must not re-parse YAML.

4. Thin runner under `processing/`

def run_distill(ctx, *, manifest_path):
    manifest = load_manifest_path(manifest_path)
    params = merged_params_from_manifest(manifest, ctx.params)
    plan = compile_…(manifest, merged_params=params)
    # open StorageSession if needed
    # call algorithms / mapping / storage helpers
    # emit metrics via framework hooks — not prometheus_client in the runner body ad-hoc
    return outcome

Rules:

Orchestration only: lifecycle, session open/close, call into owning packages.
I/O → storage/ / db/; math → algorithms/; merge → mapping/.
No domain step imports; no step_id branches.
Shared DTOs across layers → framework/contracts/.

5. Register for dispatch

Pattern path (prefer Extension SDK):

from drtoller.framework.ext import register_pattern_runner
from drtoller.framework.processing.distill.run import run_distill

register_pattern_runner("distill", run_distill)

Register builtins inside runtime/pattern_runners.ensure_builtin_runners() (or a plugin loaded by load_plugins), not by editing the if tree in run_from_manifest. processing.dispatch only re-exports the register APIs.

Entry-only path: _BUILTIN_ENTRIES is enough when dispatch falls through to import_symbol(execution.entry).

Optional legacy compatibility: register_mode_runner("distill", run_distill).

6. Params, UI, metrics

Knobs → params_defaults with an allowed namespace (runtime.*, compute.*, …).
UI fields must change merged params that the runner actually reads.
Prometheus / Grafana → fragments/metrics.drtml + framework emit; include cpu_stat and ram_stat panels when adding views.

7. Tests

lower: run.type → expected pattern / execution.entry;
validate: missing section / bad dataset refs fail loudly;
dispatch: registered pattern is selected; unknown pattern without entry fails;
runner smoke: fake storage / small fixture;
layer boundaries: test_layer_import_boundaries.py still green.

8. Docs checklist

Modes reference
How a run executes
Mode page under processing/ if the lifecycle is non-trivial
framework/FILES.md