Framework layers

Inside the framework, packages form a directed acyclic dependency graph. Shared types go through contracts/. Extension points use registries, not growing if trees.

Canon edge

drtml → contracts / storage.plan → db | storage → processing → telemetry | integration

Edge	Meaning
`drtml → storage.plan`	Compile emits plans; DRTML does not perform I/O
`storage → db`	Transport only (Postgres / Qdrant)
`processing → storage`	Session / feed orchestration
`processing → db`	Glue only (poller, registry sync) — not step I/O
`telemetry ← *`	DTO in → emit out; dashboard SQL is static codegen

Forbidden reverse edges

From	Must not import	Why
`storage`	`processing`, `telemetry`, `integration`	I/O must not know runners or metrics
`db`	`processing`, `telemetry`	Transport stays pure
`telemetry`	`processing`	Emit must not pull orchestration
`contracts`	storage / processing / telemetry / db / integration	DTOs stay layer-neutral

Allowlist exception: some db/*/connection.py may read DSN helpers from env utilities — not a license to import processing or telemetry.

Package ownership

Package	Owns	Does not own
`drtml/`	Parse, validate, compile, plans	Runtime I/O, emit
`contracts/`	Neutral DTOs (saturation, worker delta, metric rows, …)	I/O, PromQL
`runtime/`	`RunContext`, `import_symbol`, pattern/mode runner registry	Mode algorithms
`ext/`	Extension SDK (`register_*`, `load_plugins`, run hooks)	Domain transforms
`storage/`	Session, feed, parquet, backend adapters, write buffers	Partition-loop policy, Prom emit
`db/`	Engine transport (PG/Qdrant), DDL codegen helpers	Step domain SQL, dual-write
`processing/`	Mode runners, partition_loop glue, dispatch, `job_map` / `job_session`	Parquet math, feed counting (→ storage)
`mapping/`	Build / read / remap	Step merge policy in Python
`algorithms/`	Embed, cluster, evaluation formulas + registries	K / probe decide policy
`processing/cluster`	Streaming/parallel fit & assign	Choosing K
`processing/probe`	Candidate loop, nested evaluate, publish glue	Winner selection
`telemetry/`	Prometheus emit, Grafana codegen	Runtime PG I/O, step names
`integration/`	Streamlit, Airflow, Ray connect, local runner	Domain transforms

`processing/` stays thin

partition_loop/ and parallel/ orchestrate; they must not grow into:

parquet read/count, write lanes, checkpoint policy → storage/
primary shard metrics / worker scope → storage/feed/shard_*.py
import-only shim modules (re-export without logic) — forbidden; fix imports to the canonical module

Telemetry split

Subsystem	When	Forbidden
`telemetry/prometheus/`	Runtime scrape / gauges	Grafana JSON
`telemetry/dashboards/`	Server sync only	`prometheus_client`, emit during run

Prometheus emit must not know whether a slice came from Postgres or RAM.

Registries (prefer over if-forests)

Catalog	Location
Dataset write/read backends	`storage/dataset/backend_registry.py`
Pattern / mode runners	`runtime/pattern_runners.py` (re-export via `processing/dispatch`, public via `ext`)
Parallel transports	`partition_loop/parallel/backend_registry.py`
Dataset metric methods	`algorithms/…/registry` (`register_method`)

StorageSession must not branch if backend == "postgres". New sink = adapter + register.

Where to put new logic

Task	Put it in
DRTML parse / validate	`drtml/`
Shared DTO across layers	`contracts/`
Dynamic `execution.entry` import	`runtime/import_utils.py`
Pattern/mode registration	`runtime/pattern_runners.py` / `ext/`
New `datasets.*.backend`	`backend_registry` + builtins (+ `db/` transport)
Feed / WorkUnit	`storage/feed/`
Write lanes / checkpoint policy	`storage/run/`
Mode orchestration	`processing/<mode>/`
Offline map fan-out (reduce / assembly / probe)	`processing/job_map.py` (+ `job_session.py`)
Streaming cluster fit/assign	`processing/cluster/` (+ `storage/run/vector_batch_iter`)
Probe loop / row assembly	`processing/probe/`, `processing/row_assembly/`
Feed multi-worker transport	`processing/partition_loop/parallel/`
Embed / cluster / metric math	`algorithms/`
Prom emit	`telemetry/prometheus/`
Grafana JSON	`telemetry/dashboards/` (server sync)
UI / Airflow / Ray connect	`integration/`

Framework must not (anti-leak)

Forbidden in drtoller/framework/ Python (except tests):

Imports of step packages (drtoller.edge, drtoller.ops) as runtime deps
if step_id == "step01" (or any concrete step branch)
Interpreting domain.* params
Domain column defaults (doc_id, …) in storage/feed without DRTML
Domain libraries (spaCy, sklearn) inside framework
Dual-write (parquet + Postgres for the same dataset)
Runtime CREATE TABLE / ALTER (DDL is deploy codegen)
Bare Postgres names in pipeline (must be drt_tbl_* / drt_proc_* + inventory — Postgres naming)
Committing or agent-generating *_generated_dashboard.json

Tests may load step manifests for contract coverage.

Params knobs

Behavioral numbers belong in DRTML params_defaults (and UI if tunable). Python may use a named default only when the key is missing and the value matches DRTML. A divergent or 5 when DRTML says 3 is a bug.

See Params and namespaces.

Checklist (framework change)

New cross-layer type? → contracts/, not a local copy.
New backend / mode / metric method? → register; no if-forest.
Import violates the canon edge? → move code or introduce a DTO/callback.
Mention of step id / spaCy / domain.* in orchestration? → leak.
Logic in partition_loop/ that is really I/O? → storage/.
New module under framework/? → update FILES.md when the tree changes.

CI

# from DrToller.Framework repo root
pytest …/architecture/test_layer_import_boundaries.py -q
pytest …/storage/test_dataset_backend_adapter.py -q