Architecture overview

Deployment units

The framework, domain step packages, platform stack, and documentation are separate deployment concerns:

framework package        # DRTML, storage, processing, algorithms, telemetry
step packages            # DRTML contracts + pure domain processors
platform stack           # Airflow, MinIO, PostgreSQL, Qdrant, Prometheus, Grafana
website/docs             # static documentation deployment

They may share a development checkout, but no public API depends on that layout. Step discovery uses configured roots and Python import paths.

Execution flow

Airflow / Streamlit / local
        │
        ▼
 integration/  (RunContext + params overlay)
        │
        ▼
 processing/dispatch.py
        │  pattern / mode / entry registry
        ├──────────────┬──────────────┬──────────────┐
        ▼              ▼              ▼              ▼
 partition_loop    mapping      reduce_merge    embed_train
        │              │              │              │
        └──────────────┴──────┬───────┴──────────────┘
                              ▼
                    storage/ (session + feed)
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
           parquet        postgres         qdrant
           local/S3         (db/)           (db/)
                              │
                              ▼
                    telemetry/ (emit only)

Principles

1. Step = transform. No file/database/metrics/orchestrator I/O in domain code.
2. DRTML = contract. Params, datasets, metrics, UI and orchestration are declarative.
3. One dataset — one backend. Dual-write is unsupported.
4. Registries instead of if-forests. Backends, modes, patterns and metric methods are extensible catalogs.
5. Neutral contracts. Shared DTOs prevent reverse layer dependencies.
6. Algorithms = computation. Domain owns policy such as candidate K/probe loops.

Package ownership

• drtml/: parse, validate, compile manifests.
• contracts/: layer-neutral DTOs.
• runtime/: RunContext and dynamic imports.
• storage/: sessions, feeds, parquet and backend adapters.
• db/: PostgreSQL/Qdrant transports.
• processing/: execution orchestration.
• mapping/: mapping build/read/remap.
• algorithms/: embedding, clustering and dataset metrics.
• telemetry/: Prometheus emit and Grafana codegen.
• integration/: UI, Airflow, Ray connection and local runner.