Architecture overview

DrToller.Framework is a declarative execution framework for data and ML workloads. DRTML expresses the logical shape of a workload as a machine-readable execution contract; the Framework validates, compiles, and executes it across interchangeable compute, storage, parallelism, and observability backends.

Humans extend the Framework (primitives via registries). Humans and external AI compose those primitives into DRTML. The runtime does not embed an LLM. Layer relationships to Airflow, Spark, Ray, and agent stacks: Compare.

Deployment units

The framework, domain step packages, deployment environment, and documentation are separate concerns:

framework package        # DrToller.Framework — workload and runtime
step packages            # e.g. DrToller.Edge — DRTML contracts + domain processors
platform                 # DrToller.Platform — deployment environment (current reference: Compose)
website/docs             # DrToller.com — static documentation deployment

They may share a development checkout, but no public API depends on that layout. Step discovery uses configured roots and Python import paths.

Separation of concerns

Domain logic
     ↓
DRTML / execution contract
     ↓
Framework
  ├── execution modes
  ├── parallelism backends (e.g. Ray today)
  ├── storage backends
  ├── metrics / telemetry
  └── host integrations (Airflow, Streamlit, local)

Domain packages should not re-implement infrastructure inside every step. Steps interact through explicit contracts; the Framework owns feed, write, checkpoint, metrics emit, and worker transport.

Execution flow

Airflow / Streamlit / local
        │
        ▼
 integration/  (RunContext + params overlay)
        │
        ▼
 processing/dispatch.py  (+ load_plugins / ext hooks)
        │  pattern/mode via runtime/pattern_runners
        ├──────────────┬──────────────┬──────────────┐
        ▼              ▼              ▼              ▼
 partition_loop    mapping      reduce_merge    stateful/offline
                                      │          modes and patterns:
                                      │          embed_train,
                                      │          dataset_metrics,
                                      │          evaluation, probe
        └──────────────┴──────┬───────┴──────────────┘
                              ▼
                    storage/ (session + feed)
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
           parquet        postgres         qdrant
           local/S3         (db/)           (db/)
                              │
                              ▼
                    telemetry/ (emit only)

Principles

Mandatory rules for step ↔ framework and layer imports: Principles.

1. Step = transform. No file/database/metrics/orchestrator I/O in domain code.
2. DRTML = contract. Params, datasets, metrics, UI and orchestration are declarative.
3. One dataset — one backend. Dual-write is unsupported.
4. Registries instead of if-forests. Backends, modes, patterns and metric methods are extensible catalogs.
5. Neutral contracts. Shared DTOs prevent reverse layer dependencies.
6. Algorithms = computation. Domain owns policy such as candidate K/probe loops.

Package ownership

• drtml/: parse, validate, compile manifests.
• contracts/: layer-neutral DTOs.
• runtime/: RunContext and dynamic imports.
• storage/: sessions, feeds, parquet and backend adapters.
• db/: PostgreSQL/Qdrant transports.
• processing/: execution orchestration.
• mapping/: mapping build/read/remap.
• algorithms/: embedding, clustering and dataset metrics.
• telemetry/: Prometheus emit and Grafana codegen.
• integration/: UI, Airflow, Ray connection and local runner.