DrToller.Framework

Why Framework

DrToller.Framework is a reusable runtime for independently packaged pipeline steps. The framework owns:

• loading and compiling DRTML (*.drtml + fragments);
• storage (parquet / PostgreSQL / Qdrant / neo4j stub) via a single adapter registry;
• processing modes: partition_loop, mapping, reduce_merge, embed_train;
• telemetry (Prometheus emit, Grafana codegen);
• integrations (Streamlit, Airflow, Ray connect, local runner).

Step canon: receive data → transform → return data. No I/O, no Prometheus, no Airflow API in domain code.

Start here

Install

Set up a dev environment and run tests.

How a run works

From manifest to ProcessResult and write.

Boundaries

What a step may and must not do.

Cookbook

Build a complete standalone step.

Package map

drtoller/framework/
  contracts/      # layer-neutral DTOs (saturation, worker delta)
  drtml/          # load / validate / compile
  runtime/        # RunContext, import_symbol
  storage/        # session, feed, backends, registry adapters
  processing/     # dispatch + modes (orchestration only)
  mapping/        # merge / remap algorithms
  algorithms/     # embed, cluster, dataset metrics
  db/             # postgres / qdrant transport
  telemetry/      # prometheus + grafana codegen
  integration/    # UI, Airflow, Ray, env

Dependency canon:

drtml → contracts / storage.plan → db|storage → processing → telemetry|integration

Reverse edges (storage → processing, telemetry → processing, …) are forbidden and caught by CI.

Docs status

v1 covers architecture after P0–P4 remediation (contracts, backend registry, train_loop, dispatch/metric registries). REST API — Coming soon.