DrToller.Framework

A machine-readable execution contract for data workloads. Humans extend primitives; humans and AI compose pipelines.

Why Framework

DrToller.Framework is a declarative execution framework for data and ML workloads. DRTML expresses the logical shape of a workload as a machine-readable execution contract; the Framework validates, compiles, and executes it across interchangeable backends.

The framework owns:

loading and compiling DRTML (*.drtml + fragments);
storage (parquet / PostgreSQL / Qdrant / neo4j stub) via a single adapter registry;
processing paths: partition_loop, mapping, reduce_merge, dataset_metrics, evaluation, embed_train;
telemetry (Prometheus emit, Grafana codegen);
integrations (Streamlit, Airflow, Ray connect, local runner).

Step canon: domain logic interacts through explicit contracts — receive data → transform → return data. Infrastructure (I/O, Prometheus, Airflow/Ray APIs) stays in the Framework. How this layer relates to Spark, Ray, Airflow, and agent stacks: Compare.

Start here

Principles

Step ↔ framework · Layers inside framework — mandatory.

Install

Dev setup · Deploy stack

How a run works

From manifest to ProcessResult and write.

What's new

Release notes and migration actions.

Compare

Layers vs orchestration, Spark, Ray, agent stacks.

Official OSS steps

Step01 extraction and offline vocabulary metrics.

Cookbook

Build a complete standalone step.

Extend the platform

Backend · DB · Algorithm · Mode · DRTML

Package map

drtoller/framework/
  contracts/      # layer-neutral DTOs (saturation, worker delta)
  drtml/          # load / validate / compile
  runtime/        # RunContext, import_symbol, pattern_runners
  storage/        # session, feed, backends, registry adapters
  processing/     # dispatch + modes; job_map / job_session
  mapping/        # merge / remap algorithms
  algorithms/     # embed, cluster, dataset metrics
  db/             # postgres / qdrant transport
  telemetry/      # prometheus + grafana codegen
  integration/    # UI, Airflow, Ray, env
  ext/            # Extension SDK (register_*, load_plugins)
  examples/       # hello-step (dev smoke, not installed)

Dependency canon:

drtml → contracts / storage.plan → db|storage → processing → telemetry|integration

Reverse edges are forbidden. Full rules: Principles.

Docs status

The documentation covers DRTML v4, framework boundaries, storage backends, processing modes, algorithms, observability, and deployment. Recent changes: What’s new. Start and poll a run over HTTP: Run API.