What's new

This section records user-visible changes: new contracts, behavior, extension points, deployment requirements, and migrations.

Latest

August 12, 2026 — Unified `drtoller` CLI + DOKS bootstrap (Unreleased)

Canonical CLI: drtoller only (init, plan, run, validate, inspect, …). PyPI package: drtoller.
Platform stack: drtoller up|down|restart|update|verify|env|fixperms via stack/bin/drtoller.sh. Flat drt* / drtoller start removed.
KubeRay (DigitalOcean): drtoller cloud bootstrap --provider digitalocean, drtoller cloud status, drtoller ray attach; verify checks remote Ray + MinIO VPC. AWS/GCP installers remain backlog.
Platform landing: platform.drtoller.com (product + compare; no separate docs tree — operator reference stays in Deploy).
AI bootstrap: drtoller init ships AGENTS.md, Cursor skills/rules, Copilot instructions. Docs: AI quickstart.

August 11, 2026 — Multi-host Ray prep (Unreleased)

Object-store guard: storage/ray_compat.py rejects parquet backend: local when remote Ray + workers > 1, or when runtime.parallelism.require_object_store / DRTOLLER_REQUIRE_OBJECT_STORE=1. Use backend: s3 (MinIO / AWS / GCS).
ParallelPlan: num_gpus, require_object_store on runtime.parallelism.*.
CLI: drtoller ray status (+ --format json); deploy env --provider minio|aws_s3|gcp_gcs (+ --list-providers).
Preflight: partition Ray backend calls preflight_ray before actors.
Platform: Framework mount ${DRTOLLER_HOME}/drtoller.framework/src/drtoller → /drtoller/drtoller; S3 env on Airflow/ray-head; build-only ray-worker image + stack/kuberay/README.md (KubeRay target, not compose scale-out).

August 10, 2026 — LLM-friendly surface (Unreleased)

Capability catalog: drtoller.framework.capability (PrimitiveContract, build_catalog(), get_contract()); schema 1.1 with composition index + compatible_with.
CLI: drtoller inspect (--format json, --kind, --level 1..5, --schema, --docs); validate --format json emits structured {valid, errors[{code,path,available,fixes,…}]}.
Composition: generic I/O vocabulary in contracts/io_types; can_compose / compatible_consumers / assert_compatible (INCOMPATIBLE_INPUT).
AI-repairable DRTML validation: all parse/validate paths use raise_validation_error → DrtmlValidationError (still a ValueError subclass). Stable code/path/available/fixes also on compile, flush/checkpoint, feed, metrics, panels/UI, dispatch, mapping_mode, reduce_merge, embed_train.mode_spec. Docs: AI & Automation, Semantic validation.
Package layout: sources under src/drtoller/framework/ (import path drtoller.framework.* unchanged).
embed_train engine move: processing/embed_vocab/ → processing/embed_train/engine/ (compat shim removed). Metric/param ids named embed_vocab_* are unchanged.
Removed: top-level DRTML postgres: / postgres.tables and dataset postgres_table pointers — inline datasets.*.postgres only.
Example: examples/agent-pipeline-loop — discover → broken draft → structured repair → compile (no LLM).

August 8, 2026 — Developer preview `0.1.4`

Install: pip install drtoller (PyPI via GitHub Release + Trusted Publishing; see Framework docs/PUBLISHING.md).
Docs site: canonical Starlight docs at framework.drtoller.com/docs (DrToller.com/framework/docs); company landing drtoller.com.
Examples: hello-step, hello-plugin, hello-mapping; in-repo tutorial docs/tutorial-write-a-step.md.
CI: Python 3.10–3.12, ruff/mypy surface, informational coverage, MkDocs preview build.
Discover: steps roots from env only (DRTOLLER_STEPS_ROOTS* / EDGE_HOST / OPS_HOST) — no monorepo sibling scan.

August 8, 2026 — Developer preview `0.1.0`

Standalone Framework package: Apache-2.0 (LICENSE / NOTICE), SECURITY.md, Keep a Changelog CHANGELOG.md, packaging metadata (pip install -e ".[dev]" from the Framework repo root).
CLI: drtoller version and validate path/to/step.drtml (optional --compile); also drtoller.drtml ….
examples/hello-step/: minimal partition step + pytest with no Edge/Ops dependency.
Pytest marker requires_ops: contract tests that load drtoller.edge / drtoller.ops stay opt-in (RUN_OPS_TESTS=1).
Repo split: Framework, Platform, Edge, Ops, and docs (DrToller.com) are sibling repositories — not one monorepo checkout.

August 8, 2026 — Processing fan-out and registries

Two fan-out tracks: feed multi-worker stays in partition_loop/parallel/; offline map jobs (reduce buckets, row-assembly shards, probe evaluate / interleaved prepare→evaluate) share processing/job_map.py (map_jobs, pipeline_jobs, assign_round_robin). Embed PS and sticky cluster fit remain actor tracks — not copies of map.
Worker sessions: processing/job_session.open_job_worker_session for reduce / assembly children.
Pattern/mode registry lives in runtime/pattern_runners.py (ensure_builtin_runners, get_pattern_runner / get_mode_runner); processing/dispatch orchestrates and re-exports. Public Extension SDK: drtoller.framework.ext (load_plugins, register_*).
run_from_manifest calls load_plugins() and Extension SDK run hooks (emit_run_start / emit_run_end) around the runner.
Parallel transport registry: partition_loop/parallel/backend_registry.py (register_parallel_backend, ParallelBackend); alias threads → inprocess.
Evaluation runner split: thin engine + sample / vector_job / cluster_job / graph_job / write (formulas still in algorithms/evaluation/).
Row assembly spill path: row_assembly/spill.py + parallel_jobs.py + shard_plan.py for large nested evaluate inputs.

August 8, 2026 — Platform (deploy)

Standalone DrToller.Platform (Apache-2.0): Docker Compose stack, CLI, Postgres init — steps are not bundled; mount Edge/Ops via DRTOLLER_HOME / DRTOLLER_STEPS_ROOTS.
env.example: placeholders only (change-me-*, REPLACE_WITH_*); secrets live in gitignored .env (SECURITY.md).
Migrate path unchanged in spirit: codegen → bootstrap → rename-to-drt_tbl_* → framework procs → step DDL → drop-pg-orphans vs pg_object_inventory.txt.
Corpus-growth procs: sources must be pipeline.drt_tbl_* (require_source_relation); final metrics avoid an extra staging copy where the direct path applies.

August 2, 2026

Sticky cluster fit session: under compute.cluster.fit_sticky_cache (default on), vector shards load once into worker RAM (or Ray FitShardActors). The same cache serves many probe K values — no parquet re-read per candidate. Serial / threads / Ray session backends share one API (open_sticky_fit_session → fit_then_assign → close).
Warm-start centroids: when compute.cluster.warm_start is on, ascending-K init expands the previous centroid set via farthest-point selection (GEMM + running min_d in algorithms/cluster) instead of cold random init each time.
Lloyd epochs on sticky actors: each epoch broadcasts centroids (ray.put), actors run GEMM assign + local (sum, count) accumulate; the driver merges into one global model. Optional inertia early-stop (InertiaEarlyStop: relative improvement + patience) can finish before max_iter.
Assign on sticky actors: stats-only and vote collect reuse the cached matrices — int labels + np.bincount size counts on the hot path (no per-row string cleanup when keys are already clean).
Tree-merge vote maps: associative pairwise merge of assign vote/size payloads on the caller (ThreadPool). Sticky Ray gathers actor payloads then merges locally — cluster-side ObjectRef trees were tried and rolled back (OOM next to sticky actors).
LPT vector-shard balance: partition_vector_shards packs by manifest row_count (largest-first onto the lightest worker); falls back to round-robin when counts are unknown. Sticky actors see more even N.
Fit progress log: structured cluster_fit stage timings (load, epochs, assign collect, merge) for operator grepping — telemetry only, no domain policy.

August 1, 2026

Probe evaluate input cache: process-local row_feed_cache reuses identical dataset loads across nested evaluate candidates (compute.probe.cache_eval_inputs, default on). Cleared when the probe parent ends.
Bindings trim after evaluate: heavy payload keys (default assignment_map) are dropped from ProbeBindings between candidates so RAM does not grow with probe width; centroids needed for publish stay.
Nested invoke hardening: probe child launch / interleave fixes so nested evaluate entries start reliably under the shared worker budget.
Live probe gauges → Grafana sooner: status emitter flushes phase / candidate / fit-iter gauges on every note_*, not only on the heartbeat tick — long prepares no longer leave panels empty.

July 28, 2026

Cluster package layout: processing/cluster split into fit/ (incl. fit/sticky/) and assign/ with shared shards, early_stop, hash_mode_reduce. Public behavior is the streaming fit→assign path; extension imports should follow the new modules (see FILES.md).

July 27, 2026

Live probe/cluster phase status: process-local live_status tracks probe phase codes (prepare_load, prepare_fit, prepare_assign, evaluate, decide, publish), candidate id, fit iteration / max, and inertia. Prometheus optional gauges (probe_phase, probe_candidate_k, probe_fit_iter, probe_fit_inertia, …) feed Grafana value mappings — framework emit only; domain does not instrument.
Probe progress model: one progress tick unit = prepare or evaluate per candidate, plus publish; progress_total is exposed at begin so dashboards have a denominator during the first long prepare.

July 26, 2026

Streaming cluster fit/assign: learn one shared centroid model and label vectors in bounded batches — no full N×D matrix on the hot path. Optional Ray/process shard workers merge centroid stats (fit) and write assignment chunks (assign).
Vector batch iterator: storage streams numpy windows (vectors, row_ids[, metas]) for geometry work; distinct from feed / Arrow / embed-pair batches.
Offline evaluation for clusters and graphs: artifact_kind: cluster_rows (assignments joined to vectors) and graph_edges are first-class evaluation jobs with registered quality methods — not Python-only leftovers.

July 25, 2026

Probe loop: framework N+1 candidate search (prepare → nested evaluate → domain decide → publish). Winner selection stays in the step; parallelism nests under runtime.parallelism with a shared worker budget.
Row assembly: shared helper to load inputs, apply probe subset, call domain build, and optionally persist — so nested evaluate steps do not reinvent storage glue.

July 24, 2026

Shuffle writer lifecycle: after each shuffle flush, part rotation closes touched bucket writers so a run does not keep up to shuffle.buckets files open.
storage.shuffle_buckets: optional runtime override on every output that already declares shuffle:.
reduce_merge work units = shuffle buckets: round-robin across workers, private write lanes, driver manifest merge — map → shuffle → reduce.
Batch mean-pool: occurrence inference mean-pools many rows against one vocab matrix in a single NumPy pass.

July 23, 2026

Honest Ray embed_train: workers >= 2 shards embedding table W across Ray actors; workers = 1 stays a single W actor.
Default train parallelism is ray / 8 workers.
Early shuffle on parquet outputs: DRTML shuffle: { by, buckets }.
Streaming parquet reads on the hot path; materializing reads reserved for small fixtures.
Memory policy: no full-run / full-keyspace Python materialization; OOM fixes prefer streaming/spill/SQL, not “switch to inprocess”.

July 22, 2026

Embed-train metrics catalog and observation emit were unified.
Vector evaluation methods expanded; evaluation can write corpus-growth-shaped PostgreSQL observation points.
embed_train run results expose public train_stats dicts (Airflow/XCom-safe).

July 21, 2026

dataset_metrics supports separate final and milestone jobs, including per-document evidence curves.
PostgreSQL final metrics can use a direct exact path when the method supports it.
Grafana PostgreSQL curves support xychart with corpus_fraction on the X axis.
PostgreSQL object naming and inventory cleanup are enforced during deployment.

See July 2026 release notes for contracts and migration notes through July 26.

Canonical module docs:

What belongs here

Add an entry when a change affects at least one of:

DRTML syntax or validation;
public framework APIs and registries;
execution behavior or dataset semantics;
supported algorithms or metric methods;
generated dashboards and observability;
platform CLI, Docker stack, migrations, or operator actions.

Pure internal refactoring is mentioned only when it changes extension paths or removes a previously documented module.

Entry format

Each release note should state:

What changed
Who is affected
Required migration or deploy action
Links to the updated canonical documentation

Do not use release notes as the only documentation for a feature: update the relevant reference/cookbook page in the same change.