Docker stack

The stack lives in DrToller.Platform at stack/. docker-compose.yaml defines all services; all host paths come from .env (no implicit fallbacks inside compose for production paths).

Prerequisites

• Docker Engine + Compose v2.29+ (scripts use docker compose up --wait).
• Recommended: ≥ 8 GB RAM, ≥ 25 GB disk (Airflow + Ray images are large).
• Open ports on localhost (or change mappings): 5432, 8080, 8501, 9000, 9001, 8265, 10001, 9090, 3000.

Fresh Ubuntu/Debian: run bin/bootstrap-docker-debian.sh, re-login for docker group, then install CLI.

Services

Service	Image / build	Purpose
postgres	postgres:16	Airflow DB + drtoller pipeline DB
redis	redis:7-alpine	Airflow Celery broker
minio	MinIO	S3-compatible raw/artifacts/manifests
minio-init	one-shot	Creates buckets raw, artifacts, manifests, reports
qdrant	Qdrant	Vector dataset backend
ray-head	stack/docker/ray-head.Dockerfile	Ray 2.40 + Python 3.11.11 (parity with Airflow image)
ray-worker	stack/docker/ray-worker.Dockerfile	Build-only image for KubeRay (profile: build-ray-worker); not started by default compose
prometheus	Prometheus	Scrapes Ray SD + worker :9101 metrics
grafana	Grafana	Pipeline dashboards + Postgres datasource
airflow-*	apache/airflow:2.10.4-python3.11	webserver, scheduler, worker, init
streamlit	same Airflow base	Step launcher UI (platform/ui/)

All services use restart: "no" — after host reboot containers stay down until you run drtoller up.

.env contract

Copy env.example → .env and set secrets before first drtoller up. env.example contains placeholders only (change-me-*, REPLACE_WITH_*) — never commit .env.

cp env.example .env
# Edit: passwords, Airflow Fernet/secret keys, host paths

Required host paths

Variable	Meaning
DRTOLLER_HOME	Parent of step checkouts on the host (e.g. /opt/drtoller). Compose mounts ${DRTOLLER_HOME}/drtoller.edge and …/drtoller.ops by default — use those directory names or edit mounts.
DATADIR	Runtime data: Airflow logs, Grafana, spaCy cache, Qdrant, Prometheus TSDB, Redis AOF
DRTOLLER_POSTGRES_DATA_HOST	Postgres PGDATA (prefer separate block volume)
DRTOLLER_CORPORA_HOST	Host dir with workspace/ (corpus) and minio/ (object store)
DRTOLLER_ARTIFACTS_STEP0N_HOST	Per-step artifact roots → /step01…/step05 in containers

In-container paths (DRTOLLER_STEP01_ARTIFACTS_ROOT=/step01, … /step05) must match compose mount targets — change both together.

Secrets and Airflow

Set before production:

• POSTGRES_PASSWORD, MINIO_ROOT_PASSWORD
• AIRFLOW__CORE__FERNET_KEY, AIRFLOW__WEBSERVER__SECRET_KEY
• _AIRFLOW_WWW_USER_PASSWORD (admin UI)

Generate Fernet: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())".

Python dependencies in workers

AIRFLOW_PIP_ADDITIONAL_REQUIREMENTS in .env — installed on Airflow container start. After changes: drtoller env update or docker compose up -d --force-recreate airflow-webserver airflow-scheduler airflow-worker.

Code bind mounts

Compose mounts Framework (src layout), step packs, and Platform into runtime containers. Default host dirs under DRTOLLER_HOME (symlink or rename clones to match):

${DRTOLLER_HOME}/drtoller.framework/src/drtoller  → /drtoller/drtoller          (import drtoller.framework.*, ro)
${DRTOLLER_HOME}/drtoller.edge                    → /drtoller/drtoller.edge     (rw)
${DRTOLLER_HOME}/drtoller.ops                     → /drtoller/drtoller.ops      (rw)
${DRTOLLER_HOME}/drtoller.platform                → /drtoller/stack-platform    (ro)

Sibling clones (DrToller.Framework, DrToller.Platform, …) are the source of truth in git; on the host, point those compose paths at the installed/symlink layout that matches the mount contract above.

Platform is mounted as stack-platform (not /platform) to avoid shadowing Python stdlib.

PYTHONPATH includes /drtoller, edge, and ops. Step discovery:

DRTOLLER_STEPS_ROOTS_FILE=/drtoller/stack-platform/stack/config/steps_roots.txt

Edit config/steps_roots.txt to add client step roots (one path per line, in-container).

Object store env (S3 / MinIO / GCS)

Airflow and ray-head receive S3_ACCESS_KEY / S3_SECRET_KEY / S3_REGION (and related endpoint vars) from .env so parquet backend: s3 works on multi-worker Ray. Presets and comments live in Platform env.example; Framework CLI dumps the same catalogs:

drtoller deploy env --provider minio
# also: aws_s3, gcp_gcs

For multi-host, set DRTOLLER_REQUIRE_OBJECT_STORE=1 (or rely on ray_address + workers > 1) and avoid host-local parquet stores — Ray object-store guard.

Custom Docker images (Ray)

Image	When
ray-head	Always built by drtoller up / drtoller update — local single-node Ray in compose
ray-worker	Build-only (docker compose --profile build-ray-worker build ray-worker) — KubeRay worker image; not a compose scale-out service

After Dockerfile changes:

docker compose build ray-head
docker compose up -d --force-recreate ray-head
docker compose --profile build-ray-worker build ray-worker   # optional, for k8s push

Airflow uses the upstream apache/airflow image — no local Dockerfile.

Multi-host target is KubeRay on DOKS (Platform CLI: drtoller cloud bootstrap --provider digitalocean --attach). Not manual multi-VM ray up. AWS/GCP installers are backlog. See Ray and Platform stack/kuberay/ONTO_RUNBOOK.md.

Grafana provisioning

Static provisioning in platform/grafana/provisioning/:

• Datasources: Prometheus, Postgres (pipeline DB).
• Pipeline dashboards: file provider reads /drtoller.edge/static/grafana and /drtoller.ops/static/grafana (generated JSON, not committed).

Ray dashboards are embedded in Ray Dashboard via RAY_GRAFANA_IFRAME_HOST — separate from pipeline sync.

Postgres init

On first Postgres volume, stack/postgres-init/*.sql runs:

1. Create drtoller DB, pipeline schema bootstrap
2. Generic framework tables/procedures (drt_tbl_* / drt_proc_* — see naming)
3. MLflow DB (optional future use)

Step-specific CREATE TABLE DDL is not in init scripts — it is codegen’d from DRTML on drtoller up / drtoller update via migrate-postgres.sh.

Canonical allowlist: postgres-init/pg_object_inventory.txt. After migrate, drop-pg-orphans.sh removes any pipeline object missing from that file. New platform tables/procs must be added to the inventory in the same change.

Nginx (optional)

Examples under stack/nginx/ for HTTPS to Airflow, Grafana, Prometheus, MinIO, MLflow. Not required for local 127.0.0.1 access.

Changing compose

If you add volumes, services, or env keys in docker-compose.yaml, a restart is not enough — run drtoller down then drtoller up, or docker compose up -d --force-recreate … for affected services.

Next

• CLI operations
• Grafana sync