Reference Compose environment

This page documents the current default Compose environment in DrToller.Platform (stack/). Framework does not require this service list. You can use it as shipped, change parts of it, or provide a different environment that still satisfies the workload’s capabilities.

docker-compose.yaml defines the reference services; all host paths come from .env (no implicit fallbacks inside compose for production paths).

Prerequisites

• Docker Engine + Compose v2.29+ (scripts use docker compose up --wait).
• Recommended: ≥ 8 GB RAM, ≥ 25 GB disk (Airflow + Ray images are large).
• Open ports on localhost (or change mappings): 5432, 8080, 8088, 8501, 9000, 9001, 8265, 10001, 9090, 3000.

Fresh Ubuntu/Debian: run bin/bootstrap-docker-debian.sh, re-login for docker group, then install CLI.

Services (current default)

This is the service list in the shipped Compose file. It is not a Framework requirement.

Service	Image / build	Purpose
postgres	postgres:16	Airflow metadata only (no host/VPC publish)
postgres-pipeline	postgres:16	drtoller pipeline DB (pipeline schema); VPC :5432 for KubeRay
redis	redis:7-alpine	Airflow Celery broker and result backend (redis://redis:6379/1)
minio	MinIO	S3-compatible raw/artifacts/manifests
minio-init	one-shot	Creates buckets raw, artifacts, manifests, reports
qdrant	Qdrant	Vector dataset backend
ray-head	stack/docker/ray-head.Dockerfile	Ray 2.40 + Python 3.11.11 (parity with Airflow image)
ray-worker	stack/docker/ray-worker.Dockerfile	Build-only image for KubeRay (profile: build-ray-worker); not started by default compose
prometheus	Prometheus	Scrapes Ray SD + worker :9101 metrics
grafana	Grafana	Pipeline dashboards + Postgres datasource
airflow-*	apache/airflow:2.10.4-python3.11	webserver, scheduler, worker, init; depends on healthy MinIO before start
run-api	same Airflow base	Run control-plane HTTP (POST/GET /v1/runs, GET /health) on 127.0.0.1:8088
streamlit	same Airflow base	Step launcher UI (platform/ui/); calls run-api via DRTOLLER_RUN_API_BASE_URL

All services use restart: "no" — after host reboot containers stay down until you run drtoller up.

.env contract

Copy env.example → .env and set secrets before first drtoller up. env.example contains placeholders only (change-me-*, REPLACE_WITH_*) — never commit .env.

cp env.example .env
# Edit: passwords, Airflow Fernet/secret keys, host paths

Required host paths

Variable	Meaning
DRTOLLER_HOME	Parent of step checkouts on the host (e.g. /opt/drtoller). Compose mounts ${DRTOLLER_HOME}/drtoller.edge by default (OSS example pack) — use that directory name or edit mounts for additional packs.
DATADIR	Runtime data: Airflow logs, Grafana, spaCy cache, Qdrant, Prometheus TSDB, Redis AOF, kube/ (bin/kubectl + bin/doctl, kubeconfig, doctl/config.yaml for Airflow → KubeRay patch)
DRTOLLER_POSTGRES_DATA_HOST	Airflow metadata PGDATA (prefer a separate volume)
DRTOLLER_PIPELINE_POSTGRES_DATA_HOST	Pipeline PGDATA (drtoller DB); published on POSTGRES_API_BIND / MINIO_API_BIND:5432 for KubeRay
DRTOLLER_CORPORA_HOST	Host dir with workspace/ (corpus) and minio/ (object store)
DRTOLLER_ARTIFACTS_STEP0N_HOST	Per-step artifact roots → /step01…/step05 in containers

In-container paths (DRTOLLER_STEP01_ARTIFACTS_ROOT=/step01, … /step05) must match compose mount targets — change both together.

Secrets and Airflow

Set before production:

• POSTGRES_PASSWORD, MINIO_ROOT_PASSWORD
• Optional PIPELINE_POSTGRES_* if pipeline credentials differ from Airflow
• AIRFLOW__CORE__FERNET_KEY, AIRFLOW__WEBSERVER__SECRET_KEY
• _AIRFLOW_WWW_USER_PASSWORD (admin UI)

Generate Fernet: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())".

Optional run HTTP API (Compose service run-api): DRTOLLER_API_PORT (default 8088), DRTOLLER_RUN_API_BASE_URL for Streamlit. Host bind is loopback. See Run API.

Python dependencies in workers

AIRFLOW_PIP_ADDITIONAL_REQUIREMENTS in .env — installed on Airflow container start. After changes: drtoller env update or docker compose up -d --force-recreate airflow-webserver airflow-scheduler airflow-worker.

Code bind mounts

Compose mounts Framework (src layout), step packs, and Platform into runtime containers. Default host dirs under DRTOLLER_HOME (symlink or rename clones to match):

${DRTOLLER_HOME}/drtoller.framework/src/drtoller  → /drtoller/drtoller          (import drtoller.framework.*, ro)
${DRTOLLER_HOME}/drtoller.edge                    → /drtoller/drtoller.edge     (rw; OSS example pack)
${DRTOLLER_HOME}/drtoller.platform                → /drtoller/stack-platform    (ro)

Sibling clones (DrToller.Framework, DrToller.Platform, …) are the source of truth in git; on the host, point those compose paths at the installed/symlink layout that matches the mount contract above.

Platform is mounted as stack-platform (not /platform) to avoid shadowing Python stdlib.

PYTHONPATH includes /drtoller and mounted step packs (Edge by default). Step discovery:

DRTOLLER_STEPS_ROOTS_FILE=/drtoller/stack-platform/stack/config/steps_roots.txt

Edit config/steps_roots.txt to add client step roots (one path per line, in-container).

Object store env (S3 / MinIO / GCS)

Airflow and ray-head receive S3_ACCESS_KEY / S3_SECRET_KEY / S3_REGION (and related endpoint vars) from .env so parquet backend: s3 works on multi-worker Ray. Presets and comments live in Platform env.example; Framework CLI dumps the same catalogs:

drtoller deploy env --provider minio
# also: aws_s3, gcp_gcs

For multi-host, set DRTOLLER_REQUIRE_OBJECT_STORE=1 (or rely on ray_address + workers > 1) and avoid host-local parquet stores — Ray object-store guard.

KubeRay shape knobs (DRTOLLER_RAY_CPUS_PER_WORKER default 4, DRTOLLER_RAY_MAX_NODES default 23, DRTOLLER_RAY_CLUSTER_NS, DRTOLLER_RAY_CLUSTER_NAME) are fallbacks when DRTML/UI omit the keys. Live patch runs inside Airflow at Ray task start. If the DO kubeconfig execs doctl, set DIGITALOCEAN_ACCESS_TOKEN in .env. Worker yaml for s-4vcpu-8gb: CPU request/limit 3/4, memory 4Gi/6.5Gi, plasma+shm 128 MiB, maxReplicas 22.

Custom Docker images (Ray)

Image	When
ray-head	Always built by drtoller up / drtoller update — local single-node Ray in compose
ray-worker	Build-only (docker compose --profile build-ray-worker build ray-worker) — KubeRay worker image; not a compose scale-out service

After Dockerfile changes:

docker compose build ray-head
docker compose up -d --force-recreate ray-head
docker compose --profile build-ray-worker build ray-worker   # optional, for k8s push

Airflow uses the upstream apache/airflow image — no local Dockerfile.

Multi-host target is KubeRay on DOKS (Platform CLI: drtoller k8s up / k8s down with drtoller-4cpu8ram.env). Not manual multi-VM ray up. AWS/GCP installers are backlog. See Ray and Platform stack/kuberay/ONTO_RUNBOOK.md.

Grafana provisioning

Static provisioning in platform/grafana/provisioning/:

• Datasources: Prometheus, Postgres (pipeline DB).
• Pipeline dashboards: file provider reads generated JSON under each step pack’s static/grafana (for example /drtoller.edge/static/grafana). JSON is not committed.

Ray dashboards are embedded in Ray Dashboard via RAY_GRAFANA_IFRAME_HOST — separate from pipeline sync.

Postgres init

On first pipeline Postgres volume (postgres-pipeline), stack/postgres-init/*.sql runs:

1. Create drtoller DB, pipeline schema bootstrap
2. Generic framework tables/procedures (drt_tbl_* / drt_proc_* — see naming)
3. MLflow DB (optional future use)

Step-specific CREATE TABLE DDL is not in init scripts — it is codegen’d from DRTML on drtoller up / drtoller update via migrate-postgres.sh.

Canonical allowlist: postgres-init/pg_object_inventory.txt. After migrate, drop-pg-orphans.sh removes any pipeline object missing from that file. New platform tables/procs must be added to the inventory in the same change.

Nginx (optional)

Examples under stack/nginx/ for HTTPS to Airflow, Grafana, Prometheus, MinIO, MLflow. Not required for local 127.0.0.1 access.

Changing compose

If you add volumes, services, or env keys in docker-compose.yaml, a restart is not enough — run drtoller down then drtoller up, or docker compose up -d --force-recreate … for affected services.

Next

• CLI operations
• Grafana sync