Stack CLI

These commands operate the current Compose reference in DrToller.Platform. They are the default operator path, not the only way to host a Framework workload.

Install once per server:

cd /opt/drtoller.platform/stack
chmod +x bin/*.sh
sudo bin/install-stack-cli.sh

This installs only /usr/local/bin/drtoller (bin/drtoller.sh) and removes any legacy flat drt* symlinks. Scripts resolve the stack directory via readlink -f (works from any cwd). Non-default location: /etc/drtoller/stack.env with STACK_DIR=… and optional DOCKER_BIN=….

Framework subcommands (ls, clear, validate, plan, run, serve, init, inspect, …) are forwarded when the package is importable and host Python can import drtoller.framework, pyarrow. Otherwise drtoller runs them inside airflow-worker (DRTOLLER_FW_BACKEND=auto; override host or docker). Host paths under DRTOLLER_HOME/drtoller.edge rewrite to the container mount. See Framework docs/CLI_MIGRATION.md.

First-time .env from answers (see Platform stack/install.example.env):

drtoller install --config /root/drtoller-install.env --generate-secrets --up

Flags override the answers file. Secrets are not printed. --k8s-up PROFILE.env can follow the same write.

Command reference

Command	Script	When to use
`drtoller up`	`servstart.sh`	Full stack bring-up: pull, Postgres migrate, MinIO init, Airflow init, Ray build, Grafana sync
`drtoller up only SVC…`	partial `compose up -d`	Start specific services without pull/init
`drtoller down`	`servdown.sh`	`docker compose down` — stop and remove project containers
`drtoller down stop [SVC…]`	`compose stop`	Stop without removing containers
`drtoller restart [SVC…]`	`drtrestart.sh`	Restart Airflow + Ray + Streamlit (default), or named services
`drtoller update`	`drtupdate.sh`	After code deploy: PG migrate, Grafana sync, rebuild ray-head (+ optional `ray-worker` image), recreate Airflow + restart Streamlit/Prometheus
`drtoller update --git`	+ `git fetch/reset`	Same as deploy hook: sync repo then update
`drtoller update --minimal`	recreate `airflow-worker`	Python/DRTML only: restage kubectl + force-recreate worker. Do not use full `update` for a code-only pull — recreating scheduler/worker breaks Docker DNS for `minio` on the next task.
`drtoller update --env`	delegates to `drtenvupdate.sh`	Env refresh + `compose up -d`
`drtoller env`	`ensure-env.sh`	Create `.env` from `env.example` if missing (usually called by `drtoller up`)
`drtoller env update`	`drtenvupdate.sh`	Replace `.env` from `env.example` (backup `.env.bak`), then `compose up -d`
`drtoller env update --merge`	merge mode	Append only new keys from `env.example`
`drtoller verify`	`verify-stack.sh`	Health: Redis, Airflow Postgres, pipeline Postgres, MinIO, Airflow, compose Ray; if `RAY_ADDRESS=ray://…` — remote KubeRay + MinIO VPC
`drtoller fix-perms`	`fix-artifact-perms.sh`	Fix group write on `/step01`…`/step05` and spaCy volume. (`fixperms` is removed — exit 2 + hint.)
`drtoller k8s up PROFILE.env`	`cloud-k8sup.sh`	One-shot DOKS profile: kubeconfig, pool min/max, bootstrap, attach. Waits for Ray head/worker Ready and the head LB. Example: `drtoller k8s up drtoller-4cpu8ram.env`. (`k8sup` is removed.)
`drtoller k8s down PROFILE.env`	`cloud-k8sdown.sh`	Tear down RayCluster (+ operator ns unless `--keep-operator`) and shrink the DOKS pool. Default 0 nodes, autoscale off. `--min-nodes N` leaves idle capacity. Does not destroy the DOKS cluster or Compose. (`k8sdown` is removed.)
`drtoller cloud bootstrap --provider digitalocean`	`cloud-do-bootstrap.sh`	Idempotent DOKS workload: operator, pull secret, object-store secret, RayCluster (needs existing kubeconfig)
`drtoller cloud status`	`cloud-status.sh`	kubectl context, operator, ray pods/svc, `.env` `RAY_ADDRESS`
`drtoller ray attach`	`cloud-ray-attach.sh`	Head LB IP → `.env` `RAY_ADDRESS`, stage kubectl, recreate Airflow; then `ray status`
`drtoller ray bake`	`cloud-ray-bake.sh`	Build/push the Ray worker image and roll the live cluster (`--check` smoke only)
`drtoller ray sync`	`cloud-ray-sync.sh`	Debug only: patch worker vCPU / `maxReplicas` from the host. Live path is Streamlit knobs at task start
`drtoller install [options]`	`install.sh`	First-time / reconfigure: write `.env` from an answers file or flags; optional secrets, `up`, `k8s up`
`drtoller ls STEP.drtml`	Framework CLI	List `run_id`s with parquet size/object count and PG row counts (`--format json`)
`drtoller clear STEP.drtml --run-id ID[,ID…]`	Framework CLI	Per-run cleanup of parquet prefix / PG rows. Dry-run unless `--yes`. `--all` clears every id from `ls`. `--include-metrics` also clears the sibling metrics step and live metric tables

AWS/EKS, GCP/GKE, Vertex/Anyscale installers are backlog — not in this CLI yet. Current multi-host path is DigitalOcean DOKS + compose on the VM.

`drtoller up` sequence

Ensure Docker daemon
ensure-env.sh → .env
Create DATADIR subtrees (incl. kube/bin, kube/doctl) + Airflow and pipeline Postgres data dirs; stage host kubectl/doctl (sync-kube-airflow.sh)
fix-artifact-perms.sh
docker compose pull (skip local Ray images)
docker compose build ray-head (+ best-effort ray-worker via profile build-ray-worker for KubeRay)
Airflow Postgres + pipeline Postgres + Redis → migrate-postgres.sh (pipeline DB only)
MinIO → minio-init → Qdrant → airflow-init
Ray head, Prometheus, Grafana, Airflow, run-api, Streamlit
sync-grafana-dashboards.sh → restart Grafana

First run downloads large images; expect several minutes.

After deploy / git pull

cd /opt/drtoller.platform/stack
drtoller update
drtoller verify

drtoller update does not rebuild the Airflow image. It:

restages host kubectl/doctl into ${DATADIR}/kube (sync-kube-airflow.sh: binaries in bin/, doctl auth under doctl/);
runs pipeline Postgres migrate (migrate-postgres.sh against postgres-pipeline: codegen, rename-to-drt_* prefixes, framework procs, step DDL, drop orphans vs pg_object_inventory.txt);
regenerates Grafana JSON and restarts Grafana;
rebuilds ray-head (and best-effort ray-worker for KubeRay) and force-recreates Airflow services + Ray head;
restarts Streamlit and Prometheus.

After a Python/DRTML-only git pull, prefer drtoller update --minimal (or commit tag [drtoller restart]). Full update recreates the scheduler and can leave the next task unable to resolve minio until Docker DNS catches up. Airflow in the reference Compose depends on healthy MinIO before start.

Ray cluster health: drtoller ray status (Framework) or, for DOKS, drtoller verify after drtoller ray attach (see Ray).

Multi-host Ray on DOKS (stage 1)

Prerequisites: compose stack up (drtoller up), DOKS Ready, DigitalOcean token in .env.

drtoller k8s up drtoller-4cpu8ram.env
drtoller verify

k8s up polls until Ray head/worker are Ready and the head Service has an EXTERNAL-IP (no manual attach wait). Profile stack/kuberay/profiles/drtoller-4cpu8ram.env sets cluster/pool names and DOKS_MAX_NODES=23. Split form: drtoller cloud bootstrap --provider digitalocean then drtoller ray attach.
After attach, Streamlit runtime.parallelism.max_nodes / cpus_per_worker patch the live RayCluster at task start (kubectl inside airflow-worker, PATH prefers /opt/kube/bin). Put DIGITALOCEAN_ACCESS_TOKEN in .env if kubeconfig execs doctl. Host drtoller ray sync is debug-only.

Tear down Ray (keep the DOKS cluster and Compose):

drtoller k8s down drtoller-4cpu8ram.env                 # pool → 0, autoscale off
drtoller k8s down drtoller-4cpu8ram.env --min-nodes 2   # leave idle nodes

Internals: Platform stack/kuberay/ONTO_RUNBOOK.md. See Ray.

Postgres object names must use drt_tbl_ / drt_proc_ — see PostgreSQL naming.

GitHub Actions deploy (.github/workflows/deploy.yml) can trigger variants via commit tags: [drtupdate], [drtrestart], [drtrestart scheduler], [skip deploy].

Environment updates

Situation	Command
New keys in `env.example` after upgrade	`drtoller env update --merge`
Reset `.env` to template (secrets backup in `.env.bak`)	`drtoller env update`
Changed pip requirements	edit `.env` → `drtoller env update` or force-recreate Airflow
CRLF in `.env` (`$'\r': command not found`)	`drtoller env update` or `tr -d '\r' < .env`

Partial up / down

drtoller down stop ray-head airflow-worker
drtoller up only airflow-worker ray-head
drtoller restart grafana

Day-1 bootstrap (once)

After first successful drtoller up:

cd /opt/drtoller.platform/stack
./bin/day1-bootstrap.sh

Health check

drtoller verify

Reboot

Because restart: "no", reboot does not auto-start the stack. Run drtoller up after server boot (or add your own systemd unit).