Add a database / engine sink
There is no separate “database plugin” path. A new engine (Postgres sibling, Neo4j, ClickHouse, …) is a dataset backend:
datasets.*.backend: <name> → DRTML whitelist → db/<engine>/ transport → DatasetWriteBackend / DatasetReadBackend → register.
Bindings (storage.stores local/S3 profiles) are not database backends. A new MinIO bucket or path profile is configuration, not a new sink type.
Quick map
Section titled “Quick map”| Goal | Do this |
|---|---|
| Use existing PostgreSQL | PostgreSQL sink |
| Use existing Qdrant | Qdrant sink |
| Finish Neo4j stub | Follow New dataset backend with db/neo4j/ |
| Add any new engine | Same cookbook — that is the full vertical recipe |
Architecture slice
Section titled “Architecture slice”flowchart TD DRTML["DRTML datasets backend + inline block"] --> PLAN["StoragePlan / DatasetOutputSpec"] PLAN --> SESS["StorageSession write / read"] SESS --> REG["backend_registry lookup"] REG --> ADP["Adapter thin dispatch"] ADP --> RUN["storage/run engine modules"] RUN --> DB["db engine transport"]
Rules that matter for databases:
- One dataset → one backend (no dual-write parquet+PG).
- No
if backend ==inStorageSession. - Bulk writes only for SQL engines (
execute_values/COPY/ array|jsonb proc) — see postgres architecture. - No runtime DDL (
CREATE TABLEat run start). Deploy-time codegen/migrate only. - Step domain code never imports
psycopg/ engine clients. - PostgreSQL
pipelineobjects: tablesdrt_tbl_*, procsdrt_proc_*, plus a line inpg_object_inventory.txt— naming.
Follow the backend cookbook
Section titled “Follow the backend cookbook”The step-by-step implementation list (whitelist, inline block, transport, adapters, capability sets, tests) lives here:
Read next after that:
- Backend adapters
- Extend DRTML (section “Add a dataset backend”)
- Registries