Skip to content

Add a database / engine sink

There is no separate “database plugin” path. A new engine (Postgres sibling, Neo4j, ClickHouse, …) is a dataset backend:

datasets.*.backend: <name> → DRTML whitelist → db/<engine>/ transport → DatasetWriteBackend / DatasetReadBackend → register.

Bindings (storage.stores local/S3 profiles) are not database backends. A new MinIO bucket or path profile is configuration, not a new sink type.

Goal Do this
Use existing PostgreSQL PostgreSQL sink
Use existing Qdrant Qdrant sink
Finish Neo4j stub Follow New dataset backend with db/neo4j/
Add any new engine Same cookbook — that is the full vertical recipe
flowchart TD
  DRTML["DRTML datasets backend + inline block"] --> PLAN["StoragePlan / DatasetOutputSpec"]
  PLAN --> SESS["StorageSession write / read"]
  SESS --> REG["backend_registry lookup"]
  REG --> ADP["Adapter thin dispatch"]
  ADP --> RUN["storage/run engine modules"]
  RUN --> DB["db engine transport"]

Rules that matter for databases:

  • One dataset → one backend (no dual-write parquet+PG).
  • No if backend == in StorageSession.
  • Bulk writes only for SQL engines (execute_values / COPY / array|jsonb proc) — see postgres architecture.
  • No runtime DDL (CREATE TABLE at run start). Deploy-time codegen/migrate only.
  • Step domain code never imports psycopg / engine clients.
  • PostgreSQL pipeline objects: tables drt_tbl_*, procs drt_proc_*, plus a line in pg_object_inventory.txtnaming.

The step-by-step implementation list (whitelist, inline block, transport, adapters, capability sets, tests) lives here:

Add a dataset backend

Read next after that: