Skip to content
DrToller.Framework

Storage overview

For a step author, storage is invisible on purpose. You return logical row-dicts; the framework decides where they land (parquet part, Postgres upsert, Qdrant point) and how they are read back on the next run.

framework/storage/ owns that I/O: parts and manifests, feed work units, write buffers, shuffle buckets, corpus scan. Steps do not open parquet or call session.write themselves. The one intentional boundary DTO is ProcessorBatch from the public storage.feed package — already-materialized rows handed to the processor. List runs with storage/run/list_runs.py via drtoller ls. Per-run cleanup is storage/run/clear_run.py via drtoller clear (--run-id or --all; --include-metrics for sibling metrics outputs).

What you should expect

Section titled “What you should expect”
  • Logical rows — the only place Arrow/table becomes a domain dict is row_materialize (plus wide-list expand). Do not to_pylist() on hot mapping/reduce/feed paths.
  • One dataset → one backend — parquet or postgres or qdrant (neo4j stub). No dual-write.
  • Stream by default — iter_parquet_run_rows / iter_run_rows for production; materializing read_* helpers are for small fixtures. Geometry modes use iter_vector_batches (bounded numpy windows), not a full N×D load. See StorageSession.
  • Early shuffle — parquet outputs may declare shuffle: { by, buckets } so reduce can SUM per bucket instead of one giant combiner. Parts fill to the size limit while writing; no post-hoc compact. See Datasets.
  • Wide lists — DRTML wide_lists / col[N] expand on write and collapse on read inside storage; wire limits stay out of the step.

How the package is stacked (for extenders)

Section titled “How the package is stacked (for extenders)”
paths, plan, compression, backends/ (local|s3)
  → dataset/backend_registry (+ builtins)
  → access, reader, writer, wide_row, row_materialize
  → write_buffer*, routing_index/, manifest
  → run/session (+ read/write, part_rotation, write_lanes, vector_batch_iter)
  → feed/ (StepFeedPlan → WorkUnit)

You usually touch the top (StorageSession, feed iterators) or add a backend adapter — not a third ad-hoc tempfile path.

Public entry points

Section titled “Public entry points”
  • storage.run.StorageSession
  • storage.feed: ProcessorBatch, iter_feed_work_units, WorkUnit, count_feed_work_units
Section titled “Related”