Storage overview

Role

framework/storage/ owns all run artifact I/O: parquet parts, manifests, feed WorkUnits, registry store, corpus scan.
Steps do not import storage (except framework tests).

Layers (bottom-up)

paths, plan, compression, backends/ (local|s3)
  → dataset/backend_registry (+ builtins)
  → dataset/access, reader, writer, wide_row, row_materialize
  → write_buffer*, routing_index/, manifest
  → run/session (+ session_read/write, part_rotation, write_lanes)
  → feed/ (StepFeedPlan → WorkUnit)

Logical rows

Only place for batch/table → dict: dataset/row_materialize.py (+ expand_wide_lists).
Do not copy batch.to_pylist() into mapping/reduce/feed logical paths.

Wide lists

DRTML wide_lists / inline col[N] → wire columns on write (embed_wide_lists), expand on read.
Truncate/wire-limits stay in storage, not in the step.

Public API

storage.run.StorageSession
storage.feed: iter_feed_work_units, WorkUnit, count_feed_work_units (from shard_plan)