Skip to content

Feed and WorkUnit

Layer Where
Parse StepFeedPlan drtml/models/feed/
Runtime iteration storage/feed/

Unit of work for partition_loop: primary rows (+ attach join when needed).
The processor sees already materialized logical dicts.

  • partition.source=file + group_by — groups inside one shard.
  • partition.source=rows + group_by — groups across all part files; partition_id without shard.
  • corpus_scan — explicit partition in DRTML; framework does not invent domain columns.
File Role
shard_plan.py primary shards + work-unit counts (one pass)
shard_scope.py filter shards for a worker
executor.py iteration API only
attach_store.py join cache; RoutingIndexLookup.load only here
keys.py join/group/partition keys

Driver assigns primary shard metadata round-robin; the worker reads feed locally with shard_scope.
Shard counting lives in shard_plan, not processing/parallel/.