Feed and WorkUnit
Parse vs runtime
Section titled “Parse vs runtime”| Layer | Where |
|---|---|
Parse StepFeedPlan |
drtml/models/feed/ |
| Runtime iteration | storage/feed/ |
WorkUnit
Section titled “WorkUnit”Unit of work for partition_loop: primary rows (+ attach join when needed).
The processor sees already materialized logical dicts.
Partition semantics (do not break)
Section titled “Partition semantics (do not break)”partition.source=file+group_by— groups inside one shard.partition.source=rows+group_by— groups across all part files;partition_idwithout shard.corpus_scan— explicitpartitionin DRTML; framework does not invent domain columns.
Modules
Section titled “Modules”| File | Role |
|---|---|
shard_plan.py |
primary shards + work-unit counts (one pass) |
shard_scope.py |
filter shards for a worker |
executor.py |
iteration API only |
attach_store.py |
join cache; RoutingIndexLookup.load only here |
keys.py |
join/group/partition keys |
Multi-worker
Section titled “Multi-worker”Driver assigns primary shard metadata round-robin; the worker reads feed locally with shard_scope.
Shard counting lives in shard_plan, not processing/parallel/.