Feed reference

Feed configuration lives on input datasets. The compact v4 form lowers into the existing StepFeedPlan.

Primary feed

inputs:
  sentences:
    run: mapping.upstream_runs
    schema: {doc_id: string, sent_id: int64, text: string}
    feed:
      batch: 4096
      group_by: [doc_id]
      group_scope: shard
      collect: all

Field	Meaning
batch	rows per read batch
mode	batch (default) or corpus_scan
corpus: true	shorthand for corpus scan
select	optional column projection
group_by	partition grouping columns
group_scope	shard (within part file) or run (across all parts)
collect	collect policy (all, …)

Semantics that must not change:

• group_scope: shard + group_by → grouping inside one shard.
• group_scope: run + group_by → grouping across all part files; partition_id has no shard.
• corpus: true requires an explicit feed; framework does not invent domain columns.

Parquet primary feeds require run: (run-id param).

Attach feed

inputs:
  lemma_index:
    as: lemmas
    run: mapping.upstream_runs
    schema: {doc_id: string, sent_id: int64, lemma: string}
    attach:
      to: sentences
      on: [doc_id, sent_id]
      scope: primary_unit

as: becomes the processor attach alias. Attach datasets usually need a routing index on the join keys.