Feed reference

datasets.*.feed declares how input datasets become processor work. Parse lives in drtml/models/feed; runtime iteration lives in storage/feed.

Complete primary + attach example

datasets:
  documents:
    role: input
    binding: artifacts
    columns:
      document_id: string
      language: string
      text: string
    run_id_param: mapping.upstream_runs
    feed:
      loop: primary
      read:
        mode: batch
        batch_rows: 2048
        columns: [document_id, language, text]
      partition:
        source: rows
        group_by: [language]
        collect: all

  document_metadata:
    role: input
    binding: artifacts
    columns:
      document_id: string
      source: string
    indexes:
      document_id:
        kind: routing
        columns: [document_id]
    feed:
      loop: attach
      read:
        mode: scan
      attach:
        to: documents
        join_on: [document_id]
        route_by: document_id
        scope: primary_unit

Exactly one dataset must have loop: primary. Any number may use loop: attach; each attach must point to the primary.

Loop kinds

• primary: produces WorkUnits and requires read; partition defaults to {source: file}.
• attach: joins rows to each primary WorkUnit. read defaults to scan if omitted.
• none: no feed. It cannot declare read, partition, or attach.

Read modes

• materialize: materialize the selected input.
• batch: bounded record batches; requires batch_rows >= 1.
• scan: scan logical rows.
• corpus_scan: corpus reader path rather than parquet dataset iteration.

columns projects read columns. For batch, batch_rows is mandatory.

Partition sources

source: file

Each physical primary shard is an independent partition domain. With group_by, groups are formed inside each shard. The same group key in two files produces two WorkUnits.

source: rows

Rows are grouped across all parts of the selected run. With group_by, one logical group may span many files. The partition id is group-derived and has no physical shard identity.

collect currently supports all.

Attach routing

Attach feeds never fall back to full shard scans. The route column is:

1. attach.route_by, if specified;
2. otherwise the first attach.join_on column.

The attach dataset must declare indexes.<route_column>. The runtime loads RoutingIndexLookup once through the attach store cache, fetches exact row locations, materializes logical rows, and joins them to the primary unit.

ProcessorBatch mapping

processing.feed_attach maps physical dataset ids to stable logical attach keys:

processing:
  processor: acme_steps.enrich.impl:process_batch
  batch_processor: acme_steps.enrich.impl:process_batch
  feed_attach:
    metadata: document_metadata

The processor reads batch.primary and batch.attach["metadata"]; it does not access storage.

Upstream run selection

run_id_param names the merged param containing upstream run ids. Default: mapping.upstream_runs. The framework resolves it; steps do not construct object keys.

Parallel execution

For multi-worker partition_loop, the driver enumerates primary shard metadata and assigns shards to workers. Workers run the same feed executor under a shard_scope. Attach lookups remain local to each worker. source: rows and checkpoint constraints may require inprocess execution; invalid combinations fail explicitly.