Parallelism
Two orthogonal channels
Section titled “Two orthogonal channels”| Namespace | Meaning |
|---|---|
runtime.parallelism.* |
execution worker dispatch (inprocess / processes / ray) |
compute.parallelism.* |
algorithm-internal threads/processes |
The step does not pick a backend in Python — only DRTML / UI parallelism_controls.
partition_loop parallel
Section titled “partition_loop parallel”- Driver assigns primary shard metadata.
- Each worker reconstructs session/processor context and streams its shards.
- Driver merges worker manifests and metric deltas.
- Checkpoint/resume with
workers > 1is unsupported. - Run-global mutable registries are not auto-merged; prefer partial rows +
reduce_merge.
embed_train parallel
Section titled “embed_train parallel”Transport backends share ParallelPlan, but this is not partition shard mapping. Current Ray train uses one mutable-W actor per material group with bounded pair updates. runtime.parallelism.workers is metadata for that path today; do not assume N train workers are created unless the active driver path is updated.