Validation and compile

Validation layers

1. YAML root must be a mapping.
2. include expands with path/merge validation, existence checks and cycle detection.
3. Typed Pydantic models validate known sections.
4. Version validators enforce runner/entry, v2/v3 contracts and selected cross-field rules.
5. Compile parsers validate storage/feed/mapping/reduce/compute/embed backend constraints.
6. Cross-section compile checks link metrics to metric_defs, attach feeds to routing indexes, and embed datasets/columns.

New cross-field checks belong in manifest_validate/ or the owning parser, not as ad-hoc runtime guards.

Compile

compile_step_manifest(manifest, merged_params=…) builds runtime plans:

Plan	Source
StoragePlan	datasets + bindings + flush/checkpoint
MappingPlan	mapping / mapping_read
ReduceMergePlan	reduce_merge pattern
ComputePlan	compute.* params
StepFeedPlan	datasets.*.feed
embed plans	embed_train / embed_infer

Compile does not open data files or write parquet.

Backend whitelist

parquet | postgres | qdrant | neo4j

neo4j is accepted at parse time; the builtin adapter raises NotImplementedError until transport exists.

Known strictness gaps

Document these honestly when authoring contracts:

• Root StepManifest uses extra="allow". Unknown root sections can survive load until a parser/consumer exists.
• Strict UI and metrics.views.panels validators currently run on the v2 path; v3 still gets typed metrics.prometheus validation, but does not apply the same panel/UI strictness.
• Dataset materialize may appear in YAML and is stripped by the datasets bridge; it is not a StoragePlan field.
• storage.resume is typed in the plan, but the storage parser does not currently enforce a whitelist at parse time.
• mapping_read, attach/routing indexes and fixed-width wide lists are implemented and tested, but may not have many production YAML examples yet.

A field that loads but has no compiled plan or runtime consumer is not a supported feature.