Validation and compile

Validation layers

1. YAML root must be a mapping.
2. include expands path/glob fragments with existence checks and cycle detection.
3. Public v4 source is lowered by the focused source_v4/ package into the internal StepManifest.
4. Typed Pydantic models validate known lowered sections.
5. Internal validators enforce runner/entry and cross-field rules.
6. Compile parsers validate storage/feed/mapping/reduce/dataset-metrics/evaluation/compute/embed constraints.
7. Cross-section compile checks link metrics to metric_defs, attach feeds to routing indexes, and embed datasets/columns.

Failures raise DrtmlValidationError via raise_validation_error (structured code / path / available / fixes). Prefer validate --format json or validate_manifest_path() for agents — see Semantic validation (includes the stable-code table).

Compile and mode entrypoints (dispatch, mapping, reduce_merge, embed_train mode_spec, feed/checkpoint/metrics/UI) use the same structured errors — not only manifest_validate/.

New cross-field checks belong in manifest_validate/ or the owning parser, not as ad-hoc runtime guards.

Versioned validation is split by concern under manifest_validate/v3/ (core, dataset_metrics, evaluation, mapping, reduce_merge). Extend the focused validator; do not recreate a monolithic v3.py.

For run.type: dataset_metrics, validation checks:

• referenced inputs/outputs and their roles;
• non-empty jobs, keys, and metrics;
• document_counts → milestone mode + PostgreSQL executor/input + required schema columns;
• PostgreSQL executor → declared PostgreSQL output;
• Python executor → parquet/CSV job outputs.

Compile

compile_step_manifest(manifest, merged_params=…) builds runtime plans:

Plan	Source
StoragePlan	stores + inputs/outputs
MappingPlan	run.type: mapping / lookups
ReduceMergePlan	run.type: reduce
DatasetMetricsPlan	run.type: dataset_metrics
EvaluationPlan	run.type: evaluation
ComputePlan	params.compute
StepFeedPlan	inputs.*.feed / attach
embed plans	run.type: embed_train / run.infer

Compile does not open data files or write parquet.

Backend whitelist

parquet | postgres | qdrant | neo4j

neo4j is accepted at parse time; the builtin adapter raises NotImplementedError until transport exists.

Known gaps

• Extra root keys on selected integration blocks are still allowed after lowering for versioned extensions.
• A field is supported only when a parser/compiler/runtime consumer exists.