evaluation

pattern: evaluation reads upstream artifacts, keeps a bounded deterministic sample per group, runs registered quality methods, and writes normalized metric-point rows. Sampling identity combines source run and row id, so selection does not depend on read order.

Pattern

pattern: evaluation
execution:
  entry: drtoller.framework.processing.evaluation.run:run_evaluation

evaluation:
  entry: drtoller.framework.processing.evaluation.run:run_evaluation
  upstream_runs_param: metrics.upstream_runs
  sample_size_param: compute.evaluation_sample_size
  jobs:
    - input: vectors
      output: vector_metric_points
      vector_column: vector
      id_column: occurrence_id
      group_column: kind
      artifact_kind: vector_rows
      artifact: occurrence_vectors
      metrics:
        l2_mean: {method: vector_l2_mean}
        hubness: {method: vector_hubness_topk_mean}

metrics.upstream_runs must resolve to at least one run. Empty or dimension-mismatched vectors count as invalid before geometry methods run. Methods that need finite geometry select finite rows internally; methods such as vector_nonfinite_ratio still see the raw sample.

Artifact kinds

The engine (evaluation/engine.py) dispatches by artifact_kind to focused job modules. Method names must match that kind (validated at compile time). Formulas live in algorithms/evaluation/ (one module per kind family).

Module	Role
evaluation/run.py	Compile → upstream runs → bounded jobs
evaluation/engine.py	Dispatch by artifact_kind
evaluation/sample.py	Bounded deterministic heaps
evaluation/vector_job.py	vector_rows
evaluation/cluster_job.py	cluster_rows join + quality
evaluation/graph_job.py	graph_edges
evaluation/profile_job.py	profile_rows
evaluation/filter_job.py	filter_rows
evaluation/stability_job.py	stability_rows
evaluation/layer_job.py	relation / frame / candidate / assertion / constraint / identity / trust / CQ kinds
evaluation/write.py	Persist metric-point rows
evaluation/point_writer.py	Optional corpus-growth PG shape

vector_rows

Sample vector matrices and call per-method evaluators (algorithms/evaluation/vector.py).

Builtin examples: vector_l2_mean, vector_l2_p95, vector_zero_ratio, vector_nonfinite_ratio, vector_anisotropy, vector_hubness_topk_mean, vector_coverage_ratio.

cluster_rows

Sample assignment rows, join them to a vector dataset on id (vectors_input + vector column), then flatten a cluster quality report (algorithms/evaluation/cluster.py).

Required job fields include id_column, label_column, and vectors_input. Optional vector_upstream_runs_param selects which run supplies vectors when it differs from the assignment upstream.

Builtin examples: cluster_inertia, cluster_silhouette_sampled, cluster_davies_bouldin, cluster_calinski_harabasz, size / entropy / gini / empty-cluster summaries, intra/inter distance means.

graph_edges

Sample edge rows (src / dst / optional weight) and flatten a graph quality report (algorithms/evaluation/graph.py).

Builtin examples: node/edge counts, density, WCC/SCC / giant / isolated / leaf ratios, degree stats, clustering coefficient, modularity, PMI-family summaries, hub / retained-mass ratios.

Later-layer kinds (bounded row sample)

Same sample/write contract as vectors. Compile still checks required columns and that each metrics.*.method belongs to that kind.

artifact_kind	What the job scores
profile_rows	Conditional mass: coverage, entropy, sparsity of role/attribute distributions
filter_rows	Keep/drop decisions vs a score or threshold (acceptance, invalid flags)
stability_rows	Named scalar reports aligned across runs (how much the report moved)
relation_rows	Named weighted edges (duplicate ratio, family entropy, support)
frame_rows	Labeled frames / slots
candidate_rows	Proposed items and scores (invalidates bad scores)
assertion_rows	Claim rows (duplicate / unique ratios)
constraint_rows / gap_rows	Hard vs soft severity tokens (soft counted by token, not “anything not hard”)
identity_rows / dedup_rows	Link/dedup pairs (including pair-duplicate ratio)
trust_rows / uncertainty_rows	Trust flags (invalidates bad flags)
cq_rows	Competency-question coverage

Hard/soft severity strings are knobs on EvaluationOptions (hard_severity / soft_severity), not domain literals in the engine.

How sampling stays bounded

• Cap from compute.evaluation_sample_size (param name comes from the plan).
• Deterministic hash heap per optional group_column.
• Rows mark accuracy: exact|sampled from population vs sample sizes.
• Hubness and Silhouette use blocked / sampled geometry — not a full population-by-population matrix.

Extending methods

Register through register_evaluation_method with the correct artifact_kind. Keep formulas data-in / value-out in algorithms/evaluation/. The runner owns join, sample, and write.

Focused kernels that remain Python-only (classification, calibration, generic statistics helpers) must not be declared under a mismatched artifact_kind. New DRTML kinds above are registered — do not treat them as “coming later”.

Output rows

The runner writes run_id, artifact and metric names, value, group key, exact/sampled accuracy, sample/population sizes, invalid-row count, and source run ids. Storage access stays backend-neutral through StorageSession.

Optional corpus-growth observation shape

When the plan declares a separate PostgreSQL observation output, processing/evaluation/point_writer.py maps evaluation rows into corpus-growth-shaped points (corpus_fraction=1.0, order_policy=evaluation by default) so existing Grafana schemas can reuse the table. This is distinct from dataset_metrics point writers.