evaluation

pattern: evaluation reads upstream artifacts, keeps a bounded deterministic sample per group, runs registered quality methods, and writes normalized metric-point rows. Sampling identity combines source run and row id, so selection does not depend on read order.

Pattern

pattern: evaluation
execution:
  entry: drtoller.framework.processing.evaluation.run:run_evaluation

evaluation:
  entry: drtoller.framework.processing.evaluation.run:run_evaluation
  upstream_runs_param: metrics.upstream_runs
  sample_size_param: compute.evaluation_sample_size
  jobs:
    - input: vectors
      output: vector_metric_points
      vector_column: vector
      id_column: occurrence_id
      group_column: kind
      artifact_kind: vector_rows
      artifact: occurrence_vectors
      metrics:
        l2_mean: {method: vector_l2_mean}
        hubness: {method: vector_hubness_topk_mean}

metrics.upstream_runs must resolve to at least one run. Empty or dimension-mismatched vectors count as invalid before geometry methods run. Methods that need finite geometry select finite rows internally; methods such as vector_nonfinite_ratio still see the raw sample.

Artifact kinds

The engine (evaluation/engine.py) dispatches by artifact_kind to focused job modules. Method names must match that kind (validated at compile time). Formulas stay in algorithms/evaluation/{vector,cluster,graph}.py.

Module	Role
evaluation/run.py	Compile → upstream runs → bounded jobs
evaluation/engine.py	Dispatch by artifact_kind
evaluation/sample.py	Bounded deterministic heaps
evaluation/vector_job.py	vector_rows
evaluation/cluster_job.py	cluster_rows join + quality
evaluation/graph_job.py	graph_edges
evaluation/write.py	Persist metric-point rows
evaluation/point_writer.py	Optional corpus-growth PG shape

vector_rows

Sample vector matrices and call per-method evaluators (algorithms/evaluation/vector.py).

Builtin examples: vector_l2_mean, vector_l2_p95, vector_zero_ratio, vector_nonfinite_ratio, vector_anisotropy, vector_hubness_topk_mean, vector_coverage_ratio.

cluster_rows

Sample assignment rows, join them to a vector dataset on id (vectors_input + vector column), then flatten a cluster quality report (algorithms/evaluation/cluster.py).

Required job fields include id_column, label_column, and vectors_input. Optional vector_upstream_runs_param selects which run supplies vectors when it differs from the assignment upstream.

Builtin examples: cluster_inertia, cluster_silhouette_sampled, cluster_davies_bouldin, cluster_calinski_harabasz, size / entropy / gini / empty-cluster summaries, intra/inter distance means.

graph_edges

Sample edge rows (src / dst / optional weight) and flatten a graph quality report (algorithms/evaluation/graph.py).

Builtin examples: node/edge counts, density, WCC/SCC / giant / isolated / leaf ratios, degree stats, clustering coefficient, modularity, PMI-family summaries, hub / retained-mass ratios.

How sampling stays bounded

• Cap from compute.evaluation_sample_size (param name comes from the plan).
• Deterministic hash heap per optional group_column.
• Rows mark accuracy: exact|sampled from population vs sample sizes.
• Hubness and Silhouette use blocked / sampled geometry — not a full population-by-population matrix.

Extending methods

Register through register_evaluation_method with the correct artifact_kind. Keep formulas data-in / value-out in algorithms/evaluation/. The runner owns join, sample, and write.

Focused kernels that are not yet DRTML-bound (classification, calibration, statistics, generic bindings) remain callable Python APIs only — do not declare them under a mismatched artifact_kind.

Output rows

The runner writes run_id, artifact and metric names, value, group key, exact/sampled accuracy, sample/population sizes, invalid-row count, and source run ids. Storage access stays backend-neutral through StorageSession.

Optional corpus-growth observation shape

When the plan declares a separate PostgreSQL observation output, processing/evaluation/point_writer.py maps evaluation rows into corpus-growth-shaped points (corpus_fraction=1.0, order_policy=evaluation by default) so existing Grafana schemas can reuse the table. This is distinct from dataset_metrics point writers.