Algorithm catalog

Framework algorithms are data-in/result-out primitives. DRTML supplies compute.* configuration; a domain processor or a framework execution mode chooses when to invoke an algorithm.

DRTML does not dynamically execute arbitrary algorithms by name on its own. That separation keeps phase recipes, candidate search and domain policy out of generic runtime.

Public dispatch API

from drtoller.framework.algorithms import embed_rows, cluster_vectors, cluster_with_k

Method registries:

• embedding: register_embed_method, get_embed_method
• clustering: register_cluster_method, get_cluster_method
• dataset metrics: register_method, resolve_method

Embedding methods

sgns_v1

NumPy CPU skip-gram with negative sampling over MaterialRow.token_ids. It can train and infer, infer from an existing vocabulary/checkpoint, and return a serializable checkpoint.

Inputs:

• list[MaterialRow]
• method params: dim, epochs, lr, negative_samples, seed
• optional mode (train_infer / infer behavior)
• optional warm checkpoint and SGNS train options

Output: EmbedResult(method_id, row_ids, vectors, checkpoint).

stub_v1

Deterministic smoke-test embedder. It hashes integer token ids into a seeded random basis and averages per row. Parameters: dim (default 8), seed (default 0). Do not use as a production embedding method.

Clustering methods

minibatch_kmeans_v1

Pure NumPy mini-batch k-means using squared L2 assignment.

Parameters:

• k (required at policy level; effective K is min(k, n_rows))
• batch_size (implementation fallback 256)
• max_iter (implementation fallback 20)
• seed (implementation fallback 0)

Output: ClusterResult(method_id, k, row_ids, labels, inertia).

The framework supplies the implementation; domain code owns candidate K generation/probing and final K choice.

Embedding geometry diagnostics

Pure NumPy helpers in algorithms/embed/geometry_metrics.py:

• vector_l2_stats(embeddings) → mean and p95 L2 norm.
• frequent_token_indices(vocab_index, token_counts, top_tokens) → selected rows.
• hubness_topk_mean_for_indices(embeddings, indices, neighbor_k, block_cols) → mean top-k cosine score with blockwise memory use; never materializes V×V similarities.

These diagnostics are used after training. Qdrant-backed vector snapshots expose corresponding dataset metric names.

Vocabulary growth

algorithms/vocabulary_growth.py contains vocabulary-growth estimation used by dataset metric methods:

• growth exponent/beta
• fit quality (R²)
• state derived from assignment/new-key observations

Expose it through vocabulary_growth_beta and vocabulary_growth_fit_r2 metric methods rather than duplicating formulas in steps.

Dataset metric methods

Exact cardinality / assignment methods

• total_mass
• count_distinct
• assignment_homogeneity
• assignment_new_count
• assignment_count

Distribution methods (PostgreSQL snapshot path)

• singleton_ratio
• repeated_ratio
• frequency_mean
• frequency_p50
• frequency_p90
• frequency_p99
• entropy_normalized
• effective_cardinality
• frequency_gini
• top_1pct_mass_share
• top_10pct_mass_share
• aggregation_ratio

Threshold methods

• count_ge_threshold
• count_lt_threshold
• key_coverage_ge_threshold
• mass_coverage_ge_threshold

These require metric_defs.<name>.threshold_param and support checkpoint approximate/final exact modes according to registry capabilities.

Growth methods

• vocabulary_growth_beta
• vocabulary_growth_fit_r2

They require novelty/growth state and accept sampling controls.

Vector methods

• vector_l2_mean
• vector_l2_p95
• vector_hubness_topk_mean

The current catalog maps these to Qdrant/geometry snapshot keys rather than a generic row-wise Python formula.

Cross-dataset consistency and coverage

• related_key_coverage
• related_retained_mass_ratio
• single_value_consistent
• related_value_consistent

Related methods run as final exact metrics and may require a related dataset relation.

Candidate selection helper

algorithms.arbiter currently exposes select_best_candidate / select_best_k over probe rows using SelectionModel(metric, direction, tie_break, gates). Treat it as a low-level helper: the candidate loop, downstream probe and policy remain domain responsibilities and are not a generic DRTML phase recipe.

There is currently no production execution mode that calls the arbiter. It is covered by unit tests and intended for domain-owned probe loops.

Production wiring vs public API

Registration alone does not make a method a production path.

Capability	Public API	Current production wiring
sgns_v1	embed_rows / trainer	execution.mode: embed_train uses SgnsVocabTrainer directly and hard-requires sgns_v1
stub_v1	embed_rows	tests / smoke only
minibatch_kmeans_v1	cluster_vectors / cluster_with_k	no builtin cluster mode yet; call from a domain processor or custom entry
arbiter helpers	select_best_candidate	tests / domain policy only
dataset metrics with python / postgres_proc_metric	resolve_method	partition-loop dataset metric snapshots
vector methods with qdrant_metric	registry capability field	embed-train/Qdrant geometry path computes the same values; generic metric dispatcher does not currently read qdrant_metric

Adding a new embed method therefore requires both registry registration and, for embed_train, an explicit mode/backend contract change. Adding a Qdrant metric name requires a consumer path, not only a registry field.

Supporting types/utilities

• MaterialRow: row id + token ids + optional center/dependency/head context.
• EmbedResult, ClusterResult: stable algorithm result DTOs.
• BatchView, material helpers: bounded conversion/selection utilities used by processing code.
• SgnsVocabTrainer: mutable in-memory SGNS state used by embed_train transports.

Defaults

Behavioral defaults for a real run belong in the step’s params_defaults. Implementation fallbacks exist for isolated unit tests; do not rely on them as hidden production policy.