Algorithm catalog
Framework algorithms are data in → result out. They do not open datasets, pick K, or decide workflow. DRTML’s compute.* only configures method ids and knobs; a processor or execution mode chooses when to call them.
That split keeps phase recipes and domain policy out of generic runtime: registering a formula does not automatically make it a production path.
Families at a glance
Section titled “Families at a glance”| You need… | Start here |
|---|---|
| Train or apply token embeddings | Embedding methods + embed_train / pooling helpers |
| Cluster vectors at small scale | minibatch_kmeans_v1 via cluster_with_k |
Cluster a full corpus without N×D in RAM |
Same math via streaming cluster |
| Try several K and pick a winner | Probe loop (domain decide) + cluster/eval formulas |
| Offline quality on vectors / clusters / graphs | evaluation + evaluation modules below |
| Live / offline registry growth curves | Dataset metric methods (Python + Postgres dual impl) |
| Graph association, profile similarity, entropy, stability | Package APIs under algorithms/graph, profiles, information, stability |
Public dispatch for embed/cluster:
from drtoller.framework.algorithms import embed_rows, cluster_vectors, cluster_with_kRegistries: register_embed_method, register_cluster_method, dataset register_method / resolve_method. Graph/profile/… import from their packages, not the root facade.
Embedding
Section titled “Embedding”sgns_v1 (production)
Section titled “sgns_v1 (production)”NumPy skip-gram with negative sampling over MaterialRow.token_ids. Supports train/infer modes, warm checkpoints, and a serializable checkpoint in EmbedResult.
Typical knobs: dim, epochs, lr, negative_samples, seed. Production embed_train currently hard-requires this method (single-table trainer or Ray parameter-server shards) — registering another embed method is not enough for that mode alone.
stub_v1
Section titled “stub_v1”Deterministic smoke embedder for tests. Do not use in production.
Helpers you will see nearby
Section titled “Helpers you will see nearby”mean_pool_rows/mean_pool_rows_batch— occurrence mean-pool over a vocab matrix (embed_infer).geometry_metrics— L2 and blockwise hubness after train (never a full V×V similarity matrix).SgnsVocabTrainer/sgns_ps_step— mutable W for train transports.
Clustering
Section titled “Clustering”minibatch_kmeans_v1
Section titled “minibatch_kmeans_v1”Pure NumPy mini-batch k-means, blockwise assignment so temporary distance memory stays O(block_rows × k × d).
Knobs: k (domain policy), batch_size, max_iter, seed, assign_block_rows. Output: ClusterResult with labels and inertia.
The in-memory API still wants a full matrix. Corpus-scale runs reuse the same centroid helpers (centroid_stats) through streaming fit/assign. Domain owns candidate K and probe decide.
Graph, profiles, information, stability
Section titled “Graph, profiles, information, stability”Neutral kernels (no domain column names). Caller defines nodes, edges, keys, and thresholds.
| Package | For |
|---|---|
graph/ |
PMI family, filters, connectivity (WCC/SCC), degree, clustering coefficient, modularity |
profiles/ |
Normalize / conditionals; cosine, Jaccard, overlap, JS |
information/ |
Entropy, conditional entropy, Delta-H |
stability/ |
Stable subsample, shared-key align; Jaccard / ARI / NMI / Spearman |
Offline evaluation formulas
Section titled “Offline evaluation formulas”Used by pattern: evaluation when the job’s artifact_kind matches:
| Module | Answers questions like… |
|---|---|
vector.py |
Are vectors finite? How large? How hubby? |
cluster.py |
Is this clustering compact / separated / balanced? |
graph.py |
How dense? How connected? Association strength? |
statistics / classification / calibration / bindings |
Generic stats and ratios — mostly Python APIs until more artifact kinds land |
Wired kinds today: vector_rows, cluster_rows, graph_edges. Details: evaluation.
Dataset metric methods (corpus / registry)
Section titled “Dataset metric methods (corpus / registry)”These back metric_defs and offline dataset_metrics jobs. Prefer dual Python + PostgreSQL implementations; see Dataset metrics.
Cardinality / assignment — how big is the key set, how much is new, how homogeneous:
total_mass, count_distinct, assignment_homogeneity, assignment_saturation, assignment_new_count, assignment_window_new_count, assignment_count.
Distribution — shape of frequencies (singletons, entropy, Zipf, top-mass shares, …):
singleton_ratio, repeated_ratio, frequency_*, entropy_normalized, effective_cardinality, frequency_gini, top_*_mass_share, aggregation_ratio, effective_sample_size, zipf_alpha, zipf_fit_r2.
Thresholds — counts/coverage above or below a param:
count_ge_threshold, count_lt_threshold, key_coverage_ge_threshold, mass_coverage_ge_threshold.
Growth — vocabulary growth exponent and fit:
vocabulary_growth_beta, vocabulary_growth_fit_r2.
Graph / group compare (offline jobs with the right columns):
graph_degree_*, graph_leaf_node_ratio, graph_density, group_jaccard, group_weighted_overlap, group_js_divergence, pair_reciprocity_ratio.
Vector snapshots (often Qdrant/geometry keys):
vector_l2_mean, vector_l2_p95, vector_hubness_topk_mean.
Cross-dataset consistency:
related_key_coverage, related_retained_mass_ratio, single_value_consistent, related_value_consistent.
Production wiring vs “it is registered”
Section titled “Production wiring vs “it is registered””| Capability | Callable as | What actually runs in prod |
|---|---|---|
sgns_v1 |
embed_rows / trainer |
embed_train mode (trainer or Ray PS) |
stub_v1 |
embed_rows |
tests only |
minibatch_kmeans_v1 |
cluster_with_k |
small matrices; large runs → streaming cluster |
| dataset metrics | resolve_method |
checkpoint snapshots + offline jobs |
qdrant_metric field |
registry capability | specific geometry/Qdrant consumers — not a generic dispatcher |
Types and defaults
Section titled “Types and defaults”Stable DTOs: MaterialRow, EmbedResult, ClusterResult. Behavioral defaults for real runs belong in step params_defaults; Python fallbacks are for unit tests without a manifest.