Skip to content

Add an algorithm method

Use this recipe for embedding, clustering, and other pure kernels under framework/algorithms/.
For dataset metric formulas (metric_defs.method), use New metric method instead.

Framework owns Domain / step owns
Math kernel, DTO result, method registry When to call, probe loops, K choice, thresholds
compute.*ComputePlan kwargs Mapping results → logical ProcessResult rows
No I/O, no Prometheus, no DRTML parse No sklearn/torch reimplementation of a registered method

DRTML configures methods; it does not auto-execute arbitrary algorithms by name.

Family Package Registry
Embedding algorithms/embed/ register_embed_method
Clustering algorithms/cluster/ register_cluster_method
Evaluation formulas algorithms/evaluation/ register_evaluation_method
Graph / profiles / information / stability matching package public package API (no root facade yet)

Do not put kernels in processing/ or in a step package.

Example: a second cluster method.

algorithms/cluster/my_kmeans_v1.py
from typing import Any, Mapping
import numpy as np
from drtoller.framework.algorithms.types import ClusterResult
def run_my_kmeans_v1(
vectors: np.ndarray,
row_ids: list[str],
*,
k: int,
seed: int = 0,
) -> ClusterResult:
# pure computation only
...
return ClusterResult(
method_id="my_kmeans_v1",
k=k,
row_ids=list(row_ids),
labels=labels,
inertia=inertia,
)
def my_kmeans_v1_from_params(
vectors: np.ndarray,
row_ids: list[str],
params: Mapping[str, Any],
) -> ClusterResult:
return run_my_kmeans_v1(
vectors,
row_ids,
k=int(params["k"]),
seed=int(params.get("seed", 0)),
)

Rules:

  • Accept arrays / DTOs; return a typed result (ClusterResult, EmbedResult, …).
  • No StorageSession, no files, no Prometheus.
  • No domain column names (doc_id, lemma keys, …) inside the kernel.
  • Numeric knobs that steps tune must eventually map from compute.* (see §4); do not invent silent fallbacks that diverge from DRTML defaults.
# algorithms/cluster/__init__.py (or package import side-effect)
from drtoller.framework.algorithms.registry import register_cluster_method
from drtoller.framework.algorithms.cluster.my_kmeans_v1 import my_kmeans_v1_from_params
register_cluster_method("my_kmeans_v1", my_kmeans_v1_from_params)

Embedding:

from drtoller.framework.algorithms.registry import register_embed_method
register_embed_method("my_embed_v1", my_embed_v1_from_params)

Call sites stay registry-based:

from drtoller.framework.algorithms import cluster_with_k
result = cluster_with_k(
vectors, row_ids, chosen_k,
method_id="my_kmeans_v1",
params={"seed": 42},
)

4. Wire compute.* (when the method is selected from DRTML)

Section titled “4. Wire compute.* (when the method is selected from DRTML)”
  1. Add keys to step params_defaults / UI (namespace compute.*).
  2. Extend ComputePlan / *_algorithm_kwargs in drtml/models/compute/ if new knobs are needed.
  3. Domain processor (or mode) reads the plan and passes method_id + kwargs into the registry API.
params_defaults:
compute.cluster_method: my_kmeans_v1
compute.cluster_seed: 42

Do not add framework policy params such as hidden compute.cluster_k when K selection is domain responsibility.

Caveat: production embed_train currently hard-requires sgns_v1 through SgnsVocabTrainer. Registering another embed method is enough for embed_rows, not automatically for embed_train.

Graph / profile / information / stability kernels are imported from their packages:

from drtoller.framework.algorithms.graph import pmi_family
from drtoller.framework.algorithms.information import delta_h

Adding one of these:

  1. Implement the pure function in the package.
  2. Export it from the package __init__ / public module.
  3. Document inputs/outputs in catalog.
  4. Do not invent a new root algorithms.* facade method unless the family already uses one.

Minimum:

  • unit: empty input, invalid k / dims, deterministic seed;
  • registry: unknown method_id raises with known names;
  • kwargs: *_from_params matches DRTML / ComputePlan mapping;
  • optional integration: processor or mode calls the method and writes logical rows via ProcessResult (framework write path).
  • catalog — method id, inputs, outputs, params;
  • invocation — if compute.* surface changes;
  • framework/FILES.md when a new module lands.