Skip to content

July 2026

Product notes for July 2026. Later months: August 2026, September 2026. Index: What’s new.

  • Probe is a product loop. The runtime tries candidates; the step chooses the winner. Nested evaluate shares the parent’s workers instead of multiplying them. Final publish uses the full stream; trials can run on a subset.
  • Cluster at corpus scale. Fit and assign stream vector batches. Peak RAM is the model plus one batch, not the full matrix. The same path serves many probe K values once sticky cache is on (see August).
  • Evaluation covers clusters and graphs, not only vector geometry: quality of assignments, topology of edges.
  • Dashboards show probe phase while the first prepare is still running, so Grafana is not empty for the long opening stretch.

Details: Probe loop, Streaming cluster, evaluation, Grafana.

  • Embed train on many workers is real. Weights shard across workers; each batch updates the shards that own those rows. A single shared table is no longer pretending to be distributed.
  • Large keyed joins shuffle, then reduce per bucket. Producers write stable buckets; reduce sums each bucket in parallel. That is the scale-out path, not “one task per dataset”.
  • Reads stream. Production scans do not load a whole run into RAM. Optional grouped sums accumulate on the stream.
  • Occurrence pooling is batched, so embed infer does not walk the vocab one row at a time.

Details: embed_train, embed_infer, reduce_merge, StorageSession.

  • Corpus-growth curves you can plot. Offline metrics jobs distinguish a final exact point, hashed buckets, and document-progress milestones. Grafana XY uses corpus fraction vs the metric.
  • The same observation table can take embed and evaluation points, so dashboards stay on one shape.
  • Postgres migrate is boring on purpose. Managed objects follow a fixed naming scheme; leftovers from old schemas are dropped. Dashboards keep calling the same procedures.

Details: dataset_metrics, Grafana, PostgreSQL.