Official OSS steps

step01 and step01_metrics are the first official open-source pipeline steps shipped alongside DrToller.Framework.

They solve a real pipeline problem—turning an unstructured text corpus into a stable linguistic dataset and measuring whether that dataset is large and diverse enough—while serving as executable reference implementations of framework contracts.

Why publish the steps

Framework APIs are easier to understand when users can inspect a production-grade step that uses them end to end:

• DRTML v4 declares inputs, outputs, parameters, UI, orchestration, and observability;
• domain Python only transforms supplied documents into logical rows;
• framework storage owns corpus reading, parquet/CSV writing, PostgreSQL upsert, flush, and checkpoint;
• framework processing owns partition execution and offline dataset metrics;
• Streamlit and Airflow discover both steps directly from their manifests;
• Grafana dashboards are generated from DRTML rather than committed as JSON.

The steps are official examples, but their domain modules do not become dependencies of framework core. The boundary remains:

DrToller.Framework       runtime, storage, processing, algorithms, telemetry
Official OSS steps       DRTML contracts + NLP transformation policy
DrToller.Platform        Docker services, Airflow, Prometheus, Grafana

Why two steps

Extraction and analysis have different lifecycles:

text corpus
  → step01
      → document records
      → sentence/token/dependency records
      → run vocabulary
      → per-document lemma evidence
  → step01_metrics
      → exact final statistics
      → deterministic corpus-growth curves
      → CSV + PostgreSQL metric points
      → generated Grafana dashboard

Keeping metrics separate means:

• expensive spaCy parsing runs once;
• metric formulas and dashboard layouts can evolve independently;
• final statistics can be recalculated without reparsing text;
• milestone size and deterministic ordering seed can change without rerunning extraction;
• multiple Step01 runs can be evaluated together through metrics.upstream_runs.

What the pair demonstrates

Step01

• corpus-scan feed;
• batch spaCy processing;
• tenant-scoped deterministic lemma identifiers;
• nested sentence records encoded as logical list columns;
• parquet document/sentence datasets;
• PostgreSQL registry accumulation;
• durable per-document evidence;
• live progress, CPU/RAM, throughput, and spaCy timing.

Step01 Metrics

• DRTML run.type: dataset_metrics;
• direct exact PostgreSQL final metrics;
• document-milestone corpus-growth evaluation;
• dual CSV/PostgreSQL metric-point outputs;
• Prometheus publication from computed points;
• PostgreSQL-backed Grafana stats and growth curves.

Scope

This section documents only:

• step01 — corpus linguistic extraction;
• Step01 data contract;
• step01_metrics — offline quality and growth analysis.

Downstream descriptor, mapping, embedding, and other pipeline steps are intentionally outside this section.

Start here

1. Install the framework, platform stack, spaCy, and the configured model.
2. Run Step01 against a Step00 corpus run or an explicit corpus URI.
3. Keep the resulting Step01 run id.
4. Run Step01 Metrics with that id in final or milestones mode.

For deployment commands, see Deploy. For the architectural rules used by these steps, see Step ↔ framework.