Step01 — linguistic extraction

step01 converts a text corpus into reusable document-, sentence-, vocabulary-, and evidence-level datasets.

It is a DRTML v4 partition step:

drtml: 4
step: step01
schema: step01_corpus_v4
run:
  type: partition

The step receives CorpusDocument objects from framework corpus storage. Domain code never opens source files or writes output datasets.

Processing flow

corpus scan (framework)
  → document batch
  → spaCy nlp.pipe
  → token + dependency extraction
  → noun-chunk extraction
  → sentence packing
  → ProcessResult logical rows
  → framework parquet / PostgreSQL sinks

1. Corpus resolution

The source is selected in this order:

1. explicit storage.corpus_uri supplied by the user;
2. URI resolved from domain.step00_run_id;
3. deployment fallback DRTOLLER_DEFAULT_RAW_CORPUS_URI, unless S3 mode is enabled.

When domain.step00_run_id is used, the framework path helper resolves the Step00 docs corpus under the configured user/project/run layout.

The _corpus input is a corpus-scan feed grouped by doc_id across the run.

2. Processor initialization

build_run_context loads:

• spaCy model from domain.spacy_model_name;
• disabled components from domain.spacy_disable;
• tenant HMAC key from DRTOLLER_TENANT_HMAC_KEY;
• a run-local dependency-relation encoder;
• optional spaCy warm-up.

The default model is en_core_web_sm; NER is disabled by default because Step01 does not currently persist model-produced NER annotations.

If the model is missing, the processor fails with an installation hint:

python -m spacy download en_core_web_sm

3. Batched spaCy parse

process_batch calls nlp.pipe for the supplied document batch.

Two batch controls are independent:

• runtime.document_batch_size — framework read/processor batch;
• domain.spacy_pipe_batch_size — spaCy internal nlp.pipe batch.

If domain.spacy_pipe_batch_size is absent, the processor falls back to the framework document batch size.

4. Token and dependency extraction

For each sentence Step01 records:

• stable token slot and sentence root;
• lemma token id, POS, tag, punctuation/stop/number flags;
• syntactic head slot and dependency relation;
• dependency path relation codes and head lemma ids;
• bounded left/right lemma context window;
• literal value for configured POS classes;
• noun-chunk structure.

Dependency paths stop at domain.dep_path_max_len; dep_path_truncated marks paths that hit the limit.

Punctuation tokens remain present in sentence alignment. By default, they do not receive lemma ids (domain.include_punct_lemmas: false).

5. Lemma identity

lemma_token_id is a deterministic tenant-scoped HMAC over:

namespace = lemma
payload = {version, normalized lowercase lemma, normalized uppercase POS}

Consequences:

• the same lemma/POS pair is stable for the same tenant key;
• run/NOUN and run/VERB are different ids;
• raw lemma text is not used as the registry key;
• changing the tenant key changes all ids and breaks cross-run identity.

Keep DRTOLLER_TENANT_HMAC_KEY stable for every run that must be compared or merged. domain.hmac_key_id labels key configuration but does not carry the secret.

6. Noun chunks

Noun-chunk enumeration is skipped for sentences without POS values listed in domain.np_gate_pos (default NOUN, PROPN). This avoids unnecessary parser work while preserving token rows.

For each chunk the step stores:

• chunk id and token slots;
• root slot;
• head lemma token id;
• ordered lemma ids in the chunk.

7. Sentence packing

Token and noun-chunk rows are grouped into one logical row per sentence. Nested per-token/per-chunk values are packed as list columns. Ragged values use flat values plus offsets (CSR-style encoding), for example dependency paths and chunk token lists.

The framework storage layer handles physical wide-list encoding; domain code returns logical lists only.

Outputs

Dataset	Backend	Purpose
step01_corpus_v3_doc	parquet/local store	one parse-status record per document
step01_corpus_v3_sentence	parquet/local store	sentence-level packed linguistic representation
step01_lemma_vocab_run	PostgreSQL	run vocabulary: lemma id → total token count
step01_lemma_document_count	PostgreSQL	per-document lemma counts for milestone metrics

Full schemas: Step01 data contract.

Write behavior

• document and sentence parquet: 128 MiB flush/part target, segmented on doc_id;
• vocabulary: buffered 500 rows, checkpoint every 100 distinct doc_id transitions;
• document counts: buffered 5000 rows;
• both PostgreSQL outputs use bulk upsert.

step01_lemma_vocab_run adds token_count on conflict for (run_id, lemma_token_id).

step01_lemma_document_count updates the document-local token_count for (run_id, doc_id, lemma_token_id), making the evidence output idempotent for a run.

Parameters

Input and run

Param	Default	Meaning
domain.step00_run_id	empty	upstream Step00 run used to resolve corpus URI
storage.corpus_uri	empty/env fallback	explicit corpus location
domain.corpus_from_s3	false	select S3 corpus behavior
runtime.max_docs	0	process all documents; positive value caps the run
runtime.document_batch_size	4	documents passed to one processor batch
runtime.resume_enabled	true	enable framework resume/checkpoint behavior

NLP behavior

Param	Default	Meaning
domain.spacy_model_name	en_core_web_sm	installed spaCy pipeline
domain.spacy_disable	[ner]	components disabled at model load
domain.spacy_pipe_batch_size	1	spaCy internal batch
domain.context_window_k	3	lemma context radius
domain.dep_path_max_len	6	maximum dependency-chain length
domain.np_gate_pos	[NOUN, PROPN]	POS gate for noun-chunk iteration
domain.literal_pos	[NUM, SYM]	POS classes eligible for literal value
domain.include_punct_lemmas	false	assign lemma ids to punctuation
domain.parse_error_max_len	400	maximum persisted error text

Performance and diagnostics

Param	Default	Meaning
runtime.corpus_batch_max_chars	300000	framework corpus batch character bound
runtime.pipeline.read_q_max	1	reader queue capacity
runtime.pipeline.write_q_max	32	writer queue capacity
runtime.pipeline.lane_q_max	32	output-lane queue capacity
domain.profile_phases_log_every_n_docs	25	extraction phase log cadence; 0 disables
runtime.metrics_enabled	true	live Prometheus instrumentation

Behavioral defaults are DRTML-owned. Tune them through params/UI rather than changing Python constants.

Parallelism

The generic UI exposes runtime.parallelism.backend and runtime.parallelism.workers.

• inprocess is the default;
• the Step01 input is a corpus_scan, whose canonical execution path is currently inprocess;
• framework process/Ray partition dispatch is designed for parquet feed work units and may reject this corpus feed;
• vocabulary and document evidence remain PostgreSQL outputs.

Framework multi-worker limitations still apply, especially checkpoint/resume restrictions. See Parallelism.

Error behavior

A document-level parse/extraction error produces:

• parse_ok: false;
• bounded parse_error;
• document metadata;
• no sentence or lemma evidence rows for that document.

Batch-level nlp.pipe failure marks every document in that batch as failed. Wrong processor input types fail immediately because Step01 accepts only CorpusDocument.

Live observability

Step01 publishes:

• processed documents and progress;
• average throughput;
• spaCy/domain seconds per document;
• elapsed time and ETA;
• builtin CPU and RAM.

Generated Grafana UID: drtoller-step01-live.

Step01 intentionally does not compute vocabulary quality/distribution metrics during extraction. Those belong to step01_metrics.

Running Step01

Streamlit

Open the generated Step01 form, set either:

• Step00 run id, or
• Corpus URI (override).

Choose a run id, review the spaCy/batch controls, and submit. The UI triggers Airflow DAG drt_step01.

Airflow

Each discovered manifest creates its own DAG. Trigger drt_step01 with a fixed step_id and a unique run id. Example dag_run.conf:

{
  "step_id": "step01",
  "run_id": "step01_20260721_001",
  "domain.step00_run_id": "step00_20260721_001",
  "runtime.max_docs": 0
}

To bypass Step00 resolution:

{
  "step_id": "step01",
  "run_id": "step01_manual_001",
  "storage.corpus_uri": "/corpora/example/docs"
}

After success, retain the Step01 run id; it is the input identity for Step01 Metrics.

Prerequisites and troubleshooting

• PostgreSQL schema/procedures/tables must be deployed: run drtoller update.
• Airflow connection postgres_app must resolve for PostgreSQL datasets.
• MinIO connection minio_s3 is declared for corpus storage integration.
• spaCy and the selected model must exist in worker images/volumes.
• DRTOLLER_TENANT_HMAC_KEY must be present and stable.
• Artifact/corpus host paths must be writable/readable by worker users.

Common failures:

Symptom	Check
“install spaCy model”	install domain.spacy_model_name in the worker environment
no corpus / zero documents	domain.step00_run_id, storage.corpus_uri, user/project path, MinIO mount
PostgreSQL relation/schema mismatch	deploy with drtoller update; verify schema_version
ids differ between runs	tenant HMAC key changed
milestones later report no evidence	rerun Step01 after step01_lemma_document_count support was deployed
slow noun-chunk phase	inspect phase logs; tune np_gate_pos, batch sizes, model components