Step01 data contract

Step01 produces four datasets with different lifecycles:

corpus document
  ├─ document status                   parquet
  ├─ packed sentence structures        parquet
  ├─ run-level lemma vocabulary        PostgreSQL
  └─ per-document lemma evidence       PostgreSQL

The schemas below are public data contracts. Consumers should depend on dataset ids and declared columns, not on parquet part names or storage object paths.

Input: _corpus

The framework corpus scanner supplies CorpusDocument values. Logical identity is:

Field	Type	Meaning
doc_id	string	stable document id within the corpus
text	string (runtime object)	document text supplied to spaCy
provenance URI	string (runtime object)	original source location when available

The DRTML feed groups by doc_id with run scope. Corpus decoding and source I/O belong to framework storage.

step01_corpus_v3_doc

One row is emitted for every attempted document, including failures.

Column	Type	Meaning
run_id	string	Step01 run
doc_id	string	source document id
provenance_uri	string	source location or empty
char_len	int64	input character count
parse_ok	bool	complete parse/extraction succeeded
parse_error	string	bounded error message; empty on success

Use this dataset to audit corpus coverage and identify failed documents. Do not infer success from the presence of sentence rows: use parse_ok.

Storage:

• local/parquet store rooted at /step01 by default;
• flush target 128 MiB;
• maximum part target 128 MiB;
• doc_id is the flush boundary.

step01_corpus_v3_sentence

One row represents one sentence. Most token attributes are aligned lists: index i across these columns describes token slot i.

Row identity

Column	Type	Meaning
run_id	string	Step01 run
doc_id	string	source document
sent_id	int64	zero-based sentence id in the document
root_slot_idx	int64	sentence root token slot

Token-aligned arrays

Column	Type	Meaning
slot_idx	list	stable zero-based slots
lemma_token_id	list	tenant-HMAC lemma+POS ids; may be empty
pos	list	universal POS
tag	list	model-specific fine-grained tag
morph	list	reserved morphology field (currently empty)
is_punct	list	punctuation flag
is_stop	list	spaCy stop-word flag
like_num	list	spaCy numeric-shape flag
head_slot_idx	list	syntactic head slot
deprel	list	dependency label
dep_depth	list	token dependency depth when supplied by the model
dep_path_len	list	retained dependency path length
dep_path_truncated	list	path reached configured maximum
ent_iob	list	neutral entity IOB (O currently)
ent_type	list	neutral entity type (empty currently)
chunk_id	list	reserved token-level chunk assignment (currently -1)
literal_value	list	original text for configured literal POS classes

The aligned arrays have the same token cardinality. Preserve slot_idx/array position when constructing graphs or context pairs.

Ragged dependency paths

Per-token variable-length paths use flat values and offsets:

Values	Offsets	Meaning
dep_path_deprel_flat	dep_path_deprel_offsets	encoded dependency relation chain
dep_path_head_lemma_token_flat	dep_path_head_lemma_offsets	head lemma ids along the chain

For token i, values occupy flat[offsets[i]:offsets[i+1]].

Dependency relation codes come from a run-local encoder. Treat them as compact values inside a run, not as a global fixed vocabulary.

Ragged context windows

Column	Type	Meaning
lemma_token_w_flat	list	concatenated bounded context windows
lemma_token_w_valid_flat	list	valid/non-empty flags aligned to window values
lemma_token_w_offsets	list	token-to-window boundaries

The radius is domain.context_window_k. The current window includes the center slot and available neighbors; empty lemma ids remain represented and are marked invalid.

Noun chunks

Column	Type	Meaning
chunk_id_list	list	chunk ids in the sentence
chunk_root_slot_idx	list	root slot per chunk
chunk_head_lemma_token_id	list	root/head lemma id
chunk_slot_idxs_flat	list	concatenated token slots
chunk_slot_idxs_offsets	list	chunk-to-slot boundaries
chunk_lemma_token_ids_flat	list	concatenated chunk lemma ids
chunk_lemma_token_ids_offsets	list	chunk-to-lemma boundaries

Chunk ids are sentence-local. Consumers should join chunks to a sentence by (run_id, doc_id, sent_id).

Storage:

• local/parquet;
• 128 MiB flush/part target;
• doc_id flush boundary.

step01_lemma_vocab_run

Run-level registry in pipeline.drt_tbl_lemma_run_staging_v1.

Column	Type	Constraint	Meaning
run_id	string	not null	Step01 run
lemma_token_id	string	not null	stable tenant lemma+POS id
token_count	int64	not null	total occurrences in the run

Logical key: (run_id, lemma_token_id).

The framework derives this dataset from sentence.lemma_token_id using count_values. On conflict, token_count is additive. A checkpoint is taken every 100 distinct doc_id transitions.

This is the input for Step01 Metrics final mode.

step01_lemma_document_count

Durable exact evidence in pipeline.drt_tbl_lemma_document_count_v1.

Column	Type	Constraint	Meaning
run_id	string	not null	Step01 run
doc_id	string	not null	source document
lemma_token_id	string	not null	stable tenant lemma+POS id
token_count	int64	not null	occurrences inside this document

Unique key: (run_id, doc_id, lemma_token_id).

The processor computes these rows from packed sentence lemma arrays. Reprocessing the same run/document/key updates the count rather than adding it, so retries do not multiply evidence.

This is the input for Step01 Metrics milestones mode.

Invariants

Consumers may rely on:

• every attempted document has a document-status row;
• sentence rows are emitted only for successfully extracted sentence structures;
• lemma_token_id is either empty or a 64-character HMAC hex id;
• lemma identity includes POS;
• list columns within a sentence are positionally aligned;
• each offsets array starts at zero and has row_count + 1 entries;
• Step01 Metrics reads run ids from metrics.upstream_runs, not from storage paths.

Consumers must not rely on:

• physical parquet filenames or part counts;
• dependency relation integer values across separate runs;
• raw lemma text being recoverable from lemma_token_id;
• NER fields containing model output while NER is disabled/current neutral fields are used;
• a Step01 run created before document-count evidence support being valid for milestone metrics.

Schema evolution

The top-level Step01 schema id is step01_corpus_v4; historical dataset ids retain v3 in the document/sentence names for compatibility.

Breaking column or semantic changes require a schema/version migration. PostgreSQL physical schema versions are checked at run start and deployed through drtoller update, never created by the step at runtime.