RobotnixAcademy

Week 10 · lesson

Investigation: Extraction Needs a Schema and an Audit

system flow

From Text Tokens to a Bounded Extraction Claim

  1. Declared Text

    Begin with authorized text and a fixed entity schema.

  2. BIO Labels

    Assign beginning, inside, or outside labels to each token.

  3. Exact Spans

    Decode boundaries and types into evaluation units.

  4. Gold Comparison

    Count exact matches, extras, and omissions.

  5. Metric Audit

    Calculate precision, recall, and F1 without hiding boundary errors.

  6. Claim Boundary

    Do not infer factuality, privacy, safety, or general performance.

Read this concept flow as plain text
  1. Declared Text. Begin with authorized text and a fixed entity schema. LAB: 6 TOKENS.
  2. BIO Labels. Assign beginning, inside, or outside labels to each token. SCHEMA: PER LOC DATE.
  3. Exact Spans. Decode boundaries and types into evaluation units. CHECK: BOUNDARY + TYPE.
  4. Gold Comparison. Count exact matches, extras, and omissions. EVIDENCE: TP FP FN.
  5. Metric Audit. Calculate precision, recall, and F1 without hiding boundary errors. RISK: PARTIAL SPAN.
  6. Claim Boundary. Do not infer factuality, privacy, safety, or general performance. ACTION: REPAIR CLAIM.

Named entity recognition is token classification with structure. BIO labels mark the beginning and inside of an entity span; O marks tokens outside the declared entity types.

Rin       B-PER
visits    O
Harbor    B-LOC
City      I-LOC
on        O
Monday    B-DATE

This decodes into three exact spans: Rin/PER, Harbor City/LOC, and Monday/DATE.

Exact-span measurement

An entity counts as a true positive only when its start, end, and type match.

precision = TP ÷ (TP + FP)
recall    = TP ÷ (TP + FN)
F1        = 2 × precision × recall ÷ (precision + recall)

Completed boundary-error example

Suppose a system predicts Harbor/LOC but leaves City outside. It also finds Rin/PER and Monday/DATE correctly.

TP = 2
FP = 1  (Harbor/LOC is not the complete gold span)
FN = 1  (Harbor City/LOC was missed)
precision = 2/3 = 0.667
recall = 2/3 = 0.667
F1 = 0.667

Most tokens still look correct. Exact-span scoring exposes the operational boundary error.

Pre-trained models still need evidence

A pre-trained language model may perform extraction from a prompt, examples, or fine-tuning. None removes the need to define entity types, output format, allowed text, evaluation examples, privacy boundaries, and failure handling. Model families and product facts also change; verify current claims against current primary documentation instead of treating a pinned lesson as live product documentation.

Investigation

  1. Decode the gold BIO tags into exact spans.
  2. Score an output that misses Monday/DATE.
  3. Score the boundary-error output.
  4. Explain why token accuracy could hide the span error.
  5. Repair: “The model returned valid JSON, so the entities are correct.”

Vocabulary lab

Flip the idea, not just the card

Explain the term before you reveal the back. Then compare your explanation with the definition, example, and warning.

1 / 5
Read all terms without animation
named entity recognition
The task of locating text spans and assigning declared entity types. Example: Harbor City is labeled as one location span. Do not confuse it with: It is not general fact checking or intent detection.
BIO tagging
A token-label scheme marking beginning, inside, and outside positions. Example: Harbor is B-LOC and City is I-LOC. Do not confuse it with: Valid tags still must be decoded into spans.
exact span
An entity whose boundaries and type are treated as one evaluation unit. Example: Harbor City from token 2 through token 3 with type LOC. Do not confuse it with: A partial boundary is not an exact match.
pre-training
Broad training performed before adaptation or prompting for a downstream task. Example: A language model learns from a large text objective before an extraction task. Do not confuse it with: Pre-training does not certify downstream correctness.
few-shot prompting
Providing a small number of examples inside a prompt to guide a task. Example: Two tagged sentences precede a new extraction request. Do not confuse it with: Examples are instructions, not evaluation evidence.