RobotnixAcademy

Week 08 · lesson

Investigation: Order Changes Recurrent State

system flow

From Token Sequence to a Bounded Output Claim

  1. Token Sequence

    Begin with ordered supplied tokens.

  2. Input Value

    Represent the current token with a declared teaching value.

  3. Recurrent Update

    Combine current input with the carried prior state.

  4. Final State

    Preserve the ordered trace rather than treating tokens as a bag.

  5. Candidate Scores

    Calculate supplied comparison scores from final state.

  6. Claim Boundary

    Do not infer understanding, truth, safety, or real-model performance.

Read this concept flow as plain text
  1. Token Sequence. Begin with ordered supplied tokens. LAB: 3 SEQUENCES.
  2. Input Value. Represent the current token with a declared teaching value. BOUNDARY: INVENTED VALUES.
  3. Recurrent Update. Combine current input with the carried prior state. RULE: 0.5S + X.
  4. Final State. Preserve the ordered trace rather than treating tokens as a bag. TEST: REVERSE ORDER.
  5. Candidate Scores. Calculate supplied comparison scores from final state. RISK: NOT PROBABILITY.
  6. Claim Boundary. Do not infer understanding, truth, safety, or real-model performance. ACTION: REPAIR CLAIM.

A language model estimates which tokens fit a context. A recurrent network processes one token at a time and carries a state forward. The same update rule is reused at every step, but the state arriving at each step depends on all earlier updates.

For this investigation, use an invented one-number recurrence:

new state = 0.5 × previous state + token value

This is small enough to calculate by hand. It demonstrates sequence mechanics, not the behavior of a trained RNN.

Completed trace

Use robots = 0.8, build = 0.4, and start with state 0.

robots: 0.5 × 0.0 + 0.8 = 0.8
build:  0.5 × 0.8 + 0.4 = 0.8
final state = 0.8

Reverse the same tokens:

build:  0.5 × 0.0 + 0.4 = 0.4
robots: 0.5 × 0.4 + 0.8 = 1.0
final state = 1.0

The tokens are unchanged, but their order changes the trace and final state.

Scores are not facts

The Lab turns the final state into two invented candidate scores:

safely = 1 - |state - 0.8|
quickly = 1 - |state - 0.3|

These scores expose how context state can affect an output comparison. They are not probabilities, meanings, recommendations, or evidence about real language.

Investigation

  1. Trace robots build and build robots by hand.
  2. Calculate both candidate scores from each final state.
  3. Explain exactly where order enters the mechanism.
  4. Add quickly = 0.2 as a third token and predict the new final state.
  5. Repair: “The state remembers the sentence, so the network understands it.”

Vocabulary lab

Flip the idea, not just the card

Explain the term before you reveal the back. Then compare your explanation with the definition, example, and warning.

1 / 5
Read all terms without animation
language model
A model that assigns scores or probabilities to token sequences or token predictions. Example: Estimating a next token from preceding context. Do not confuse it with: A score does not establish truth or human understanding.
recurrent state
A carried numeric representation updated as sequence elements are processed. Example: The teaching state changes from 0 to 0.8 to 0.8. Do not confuse it with: It is not a verbatim memory or explanation of meaning.
shared weights
The same learned transformation is reused at each sequence step. Example: One recurrent cell rule processes every token. Do not confuse it with: Shared weights do not make each step's state identical.
next-token score
A numeric value used to compare candidate continuations for a context. Example: Safely receives 1.0 from final state 0.8 in the teaching rule. Do not confuse it with: A score is not automatically a calibrated probability.
exposure error
Error that can compound when generated outputs become later inputs. Example: One poor token changes the context used for the next prediction. Do not confuse it with: It is different from one isolated prediction error.