RobotnixAcademy

Week 07 · lesson

Investigation: Similarity Depends on Representation

system flow

From Text Token to a Bounded Similarity Claim

  1. Text

    Begin with declared supplied text and context.

  2. Tokenization

    Select units and map known tokens through a vocabulary.

  3. Vector Representation

    Use sparse identity or dense fitted coordinates.

  4. Similarity Calculation

    Preserve dot products, norms, and cosine values.

  5. Context Test

    Challenge whether the relationship survives different meanings or corpora.

  6. Claim Boundary

    Do not infer synonymy, truth, fairness, or causation from closeness.

Read this concept flow as plain text
  1. Text. Begin with declared supplied text and context. LAB: 3 TOKENS.
  2. Tokenization. Select units and map known tokens through a vocabulary. BOUNDARY: TOKEN RULES.
  3. Vector Representation. Use sparse identity or dense fitted coordinates. COMPARE: ONE-HOT · DENSE.
  4. Similarity Calculation. Preserve dot products, norms, and cosine values. METRIC: COSINE.
  5. Context Test. Challenge whether the relationship survives different meanings or corpora. RISK: CONTEXT SHIFT.
  6. Claim Boundary. Do not infer synonymy, truth, fairness, or causation from closeness. ACTION: REPAIR CLAIM.

A token is a selected text unit. A vocabulary maps each known token to an ID. An ID is an address, not meaning. A one-hot vector preserves identity but makes every different token orthogonal. A learned embedding maps tokens to dense vectors whose geometry reflects the training objective and data.

Cosine similarity

Cosine compares direction rather than raw magnitude:

cos(a,b) = dot(a,b) / (norm(a) × norm(b))

Values near 1 point in similar directions, 0 indicates orthogonality, and -1 indicates opposing directions. Interpretation still depends on the model.

Completed calculation

Use robot = [0.9, 0.8] and machine = [0.8, 0.9].

dot = (0.9×0.8) + (0.8×0.9) = 1.44
norm(robot) = sqrt(0.9² + 0.8²) ≈ 1.204
norm(machine) ≈ 1.204
cosine = 1.44 / (1.204×1.204) ≈ 0.993

The supplied vectors are geometrically close. That does not prove the tokens are synonyms, interchangeable, true, fair, or close in another embedding.

Context and missing tokens

Embedding behavior inherits a corpus, tokenization system, objective, and training procedure. Rare terms may be weakly represented. Out-of-vocabulary tokens may be split, replaced, or omitted. A single static vector also cannot fully represent a word such as bank in both river and financial contexts.

Investigation

  1. Predict similarities for robot, machine, and banana under one-hot vectors.
  2. Calculate both dense-vector cosine values.
  3. Name exactly what changed between representations.
  4. Define a context test using two meanings of one spelling.
  5. Repair: “The vectors are close, so the words mean the same thing.”

Vocabulary lab

Flip the idea, not just the card

Explain the term before you reveal the back. Then compare your explanation with the definition, example, and warning.

1 / 5
Read all terms without animation
token
A text unit selected by a tokenizer for representation. Example: A word, subword, character, or punctuation mark. Do not confuse it with: A token is not always a whole human word.
vocabulary
The declared set mapping known tokens to identifiers. Example: Robot, machine, and banana receive separate IDs. Do not confuse it with: An ID records identity, not semantic meaning.
one-hot vector
A sparse identity vector with one active position. Example: Robot can be represented as 1,0,0. Do not confuse it with: Different one-hot vectors carry no graded similarity.
embedding
A dense numeric representation fitted for an objective and data context. Example: Robot receives the supplied vector 0.9,0.8. Do not confuse it with: An embedding is not a dictionary definition or fact database.
cosine similarity
The normalized dot product comparing vector direction. Example: Robot and machine score about 0.993 in the supplied dense space. Do not confuse it with: It does not prove synonymy, truth, or causation.