RobotnixAcademy

Week 11 · lesson

Investigation: An Optimizer Follows the Objective

system flow

From Candidate Behavior to a Bounded Optimization Claim

  1. Candidate Evidence

    Begin with fixed, fictional behavior measurements.

  2. Declared Objective

    State every reward, penalty, and omitted consequence.

  3. Score Calculation

    Preserve each operand rather than only the total.

  4. Ranking

    Order candidates under the selected formula.

  5. Loophole Test

    Inspect repeated reward, hazards, and missing constraints.

  6. Claim Boundary

    Do not infer safety, true optimality, transfer, or authorization.

Read this concept flow as plain text
  1. Candidate Evidence. Begin with fixed, fictional behavior measurements. LAB: 3 BEHAVIORS.
  2. Declared Objective. State every reward, penalty, and omitted consequence. COMPARE: 2 FORMULAS.
  3. Score Calculation. Preserve each operand rather than only the total. EVIDENCE: 6 SCORES.
  4. Ranking. Order candidates under the selected formula. CHECK: REVERSAL.
  5. Loophole Test. Inspect repeated reward, hazards, and missing constraints. RISK: REWARD HACK.
  6. Claim Boundary. Do not infer safety, true optimality, transfer, or authorization. ACTION: REPAIR CLAIM.

A genetic algorithm uses a fitness function to compare candidate genes. A reinforcement-learning agent uses reward to update a policy. These mechanisms differ, but neither can infer an unstated goal. If safety, uniqueness, or task completion is absent from the objective, optimization may ignore it.

Use three fictional navigation behaviors:

BehaviorGoalStepsCheckpoint hitsUnique checkpointsHazards
Shortcut13222
Safe route16330
Checkpoint loop08810

Two declared objectives

naive = 10×goal + 2×checkpoint hits - steps
safety-aware = 10×goal + 2×unique checkpoints - steps - 6×hazards

Completed comparison

For the shortcut:

naive = 10×1 + 2×2 - 3 = 11
safety-aware = 10×1 + 2×2 - 3 - 6×2 = -1

For the safe route:

naive = 10×1 + 2×3 - 6 = 10
safety-aware = 10×1 + 2×3 - 6 - 6×0 = 10

Changing the objective changes the ranking without changing any behavior. That is evidence about objective design, not proof that either formula captures every real consequence.

Search and learning still require tests

Mutation and exploration can discover novel candidates, including loopholes. Selection and exploitation can amplify whatever scores well. Test objectives against edge cases, preserve constraint violations separately, and require human authority before any real-world action.

Investigation

  1. Calculate both scores for all three behaviors.
  2. Rank the behaviors under each objective.
  3. Explain the loop's repeated-checkpoint loophole.
  4. Identify one important consequence missing from both formulas.
  5. Repair: “The optimizer found the highest score, so it found the best action.”

Vocabulary lab

Flip the idea, not just the card

Explain the term before you reveal the back. Then compare your explanation with the definition, example, and warning.

1 / 5
Read all terms without animation
fitness function
A declared numeric rule used to compare candidate solutions in evolutionary search. Example: The safety-aware formula ranks the three supplied behaviors. Do not confuse it with: Fitness is not an automatic measure of every real consequence.
mutation
A variation operation that changes part of a candidate representation. Example: A route gene changes one move. Do not confuse it with: Mutation does not guarantee improvement or safety.
reward function
A declared signal used to reinforce behavior in an environment. Example: Goal completion adds ten points. Do not confuse it with: Reward is not the same as human value or approval.
policy
A rule or model that selects actions from observed states. Example: A navigation policy chooses the next move. Do not confuse it with: A policy is not the environment or reward function.
reward hacking
Behavior that scores well by exploiting a mismatch between the metric and intended outcome. Example: Repeated checkpoint hits earn naive reward without reaching the goal. Do not confuse it with: The optimizer is following the written objective, not the unstated intent.