Poison a model: clean versus attacked

Course · Part 4 · Part 4 exercises · Notebook · Example bank

Self-study after Part 4 · slides 15–17 (notebook lesson 4). Mina’s accepted training labels are an integrity boundary. An attacker lowers selected priority labels; the optimiser then learns from the altered examples.

Teaching steps: predict, do, observe, explain

1. Set 6 labels, lowering strength 0.9 and threshold 0.35. Predict which region will move; clean evaluation labels stay fixed.

2. Train both 1,000 updates. Same seed, model and budget. Compare clean-evaluation MSE and missed priority alerts.

3. Change only label count to 3; train again. Record the difference. Changing labels resets both models; a threshold edit does not.

4. Restore trusted labels, then retrain. Explain why the matched runs agree and what trustworthy provenance the restoration assumes.

The in-demo step controls move between instructions. They never perform the next experiment for you. Record one observation before moving on.

Predict, try, explain

1. Before training, keep 6 edited labels, label shift 0.9, threshold 0.35. Predict the region of the curve that will move. Identify crosses and original labels.

2. Click Train both · 1,000 updates. Both models use the same 12-unit architecture, initial seed and update budget. Compare clean-evaluation MSE and missed priority alerts. Do not use the attacked training loss as the only judgement.

3. Change only the number of poisoned labels to 3 and train again. Compare with the six-label run. Then restore trusted labels and retrain. Explain what evidence the recovery assumes.

Record: I changed ___. I predicted ___. I observed ___. This does not establish ___.

More experiments

Try 0, 3, 6 and all 13 targetable labels; compare shifts 0.3 and 0.9; move the routing threshold after training; explain which comparison isolates data integrity and which only changes analyst workload.

What this example establishes

The fixed six-label run misses 11 reference-priority alerts versus zero for clean training. These are measured outcomes of this seeded generated task, not a production attack success rate. Restoring trusted labels requires trustworthy provenance; a neat curve does not prove label integrity.

Security practice context: OWASP LLM04:2025 data and model poisoning. This numerical label-edit experiment illustrates training-data integrity; it does not reproduce an LLM backdoor.

Explain it to your partner

Question: Why can a smaller loss against edited labels be bad news for the analyst?

Take 30 seconds each: point to the input, name the one change, then cite an observation from your run. A useful explanation: Training fits the accepted labels, including corrupted ones. Judge the changed model against unchanged clean labels.

Read next: Concept and research connection. The reading is optional; you do not need its mathematics for this exercise.