Split the campaigns to expose evaluation leakage

Course · Part 3 · Part 3 exercises · Notebook · Example bank

Self-study after Part 3 · slides 51–52 (notebook lesson 4). Core comparison: steps 1–3; moving individual cards in step 4 is further practice. Eight message cards belong to four near-duplicate campaigns. Our small lookup remembers a campaign’s training label; unseen campaign fingerprints default to normal.

Teaching steps: predict, do, observe, explain

1. Choose Split near-duplicate rows. Count which campaign letters occur in both partitions. Predict before Evaluate this split.

2. Evaluate. The lookup scores 4/4 on familiar campaigns. Does that answer the question about an unseen campaign?

3. Choose Hold out whole campaigns, then evaluate. Compare accuracy and familiar-campaign count; the learner did not see evaluation labels.

4. Move one card, evaluate again and explain the changed question. At least one training and evaluation example must remain.

The in-demo step controls move between instructions. They never perform the next experiment for you. Record one observation before moving on.

Predict, try, explain

1. Click Split near-duplicate rows. Count campaigns shared across training and evaluation. Predict the score, then Evaluate this split.

2. Click Hold out whole campaigns. Confirm A/C are training and B/D are evaluation. Predict which examples the lookup has never seen, then evaluate.

3. Move one message between partitions and evaluate again. Does a better score answer the intended unseen-campaign question, or did you make the exam more familiar?

Record: I changed ___. I predicted ___. I observed ___. This does not establish ___.

More experiments

Create a split with one overlapping campaign; create one with none; try an empty partition and read the feedback. Write the intended future-use question before choosing a split.

What this example establishes

The row split scores 4/4 with 4/4 familiar campaigns. The held-out-campaign split scores 2/4 with zero familiar campaigns. These tiny purpose-built results illustrate a mechanism; they do not estimate production accuracy.

Primary context: scikit-learn on data leakage. This original campaign lookup makes overlap visible; the lab does not use scikit-learn.

Explain it to your partner

Question: Which split answers whether the detector handles a previously unseen campaign?

Take 30 seconds each: point to the input, name the one change, then cite an observation from your run. A useful explanation: Keep whole campaigns together for the unseen-campaign question. Familiar copies answer a different question.

Read next: Concept and research connection. The reading is optional; you do not need its mathematics for this exercise.