Part 2 · Detection and its evaluation

Count the decisions · slides 35–40

Slide 35 · Misuse or anomaly?

Open slide 35

One flow record: workstation ws-114 sends 95 MB/min to 203.0.113.45 at 09:05. Detection objective and training signal are separate axes and do not map one-to-one.

Step by step, as on the slide:

1. Misuse detection asks whether the event matches known malicious behaviour. Signatures are one implementation, not the definition. Nothing known matches here.
2. Anomaly detection asks whether the event is unusual for ws-114. Its Lab 04 office history has mean \mu = 10 MB/min and \sigma \approx 1.225, so z = (95 - 10)/1.225 \approx 69.4. Anomaly methods need not be unsupervised; one-class training assumes a clean baseline. A scheduled backup is also unusual for an office workstation.
3. Neither result proves intent. Check the process, the destination and the job schedule. The objective does not fix the method; the per-entity baselines return on slide 46.

False-alert load and coverage of new attacks depend on features, data, threshold and deployment. Sources: Denning (1987), anomaly model · Kumar & Spafford (1994), misuse model · Sommer & Paxson (2010).

Slides 36–38 · Count decisions → precision → recall and false-positive rate

Open slide 36

A mail filter at threshold 0.50 over 1,000 emails, 100 of them phishing. Step by step, as on slide 36:

1. One square per email: red squares are the 100 phishing emails, grey squares the 900 legitimate ones.
2. Every email has two facts: truth and decision. The outlined squares are the filter's 100 alerts: 90 on phishing emails, 10 on legitimate ones.
3. Sorted by truth (columns) and decision (rows): TP + FN = 100 phishing emails, FP + TN = 900 legitimate, all = 1{,}000.
4. The outlined Alert row is what an analyst sees: TP + FP = 90 + 10 = 100 alerts, and precision divides by it.
5. The outline clears and the 10 misses are marked: they were delivered and raised no alert. Later investigation, other controls or audits may still reveal them, and the “actual” labels themselves need evidence.
6. Vote: which costs more, missed phishing (FN) or a quarantined invoice (FP)?
7. Answer to the vote: no universal answer; impact and workflow set the cost.

Other settings: which error costs more for ransomware on a file server? What about quarantined legal contracts? There is no universal answer: a missed ransomware event (FN) can halt the file server; a false alert that quarantines legal contracts (FP) blocks work. Impact, availability and workflow set the cost.

Every metric is a ratio of these cells. Start with the denominator.

Slide 37, step by step:

1. The outlined Alert row is the review queue: 90 TP + 10 FP = 100 alerts. Precision divides by these; the 10 FN and 890 TN fade because they never reach review.
2. Numerator: TP, the 90 alerts that are real phishing emails. \text{Precision} = \frac{TP}{TP + FP}.
3. \frac{90}{90 + 10} = 0.90. The 10 hollow squares are reviews spent on legitimate mail: the false discovery rate, 1 - \text{precision} = 10\%.

Slide 38, step by step:

1. Recall: of all the real phishing emails, how many did we catch? Also called detection rate or true-positive rate (TPR); the denominator is the 100 phishing emails. \frac{TP}{TP + FN} = \frac{90}{100} = 90\%; the 10 misses were delivered and raised no alert.
2. False-positive rate: the denominator is the 900 legitimate emails. \frac{FP}{FP + TN} = \frac{10}{900} \approx 1.1\%; 890 were delivered. The ROC curve on slide 39 plots one against the other.
3. Queue share: the same 10 false alerts are 1.11% of legitimate emails but \frac{FP}{TP + FP} = \frac{10}{100} = 10\% of the alerts: the false discovery rate.

Precision and recall both equal 0.90 here because FP and FN happen to be equal; that is not an identity. Accuracy can hide missed rare attacks, and F_1 does not express false-alert workload or the cost of errors. These counts carry a 10% phishing share; the million-event day on slides 39–41 changes the prior, so do not carry 90% precision across. Sources: Fawcett (2006) · Arp et al. (2022), P7.

Slide 39 · What does an ROC curve show?

Open slide 39

In plain words: an ROC curve shows every threshold at once. Higher on the curve means more attacks caught; further left means fewer false alerts. The threshold is the score an event needs to raise an alert. A low threshold catches almost everything but floods the queue; a strict one gives about 1,000 false alerts a day, and only about 4 in 100 alerts are real. The AUC (area under the curve) of 0.98 grades the whole curve and hides this.

In detail: a receiver operating characteristic (ROC) curve plots recall against false-positive rate as the threshold t moves. One day has 1,000,000 events and 100 attacks; benign scores follow \mathcal N(0,1) and attack scores \mathcal N(3,1), so

where \Phi is the standard normal distribution function. The slide's x-axis is logarithmic so the low-FPR end, where operations live, is visible.

Step by step, as on the slide (the interactive version below computes the same points):

1. t = 1.0: FPR 15.87%, recall 97.7%. 999{,}900 \cdot 0.1587 \approx 158{,}639 false alerts against about 98 true ones; precision 0.06%.
2. t = 3.09, the 99.9th percentile of benign scores (\Phi^{-1}(0.999) = 3.090232): FPR 0.100%, recall 46.4%; about 1,000 false and 46 true alerts, precision 4.4%. \text{AUC} = \Phi(3/\sqrt{2}) \approx 0.983 is one number for the whole curve, the same at every threshold, and it ignores the base rate.

Report recall at a fixed low FPR, or the partial AUC over that range (McClish, 1989), plus precision at the deployment base rate (Davis & Goadrich, 2006). The next slide deliberately uses a different operating point (99% recall, 1% FPR). Sources: Fawcett (2006) · McClish (1989) · Davis & Goadrich (2006).

Slide 40 · Rare attacks, mostly false alerts

Open slide 40

In plain words: a good detector catches 99% of attacks and flags 1% of normal events by mistake. Attacks are rare (1 in 10,000), so the mistakes on the many normal events swamp the few real alerts: only about 1 in 100 alerts is a real attack.

In detail: the base-rate fallacy (Axelsson, 2000); the title states the result. 1,000,000 events/day · 100 attacks. Recall 99% gives 99 true alerts. FPR 1% on 999,900 benign events gives 9,999 false alerts. Precision \approx 1\%: about 99 of every 100 alerts are false. The base rate changes what an alert means. This is a different population from the 1,000 emails: 100 attacks in 1,000,000 events instead of 100 in 1,000, so their 90% precision does not carry over.

Step by step, as on the slide:

1. One day: 1,000,000 monitored events, for example sign-ins or process starts, watched by a detector with 99% recall and 1% false-positive rate.
2. Split by truth: only 100 of 1,000,000 events are attacks (base rate 0.01%); 999,900 are benign.
3. Recall 99% acts on the 100 attacks: 99 alerts, 1 missed.
4. FPR 1% acts on 999,900 benign events: 9,999 false alerts; 989,901 benign events raise none. Before the next step: which box makes the title true?
5. One queue: 99 + 9{,}999 = 10{,}098 alerts per day, and

the 9,999 false alerts, not the 99 true ones, make the title true. The final caption: all 10,098 alerts land in one queue: only about 1 in 100 is real.

This is Bayes' theorem in counting form, “natural frequencies” (Gigerenzer & Hoffrage, 1995). This day follows Axelsson's argument, not his numbers. His conclusion: the false-positive rate, not the detection rate, is the limiting factor; in his scenario (base rate 2 in 100,000) it must be about 1 in 100,000 for two in three alarms to be real. A score of 0.9 from a model trained on balanced data is not a 90% chance of attack at a 0.01% base rate. Arp et al. (2022) list the base-rate fallacy as pitfall P8. Sources: Axelsson (2000), ACM TISSEC 3(3) · Gigerenzer & Hoffrage (1995) · Arp et al. (2022).

Can analysts keep up? · slides 41–43

Slide 41 · Exercise · how low must the false-positive rate be?

Slide 41 is an exercise for this page: it is hidden in the deck, so the slide footers jump from 40 to 42.

Pairs · 3 minutes. Same day: 1,000,000 events, 100 attacks, recall 99%. Complete the table before opening the answers.

Slide 42 · Live demo · from too many alerts to a workable queue

Open slide 42

In plain words: on lab data, where 1 in 10 events is an attack, model A looks fine. On real traffic, where 1 in 10,000 is an attack, the same model raises far more alerts than analysts can check. Raising the threshold until analysts can cope means missing 57 of 100 attacks. Only a better model (B) fits within 200 alerts a day and still catches 97 of 100.

In detail: two models score the same events: A (overlapping score distributions) and B (better separated). The SOC can review 200 alerts per day.

Step by step, as on the slide:

1. Start, base rate 10%, an enriched lab dataset: model A at 0.50 gives precision about 69%.
2. Base rate 0.01%, the day from slide 40: the curves do not move; only the counts behind them change. Precision falls to about 0.2%: roughly 97 true and 48,000 false alerts, 47,883 in total, about 239 times what the SOC can review.
3. Raise the threshold to fit capacity: 0.74 is the first value within capacity: 43 + 123 = 166 alerts, precision about 26%, but 57 of 100 attacks per day are missed. At 0.75: 31% and 60 missed. That is the operational price of a low base rate.
4. A better-separated model fits capacity and keeps detection: model B at 0.70 detects about 97% of attacks with 97 + 32 = 129 alerts and precision about 75%, 3 missed.

In practice the other levers are analyst capacity and raising the base rate by scoping and correlation. The risk owner, not the vendor default, should decide where the threshold sits. The rate options are assumptions for this demonstration, not universal deployment base rates. Lab 05 lets you repeat this with a concrete queue. Sources: Axelsson (2000) · Alahmadi et al. (2022).

Open the interactive lab

Slide 43 · Analysts cannot review everything

Open slide 43

In plain words: 67 alerts arrive in one shift, but there is time to check only 50. If false alerts come first, only 1 of the 18 real attacks gets checked; in random order about 13; if real attacks come first, all 18. The order of the queue decides what gets checked.

• Set thresholds using error costs, review time and capacity: the expected cost of a missed attack, the cost of investigating a false alarm, and alerts per analyst per shift.
• Alert fatigue: SOC practitioners (survey n = 20, interviews n = 21) perceived most alerts as false positives, but most were benign triggers: the rule fired correctly on legitimate activity. Analysts learn to ignore them, may disable noisy rules, and real alerts get buried (Alahmadi et al., 2022). Tuning down a noisy rule is a rational response to low precision, and also how detection coverage erodes.
• Measure the decisions improved, alongside detector performance.

One analyst's shift: 67 alerts against 50 review slots, one analyst's share of the 200 alerts a day on slide 42. Lab 05's rates give the 67: 20 attacks and 980 benign events, recall 90% and FPR 5%, so 0.90 \cdot 20 = 18 true and 0.05 \cdot 980 = 49 false alerts. This is a different population from the 1,000 emails on slide 36. The tiles are grey because the analyst does not yet know which alerts are true. Which 50 get reviewed?

Step by step, as on the slide (the interactive version below draws the same queue):

1. Worst order: every false alert first. The 49 false alerts take 49 slots, 17 of the 18 true alerts land outside the slots, and only 1 is reviewed.
2. Order unrelated to risk, here the order in which the alerts arrived: 13 true alerts fall inside the slots. The expected value is \mathbb E[\text{true in 50}] = 50 \cdot \frac{18}{67} \approx 13.4.
3. Perfect ranking: all 18 true alerts first. This is an upper bound, not a property of this detector. Ranking quality, not only the threshold, decides what capacity buys.

Sources: Alahmadi et al. (2022) · Lab 05.

In the interactive version, Random order draws a different arrival order from each seed: the count inside the 50 slots varies around 13.4.

Three alerts, worked through · slides 44–47

Slide 44 · Alert 1 · a reported email

Open slide 44

Slides 44–46 follow one incident through three alerts the analyst checks: an email, an attachment, an upload. A detector is useful only if its alerts can be verified; these slides show what verifying an alert takes. Alert 1 is the phishing email from slide 14, now reported by a staff member. Would you click? An analyst does not judge the prose but opens the message source.

Step by step, as on the slide:

1. Identity. “Service desk” is a display name: free text the sender types. The From header carries the address, it-desk@exampIe.com.
2. Read the domain letter by letter: “exampIe with a capital I”, U+0049, where example.com has a lowercase l, U+006C. In the sans-serif font of a mail client the two are almost identical; in the monospace source they are not.
3. Authentication. Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM) and Domain-based Message Authentication, Reporting and Conformance (DMARC) all pass, because the attacker registered exampIe.com and configured it correctly: Authentication-Results: spf=pass dkim=pass dmarc=pass header.from=exampIe.com. Authentication confirms the message came from exampIe.com; it says nothing about intent, and a compromised legitimate domain passes as well.
4. Destination. The link text says mail.example.com; the target is mail.exampIe.com/reset, which is not the organisation's known portal. Decision: report the message and block the domain. Then: did anyone open mail from this sender? That is alert 2. The final caption: verified by identity, authentication and destination, not prose.

Learned filtering dates back at least to Sahami et al. (1998); for AI-written phishing, rely on the later Heiding study (slide 66), which measures clicks, not compromise (Heiding et al., 2024 [preprint]).

Self-study: train a two-clue phishing classifier and find a mistake its features cannot explain.

Open the interactive lab

Slide 45 · Alert 2 · the attachment ran

Open slide 45

Alert 2: the same sender sent the attachment invoice_7731.pdf.exe, and workstation ws-114 opened it.

Step by step, as on the slide:

1. Static inspection reads the file without running it: a double extension, an import of CreateRemoteThread, a packed section with entropy 7.6 of a maximum 8 bits per byte. Importing a thread-creation function is not proof of injection, and benign installers are packed too. Static models can learn the packer as a shortcut instead of the behaviour (Aghakhani et al., 2020). Limit: packing hides features.
2. Dynamic inspection runs it in a controlled sandbox: at t+2 s it writes an executable, at t+3 s it launches a child process, at t+4 s it connects to 203.0.113.45, the destination of the ws-114 flow on slide 35. A connection is not proof of command-and-control. Samples can detect the sandbox and delay or alter what they do; Lindorfer et al. (2011) compared behaviour across several analysis sandboxes to find such environment-sensitive samples. Limit: it may behave differently outside the sandbox.
3. Three facts from other sources corroborate the clues: the sender is the lookalike domain it-desk@exampIe.com (capital I); the parent process is the mail client on ws-114, so a user opened the attachment; and the proxy log shows an outbound upload to the same address seen in the sandbox, 203.0.113.45. Together they justify escalating for review; no single clue does.

Labels remain noisy and delayed (Zhu et al., 2020), and new benign software as well as new malware causes drift. Sources: Anderson & Roth (2018) [preprint] · Raff et al. (2018) · Aghakhani et al. (2020) · Lindorfer et al. (2011).

Open the interactive lab

Slide 46 · Alert 3 · the upload from ws-114

Open slide 46

• A baseline is specific to a host, user or service.
• Isolation forests separate unusual points: an event that a few random splits isolate from all others is unusual (Liu et al., 2008). Autoencoders learn to compress and rebuild normal traffic and flag traffic they rebuild badly, as in Kitsune (Mirsky et al., 2018).
• The same traffic volume can be unusual for one entity and normal for another.
• Unusual does not establish malicious intent. Dataset artefacts can also inflate evaluations.

Alert 3: ws-114, the workstation that opened the attachment, uploads 95 MB/min to 203.0.113.45, the flow from slide 35. Is that unusual for this host? Each host gets eight observations of outbound traffic from Lab 04, and the score is the distance from that host's own mean in standard deviations (population standard deviation, cutoff 3):

Step by step, as on the slide (the interactive version below builds each baseline one observation per Step):

1. Eight observations per host, in MB/min, from Lab 04.
2. Each host has its own normal: \mu = 10, \sigma = 1.22 for ws-114; \mu = 97.5, \sigma = 9.54 for the backup server. The band is \mu \pm 3\sigma.
3. The same 95 MB/min event lands on both baselines.
4. ws-114: z \approx 69.4, unusual. Backup server: z \approx 0.26, ordinary. The anomaly score says “unusual for ws-114”; alone it does not establish intent. With alerts 1 and 2 (the lookalike sender, the attachment opened on ws-114, the same destination) it points to one incident: escalate and request isolation of ws-114 from the authorised approver.

Strengths: no attack labels, and novel behaviour may surface. Weaknesses: benign change is constant, per-entity baselines are noisy, and results are hard to explain. Many published intrusion-detection results were later shown to exploit artefacts of how public datasets were generated (McHugh, 2000; Engelen et al., 2021). Arp et al. (2022) re-evaluated Kitsune on its own Mirai data: a one-feature boxplot of packet rate reached 0.996 detection at 0.1% FPR against Kitsune's 0.882, because the attack was overly conspicuous (pitfalls P6 and P9). Sources: Liu et al. (2008) · Mirsky et al. (2018) · McHugh (2000) · Engelen et al. (2021).

In the interactive version, Host switches between ws-114 and the backup server.

Open the interactive lab

Slide 47 · Explanations are leads, not proof

Open slide 47

• Interpretability can be built into rules, small trees and linear models.
• Post-hoc methods assign each feature a share of the difference between this score and a baseline score; the assignment depends on the method's assumptions and is not causal proof.

Why did the gateway flag the “Your mailbox expires today.” email? This is gateway v2, the five-feature production model announced on slide 27. The axis is log-odds z; the score is \sigma(z). The review threshold is z = 0, \sigma(0) = 0.50. For a logistic model the additive decomposition lives in log-odds, not in the 0 to 1 score.

Step by step, as on the slide:

1. Start from the bias b = -3, the log-odds of an email with none of the features, f(\mathbf 0); score 0.05.
2. Each contribution is \phi_j = w_j x_j. Three urgent phrases at weight 0.6 give +1.8; two spelling mistakes at 0.5 give +1.0. The running total is -0.2, still below the threshold.
3. Link mismatch +0.9 and new sender domain +1.2: the running total passes the threshold. The order is for display; a running total is not the verdict of any single feature.
4. Password form +1.5 completes it: z = -3 + 1.8 + 1.0 + 0.9 + 1.2 + 1.5 = 3.4 and \sigma(3.4) = 0.97, review. The decomposition is exact, \sum_j \phi_j = f(\mathbf x) - f(\mathbf 0), which is SHAP's local accuracy with the all-zero email as the reference (Lundberg & Lee, 2017). A different reference gives different contributions; for nonlinear models SHAP and LIME only estimate them.
5. First reading: the largest feature contribution, urgent phrases (+1.8), is the cheapest for the attacker to remove. An evasion attack rephrases exactly these.
6. Second reading: the bars to verify are the link mismatch and the new domain. Check the link target and the gateway record from slide 44. Explanations are leads, not proof.

Explanations approximate the model; they can be unfaithful, unstable and even deliberately manipulated (Slack et al., 2020). In a security comparison, gradient-based methods were more accurate and stable (Warnecke et al., 2020). A good explanation tells the analyst which evidence to check next. Sources: Lundberg & Lee (2017) · Lundberg et al. (2020) · Ribeiro et al. (2016) · Guo et al. (2018) · Warnecke et al. (2020) · Slack et al. (2020).

Back to accuracy · slide 48

Slide 48 · Is 99.9% accuracy good?

Open slide 48

Question. A vendor tells you its new network detector is “99.9% accurate”. Would you buy it? Which three questions would you ask first, before believing the claim is relevant to your organisation? The slide's answer chips: alert load at our base rate? Whose data, when? Drift and adaptive attackers?

Part 2 answer · Is the detector useful? For analyst triage: only if precision at our base rate fits capacity.

The scope matters: correlated low-precision clues (slide 45) and inline blocking are judged on other terms, such as the cost of a wrong block. Accuracy, AUC and a vendor's headline number do not answer the question.

Self-check · Part 2

Practise Part 2

Notebook: Lesson 3 · Investigate behaviour and workload. Worked examples: Lesson 3.