Explainability Under Pressure
A security system gives an alert and a reason. Can you trust the reason? Three short, hands-on tests.
Interactive Lab · Toy model
A security system raises an alert and tells the analyst why. The analyst trusts that reason and investigates it. But can the reason be trusted? Try three quick tests below.
The starting point
This is what the analyst sees
Someone may be stealing data. The system flags the event and lists the reasons, largest first.
Test 1 of 3
Ask the same question 20 times. Do you get the same answer?
Nothing about the event changes. We only ask the system “why?” again. Many real tools save time by making a quick estimate instead of a careful calculation.
What the 20 answers said
In my research: KernelSHAP, a widely used estimating tool, passed a small check but gave unstable answers at full scale. LIME varied so much on its own that any effect of an attack was hidden inside that variation.
Test 2 of 3
Change the event very slightly. Does the reason hold?
Real events are never identical. Each try below moves every signal up or down by at most one step, at random. The careful calculation is used, so any change comes from the event itself.
In my research: on real intrusion data, TreeSHAP explanations changed a great deal under standard small-change attacks, far more than in this toy.
Test 3 of 3
Can an attacker change the reason on purpose?
The attacker is still stealing the same amount of data, so the alert stays high. But they can make small changes to the other signals. Their goal is to make the analyst look somewhere else.
Before
After the attacker's changes
What the attacker changed
Try it yourself: the attacker's changes
Each signal can move one step up or down. Data leaving the network is fixed.
In my research: a targeted attack pushed the top reason out of the top ten in about one in five network flows on one dataset, with the alert unchanged. On the other dataset, the same attack barely worked.
What the three tests show
- A quick estimate can give a different reason every time you ask, even when nothing has changed. Before you read meaning into a reason, check how much it varies on its own.
- Tiny, random changes usually leave the reason alone, but not always.
- A deliberate attacker can change the reason while the alert stays the same. Random changes rarely do this. A targeted search does.
These are three different weaknesses. A system can pass one test and fail another, so each one needs its own check. That was the central finding of my MSc research, which tested real explanation tools (SHAP and LIME) on real intrusion-detection data.
For the technical reader
Model. Risk = logistic(−6.0 + 0.22·logins + 0.06·unusual time + 0.20·data leaving + 0.39·scans + 0.08·new device + 0.044·data leaving·new device + 0.054·unusual time·logins), with every signal on a 0 to 10 scale. High risk is 70% or above; needs review is 40% to 69%.
Reasons. Shapley values of the risk score, measured against a quiet reference event with every signal at 1. The percentages are each signal's share of the positive contributions. “Careful calculation” is exact (all 32 coalitions), as with TreeSHAP. “Quick estimate” uses permutation sampling with 5 samples, which behaves like the sampling in KernelSHAP and LIME.
Test 2. Each signal moves by −1, 0 or +1, chosen at random.
Test 3. The attacker may move each signal except data leaving by at most one step. An exhaustive search of all 81 combinations finds 3 that keep the alert high and change the top reason. The interaction terms make this possible: the same data volume gets less of the credit when the device signal drops and the login signals rise.
Reproducibility. All random draws use a fixed seed, so the page gives the same sequence of results each time it loads.
Limits. The attacker here knows the full model. My research used an attacker who could only send inputs and read the outputs. Real network signals cannot always change freely.
Read the research case study and the benchmark write-up for the real experiments and findings.