All articles

Blog

How to measure training effectiveness: completion isn't enough

Measure training effectiveness in security awareness: why completion rates mislead, and which behavioural indicators to track instead.

  • By Azziz ERRIME
  • 11 min read
Measure training effectiveness: a hospital workstation at night, a blurred training page closing on screen

TL;DR

  • To measure training effectiveness, the completion rate isn’t enough: a module closed straight away can count for as much as one that was read.
  • Across 19,500 employees tracked for 8 months, annual training showed no significant relationship with the risk of clicking on a simulated phishing email.
  • I’d present the executive committee with a behavioural indicator, measured without warning and repeated every 4 to 6 months.

In May 2025 in San Francisco, a team from the University of California San Diego presented a study run in its own teaching hospital. Its researchers followed 19,500 employees for 8 months. The training they had all taken made almost no difference at the moment of the click. So to measure training effectiveness, the completion rate isn’t enough.

The venue was the 46th IEEE Symposium on Security and Privacy. Grant Ho, Ariana Mirian and their colleagues described an experiment carried out at UC San Diego Health. Over 8 months, employees received ten simulated phishing campaigns, no two of the same type. When an employee in the trained group clicked, a training page opened to explain what they should have spotted.

The researchers then looked at what happened on that page. According to the university’s press release, 75% of those who received it spent a minute or less on it, and a third closed it immediately. In an activity report, every one of those page views can be listed as training delivered.

Their conclusion fits in a single sentence: “Anti-phishing training programs, in their current and commonly deployed forms, are unlikely to offer significant practical value in reducing phishing risks.” That doesn’t mean I’m shelving training. I’m looking instead at what gets measured when people say it works.

In San Diego, annual training had no effect on clicks

The study separates two formats that many companies run side by side. The first is mandatory annual training, the course every employee takes once a year and the one that feeds the completion rate. The researchers found no significant relationship between having taken it and the likelihood of clicking on a simulated phishing email.

The second is embedded training, which opens at the moment of the click. It does have an effect, but a slim one: 2% fewer clicks, according to the university. Yet on paper it’s the ideal format, since it arrives at the exact moment of the mistake. The reading times noted above offer a clue as to why it achieves so little.

The third finding is the one I’d show an executive committee. In the first month, 10% of employees clicked on a malicious link. After 8 months, more than half had clicked at least once. The risk isn’t confined, then, to a handful of careless people who just need retraining. Over successive campaigns it reaches more than half the workforce, whenever the right lure lands at the wrong moment.

A 100% completion rate is still consistent with all three findings. It tells you each person opened the module. It tells you nothing about how that same person will react to the next phishing email. To measure training effectiveness, that reaction is what you would need to observe.

Over 14,000 employees tracked for fifteen months, and the same finding on embedded training

You could see this as an isolated case, specific to one hospital and one period. ETH Zurich had run a study on a similar scale a few years earlier. Lain, Kostiainen, Čapkun and their colleagues followed more than 14,000 employees of a large company for fifteen months, and presented their results at the 2022 IEEE symposium.

Their finding on embedded training goes further than San Diego’s. As deployed in industry, it doesn’t make employees more resistant, and it “can make employees even more susceptible to phishing”. The authors do, however, identify an approach that holds up. Relying on employee reports as a collective phishing sensor remains workable in a large organisation.

Time works against once-a-year training too. Reinheimer and colleagues tracked the effect of awareness training over several months, in a study presented at the SOUPS conference in 2020. After 4 months, participants were still significantly better than before the training. After 6 months, the gap with their starting level had gone.

I run the numbers for a single annual session. If the effect fades somewhere between the fourth and sixth month, at least half the year goes by with no measurable effect. The attendance record, meanwhile, stays up to date for all twelve months.

The Kirkpatrick model doesn’t count completed modules

Training managers have long had a framework for measuring training effectiveness. Donald Kirkpatrick described it in the late 1950s, and it splits evaluation into four levels. The definitions below come from Kirkpatrick Partners, the organisation behind the Kirkpatrick model today. The last column gives a security example.

Level What it measures Security example
1. Reaction Whether the audience finds the training “favorable, engaging, relevant” Satisfaction score at the end of the module
2. Learning The knowledge and skills acquired Quiz passed after the module
3. Behavior Whether the audience “performs the critical behaviors in its environment” Real phishing email reported, vulnerability spotted in code review
4. Results The outcomes the organisation expects Fewer incidents, fewer vulnerabilities in production

The completion rate doesn’t appear in any row. It counts the people who went through the module, without even asking what they thought of it. It’s often the figure that gets presented, because it’s what audits check.

The PCI DSS standard for payment cards is a case in point. To assess a security awareness programme, its testing procedure 12.6.3.a tells the assessor to “Examine security awareness program records to verify that personnel attend”. So attendance records are enough to tick the box.

Entry turnstile with a mechanical counter in front of an empty corridor

The counter logs every pass through the turnstile, and none of what happens in the corridor.

“But our phishing simulations show a drop”

That’s the objection I’d expect to hear in the committee room, and it rests on real figures. KnowBe4, a security awareness training vendor, publishes a benchmarking report by industry. Its 2025 edition reports the share of employees likely to click falling from 33.1% to 4.1% after twelve months of training.

That figure comes from a vendor that sells this training, and it’s measured on the vendor’s own simulations. I suspect employees end up recognising the style of the campaigns. The San Diego and Zurich studies also measure clicks on simulations. San Diego, however, rotated through ten types of lure, and more than half the workforce eventually clicked.

Ariana Mirian took the subject further in August 2025 at Black Hat USA, with Christian Dameff. The title of their talk says it all: “Pwning Phishing Training Through Scientific Lure Crafting”. What I take from it is that if you know how to craft a lure, you also know how to craft an easy one. An easy campaign brings the figure down without anyone becoming more careful.

A similar bias shows up among developers who review code. An explicit instruction to look at security increases vulnerability detection in code review eightfold, according to a study presented at ICSE in 2022. So a test people know is coming measures test-day vigilance. It overestimates vigilance on an ordinary Tuesday, when the same pull request arrives in the middle of all the others.

Measure training effectiveness without warning, every 4 to 6 months

The frameworks already point this way. NIST SP 800-50 Rev. 1, a US guide to building a training programme, was published on 12 September 2024. It includes “suggested metrics and evaluation methods to regularly improve and update the program”. ISO/IEC 27001, in clause 7.2, requires organisations to evaluate the effectiveness of the actions taken to acquire competence. Article 21 of the NIS2 Directive requires “policies and procedures to assess the effectiveness of cybersecurity risk-management measures”. Among those measures, the same article lists “basic cyber hygiene practices and cybersecurity training”.

To evaluate training effectiveness at level 3, I’d stick to four steps:

  • choose an observable behaviour for each population: for all employees, phishing reports and how quickly they arrive; for developers, what code review lets through;
  • measure without warning, in situations close to everyday work, so you aren’t measuring test-day vigilance;
  • repeat the measurement every 4 to 6 months, since the effect of awareness training fades between those two points;
  • track reports as a collective sensor, rather than clicks as a list of culprits.

Measuring doesn’t remove the need for the technical protections the San Diego researchers recommend. They cite two-factor authentication, and password managers that only fill in credentials on the right domain. The completion rate keeps its place too: auditors need it to verify that the programme exists.

For developers, code review already provides the raw material. Findings from code reviews and code testing, sorted by vulnerability type, show what still slips through after training. It’s also what I’m measuring at the moment: what developers spot, or let through, when they review code written by AI. I wrote about it in an earlier article on code review.

Panel of needle dials, one pinned against its maximum stop

A needle pinned against its stop no longer moves, whatever happens.

A page closed straight away can count as completed training

I keep coming back to the third of trained employees in San Diego who closed the training page as soon as it opened. Each of them received the training, in the sense that the tool displayed it. A tool that counts displayed pages lumps them in with the people who read it to the end.

A completion rate can count that displayed page. So alongside it, I’d present the committee with a behavioural indicator, measured without warning and repeated every 4 to 6 months. For employees, I’d use phishing reports. For developers, the vulnerabilities that get past review.

I still have one doubt, because measuring behaviour costs more than counting modules. I don’t yet know what pace a mid-sized team can sustain without wearing anyone out. To measure training effectiveness, I’d start with a single population and a single behaviour. Then I’d see whether the curve moves.

Frequently asked questions about how to measure training effectiveness

How do you measure training effectiveness?

Measuring training effectiveness means checking that behaviour has changed on the job, which the Kirkpatrick model calls level 3. In security, you track an observable action, such as reporting a phishing email or spotting vulnerabilities in code review. You measure it without warning, and repeat the measurement every 4 to 6 months.

What is the Kirkpatrick model?

Donald Kirkpatrick described this model in the late 1950s. It evaluates training on four levels: participants’ reaction, what they learned, their behaviour at work, then the results for the organisation. The reference definitions are published by Kirkpatrick Partners.

What is a training completion rate?

The completion rate is the share of enrolled people who have finished a module. It shows that people got through the module, which is often what audits check, without measuring learning or behaviour. Across 19,500 employees, the UC San Diego study found no significant relationship between annual training and the risk of clicking.

What metrics should you track for security awareness training?

For all employees, I’d track the phishing reporting rate and time to report, along with the share of repeat clicks, monitored at team level. For developers, I’d look at the share of vulnerabilities that get past code review. The NIST SP 800-50 Rev. 1 guide also suggests metrics and evaluation methods for keeping a programme up to date.

Sources