All articles

Blog

Secure coding training: the score is no measure of vigilance

Secure coding training compared: PortSwigger, Juice Shop, Secure Code Warrior, classroom. What each option measures, and what it misses in review.

  • By Azziz ERRIME
  • 11 min read
Secure coding training: a review table seen from above, with a single stack of pages lit by a lamp

TL;DR

  • Secure coding training should be judged by what developers do when nobody prompts them, and none of the options compared here measures that in real code review.
  • In an experiment published in 2022, asking reviewers to focus on security made them eight times more likely to detect the vulnerabilities.
  • I’d choose the platform for what it teaches the team, and measure vigilance on top of that, in real code reviews.

In an experiment published in 2022, researchers asked 150 developers to review a vulnerable code change, and told only some of them to think about security. That group spotted the flaws eight times as often. Every secure coding training programme runs into this result: it measures what developers can do when they’re asked to.

Larissa Braz, Christian Aeberhard, Gül Çalikli and Alberto Bacchelli recruited 150 developers, 71% of whom reported at least three years of professional experience. Each was given a code change to review, the kind that goes through a pull request every day, and the change contained vulnerabilities. Some participants read one extra instruction, to focus on security, and others were also handed a checklist. Then each of them decided, alone with the code, what to flag.

The result was presented at ICSE in 2022. Their paper sums it up: “simply asking reviewers to focus on security during the code review increases eight times the probability of vulnerability detection”. The checklist, even when tailored to the change under review, added nothing significant. The authors also point out in passing that developers “often miss even popular and easy-to-detect vulnerabilities during code review”.

The reviewers who got no instruction had no reason to be any less competent than the rest. They were simply looking elsewhere, a situation that none of the training options compared below reproduces.

A reviewer’s attention matters more than what they know

The authors frame this as a “mental attitude” hypothesis: reviewers find what they look for, and they look for what they’ve been asked to look for. Code review, the moment when a developer reads a colleague’s change before it joins the shared codebase, is mostly about logic, naming and tests. Security only comes into it when someone brings it in.

In practice, an injection flaw in a pull request looks like any other line. A variable travels from a form to a database query without being sanitised, and nothing about it catches the eye of someone checking that the feature works. A reviewer who knows how to spot an injection will let it through if they aren’t looking for one. The attacker looks for nothing else: they find the fields that accept input and follow the data all the way to the database.

Yet the platforms compared below ask the question for the reviewer: here is some code, find the flaw. That is exactly the instruction given to some of the reviewers in the experiment. So secure coding training can improve knowledge while telling you nothing about attention.

Secure coding training platforms mostly measure performance on demand

In this table of application security training options, I only include what vendors or the specialist press publish. The classroom row describes common practice, not a particular vendor.

Option What you do What is measured and reported Does the developer know they’re being assessed?
PortSwigger Web Security Academy Web application attack labs, for example 30 on cross-site scripting Labs solved, progress, Hall of Fame ranking; the Academy is free Yes
OWASP Juice Shop Challenges on a deliberately vulnerable application, free and open source Challenge score board Yes
Snyk Learn Lessons and learning paths, free Completion percentage Yes
Security Journey Secure coding training paths, priced on request Assessments, reports to the LMS (SCORM, xAPI) Yes
Secure Code Warrior Code challenges Trust Score per developer; Trust Agent: AI links real commits to the AI tools used Yes for the training, partly for the tracked commits
Immersive (Developer AI Ranges) Pull requests and backlog triage, facing coding agents What was caught or missed, with a reference verdict Yes, the exercise is announced
Classroom training Taught sessions and hands-on exercises with a trainer Attendance sheet, end-of-session assessment Yes

Each has its strength. PortSwigger teaches the mechanics of web attacks without costing a penny. OWASP Juice Shop, free and open source, works well as a workshop playground. Snyk Learn offers lessons and learning paths, also free. Security Journey feeds its results into the company’s training-tracking tool. Secure Code Warrior has developers take on code challenges. Classroom training lets people put questions to a trainer during the hands-on sessions.

Immersive goes furthest towards real conditions. Since 29 July 2026, its Developer AI Ranges have put developers in front of pull requests and coding agents. Each exercise ends with a reference verdict on “what was caught or missed”. It’s the closest public approach to a real review that I’ve found. Even so, the developer knows they’re in an exercise. Only Trust Agent: AI, from Secure Code Warrior, looks at real commits. It cross-references the Trust Score and AI usage with vulnerabilities, but doesn’t say what the review let through.

If none of these measures observes the review itself, what does the auditor look at when checking developer security training?

Fairground shooting gallery with the targets hit dead centre, one target knocked over in the shadows

The announced targets are hit dead centre. The everyday one waits in the shadows.

The audit signs off the training without seeing Monday’s review

Take PCI DSS, the payment card security standard. Its requirement 6.2.2 calls for developers working on bespoke and custom software to receive security training at least once every 12 months. To check it, the assessor follows a specific procedure, 6.2.2.b: “Examine training records and interview personnel”.

The records show who completed which training, on what date and with what score. That is precisely what the platforms in the table report, in Security Journey’s case straight into the training management tool. Nothing in this process looks at what became of the last pull request that touched a database query.

I have no quarrel with this process: it proves that training took place, which is what it’s asked to do. The risk lies in what people infer from it: a 100% completion rate reassures the executive committee about a skill nobody has seen in practice. The gap widens with AI, which increases the volume of code to review, as I explained in an earlier article on code review.

That leaves the strongest objection: some platforms claim they do measure an effect on vulnerabilities, not just attendance.

A drop in vulnerabilities after secure coding training doesn’t show where the vigilance comes from

On 15 October 2024, Secure Code Warrior published a study, reported by CyberScoop, based on data from 600 enterprise customers. In organisations with more than 7,000 developers following the same programme, the vendor measured 47 to 53% fewer vulnerabilities.

I take it seriously, because it measures an outcome, vulnerabilities, where many training studies stop at satisfaction. It is also a vendor study, based on data from its own platform, and the public documents I’ve been able to read don’t explain how the vulnerabilities were counted. It may be accurate and still not answer the question this article asks.

A fall in vulnerabilities across the code of a large, trained organisation mixes several causes: better coding habits, better-tuned analysis tools, more attentive reviews. The study doesn’t tell you which of them was at work. And vigilance in review is precisely the part that no row of the table observes within the review itself.

Secure coding training is still necessary, then: without it, nobody knows what an injection looks like in the middle of a pull request. What remains is to add a measure of what happens outside the exercise.

A security line in the pull request template is the cheapest step

You learn on the platform that best fits the team’s tech stack. You measure elsewhere, in real work, what the review lets through. For a team, here is what I’d do, in this order:

  • start with the free resources, PortSwigger, Juice Shop and Snyk Learn, to see what engages the team before signing a contract;
  • choose the paid platform for your languages and frameworks, not for the size of its catalogue;
  • ask the vendor what can be exported: a completion rate, or per-skill results you can cross-reference with your reviews;
  • track separately what the review lets through, by linking each vulnerability found later, by a test or in production, to the review that let it pass;
  • add a line to the pull request template asking whether security has been reviewed.

That last line is the cheapest step, and the closest to the instruction tested in 2022. I don’t yet know whether it keeps its effect once it becomes routine.

For the record: I run training sessions and I measure this kind of gap, what developers let through when they review AI-written code. Training gives them the vocabulary. What I’m trying to find out is when they use it without being asked.

Empty railway signal box, one lever marked with a red ribbon

The step that counts happens in the signal box, as the train goes through.

The reviewer with no instruction is your team on a Monday morning

Back to the 150 developers in the experiment. Those who received no instruction look exactly like your teams on a Monday morning, with a queue of pull requests to clear before the meeting. They may have learned to recognise a vulnerability, but nobody asked them to look for one at that moment.

Well-chosen secure coding training teaches them what to look for. No training guarantees they will look. So I wouldn’t choose a platform for its catalogue, or for the average score it displays. Against every euro spent on training, I’d set a measure of what gets through real review.

I still have one doubt: as soon as developers know their reviews are being watched, they review more carefully. The code gains, and the measurement loses, because it turns back into an instruction. I haven’t yet found a way to observe vigilance without changing it.

Frequently asked questions about secure coding training

What is secure coding?

Secure coding covers the practices that keep vulnerabilities out of code as it is written: validating input, enforcing access control, choosing safe dependencies. The PCI DSS standard makes it a mandatory training topic for developers of bespoke and custom software. Its requirement 6.2.2 calls for training at least once every 12 months, covering, among other things, “secure coding techniques”.

What secure coding training should developers take?

For a first round of secure coding training with no budget, PortSwigger’s Web Security Academy is entirely free, as are OWASP Juice Shop and Snyk Learn. Paid platforms add, among other things, assessments and reports for training tracking. Either way, measure separately what gets through your code reviews, where a simple security instruction makes a marked change to detection, according to Braz and colleagues.

Is there free OWASP secure coding training?

Yes. OWASP Juice Shop, an OWASP Foundation project released under the MIT licence, is a deliberately vulnerable application whose challenges you track on a score board. For a more guided path, PortSwigger’s Web Security Academy, also free, covers the main families of web vulnerabilities.

What is Secure Code Warrior?

Secure Code Warrior is a secure coding training platform that gives each developer a Trust Score. Since March 2026, its Trust Agent: AI module has been linking commits to the AI tools that influenced them, according to Help Net Security. A vendor study published in October 2024 claims 47 to 53% fewer vulnerabilities in large trained organisations.

Sources