NCC Group says human testers still vital in AI security

NCC Group has outlined why it believes human judgement remains essential in AI-driven security testing, arguing that AI systems still struggle to find context-specific flaws.

AI tools now play a significant role in routine testing, vulnerability detection and exploit generation, the cyber security consultancy said. But human testers are still needed to identify business logic errors, novel attack chains and risks shaped by organisational context.

Steven van der Baan, Principal Security Consultant at NCC Group, framed the issue around the idea that digital models of a system are only partial representations of how it behaves in practice. He argued that AI can map known patterns from training data, but cannot fully account for the more complex and less predictable conditions in which many serious weaknesses emerge.

He cited one example in which a penetration tester found a flaw in an online checkout process after automated tools failed to detect anything significant. The issue allowed a discount code to be applied repeatedly during a cart abandonment sequence, creating an indefinitely compounding price reduction.

According to van der Baan, the vulnerability arose from the interaction of features built by different teams over several years. Problems like this often sit outside standard databases of known vulnerabilities and beyond the range of tools trained to recognise established patterns, he said.

AI adoption

The comments come as AI tools become more common in security research and testing. Van der Baan pointed to examples including Google's Big Sleep, which he said found a confirmed zero-day vulnerability in SQLite, and AI-enhanced fuzzing systems that have uncovered vulnerabilities in open-source software including OpenSSL.

He also cited autonomous testing products and AI features in established security platforms. These systems are increasingly used for early-stage scanning, matching exposed software against known weaknesses, reducing false positives and examining findings generated by existing tools, he said.

The wider market has moved quickly in that direction. NCC Group said 70% of security researchers now use AI tools in their daily work, reflecting how deeply the technology has become embedded in standard workflows.

NCC Group is not arguing that AI has little value in security testing. Instead, it says the strongest use case is to augment human researchers rather than replace them.

Known patterns

At the centre of van der Baan's case is the distinction between recognised attack types and flaws that depend on a specific business process or environment. He said AI systems are effective when an application's weaknesses resemble material already present in sources such as CVE databases, exploit archives and published vulnerability disclosures.

That approach works well for established classes of weakness. It becomes less reliable, he argued, when the vulnerability lies in how a particular organisation has designed a workflow, connected systems or set assumptions across departments.

Business logic flaws are a common example. A failure in one team's handling of account status, pricing rules or permissions may not become visible until another system interacts with it in an unintended way. Such issues may bear little resemblance to a previously documented software defect, even if the commercial impact is severe.

Van der Baan also argued that experienced testers are better placed to combine several modest findings into a more serious attack path. In many cases, account takeover or privilege escalation does not stem from one obvious vulnerability but from a sequence of smaller issues that only become significant when linked together.

Risk context

NCC Group also said human judgement matters when deciding what a finding means for a particular organisation. A technical weakness may carry sharply different consequences depending on the type of data involved, the number and profile of users with access, and the business processes tied to the affected system.

A vulnerability exposing email addresses, for example, may have very different implications in healthcare than in a public-facing sports data service. Likewise, a stored cross-site scripting flaw in an administration panel may pose one level of risk in a small office and another in a distributed organisation with thousands of contractors.

The consultancy extended the argument beyond software testing to social engineering and behavioural risk. It said phishing resilience, exposure to pretexting and the effectiveness of physical security procedures all depend on human behaviour and company culture, which cannot be derived from application telemetry alone.

Van der Baan acknowledged that AI systems are becoming more adaptive, citing tools that reason about application behaviour rather than relying only on fixed signatures. Even so, he argued that they still operate within learned representations rather than a direct understanding of every real-world condition in which software is deployed.

That is especially relevant to the discovery of entirely new classes of weakness. He said novel forms of attack often emerge when a researcher questions assumptions that existing models do not capture, rather than when a system searches for a known pattern at greater scale.

"AI gives us better maps. Faster maps. Maps that cover more known terrain with less effort. That is genuinely valuable, and anyone who tells you otherwise is not paying attention to what is actually happening in the field. But the territory is always larger than the map. It always will be. And that is why the human security tester will always have work to do," said van der Baan.