The Decline of Real Penetration Testing

As the leader of Cyber Agency, one of the largest penetration testing and red teaming specialist outfis, I have a bone to pick with where our industry is headed. Back in the late 90s and early 2000s, “pentesting” actually meant breaking into systems (and sometimes buildings!) with skill and creativity. Today, I’m watching a disappointing shift: more and more so-called pentests have turned into cheap, automated, check-the-box exercises that barely scratch the surface. This isn’t just a minor annoyance; it’s an affront to everything we hackers-for-hire used to stand for.

When Hacking Was an Art, Not a Checklist

Let me paint a picture of the glory days: At Cyber Agency, a penetration test was all-out warfare (in the best sense). We never failed to capture the flag on an engagement – ever. If a client challenged us to breach their fortress, we took that as a personal dare. My team would spend 12 to 18 months on a single project, meticulously plotting and probing every layer of security. We wrote custom exploits from scratch when no tool could get us in. We phished employees with crafty social engineering ploys. We even packed our bags for physical penetration ops – sneaking into offices at night, evading alarms and guards, just to prove we could. Nothing was off-limits if it helped us uncover a vulnerability. It wasn’t illegal; it was authorized mayhem in the name of better security.

And guess what? These extensive engagements didn’t come cheap. A full red-team project from us ran in the ballpark of $2M and could last over a year. By the end, we’d hand over a report that often weighed in at 1000+ pages – no fluff, just detailed documentation of every hole we found, exactly how we exploited it, and step-by-step recommendations to fix it. Our reports were basically a comprehensive blueprint for securing the organization. Clients got their money’s worth in blood, sweat, and zero-days. This was penetration testing as an art form and a science. It felt good to do it right.

Now, compare that to what I’m seeing creep into the market… cue eye roll.

Enter the Check-Box Charlatans

Somewhere around the turn of the millennium, a new breed of “penetration testing” providers started popping up. I call them check-box charlatans. These folks aren’t here to truly challenge your security – they’re here to tick off a compliance checklist and call it a day. It’s pentesting for those who want the sticker without the work. And it’s driving me up the wall.

Here’s the typical scenario: a CIO (who’s more interested in pleasing auditors than actually securing systems) decides they need a pentest report on file. Not an actual security assessment, mind you – just a document to wave at senior execs and regulators to say, “Look, we did it!” Instead of hiring an expert team for months, they opt for a bargain-basement gig. Some firm comes in, runs a quick automated scan of the network, maybe performs a handful of trivial password tests, and boom – two weeks later you get a shiny report. Everyone checks the box, “security audit completed.” Job done, right?

These engagements are cheap and quick by design. They have to be – their only goal is to produce something (anything!) that looks like a pentest report, as fast as possible. Whether it provides real insight is apparently irrelevant. The sad part is, clients are eating it up because it’s easy. Why spend $2M and a year of effort on a thorough test when a $50,000 automated scan will get the auditors off your back next month? Never mind that the latter is practically worthless for actual security – it’s all about that paper trail.

Let’s compare the two approaches side by side for clarity:

Real Penetration Testing (Our Approach)

• Scope & Duration: Engagements running 12+ months, deep-diving into networks, applications, and even physical facilities. No stone unturned.
• Techniques: Custom exploits developed for unique client systems; full red-team ops including social engineering, on-site intrusion, and stealthy persistence. We simulate real adversaries.
• Expertise: Highly skilled team of professionals brainstorming creative attack vectors daily. It’s a craft honed by experience and curiosity, not just tool output.
• Deliverable: Massive, bespoke report (often 1000+ pages) detailing every finding with proof-of-concept, impact analysis, and tailored remediation steps. Essentially a security improvement roadmap.

“Check-the-Box” Pentesting (The New Fad)

• Scope & Duration: A few days to a week, focused on whatever will satisfy a minimal requirement. Very narrow and rushed.
• Techniques: Largely automated scanning tools (point-and-click hacks). Little to no manual exploration, and certainly no physical tests or creative social engineering.
• Expertise: Maybe one or two junior consultants following a script or using tools they barely understand. It’s more factory assembly line than craft.
• Deliverable: A slim, generic report (perhaps 20-30 pages) with a bunch of scanner printouts and boilerplate text. It checks the compliance boxes but offers virtually no insight. It’s security theater, printed and bound.

As you can tell, I have opinions about this.

Security Theater at Its Finest (And Most Dangerous)

The biggest problem with these drive-by pentests isn’t just that they’re low-effort; it’s that they give a false sense of security. Executives see the word “Penetration Test” on a report and think they’re safe for another year. News flash: you’re not. A superficial test that exists solely to meet an audit requirement will not stop real attackers. In fact, it may be leaving you even more vulnerable by lulling you into complacency.

Let’s be clear: a “check-the-box” pentest often gives organizations a misleading sense of safety by focusing on meeting audit expectations rather than simulating real-world threats. It’s basically security theater. You go through the motions, you get the certificate, but nothing really changes. Meanwhile, the bad guys, who don’t give a damn about your compliance checklist, are laughing all the way in.

These minimal tests usually cover only the surface-level vulnerabilities, the low-hanging fruit. Anything complex, unique, or requiring real attacker creativity is missed entirely. The testers aren’t incentivized to think outside the box; they just need to say you’re “compliant.” They fail to consider your organization’s unique environment and threat landscape. Context matters in security – a generic scan won’t reveal an exploit chain that uses your custom software and a clever phishing ploy, for example. But hey, those things aren’t on the checklist, so they get ignored.

The result? Companies end up with generic reports lacking any real insights or actionable recommendations . I’ve seen some of these reports – they’re practically templated. Page 1: Executive Summary saying “Overall risk: Low.” Pages 2-10: a bunch of cookie-cutter findings copy-pasted from a tool (half of which might be false positives or trivial issues). No prioritization, no context, no deep dive into what a breach would actually look like in that environment. No discussion of the human element, or how an attacker could chain issues together. Essentially, it’s a participation trophy in the security world. “Thanks for playing, here’s your certificate of attendance.”

This is dangerous. It breeds complacency. Management thinks, “We passed the test, so we must be secure, right?” Wrong! Those superficial assessments often overlook critical vulnerabilities and give a snapshot in time that can be obsolete the minute a new exploit is discovered. It’s all for show – a fancy (and expensive) method of putting their minds at ease – at least for a little while. Meanwhile, actual attackers are constantly evolving, finding the gaps left by compliance-focused checkups. While you’re proudly displaying your pentest report in a binder, attackers are exploiting the very things it failed to catch.

Why Did We Let This Happen?

So how did we go from Pentesting-as-an-Art to Pentesting-as-Paperwork? A few reasons, from my admittedly cynical point of view:

The Rise of Compliance Regulations: As soon as regulations and industry standards started mandating security testing, companies scrambled for the easiest way to comply. The path of least resistance? Hire someone cheap to do a quick scan and check the required boxes. It’s not about security, it’s about avoiding fines and bad press. Ironic, since a real breach causes far worse press than a failed audit… but hey, humans are fantastic at short-term thinking.

Budget Crunch and Shortcut Mentality: The early 2000s haven’t been kind economically (dot-com bust, anyone?). CFOs are pinching pennies, so the idea of a “low-cost pentest” gained traction. Never mind that you get what you pay for. Try explaining to a boardroom that they should spend millions on something when a $20k scan gives them a report too. They’ll choose the cheap route nine times out of ten, especially if they don’t understand the technical nuance (which, let’s face it, many don’t).

Vendor Proliferation and Marketing Hype: Suddenly every IT consultancy and their cousin is offering “penetration testing services.” Some of these folks have no business doing so. I’ve encountered “pentesters” who basically run Nessus (a popular vulnerability scanner) and generate a report directly from it. That’s it. No manual testing whatsoever. And they market this as a thorough security assessment! The audacity is astounding. But with slick marketing and glossy reports, they convince clients that this is the real deal. It’s not – it’s Vulnerability Scanning 101, masquerading as elite hacking.

Client Complicity: I hate to say it, but some clients want it this way. They don’t want a team like ours coming in and tearing apart their networks for a year, finding inconvenient truths. They prefer a quick pat on the back. I’ve had potential clients explicitly say, “We’re just looking to satisfy our auditors.” At which point I usually bite my tongue to avoid saying something that would definitely not land us the contract. If the market is demanding fast food instead of a gourmet meal, is it any surprise so many junk-food security firms popped up?

An Industry at a Crossroads (Time to Wake Up)

Here we are in 2003, at a crossroads in the security industry. On one hand, the threats out there are real and growing. Organized cybercriminals, state-sponsored hackers, hacktivists – the attacker landscape is getting fiercer by the day. On the other hand, too many businesses are doubling down on appearing secure rather than being secure. It’s all form over substance. And the trend of quick-and-dirty pentesting is a symptom of that broader disease.

My rant (and let’s be honest, this absolutely is a rant) comes from a place of passion for doing things right. I know not every organization can afford a Cyber Agency-style year-long engagement – I’m not totally out of touch. But when you reduce security testing to a mere formality, you’re shooting yourself in the foot. Compliance is not the same as security, and a report is not the same as actual resilience. A cheap pentest that finds nothing (or only trivial issues) doesn’t mean you’re secure; it likely means the test was crap.

As a CEO who built a career on outsmarting bad guys, it drives me nuts to see our craft diluted like this. Penetration testing should be about finding and fixing weaknesses – truly testing your defenses – not just generating paperwork. The value of a real pentest is in the pain it inflicts (uncovering things you never imagined could be exploited) and the improvements it forces you to make. If you’re not getting that, you’re being sold snake oil.

So here’s my plea (or challenge) to organizations and fellow security professionals alike: Don’t settle for check-box security. Demand tests that actually test you. If you’re hiring a firm, ask them how they conduct tests. Do they just run automated tools, or do they employ seasoned experts who think like attackers? Will they tailor their approach to your environment, or is it one-size-fits-all? And when you get that report, it better not be generic fluff. It should contain both bad news and a path to make things better.

Finally, a message to the check-box pentesters themselves (if any of you are reading this): You’re giving our industry a bad name. Cut it out. Yes, you can make a quick buck doing the bare minimum, but you’re not actually helping anyone in the long run. Have some pride – push your clients to do more, or at least be honest about what you’re delivering. Because when a real attack hits (and it will), that flimsy report isn’t going to save your client’s bacon. And guess what? They’ll come looking for folks like us to clean up the mess.

End of rant. I know this has been a long one, but hey, I had a lot to get off my chest. If you’ve made it this far, thanks for listening to an old-school hacker CEO’s grievances. Let’s try to restore some honor to penetration testing – before it completely devolves into a checklist cult. Your security (and mine) depends on it.