Why Most Penetration Tests Are Theater: A CISO's Guide to Vetting a Real Offensive Security Partner

Somewhere in the last decade, "penetration testing" became a category label loose enough to cover everything from a genuinely manual, attacker minded engagement run by skilled testers, to an automated vulnerability scan that gets a report template, a logo, and an invoice attached to it. Both get sold under the same name. Both satisfy a compliance checkbox that just says "penetration test completed." Only one of them actually tells you whether your organization can withstand a real attack.

If you are a CISO or security leader in the US or UK evaluating offensive security vendors, this post is a direct, practical guide to telling the two apart before you sign a contract, not after a breach reveals the gap.

The economics here are straightforward, and worth understanding before you evaluate any vendor. A fully automated scan run through a commercial tool costs a vendor very little in tester time and can be sold at a price point that undercuts genuinely manual testing significantly. Wrap that scan output in a branded PDF, add a short executive summary, and the result reads like a penetration test report to anyone who has not seen the difference up close.

The tell is almost always in the findings themselves. Automated scan based reports tend to list a high volume of low severity, often informational findings, such as missing security headers or outdated software version banners, with few or no findings that demonstrate an actual exploit chain or business impact. They rarely include evidence of manual testing such as business logic abuse, authorization bypass across multiple steps, or chained exploitation of the kind we documented in our recent post on real OWASP Top 10 exploit chains found in production environments. If a report reads like a list scanners could have generated with no human judgment applied, that is usually exactly what it is.

What percentage of this engagement is manual testing versus automated scanning. A credible vendor should be able to give you a clear answer, generally the majority of engagement time for a proper penetration test should be manual, with automated scanning used as a starting point for coverage rather than the deliverable itself.

What certifications do the specific testers assigned to my engagement hold. Ask for testers by name and certification, not just a generic statement that "our team is certified." OSCP and CEH are widely recognized benchmarks for hands on offensive security skill, and a vendor confident in their team will share this without hesitation.

Can I see a sample report with the sensitive client details redacted. This is the single fastest way to evaluate a vendor. A real manual engagement report documents methodology, includes screenshots or proof of concept evidence for each finding, and explains business impact in plain language a non technical stakeholder can understand. A scan based report typically looks thin, generic, and heavy on tool output formatting.

What is your retest policy once we remediate findings. A vendor who does not include retesting, or charges a substantial additional fee for it, is signaling that closure and verification are not really part of their process, which matters enormously for any compliance framework that requires documented remediation evidence.

Is your testing methodology transparent and documented, such as alignment with OWASP, PTES, or NIST frameworks. Vendors following a recognized methodology can explain their process step by step. Vendors who cannot describe their methodology beyond "we run our tools and review the results" are telling you something important about the depth of the engagement.

How do you scope the engagement, and does scope include business logic testing specific to my application, not just generic vulnerability categories. Every application has unique business logic, the rules around what a given user should and should not be able to do, and that logic cannot be tested by a generic scanner. A vendor who scopes purely around checklist categories without first understanding your application's specific workflows is not set up to find your most damaging vulnerabilities.

A scan based engagement typically takes one to two days of actual tester effort, produces a report dominated by low and informational findings, includes little to no exploit chaining, and offers retesting only as a paid add on. The price reflects the low effort involved, and the report, while official looking, tells you very little about whether a skilled attacker could actually compromise your environment.

A genuinely manual engagement typically runs a full week or more depending on application complexity, involves testers actively reasoning about your specific business logic and combining findings into realistic attack chains, includes detailed proof of concept evidence for each finding, and bundles retesting into the engagement as a standard part of the process rather than an upsell. The findings are fewer in number but dramatically higher in actual risk relevance, which is the entire point of the exercise.

Every engagement we run is led by OSCP and CEH certified testers performing genuinely manual, scenario based testing, the same approach documented in the real exploit chains we have published from past engagements. Across more than 18,000 vulnerabilities identified for our clients, our focus has consistently been on findings that demonstrate actual business impact through chained exploitation, not high volume, low value scan output. Retesting is included as standard, and our reports are built to satisfy both technical engineering teams and the compliance and audit requirements covered in our recent guide to GDPR and NIS2 penetration testing obligations.

Take the six questions above into your next vendor conversation, whether that is with us or with anyone else you are evaluating. A vendor confident in the depth of their testing will answer directly and will be glad to share a sample report. A vendor who hesitates, deflects, or cannot name the certifications their testers hold is telling you, indirectly, which category of engagement you are actually buying.

Request a sample report from Nuage Security and compare it directly against whatever you have on file from your last engagement. The difference is usually immediately obvious, and it is the fastest way to know whether your organization's last "penetration test" actually tested anything at all.