Most organizations test once a year. A low-yield engagement leaves you with a 12-month blind spot in your defensive posture, and you rarely find out it was wasted until an incident forces the issue.
The outcome depends far more on your preparation and oversight than on the vendor’s wizardry.
Before You Buy
Scope Determines Value
The single biggest lever is scope. A narrow scope yields a clean report and zero security insight. We covered scoping strategy in How to Scope a Penetration Test, which is the place to start. The advice below assumes you have already defined what is being tested and why.
Vet the Vendor on Evidence
Certs prove an individual passed an exam; they do not guarantee rigorous fieldwork or an actionable report. Ask for:
- A redacted sample report. It is the most informative thing you can inspect. If it reads like a Nessus or Qualys dump, pass.
- Who actually performs the test. The person on the sales call is often not the person doing the work. Ask who is assigned and what they have tested recently.
- Their methodology, and whether it is a recognized framework (PTES, the OWASP Testing Guide) or something invented in-house.
- The ratio of manual to automated work. Every test starts with scanning. The question is what happens next.
- How false positives are handled before the report reaches you.
- Whether retesting is included, and for how long afterwards.
What a Bad Report Looks Like
You can usually tell from the sample. Warning signs:
- It is a scanner export with a cover page and a logo
- Findings have no reproduction steps you could follow yourself
- Recommendations are boilerplate: apply vendor patches, implement least privilege
- Severity is raw CVSS with no reference to your environment
- Nothing in it demonstrates the tester understood what your business does
A good report contains findings a tool could not have produced on its own: chained access, broken business logic, an authorization boundary that only makes sense once you know how the application is meant to work.
Prepare the Environment
Fix What You Already Know About
Paying a third-party specialist to find unpatched CVEs you already know about is a waste of your budget.
Run internal vulnerability scans first, patch outdated dependencies, and lock down obvious configuration drift. Clear the baseline noise so the assessment team spends their billable hours breaking your actual logic instead of rediscovering technical debt.
Have the Logistics Done on Day One
Tester idle time is expensive. Have this checklist verified 48 hours prior to kickoff:
| Ready before kickoff | Why it matters |
|---|---|
| Test accounts at every privilege level | Provisioning mid-test can cost days |
| VPN or network access, verified by someone | “It should work” is not verification |
| Source IPs allowlisted where needed | Or a decision that they deliberately are not |
| An asset list with named owners | So questions reach whoever can answer them |
| Backups and a rollback path | For anything being tested in production |
| A named point of contact, reachable | Not a shared inbox |
Make Deliberate Choices on Defenses
If a WAF, EDR, or IPS sits between the tester and the target, you have a choice, and either answer is defensible as long as it is deliberate:
- Leave it on to see what your stack actually stops in practice.
- Allowlist the tester to test the application itself rather than the shield in front of it.
The worst outcome is not choosing: spending half the engagement getting rate-limited and calling the result a clean test. A common compromise is to allowlist for the bulk of testing, then spend a defined block of time with protections live to see what gets caught.
During the Test
Be responsive. Answer questions the same day, because a blocked tester is burning hours you paid for.
Use short syncs. A 15-minute daily or alternate-day standup catches wrong targets, credential rot, and testing hurdles while remedies are still cheap.
Do not quietly fix things mid-test. It is tempting when a critical lands on day two, but silently patching breaks reproduction, invalidates the finding, and can make the tester think their access was revoked. If something is urgent enough to fix immediately, fix it and tell them, so the report records both the finding and the remediation date.
Expect the critical call before the report. Anything severe should reach you during testing. If you get to the debrief and hear about a critical for the first time, that is a process problem worth raising.
Reading the Report
Look past the executive summary. The technical appendix is where your engineering team lives.
Push back on severity where the context is wrong. Testers rate findings from outside your business. An issue on an isolated lab system and the same issue on your payment path do not deserve the same rating, and a good tester will happily adjust when you explain why.
Ask for reproduction on anything you doubt. A finding that cannot be reproduced on request should not be in the report. Asking is reasonable and a competent tester will not take it personally.
Look for what is missing. If the report contains nothing about authorization, or nothing about your identity provider, ask whether those were tested and found sound, or simply not reached. There is a large difference and it will not always be stated.
After the Engagement
Fix the class, not the instance. If SQL injection turned up in one endpoint, the real finding is that your data access layer allows string-concatenated queries. Fix the one that was found and the pattern that produced it, or you will buy the same finding again next year with a different URL.
Assign an owner and a date to every finding. Findings without a named person do not get fixed.
Use the retest. It is usually included and it routinely goes unused. It is also the only thing that converts “we fixed it” into evidence.
Feed the findings into detection. Every technique the tester used successfully is one your monitoring did not catch, which makes the report a free, environment-specific detection backlog, often worth more than the vulnerability list itself. That work is a whole discipline of its own, which we wrote about in tuning out alert noise.
Track recurrence over time. Count how many of last year’s findings came back; that tells you more than the size of this year’s list.
When to Test
An annual audit satisfies compliance frameworks, but changes introduce risk. Schedule targeted tests:
- Before a major application or infrastructure launch
- After a significant architecture change, cloud migration, or acquisition
- When the previous test’s critical findings have been remediated and need validating
- Annually regardless, as the baseline
What Success Actually Looks Like
A clean report is not success. Against a real scope, it usually means the scope was too narrow.
A successful test gives you findings you could not have found yourself, in enough detail to fix them, prioritized in a way that matches your business, and a clear path to confirming the fixes worked. If you finish the engagement knowing something you genuinely did not know before, and you have owners assigned to act on it, it worked.
If you are planning your next penetration test and want a second opinion on scope, on a proposal you have been sent, or on what to do with a report you have already received, get in touch.