Choose the right security assessment
Compare vulnerability scanning, penetration testing, red-team exercises and collaborative testing with three worked commissioning scenarios.
Choose the assessment by the question it must answer and the evidence you need afterwards. Buying the most elaborate sounding exercise can leave a simple, important question unanswered.
The comparison below is a commissioning aid. Suppliers use these terms differently, so put the intended activities and deliverables in the scope rather than relying on the label alone.
Compare the questions each approach answers
| Approach | Useful question | Typical output | Important limitation |
|---|---|---|---|
| Vulnerability scanning | Which known weaknesses or exposed services should we investigate across these assets? | Repeatable observations, affected assets and candidate findings | Automated results need validation; absent findings do not establish that business logic is safe |
| Penetration testing | Can an authorised tester demonstrate a significant security failure within this boundary? | Supported findings, impact, coverage and remediation advice | The agreed scope, available access and time constrain the result |
| Adversary simulation or red-team exercise | Can we prevent, detect and respond to an agreed realistic attack objective? | An account of the attempted objective, control observations and response gaps | It does not promise a complete inventory of vulnerabilities |
| Collaborative defensive exercise, often called purple teaming | Can testers and defenders improve a particular detection or response together? | Tested observations, revised controls and evidence of repeatability | Coaching and shared knowledge change the conditions; it is not a blind assessment |
The NCSC's adversary-simulation guidance emphasises prevention, detection and response against realistic objectives. It distinguishes this from a penetration test's emphasis on technical vulnerabilities. That distinction matters when you decide what a successful engagement should produce.
Scenario 1: a small organisation with an uncertain inventory
Fictional situation: Cedar Arts has a public website, cloud email and several old supplier-managed services. Nobody can confidently list every internet-facing system. The request is “give us a red team”.
Better first question: What do we operate, who is responsible for it, and which known issues need attention?
Start with an agreed asset-discovery and vulnerability-management task on systems the organisation is authorised to assess. Resolve ownership and unsupported services. Commission targeted testing where the inventory and business risk justify it. A covert exercise would be difficult to interpret while ordinary asset ownership remains unknown.
Decision evidence: a checked inventory, named owners, confirmed exposure and a remediation list. This is a foundation for further testing, not a claim that the whole organisation is secure.
Scenario 2: a new customer portal
Fictional situation: Harbour Workshop is about to release a portal containing customers' job records. The team has automated checks, but has not independently examined cross-customer permissions.
Question: Can one customer view or change another customer's records through the supported portal functions?
Choose an application penetration test with representative customer, staff and administrative roles. Include authentication, authorisation and the associated APIs in the agreed boundary. The testers need accounts and realistic test data. A scan alone may identify useful technical issues without answering this particular relationship between users and records.
Decision evidence: which roles and functions were examined, supported findings, remaining limitations and retest results for issues affecting release. Continue the service team's ordinary testing alongside the independent assessment.
Scenario 3: an established monitoring team
Fictional situation: A larger logistics organisation already maintains its asset inventory and vulnerability process. It has a staffed monitoring function and an exercised response plan, but wants to examine whether those capabilities work together.
Question: Would the organisation detect and contain a specified route to unauthorised access to a representative scheduling system?
Consider a tightly controlled adversary simulation. Agree the objective, permitted routes, safety controls and who is aware. Where the immediate goal is to tune a known weak detection, a collaborative exercise may produce more useful learning than a covert assessment.
Decision evidence: the attempted sequence, telemetry available to defenders, decisions made and improvements that can be verified. Success or failure on this objective does not establish the security of every system.
Information level is a separate choice
A supplier may describe testing as black box, grey box or white box. These labels describe how much information or access is supplied; they do not define a reliable level of quality. Ask exactly what the tester receives: architecture, source code, accounts, configuration, monitoring information or none of these.
Providing relevant information can let a time-limited assessment spend more effort on the question you care about. If withholding information is necessary to the objective, explain why and record the resulting coverage limits. NIST's technical assessment guide is an older reference for the benefits and limits of different assessment techniques; use current supplier and platform guidance for the specific technology.
Write a one-sentence commissioning decision
Use this pattern: “We need to decide [business decision], so we will assess [systems and behaviours], expecting [evidence], while accepting [limits].”
For the portal: “We need to decide whether customer access controls are ready for release, so we will assess agreed roles and record functions in the representative environment, expecting reproducible findings and a coverage statement, while recording any production-only dependencies left untested.”
Take that sentence into scoping and authorisation, then compare providers against it.
Sources and further reading
- NCSC: Adversary simulation: what you need to know — checked 2 November 2026
- NIST SP 800-115: Technical Guide to Information Security Testing and Assessment (2008) — checked 2 November 2026