Secure Software Development: What an Assessment Actually Tests

Most secure development programmes are measured by what they run - a scanner in the pipeline, a training course, a threat model filed at design time - and not by what those controls actually catch. Two teams can report identical tooling and have wildly different exposure, because the difference is not which scanners exist but which findings reach a developer before release, how fast they are fixed, and whether anything at all is tested for the flaw class that scanners cannot see.

This guide is a working description of what a secure software development assessment measures in 2026, in the order the failures actually happen: the process that produces the requirement, the build that produces the artefact, the code that produces the defect, and the response that decides how long it lives. It is written for the person who has to sign the report and defend it to an auditor, a customer's procurement team or a regulator, and it is explicit about which claims can be proved from artefacts and which are assertions dressed as evidence.

What a secure software development assessment actually measures

Four questions decide the outcome of an assessment, and a report that does not answer all four is a tooling inventory with a cover page. First: does the organisation know what it ships, down to the component and the version, and can it produce that list on demand for a specific release rather than for a repository at some point in the past? Second: does a defect that a tool already reports reach a developer before the release, and does it stay fixed? Third: what does the process do when the flaw is one no tool reports - a missing authorization check, a business-logic shortcut, an unsafe default that is technically documented? Fourth: after a vulnerability is reported by someone outside the team, what is the measured time to a patched release, and is the evidence of that fix preserved?

Each question decomposes into tests with a numeric or binary result, which is what makes the work assessable rather than advisory. For the first: pick one production artefact at random and ask for its software bill of materials, then check whether the listed versions match what is actually deployed. For the second: take twenty findings the scanner reported six months ago and establish, file by file, whether the code changed, whether the rule was suppressed, and whether the suppression expires. For the third: read five features that changed authorization decisions and ask for the test that would fail if the check were removed. For the fourth: measure the interval from a public disclosure to a patched release across the last ten incidents, not the policy that states the intent.

The output is a matrix, not a narrative. Rows are the phases a defect can enter or leave the process - requirements, design, implementation, build, test, release, operation - and columns are the properties that matter in each: whether a control exists, whether it is enforced rather than advisory, what it misses, and what artefact proves it ran. A cell is either measured with a value, or marked untested with the reason and the residual risk. The difference between a matrix and a maturity score is that a maturity score can be raised by opinion, and a matrix cannot.

One boundary belongs at the front, because assessments fail on it constantly: this work is about software the organisation owns or is authorised to test. Reading source you do not have rights to, testing a third-party product without permission, or probing a supplier's environment without a signed scope is not an assessment, it is a liability. Everything described below assumes written authorisation from the system owner, and the section on the lawful boundary at the end states where that line sits.

The 2026 record: three failure modes that keep repeating

The incidents that define the current baseline do not share a single cause, but they cluster into three failure modes, and each one maps to a different part of the process. The first is the trusted component that was not maintained: a library, an appliance or an open-source package that was current when it was adopted and quietly stopped receiving fixes. The second is the build and distribution path treated as infrastructure rather than as production: credentials with publish rights sitting in a continuous integration runner, artefacts assembled from unpinned inputs, and no way to prove after the fact what went into a release. The third is the defect class that no scanner reports: an authorization check missing on one endpoint, a token accepted from the wrong audience, a business rule that lets a refund be issued twice.

Verizon's 2026 Data Breach Investigations Report puts exploitation of vulnerabilities at the centre of 31 per cent of breaches, up from 20 per cent the year before, and the shift is not driven by novel techniques - it is driven by the speed at which a public proof of concept becomes a working exploit against systems whose owners had not yet patched. That number matters for software producers specifically, because the vulnerable component is frequently one they did not write and cannot patch alone; the only controls that help are knowing what is inside the release and being able to rebuild it quickly.

The second and third modes are less visible in breach statistics and more expensive per incident, because they are not fixed by a patch. A build pipeline that can be written to by anyone who can open a pull request is a supply-chain vulnerability in the literal sense: the attacker changes the input, the pipeline signs the output, and the customer installs it with the organisation's own signature attached. An authorization flaw is the same shape of problem from the other direction - the code does exactly what it was told, and the test suite proves it does the wrong thing correctly.

The practical consequence for an assessment is that scanning coverage is a weak proxy for any of these three. A mature scanning programme can coexist with all of them: the trusted component is in the SBOM but nobody reads the SBOM, the pipeline has no provenance because provenance was never a requirement, and the authorization flaw is invisible to a scanner that does not know the intended access model. Ask about the three modes directly and the conversation changes from tools to evidence.

NIST SSDF is the spine, and it is not a checklist

The reference framework in 2026 is NIST SP 800-218, the Secure Software Development Framework version 1.1, published in February 2022. It organises practices into four groups: Prepare the Organization (PO), Protect the Software (PS), Produce Well-Secured Software (PW) and Respond to Vulnerabilities (RV). Its own introduction is explicit that the intent is not a checklist but a basis for planning a risk-based approach, which is the sentence that most assessments skip.

The reason it is the spine rather than an option is that it is the vocabulary procurement and regulation now use. CISA's Secure by Design pledge, announced on 8 May 2024 with more than 250 manufacturers signed on by the end of that year, pushes the same practices from the buyer side, and the European Cyber Resilience Act turns a subset of them into legal obligations with dates attached. An assessment that scores an organisation against SSDF practice groups produces a result that can be reused in a customer questionnaire, an ISO 27001 evidence pack and a CRA conformity file, which is why it is worth using even where no one requires it yet.

Two things the framework does not do, and assessments routinely pretend it does. It does not tell you how much of each practice is enough - PO.4 on defining criteria for software security checks is satisfied by a policy nobody enforces, and by a pipeline that blocks a merge on a failed check, and those are not the same organisation. And it does not measure outcome: RV.1 covers identifying and confirming vulnerabilities, not whether the confirmed ones were fixed within a window that matches the exploitation risk. A score derived from practice presence alone is therefore a floor, and a low one.

The way to use it as a spine rather than a certificate is to map each practice to a measurable artefact and treat absence of the artefact as absence of the practice. PO.1 on defining security requirements maps to a document a developer can point at for a specific feature. PS.2 on verifying software integrity maps to a signature and a verification step in the deployment job. PW.7 on reviewing and analysing code maps to review records and to a rule set whose findings have owners. RV.2 on assessing and prioritising remediation maps to the interval between disclosure and patched release. Where no artefact exists, the honest cell reads "asserted, not evidenced" - and that phrase, once it appears in a report, is usually the most useful line in it.

Where the four practice groups fail in practice

The failures inside each group are specific enough to look for directly, which is what makes an assessment faster than a full maturity programme. Under Prepare the Organization, the recurring failure is that security requirements exist for the product but not for the change: a team can produce a security architecture document and still merge a feature that stores a session token in local storage, because the document was written for the product as a whole and nobody translated it into criteria for the pull request. The second failure is role confusion - a named product security owner who has no authority to block a release.

Under Protect the Software, the failure is almost always the CI/CD system. Build runners hold credentials that can publish to package registries and push images; they accept jobs from branches; they cache dependencies; they run third-party steps with repository secrets in scope. An attacker who can open a pull request, or who can compromise one dependency of an action used by the pipeline, inherits the release authority of the whole organisation. The fix is not exotic - short-lived credentials, environment-scoped secrets, actions pinned by digest, and a separate privileged job for publishing - but it has to be verified, because most pipelines look fine in a diagram.

Under Produce Well-Secured Software, the failure is the gap between the control and the population it covers. Static analysis runs on the twelve services in the main repository and not on the four that were migrated from a previous platform and live in a separate organisation. Dependency scanning runs on application code and not on the container base image, which is where the exploitable library actually sits. The population is the measurement, and it is the number an assessor should ask for first: not how many findings were fixed, but what fraction of production artefacts were examined at all.

Under Respond to Vulnerabilities, the failure is the intake path. There is a security contact, but it is a shared mailbox nobody owns; there is a vulnerability disclosure policy, but it promises a response time the team cannot meet; and internally generated findings and externally reported ones live in different systems with different severity scales, so the same defect can be both "high, due next week" and "won't fix". The CRA reporting clock, covered later in this guide, does not care about that distinction: it starts when the manufacturer becomes aware, and awareness does not arrive in a convenient format.

Threat modelling that survives contact with a sprint

Threat modelling fails in most organisations for a reason that has nothing to do with skill: it is performed once, at design time, by people who will not be in the room when the design changes. The model describes the system as intended, the system drifts, and the document becomes a historical artefact that satisfies an audit question about whether threat modelling was performed.

The version that survives is smaller and more frequent. Instead of a full data-flow diagram per product per year, the model attaches to the change that alters trust boundaries - a new external integration, a new authentication path, a new data store holding regulated data, a new background job that acts with elevated rights. The questions are four, and the engineer writing the change can answer them in twenty minutes: what does this change accept from outside the boundary, what does it decide without re-checking identity, what does it write that someone else will later trust, and what happens when each of those is wrong? The output is a short list of abuse cases, and each abuse case becomes either a test or a documented accepted risk.

NIST SP 800-218 practice PW.1.1 names the techniques - threat modelling, attack modelling, attack surface mapping - and treats them as equivalent routes to the same assessment of risk. The practical difference is that attack surface mapping scales better for small teams: enumerate the entry points, name the data that crosses each one, and rank by what an attacker gains rather than by what is easiest to draw. For a microservice estate the entry points are the API routes, the message consumers, the scheduled jobs with external input and the webhooks, and most teams discover during the enumeration that they have more of each than they thought.

Two checks separate a real model from a document. Does the model name the trust boundary in terms a developer recognises in the code - a middleware, a decorator, a gateway rule - rather than in an architecture diagram? And does each mitigation have an owner and a test? A threat model whose mitigations read "implement authorization" has produced a wish; one whose mitigations read "the invoice endpoint rejects a request where the invoice identifier belongs to another tenant, asserted in invoice-authz.spec.ts" has produced work.

Security requirements: writing them so they can be failed

A requirement that cannot be failed is not a requirement. "The application must be secure" and "developers must follow OWASP" are unenforceable, and their presence in a document is evidence of a template rather than a process. The ones that work have three properties: a named artefact that shows compliance, a named check that shows non-compliance, and a scope that says which systems it applies to.

OWASP ASVS 5.0.0, dated May 2025, is the most useful source of pre-written requirements because it is structured for exactly this. It contains roughly 350 requirements across 17 chapters and three levels, and its citation format - chapter, section, requirement, such as 1.2.5 on OS command injection - lets a requirement be quoted in a ticket and tested in a pipeline. The 5.0 revision removed the former V1 Architecture chapter, added chapters for OAuth and OIDC, self-contained tokens and WebRTC, and redefined Level 1 as a first-layer defence baseline rather than the minimum, which matters when a team argues about which level applies.

The mistake is scoping the whole standard to every system. A practical rule: Level 1 for anything handling internal data, Level 2 for anything handling personal, financial or health data or reachable from the internet, and Level 3 only where a compromise is a company-level event. Then measure coverage per system rather than as an organisational average, because an average hides the one service at Level 0 that happens to hold the customer database.

Where the requirement is architectural rather than a control, it needs a different form. "Passwords must be hashed with Argon2id or bcrypt at a work factor of 12 or higher" is checkable in review and in a test. "Tenant isolation must be enforced at the data layer rather than in application code" is checkable only by reading five representative queries and by attempting a cross-tenant read with a test account. Both are legitimate requirements; the second is the class assessments most often find unmet while every control-level requirement passes.

The build system is production: CI/CD as attack surface

A build pipeline is a production system with an unusual risk profile: it is the only system in the estate whose normal operation produces signed artefacts that customers install, and it is usually run by a team that does not think of itself as operating production infrastructure. Everything that makes it convenient makes it dangerous - it accepts input from contributors, executes code from the repository it is building, holds credentials for registries and cloud accounts, and caches state between runs.

The concrete tests are unglamorous and they find real problems. Can a pull request from a fork reach a job that has repository secrets in scope? Are third-party actions and images pinned by commit digest or by a mutable tag? Does the build run in an environment with internet access and access to the production cloud account at the same time? Is the publish step a separate job with its own identity, or does every test job carry the publishing token? Can a lower-privileged job poison a cache that a higher-privileged job later restores? Each of these has a documented answer in the platform's own guidance and a different answer in most real pipelines.

Two design changes remove most of the exposure. The first is separating build from publish: the build job produces an artefact and its provenance, the publish job verifies the provenance and holds the only credential that can write to the registry. The second is making the runner disposable and network-restricted, with an allowlist for the registries it needs and no route to production. Neither is expensive, and both are far cheaper than the incident where an attacker with a merged pull request publishes a backdoored release under the organisation's own signing key.

The human side matters as much. A pipeline that blocks a merge on a failed security check is a control; one that posts a warning is a notification. Teams optimise for the fastest path to merge, so any check that can be bypassed with a comment will be - and the standard bypass, a suppression with no expiry and no owner, is the single most common way a mature-looking programme ships the same defect for years.

Provenance and signing: what an attestation actually proves

Provenance answers one question: who built this artefact, from what inputs, using what process. It does not answer whether the code is free of vulnerabilities, whether the inputs were trustworthy, or whether the maintainer was honest - and assessments that treat a signed attestation as a security guarantee make a claim the technology does not support. What it does answer is the question that decides the blast radius of a supply-chain compromise: did the thing the customer installed come from the pipeline we think it came from?

SLSA is the specification most organisations use for this. Version 1.1 was approved on 21 April 2025 and replaced 1.0; the current published version is 1.2. Its levels sit on tracks, and the build track is the one that matters here: Build L1 requires provenance showing how the package was built, L2 requires signed provenance generated by a hosted build platform, and L3 requires a hardened build platform that resists tampering during the build itself. The progression is not about cryptographic strength, it is about which attacker each level excludes - the honest mistake, the post-build tamperer, and the attacker inside the build.

The practical assessment question is narrower than "what SLSA level are we". Take a release the organisation shipped last month and ask: can we produce its provenance, does the provenance name the source revision and the dependency inputs by digest, is it signed by an identity a customer can verify, and does the deployment path verify the signature before running the artefact? A programme that answers yes to all four has something an attacker cannot easily imitate. A programme that publishes a bill of materials but does not verify signatures has a document, not a control.

Signing also fails in a specific and common way: the key is available to the pipeline that builds untrusted code. A signing step that runs in the same job as the build, using a long-lived key stored as a secret, means anyone who can influence the build can sign the result. The mitigation is the same as in the previous section - separate the privileged step, bind the signing identity to the workload rather than to a static key, and make verification part of installation rather than part of documentation.

Dependencies: a bill of materials nobody reads is a filing exercise

Every assessment finds the same thing about software composition analysis: the scanner is installed, the bill of materials is generated, and nobody acts on either until a headline vulnerability makes the question urgent. The control that matters is not generation, it is the ability to answer three questions in under an hour: which of our shipped artefacts contain component X at a vulnerable version, who is the internal owner of each, and what is the rebuild path for each?

Answering them requires the bill of materials to be attached to the artefact rather than to the repository. A list generated from a lock file at build time describes what the build intended to include; the list generated from the built image or package describes what is actually inside. Those differ more often than teams expect - transitive dependencies resolved differently across platforms, packages installed by base-image layers, vendored code copied into the tree, and binaries bundled by a framework. The assessment test is to pick a component that appears in the image but not in the lock file, or the reverse, and see whether the process notices.

The wider problem, including the third-party paths that reach production without crossing your repository, is set out in supply chain attack prevention. The second gap here is the maintenance question, which is a decision rather than a scan. A dependency with no release in two years is not automatically a risk, but it is automatically a candidate for the question: who would fix this if a vulnerability were published tomorrow, and how long would it take? For the components that carry the most exposure - the ones parsed by untrusted input, the ones running with elevated privilege, the ones shipped to customers - the answer should be known, written down, and revisited. This is the same judgement the CISA Secure by Design pledge pushes manufacturers toward when it asks for a measurable reduction in the prevalence of a whole vulnerability class rather than a lower finding count.

Two habits make the difference between a programme and a compliance artefact. Triage by exploitability rather than severity alone - a high-severity finding in a code path that is not reachable from untrusted input is not the same priority as a medium finding on the authentication path - and record the reasoning, because "not exploitable" with no analysis becomes "ignored" the first time a manager reads the backlog. Then set the rebuild cadence from the threat: a component with public proof-of-concept exploits needs a rebuild path measured in hours, not in the next quarterly release.

Secrets: the cheapest finding and the most common one

Hardcoded credentials remain the most reliable finding in any code review, and the reason is structural rather than careless: secrets arrive through convenience. A developer needs a database connection at 11pm, the configuration system is awkward, the value goes into a file that is later committed, and the commit history keeps it forever. Rotating the key does not clean the history, and cleaning the history does not rotate the key, so the two steps are usually done separately and one of them is skipped.

The assessment tests are cheap and decisive. Does a secret scanner run on every commit and on the full history, or only on new changes? When a finding appears, is there a documented rotation path that a developer can complete in under an hour without a ticket to another team? Do build logs, error trackers and analytics events scrub values that look like tokens? Does the deployment configuration inject secrets at runtime, or bake them into the image layer where anyone with pull access can read them?

Detection without rotation is theatre, and this is where most programmes stop. A secret found in a 2019 commit is still live if the key was never changed, and the finding is closed as "historical" while the credential continues to authenticate. The workable rule is that a secret-touching finding is not resolved until the credential is rotated and the rotation is recorded - and that rule needs an owner with the authority to rotate, which is usually not the security team.

The structural fix is to make the safe path shorter than the unsafe one. Runtime injection from a managed secret store, short-lived credentials for service-to-service authentication, and a local development configuration that pulls from the same store with a personal identity removes the reason to hardcode anything, and the vault side of the same problem - what happens when a credential does have to live in a human's browser - is covered in password manager security. Where that is not possible - legacy systems, vendor appliances, embedded devices - the compensating control is a documented inventory of long-lived secrets with owners and rotation dates, which is a worse answer but an honest one.

Static analysis: measure the programme, not the product

A scanner's marketing page and its behaviour in a real repository are different documents. What decides whether static analysis reduces risk is not detection breadth but three operational numbers: the false-positive rate on this codebase's languages and frameworks, the latency of a scan on a typical pull request, and the fraction of reported findings that receive a disposition before their next release. A tool that is accurate but slow gets bypassed; a tool that is fast but noisy gets ignored; a tool that is both accurate and fast but has no triage owner produces a dashboard nobody reads.

Measuring it takes a week and is worth more than any vendor comparison. Take one mature repository and one recently created one. Run the candidate on the same code, then have a senior engineer triage fifty findings and record how many were real. Time the scan on a real pull request, not on a nightly full scan. Check whether findings appear where developers work - the pull request, the editor - or only in a separate portal they have to remember to open. Then check the suppression mechanism: can a rule be silenced inline, who reviews that, and does the suppression expire?

That last point separates programmes that improve from programmes that accumulate debt. An inline suppression with no expiry is a permanent decision made under time pressure by the person least able to assess it. Rules carrying the largest false-positive burden should be tuned or disabled with the reason recorded, and the decision revisited when the detector or the framework changes, because a rule that was noisy against one framework version can be precise against the next.

Coverage is the other number that gets reported dishonestly, usually by accident. "We scan all our code" frequently means "we scan every repository that has the workflow file", which excludes the services migrated from a previous platform, the infrastructure templates, the mobile clients and the scripts that run in production. The population should be enumerated from the deployment inventory rather than from the CI configuration, and the gap between the two is itself a finding.

Dynamic testing, fuzzing and the coverage a scanner cannot reach

Dynamic analysis sees the application as a running system, which makes it the only automated technique that observes configuration, middleware, framework defaults and runtime data flow together. That breadth has a cost: it needs a deployed instance, credentials and seed data to reach authenticated functionality, and it will not find the endpoint a crawler never discovers. An assessment that reports a clean dynamic scan usually means the crawler reached the login page and the public marketing routes.

For APIs specifically, API security testing covers the request-shape and authorization tests that belong alongside the scan. The practical fix here is to drive the scan from the specification rather than from the crawl. A well-maintained API description, imported into the scanner, gives it every route and every parameter, including the ones no link points to. Then add authenticated sessions for each role, so the scan exercises the paths where authorization and business logic live. Where no specification exists, the honest statement is that authenticated coverage is unmeasured - and that statement is more useful than a green dashboard, because it names the work: write the specification, or derive the route inventory from gateway logs.

Fuzzing covers the class both other techniques miss: parsers and decoders that fail on input nobody imagined. The investment is real - a harness, a corpus, a way to run it continuously - but the return is concentrated where the organisation processes untrusted structured input: file parsers, protocol handlers, image and document decoders, serialization layers. The memory-safety goal in CISA's Secure by Design pledge points at the same risk from the language side; fuzzing is what you do about the code already written.

None of these techniques finds the flaw that matters most in business applications: a correct implementation of the wrong rule. A refund endpoint that validates the amount, authenticates the user and checks the role, but never checks that this user owns this order, passes every scanner and every fuzz corpus. That gap is why the next section exists.

Code review is the only control that finds authorization logic

Authorization defects are the most expensive class to miss and the least amenable to automation, because the intended access model is not in the code. A scanner can see that a query has no tenant filter; it cannot know whether the table is global by design. A person reviewing the change with the access model in hand can answer in a minute, and that is the entire justification for keeping human review as a control rather than as a formality.

Making review effective is a scoping problem. The features that deserve a security-focused review are the ones that change who can do what: a new role, a new endpoint that reads or writes another user's data, a permission check moved or removed, a token audience or scope change, a new administrative function, an export or bulk operation. In a typical quarter that is a small fraction of the diff, and it can be requested explicitly - a review label, a named reviewer, three questions rather than thirty.

The three questions that catch most of it: where is the identity established, and can it be influenced by the request rather than by the session? Where is the authorization decision made, and is it on the server side of every path that touches this data, including the background job and the admin tool? What does the query return when the filter is missing - an error, or everything? The third question is the one that finds defects, because the dangerous default is silent: a missing filter that returns all rows looks like a working feature in every environment where the test data has one tenant.

Review also has a supply side. AI-assisted code generation produces plausible code quickly, and plausible code is what a reviewer skims. The mitigation is not to ban the tooling but to review the diff with the same three questions and the same rigour, and to require that generated code touching authentication, authorization, cryptography or query construction gets a named human reviewer who is accountable for it. The CRA and the SSDF apply identically to code a model wrote, because the obligation attaches to the product, not to the author.

Secure defaults and the memory-safety roadmap

A secure default is worth more than a control, because it does not depend on a decision being made correctly at the right moment. The identity side of the same principle is covered in zero trust security. The highest-yield defaults in application development are well known and rarely all present: deny-by-default authorization with explicit grants, session cookies marked HttpOnly, Secure and SameSite, parameterised queries as the only available database interface, a templating layer that escapes by default rather than on request, and a cryptography library whose safe mode is the default constructor.

The assessment question is whether the default is structural or documentary. If the framework offers an unsafe and a safe way to do the same thing and the unsafe way is shorter, the codebase will contain the unsafe way. The fix is to make the safe path the shortest one - a shared internal library, a lint rule, a template that already does it - and then to measure adoption across the population, because a library introduced last quarter covers new code and nothing else.

Memory safety belongs in the same conversation for a different reason. CISA's guidance on memory safe roadmaps asks manufacturers to plan a transition for components where memory-safety defects are both likely and dangerous, and the pledge's vulnerability-class goal makes that measurable. For most organisations the honest roadmap is not a rewrite: it is a prioritisation of the parsers, protocol handlers and native extensions that process untrusted input, are written in a memory-unsafe language, and ship to customers. A roadmap that starts from the inventory of those components is credible; one that starts from a promise to migrate everything is a press release.

Vulnerability response, and the CRA clock that now starts on awareness

Response is the practice group where a weak programme is visible to customers, because customers see the patch and they see the advisory. The measurable question is the interval between a vulnerability becoming known and a fixed release being available, split by severity and by whether the vulnerable component is the organisation's own code or a dependency. Most teams cannot answer it from data, only from memory, which is itself the finding.

The regulatory clock changed the stakes in 2026. Under the European Cyber Resilience Act, reporting obligations for manufacturers of products with digital elements apply from 11 September 2026, while the main obligations apply from 11 December 2027 and the chapter on notifying conformity assessment bodies applied from 11 June 2026. The reporting duties are short: an early warning within 24 hours of becoming aware of an actively exploited vulnerability or a severe incident, a fuller notification within 72 hours, and a final report within 14 days of a corrective or mitigating measure becoming available. Notifications go once, through ENISA's CRA Single Reporting Platform, to the CSIRT of the member state of main establishment and simultaneously to ENISA.

Two details decide whether an organisation can comply. The clock starts at awareness, not at confirmation, so the intake path has to capture a credible report from a customer, a researcher, a forum post or a supplier and route it to a named person within hours. And the reporting obligation applies to products already on the market, which means the inventory is a legal prerequisite: a manufacturer who cannot say which shipped versions contain the affected component cannot write the early warning. The same inventory gap that makes an SBOM useless for engineering makes the reporting duty impossible in practice.

The operational design that survives contact is a standing decision tree written before it is needed: who receives the report, who decides whether it is in scope, who drafts the early warning, who can authorise a release out of band, and what the customer-facing advisory says. Practices that carry no legal weight but a great deal of practical weight sit alongside it - a published vulnerability disclosure policy with a response commitment the team can meet, a security contact that a researcher can find in ten seconds, and a statement that good-faith research will not be pursued legally. Those are the CISA pledge's disclosure goal in plain terms, and they are what turns a researcher's private report into a fix instead of a public disclosure.

Metrics that predict the next defect rather than describe the last

Security dashboards usually measure activity: scans run, findings closed, training completed, tickets opened. Those numbers rise when the programme is healthy and also when it is failing loudly, which makes them useless for deciding what to do next. The metrics that inform a decision share a property: they measure a population against a standard, or they measure time, and they cannot be improved by relabelling.

Six are worth reporting, and each has an owner. Coverage: the fraction of production artefacts that were scanned by each control in the last thirty days, with the denominator taken from the deployment inventory. Escaped defects: vulnerabilities found in production or by a customer that a control should have caught, counted per quarter and analysed for why the control missed them. Time to disposition: median and 90th percentile from finding creation to a recorded decision, split by severity. Time to remediation: from decision to deployed fix, which is a different number and usually the embarrassing one. Suppression debt: the count of active suppressions, the number without an owner, and the age distribution. Recurrence: the share of findings that repeat a weakness class already fixed elsewhere, which measures whether the organisation learns or just patches.

One more belongs to the products rather than the process: the share of customer-facing releases whose provenance and bill of materials can be produced on request. That number is what a procurement questionnaire, an incident responder and a regulator all eventually ask for, and it is the only metric on the list that a customer can verify independently.

Report them as trends with the denominator attached, and resist the temptation to turn them into targets. A team told to reduce open findings will close findings, not fix vulnerabilities; a team told to raise scan coverage will add the workflow file to repositories it does not build. The targets that hold up are about time and about the population, because those cannot be satisfied without doing the work.

One assessment, many frameworks: mapping without duplicating work

By the time an organisation has answered to SSDF, ASVS, ISO 27001, NIS2, DORA and now the CRA, the same evidence has been produced four times in four formats. The efficient approach is to collect evidence once, at the artefact level, and map outward from it: a signed build with provenance answers a CRA integrity question, an SSDF PS.2 practice, an ISO control on change management and a customer's secure development questionnaire in the same document.

The mapping is mechanical if the artefacts are real, and impossible if they are not. Evidence that maps well: the bill of materials per release, provenance with verified signatures, the list of suppressions and their owners, the remediation time series, the threat models attached to the changes that altered trust boundaries, the review records for authorization-touching changes, and the disclosure policy with its measured response times. Each of these is produced by doing the work and is reusable; none can be produced by writing a policy document, which is why programmes that start from the framework rather than the artefact end up with four contradictory narratives.

The frameworks do differ in obligation, and an assessment should say where. SSDF describes practices and does not certify anything. ASVS gives testable requirements and does not audit. ISO 27001 certifies a management system, not a product. NIS2 and DORA impose duties on entities in specific sectors, with DORA adding a register-of-information requirement and incident reporting for financial entities. The CRA imposes product duties on manufacturers and importer obligations along the chain, with reporting from 11 September 2026 and conformity from December 2027. Treating any of these as a substitute for another is the most common cause of a compliance file that cannot survive a second reader.

AI-assisted development changes the inputs, not the standard

Generated code shifts the economics of secure development in two directions at once. It increases volume, which increases the population a control must cover; and it produces code that looks idiomatic, which reduces the vigilance a reviewer applies. Neither effect changes the requirement, but both change where the assessment should look.

The questions that produce a useful answer are about governance of the toolchain rather than about the model: is the assistant permitted to see the repository, and under what terms; are prompts and completions retained, and where; is generated code distinguishable in review, so that a reviewer knows which parts were written by a person with context and which were assembled from patterns? Then the coverage questions: do the scanners run on the generated code with the same rules, and does the review checklist still apply to the harness, the prompt template and the glue code? That last category is where generated infrastructure code carries risk that nobody owns, because it is not a product feature and no team feels responsible for it.

A specific pattern deserves naming. A generated endpoint will typically include authentication and may include authorization, because examples show the former and omit the latter. It will use the framework's default for escaping, which is correct until the change is to bypass it. It will create a query with parameters, which is correct until the generated code composes a filter as a string. Each of these is the same defect the previous sections describe, produced faster - which is an argument for the review control, not an argument against the tool.

Scope, effort and what an assessment realistically costs

An assessment is a defined engagement with a stated scope, not an open-ended audit. Scope it by decision: what is the organisation trying to decide? Whether a product can be sold to an enterprise customer, whether it meets a regulatory obligation starting on a date, whether an acquisition target's development practice is as advertised, or where to spend the next quarter of engineering effort. Each decision needs a different depth, and naming it prevents the assessment that produces a hundred pages nobody can act on.

For a single product with a small engineering team, the work is typically two to three weeks: a week of reading artefacts and interviewing, a week of tests against the population, and a few days of writing and reviewing findings with the team that has to fix them. For a multi-product manufacturer, the shape changes: sample products by risk, assess the shared platform once because the pipeline is shared, and measure the population across all products so the report can say what fraction was examined. The number that should never be quoted is a fixed count of days per developer, because the effort tracks the number of distinct build pipelines and distinct frameworks far more closely than it tracks headcount.

Two costs are routinely left out and both are real. The first is the remediation the assessment will surface - typically the build pipeline separation and the suppression cleanup, both of which take engineering time that was not in anyone's plan. The second is the internal cost of sustaining the evidence: someone has to own the bill of materials, the provenance verification and the remediation clock between assessments, or the next assessment re-discovers the same gaps. An engagement that does not hand over that ownership has produced a report rather than a capability.

The lawful boundary: authorisation is not a formality

Everything in this guide assumes the assessor has written permission from the system owner. That is not a disclaimer bolted onto the end; it is the line between professional work and a criminal offence in most jurisdictions, and it is the first thing a competent assessor establishes and the first thing a careless one skips. Testing a supplier's software, reading a competitor's source, probing an endpoint that belongs to a third party, or running a dependency attack against a system that merely integrates with your own is outside the scope of an assessment regardless of intent.

Practical authorisation has four elements: a named person with authority to grant it, a written scope naming systems and time windows, a statement of what the assessor may do - including whether exploitation, persistence or data access is permitted - and an out-of-band contact for when something breaks. For internal work the same discipline applies in a different form: a change record, an environment owned by the organisation, and data that is either synthetic or approved for use. Production customer data is not test data, and an assessment that reads it has changed the privacy footprint of the organisation whether or not anything is disclosed.

For organisations that want external verification without hiring an attacker, the disclosure policy is the mechanism: a published channel, a commitment, and a statement that good-faith research following the policy will not be pursued legally. That combination is what the CISA pledge's disclosure goal asks for, and it is what makes a report from a stranger arrive at the right desk within hours rather than on a public forum six months later. For defensive work in the home or a small business, the same boundary applies inversely: the tools that inspect traffic, review code or test a device belong on systems the person owns or administers, and anything else needs the owner's explicit agreement.

A first thirty days, cheapest first

Most of the value in this guide can be captured in a month without new tooling, because the cheapest findings are the ones nobody has looked for. The sequence below is ordered by cost rather than by elegance, and each step produces an artefact the later steps need.

Days 1 to 5: enumerate. Build the list of production artefacts from the deployment inventory, not from CI: services, mobile clients, infrastructure templates, scheduled jobs, customer-facing binaries. Then list the build pipelines that produce them. The gap between those two lists is the first finding, and in most organisations it is non-trivial.

Days 6 to 12: measure coverage. For each artefact, which controls ran in the last thirty days - static analysis, dependency scanning, secret scanning, dynamic testing - and what fraction of the population each reached. Record the result as a table with denominators. Do not change anything yet; the point is a baseline that can be compared later.

Days 13 to 18: read the suppressions. List every active suppression and inline ignore, with its age and its owner. Expire the ones with no owner, and re-open the ones older than a year. This is the fastest visible improvement available, and it usually produces a handful of findings that were real all along.

Days 19 to 24: test one release end to end. Pick the most recently shipped customer-facing release. Produce its bill of materials and its provenance. Then verify the signature at install time, not in a document. Whatever breaks in that exercise is the honest state of the supply chain, and fixing it is the highest-value work in the month.

Days 25 to 30: write the decision tree. Who receives a vulnerability report, who decides whether the CRA clock is running, who can authorise an out-of-band release, and who writes the customer advisory. Then review the three authorization-touching changes from the last quarter with the three questions from this guide, and turn each answer into a test. At the end of thirty days the organisation has a measured baseline, a supply chain it can describe, and a response path that exists before it is needed - which is more than most programmes achieve in a year of buying tools.

Where to go from here

The pattern across every section is the same: the control that matters is the one that is enforced, measured and owned, and the difference between a programme and a document is whether an artefact can be produced on request. Frameworks give the vocabulary, regulation sets the clock, and tooling supplies the findings, but none of them decides whether a defect reaches a customer. That decision is made by the pipeline, the reviewer and the response path - and those are the three things an assessment should test first.

Organisations that want external help with the reading and the testing should look for a scope that names the systems, the artefacts that will be produced, and the authorisation in writing before the work starts. Organisations that want to do the first pass themselves can follow the thirty-day sequence above and should expect the enumeration step to take longer than planned, because that is where the real gap usually is.