Testing, Fuzzing, and Security Verification
No single tool proves an application secure.
13.1 Security testing pyramid
Unit tests: small validation and policy functions.
Integration tests: identity, database, storage, and external-service boundaries.
End-to-end tests: full user flows and negative authorization paths.
Static analysis (SAST): source or bytecode patterns.
Dynamic testing (DAST): behavior of a running application.
Software composition analysis (SCA): dependency inventory and known vulnerabilities.
Secret scanning: accidental credentials.
Fuzz testing: unexpected or malformed inputs.
Penetration testing: authorized, scoped assessment by skilled testers.
Configuration and infrastructure testing: deployment and policy checks.
No single tool proves an application secure.
13.2 Positive and negative tests
For each feature, test expected success and expected denial:
Valid input / malformed input Authorized user / unauthorized user Owner / non-owner Same tenant / different tenant
Normal size / boundary size / excessive size First request / duplicate request
Valid state transition / invalid state transition Service available / timeout / partial failure
13.3 Property-based testing
Test general properties across many generated inputs.
Examples of properties:
A parser never accepts data exceeding configured limits.
An unauthenticated caller never receives a private record.
A user from tenant A never receives tenant B data.
A single-use reset token cannot be accepted twice.
A rejected operation does not partially mutate state.
A property checked against generated inputs Output (CPython 3.13.1):
Libraries such as Hypothesis (Python), fast-check (JavaScript) and jqwik (Java) generate inputs more cleverly and *shrink* a failing case to a minimal example; this plain loop shows the idea with only the standard library.
import random
MAX_LEN = 64
ALLOWED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-")
def is_valid_course_id(value: str) -> bool:
return 1 <= len(value) <= MAX_LEN and set(value) <= ALLOWED
rng = random.Random(2024) # fixed seed: failures are reproducible
alphabet = "".join(sorted(ALLOWED)) + " ./\\'\"<>;\x00é"
violations = 0
for _ in range(20_000):
candidate = "".join(rng.choice(alphabet) for _ in range(rng.randint(0, 80)))
accepted = is_valid_course_id(candidate)
# Property: anything accepted is within the length limit and uses only allowed characters.
if accepted and (len(candidate) > MAX_LEN or not set(candidate) <= ALLOWED):
violations += 1
accepted_count = sum(is_valid_course_id("".join(rng.choice(alphabet) for _ in range(rng.randint(0, 80)))) for _ in range(1_000))
print("generated cases: 20000, property violations:", violations)
print("of 1000 further random strings,", accepted_count, "were accepted")generated cases: 20000, property violations: 0
of 1000 further random strings, 72 were accepted13.4 Fuzzing
Fuzz parsers and input-handling code with generated malformed data. Use safe test environments, limits, reproducible seeds, and crash triage. Fuzzing is especially useful for file formats, binary protocols, compilers, parsers, and native code.
13.5 Static analysis and sanitizers
Use language-specific tools, compiler warnings, memory/undefined-behavior sanitizers, type checking, linting, and code review. Tune rules with documented justification; do not silence a finding simply to obtain a green build.
13.6 Regression tests
Every confirmed security defect should produce a test where feasible. The test should fail before the fix and pass afterward, and it should target the root cause rather than only the exact observed payload.
A regression test aimed at the root cause Output (CPython 3.13.1):
The tests describe the *rule* (owner and tenant must match) rather than the one request that exposed the bug, so a different route to the same weakness also fails.
import unittest
def can_download(user, report):
# Fix for finding SEC-42: previously only checked that user was logged in.
return user is not None and user["tenant"] == report["tenant"] and user["id"] == report["owner"]
class DownloadAuthorization(unittest.TestCase):
report = {"owner": "u1", "tenant": "t1"}
def test_owner_can_download(self):
self.assertTrue(can_download({"id": "u1", "tenant": "t1"}, self.report))
def test_other_user_same_tenant_is_denied(self): # the original finding
self.assertFalse(can_download({"id": "u2", "tenant": "t1"}, self.report))
def test_same_id_other_tenant_is_denied(self): # root cause, not just the payload
self.assertFalse(can_download({"id": "u1", "tenant": "t2"}, self.report))
def test_anonymous_is_denied(self):
self.assertFalse(can_download(None, self.report))
unittest.main(argv=["regression"], verbosity=2, exit=False)test_anonymous_is_denied (__main__.DownloadAuthorization.test_anonymous_is_denied) ... ok
test_other_user_same_tenant_is_denied (__main__.DownloadAuthorization.test_other_user_same_tenant_is_denied) ... ok
test_owner_can_download (__main__.DownloadAuthorization.test_owner_can_download) ... ok
test_same_id_other_tenant_is_denied (__main__.DownloadAuthorization.test_same_id_other_tenant_is_denied) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK13.7 Authorized penetration testing
Define written scope, test accounts, permitted techniques, time window, data-handling rules, escalation contacts, and stop conditions. Do not test systems without authorization. Findings should include impact, reproducibility, affected components, remediation, and verification.
A CI security gate Text equivalent: each pull request is linted and type-checked, tested, scanned for code patterns, secrets and vulnerable dependencies, built with an SBOM, deployed to a non-production environment for a controlled dynamic scan, and only then approved for release. A failing stage stops the pipeline.
flowchart TD
PR[Pull request] --> L[Lint and type check]
L --> T[Unit and integration tests]
T --> S[SAST and secret scan]
S --> D[Dependency / SCA scan]
D --> B[Build and SBOM]
B --> ST[Deploy to non-production]
ST --> DA[Controlled DAST scan]
DA --> REL[Release approval]Exercise
Create a CI security gate for a small API: unit tests, dependency scan, secret scan, static analysis, build, integration tests, and a controlled dynamic scan in a non-production environment.
Quiz
? A confirmed bug let users download other users' reports. What makes a good regression test?
+ Tests that fail before the fix and pass after, covering the rule (owner and tenant) rather than one URL :: Root-cause tests catch other routes to the same weakness.
- One test that replays the exact reported request :: Too narrow — a variant may slip through.
- Disabling the endpoint in tests :: That tests nothing.
- A comment in the code explaining the fix :: Comments are not verification.
? Which testing activity requires written authorization and scope?
- Unit tests on your own code :: Normal development.
+ Penetration testing of a deployed system :: Scope, accounts, windows and stop conditions must be agreed in writing.
- Running a linter :: Normal development.
- Reading the OWASP Cheat Sheets :: No authorization needed.
? Why use a fixed seed in generated tests?
+ A failure can be reproduced exactly. :: Reproducible seeds make triage possible.
- It makes the inputs more random. :: It makes them repeatable.
- It is required by the language. :: It is a testing practice.
- It hides failures. :: It does the opposite.