websec-validator
Technical Brief · v0.21.0 · 1 Oct 2026

A scanner gives you findings. A pentester gives you a plan.

websec-validator reads a repository the way a senior pentester reads it on day one, then hands your AI coding agent a plan: the supported attack surface, which leads need review, and how to test them. No model inside; the core pass needs no server or running app.

The problem

AI agents now write the code. Nobody briefs them on what to attack. Static scanners answer “which patterns match?” and bury the answer under volume. A pentest answers “what matters, and how would I prove it?” and happens once a year, if at all.

A real count, from this project's own changelog (0.17.0, bug-311)
findings-ledger.json   total: 163            # what the scanners said
cluster_summary        distinct issues: 13   # what a reviewer actually has to fix

Same fixture, same run. The 163 is not wrong, and every one is kept for the machine consumers. But a human, or an agent, has to be told they are 13 problems, where they live, and what would confirm each one. That handoff is the product.

A scanner asks “Which patterns match?” 163 findings — all retained, not yet validated websec asks “What would a pentester test first, and how?” 13 distinct issues — each with a location and a probe

Fig. 1 — The same ledger, under two different questions. The second one is what a pentest report contains.

The method

code in

Read supported source. Map visible routes, auth decisions, tenant boundaries, sinks, secrets and dependency. Run the installed static scanners, de-duplicate them, and rank what is left by whether it is reachable and exploited in the wild.

artifacts out

A fact file, a findings ledger, a SARIF report, a probe library tailored to what was found, and a briefing for the agent that will do the fixing. Then an exit code that says whether the picture is complete.

The rule that surprises people: there is no model in the tool. Analysis is reproducible with the same inputs and tooling; run metadata and dated advice can vary. The model sits on the other side of the briefing, with a human beside it.

A tool that never says “incomplete” is a tool you cannot trust when it says “clean.”

The design constraint everything else follows from
websec-validator
02 — What it refuses to say

Every ambiguity rounds toward “incomplete”

Most security tools have two outputs: findings, or a green check. This one has a third, and it is what makes the other two believable.

What this tool said about a deliberately vulnerable Elixir app, until September 2026
# raw SQL interpolation · System.cmd injection · a hardcoded credential · no auth plug
unsupported: []   gaps: []
REQUESTED CHECKS COMPLETED — 0 files read

A suffix list had been absorbed into another, leaving the “language I cannot read” branch reachable for one language. The fix does not make the tool read Elixir. It makes it say it did not: the banner leads with NO ANALYZABLE SOURCE, and --require-analyzed turns that into exit 3.

Requested analysis incomplete, with no threshold finding? yes exit 3 · incomplete a missing scanner, or nothing analyzable, is not a clean run Invalid flags or incompatible selected scanner? yes exit 2 · usage wrong flags, or a scanner below its minimum version A finding crosses the selected threshold? yes exit 1 · findings gaps retained if findings + incomplete Everything ran, nothing crossed the threshold. exit 0 · clean “nothing blocking was found” — never “verified safe”

Fig. 2 — With --require-complete and --fail-on. Usage is checked first; findings + incomplete exits 1, with both recorded. Ungated exit 0 is not a completeness verdict.

Nine things the tool will not do, and where that is enforced

RefusalWhy, and where
say “compliant”The word appears in no output, and a test asserts that, so it cannot come back by accident. tests/test_attest.py
claim complete protectionprotection_complete is always false: static analysis cannot see a running system. docs-canonical/SECURITY.md
fill an evidence gapThe audit projection has 12 control rows; 5 carry an empty evidence list, with the reason. attest.py
count “unknown” as “false”The old rule scored every unmatched finding as a false positive. Relabelling against pinned source left 21 reviewed labels and 35 findings still unknown, excluded from both sides. calibration.json
optimize its own scoreAnything fitted from labels uses a strictly proper rule, Brier or log score, never accuracy or F1, which reward confident wrongness. It is printed, never optimized. calibration.SCORING_RULE
run a scanner unaskedTruffleHog verifies secrets against live services, so it runs only when named. scanners.py
read a cell too thin to trustA per-class probability needs at least five samples. One class reaches that today; the rest fall back a tier, and the output names which. calibration.MIN_N
guess reachabilityA vulnerable dependency is tagged imported, no-import-found or n/a. It never rounds n/a to safe. enrichment.py
claim a benchmark it has not runAn offline captured-report harness preserves reviewed TP/FP/unknown labels and unscored partial captures. No head-to-head or agent-benefit experiment has run. BENCHMARKS.md

The one place it fails open, on purpose

The in-editor hook blocks on findings or completed-but-incomplete analysis, including selected source it could not read. Only unexpected internal crashes fail open, loudly: the edit was not checked. Client-side settings remain editable, so production enforcement belongs in required CI checks, not this convenience hook.

websec-validator
03 — Anatomy of a run

One repository, start to briefing

In → your repository, read as text Out → websec-out/ Recon 22 extractors → FACTS.json Scan installed scanners, de-duplicated Enrich reachable × exploitable Stage probes from 24 templates Brief AGENT-BRIEFING.md Exit 0 · 1 · 2 · 3 Zero runtime dependencies The tool shells out to Trivy, Gitleaks, Semgrep, Checkov and the per-stack SAST tools it finds. Required missing tools make the run incomplete. Enrichment is local too EPSS scores and the CISA KEV list come from an on-disk cache. The default pass makes zero network calls. Core pass: source and local artifacts. External scanners and online metadata require separate opt-in. Below the line: your agent and you, running the staged probes against a TEST instance you own, with credentials you supply. Dynamic data probes default to GET; configured TEST login may POST. Mutating probes stay localhost-only at every transport.

Fig. 3 — Staging sends no requests. Review and authorization precede execution; optional external scanners are not a sandbox.

How a finding earns its severity

Severity is a rule, not a score. The same evidence yields the same label every run, and the rule is written in the methodology before it is written in the code.

Confirmed dynamically, a verified live secret, or a fixed-version HIGH/CRITICAL CVE? yes HIGH something outside the code agrees Concrete static evidence: a sink fed by request input, a route with no auth decision? yes MEDIUM gate threshold is operator-selected Otherwise: a single-source hypothesis. LOW kept visible, ranked last, never dropped Then, within the label Imported package? KEV-listed CVE and EPSS score? An invariant flips to VIOLATED only with dynamic confirmation, verified evidence and confirmed-vulnerable status. Static hypotheses never flip it. P(real), measured LOW 0.545 [0.28, 0.787] · n=11 MEDIUM 0.7 [0.397, 0.892] · n=10

Fig. 4 — Default evidence tiers; individual scanner/detector rules retain their documented classifications. Severity is not vulnerability confirmation.

What lands in websec-out/

ArtifactWhat it is
AGENT-BRIEFING.mdThe product. Detected surface, the access-control map, targeting, findings, the method, and the staged probe list, written for the agent that will act on it.
FACTS.jsonThe full structured recon: every fact the 22 extractors produced, schema-versioned, with a coverage block that says what was skipped and why.
findings-ledger.jsonThe traceable ledger: evidence chain, CWE / ASVS / OWASP-API citation, remediation and bucket-based confidence with provenance, interval and sample size. Priors are not measurements. REPORT.md is the same ledger for humans.
results.sarifSARIF 2.1.0, always written. GitHub code scanning, GitLab, Azure DevOps and the VS Code SARIF viewer read it as-is.
probes/Scripts staged from the 24 templates, pre-filled with the endpoints the recon found, for a human to review and run against a test instance.
websec-validator
04 — What it reads

Twenty-two extractors, seventeen sink classes, one fact file

Each extractor answers one question a pentester asks on day one, and writes the answer into FACTS.json as data, never prose.

Identity & access · 5 auth authz authz_dataflow tenant password_policy Who may call what, and does the code agree with itself about it. Surface & data · 7 routes surface schemas graphql upload_security pii_exposure transport_security Every entry point, every sink, every field that carries PII. Client · 3 client_exposure client_integrity webext What the browser can see, tamper with, or be sold. Platform & supply chain · 5 stack iac_ci integrations dependencies crypto_usage Infrastructure as code, CI, webhooks, lockfile drift, install scripts, hashing. AI & agents · 2 llm_security agent_config Prompt construction, tool dispatch, and whether the repo's own agent wiring has been poisoned.

Fig. 5 — The 22 extractors. Groups are editorial; the registry is flat. Each is isolated: one that crashes records an error in coverage and the run continues.

The seventeen sinks: request input reaching something dangerous, traced per route

ssrfcommand-injectionsql-injectionnosql-injection path-traversalsstiopen-redirectinsecure-deserialization xxeprototype-pollutionmass-assignmentredos eval-injectionssrf-outbound-httperror-disclosurexsslog-injection

Why these twenty-two

Not designed at a whiteboard. Three rows exist because dogfooding on a large LLM-agent monorepo, verified by a 15-agent adversarial pass, exposed two classes the route-and-auth model could not see. Four more grew because a real authenticated pen test found its two Criticals and two Highs in files the recon never parsed: AWS-CDK auth config, AppSync GraphQL schema, VTL resolvers. The walker reads those now.

Mapped to what a reviewer will ask about

Every finding carries a CWE, an ASVS 4.0.3 reference and, where they apply, the OWASP API and OWASP LLM Top 10 entries. The methodology publishes the map as a 41-row table; the findings module cites 77 distinct CWEs. Two limits are stated up front: regex over CDK TypeScript is not an AST and can be evaded by aliasing, and man-in-the-browser can never be a confident static catch, so it ships as LOW.

The scanners it shells out to, and the version each one must be

Presence on PATH is not compatibility, so each entry carries the lowest version whose CLI matches the invocation the tool builds, and doctor reports it.

ScannerCategory · minimum
Trivyvulnerabilities, secrets, misconfig · 0.38
Gitleakssecrets in history · 8.0; plus a working-tree pass
Semgrep/OpenGrepSAST, with two shipped rules · 1.0
Checkovinfrastructure as code · 2.0
OSV-Scannerdependency advisories · 2.0
Prowlercloud posture · detect only
Per stackRuns only when the language is present
BanditPython · 1.7
gosecGo · 2.0
BrakemanRuby · 4.0
TruffleHoglive secret verification · opt-in only, never implicit
websec-validator
05 — Evidence from the field

Six public repositories, 116 leads, zero verified critical issues

Dated source-only review: 14 September 2026. A tool that reported a critical in every public project it touched would be a tool that mislabels. The review ran source-only against pinned revisions, executed nothing, sent nothing upstream, and wrote down what it could not conclude.

Linkding27b7303 · 21 Rustpad54e4a93 · 5 Actualf8ad7c0 · 70 Flaskd73fa1c · 10 Angular RealWorlddd99ed2 · 7 Ignitee829d2f · 3 incomplete detector changed mid-scan; a Yarn bundle hit the source cap an emitted lead, not a confirmed bug a false-positive class, adjudicated and fixed

Fig. 6 — One dot per lead. Counts are detector output at a pinned revision; the review says so on its first line.

116
leads emitted across six repositories, none executed, nothing sent upstream
docs/security-review/public-repository-review.md
0
verified critical issues. Written as the headline, not buried in a footnote
same document, first paragraph
3 → 10
false-positive classes adjudicated with source links, each turned into paired regression tests
tests/test_public_precision.py

What the false positives taught

  • An archive error helper that formats ZIP messages was read as an authentication decoder because its strings said “unauthorized.”
  • A local file stream piped to stdout was read as HTTP file serving because createReadStream alone counted as serving.
  • A numeric GitHub expression in a workflow was read as script injection.

Each fix ships with its opposite: the test that makes the tool quieter is paired with one that proves the real case, the JWT decoded without verification, the stream piped to a response, still fires. The review also records its own gap: @angular/core was not recognized in the framework inventory.

The corpus, pinned

Dated v0.14.0 proof, not a v0.21.0 rerun or vulnerability recall. Three deliberately vulnerable apps at fixed commits, retrieved 2026-09-12: VAmPI f16052d, NodeGoat c5cb68a, DVGA a961308. The proof harness passes 10/10 surface checks with Noir present and 8/10 without it, with 7 truth labels left unknown. Both numbers are committed as JSON with the detector revision hash. DocGuard binds their validation-record statements to JSON fields; this brief uses separate fact tests and a PDF freshness manifest.

In September 2026 those 56 corpus findings were relabelled one at a time against the pinned source, each with a note and the revision it was read at. That produced 21 reviewed labels and a measured probability per severity, replacing a prior that had been guessed. The other 35 stay unknown and are counted on neither side.

And one real pentest: an authenticated engagement whose two Criticals and two Highs sat in the managed-cloud boundary the recon did not yet parse. Those findings became extractor rows, and two of the retest lessons corrected the tool's own earlier output.

websec-validator
06 — How the tool guards itself

The suite is guarded the way the tool guards you

Five checks gate the main branch. Two of them exist because of a specific, recorded failure.

tests · py3.11 · py3.12 · py3.13 Install editable, byte-compile the package, smoke the CLI entry point, run the suite. three required checks hermeticity · hostile git config A global hooksPath, gpgsign, excludesFile and a renamed default branch. The suite must still pass, and sentinel hooks outside the temp repos must checksum clean. suite-integrity Derived test floor · 60 s ceiling · required-check names cross-checked against the jobs this run emits · unit tests for the bot that merges without a human. Why the floor has no number in it A stored MIN_TESTS was bumped 1241 → 1243 by two branches at once. Both merged cleanly, because the values matched, and the floor ended two below the real suite: two tests deletable unnoticed. So the baseline is now counted from the base commit at CI time, with the unittest loader, in a separate process per tree. A module that fails to import is a hard failure, not a count of one. Why sixty seconds Runtime varies by host. A recorded 7.6× blowup stayed green; the sixty-second CI ceiling detects that failure shape without claiming a universal runtime.

Fig. 7 — The gate stack. The names in required-checks.json are the single source of truth; a required check that never runs would block every PR forever.

1,851
tests across 99 files, stdlib only, no public network, at this snapshot
tests/ · one opt-in lifecycle skipped in the default suite
0
runtime dependencies. The tool shells out to scanners and never imports them
pyproject.toml · dependencies = []
5
required checks, cross-verified against the jobs the run actually emits
.github/required-checks.json

Docs are checked like code

DocGuard runs with 27 of its 29 validators on, pinned explicitly rather than inherited. Four canonical documents are review-gated. Five sentences in the validation record are bound to JSON pointers inside the committed proof artifacts, so a number in prose cannot differ from the number in the file. Every conscious deviation from those documents is logged in a drift file with its reason.

Publication has separate safeguards. Selected facts are checked by test_explained_brief.py; a hash manifest binds the HTML, PDF and renderer script. CI rejects a stale pair. DocGuard checks marked Markdown sections, not PDF rendering. Prose accuracy and all eight rendered pages still need human review.

A release train that stops for a human

Every Monday a workflow proposes a version bump as a pull request and stops. A tag is cut only when a commit whose subject starts with release: lands on main, never from a version diff. Publishing to PyPI uses Trusted Publishing over OIDC, with no long-lived token, and refuses if the tag does not match the package version. The triage bot that labels and merges bot PRs never checks out the PR's code, and its classifier has its own unit tests in the required checks.

Every fix cites its bug, and every quieting test has a loud twin

A field report from 2026-09-18 listed nine problems. They became 50 tests in one file, and the fixes carry the bug number as a comment at the line that changed, in the CLI, the scanner adapters, the findings module, the coverage contract and the dynamic phase, so a reader can walk from the report to the code and back.

The test file's own docstring states the rule: every “this should be quieter” assertion is paired with a “this must stay loud” one. A precision fix that is not paired with the true-positive it must preserve is not accepted. That is the mechanical form of the no-regression bar this project holds itself to.

websec-validator
07 — Prior art

What is not new, and who solved which part

Static analysis, secret detection, dependency scanning and attack-surface mapping are all solved problems with excellent open-source tools. Anyone claiming otherwise is selling something. This tool uses them and adds the a repository-aware planning layer. The benchmark protocol exists; no unrun head-to-head result is claimed.

The scanners

Semgrep, Trivy, Gitleaks, Checkov, OSV-Scanner, Prowler and the per-stack SAST tools: Bandit, gosec, Brakeman. Eleven registry entries, each with a minimum version that doctor checks, because presence on PATH is not compatibility. The tool shells out, parses, de-duplicates, and cites each finding back to the scanner that produced it.

Attack-surface detection

OWASP Noir extends route discovery beyond the fallback patterns. In the dated v0.14.0 proof the harness scores 10/10 with Noir and 8/10 without it. Both numbers are published, because a user without Noir deserves to know what they are missing.

Dynamic scanners

ZAP, Nuclei and sqlmap test a live system. The briefing suggests relevant signatures, not guaranteed detections. BOLA, missing auth, mass assignment and row-level-security gaps often need identity-aware manual controls. A clean DAST run is not “safe,” and the briefing says why.

What is different: the question being asked

A scanner

Matches patterns, reports all of them

“Which lines look dangerous?”

→ 163 findings, one severity each

A list. Every entry may be true. Nothing says which one a pentester would open first, what request would prove it, or which findings are the same bug seen from six files.

websec-validator

Assembles a pentester's handoff

“What would I test first, against which endpoint, and what would confirm it?”

→ 13 issues, each with a location, a phase, an oracle, and a probe file

A briefing. Phase 1 safe recon, Phase 2 authorization with two identities, Phase 3 injection fired only at sink-backed endpoints and gated behind an explicit warning.

Illustrative briefing item — the shape, not a real finding §5b · Phase 2 · authz · two identities GET /api/orgs/{id}/invoices ← route with no per-object ownership check (authz_dataflow) Oracle: tenant B reads tenant A's protected object, verified against owner and denied-access controls. probes/bola-cross-tenant.sh · status alone is inconclusive · identity, ownership and content evidence required

Fig. 8 — A finding keeps its name, its endpoint and its oracle. That is the difference between a metric and a plan.

What it deliberately is not

websec-validator
08 — Where it plugs in

One engine, seven surfaces

SurfaceWhat it gives you
CLIwebsec run . is the whole pipeline. gate is the fast scoped pass on the files you just changed; attest projects a run into an audit-evidence table; feedback records that a finding is wrong, metadata-only by default, because a hardcoded-secret finding's snippet is the secret.
SARIF 2.1.0Written on every run and importable back in, so GitHub code scanning, IDEs and other tools read the same ledger.
GitHub ActionComposite action with a fail-on threshold, a baseline for legacy repositories, and SARIF upload. Every third-party action it calls is pinned by commit SHA. It installs the engine from its own checkout, never from the target repository.
DockerCore-only or bundled, non-root Python 3.14 images. Noir 1.0.0, Gitleaks 8.30.1 and Trivy 0.74.0 archives are checksum-bound. Native amd64/arm64 checks run; transitive builds and every scanner adapter are not certified.
MCP serverFour read-only repository tools, websec_recon, websec_findings, websec_sarif and websec_briefing, plus advisory websec_check_updates, over stdio or loopback-only HTTP that requires a token. Release checks default to offline metadata; online checks require consent and never install packages. Non-loopback bindings are rejected before listening.
pre-commitOne hook id, defaulting to --require-complete --fail-on high. Real disposable commit/pre-push checks pass, including staged/unstaged restoration; no consumer hooks are activated.
Claude Code pluginA security-pass skill and a PostToolUse hook: findings and incomplete analysis block; unexpected internal crashes fail open, loudly.

Four months, thirty-three releases, one breaking change

0.2 · 30 May 0.3 · 7 Jun 0.7 · 22 Jun 0.10 · 3 Jul 0.11 · 11 Jul 0.12 · 11 Sep 0.14 · 14 Sep 0.16 · 18 Sep 0.17 · 22 Sep 0.21 · 1 Oct First public line. PyPI via Trusted Publishing from day one. 0.3 walks AWS-CDK, AppSync SDL and VTL: the managed-cloud boundary a real pentest exposed. 0.7 adds the LLM / agent extractor. 0.10 adds SARIF output and the CI gate. 0.11 adds MCP over HTTP and the benchmark protocol. 0.12 fixes the hermeticity bug and adds branch protection. 0.14 ships schema 2.0. 0.16 adds the in-loop gate and attest. 0.17 splits the exit codes: the one breaking change. 0.18: reviewed calibration. 0.19.2: boundary fixes. 0.20: consent-driven advice. 0.21: bounded framework coverage and integration evidence. ← gap: July to September →

Fig. 9 — Release history through 1 October 2026. Releases do not imply rerunning dated proof or public review.

Check, do not update

Before review, the AI offers a release check and respects a decline. websec update-check is offline; --online requires consent. Only version/check-time metadata is cached; stale observations are dated. Checking never installs anything. An upgrade needs separate approval; refresh installed guidance explicitly afterward.

New bounded source coverage: registered Connexion JSON/literal YAML, FastAPI dependencies, mounted tRPC chains, Django returned-template CSP and Flask columns. Assigned SQL/command/upload flow is local, not whole-program proof. Dynamic bindings, middleware, context identity and deployment still need review. This reviewed v0.21.0 snapshot uses Selected fact tests and the PDF manifest; neither proves every sentence.