websec-validator reads a repository the way a senior pentester reads it on day one, then hands your AI coding agent a plan: the supported attack surface, which leads need review, and how to test them. No model inside; the core pass needs no server or running app.
AI agents now write the code. Nobody briefs them on what to attack. Static scanners answer “which patterns match?” and bury the answer under volume. A pentest answers “what matters, and how would I prove it?” and happens once a year, if at all.
findings-ledger.json total: 163 # what the scanners said cluster_summary distinct issues: 13 # what a reviewer actually has to fix
Same fixture, same run. The 163 is not wrong, and every one is kept for the machine consumers. But a human, or an agent, has to be told they are 13 problems, where they live, and what would confirm each one. That handoff is the product.
Fig. 1 — The same ledger, under two different questions. The second one is what a pentest report contains.
Read supported source. Map visible routes, auth decisions, tenant boundaries, sinks, secrets and dependency. Run the installed static scanners, de-duplicate them, and rank what is left by whether it is reachable and exploited in the wild.
A fact file, a findings ledger, a SARIF report, a probe library tailored to what was found, and a briefing for the agent that will do the fixing. Then an exit code that says whether the picture is complete.
The rule that surprises people: there is no model in the tool. Analysis is reproducible with the same inputs and tooling; run metadata and dated advice can vary. The model sits on the other side of the briefing, with a human beside it.
A tool that never says “incomplete” is a tool you cannot trust when it says “clean.”
Most security tools have two outputs: findings, or a green check. This one has a third, and it is what makes the other two believable.
# raw SQL interpolation · System.cmd injection · a hardcoded credential · no auth plug unsupported: [] gaps: [] REQUESTED CHECKS COMPLETED — 0 files read
A suffix list had been absorbed into another, leaving the “language I cannot read” branch reachable for one language.
The fix does not make the tool read Elixir. It makes it say it did not: the banner leads with
NO ANALYZABLE SOURCE, and --require-analyzed turns that into exit 3.
Fig. 2 — With --require-complete and --fail-on. Usage is checked first; findings + incomplete exits 1, with both recorded. Ungated exit 0 is not a completeness verdict.
| Refusal | Why, and where |
|---|---|
| say “compliant” | The word appears in no output, and a test asserts that, so it cannot come back by accident. tests/test_attest.py |
| claim complete protection | protection_complete is always false: static analysis cannot see a running system. docs-canonical/SECURITY.md |
| fill an evidence gap | The audit projection has 12 control rows; 5 carry an empty evidence list, with the reason. attest.py |
| count “unknown” as “false” | The old rule scored every unmatched finding as a false positive. Relabelling against pinned source left 21 reviewed labels and 35 findings still unknown, excluded from both sides. calibration.json |
| optimize its own score | Anything fitted from labels uses a strictly proper rule, Brier or log score, never accuracy or F1, which reward confident wrongness. It is printed, never optimized. calibration.SCORING_RULE |
| run a scanner unasked | TruffleHog verifies secrets against live services, so it runs only when named. scanners.py |
| read a cell too thin to trust | A per-class probability needs at least five samples. One class reaches that today; the rest fall back a tier, and the output names which. calibration.MIN_N |
| guess reachability | A vulnerable dependency is tagged imported, no-import-found or n/a. It never rounds n/a to safe. enrichment.py |
| claim a benchmark it has not run | An offline captured-report harness preserves reviewed TP/FP/unknown labels and unscored partial captures. No head-to-head or agent-benefit experiment has run. BENCHMARKS.md |
The in-editor hook blocks on findings or completed-but-incomplete analysis, including selected source it could not read. Only unexpected internal crashes fail open, loudly: the edit was not checked. Client-side settings remain editable, so production enforcement belongs in required CI checks, not this convenience hook.
Fig. 3 — Staging sends no requests. Review and authorization precede execution; optional external scanners are not a sandbox.
Severity is a rule, not a score. The same evidence yields the same label every run, and the rule is written in the methodology before it is written in the code.
Fig. 4 — Default evidence tiers; individual scanner/detector rules retain their documented classifications. Severity is not vulnerability confirmation.
| Artifact | What it is |
|---|---|
| AGENT-BRIEFING.md | The product. Detected surface, the access-control map, targeting, findings, the method, and the staged probe list, written for the agent that will act on it. |
| FACTS.json | The full structured recon: every fact the 22 extractors produced, schema-versioned, with a coverage block that says what was skipped and why. |
| findings-ledger.json | The traceable ledger: evidence chain, CWE / ASVS / OWASP-API citation, remediation and bucket-based confidence with provenance, interval and sample size. Priors are not measurements. REPORT.md is the same ledger for humans. |
| results.sarif | SARIF 2.1.0, always written. GitHub code scanning, GitLab, Azure DevOps and the VS Code SARIF viewer read it as-is. |
| probes/ | Scripts staged from the 24 templates, pre-filled with the endpoints the recon found, for a human to review and run against a test instance. |
Each extractor answers one question a pentester asks on day one, and writes the answer into FACTS.json as data, never prose.
Fig. 5 — The 22 extractors. Groups are editorial; the registry is flat. Each is isolated: one that crashes records an error in coverage and the run continues.
Not designed at a whiteboard. Three rows exist because dogfooding on a large LLM-agent monorepo, verified by a 15-agent adversarial pass, exposed two classes the route-and-auth model could not see. Four more grew because a real authenticated pen test found its two Criticals and two Highs in files the recon never parsed: AWS-CDK auth config, AppSync GraphQL schema, VTL resolvers. The walker reads those now.
Every finding carries a CWE, an ASVS 4.0.3 reference and, where they apply, the OWASP API and OWASP LLM Top 10 entries. The methodology publishes the map as a 41-row table; the findings module cites 77 distinct CWEs. Two limits are stated up front: regex over CDK TypeScript is not an AST and can be evaded by aliasing, and man-in-the-browser can never be a confident static catch, so it ships as LOW.
Presence on PATH is not compatibility, so each entry carries the lowest version whose CLI matches the invocation the tool builds, and doctor reports it.
| Scanner | Category · minimum |
|---|---|
| Trivy | vulnerabilities, secrets, misconfig · 0.38 |
| Gitleaks | secrets in history · 8.0; plus a working-tree pass |
| Semgrep/OpenGrep | SAST, with two shipped rules · 1.0 |
| Checkov | infrastructure as code · 2.0 |
| OSV-Scanner | dependency advisories · 2.0 |
| Prowler | cloud posture · detect only |
| Per stack | Runs only when the language is present |
|---|---|
| Bandit | Python · 1.7 |
| gosec | Go · 2.0 |
| Brakeman | Ruby · 4.0 |
| TruffleHog | live secret verification · opt-in only, never implicit |
Dated source-only review: 14 September 2026. A tool that reported a critical in every public project it touched would be a tool that mislabels. The review ran source-only against pinned revisions, executed nothing, sent nothing upstream, and wrote down what it could not conclude.
Fig. 6 — One dot per lead. Counts are detector output at a pinned revision; the review says so on its first line.
createReadStream alone counted as serving.Each fix ships with its opposite: the test that makes the tool quieter is paired with one that
proves the real case, the JWT decoded without verification, the stream piped to a response, still fires. The review
also records its own gap: @angular/core was not recognized in the framework inventory.
Dated v0.14.0 proof, not a v0.21.0 rerun or vulnerability recall. Three deliberately vulnerable apps at fixed commits, retrieved 2026-09-12: VAmPI f16052d,
NodeGoat c5cb68a, DVGA a961308. The proof harness passes 10/10 surface checks with
Noir present and 8/10 without it, with 7 truth labels left unknown. Both numbers are committed as
JSON with the detector revision hash. DocGuard binds their validation-record statements to JSON fields;
this brief uses separate fact tests and a PDF freshness manifest.
In September 2026 those 56 corpus findings were relabelled one at a time against the pinned source, each with a note and the revision it was read at. That produced 21 reviewed labels and a measured probability per severity, replacing a prior that had been guessed. The other 35 stay unknown and are counted on neither side.
And one real pentest: an authenticated engagement whose two Criticals and two Highs sat in the managed-cloud boundary the recon did not yet parse. Those findings became extractor rows, and two of the retest lessons corrected the tool's own earlier output.
Five checks gate the main branch. Two of them exist because of a specific, recorded failure.
Fig. 7 — The gate stack. The names in required-checks.json are the single source of truth; a required check that never runs would block every PR forever.
DocGuard runs with 27 of its 29 validators on, pinned explicitly rather than inherited. Four canonical documents are review-gated. Five sentences in the validation record are bound to JSON pointers inside the committed proof artifacts, so a number in prose cannot differ from the number in the file. Every conscious deviation from those documents is logged in a drift file with its reason.
Publication has separate safeguards. Selected facts are checked by
test_explained_brief.py; a hash manifest binds the HTML, PDF and renderer script. CI rejects a stale pair.
DocGuard checks marked Markdown sections, not PDF rendering. Prose accuracy and all eight rendered pages still need human review.
Every Monday a workflow proposes a version bump as a pull request and stops. A tag is cut only when a commit
whose subject starts with release: lands on main, never from a version diff. Publishing to PyPI uses
Trusted Publishing over OIDC, with no long-lived token, and refuses if the tag does not match the package version.
The triage bot that labels and merges bot PRs never checks out the PR's code, and its classifier has its own unit tests
in the required checks.
A field report from 2026-09-18 listed nine problems. They became 50 tests in one file, and the fixes carry the bug number as a comment at the line that changed, in the CLI, the scanner adapters, the findings module, the coverage contract and the dynamic phase, so a reader can walk from the report to the code and back.
The test file's own docstring states the rule: every “this should be quieter” assertion is paired with a “this must stay loud” one. A precision fix that is not paired with the true-positive it must preserve is not accepted. That is the mechanical form of the no-regression bar this project holds itself to.
Static analysis, secret detection, dependency scanning and attack-surface mapping are all solved problems with excellent open-source tools. Anyone claiming otherwise is selling something. This tool uses them and adds the a repository-aware planning layer. The benchmark protocol exists; no unrun head-to-head result is claimed.
Semgrep, Trivy, Gitleaks, Checkov, OSV-Scanner, Prowler and the per-stack SAST tools: Bandit, gosec, Brakeman.
Eleven registry entries, each with a minimum version that doctor checks, because presence on PATH is not
compatibility. The tool shells out, parses, de-duplicates, and cites each finding back to the scanner that produced it.
OWASP Noir extends route discovery beyond the fallback patterns. In the dated v0.14.0 proof the harness scores 10/10 with Noir and 8/10 without it. Both numbers are published, because a user without Noir deserves to know what they are missing.
ZAP, Nuclei and sqlmap test a live system. The briefing suggests relevant signatures, not guaranteed detections. BOLA, missing auth, mass assignment and row-level-security gaps often need identity-aware manual controls. A clean DAST run is not “safe,” and the briefing says why.
→ 163 findings, one severity each
A list. Every entry may be true. Nothing says which one a pentester would open first, what request would prove it, or which findings are the same bug seen from six files.
→ 13 issues, each with a location, a phase, an oracle, and a probe file
A briefing. Phase 1 safe recon, Phase 2 authorization with two identities, Phase 3 injection fired only at sink-backed endpoints and gated behind an explicit warning.
Fig. 8 — A finding keeps its name, its endpoint and its oracle. That is the difference between a metric and a plan.
| Surface | What it gives you |
|---|---|
| CLI | websec run . is the whole pipeline. gate is the fast scoped pass on the files you just changed; attest projects a run into an audit-evidence table; feedback records that a finding is wrong, metadata-only by default, because a hardcoded-secret finding's snippet is the secret. |
| SARIF 2.1.0 | Written on every run and importable back in, so GitHub code scanning, IDEs and other tools read the same ledger. |
| GitHub Action | Composite action with a fail-on threshold, a baseline for legacy repositories, and SARIF upload. Every third-party action it calls is pinned by commit SHA. It installs the engine from its own checkout, never from the target repository. |
| Docker | Core-only or bundled, non-root Python 3.14 images. Noir 1.0.0, Gitleaks 8.30.1 and Trivy 0.74.0 archives are checksum-bound. Native amd64/arm64 checks run; transitive builds and every scanner adapter are not certified. |
| MCP server | Four read-only repository tools, websec_recon, websec_findings, websec_sarif and websec_briefing, plus advisory websec_check_updates, over stdio or loopback-only HTTP that requires a token. Release checks default to offline metadata; online checks require consent and never install packages. Non-loopback bindings are rejected before listening. |
| pre-commit | One hook id, defaulting to --require-complete --fail-on high. Real disposable commit/pre-push checks pass, including staged/unstaged restoration; no consumer hooks are activated. |
| Claude Code plugin | A security-pass skill and a PostToolUse hook: findings and incomplete analysis block; unexpected internal crashes fail open, loudly. |
Fig. 9 — Release history through 1 October 2026. Releases do not imply rerunning dated proof or public review.
Before review, the AI offers a release check and respects a decline. websec update-check is
offline; --online requires consent. Only version/check-time metadata is cached; stale observations are dated.
Checking never installs anything. An upgrade needs separate approval; refresh installed guidance explicitly afterward.
New bounded source coverage: registered Connexion JSON/literal YAML, FastAPI dependencies, mounted tRPC chains, Django returned-template CSP and Flask columns. Assigned SQL/command/upload flow is local, not whole-program proof. Dynamic bindings, middleware, context identity and deployment still need review. This reviewed v0.21.0 snapshot uses Selected fact tests and the PDF manifest; neither proves every sentence.