websec-validator reads a repository the way a senior pentester reads it on day one, then hands your AI coding agent the plan: what the attack surface is, which findings are real, and exactly what to test. No model inside, no server, no running app.
AI agents now write the code. Nobody briefs them on what to attack. Static scanners answer “which patterns match?” and bury the answer under volume. A pentest answers “what matters, and how would I prove it?” and happens once a year, if at all.
findings-ledger.json total: 163 # what the scanners said cluster_summary distinct issues: 13 # what a reviewer actually has to fix
Same fixture, same run. The 163 is not wrong, and every one is kept for the machine consumers. But a human, or an agent, has to be told they are 13 problems, where they live, and what would confirm each one. That handoff is the product.
Fig. 1 — The same ledger, under two different questions. The second one is what a pentest report contains.
Read the repository. Map every route, auth decision, tenant boundary, sink, secret and dependency. Run the installed static scanners, de-duplicate them, and rank what is left by whether it is reachable and exploited in the wild.
A fact file, a findings ledger, a SARIF report, a probe library tailored to what was found, and a briefing for the agent that will do the fixing. Then an exit code that says whether the picture is complete.
The rule that surprises people: there is no model in the tool. Same repository, same output, every run. The model sits on the other side of the briefing, with a human beside it.
A tool that never says “incomplete” is a tool you cannot trust when it says “clean.”
Most security tools have two outputs: findings, or a green check. This one has a third, and it is what makes the other two believable.
# raw SQL interpolation · System.cmd injection · a hardcoded credential · no auth plug unsupported: [] gaps: [] REQUESTED CHECKS COMPLETED — 0 files read
A suffix list had been absorbed into another, leaving the “language I cannot read” branch reachable for one language.
The fix does not make the tool read Elixir. It makes it say it did not: the banner leads with
NO ANALYZABLE SOURCE, and --require-analyzed turns that into exit 3.
Fig. 2 — The exit-code contract. Incomplete outranks clean, so a check that silently did not run cannot produce a green build.
| Refusal | Why, and where |
|---|---|
| say “compliant” | The word appears in no output, and a test asserts that, so it cannot come back by accident. tests/test_attest.py |
| claim complete protection | protection_complete is always false: static analysis cannot see a running system. docs-canonical/SECURITY.md |
| fill an evidence gap | The audit projection has 12 control rows; 5 carry an empty evidence list, with the reason. attest.py |
| count “unknown” as “false” | The old rule scored every unmatched finding as a false positive. Relabelling against pinned source left 21 reviewed labels and 35 findings still unknown, excluded from both sides. calibration.json |
| optimize its own score | Anything fitted from labels uses a strictly proper rule, Brier or log score, never accuracy or F1, which reward confident wrongness. It is printed, never optimized. calibration.SCORING_RULE |
| run a scanner unasked | TruffleHog verifies secrets against live services, so it runs only when named. scanners.py |
| read a cell too thin to trust | A per-class probability needs at least five samples. One class reaches that today; the rest fall back a tier, and the output names which. calibration.MIN_N |
| guess reachability | A vulnerable dependency is tagged imported, no-import-found or n/a. It never rounds n/a to safe. enrichment.py |
| claim a benchmark it has not run | The competitor comparison exists as a written protocol. Until it runs, no head-to-head claim is made. BENCHMARKS.md |
The in-editor hook that checks an agent's edit exits 0 when it cannot run, and says on stderr that the edit was not checked: a check that blocks every edit when broken gets uninstalled, and then there is no check at all. It refuses an environment-variable escape hatch, because the agent is what sets environment variables.
Fig. 3 — Code in, artifacts out. The dashed line is the whole safety model: nothing above it can touch a running system.
Severity is a rule, not a score. The same evidence yields the same label every run, and the rule is written in the methodology before it is written in the code.
Fig. 4 — Severity order. A HIGH always has something outside the static analysis agreeing with it.
| Artifact | What it is |
|---|---|
| AGENT-BRIEFING.md | The product. Detected surface, the access-control map, targeting, findings, the method, and the staged probe list, written for the agent that will act on it. |
| FACTS.json | The full structured recon: every fact the 22 extractors produced, schema-versioned, with a coverage block that says what was skipped and why. |
| findings-ledger.json | The traceable ledger. Each finding carries its evidence chain, CWE / ASVS / OWASP-API citation, remediation, and a calibrated probability with its interval and sample size. REPORT.md is the same ledger for humans. |
| results.sarif | SARIF 2.1.0, always written. GitHub code scanning, GitLab, Azure DevOps and the VS Code SARIF viewer read it as-is. |
| probes/ | Scripts staged from the 24 templates, pre-filled with the endpoints the recon found, for a human to review and run against a test instance. |
Each extractor answers one question a pentester asks on day one, and writes the answer into FACTS.json as data, never prose.
Fig. 5 — The 22 extractors. Groups are editorial; the registry is flat. Each is isolated: one that crashes records an error in coverage and the run continues.
Not designed at a whiteboard. Three rows exist because dogfooding on a large LLM-agent monorepo, verified by a 15-agent adversarial pass, exposed two classes the route-and-auth model could not see. Four more grew because a real authenticated pen test found its two Criticals and two Highs in files the recon never parsed: AWS-CDK auth config, AppSync GraphQL schema, VTL resolvers. The walker reads those now.
Every finding carries a CWE, an ASVS 4.0.3 reference and, where they apply, the OWASP API and OWASP LLM Top 10 entries. The methodology publishes the map as a 41-row table; the findings module cites 77 distinct CWEs. Two limits are stated up front: regex over CDK TypeScript is not an AST and can be evaded by aliasing, and man-in-the-browser can never be a confident static catch, so it ships as LOW.
Presence on PATH is not compatibility, so each entry carries the lowest version whose CLI matches the invocation the tool builds, and doctor reports it.
| Scanner | Category · minimum |
|---|---|
| Trivy | vulnerabilities, secrets, misconfig · 0.38 |
| Gitleaks | secrets in history · 8.0; plus a working-tree pass |
| Semgrep/OpenGrep | SAST, with two shipped rules · 1.0 |
| Checkov | infrastructure as code · 2.0 |
| OSV-Scanner | dependency advisories · 2.0 |
| Prowler | cloud posture · detect only |
| Per stack | Runs only when the language is present |
|---|---|
| Bandit | Python · 1.7 |
| gosec | Go · 2.0 |
| Brakeman | Ruby · 4.0 |
| TruffleHog | live secret verification · opt-in only, never implicit |
That is the headline, and it is the right one. A tool that reported a critical in every public project it touched would be a tool that mislabels. The review ran source-only against pinned revisions, executed nothing, sent nothing upstream, and wrote down what it could not conclude.
Fig. 6 — One dot per lead. Counts are detector output at a pinned revision; the review says so on its first line.
createReadStream alone counted as serving.Each fix ships with its opposite: the test that makes the tool quieter is paired with one that
proves the real case, the JWT decoded without verification, the stream piped to a response, still fires. The review
also records its own gap: @angular/core was not recognized in the framework inventory.
Three deliberately vulnerable apps at fixed commits, retrieved 2026-09-12: VAmPI f16052d,
NodeGoat c5cb68a, DVGA a961308. The proof harness passes 10/10 surface checks with
Noir present and 8/10 without it, with 7 truth labels left unknown. Both numbers are committed as
JSON with the detector revision hash, and the sentences that quote them are bound to those JSON fields by DocGuard,
so the prose cannot outrun the artifact.
In September 2026 those 56 corpus findings were relabelled one at a time against the pinned source, each with a note and the revision it was read at. That produced 21 reviewed labels and a measured probability per severity, replacing a prior that had been guessed. The other 35 stay unknown and are counted on neither side.
And one real pentest: an authenticated engagement whose two Criticals and two Highs sat in the managed-cloud boundary the recon did not yet parse. Those findings became extractor rows, and two of the retest lessons corrected the tool's own earlier output.
Five checks gate the main branch. Two of them exist because of a specific, recorded failure.
Fig. 7 — The gate stack. The names in required-checks.json are the single source of truth; a required check that never runs would block every PR forever.
DocGuard runs with 27 of its 29 validators on, pinned explicitly rather than inherited. Four canonical documents are review-gated. Five sentences in the validation record are bound to JSON pointers inside the committed proof artifacts, so a number in prose cannot differ from the number in the file. Every conscious deviation from those documents is logged in a drift file with its reason.
This brief is held to the same standard. Every count on these pages, from the 22 extractors to the
116 leads, is asserted against the source tree by tests/test_explained_brief.py. If the code changes and the
page does not, the suite fails.
Every Monday a workflow proposes a version bump as a pull request and stops. A tag is cut only when a commit
whose subject starts with release: lands on main, never from a version diff. Publishing to PyPI uses
Trusted Publishing over OIDC, with no long-lived token, and refuses if the tag does not match the package version.
The triage bot that labels and merges bot PRs never checks out the PR's code, and its classifier has its own unit tests
in the required checks.
A field report from 2026-09-18 listed nine problems. They became 50 tests in one file, and the fixes carry the bug number as a comment at the line that changed, in the CLI, the scanner adapters, the findings module, the coverage contract and the dynamic phase, so a reader can walk from the report to the code and back.
The test file's own docstring states the rule: every “this should be quieter” assertion is paired with a “this must stay loud” one. A precision fix that is not paired with the true-positive it must preserve is not accepted. That is the mechanical form of the no-regression bar this project holds itself to.
Static analysis, secret detection, dependency scanning and attack-surface mapping are all solved problems with excellent open-source tools. Anyone claiming otherwise is selling something. This tool uses them and adds the layer none of them ship: the plan.
Semgrep, Trivy, Gitleaks, Checkov, OSV-Scanner, Prowler and the per-stack SAST tools: Bandit, gosec, Brakeman.
Eleven registry entries, each with a minimum version that doctor checks, because presence on PATH is not
compatibility. The tool shells out, parses, de-duplicates, and cites each finding back to the scanner that produced it.
OWASP Noir finds routes in a codebase better than a fallback regex ever will. When it is installed the proof harness scores 10/10; when it is absent, 8/10. Both numbers are published, because a user without Noir deserves to know what they are missing.
ZAP, Nuclei and sqlmap rediscover at runtime, expensively, much of what is visible in the source. So the briefing aims them: each static finding is mapped to the scanner signature it will produce, and the classes no scanner finds, BOLA, missing auth, mass assignment, row-level-security gaps, are listed as blind spots. A clean DAST run is not “safe,” and the briefing says why.
→ 163 findings, one severity each
A list. Every entry may be true. Nothing says which one a pentester would open first, what request would prove it, or which findings are the same bug seen from six files.
→ 13 issues, each with a location, a phase, an oracle, and a probe file
A briefing. Phase 1 safe recon, Phase 2 authorization with two identities, Phase 3 injection fired only at sink-backed endpoints and gated behind an explicit warning.
Fig. 8 — A finding keeps its name, its endpoint and its oracle. That is the difference between a metric and a plan.
| Surface | What it gives you |
|---|---|
| CLI | websec run . is the whole pipeline. gate is the fast scoped pass on the files you just changed; attest projects a run into an audit-evidence table; feedback records that a finding is wrong, metadata-only by default, because a hardcoded-secret finding's snippet is the secret. |
| SARIF 2.1.0 | Written on every run and importable back in, so GitHub code scanning, IDEs and other tools read the same ledger. |
| GitHub Action | Composite action with a fail-on threshold, a baseline for legacy repositories, and SARIF upload. Every third-party action it calls is pinned by commit SHA. It installs the engine from its own checkout, never from the target repository. |
| Docker | Everything bundled on a slim Python 3.14 image, running as a non-root user, with Noir 1.0.0, Gitleaks 8.30.1 and Trivy 0.74.0 pinned by version. |
| MCP server | Four read-only tools, websec_recon, websec_findings, websec_sarif and websec_briefing, over stdio or loopback-only HTTP that requires a token. Non-loopback bindings are rejected before listening. |
| pre-commit | One hook id, defaulting to --require-complete --fail-on high, so a partial run cannot pass the commit. |
| Claude Code plugin | A skill that runs the security pass, and a PostToolUse hook that checks each edit as it lands, failing open and loudly. |
Fig. 9 — Release cadence. Two of the last three releases corrected what the tool says about itself rather than what it detects.
Code in, briefing out. Everything it could not check, it says so.
Every number on these pages carries a path in the repository and is asserted by the test suite against the source tree at the version in the masthead. The field review, the proof artifacts, the calibration table and the CI workflows are all committed and readable. If you find a claim here that the code does not back, that is a bug: please open an issue.