Audit AI in numbers

What the scanner has looked at

What the rules found

By rule

Share is the rule's part of all open findings. Precision is real ÷ (real + false positive), from votes on report pages; unsure votes are listed but never counted. n is the number of real and false-positive votes behind a figure. Votes on earlier scans of a repository count too, so a rule can have votes and no open findings.

Sandbox proofs

A finding is proven only when we reproduce it. "Prove it" on a report runs the app in an isolated sandbox with two synthetic tenants and tries the attack before and after the fix.

Nobody has requested a proof yet. Until someone does, every finding above stays "likely".

Are the findings real?

Anyone reading a report can mark each finding as real, false positive or unsure. This is the public, self-reported view of precision; it is not checked by us, and people who vote are not a random sample.

No votes yet. The first ones will show up here, next to the rule they are about.

Precision on code we had never seen

On 23 Sep 2026 we scanned 30 public Next.js + Supabase repositories the engine had never run on, none of them in an earlier sample. They were picked mechanically from GitHub code search. The selection rules, how many findings would be labeled and what counts as a real finding were all written down before the scan, so the result could not be tuned after the fact. The scan produced 4,428 findings; the code behind a random 100 of them was then read and each one labeled real, a false positive or unsure, with a sentence of reason that cites the line. A second labeller, who did not see those labels, labeled 30 of them independently and gave the same label to 27 (90%); the figures below are the first labeller's.

Every blind sample so far

Samples 1 to 3 took at most 20 findings from one repository out of 20, and three repositories ended up supplying 60 of the 100 findings of the third. From sample 4 on it is at most 10 from one repository out of 30, so no single project moves the figure by more than a tenth. That change was written into the protocol before the fourth sample was picked. Sample 4 came out the same as sample 3 (30% against 31%, p = 0.87): the fixes made after the third sample did not raise precision on code the engine had never seen. Sample 5 came out lower, 22% against 30%, but within noise (p = 0.19): the engine changes made since then, including the split into headline findings and leads, did not raise precision on new code either. Because leads never block, only 17 of its 100 findings are in the blocking tier, so that column says little for sample 5.

By rule, sample #5

Between the samples: the 64 false positives of the first one named gaps in the engine (credential checks other than Supabase auth, guards in a layout, a schema parse as an allow-list, a row picked by the caller's own identity, a write that Row Level Security refuses anyway). Two rounds of fixes removed 28 of those findings and lost none of the 36 real ones, which put precision at 50% on the same labels (n = 72). That number was not blind, because the fixes were made with those labels in view, so the fixed engine was run on new repositories instead, and every later round of fixes was measured the same way: those are the later samples above.

What these figures are not: they do not say we find everything. Measuring that would take a full manual audit of each repository. They also say nothing about findings we prove: a sandbox reproduction is real by construction, which is the whole point of the proof step. The rules with the lowest precision are where the noise lives, and they are what we are fixing next.

The older sample the engine was tuned against

The blind measurements above are the honest ones. This older count is kept because the engine was tuned against it, which is exactly what makes it worth showing next to a blind figure: on 13 Sep 2026 we scanned 26 public Next.js + Supabase repositories, then the code behind every one of the 183 findings was read and labeled real, false positive or unsure. As in the blind samples above, the reading was done by a model working from the code, not by a person. The figures below cover the 131 of them that scanner build ebf88ea still reports; the rest were false positives it no longer raises, and no finding labeled real was lost. Those fixes were made while looking at these labels, so this is not a blind test.

By rule

Repositories were chosen by GitHub search for Next.js + Supabase, not at random, and labeled by reading code, not by running attacks; a label can be wrong. A false positive here means a control exists that the rule missed, or the data is public by design. Rules with no row matched nothing in this scan. Method and label definitions: real-world labeled sample.

Methodology

The rules, and a vulnerable and a secure test app for each, are open source: what each rule catches · rule source. Numbers are regenerated at most every five minutes.