Verification

A fix is trusted because it passed reproducible checks, not because a model described it confidently. This page is the protocol behind the verified status and the "Prove it" button on a report page.

The protocol

1
Baseline
The app's own test suite runs first. If it doesn't pass, nothing downstream is trustworthy.
2
Security test, before
A generated request — Alice against Bob's resource — runs against the vulnerable code.
3
Expected: fails
The DENY assertion must fail (200, data returned). If it doesn't fail here, the bug wasn't reproduced, and nothing gets marked verified.
4
Patch applied
The minimal diff is applied to a fresh copy of the tree, not the one you're looking at.
5
Security test, after
The same request runs again against the patched code.
6
Expected: passes
The DENY assertion must now pass (403/404), and the paired ALLOW assertion — Alice reading her own data — must still pass.
7
Existing tests
The app's own suite runs again. A patch that breaks legitimate access is not a verified fix.
8
Re-scan
A deterministic scan of the patched tree must no longer show the original reachable path.
9
Proof
Before/after result, test output and the diff are recorded as the proof artifact.

Passing the generated security test alone is not enough. Existing tests passing and the deterministic re-scan coming back clean are both required — the definition Audit AI holds itself to:

A fix is verified only if all of the following hold:

the security regression test fails against the code before the fix;
the same test passes after the fix;
the existing test suite passes after the fix;
a deterministic re-scan no longer shows the original reachable path;
the diff matches the problem and contains nothing extra.

If any one of these doesn't hold, the status is not verified — it stays unverified, and says so.

Two tenants, 200 before, 403 after

v0 verification covers one scenario shape: a cross-tenant read. The sandbox seeds two synthetic tenants — Alice (tenant A) and Bob (tenant B) — with one user and one resource each, boots a throwaway local Supabase project (Postgres, GoTrue, PostgREST, Kong only) and the fixture app with next dev, then sends Alice's credentials at Bob's resource. Before the patch that request returns 200 with Bob's data; after the patch it returns 403 or 404. The paired ALLOW test — Alice reading her own resource — must pass in both phases, or the run is reported as an error rather than a verified anything: a broken ALLOW test in the "before" phase means the app never came up correctly; a broken ALLOW test "after" means the patch broke real access.

The fuller authorization matrix Audit AI checks against, where the application context supports it:

Actor	Own object	Other user	Other tenant	Admin
anonymous	DENY	DENY	DENY	DENY
user	ALLOW	DENY	DENY	DENY
manager	ALLOW	policy	DENY	DENY
admin	policy	policy	policy	ALLOW

"policy" means the correct answer depends on the app's own rules (a manager reading a peer's record, say) rather than being a flat allow or deny.

Sandbox isolation

Ephemeral: a fresh temporary directory per run, torn down unconditionally afterward.
No production credentials anywhere in the sandbox; synthetic seed data only.
The running application container has no network access.
Process timeout, memory limit, CPU limit, and --cap-drop ALL on the container that runs the app and the tests.
Filesystem isolation between the before/after copies of the tree.

What a proof request can come back as

"Prove it" on a report page queues a job for the worker. Three outcomes exist today, and the difference between them is the whole point of this page:

Verified fix. The repository sits in a known fixture layout (evals/fixtures/<name>/{vulnerable,secure,security-test,supabase}), so the worker can run the attack, apply the fix and run the attack again: HTTP 200 before, 403 after, existing tests still green, deterministic re-scan clean.
Reproduced in the sandbox (no fix applied). For repositories whose owner is on the worker's allow-list, the sandbox installs the app from its lockfile, applies its migrations, seeds two tenants from the parsed schema, generates the DENY/ALLOW test and runs it against the unmodified code. If another tenant's data comes back, the finding becomes confirmed. Nothing is fixed, so this is never called verified.
Needs setup. Every other repository. Audit AI does not run code from unknown owners unattended: a person reviews the request, prepares the seed and the test, and follows up through the contact you leave. A run the sandbox could not judge (the app did not start, the generator refused the finding) is reported as such, never as safe.

Limits

One scenario shape in v0: cross-tenant read. Cross-tenant write, privilege escalation and the storage/RPC classes are not yet proven end to end in the sandbox, even once the corresponding rule exists.
The dependency-install step of the sandbox has network access, to run the package manager; the application under test does not, once it's running.
Local Supabase uses fixed ports, so a proof run conflicts with an already-running supabase start on the same machine.
A proof run takes minutes, not seconds — it boots a real database and a real dev server.