Evidence
A benchmark cannot tell you the agent built the wrong thing.
Neither can a scope-creep count, if the agent did not add anything. It replaced. Those are two different failures, and measuring only the first is how an AI system passes every test while shipping something nobody asked for.
This page is the evidence for the work behind that sentence. Every claim below links to something you can check without asking me.
The distinction
Two failures, not one
An instrument that measures addition and calls it governance will miss the more serious failure entirely.
Addition
The agent ships what you asked for, plus things you did not. An extra endpoint here, an admin route there. Each one widens the attack surface, has to be maintained, and was never reviewed because nobody knew to look for it.
Substitution
The agent ships something else entirely. Asked for a command-line tool, it builds a scheduled data pipeline. It added nothing, so a scope-creep count reports zero — the same score as the tools that got it right. Measured behaviour, not a hypothetical.
Both are measured, separately, because they fail differently and are fixed differently.
Validation
What has been tested, and how
Agreement with human judgement
Two people labelled the same sample independently, without seeing the instrument's answers. Agreement between them was κ = 0.870; between each of them and the instrument, κ = 0.853 and κ = 0.727. All three clear the threshold conventionally read as substantial agreement, which is what moves a measure from exploratory to usable.
The figures reproduce from the public repository with one command, on a clean install.
What the disagreement found
The raters disagreed with the instrument on one item, and they were right: its route detector could not see a whole class of web framework, so it had scored a case zero by construction rather than by judgement. That defect is published as an erratum, the repair is in the code with a regression test, and the claim it overturned — one the study had advanced as its cleanest result — was withdrawn in the text rather than quietly softened.
Receipts
Check any of it yourself
Nothing here needs to be taken on trust.
| Claim | Where to verify |
|---|---|
| The instrument is open source under MIT | github.com/dominicrume/ai-code-quality-auditor |
| It is published and installable | pypi.org · ai-code-quality-auditor |
| Download counts are third-party measured | pepy.tech |
| KYA Rails is in the official Canton Developer Hub catalogue | merged pull request #156, canton-network-devs |
| The mandate-and-receipts engine is published | pypi.org · knowyouragenticai-receipts |
| The agreement figures reproduce from the repository | scripts/compute_kappa.py --pre-erratum002 |
Limits
What is deliberately not claimed
An instrument built to audit other people's work has to hold itself to the standard it enforces.
Not "patented"
KYA is a UK patent application, GB2611754.9, filed in May 2026 and pending. An application is not a grant, and calling it one would be the first thing a diligence process caught.
Not a perfect agreement score
Repairing a defect the raters exposed lifts agreement with the instrument to κ = 1.000. That figure is circular — the defect was found by the raters and the repair then measured against the same labels — so the published figure remains the weaker 0.853.
Not adopted by Canton
KYA Rails is catalogued as a partner tool in the Canton Network developer hub repository. That is a listing, not an endorsement, a contract, or a statement about the Foundation.
Not a whole-project security score
Vulnerability density is measured per language. A JavaScript project scanned by a Python analyser reports a low number because of what the scanner cannot see, not because the code is safe.
Not yet validated for architectural substitution
Shape detection identifies 6 of 6 deliberately mispaired deliverables and flags none of 24 correct ones. Those controls were built by the same hand as the detector. Independent raters are being recruited, and until they have worked the result stands as a self-check.
If your agents are making decisions you would have to defend
The measurement layer is free and open source. What is worth paying for is the assurance built on top of it.