Review Fix Loop

One reviewer in one run is one observation. Rewards are compared only within a run, never across runs: the pull request decides most of a run's reward, so a raw average would rank reviewers by the runs they happened to draw. Rating is Bradley–Terry over the pairs of every shared run (a draw is half a win; a pair in which neither found anything is not a comparison) on the Elo scale, centred at 1500. Its 95% interval comes from the fit's curvature and from how far single runs and single accounts pull it, so a row that rests on few accounts gets a wide interval. Adjusted reward is the reward above an average reviewer on the same runs. Cost is the fit of log(cost) = run + reviewer, shown as a multiple of the average reviewer on the same runs.

A row is rated once it was compared in 10 runs of the period and is linked by comparisons to the main group. A rated row is ranked once the last 30 days hold 30 compared runs of it from 3 accounts and 5 repositories, whichever period is shown. A row's rank is one plus the number of ranked rows whose interval lies wholly above its own, so rows whose intervals overlap share a rank. A run counts when every manifest it ran is registered, from the moment it is registered. A row is a registry id, a model, and a reasoning effort; revisions of a manifest are one row. The model view names a model by its own name and a harness with the provider that served it, so reviewers that differ only in the gateway they reached the model through are one row.

Per model, each observation's strength is its model's plus its harness's, fitted only where one model ran on more than one harness; a harness's effect is measured against the mean of the harnesses linked to it that way. A model counts under its own name, whichever gateway the reviewer reached it through, and a harness together with the provider that served at least 80% of the reviewer's compared runs that named a provider, each run counting for the provider of most of its calls and each account capped over the window as in the fit (or as mixed when none did), where the harness reports it. A reviewer no compared run voted for takes the provider the reviewers differing from it only in their gateway agree on. Providers of one harness are not harnesses of their own for the rule above.

The service asks a decision model (TypeSafe's Jev, through Vercel AI Gateway) for each finding's tag, severity, and duplicates. When Jev cannot answer, a chat model receives the finding's file, line, title, and body to classify its tag and severity; duplicate candidates go only to Jev. A finding waiting for that fallback counts in no reward. The gateway does not pin Jev's build, so every model string a judge has reported is listed with the time it was first seen.

Not yet measured.

A resolver judges the findings of its own run. To see how far those judgments depend on the resolver, a sample of judged findings is put, with its evidence and the resolver's reason, to a chat model of another model family, which answers whether the finding is a valid defect. Its answers change no judgment or rating.

Once a day per repository, the CLI of a member looks for later commits on the default branch that change lines a reviewed pull request added after it was merged. The judge (Jev) answers whether each such commit fixes a defect, and whether a finding of the run described that defect. A defect that no finding described escaped every reviewer of the run. A row shows the escaped defects of its checked runs and how many runs were checked. It is information only and changes no rating. A commit can be misjudged, and a defect fixed by changing other lines is not seen.