Dashboard

Normalization sandbox

Snapshot 2026-10-02 15:24 UTC. Refresh progress. Comparisons include skips; results count decided comparisons.

Judges

Is this panel ready to publish?

A verdict per judge, worst first, and the one thing to do about it. This is the section to read before the standings: everything below is about the field and the fit, and this is about whether the panel that produced them is finished. Nothing here can move a ranking - the pass refits the pairwise model once per judge, to measure each of them against a consensus that excludes their own decisions, and throws every refit away. It is also organizer-only in a stronger sense than the rest of this page: a verdict is never shown to the judge it is about, because a judge who reads one starts judging the report instead of the work.

Method

per-judge-calibration+leave-one-out(surprise>=1.5x, cycles>=0.15 over 8 triples, one ballot>=4sd, bloc>=0.2 at 0.05 over the table)

Panel surprise

-

Measured out of their own fit

0 of 0 judges

Surprise is how unlikely this judge's duels are under a model fitted without them, per duel, so it is comparable across judges who decided different numbers. A judge measured in sample rather than held out is one whose removal disconnects the field: their own decisions are in the model they are being scored against, which flatters them, and the row says so.

No pair of judges shares enough comparisons to test for agreement beyond what this panel predicts.

Everything the pass said

Each of these is said once here and nowhere else on the page. The tagged ones are what the verdicts above are made of; the rest are printed so that noticing them in the raw ballots does not look like a discovery.

• Judge Amara Silva scores 22.7% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Anya Sokolova's slope was not fitted at all and is held at 1.00, because 1 ballot is not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Anya Sokolova scores 25.1% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Bruno Costa's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Bruno Costa scores 26.3% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Dilan Yilmaz's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• act on this Judge Emeka Adeyemi's scores barely move with the field: their fitted slope is 0.35, so their ballots separate the projects they saw much less than the panel does. The fit weights them down accordingly, which means the placement of those projects is resting on the other judges who saw them.
• Judge Felix Roth scores 20.8% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Hiro Tanaka scores 64.7% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• act on this Judge Iva Petrova's scores barely move with the field: their fitted slope is 0.40, so their ballots separate the projects they saw much less than the panel does. The fit weights them down accordingly, which means the placement of those projects is resting on the other judges who saw them.
• Judge Iva Petrova used 0.0% of the rubric's range across 3 ballots — the distance between their own lowest and highest score, averaged over the criteria. The fit rescales that, so it costs nothing on its own.
• Judge Kofi Mensah's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Kofi Mensah scores 26.3% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• act on this Judge Lars Berg's scores barely move with the field: their fitted slope is 0.35, so their ballots separate the projects they saw much less than the panel does. The fit weights them down accordingly, which means the placement of those projects is resting on the other judges who saw them.
• Judge Leila Nasser's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Leila Nasser scores 16.8% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Lena Kovac scores 69.0% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Marek Nowak scores 21.7% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• act on this Judge Mira Kaur's scores barely move with the field: their fitted slope is 0.35, so their ballots separate the projects they saw much less than the panel does. The fit weights them down accordingly, which means the placement of those projects is resting on the other judges who saw them.
• Judge Nadia Rahman scores 17.4% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• act on this Judge Nils Berg is on the panel with nothing assigned and nothing filed. Their place on the roster is counted in every "how many judges" figure on this page.
• act on this Judge Noor Haddad's scores barely move with the field: their fitted slope is 0.35, so their ballots separate the projects they saw much less than the panel does. The fit weights them down accordingly, which means the placement of those projects is resting on the other judges who saw them.
• Judge Pavel Ivanov's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Priya Nair's slope was not fitted at all and is held at 1.00, because 2 ballots are not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Priya Nair scores 19.6% of the rubric's span above the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Thandi Dlamini scores 15.3% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• Judge Tomas Varga's slope was not fitted at all and is held at 1.00, because 1 ballot is not enough to fit one. Their scores go into the ranking with only their offset removed.
• Judge Tomas Varga scores 45.1% of the rubric's span below the panel. This is the difference the fit exists to remove, and it is removed: their offset is subtracted before any project is ranked.
• There are no duels on this event, so the pairwise half of this report — surprise, self-consistency and who agrees with whom — is not measured.
• 8 judges have too little on file to calibrate — under 3 ballots and under 5 duels: Tomas Varga, Priya Nair, Kofi Mensah, Dilan Yilmaz, Bruno Costa, Anya Sokolova and 2 more. An empty row is the absence of a clean bill of health rather than one.

Coverage

8 of 41 projects are short of the 3 reviews this event asks for. Only those are listed.

Draw assignments

Assigning judges to projects is idempotent in effect but not in cost: drawing again after ballots exist adds assignments rather than replacing them. Preview first.

Add a track

Exports & Integrations

Full RFC 4180 CSV export at every stage:

Registrations · Teams · Projects · Assignments · Ballots · Results · Votes · Audit

Offline Certificate Verifier · Server Ed25519 Public Key · Embeddable Gallery Widget

Standings · Rubric · Projects