Assay

WHY I DESIGNED IT

I have previously evaluated model outputs against rubrics and most of what exists is repurposed annotation software built for labeling images. It treats scoring as data entry when it is judgment. Agent evaluation is different. The sequence is not a label you read or an image you see but a trajectory: a sequence of calls where the failure at step 5 is a consequence of step 1 and no interface built for labeling text spans or images will show you that.

WHO IT'S FOR

• Reviewers: not to be slowed down and not to be blamed for an unclear instrument.
• Adjudicators: needs to see what each reviewer looked at not just where they landed.
• Program leads: throughput and quality at once, decided weeks before seeing the result.

THE PROBLEM

Evaluation data is the input to every decision a lab makes about whether a model ships. Unreliable scores are indistinguishable from reliable ones until the model trained on them starts behaving oddly.

Four causes, all of them design problems. Rubrics are ambiguous on first contact with no way to say so. The person who wrote the criterion knows what they mean. The person applying it hours later doesn't. Scores drift without anyone lying. Disagreements go unresolved because there is nowhere to put them. And the rubric itself never gets updated because nobody can prove it is the rubric that is wrong.

There is a real tension that comes with all four. Throughput matters, there are cold hours against deadlines and a tool that makes careful work slow makes careful work not happen. Care matters more because unreliable labels are identical to reliable ones until much later.

SIX DESIGN DECISIONS

• Deterministic checks run before a human sees the item. Machine-settleable violations are settled by a machine.
• Evidence first, score second. The reviewer attaches steps to criteria, then scores. Not the reverse. Justification stops being a chore because the attachment is the justification and Adjudication becomes concrete
• Failures are traced to their cause not just their location. A failure late in a trajectory is usually caused by something early like a constraint that arrived and was never carried forward. Linking them means the evidence records what went wrong rather than where it surfaced.
• The rubric is treated as a thing that can be wrong. Most tools treat the rubric as fixed and the reviewer as the variable, so low agreement becomes a training problem. Assay assumes the opposite is often true.
• Reviewers can dispute a check not just a score. Any check can be disputed from inside the task, carrying the item and the evidence so far. A reviewer who can't push back on a bad instrument will guess or leave.
• Data can't leave without passing a gate. Agreement per criterion, unadjudicated items, unresolved disputes, version pinning. When the numbers don't hold, release is blocked.

DESIGNING ASSAY

In order to get the screens designed I built 50+ components with variants across nine groups, 84 typography variations and a tiered token system cutting across brand, alias, mapped and responsive collections. It was either this or having to use 80 slightly different unbound grays for the screens.

SCREENS & USE CASE EXAMPLE

The screens below shows what happens in a batch from setup to a dataset release across three different roles on the platform. A program lead, reviewer and adjudicator.

ONE PLATFORM ANY DOMAIN

The thing being evaluated changes completely between domains. What a reviewer has to do doesn't.

Agent failures have the same shape wherever they happen. A constraint arrives early, gets dropped somewhere in the middle and the consequence lands several steps later. Rotterdam is a cold chain requirement retrieved at step 1 and ignored at step 5. The same structure holds whether the agent is handling a financial reconciliation, a clinical intake or a support escalation.

So only three things swap between domains: the tool registry, the check suite and the rubric. The interface doesn't move. A domain expert in one field and a domain expert in another are learning the same tool and a program running several isn't maintaining several of everything.

The same problem outside a lab

Labs evaluating models before release are the obvious user, but they aren't the only one.

Any company running an agent inside its own product has the same problem in a different shape: it could be a bank triaging disputes, a helpdesk resolving tickets. Someone internally has to answer whether the agent is behaving and without an evaluation instrument that answer tends to be a sample read by one person, which can't be compared across prompt changes or across months.

Most of Assay applies unchanged and the qualification problem shrinks because the reviewers are already the domain experts. Two things differ. The batch becomes a continuous sampling policy rather than a fixed set. And the release gate becomes a deployment gate: the same checks, reading the same agreement numbers, deciding whether a new prompt or model version reaches production instead of whether a dataset ships.

WHAT I WOULD AND WOULD NOT VALIDATE

Does evidence-first cost more time than it saves? My assumption: slower on the first pass, faster on adjudication, net positive. I haven't measured it.

Do reviewers use cause links or attach and move on? This is my most distinctive decision and the one I'm least sure of. It asks a reviewer to form a theory about causation, not just notice a failure. That's real cognitive work nobody wants to do.

Is ambient calibration reassuring or is it surveillance? Showing a reviewer they're scoring one criterion harder than their cohort could read as useful or as a performance metric with a friendly face. The answer depends almost entirely on what who runs the program does with the number.

Will a program lead accept that the rubric is the problem? That's a claim about organisational behaviour and not about the interface.

The release gate. Whether data should be blocked from leaving without passing quality thresholds isn't a usability question but a position and testing it would only tell me whether people under deadline find it inconvenient.

Additionally I would also keep my eyes on three numbers:

• Agreement per criterion, tracked per rubric version. A criterion whose agreement jumps after its anchors are rewritten is the whole thesis working.
• Adjudication rate over time. It should fall. If it's flat, reviewers are disagreeing about the same things and nothing upstream is being fixed.
• Days between a reviewer raising a dispute and the rule changing. The loop the whole tool exists to close. If that number falls, the instrument is improving. If it climbs, disputes are being collected rather than answered and Assay has become an elaborate suggestion box.