Northbeam Precision / Packing Station 1
The expected quantity is never sent to the model. It reports what it sees, per line, and code does the comparing. The check is blind on quantity and slip number. The expected lot is shown to the model, and the lab below shows what that leaves open.
Each card replays a real failure with recorded answers, so it behaves the same every time. Steel blue is what the model said. Copper is what code decided. No live model runs here.
Blank lot
Carton fronts with no lot on them. The model cites the slip and answers 0.95. A slip is printed from the order, so it can't prove what is in the box.
Launder a stop
Identical retries can flip a verdict, so resubmitting a stop can produce a pass. Submit the same photos and watch the raw answer change.
Ghost frame
The goods get moved, then the packer presses capture. A queued camera stream can hand back the table as it was before the press.
Two lots
One product in two lots, 5 cartons of each. A strict schema writes fields in order, so a count placed before the reasoning commits the model early.
Slip misread
Lookalike characters fold only where the expected number has a letter. Digits must match as read, because a misread digit and a real wrong slip look the same to code.
Format break
Without a schema, some answers arrive with prose around the JSON or cut off. Recovery is flagged and never silently trusted. A schema fixes this and nothing else.
Each candidate runs against archived orders, next to the serving prompt. These are recorded results from real replay runs, shown on the fictional fixtures.
| Gate | Required | Result | Status |
|---|
Test the test
Two different candidates caused the same false stop. Either the order is fragile or the rig favors one arm. An A/A run gives both arms the live prompt, with only a code comment added to one.
How this works
Shape. A small PC and a touchscreen at the packing bench, a camera on a mast, and one pipeline in a self-hosted workflow engine: 72 nodes, five webhook entry points, and two code nodes that carry all the judgment. One builds the vision request. One parses the reply and renders the verdict.
Ground truth is the order system, not the paperwork. The packing slip is a printed artifact that can be wrong, and worse, it shows the expected quantity in a place a vision model can read. Counts come from the order lines. The slip is photographed for the record and the prompt carries an explicit prohibition on deriving counts from it.
Confidence has a floor per product category. High risk categories get a strict floor, low risk categories get a loose one, and some categories are marked manual-confirm-only and bypass the model entirely by design, with a written human note required.
Deterministic checks sit on top. Each exists because of a real confirmed wrong shipment, and all of them have the same signature: the model read the discriminating text correctly and then reasoned past it in its own summary sentence. The variant check here is the third of that family. A premium variant and a standard variant have names where the standard is a strict substring of the premium, and a fuzzy name match has no penalty for an absent qualifier, so the cheaper part scores near perfect against the premium line. The check reads the variant off the SKU, which is order-system ground truth, and hard fails when the transcribed marking disagrees. It is a gate: it can only add a stop, which is what makes it safe to ship. The engine also has narrow lifts, which can turn a stop into a pass and each need stated evidence. In all, the engine has 9 hold conditions and more than a dozen review conditions.
Two live subtleties this demo reproduces exactly. A merged order can put the same product on two separate lines, so keying by product identity compares a frame total against one line and holds correct goods. Measured on real traffic: 4 orders in 569 carry a duplicate key and none had ever passed. And the camera sees raw units while the order line is in sale units, so a perfect read scores as a mismatch on every case-packed line unless the count is normalized before it is compared.
Release discipline. The verdict engine has reached v2.66, with more than a dozen releases served since August. Every engine change is replayed against stored history before it goes live: up to 232 retained runs and 2,830 stored final answers, which compare old code with new and nothing else. Prompt changes run through a paired rig with a control arm and gates written before the run. Deploys check the engine is idle, write both version rows in one transaction, need no restart, and roll back with one command.
Measured in production. Roughly 40 orders a working day. First-try pass ran 78.5 to 87.5 percent by month and station from August to October. About 9 seconds a verify on structured outputs, from one partial day, and about $0.15 of model spend per order at list rates. It was built to be time neutral on purpose, because a check that slows the floor gets ripped out.
Also measured, and published on purpose. Of 16 override-and-ship finalizes, 7 overrode a hold. The 3 of those checked from photos all had correct goods. A stop that is wrong often enough teaches operators to click through, so the false-stop classes were then chased down. Anyone who has run a quality gate recognizes that risk.
Sign in
No PIN, by decision. Every action is attributed to a name, and the server rejects any state change that arrives without one, so the gate cannot be bypassed by hitting the API.