Shipping verification kiosk
- The system
- Two camera stations where a vision model reports what it sees and code makes the decision, on roughly 40 orders a working day from a catalog full of near-identical variants.
- How it works
- The model reads, and the order system stays ground truth. 9 conditions force a hold and more than a dozen send an order to review.
- Results
- 17 wrong boxes stopped and fixed in 12 weeks, each proved by photos, at about $0.15 of model spend per order and about 9 seconds a verify.
- My role
- Architect and engineer. I designed the system, set its rules and made the release calls.
What it replaced
An operator used to compare a packed box against a screen. Products in this catalog can differ by one character: the same part in another size, or two versions with nearly the same name. Packers get it right almost every time, and when they miss, a customer has to be made whole. If someone said a box was short, nobody could prove otherwise. The check also had to fit the floor, or it would get ripped out.
How it works
The packer signs in, scans the order and takes a short photo sequence. The order lines come from the order system, never from the printed slip, because the slip is printed from the order and can't prove what is in the box.
The model is not told the expected quantity or slip number. It reports what it sees for each line through a schema-checked answer, and code compares that with the order. Confidence is scored per line against a floor for each product type.
Rules come in two kinds. Gates only add stops, such as a lot that must be read off the box. Narrow lifts can turn a stop into a pass, and each needs stated evidence, such as a QR count that matches the order on a low-risk part. An override with a written note always lets the packer continue, so the system can block but never leaves a dead end. A local model on a GPU also reads each verification in shadow, and its disagreements go to a human queue.
The verdict engine has reached v2.66, with more than a dozen releases served since August, in one workflow of 72 steps across 5 webhooks.
Results
| What | Value | How it was measured | Tag |
|---|---|---|---|
| Wrong boxes stopped and fixed | 17 | Over 12 weeks, each proved by comparing stopped and corrected photos. A floor | Measured |
| Stops on correctly packed boxes | 89 | From the same review | Measured |
| Time per verify | about 9 seconds | Station log, 35 verifies, one partial day. The wait for a verdict, not the whole order | Measured |
| Model spend per order | about $0.15 | Token usage on 239 retained calls at list rates | Measured |
| First-try pass | 78.5 to 87.5 percent | By month and station, August to October | Measured |
| Verifies per order | 1.15 to 1.29 | Same months and stations | Measured |
How a change earns its way to the floor
A packer asked why the same order sometimes passes and sometimes stops, so I measured it. 41 of 111 retries with byte-identical photos changed verdict, in pairs that come mostly from retries after a bad result. 10 flips were purely about format and 2 partly. A schema fixes format and nothing else, and I said so before building it. Structured output went live on the goods step in November 2026.
Gates are written before each run and don't change after it. One candidate prompt took its target fault from 14 samples to 0, and I rejected it, because a written gate caught a new false stop: a good two-lot order went from 1 hold in 10 to 10 in 10. When a later schema run moved the count on 2 lines, that gate stayed failed as written, and the decision to ship was made with those facts on the table.
Two different candidates caused the same false hold about 9 times in 10, so I tested the rig itself: the live prompt on both arms, with only a code comment added to one. Requests matched byte for byte on 8 of 8, the arms came out about even, and the rejections stood. It cost $3.47.
Engine changes replay against up to 2,830 stored final decisions, which compare old code with new.
Problems it surfaced, and what I did about them
Measuring frame age found a quiet evidence bug on one station: old frames queued behind a network camera, so a photo could show the table up to 7.7 s before the press. On the bench the old reader took 68 of 300 frames and the new one took 300 of 300. Each capture now waits for a newer frame, and after 2 seconds it reports a camera fault.
At least four wrong shipments got through before the watcher existed, three of them after the pipeline had correctly named the problem. That is why a watcher now looks for exactly that pattern. Separately, 7 of 16 overrides overrode a hold, which led me to trace each false-hold class. The 3 of those checked from photos all had correct goods.
A station lock failed a real keystroke test, and I rolled it back on both stations the same night, with the page dark for about 2 minutes after hours. A release kit that rolled back a healthy service, after 31 seconds of startup against a 30-second wait, now waits longer.
Where I would take it next
- Make the lot read blind as well, since the model is still shown the expected lot and can copy it.
- Promote the slip barcode check from logging to deciding, once it reads more slips.
- Bring the shadow reader's human review queue back up to date, since its agreement alone proves nothing.
- Widen live QR scanning, which is limited because most stock has no QR label.
Stack
Claude vision, Python and JavaScript, a small web app on each station, a self-hosted workflow engine, a local open-weights model on a GPU as a shadow reader, and Claude Code for the build.
My role
I designed this system and I own it. I decided what the model may and may not decide, defined the gates before each run, read the failures, and made the release calls. I direct Claude Code agents to write much of the code and run the replays and reviews inside guard rails I set, and I measure drift with replays and live watchers. Ask me why any part works the way it does.
Run the kiosk demo on made-up orders.