06 · Development · Eval before posting
If the cost of one error cannot be stated, do not fully automate
The algorithm engineer puts recognition and recommendation where they can be reversed. The model may give a suggestion. Before a person confirms, it is not a fact, and it cannot directly change stock rules.
Eval set: Stratify by supplier layout. Do not treat fonts from the web as on-site dot-matrix print.
Cost: How many documents a wrong once must correct, and whether it changes the books, written as numbers, not as a feeling.
Version: Every confirmed document can go back to the model version of that time, or it cannot be replayed.
The algorithm engineer owns eval design
Recognizing delivery notes, recommending locations, forecasting orders: these capabilities are easy to call automation. The algorithm engineer’s mark is not putting a prefill on the page. It is first defining what happens when it is wrong, which set proves it, and what one error is worth. Output is a suggestion. Fact comes from the confirm action. Automatic posting with no confirm action must be reviewed on its own, and must have a closed loop of automatic correction. If the loop cannot be built, do not auto-post. Prefill on the page and compare to the original image is implemented by frontend. When the books stand is decided by the Backend engineer’s transaction.
The confidence threshold is calculated from historical documents’ error cost, not from a feeling. Below the threshold goes to a human queue. Backlog count and wait time must be visible. Hundreds in the queue means the threshold or the people do not match. You cannot clear the queue by raising auto-pass rate. While auto-pass rate rises, sample error rate must stay visible. Human-corrected results must feed the next eval. They cannot be dropped as anomalies.
Samples, replay, and the trigger to re-eval
Training samples come from the customer’s real documents, used after de-identification. Before go-live, replay the last month’s documents and report which layouts fail completely. Completely failed layouts go to humans, not into the model. When a layout changes, there must be a trigger to re-eval. The model cannot stay forever at the accuracy of acceptance day. Recommended locations cannot overwrite a putaway a person already confirmed. The floor is the same as recognition.
The cost function stopped one full automation
The business asked that recognized delivery qty no longer be seen by a person. The algorithm engineer first delivered the eval: error rate stratified by supplier, how many documents each error class must correct, and whether one error changes stock. One supplier’s layout stably read 10 as 18. Errors clustered, not even. Overall accuracy looked as if it could pass the line. Cost calculated out to correcting a batch of already-confirmed documents. He refused full automation, allowed only prefill, and stopped auto-pass for this layout. Model version is written into the confirm record. Later, to replay a version, find those documents by version, not rummage the whole database.
A recognition result with no version is, to the algorithm engineer, not delivered. If eval set, cost table, and version number are not all three present, this role’s work is still at demo accuracy.
Recommended location and recognition are not the same eval. A wrong recognition changes qty. A wrong recommendation must not let the model overwrite a putaway a person already confirmed. The recommendation cost function looks at “after a person corrected, did the model drop the change as an anomaly.” If it dropped it, the next eval set becomes more fake. The algorithm engineer forces human-corrected samples to stay in the eval set, and reports stratified by supplier and by shift. If one shift’s errors cluster, stop auto-pass for that shift first. Do not adjust overall accuracy to hide it. Handoff to backend: model version writes into the confirm record. When stock transactions stand is still decided by the server. Without these three columns — predicted, human value, finally adopted — recommendation and recognition are still only a demo.
Fonts casually found on the web cannot pretend to be on-site dot-matrix print in the eval set. When a layout changes, there must be a re-eval trigger. It cannot stay forever at the accuracy of acceptance day. Completely failed layouts go to humans, not into the model. If automatic posting cannot close a correction loop, do not do it.