09 · Run & release · Grey release
Switch eight warehouses together and rollback does not know whom to save first
The release manager decides who uses the new version first and how to roll back, and forbids eight warehouses changing in the same minute. How the pipeline packs sits with the DevOps engineer.
All at once: At ten at night every warehouse switches together. A problem means rolling everything back, after the floor has already received goods for an hour.
One warehouse first: One warehouse runs through morning peak, then widen. Rollback only hits that warehouse. The artifact number is in the notice.
The release manager owns the window and the rollback
Test sign-off says a version may be a candidate. The DevOps engineer supplies an artifact with a build number. The release manager decides when, for which warehouses, in what order to open, and which signal triggers rollback. The mark of the role is refusing “switch everyone tonight.” The rollback cost of switching everyone is every in-flight document across all warehouses. He first picks a warehouse that is representative and not the only peak, runs a full shift, then widens.
Release notes use business language: this version fixed duplicate receiving; it does not include a new export. The note has the build number, start time, observer, and rollback owner. A window with no rollback owner does not open. Rollback has been drilled. It is not the sentence “roll back if needed.”
And the watch metrics
For thirty minutes after opening, watch SRE business errors and pending write-back, not only whether the service started. Started but double-posted: roll back. After rollback he confirms the next shift is on the old build number, and tells support how to explain documents already produced. Documents that already double-posted go on a repair list. Do not pretend rollback automatically wipes the books.
The lesson of eight warehouses at once
Once eight warehouses switched in the same minute. One warehouse’s printer template was incompatible. Receiving still worked; the slip would not print. Because it was a full release, rollback would hit the seven warehouses that were already fine. After that the release manager put floor dependencies such as printing on the lead-warehouse checklist. The lead warehouse must print one real delivery note before a second warehouse is allowed. That day’s template problem was found in the lead warehouse. The others stayed on the old version until the next day. The blast radius was one warehouse, one shift.
The decision to widen is written down. Saying it once in a group chat does not count. Before widening, the release manager checks print, write-back, and privilege in the lead warehouse, not only whether the service started. Started while the slip will not print is a failure for the warehouse. The list of warehouses to widen is published by him. Nobody else can add an eighth warehouse in the group. A warehouse added that way has no rollback owner. Emergency fixes also go through the lead warehouse. Only the window can be shortened. A reason to skip the lead warehouse must be written: not fixing it will stop the line, and the risk is smaller than an all-warehouse fault. If those two sentences cannot be written, it is not an emergency. It is hurry. On release day he puts the business contact, lead warehouse, rollback owner, and artifact number on one page and sends it to support and implementation. Support that does not know the artifact number cannot tell whether a floor problem belongs to the new version or the old. Implementation that does not know the lead warehouse will blame other warehouses’ exceptions on the release.
Missing one of those facts, the first hour after release is spent confirming who is on which version. He also writes “do not release” as a decision: this version stays in test, not production, because lead-warehouse printing did not pass. Write the non-release in the open, or the team will quietly replace files at night. One quiet replace, and the grey list never means anything again. He hard-writes the gap between widenings so it covers at least one full shift. Compress the gap to ten minutes, and the lead warehouse has not had time to show write-back delay. Once that delay reaches a second warehouse, rollback involves two batches of in-flight documents.
Release notes use business language for build number, start time, observer, and rollback owner. A window with no rollback owner does not open. Rollback has been drilled. It is not “roll back if needed.” Wrong documents already produced go on a repair list. Do not pretend rollback automatically wipes them. “Do not release” is also a written decision: this version stays in test because lead-warehouse printing did not pass. Write that in the open so the team does not quietly replace at night. Which pipeline the artifact came from is DevOps. Whether the business is hurt after opening is SRE. This role only sequences the window and the rollback.