09 · Run & release · Business signals
The machine is still alive. The write-back queue has been blocked all morning
SRE alerts on signals the business can feel, such as receiving failures and write-back backlog. A live machine is not availability. The first action is not to restart every service.
SRE defines an alert as the business already being hurt
The operations engineer guarantees machines and backups can be restored. The mark of SRE is choosing a few metrics that actually mean business failure: receiving error rate, pending write-back backlog, login failures, disk about to fill. Metrics need thresholds and an owner. CPU that spikes now and then, if receiving is fine, should not wake anyone. After a few irrelevant wake-ups, nobody looks when the queue is actually blocked.
Every alert writes the first action. For pending write-back backlog, first see whether it is the network, the other side closing the books, or a program error — not restarting every service. A restart turns in-flight requests into unknown. SRE writes that action with the integration engineer. An alert with no first action is only a message.
Error budget is spent on releases
If receiving failures in a week already exceed the agreement, SRE can ask to pause non-urgent releases. This is not a fight over authority. Continuous releases make it impossible to tell which version introduced the problem. He splits metrics by release number so they can be compared. User-side incident times must line up with alert times. If they do not, the alert is not the thing users actually feel, and the metric needs to change.
The queue was blocked all morning
Kingdee maintenance made every write-back time out. Application-server health checks stayed green. Warehouse clerks only saw documents stuck in pending write-back. SRE turned backlog count and the age of the oldest item into alerts, and after fifteen minutes notified integration and finance. That day the alert fired soon after maintenance started. Support followed the playbook and told the warehouse: receiving can continue, vouchers will be delayed, do not tap write-back again. When maintenance ended the queue drained. There were no duplicate drafts. Watching only the homepage health check, this morning would have been found at reconciliation.
The playbook goes into on-call notes, next to operations restore drills. Each quarter SRE deletes alerts that have never triggered an action. Before deleting he asks: if this fires tonight, who does what. If nobody can answer, delete it. Remaining alerts must have a first action in the on-call notes, and that first action is not restarting every service. False positives are logged with a reason. Unhandled false positives only push the threshold higher, and a real backlog is swallowed too.
He overlays alerts with build numbers. If business metrics have not moved but the error rate rose, first check the statistic definition. Do not wake the night shift to restart. If the step users report most has no matching metric, either add a metric or admit that step is unmonitored. Clustering of reports points at a collection blind spot, not at support attitude. After an alert recovers, write which action actually worked. If that cannot be written, the next time will still try from the start through round three. Whether a backup can be unpacked is an operations drill. This role owns who does what when the business is already hurt. The first person on the roster must be able to say who to call first. Whether duplicate drafts appear is still proved by integration with an idempotency key. Backlog age and backlog count need separate alerts. A small count whose oldest item is already four hours old may have already missed the reconciliation window. On-call notes must say: call integration first, or tell finance the books-close time first. Get the order of those two calls wrong, and the warehouse will keep tapping write-back. The other system’s maintenance window must be on the calendar in advance. On a maintenance day the backlog alert will fire. The playbook must say “expected — do not restart.” Without that line, on-call will treat it as a fault and scramble the queue.