09 · Run & release · Restore
A backup that has never been restored does not count as a backup
The operations engineer owns backups, restores, disks, and account expiry, so a broken system can come back by steps already drilled. Receiving failures and similar business alerts go to SRE.
The operations engineer owns “it can come back”
After the app is live, machines, disks, certificates, backups, and accounts still need an owner. The mark of this role is drill instead of assumption. A backup job that shows success does not mean yesterday’s stock can be restored. He regularly restores backups into an isolated environment, checks document counts, and records how long it took. That time decides how long the business must be told to stop when something breaks. Without that time, the incident meeting can only say “we are working on it.”
He also owns capacity. When photos and attachments fill a disk, receiving fails. Monitoring must warn before the disk is full, and must know which directories can be cleaned and which are business attachments that must not be deleted. Deleting attachments is deleting evidence. Certificates, domains, and the accounts and passwords for access-control interfaces all sit on a calendar. They do not rely on one person’s memory.
Change and privilege
Production changes go through the release manager’s window. The operations engineer does not hand-edit configuration at business peak. Places that were hand-edited go into a record, so the next release does not overwrite them while nobody knows. Server login uses personal accounts. Shared root passwords must be retired. Accounts left by implementation expire under him; he closes them and checks once against the security architect’s list.
Photos on the disk
Original images of delivery notes live on the application disk. A month after go-live the disk alert was ignored until writes failed and the receiving page errored. The operations engineer first cleaned installer packages that were not business data and expanded capacity, then put the original-image directory onto capacity alerts, with the threshold moved up to seventy percent. A later restore drill proved the database could return to the point before the failure, and the photo directory was in the backup too. A plan that backed up only the database and not the originals was changed. Otherwise there is nothing to compare when a recognition error is corrected.
Drill records go to SRE and the project manager. Restore is what makes an availability promise possible. The operations engineer puts certificates, domains, and backup drills on the same calendar, with an alert two weeks ahead. A certificate found only on the day it expires can only be replaced by stopping the business. He writes the last restore duration next to the calendar so the business knows roughly how long a restore will stop them. A restore of unknown length is something the warehouse cannot plan holding space for. Log disks and backup disks do not sit on the same volume. When that volume fills, the common fix is to delete logs to keep the backup, and then the cause of the incident is gone too. He would rather expand early than let restore ability and investigation ability fight for space. He checks alert contacts by shift. A midnight disk alert that only goes to an implementer who already left means even a complete backup has nobody to act. The person the alert reaches must be able to execute. Reaching a manager is not enough. A manager who sees it and then forwards it to a group has already missed the cleanup window.
He picks a business trough for restore drills, but not a perfectly prepared afternoon, and he leaves a document-count comparison before and after the restore. If the counts do not match, the restore is not done; it only means the backup file was unpacked once. Steps are written as a sequence anyone can run. Who gets the midnight alert, and whether the first action is a reboot, sits in SRE’s on-call notes. This role guarantees the backup can actually be unpacked, and that disks and certificates are still on the calendar. Database backups and original-image object storage must return to the same business day. A database back to yesterday with photos still on last week leaves nothing to compare when correcting recognition. After a bastion session times out, unfinished restore steps must continue from the drill record. They cannot depend on the on-call person remembering where they stopped. Restore steps must write where the account is taken from. They cannot write “use the old password.”