09 · Run & release · The database at peak
While a report holds the lock, the dock is still waiting to receive
Database operations watches backups, slow queries, replica lag, and locks, so receiving at peak is not killed by one report. How tables are designed and how constraints are added is the database engineer’s work.
Database operations guards the production database by day and by night
The database engineer designs constraints and migrations. The mark of database operations is watching in production whether those designs get hurt at peak: slow queries, lock waits, replica lag, disk, whether the backup actually finished. Before morning peak he looks at last night’s backup. He does not run heavy reports at peak. Large queries the business wants go to a replica or a trough. When replica lag exceeds the threshold, the report must say it is not real-time. It cannot be used to judge whether the receiving document just now succeeded.
Change scripts are run by him in a window. Before they run there is a backup or a statement that can be rolled back. He refuses “try adding an index on production first.” Even a trial statement goes through release. If an emergency index truly must be added, it is logged as a change, and he watches whether writes slowed. Slow receiving makes people tap again.
Accounts and restore
He runs restore drills with the operations engineer, checking whether transactions are complete, not whether a file can be unpacked. Personal accounts, least privilege, and reclaim when someone leaves apply here too. When the application account password rotates, he changes config with DevOps. The new password is not sent in a group chat.
Replica lag treated as stock
Someone pointed available-to-sell queries at a replica. With two minutes of lag, goods just received were still invisible. The warehouse clerk thought they had not arrived and received them again. The primary already had them. Database operations pointed available-to-sell back at the primary. Reports stayed on the replica and showed the lag. At peak he killed a full-table reconciliation and moved it to night. That day a report did not lock receiving transactions.
He leaves the waiting statement with development as evidence of a missing index or a bad query, instead of only restarting on the floor. Before peak starts, database operations confirms there is no unfinished statistics job. After peak he looks at the slow-query ranking. If first place is a report, he drives it to the replica. If it is receiving, he hands it to development. He does not try a pile of indexes on production first. A tried index may slow writes. The dock feels it first. A primary/replica failover is drilled at least once a year. After the switch the application must point at the new primary, and the queue must not drop. Writing “can fail over” in a document and never switching means everyone meets the procedure for the first time on the day of the fault. Every day he glances at whether yesterday’s ten heaviest statements changed. A stable ranking means load is under control. A sudden full-table scan is usually a new feature or a new report going live.
After finding it he first blocks retries so the floor does not keep tapping, then hands the statement and the plan to development. Repeatedly tapping the same heavy statement fills database CPU and times out innocent writes with it. Archive windows are scheduled from the migration duration the database engineer gave. This role does not change constraint design on production. Backing up only the live database and not history breaks audits of old documents. He plots slow queries against the shift’s business peak on one chart. Overlaying them shows whether that statement slowed when writes were densest. When replica lag exceeds what the report labels, the report is degraded first. It is no longer labelled real-time. How tables get constraints, and how a request ID stays unique, is still the database engineer’s design. This role only keeps those designs alive at peak. When connections fill, limit report sessions first. Do not kill receiving sessions first. If the backup window hits day-end close, move the backup to half an hour after close. Do not change day-end close to lock-free on production. Archive windows follow migration duration, not feel.