Two clinicians opened the same patient record, both edited the medication list, and both hit save within about a second of each other. One of the two edits silently vanished. No error, no warning, nothing in the logs. The database did exactly what we told it to. The problem was that we'd told it something dumb.
The lost update
The shape of this bug is everywhere once you know to look for it. You read a row, change it in application code, and write it back. Request A reads version 5. Request B reads version 5. Request A writes its change. Request B writes over it. Last write wins, and “wins” is carrying a lot of weight in that sentence, because A's edit is just gone.
Under light traffic you will never see this. The window between read and write is a few milliseconds, and two people rarely hit it at once. Under load, that window is a coin flip you keep losing, and the symptom is the worst kind: not a crash, just data that's quietly wrong.
Optimistic concurrency: assume you're fine, then check
The cheapest fix that actually holds is optimistic concurrency. Add a version column. Read it along with the row. When you write, only write if the version is still what you read, and bump it in the same statement:
UPDATE records
SET data = $1, version = version + 1
WHERE id = $2 AND version = $3;
-- 0 rows affected → someone changed it under you → re-read and retry, or rejectIf the update touches zero rows, somebody beat you to it, and now you get to decide what that means: re-read and reapply, or tell the user their view was stale. Either way, nothing gets silently clobbered. No locks, no held connections, and it scales, because in the common case there's no conflict and you pay almost nothing.
When optimistic isn't enough
Sometimes one atomic UPDATE can't express what you need. The operation spans multiple systems, or several writes that have to happen together, and you genuinely need mutual exclusion across services. That's when a distributed lock earns its place. In Redis it's a key you set only if it doesn't exist, with an expiry so a dead holder can't block everyone forever:
SET lock:invoice:4123 <token> NX PX 5000 // acquire, auto-expire after 5s
// ... do the protected work ...
// release ONLY if we still hold it (compare token), never a blind DELA lock is a loaded gun
Here's the part people skip when they excitedly add a lock. A distributed lock is itself a distributed systems problem. What if the holder crashes? The expiry saves you, but now your work can outlive the lock, and a second worker can start while the first is still going. What if the work takes longer than the expiry? Same thing, two holders. And if you release with a blind DEL, you might delete a lock that someone else already acquired. That is the whole Redlock argument in miniature, and the practical takeaway is simple: reach for optimistic concurrency first, use a lock only when you truly can't model the operation as one atomic write, and keep the critical section short.
What we shipped
Most of our races were single-row and disappeared the moment we added a version column. A smaller set genuinely needed a lock, the cross-service operations that touched more than one system at once. The real work was telling those two cases apart, not the code for either. The races went away, and more importantly, they stopped going away silently.
A lost update has no stack trace and no error. Just data that's wrong. Make your writes assert what they expected to be true, and let them fail loudly when it isn't.