The Day Deposits Doubled: Idempotency, Race Conditions, and a Lock That Saved Us

Published at Jun 24, 2026

#idempotency#concurrency#race-condition#postgres#csharp#payments#production

Confidentiality note: I can’t disclose the real company or provider involved. The numbers, names, and product are fictional. Every technical decision, failure mode, and fix described here is real and was applied in production.

There’s a specific kind of silence that falls over a team when money is involved.

Not the calm kind. The kind where everyone is staring at the same dashboard, the count is going up, and nobody is typing yet because nobody wants to be the one who makes it worse.

This is the story of that day — the day our players started receiving duplicated deposits — and the small, almost boring change that stopped it: stop checking before you insert. Insert first, then check, under a lock.

1) The setup

I’m a tech lead on a team that integrates with external payment providers. Player deposits money, the provider processes it, the provider calls us back, we credit the player’s wallet. Standard stuff.

The flow looked like this:

Player → Provider → [callback] → Our API → Credit wallet → Done

The provider sends us a callback (a webhook) for every successful payment. That callback carries a transactionId — a unique identifier for that specific payment. Our job: receive it, credit the balance, exactly once.

And we knew callbacks could be retried. Networks fail, timeouts happen, providers re-send. So from day one we had an idempotency check in place. We weren’t naive about it.

Or so we thought.

2) The morning it broke

The provider shipped a deploy to production. We didn’t know that yet.

Their deploy had a bug: every callback was being sent twice. Not “occasionally retried” — every single payment fired two near-simultaneous callbacks at us, milliseconds apart.

Our idempotency check was built for sequential retries. It was never built for two identical requests arriving at the same time.

The dashboard started ticking. Players were getting credited twice. Real balances, real money, climbing in real time.

The client noticed before our alerts did. That’s the worst way to find out.

3) The war room

Within minutes I was in a room — physical and virtual — surrounded by half the engineering team and the PM. Everyone talking at once.

“Pause the consumer!” “No, we’ll lose callbacks — we don’t know if the provider replays.” “Add a distributed lock in Redis.” “Just put a lock around the method.” “Roll back our last deploy” — except the bug wasn’t ours. “Reconcile afterwards and claw the money back.”

Every idea had a grain of truth and a sharp edge. We couldn’t afford a wrong move — each mistake here had a currency sign in front of it.

So I did the thing that actually works in those moments: I stopped talking and ran the code in my head. I needed to see the exact instruction where two requests collided.

4) Why our idempotency check wasn’t enough

Here’s what the original handler looked like, simplified:

public async Task HandleAsync(DepositCallback callback)
{
    // 1. Has this transaction already been processed?
    var existing = await _db.Deposits
        .FirstOrDefaultAsync(d => d.TransactionId == callback.TransactionId);

    if (existing is not null)
        return; // already processed, ignore

    // 2. Process it
    _db.Deposits.Add(new Deposit
    {
        TransactionId = callback.TransactionId,
        PlayerId = callback.PlayerId,
        Amount = callback.Amount,
    });

    await _db.Wallets.CreditAsync(callback.PlayerId, callback.Amount);
    await _db.SaveChangesAsync();
}

Read it slowly. It looks correct. It is correct — for one request at a time.

This is a textbook check-then-act race condition (a TOCTOU bug — time-of-check to time-of-use). Lay two concurrent callbacks over the same timeline:

 time │  Request A                  Request B
──────┼──────────────────────────────────────────────────
  t0  │  SELECT ... → null
  t1  │                             SELECT ... → null
  t2  │  INSERT deposit
  t3  │                             INSERT deposit
  t4  │  CREDIT wallet  (+100)
  t5  │                             CREDIT wallet  (+100)

Both requests check at t0/t1. Neither has inserted yet. Both see null. Both conclude “fresh transaction.” Both credit the wallet.

The idempotency check was guarding a door that both requests walked through before either one closed it. The window is tiny — a few milliseconds. Under normal sequential retries you’d basically never hit it. Under a provider sending perfectly-paired duplicate callbacks, you hit it every single time.

No amount of “check harder” fixes this. The check itself is the problem. You can’t ask “does it exist?” and then create it as two separate steps when other writers are racing you.

5) The idea: insert first, then check — under a lock

Standing there debugging in my head, I took the good parts of what people were shouting:

  • The Redis-lock crowd was right that we needed serialization — collisions had to be forced into a line.
  • The “use the database” instinct was right that the lock should live where the data lives, not in a separate system that can drift out of sync.
  • The reconcile crowd was right that we needed a single source of truth for “was this processed.”

The database already had all three. We just had to use it correctly.

The fix is to invert the order. Don’t check then write. Write first — and let the database’s atomicity be the gatekeeper — then check under a lock.

Two pieces make it work.

Piece one: a unique constraint that makes duplicates impossible

First, the database must physically refuse a second row for the same transaction:

ALTER TABLE deposits
    ADD CONSTRAINT uq_deposits_transaction_id UNIQUE (transaction_id);

This is the non-negotiable foundation. With this in place, two concurrent inserts for the same transaction_id can no longer both succeed. One wins; the other gets a unique-violation error. The race at t2/t3 is now decided by the database engine, atomically, instead of by application code that already made up its mind at t0.

Piece two: insert-then-lock, so the crediting is serialized too

A unique constraint stops duplicate rows. But we also credit a wallet — that side effect still needs to happen exactly once. So we insert the deposit row in a Pending state, then take a row lock on it (SELECT ... FOR UPDATE) and decide what to do based on its real, committed state.

public async Task HandleAsync(DepositCallback callback)
{
    await using var tx = await _db.Database.BeginTransactionAsync();

    try
    {
        // 1. INSERT FIRST. The unique constraint is the gatekeeper.
        _db.Deposits.Add(new Deposit
        {
            TransactionId = callback.TransactionId,
            PlayerId      = callback.PlayerId,
            Amount        = callback.Amount,
            Status        = DepositStatus.Pending,
        });

        await _db.SaveChangesAsync(); // throws on duplicate transaction_id
    }
    catch (DbUpdateException ex) when (ex.IsUniqueViolation())
    {
        // A row already exists → this is a duplicate callback. Stop.
        await tx.RollbackAsync();
        return;
    }

    // 2. Lock the row we just created and re-read its committed state.
    var deposit = await _db.Deposits
        .FromSqlInterpolated($@"
            SELECT * FROM deposits
            WHERE transaction_id = {callback.TransactionId}
            FOR UPDATE")
        .SingleAsync();

    // 3. Only credit if it hasn't been credited yet.
    if (deposit.Status == DepositStatus.Pending)
    {
        await _db.Wallets.CreditAsync(callback.PlayerId, callback.Amount);
        deposit.Status = DepositStatus.Completed;
        await _db.SaveChangesAsync();
    }

    await tx.CommitAsync();
}

Walk the same two concurrent requests through it now:

 time │  Request A                       Request B
──────┼──────────────────────────────────────────────────────────
  t0  │  INSERT (Pending) → OK
  t1  │                                  INSERT (Pending) → 💥 UNIQUE VIOLATION
  t2  │  SELECT ... FOR UPDATE (locks)   rollback + return
  t3  │  status == Pending → CREDIT
  t4  │  status = Completed, COMMIT

Request B never even reaches the wallet. It’s rejected at the door — by the database, atomically — the instant it tries to insert. There is no window left to race through, because the decision is now a single atomic operation instead of two separate ones.

6) Why the lock still matters

You might ask: if the unique constraint already blocks duplicate rows, why bother with FOR UPDATE?

Because the constraint protects the row. The lock protects the side effect.

Consider a different concurrency path: a duplicate that arrives after the first insert commits but while the first request is still mid-crediting. Or a retry that shows up minutes later, after a crash, when the row exists but its status is still Pending because the original process died before finishing.

SELECT ... FOR UPDATE forces every handler touching that transaction into a single line. Whoever holds the lock reads the true committed status, acts on it, and updates it before releasing. The next one in line reads the updated status — Completed — and correctly does nothing.

Unique constraint = “there is exactly one record.” Row lock = “exactly one process acts on it at a time.”

You need both. One guarantees identity; the other guarantees serialized execution of the money-moving part.

7) Don’t trust me — run it yourself

Reading about a race condition is one thing. Watching a wallet balance double on your own machine is another.

I put together a small, self-contained .NET 8 + Postgres project that reproduces this exact incident and the fix. It exposes both handlers side by side:

  • POST /deposits/buggy — the original check-then-act guard, no unique constraint
  • POST /deposits/safe — insert-first, then SELECT ... FOR UPDATE

A stress test fires 200 transactions, each as a pair of identical concurrent callbacks — 400 requests — at each endpoint, exactly like the provider’s double-send bug. Then it checks the wallet.

Everything runs in Docker, so there’s nothing to install but Docker itself:

git clone https://github.com/Thyago-Oliveira-Perez/idempotent-deposits
cd idempotent-deposits
docker compose --profile test up --build --abort-on-container-exit

The output speaks for itself:

[safe]  honest=2000 balance=2000.00 rows=200   ✅ credited exactly once
[buggy] honest=2000 balance=3960.00 rows=396   ❌ nearly double

Same load, same concurrency, same database. The only difference is the order of two operations and one row lock. That’s the whole lesson, made tangible.

The repo has the full source, the stress suite, and a README that walks through reproducing it by hand with curl.

8) The cleanup

The fix stopped the bleeding immediately. But we still had hours of duplicated deposits sitting in production.

That part wasn’t clever engineering — it was careful accounting:

  • We queried every wallet credited more than once for the same logical payment.
  • We built a reconciliation report and reversed the duplicate credits in a single audited batch.
  • We notified the provider, who confirmed (and rolled back) their double-send bug.
  • We added an alert on duplicate-rate per transaction so the database itself would scream long before a client noticed again.

The provider’s bug was out of our control. Our handler’s fragility was not. We owned our half.

9) What I took away from it

Idempotency is not a flag you tick. It’s a property you have to prove under concurrency. “We have an idempotency check” meant nothing the moment two requests arrived at once. The check existed; the guarantee didn’t.

Check-then-act is a lie when other writers exist. Any time you read state, make a decision, then write based on that decision in separate steps, you have a race. The fix is almost always to collapse the read-decide-write into one atomic operation — and the database already gives you the tools: unique constraints and row locks.

Put the lock where the data is. A Redis lock would have worked too, but it introduces a second system that can fail, drift, or expire independently of your data. The deposit lives in Postgres; the truth about whether it was processed should live there too.

In a war room, debug, don’t shout. The best ideas were already in the room — scattered across five people talking over each other. My job as tech lead wasn’t to out-shout them. It was to go quiet, trace the exact line where the race happened, and assemble the good fragments into something that actually held.

The whole fix was, in the end, a reordering and a lock. Insert first. Then check. Hold the row while you do the thing that matters.

Boring. Bulletproof. Exactly what you want standing between two requests and someone’s money.