Your idempotency test probably cannot fail

I had a test called "stays quiet when the plan is already what the event grants". It passed on every run for two weeks.

Then two identical "payment received" messages arrived on my phone, 142 milliseconds apart.

The test was not weak. It could not have failed. And that is a more interesting problem than a missing test, because a missing test at least looks like a gap — this one looked like proof.

What the code did

Paddle sends subscription.created and subscription.activated for a single purchase, and it retries anything that does not answer 200. So two deliveries for one payment is not an edge case, it is Tuesday.

I knew that. I had written the dedupe:

const { data: profile } = await supabase
  .from('profiles')
  .select('plan')
  .eq('id', userId)
  .single()

const alreadyOnPlan = profile?.plan === plan

await supabase.from('profiles').update({ plan }).eq('id', userId)

if (!alreadyOnPlan) await notifyPayment({ userId, plan, amount })

Read the current plan, compare, write, announce only if it changed. It reads correctly. It is correct, for one caller.

Two deliveries arrive. Both read free. Both see alreadyOnPlan === false. Both write basic. Both announce.

The window between the read and the write is where the whole bug lives, and it is exactly as wide as one round trip to the database.

Why the test agreed with me

Here is what I had written:

it('stays quiet when the plan is already what the event grants', async () => {
  db.setRow('profiles', { id: 'user-1', plan: 'basic' })
  await post(activation())
  expect(notifyPayment).not.toHaveBeenCalled()
})

Read it again with the bug in mind. It calls the handler once, against a row that is already in the target state. It asks: if I run this after the plan is already set, does it stay quiet?

That is a real question. It is not the question the bug is about.

A race requires two things to overlap. await post(...) runs one handler to completion before the next line executes, so the two deliveries never coexist. I could have added a second call, a third, a hundred — sequentially they would all pass, forever, while production kept sending doubles.

This is the part worth taking away. My test was not a bad test of concurrency. It was a test of something else that I had filed under concurrency, and the passing green tick was doing active harm: it told me the case was covered.

Making it fail first

The fix in the product code is small, and I will get to it. But the test had to fail before the fix, or I would have no evidence the fix did anything.

Two handlers have to overlap. In JavaScript that does not need threads — it needs the first handler to yield at an await while the second one starts:

it('announces once when two deliveries race each other', async () => {
  db.setRow('profiles', { id: 'user-1', plan: 'free' })

  await Promise.all([
    post(activation()),
    post(activation('subscription.created')),
  ])

  expect(notifyPayment).toHaveBeenCalledTimes(1)
})

Promise.all starts both, and the first one suspends at its first await — the read. The second handler runs its own read against a row nobody has written yet. That is the production interleaving, reproduced deterministically, in a unit test, with no timing hacks.

Run that against the old code and it fails: two announcements. Which is what a test is for.

The part I did not expect

It did not fail. It passed against the broken code too.

The mock database ignored the condition I was about to rely on. It recorded that an update happened and returned success; whether the row actually matched was not something it modelled. So a conditional write and an unconditional one produced identical results, and no test on earth could tell them apart.

I want to be precise about how bad this is. A missing test leaves a known hole. A test double that quietly simplifies the thing you are testing produces confident wrong answers, and it produces them in the exact area you thought you had covered. It is the same failure mode as the original bug, one level up.

So the mock had to learn the one behaviour that matters here: a conditional update is a check and a write in a single step, and the row is changed before anyone else can read it.

if (pendingWrite === 'update' && notEquals.length) {
  const current = state.rows[table]
  const blocked = notEquals.some(([col, val]) => current?.[col] === val)
  if (blocked) return { data: [], error: null }
  if (current && writeValues) state.rows[table] = { ...current, ...writeValues }
}

Now the second caller in the Promise.all sees what the first one wrote. Now the test fails on the old code and passes on the new one, which is the only property that makes a test worth keeping.

The fix

Once check-and-act has to be one step, the shape of the answer is forced. The condition moves out of the process and into the write:

async function setPlan(userId: string, plan: Plan): Promise<{ changed: boolean }> {
  const { data, error } = await supabaseAdmin
    .from('profiles')
    .update({ plan })
    .eq('id', userId)
    .neq('plan', plan)          // ← the whole fix
    .select('id')

  if (error) throw new Error(`Failed to update plan: ${error.message}`)
  return { changed: Array.isArray(data) ? data.length > 0 : Boolean(data) }
}

Postgres decides who was first. The update that actually matched a row returns it; the loser gets an empty list. Only the winner announces.

Note what did not change: both deliveries still attempt the write. That matters — if the first one's write is lost to a network failure, the second one heals it. A dedupe that skips the write on the second delivery would turn a lost message into a lost purchase.

Three questions I now ask my tests

Can this test fail? Not "does it pass" — can I write a version of the product code, plausible enough that I might have written it, that this test does not catch? If the answer is no, the test is decoration. Deleting the fix and watching the test go red takes ten seconds and is the only way to know.

Does the shape of the test match the shape of the bug? Races need overlap. Retries need repetition. Ordering bugs need the wrong order. A sequential test cannot express a race, in the same way a unit test cannot express a deployment problem — not "is unlikely to catch", cannot express.

Does my test double model the property I am relying on? I relied on a conditional write. My mock did not have conditions. Every assertion built on top of that was measuring the mock, not the code. If the fix depends on a database guarantee, the double has to implement that guarantee or the test is theatre.

Why this keeps happening

I write this product alongside an AI, and this bug is a good example of what that changes and what it does not.

The dedupe it wrote was reasonable. Read, compare, write — that is what the task sounded like, and it is what most people write by hand too. The failure needed context nobody stated: that this specific provider fans one purchase into two events and retries on top of that. Nothing in the code says so. It is knowledge about the outside world, and it is exactly the kind of thing that will not be in the diff.

The test was mine. I wrote it to feel covered, and it worked — I felt covered for two weeks, until my phone buzzed twice.

The lesson is not "review AI code more carefully". It is narrower and more useful: a passing test is a claim, and claims about concurrency made by sequential code are worth nothing. That was true before any of this, and it will be true after.