A friend once had a fight over text, not because of what she said, but because of the order it arrived in.
She sent "I'm sorry," then "but you were also wrong." The signal was bad, and the second message arrived first.
Same words, different order, and a much worse evening.
I spent days debugging the same fight inside a distributed system. It convinced me that most of us get one of the most common patterns in software quietly wrong: retry is not a loop, it's a data structure. Treat it like a loop and your system will lie to you.
The retry everyone writes
You've written this code: try the thing, wait if it fails, try again with backoff, give up after five attempts. For one isolated operation this is correct, and nobody should talk you into more machinery.
The trouble starts when you have a stream of operations and only some of them fail. While item #3 waits for its second chance, items #4 through #9 sail through and land. Thirty seconds later, #3 succeeds and lands after them.
Your log now reads: 1, 2, 4, 5, 6, 7, 8, 9, 3.
If those items are cache warmups, nobody cares. If they're a timeline, a ledger, a chat, or an audit trail, you've built a machine that rearranges history: quietly, only under failure, and only sometimes. No test will catch it. Production will, months from now, with an effect showing up before its cause, and you staring at the screen the way my friend stared at her phone.
Three questions before you retry anything
These three questions take a minute. They have saved me from at least two designs that would have failed in this slow, silent, unreproducible way.
1. Does order matter?
Would a reader reach a different conclusion if the sequence changed? Then it matters, and anything with the word "history" in it qualifies.
The moment order matters, your retry stops being a loop. A failed item can't rejoin whenever it recovers. Either it holds its place and everyone behind it waits, or you add sequence numbers so readers can reconstruct the truth. Either way you're now managing state and invariants: a data structure.
2. Does duplication matter?
You often can't know that the failed attempt actually failed. Maybe it succeeded and the response died, in which case retrying means doing it twice.
If the operation is idempotent, retry freely. If it appends, increments, or charges a credit card, every retry needs a dedup key on the other side. The loop was never the hard part; the dedup table is.
3. Does staleness matter?
An item that's been sitting in a retry buffer for five minutes is not the item it was five minutes ago. The file changed, the user cancelled, a newer item superseded it.
A naive loop delivers stale news anyway. A correct retry checks at send time whether the item is still true and still wanted. Sometimes the correct retry is a drop, and dropping correctly requires knowing what supersedes what. Again, a data structure.
The line this pattern draws
A retry loop is what you write when you think about the operation. A retry data structure is what you write when you think about the observer: whoever later reads what you produced and tries to reconstruct what happened.
That shift is one of the quiet lines between mid-level and senior systems work. Mid-level code asks "did the call succeed?" Senior code asks "will the story this system tells still be true when the network has a bad day?" Idempotency keys, sequence numbers, write-ahead logs, vector clocks: all of it is machinery built for observers. Retries under failure are where the cheap version and the correct version part ways.
Observer-thinking keeps getting more relevant, too. A whole category of tooling now exists purely to reconstruct stories from event streams. Git AI, for instance, watches AI coding agents work and answers "which lines did the human write, and which did the agent?" That answer is only as trustworthy as the ordering of the events underneath it. If a retried event can land out of order, the story lies.
Humans annotate their retries
Back to the text message. Humans retry all the time: we repeat ourselves, follow up, resend. But notice what we attach when we do:
- "As I mentioned earlier..." -> sequence numbers
- "Sorry if you already got this..." -> idempotency keys
- "This may no longer be relevant, but..." -> TTLs
We instinctively know a bare retry can mislead, so we annotate. A retry without metadata is a rumor; a retry with metadata is a record.
What to do on Monday
This takes about half an hour:
- Grep for
retry,attempt,backoff. There are more than you remember writing. - Answer the three questions for each one, and write the answers as a comment right there.
- Every "yes" without machinery is a scheduled incident. If order matters and there are no sequence numbers, you're waiting for the failure, not preventing it.
- Every "no": leave the loop alone. Simple retries for independent, idempotent, evergreen work are correct. The skill isn't building the data structure everywhere. It's knowing which retries need one.
Most "weird intermittent data issues" aren't weird and aren't intermittent. They're retries doing exactly what they were told: try again, whenever, in whatever order, with no memory of what the world looked like the first time.
When failure enters the picture, again is not a repetition. It's a new event. Treat it like one.
Ever debugged a timeline where the effect showed up before its cause? I collect these. Find me on X or LinkedIn.