The receipt is what makes a retry safe
A vendor sends us webhooks when inventory changes. The obvious design is to receive each event and write it down, and that design is wrong in a way that takes months to notice.
It is wrong because delivery is not guaranteed and failure is not loud. A webhook that never arrives raises nothing. No error, no alert, no retry you can see. The database simply drifts away from the freezer, and the first symptom is a scientist being told a sample exists when it does not. A system that is wrong but confident is worse than one that is visibly down, because nobody goes looking.
So the receiver does almost nothing. It accepts the payload, puts it on a queue, and returns. No parsing, no business logic, no validation that could reject something surprising. Everything that could go wrong has moved out of the one path we do not control.
Reconciliation is what makes completeness a property
The real work is a scheduled job that walks the vendor's API over a look-back window and compares it against what we hold. Anything missing gets pulled in.
This is the part that changes the guarantee. With only webhooks, completeness is a hope. With reconciliation, the steady state is correct even if every webhook in a window is lost, and losing an event stops being data loss and becomes latency.
A second scheduled job asks a different question. Reconciliation asks whether anything is missing. Change detection asks whether anything I already hold has been edited since. Those have different look-back windows and different costs, and one job doing both does neither well.
Reconciliation enqueues into the same state a webhook does, deliberately. Two paths that both write records have to agree with each other forever. One path that two things can feed does not.
The receipt
Every event gets a row recording where it is: queued, processing, or finished. That row is what makes reprocessing safe. The same event arriving twice finds its own receipt instead of creating a second sample.
The interesting case is a crash halfway through.
The queue makes a message visible again after a visibility timeout, so the event comes
back. Meanwhile the receipt is still sitting in processing, because the
process that was working on it died without finishing. If the retry only matches receipts
that are untouched, it sees work that looks claimed, assumes someone else has it, and
drops the message. Silently. Again.
So the match has to include processing, not just queued.
Recognising a half-finished receipt as resumable is the whole thing, and it is the kind
of detail that is obvious once written down and invisible in a design review.
What I would tell someone starting this
The instinct is to make the receiver clever. Validate harder, deduplicate, add retries. That is effort spent in the one place that cannot be made reliable, because you do not control the sender.
Put the cleverness in reconciliation. Keep the receiver boring. Give every event a receipt, and be careful about what a retry is allowed to assume.
And archive the raw payload before you interpret it. Parsing logic is the thing most likely to be wrong months later; the raw event is the thing least likely to be. Ours goes to object storage untouched, and there is an endpoint that feeds it back through the same path to rebuild history when a mapping turns out to have been wrong.