Ingest before you route

Two operations happen when a lead arrives: you store it, and you try to sell it. The order you do them in decides what a buyer outage costs you.

Route first and a failure downstream can take the lead with it. Store first and the worst case is a lead you have to sell later.

That is the whole principle. The rest is what it implies.

Respond before you sell

The visitor who filled in your form is waiting. Whatever you do about buyers, do it after they have seen the thank-you page.

1receive → normalise → store → respond   ← the visitor's experience ends here
2                                ↓
3                          route to buyers ← everything commercial happens here

There is no version of the waterfall the visitor should wait for. A three-buyer waterfall with ping/post and ten-second timeouts has a worst case measured in the tens of seconds, and a person staring at a spinner for forty seconds concludes the form is broken and submits again.

Splitting it also means a buyer being down is invisible to the person who just converted. They get their confirmation. You get a lead marked unsold and a log row explaining why, which you can act on at your leisure.

Log before you persist

A smaller ordering decision with an outsized payoff: write the "a lead arrived" log entry before you write the lead itself.

If the persist then fails — a validation error, a database hiccup, a duplicate key — you still have evidence something arrived. Do it the other way round and a failure at the persist step produces complete silence, which is indistinguishable from nothing having happened.

The cases where you most need to know a lead existed are exactly the cases where the code that records it did not finish.

Store what arrived, not just what you understood

Normalisation is lossy by design. It keeps the fields you mapped, in the shape you specified, using the configuration that was live at the time.

Keep the raw payload too, unmodified, forever. It is small, it compresses well, and it is the only artefact that can answer:

• Why is this field empty? Because it arrived under a key your mapping did not know.
• Did the partner change their format? Compare a lead from March to one from August.
• Can we fix the mapping and re-derive the last two weeks? Only if the originals exist.

Every operator who deleted raw payloads to save space learned why not during the incident that needed them.

Idempotency is not optional

Anything that retries will eventually deliver the same lead twice. Meta retries. Partners retry. Your own reconciliation sweep is a deliberate second delivery of things you may already have.

So dedupe on something stable from the source — the provider's own lead ID — and make the database the authority rather than a prior lookup. A check-then-insert has a race window; a unique index does not. Catch the duplicate-key error and treat it as the duplicate it is.

The consequence of getting this wrong is not a tidy data-quality problem. It is selling the same person to the same buyer twice, which is the fastest way to lose a buyer.

One sale, many calls

Worth stating as an invariant, because it is easy to lose during a refactor: a lead may generate any number of outbound calls, and at most one sale.

A three-buyer waterfall with ping/post might make five HTTP calls plus two event postbacks. All seven are real, all seven should be logged, and exactly one of them — the accepted post — is the sale. The moment a lead is accepted, the waterfall stops.

Keeping "how many calls did we make" and "did we sell it" as separate concepts is what lets your logging be complete without your revenue being wrong.

Snapshot every attempt

When a call fails and you want to retry it, the question is: retry with what?

Not with the current configuration. The buyer's body template may have changed, their endpoint may have moved, the accept rule may have been rewritten. Replaying a two-day-old failure against today's config is not a retry, it is a new and different request that happens to reference the same lead.

Freeze the configuration used for each attempt and store it with the log row. A retry replays the snapshot. And write the retry as a new row rather than updating the original — attempts are history, and overwriting history is how you lose the ability to explain what happened.

What this buys you

Order the operations this way and a buyer outage is an inconvenience: leads keep arriving, keep being stored, and sit marked unsold with a clear reason until the buyer comes back or you route them elsewhere.

Order them the other way and the same outage is lost inventory you already paid to acquire, with no record that it existed.

The code is barely different. The failure mode is completely different.