Dedupe chat messages by channel message id, never by text and time

A customer on WhatsApp replies "yes" to a bot that has just asked them to confirm an order of £48.00. The channel posts the message to my webhook with message id 7Qx1 at 14:02:10. The handler parses the intent, places the order, sends the confirmation back through the channel's outbound API, and only then returns 200. That round trip takes 2.4 seconds. Suppose the channel waits 2 seconds for the acknowledgement, gives up, and 20 seconds later posts the same message again: same id, same text, same timestamp. The rule I want in that handler is this: a message whose channel and message id pair is already stored is dropped before anything parses it, and the pair is stored before the webhook is acknowledged. Not a hash of sender, text and time, and not a check after intent detection.

Walk the redelivery through a handler without that rule. The second webhook arrives at 14:02:30. If the first pass has finished, the conversation is in the state that awaits a delivery slot, so "yes" is a miss for the slot parser and the customer is asked "Which delivery slot would you like?" a second time, as if the bot had forgotten. If the first pass is still mid-flight, because the outbound confirmation is slow, the second pass reads the conversation as still awaiting confirmation, parses "yes" as a confirmation, and calls the order handler again. The customer sees two confirmations and two orders of £48.00. The bot saw two valid messages; nothing it inspected looked wrong.

Where the text hash breaks

The tempting substitute is a fingerprint: hash the sender, the text and a 30 second window, and drop a match. It fails in both directions. A customer asked "How many?" answers "2", then is asked "Morning or afternoon, 1 or 2?" and answers "2" eleven seconds later. Same sender, same text, inside the window: the second answer is dropped and the slot question is asked again. In the other direction, a channel can redeliver a message minutes or hours after the first copy, far outside any window a live conversation can tolerate, and the fingerprint lets it through. The channel's message id is the channel's own statement that this is the same message. Two different ids are two messages, however identical the text; one id is one message, however long the gap.

Each channel's ids live in their own namespace, so the key is the pair (channel, message id), never the id alone: one channel issues integers, another opaque strings, and nothing stops them colliding. Your own web chat gives you no id unless the client generates one, so generate a UUID per send in the client and keep it across the client's own retries, which are redeliveries too. I have led an omni-channel chatbot across Messenger, WhatsApp, Telegram and web chat, and the rule I apply is that no message reaches the intent step until this pair has been written.

Store the id before you acknowledge

Insert the pair, with a unique constraint on it, as the first statement of the handler: before parsing, before reading the conversation state. On success, continue. On a constraint violation, return 200 and do nothing else; the first delivery owns the message, and a 4xx or 5xx here would make the channel redeliver yet again. For the first delivery, return 200 as soon as the row and the raw payload are stored, and run the parse and the reply from a queue, so the acknowledgement never waits on the channel's outbound API. The row has to outlive the channel's redelivery horizon, which runs to days for some channels, so keep it for the life of the conversation plus a week rather than for a few minutes.

flowchart TD
  C[Channel webhook] --> I[Insert channel and message id]:::accent
  I --> D{Row already there}
  D -- yes --> A[Return 200 and drop]
  D -- no --> Q[Store raw payload and return 200]
  Q --> P[Parse intent from queue]
  P --> R[Reply via channel API]

// simplified: the unique index on (channel, channel_message_id) is the lock
async function onInbound(channel: string, msg: InboundMessage) {
  const inserted = await db.insertIgnore('inbound_messages', {
    channel,
    channelMessageId: msg.id,
    conversationId: msg.conversationId,
    raw: msg,
  });
  if (!inserted) return ack(200); // redelivery: the first copy owns it
  await queue.publish('inbound.stored', { channel, id: msg.id });
  return ack(200); // parse and reply happen off the request path
}

Add the inbound messages table with the unique index today, and move the insert to the top of every channel's webhook handler. Then take the reply off the request path, so the acknowledgement no longer depends on how quickly the channel accepts your outbound message.