An intent threshold belongs to the intent, not to the chatbot

A customer on WhatsApp writes "cancel the refund, I want to keep the order". The intent classifier returns cancel_order at 0.83 and cancel_refund at 0.09, the bot's one threshold is 0.80, so the cancel_order handler runs. Order 48213 is cancelled and the customer is told so, which is the opposite of what they asked for. Twenty minutes earlier the same bot received "where's my parcel", scored track_order at 0.78, and asked the customer to rephrase, because 0.78 is below 0.80. One number caused both failures.

A confidence threshold belongs to the intent, not to the chatbot, because the right number depends on what the handler does when the classifier is wrong. A wrong track_order costs one wasted turn. A wrong cancel_order costs a cancelled order, a refund, a support ticket and a customer who no longer trusts the channel. Those two intents cannot share a threshold and both be right.

Fit each threshold to the price of being wrong

The score a classifier emits is not a probability of being right. A softmax of 0.83 from one model and 0.83 from its retrained successor can mean different error rates, so the threshold has to be measured, not chosen. Take the last month of labelled turns, and for each intent and each candidate cut-off from 0.50 to 0.99, compute precision: of the turns scored at or above the cut-off for that intent, how many were really that intent. The act threshold is the lowest cut-off whose precision meets the target the handler's cost demands.

Three bands fall out of that. Above the act threshold the handler runs. Between the confirm threshold and the act threshold the bot restates the intent with the real values ("Cancel order 48213, two items, £64.90?") and waits for yes or no. Below the confirm threshold it asks an open question. Destructive intents get a fourth check: the top score must beat the second intent by a margin, so a 0.83 against a 0.79 confirms even when 0.83 alone would act.

The numbers are example values; yours come from your own transcripts. The shape carries across: a read-only intent can run on a score that a destructive one would not even confirm on.

flowchart TD
  M[Customer message] --> K[Intent classifier]
  K --> T{Top score at or above act}
  T -- yes --> G{Margin over second intent}
  T -- no --> F{Top score at or above confirm}
  G -- yes --> A[Run the handler]
  G -- no --> C[Confirm with real values]
  F -- yes --> C
  F -- no --> Q[Ask an open question]
  P[Per intent thresholds]:::accent --> T
  P --> F

Replay the cancellation with per intent thresholds

Run the opening message again with the table above. The classifier still returns cancel_order at 0.83. The act threshold for cancel_order is 0.95, so the handler does not run. The score is above the confirm threshold of 0.70, so the bot replies "Do you want me to cancel order 48213 (two items, £64.90)? Reply yes or no." The customer replies no, the bot asks what they would like to do with the refund, and the order survives. The track_order message at 0.78 is above its act threshold of 0.65 and gets the tracking status without a detour.

I led an omni-channel chatbot across Messenger, WhatsApp, Telegram and web chat, and the rule I apply is that any intent whose handler moves money or changes an order gets its own fitted act threshold, confirm threshold and margin, and never inherits the bot's default. Every turn's log line must carry the top two intents, their scores and the threshold set version, because a threshold is only valid for the model it was fitted against. After a retrain the old 0.95 may mean 0.93 precision, so re-fit the table on the same labelled turns before the new model takes traffic.

The table breaks in one more way. If cancel_order lands in the confirm band on a quarter of its turns, the temptation is to lower the act threshold until the confirmations stop. Resist it: a high confirm rate means the classifier lacks examples of the phrasings customers use, and a lower threshold buys back the wrong cancellations the table exists to prevent. Fix the model, then re-fit.

Start by moving the single threshold into a per intent table where every row carries today's global value, so nothing changes on the first deploy. Then add the top two scores and the threshold version to the turn log, fit precision per intent on last month's labelled turns, and lower the read-only rows before you raise the destructive ones.