The Metrics That Actually Predict OTP Success
Delivery rate isn't the right number. Here are the four metrics that actually correlate with OTP conversion — and how to instrument them.
Why "delivery rate" is misleading
The default OTP dashboard shows delivery rate — the percentage of messages that returned a "DELIVRD" status from the carrier. Most teams report this to management. Most management thinks 98%+ delivery means the system is healthy.
It often doesn't. Here are four metrics that correlate far better with whether users actually log in.
1. Time-to-deliver, p95
OTPs typically expire in 60 seconds. Anything that adds latency between send and arrival eats into the window the user has to enter the code.
Track p95 — not average — time from API request to handset arrival, segmented by:
- Operator (within each country)
- Vendor route
- Hour of day
- Traffic class
What you'll find: average looks fine (3–5 seconds), but the p95 on one operator-route combination has crept up to 25–40 seconds. That's where conversion is silently dropping.
Alert threshold: p95 over 10 seconds for any operator-route combination over a 5-minute window.
2. Conversion rate per route
The only honest measure of OTP success is whether the user submitted the correct code in time. This is data you have in your application — not your messaging vendor.
Join your message-send log with your OTP-verification log on a request ID. Then aggregate conversion by:
- Route (which vendor delivered)
- Country
- Hour of day
- Sender ID
What you'll find: one route delivers 99% according to DLRs but only 84% of users actually verify. Another route delivers 96% according to DLRs but 94% of users verify. The second route is better despite the worse delivery number — because the messages that do arrive, arrive on time and look legitimate.
This is the single most important number in OTP operations and almost nobody tracks it.
3. First-attempt verification rate
When users don't get their OTP in time, they hit "resend". Tracking the rate of resends per attempted login surfaces problems before they show up in conversion.
A healthy OTP flow has 5–10% of users requesting a resend (typos, switching device, etc.). When that number jumps to 20–30% on a specific route, you have a delivery problem the DLRs are hiding.
Alert threshold: resend rate over 15% for any operator-route combination, sustained for 10 minutes.
4. Time-of-first-arrival distribution
Beyond p95, the shape of the latency distribution tells you whether problems are systemic or sporadic.
Plot a histogram of time-to-deliver per operator. A healthy route shows a tight peak around 1–3 seconds with a long thin tail. A degrading route shows a bimodal distribution — most messages still fast, but a growing cluster at 30+ seconds.
Bimodal distributions are early warning. By the time the average has moved, conversion has already dropped on the slow cluster.
Vendor SLOs vs your SLOs
Vendor SLAs are typically:
- 99% delivery rate, measured monthly
- No latency commitment beyond "best effort"
- No commitment on conversion (which they don't have visibility into anyway)
Your internal SLO for OTP should be tighter:
- p95 delivery time under 10 seconds, per operator
- Conversion rate within 2 percentage points of your 30-day baseline, per operator
- First-attempt verification rate above 85%, per operator
When you breach your SLOs but not the vendor's, you've found a real operational gap that doesn't appear in the contract.
Building the dashboard
Minimum viable OTP observability:
- Tag every message with: route ID, operator (post-lookup), traffic class, sender ID, country
- Join with verification events on a request ID
- Build per-route, per-operator dashboards for the four metrics above
- Alert on the per-route thresholds, not aggregate
This isn't difficult — but most teams haven't done it because the default vendor dashboards looked fine.
What good looks like
A mature OTP operation should be able to answer, in under a minute:
- Which route is currently delivering best on Vodafone Spain?
- What's our conversion rate on T-Mobile US compared to last week?
- Which operator-route combination has the worst p95 right now?
- Did the resend rate move after the last vendor change?
If you can't, your observability is the lowest-leverage place to invest in your messaging stack.
If your OTP dashboards stop at "delivery rate" and you suspect there's silent loss happening underneath, book a 30-minute messaging review. We'll show you what we instrument by default.