Most messaging programmes are measured well enough to produce a report and not well enough to produce a decision. Send counts, delivery percentages and a click rate get circulated weekly, everyone agrees the numbers look broadly fine, and nothing changes until something breaks loudly.
A feedback loop is the thing that turns that reporting into decisions. It does not improve results on its own. What it changes is the quality and the speed of the decisions you make — how quickly you notice a problem, how confidently you can name its cause, and whether the fix lands in the system that actually controls the behaviour.
Start from the outcome, not the message
Before a message is designed, write down the outcome it exists to produce, in the product's own terms: the appointment is confirmed, the code is entered, the invoice is paid, the delivery slot is chosen, the account is reactivated. That sentence determines everything downstream — what you instrument, what counts as success, and what a "good" delivery rate is actually worth.
If nobody can state the outcome, the message is usually being sent because it has always been sent. That is a finding in itself, and it is cheaper to act on than any optimisation.
Two things follow from naming the outcome. First, you need a way to observe it, which almost always lives in the application or product database rather than in messaging data. Second, you need to accept that a delivered message and a completed outcome are different events with different owners.
Capture the whole chain under one correlation ID
The single most valuable piece of instrumentation is a correlation ID generated by your application when it decides to send, carried through every subsequent record, and stored on both sides. Without it, you have several partial datasets that cannot be joined and analysts guess at the joins.
The chain worth capturing:
- Triggering event — what happened in the product, when, and the state that made the message eligible.
- Message request — destination, channel, template or content version, traffic class, sender identity, timestamp, your client reference.
- Provider and route — which provider and route handled the attempt, and any provider-side identifier.
- Canonical statuses — accepted, submitted, delivered, failed, expired, rejected, plus your canonical failure reason and the raw provider value kept for debugging.
- Engagement — clicks with the link resolved back to the message, and replies or inbound events.
- Outcome — the product event you defined, with its own timestamp.
- Negative signals — complaints, opt-outs, support contacts, and repeated retries against the same subject.
The discipline is that all of these land in one place keyed on the same ID. A programme that can answer "for this cohort of triggered messages, what happened to the person" is in a completely different position from one that can answer "what percentage was delivered".
Delivery status is not an outcome
Provider and network delivery receipts are evidence about the network, not about the human. Treating them as an outcome proxy is the most common measurement error in messaging.
What delivery receipts genuinely tell you varies by channel and by market, and it varies in ways that are not always visible: some networks confirm handset delivery, some confirm handoff, some return an optimistic final status, some return nothing at all and leave you to reconcile. A route that returns a suspiciously perfect delivery rate deserves more scrutiny than one with an honest failure distribution.
So use delivery data for what it is good at — detecting route and network problems, comparing providers on the same traffic, spotting registration and filtering issues — and use application outcomes to judge whether the message worked. When the two diverge, that divergence is the interesting signal. A high delivery rate with a collapsed outcome rate points at content, timing, audience or a broken downstream product step, not at the network.
Build a failure taxonomy you can act on
One "failed" bucket is unusable. The point of a taxonomy is that each category maps to a different owner and a different action:
| Category | Typical cause | Owner and action |
|---|---|---|
| Invalid destination | Wrong, disconnected or malformed number or address | Data quality; validate at capture, suppress after repeat |
| Blocked or filtered | Content, sender or category mismatch with route rules | Messaging ops; sender/route/template review |
| Rejected pre-send | Registration, sender identity or template not valid for market | Messaging ops; registration and template governance |
| Capability mismatch | Channel not available for that subject | Orchestration; eligibility rules and fallback |
| Provider or route error | Transient provider fault, bind or throughput issue | Messaging ops; retry policy, failover, vendor escalation |
| Expired | Handset unreachable for the validity period | Policy; validity window and fallback decision |
| No final status | Lost callback, unreconciled attempt | Engineering; reconciliation job and monitoring |
| Suppressed | Opt-out, complaint or quiet-hours rule applied | Compliance and orchestration; confirm rule was correct |
Two rules keep it useful. Never let an unmapped provider value fall into a generic bucket — alert on it instead, because unmapped values are how genuine new failure modes hide. And review the shape of the distribution, not only the total: a stable failure percentage made of a different mix than last month is a change.
Separate the layers before assigning blame
When an outcome rate drops, the argument about cause is usually unproductive because the layers are not separated. Ask, in order, whether the change is in:
- Content — template version, wording, link position, sender name presentation, length or segmentation change.
- Audience — cohort composition, a new acquisition source, a segment that was previously excluded.
- Timing — send window, local time handling, batching or scheduling change, quiet-hours rule.
- Sender identity — registration status, sender ID changes, brand or campaign changes.
- Route or provider — routing change, provider incident, filtering change, throughput throttling.
- Downstream product — the landing page, the app flow, the payment step, the code-entry screen.
The last one is worth checking first more often than teams expect. A collapsed conversion rate with unchanged delivery and click rates is usually a product problem being reported as a messaging problem.
Separating layers also requires change control: routing, template and audience changes should all be timestamped somewhere you can overlay on the metrics. Most of the ambiguity in messaging analysis comes from not knowing what changed when.
Turn signals into named actions
A signal without a named action and an owner is decoration. For each metric you keep, write down the threshold, the owner and the action:
- Delivery rate for a country/route falls below its own established baseline → messaging ops → compare against a second route on the same traffic, escalate to vendor, consider failover.
- Unmapped status values appear → engineering → map or reject, do not absorb.
- Reconciliation gap grows → engineering → check callback endpoint and reconciliation job.
- Complaint or opt-out rate rises for a template → template governance → pause or revise template, review frequency.
- Outcome rate falls with delivery and clicks stable → product → inspect the downstream flow.
- Repeated retries against the same subject → orchestration → check stop conditions and state freshness.
- Rejection at send for a market → registration owner → re-verify sender and template registration status.
Thresholds should be based on each programme's own history rather than an industry figure. Baselines differ enough by market, channel, traffic class and audience that borrowed benchmarks mislead more often than they help.
Test in a way that does not manufacture false causation
Comparing this month to last month attributes every difference to whatever you happened to change. If a decision matters, structure it:
- Controlled holdout — a randomly selected group that receives nothing, or the existing treatment, for a defined period. This is the only way to see what the message itself contributes rather than what the underlying process would have produced anyway.
- Staged rollout — release the change to a fraction of traffic, hold the rest, and compare in the same window rather than across windows.
- One variable at a time — a change to content, timing and route at once yields a result you cannot attribute.
- Same-window route comparison — split identical traffic across two routes concurrently, because sequential route comparisons are contaminated by everything else that moved.
Be explicit about what a test cannot tell you. Small volumes, short windows and rare outcomes produce differences that are noise. It is more useful to say "we cannot tell yet" than to record a conclusion the data does not support.
Watch the delayed and negative signals
Immediate metrics are biased towards optimism because the costs of over-messaging arrive later. Track, on a lag:
- complaints and opt-outs, by template and by frequency band;
- support contacts referencing your messages, which is where confusing content becomes visible;
- repeated attempts against the same subject, a common sign of stale state or missing stop conditions;
- suppression list growth rate, which is a slow, hard-to-reverse cost;
- downstream trust effects such as falling click rates on the same template over time.
A change that raises a short-term conversion rate while accelerating opt-out growth is not obviously a win, and a loop that only reports the first number will keep recommending it.
Close the loop into systems that control behaviour
The step most programmes skip is routing the finding into the system that governs the behaviour, with an owner. Findings should land in:
- Routing policy — per-market and per-traffic-class route selection, throughput allowances, failover order.
- Template governance — versioned templates, approval, retirement of underperformers, per-market variants.
- Suppression and consent — opt-out propagation across channels, complaint handling, quiet-hours and frequency rules.
- Product workflow — trigger conditions, state freshness, stop conditions, the downstream flow itself.
- Vendor management — escalation with evidence, route performance reviews, commercial and coverage decisions.
If a finding cannot be routed to one of these, it is unlikely to change anything. This is also where the operating model matters: whether you use routes we supply, keep your own vendors under BYOV, or run a hybrid of both, someone has to own translating evidence into routing, registration and escalation changes. That ownership is the loop.
A minimum measurement schema
| Field | Source | Why it is needed |
|---|---|---|
| correlation_id | Application | Joins every record for one send decision |
| trigger_event, trigger_at | Application | Establishes eligibility and timing |
| subject_ref | Application | Frequency, suppression and repeat detection |
| channel, traffic_class | Application | Isolation and comparison |
| template_id, template_version | Template registry | Attributing content changes |
| sender_identity | Routing config | Registration and identity analysis |
| provider, route | Gateway | Route comparison and escalation evidence |
| submitted_at, status, status_at | Gateway | Latency and delivery analysis |
| canonical_reason, raw_reason | Gateway | Failure taxonomy without losing detail |
| click_at | Link layer | Engagement attributable to the message |
| outcome_event, outcome_at | Product | The actual objective |
| complaint, opt_out, support_contact | Multiple | Delayed cost signals |
Nothing here is exotic. The reason it is often missing is that each field lives with a different owner, and no one is accountable for the join.
Weekly operating review checklist
- Volume by channel and traffic class, against the previous period, with known changes annotated.
- Delivery and failure distribution by market and route — shape, not just the headline number.
- Any unmapped status values, and the reconciliation gap trend.
- Outcome rate for each defined objective, compared with delivery and click rates.
- Complaints, opt-outs and support contacts, on a lag, by template.
- Repeat-attempt anomalies and suppression growth.
- Open vendor escalations with evidence attached and a next action.
- Registration and template expiries approaching, by market.
- Decisions taken since the last review, and their owners.
The review has done its job when it produces a short list of decisions with owners, not when everyone has seen the charts. A loop that improves the quality and speed of those decisions is worth building. It will not, by itself, improve churn or revenue, and any measurement programme sold on that basis is being oversold.