Trace the journey, not the service

At 09:02, checkout latency crosses its alert threshold. The API dashboard is green. Payments are green. The queue is healthy. At 09:11, support reports that customers are seeing an empty confirmation screen after paying.

Nothing in that sequence is unusual. Each service can meet its local target while the journey between them breaks.

Reconstruct the failure from the user's side

Take one affected checkout and write the path as the customer experienced it:

tap Pay
→ create payment intent
→ complete provider challenge
→ confirm order
→ enqueue fulfilment
→ render confirmation

Now attach evidence to every transition. Which release rendered the screen? Which payment intent was used? Did order creation return before the client navigation timed out? Did the queue receive the order? Did the app restore the pending checkout after it restarted?

That exercise changes observability from “collect more telemetry” into “prove or disprove each transition”.

Propagation is the join

OpenTelemetry describes the purpose neatly: “With context propagation, signals (traces, metrics, and logs) can be correlated with each other”.

The browser or mobile client should start a trace context for the journey. APIs preserve it. Messages carry it. Workers extract it. Downstream calls become child spans or linked traces where the transport requires a new root.

Without that join, searching by timestamp becomes the incident response strategy. With it, a responder can move from a failed confirmation event to the payment call and the order worker that belong to the same attempt.

Do not put customer names, tokens or arbitrary request bodies into propagated baggage. OpenTelemetry explicitly warns against placing credentials, API keys or personal data there. Carry identifiers that correlate the work; fetch sensitive context through authorised systems when it is actually needed.

Record decisions, not narration

Compare these two log lines:

OrderService.confirm started
OrderService.confirm completed

order.confirmation.deferred reason=payment_outcome_pending retryAfterMs=2000

The second line explains a branch in the product. Useful events capture validation failures, fallbacks, retries, stale state and terminal outcomes. Their names should remain stable while implementation details change.

An event is not automatically a metric. Decide which fields have bounded values before they become labels, otherwise one customer or request ID can create an expensive cardinality problem. Keep the detailed identifiers in traces and logs; aggregate the dimensions used for operational decisions.

A concrete dashboard

For checkout, a journey-level view should answer four questions without switching tools:

1. What percentage of payment attempts reached a confirmed order?
2. How long did successful journeys take at the slower end of the distribution?
3. Where did unsuccessful journeys stop?
4. Which release, platform or dependency changed with the regression?

Service latency and error rates still matter, but they become diagnostic layers beneath the outcome. Google's SRE guidance recommends defining service-level objectives around what users care about, then choosing indicators that measure those objectives. A database availability chart is evidence; “order confirmed within the expected time” is the promise.

The client is part of the trace

Server-only telemetry cannot see a disabled button, a deep link that opened the wrong route or an app that lost pending state on restart. Capture client navigation, network attempts and visible outcomes with the same release and journey identifiers used by the backend.

For mobile, test the path while moving between Wi-Fi and cellular, backgrounding during a provider challenge and reopening from a notification. For web, include back navigation, duplicate submissions and a tab closed before the final response. These are ordinary user behaviours, not exotic test cases.

Our backend engineering work treats this path as part of the feature. The same principle is visible in products such as Virgin Trains Ticketing, where the meaningful unit is the journey a person completes, not the service boundary a team owns.

What the alert should hand you

A good journey alert contains a representative trace, the affected outcome, the current release and the owning response path. The responder should not need to rediscover the system map while customers wait.

If every box is green and the user still cannot finish, the boxes are not the right top-level model. Trace the promise.

Sources

• Context propagation — OpenTelemetry
• Service Level Objectives — Google SRE book
• W3C Trace Context