A trace is a story told across processes

A request reaches the bfstore order service:

CreateOrder

The order service validates the basket.

It asks inventory to reserve stock.

It asks payment to authorise the charge.

It writes the new order to MySQL.

It publishes an event.

CLIENT
  │
  ▼
ORDER SERVICE
  │
  ├──► INVENTORY SERVICE
  ├──► PAYMENT SERVICE
  ├──► ORDER DATABASE
  └──► KAFKA

From the customer’s perspective, this is one operation:

Place my order.

From the system’s perspective, it is work performed by several processes, potentially running in different containers, on different nodes and under different failure conditions.

Each process sees only its part.

The order service knows that it called payment.

The payment service knows that it called the provider.

The provider client knows that it waited 1.8 seconds.

No single process naturally owns the complete execution history.

A distributed trace reconstructs the observed parts of that history.

A trace is a story told across processes. Each process records a span, and propagated context tells the observability system that those spans belong to the same execution.

Without shared context, the services produce separate fragments.

With it, their fragments become one navigable account of what happened.

One operation, several local observations

Suppose a checkout takes 2.4 seconds.

The order service can measure:

CreateOrder duration:
    2.4 seconds

That tells me the request was slow.

It does not reveal where the time went.

The inventory service records:

ReserveStock duration:
    70 milliseconds

The payment service records:

AuthorisePayment duration:
    2.1 seconds

The payment service’s HTTP client records:

provider request duration:
    1.9 seconds

These observations exist in different processes.

A trace assembles them:

CreateOrder                           2.4 s
├── ReserveStock                     70 ms
├── AuthorisePayment                 2.1 s
│   └── Payment provider request     1.9 s
├── Insert order                     35 ms
└── Publish OrderCreated             20 ms

Now the slow path is visible.

The order service was not spending 2.4 seconds performing local computation.

Most of the observed time belonged to the external payment call.

The provider request span measures the payment service’s view of that call. It does not reveal the provider’s internal processing unless the provider also participates in compatible tracing.

The trace turns several local observations into one causal picture.

Spans and identifiers reconstruct the execution

A span represents one unit of work within a trace.

Examples include:

HTTP request

gRPC method

database query

message publication

message consumption

cache lookup

application operation

A span commonly contains:

• a trace ID
• its own span ID
• a parent span ID where applicable
• operation name
• start and end time
• service identity
• status
• attributes
• recorded events

A simplified span might look like:

{
  "trace_id": "4f918c...",
  "span_id": "f20a7e...",
  "parent_span_id": "8b71d2...",
  "service": "payment-service",
  "operation": "AuthorisePayment",
  "duration_ms": 2100,
  "status": "error"
}

The span says:

The payment service performed this operation,
for this long,
under this trace context.

Every span recorded under one shared trace context carries the same trace ID.

The spans have separate span IDs:

ORDER SPAN

span ID:
    8b71d2...


PAYMENT SPAN

span ID:
    f20a7e...


PROVIDER SPAN

span ID:
    01c94a...

The relationships form a hierarchy:

trace 4f918c...

CreateOrder
└── AuthorisePayment
    └── Provider request

The trace ID identifies the spans recorded as part of the same observed execution.

The span IDs identify individual pieces of work within it.

The parent relationship records which operation caused another piece of work.

The shape is reconstructed from identifiers contributed by separate participants rather than from every service writing into one shared record.

Context propagation carries the story

The order service cannot create the payment service’s server span.

It runs in another process.

Instead, the order service injects trace context into the outgoing request.

ORDER SERVICE
      │
      │ gRPC request
      │ trace context
      ▼
PAYMENT SERVICE

The payment service extracts that context and creates a related span.

incoming trace:
    4f918c...

new span:
    AuthorisePayment

parent:
    order service client span

The same pattern continues when payment calls the external provider.

The story survives because context travels with the work.

Trace propagation is the stitching that turns local spans into a distributed trace.

Without propagation, each process starts a new fragment.

The tracing backend may receive:

TRACE A

CreateOrder
└── ReserveStock

and separately:

TRACE B

AuthorisePayment
└── Provider request

The operations are related in reality.

They appear unrelated in telemetry.

An operator may still infer the relationship using timestamps, order IDs or logs, but the trace no longer provides it directly.

Propagation gaps can occur when:

• a client library is not instrumented
• metadata is removed by an intermediary
• a custom protocol does not carry context
• a message producer omits trace headers
• a consumer fails to extract them
• incompatible propagation formats are used

Standard formats prevent each team from inventing a private dialect.

For HTTP-style requests, W3C Trace Context defines the traceparent header. OpenTelemetry instrumentation can inject this context into outgoing requests and extract it from incoming ones.

For gRPC, context travels through request metadata.

For messaging, it can travel through message headers.

The transport changes.

The requirement does not:

The next process must receive enough context
to continue the observed execution.

The trace shape should reflect causality

Suppose the order service calls inventory and payment sequentially.

CreateOrder
├── ReserveStock
└── AuthorisePayment

Both calls are children of the order operation.

The hierarchy communicates more than timing.

It records causality:

CreateOrder caused ReserveStock.

CreateOrder caused AuthorisePayment.

If the payment service then calls a provider:

CreateOrder
└── AuthorisePayment
    └── Provider request

the provider client span belongs beneath payment because payment initiated that work.

This helps distinguish time spent locally from time spent waiting on a dependency.

Parallel work changes the shape.

CreateOrder
├── ReserveStock          90 ms
├── CalculateShipping     120 ms
└── AuthorisePayment      1.8 s

The child spans overlap, so their durations should not simply be added together.

CreateOrder       ├──────────────────────────┤

ReserveStock      ├───┤

Shipping          ├─────┤

Payment           ├──────────────────────┤

The request completes when the necessary parallel work completes.

Tracing can reveal one dominant dependency, repeated retries, queueing, lock contention or sequential calls that could have overlapped.

The shape of the story matters as much as the length of each chapter.

Something meaningful may also occur inside a span without deserving another child operation.

An AuthorisePayment span might record events such as:

validation completed

provider request sent

retry scheduled

outcome marked unknown

A child span is more appropriate when the work has its own duration and execution boundary, such as the provider request itself.

Not every line of application code needs its own brightly lit box.

Errors should preserve technical and business meaning

Suppose the payment provider times out.

The provider client span might record:

Provider request

status:
    error

error type:
    deadline exceeded

The parent payment span may record a domain consequence:

AuthorisePayment

status:
    error

payment outcome:
    unknown

The order span may then record:

CreateOrder

status:
    error

result:
    checkout incomplete

These spans describe different levels of the same failure.

PROVIDER CLIENT

network deadline exceeded


PAYMENT SERVICE

authorisation outcome unknown


ORDER SERVICE

checkout could not be completed safely

Technical failure and business consequence are related but not identical.

Span status should follow the operation’s contract and the relevant semantic conventions. A valid negative business result is not automatically an instrumentation error.

For example, an inventory response saying that stock is unavailable may be an expected domain outcome rather than a failed span.

The trace should preserve each layer’s meaning instead of replacing every outcome with the same generic error=true.

Traces connect to logs rather than replacing them

It is tempting to attach every diagnostic detail to spans.

That creates traces containing complete request bodies, response payloads, authentication tokens, personal data and large stack traces.

The result is expensive, risky and difficult to navigate.

Spans should describe the execution path.

Logs can preserve richer event details where appropriate.

TRACE

which path,
which timing,
which status


LOG

which detailed event,
which identifiers,
which diagnostic context

Suppose the payment span contains:

trace_id:
    4f918c...

span_id:
    f20a7e...

The payment logs include the same values:

{
  "event": "payment_outcome_unknown",
  "payment_id": "pay_91a2c",
  "trace_id": "4f918c...",
  "span_id": "f20a7e..."
}

The tracing interface can link from the failed span to the logs emitted during that operation.

The trace provides the map.

The logs provide notes made along the route.

Neither needs to impersonate the other.

A business workflow may outlive one trace

A checkout request may finish after three seconds.

The business workflow may continue for hours.

Suppose payment authorisation times out with an unknown result.

The request returns an incomplete outcome.

A reconciliation worker checks the provider later.

INITIAL REQUEST TRACE

CreateOrder
└── AuthorisePayment
    └── timeout


LATER RECONCILIATION TRACE

ReconcilePayment
└── QueryProvider

These are separate executions.

They may share a stable business identifier:

payment_id:
    pay_91a2c

but they do not need to be one trace lasting several hours.

TRACE ID

identifies one observed execution


BUSINESS OPERATION ID

identifies the logical workflow
across several executions

Logs can carry both identifiers.

Trace links can connect related executions where the tracing model supports it.

The complete business story may span several traces.

Each trace should still give a coherent account of one execution.

Retries, deadlines and asynchronous work

Retries need visible attempt boundaries.

AuthorisePayment
├── Provider attempt 1     timeout
└── Provider attempt 2     success

Without separate attempt spans, the trace might show only:

Provider request:
    3.8 seconds

That hides how many calls were made, which attempt failed, whether backoff occurred and whether the same idempotency key was reused.

A trace can reveal amplification:

one checkout

produced

three payment requests

That matters during dependency incidents, when retries can turn a small failure into a traffic anvil.

Deadlines should also appear in the story.

CreateOrder                    3.0 s budget
└── AuthorisePayment           2.8 s remaining
    └── Provider request       2.5 s remaining

Without deadline propagation, each service may start a fresh timeout and allow the complete path to exceed the caller’s intended limit.

Tracing can show whether the latency budget was propagated, reset incorrectly or consumed by one dependency.

Asynchronous systems need a different relationship model.

The order service may publish:

OrderConfirmed

Shipping and notification consume it independently.

ORDER SERVICE
      │
      ▼
OrderConfirmed
      │
      ├──► SHIPPING
      └──► NOTIFICATION

The consumers may start after the producer request has completed.

Asynchronous relationships should preserve causality without pretending there was one synchronous call stack.

Span links are particularly useful when separate traces, fan-out or batch processing make a simple parent-child relationship misleading.

ORDER TRACE
    │
    └── publishes OrderConfirmed
              │
              ├── linked shipping trace
              └── linked notification trace

The message carries enough origin context for the consumers to record the relationship honestly.

Sampling decides which stories survive

A busy platform may produce more traces than it can afford to retain.

Sampling chooses a subset.

Head sampling decides near the beginning of the trace.

It is efficient, but it does not yet know whether the request will become slow or fail.

Tail sampling waits until more spans arrive.

It can make decisions using the completed outcome, such as:

retain if:

status = error

or

duration > threshold

or

service = payment

A wider policy might retain a small percentage of routine success, all errors and more slow requests.

Sampling means the tracing system does not preserve every story.

Operators need to know which policy applies.

No matching trace

may mean:

the request was not sampled

rather than:

the request never occurred

Names, attributes and baggage make traces useful

A trace full of spans named:

request

call

execute

handler

reveals very little.

Operation names should describe stable work:

OrderService/CreateOrder

InventoryService/ReserveStock

PaymentService/Authorise

HTTP POST payment-provider.example/authorisations

Names should not include unbounded identifiers.

Weak span name:

GET /products/7f31c2

Stronger span name:

GET /products/{product_id}

The specific product ID can be an attribute where appropriate.

A useful distinction is:

Useful attributes might include:

service.name:
    payment-service

deployment.environment:
    prod

service.version:
    1.6.0

rpc.method:
    Authorise

payment.provider:
    provider-a

result:
    unknown

These support investigations such as:

Show failed Authorise traces

for provider-a

after version 1.6.0

in production.

Attributes should be bounded and selected deliberately.

A trace does not need the customer’s full payment details to explain that a provider request timed out.

Baggage carries selected key-value context across service boundaries.

Possible examples include:

tenant tier

experiment cohort

support case identifier

Baggage is propagated separately from span attributes and is not automatically recorded in every trace. Instrumentation must deliberately decide whether to copy a baggage value into telemetry.

It may also travel widely, so it should not contain credentials, authentication tokens, card data, private customer details or large payloads.

The question is not only whether a value would help the trace.

It is whether every participating process needs to receive it.

Tracing has cost, privacy and reliability limits

Tracing adds work:

• creating spans
• collecting attributes
• propagating context
• buffering telemetry
• exporting data
• storing and querying traces

Poor instrumentation can increase latency or memory use.

A healthy design usually exports spans asynchronously and applies bounded buffering.

APPLICATION WORK
      │
      ├── complete operation
      │
      └── enqueue telemetry

When buffers fill, dropping telemetry may be safer than exhausting application resources.

The workload should not fail merely because the tracing backend is unavailable.

Observability exists to support the service, not to become a surprise synchronous dependency.

Separate processes also record timestamps using separate clocks.

Their clocks may not be perfectly aligned, so a child span can appear to start before the parent sent the request.

Causal relationships and measured durations should be interpreted alongside wall-clock timestamps.

A trace is a distributed account assembled from cooperating narrators.

Occasionally, one narrator’s watch is fast.

A bfstore checkout trace

A successful trace might look like:

CreateOrder                                      420 ms
├── GetBasket                                     35 ms
├── ReserveStock                                  82 ms
│   └── inventory MySQL transaction               26 ms
├── AuthorisePayment                             210 ms
│   └── provider authorisation request           180 ms
├── order MySQL transaction                       41 ms
└── publish OrderConfirmed                        18 ms

A failing trace might look like:

CreateOrder                                      2.4 s
├── GetBasket                                     32 ms
├── ReserveStock                                  79 ms
├── AuthorisePayment                              2.1 s
│   └── provider authorisation request            2.0 s
│       status: deadline exceeded
└── release stock reservation                     96 ms

The failing trace shows that:

• the inventory reservation succeeded
• payment consumed most of the request time
• the provider call exceeded its deadline
• order creation did not proceed
• compensating release work occurred

The payment span records:

payment outcome:
    unknown

The logs contain the payment and order identifiers.

A later reconciliation trace connects the recovery attempt to the same payment operation.

The complete business story spans more than one trace.

Each trace still gives a coherent account of one execution.

A compact tracing review

A trace is useful when it preserves the shape and meaning of distributed work without attempting to record the entire universe around it.

The mental model I am keeping

My earlier model was:

TRACE

a timing diagram for one request

The stronger model is:

                          LOGICAL OPERATION
                                  │
                                  ▼
                              ROOT SPAN
                                  │
                  ┌───────────────┼───────────────┐
                  │               │               │
                  ▼               ▼               ▼
             PROCESS A       PROCESS B       PROCESS C
               SPAN            SPAN            SPAN
                  │               │               │
                  └───────────────┼───────────────┘
                                  ▼
                         PROPAGATED CONTEXT
                                  │
                                  ▼
                         ONE DISTRIBUTED STORY

A distributed system does not produce one natural call stack.

Each process knows its own work and little more.

Tracing creates a shared narrative by passing context from one process to the next and allowing each participant to record the chapter it owns.

The resulting trace is not a perfect transcript.

It is a structured account of causality, timing and boundaries.

One service begins the sentence.

Another service continues it.

A database, broker or external provider may add a difficult paragraph.

When the context survives the journey, the operator can finally read the operation as the customer experienced it:

not as a collection of isolated processes, but as one story moving through them.

References and further reading

OpenTelemetry: Traces Introduces traces, spans, span context, attributes, events, links and status.

OpenTelemetry: Context propagation Explains how tracing context moves between services and processes.

W3C Trace Context Defines standard headers for carrying trace identifiers and flags across distributed requests.

OpenTelemetry semantic conventions Documents common naming and attribute conventions for services, RPC calls, messaging, databases and other operations.

Google Dapper paper Describes the tracing model that influenced many modern distributed-tracing systems.