A service records a request span.
The first implementation sends it directly to one tracing backend:
APPLICATION
│
▼
VENDOR-SPECIFIC TRACING SDK
│
▼
VENDOR BACKEND
The application now depends on more than a storage destination.
Its code may depend on:
- the vendor’s tracing API
- the vendor’s context propagation
- the vendor’s span model
- the vendor’s configuration
- the vendor’s exporter
- the vendor’s sampling controls
Later, the platform team wants to evaluate another backend.
Perhaps the original system became too expensive.
Perhaps the team wants to self-host its observability stack.
Perhaps traces need to be sent to two destinations during a migration.
Perhaps one backend is excellent for metrics but less suitable for long-term trace storage.
Changing the backend now means revisiting application code.
NEW BACKEND
requires
NEW SDK
NEW INSTRUMENTATION
NEW CONFIGURATION
NEW DEPLOYMENT
That feels backwards.
The application should describe what it is doing:
handling a gRPC request
calling MySQL
publishing a Kafka message
waiting for a payment provider
The observability platform should decide where those descriptions are processed and stored.
This is the separation OpenTelemetry offers.
OpenTelemetry allows applications to instrument their behaviour through a common telemetry model while leaving export, processing and backend selection to the platform.
The phrase I keep encountering is:
instrument once,
choose the backend later
That is directionally useful.
It does not mean every backend is interchangeable or that migrations become free.
It means the application no longer needs to speak the private language of every storage system it may use.
Instrumentation and storage are different concerns
Instrumentation describes what happened inside the application.
Storage preserves and queries the resulting telemetry.
Those are related responsibilities, but they do not need to be fused together.
APPLICATION CONCERN
Which operation occurred?
How long did it take?
Which dependency was involved?
Did it succeed?
Which trace did it belong to?
PLATFORM CONCERN
Where should telemetry be sent?
How long should it be retained?
How should it be sampled?
Which backend should store it?
Who may query it?
A catalog service should know that it is handling:
CatalogService/GetProduct
It should not need to know that the platform currently stores traces in one particular database.
Similarly, the payment service should know that it is calling:
payment-provider/authorise
It should not contain backend-specific query syntax or authentication logic for the tracing platform.
OpenTelemetry creates a boundary between those concerns.
The application emits telemetry through APIs and SDKs
At the application layer, OpenTelemetry provides APIs for creating and enriching telemetry.
For tracing, application code may create a span around meaningful work:
ctx, span := tracer.Start(ctx, "PaymentService/Authorise")
defer span.End()
The span can record attributes:
span.SetAttributes(
attribute.String("payment.provider", providerName),
attribute.String("payment.operation", "authorise"),
)
It can record an error:
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "payment authorisation failed")
return err
}
The application describes the operation.
The SDK handles concerns such as:
- creating span identifiers
- applying sampling decisions
- processing completed spans
- batching telemetry
- passing data to an exporter
APPLICATION CODE
│
▼
OPENTELEMETRY API
│
▼
OPENTELEMETRY SDK
│
▼
EXPORT PIPELINE
The API is the instrumentation contract used by the code.
The SDK supplies the runtime implementation behind that contract.
Automatic instrumentation covers common boundaries
Many frameworks and libraries already have recognisable operational boundaries:
- incoming HTTP requests
- outgoing HTTP requests
- gRPC clients and servers
- database calls
- message producers and consumers
- runtime behaviour
Automatic instrumentation can create telemetry around those boundaries without requiring every span to be written manually.
For example, a gRPC server interceptor may create a span for:
bfstore.catalog.v1.CatalogService/GetProduct
A database instrumentation library may create a child span for the MySQL query.
GetProduct
└── MySQL SELECT
This gives useful technical coverage quickly.
It can reveal:
- request duration
- response status
- dependency latency
- network failures
- database errors
- trace relationships
But automatic instrumentation understands frameworks better than it understands the business.
It may know:
gRPC request returned DeadlineExceeded
It may not know:
payment outcome is now unknown
That meaning belongs to the application domain.
Manual instrumentation adds domain meaning
A useful trace should contain more than framework activity.
Suppose the order service coordinates checkout.
Automatic instrumentation may show:
OrderService/CreateOrder
├── InventoryService/ReserveStock
├── PaymentService/Authorise
└── MySQL INSERT
Manual instrumentation can add events or attributes describing the workflow:
inventory reservation:
confirmed
payment outcome:
unknown
order result:
pending reconciliation
These details explain the business consequence.
TECHNICAL SIGNAL
provider request exceeded deadline
DOMAIN SIGNAL
payment may still have been authorised
Both matter.
Automatic instrumentation provides breadth.
Manual instrumentation provides meaning.
A good OpenTelemetry design uses both rather than asking every developer to handcraft network spans or expecting framework instrumentation to understand the business.
Context propagation keeps the trace intact
A trace crosses process boundaries only when context travels with the request.
ORDER SERVICE
│
│ trace context
▼
PAYMENT SERVICE
│
│ trace context
▼
PAYMENT PROVIDER CLIENT
OpenTelemetry supports propagators that inject context into outgoing requests and extract it from incoming ones.
For HTTP-style communication, this often involves standard trace headers.
For gRPC, the context travels through metadata.
For Kafka, it can travel through message headers.
SENDER
inject context
│
▼
TRANSPORT
│
▼
RECEIVER
extract context
The receiving process can then continue the existing trace rather than creating an unrelated one.
Without propagation:
order trace
payment trace
provider trace
With propagation:
checkout trace
OrderService/CreateOrder
└── PaymentService/Authorise
└── provider request
OpenTelemetry does not make distributed causality automatic.
Every relevant boundary still needs compatible instrumentation and propagation.
Resources describe where telemetry came from
A span named:
GetProduct
is useful.
It becomes much more useful when the platform also knows:
service.name:
catalog-service
service.version:
1.8.0
deployment.environment:
prod
cloud.region:
eu-west-2
OpenTelemetry resources describe the entity producing telemetry.
TELEMETRY RECORD
│
├── operation attributes
└── resource attributes
Operation attributes describe this particular work.
Resource attributes describe the service or runtime that emitted it.
For bfstore, a common resource model might include:
- service name
- service version
- environment
- cluster
- Namespace
- cloud account
- region
- container image digest
This supports questions such as:
Did catalog latency increase
only in production?
Did failures begin after version 1.8.0?
Are traces missing from one cluster?
Is one image digest behaving differently?
Consistent resource attributes make telemetry from different services easier to compare.
Semantic conventions create a shared vocabulary
A common API is useful.
A common vocabulary is equally important.
One service could describe an HTTP method as:
method
Another might use:
http_method
A third might choose:
request.verb
All three values may be technically correct.
Queries across services become awkward.
Semantic conventions define common names and meanings for attributes associated with technologies such as:
- HTTP
- RPC
- databases
- messaging
- cloud resources
- service identity
WITHOUT CONVENTIONS
every service invents its fields
WITH CONVENTIONS
common operations use
a shared telemetry vocabulary
This improves interoperability between:
- libraries
- collectors
- backends
- dashboards
- alerting rules
Custom business attributes remain useful.
They should complement the shared vocabulary rather than replace it.
OTLP provides a common transport
OpenTelemetry telemetry can be exported using the OpenTelemetry Protocol, commonly shortened to OTLP.
A simplified path is:
APPLICATION
│
│ OTLP
▼
COLLECTOR
The application does not need a direct integration with every backend.
The collector can receive OTLP and export the data elsewhere.
APPLICATIONS
│
▼
OTLP RECEIVER
│
▼
COLLECTOR PIPELINES
│
├── traces backend
├── metrics backend
└── logs backend
This is one of the most important portability boundaries.
The services speak a common telemetry protocol.
The platform translates or routes that telemetry to the selected storage systems.
Changing the backend may require collector configuration rather than application changes.
The Collector is telemetry middleware
The OpenTelemetry Collector is more than a forwarding process.
It can receive, process and export telemetry.
RECEIVERS
│
▼
PROCESSORS
│
▼
EXPORTERS
A traces pipeline might look like:
service:
pipelines:
traces:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- trace_backend
A metrics pipeline might use different processors and exporters:
service:
pipelines:
metrics:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- metrics_backend
The collector can perform tasks such as:
- batching
- retrying exports
- limiting memory use
- filtering
- enrichment
- attribute transformation
- sampling
- routing
- exporting to multiple destinations
It creates a controlled platform boundary between applications and telemetry storage.
Direct export couples the application more tightly
An application can export telemetry directly to a backend.
APPLICATION
│
▼
BACKEND
This may be simple for a small environment.
It also places more backend responsibility inside each workload:
- endpoint configuration
- authentication
- retries
- exporter behaviour
- migration changes
- destination-specific settings
Using a collector creates another component to operate.
It also centralises those concerns.
APPLICATION
│
▼
COLLECTOR
│
▼
BACKEND
The choice is not:
collector good
direct export bad
The useful question is:
Where should telemetry routing
and backend coupling live?
For a platform with many services, a collector layer often provides a cleaner operational boundary.
Agent and gateway patterns solve different problems
Collectors can be deployed close to workloads or as shared gateways.
An agent-style collector runs near the workload:
APPLICATION
│
▼
LOCAL COLLECTOR
│
▼
CENTRAL GATEWAY
A gateway-style collector receives telemetry from several workloads:
APPLICATION A
APPLICATION B
APPLICATION C
│
▼
SHARED COLLECTOR GATEWAY
A combined model may use both:
WORKLOADS
│
▼
NODE OR SIDECAR COLLECTORS
│
▼
CENTRAL COLLECTOR GATEWAY
│
▼
BACKENDS
Local collectors can reduce repeated exporter logic and gather host-level context.
Central gateways can apply organisation-wide:
- routing
- filtering
- tail sampling
- authentication
- export policy
Every additional layer adds capacity and failure considerations.
The topology should follow the platform’s scale, security boundaries and operational needs.
One application can feed several backends
A collector can export the same telemetry to more than one destination.
APPLICATION
│
▼
COLLECTOR
│
├──► BACKEND A
└──► BACKEND B
This is useful during:
- backend evaluation
- migration
- disaster recovery testing
- specialised analysis
- temporary dual running
The application remains unchanged.
The collector configuration controls the destinations.
For example:
service:
pipelines:
traces:
receivers:
- otlp
processors:
- batch
exporters:
- current_backend
- candidate_backend
Dual export is not free.
It increases:
- network traffic
- collector work
- storage volume
- backend cost
But it allows a platform to compare backends using the same telemetry population.
That is much safer than replacing every service’s instrumentation before discovering that the new backend cannot support an important investigation.
Backend independence is not backend equivalence
The phrase:
choose the backend later
needs a boundary.
OpenTelemetry can reduce instrumentation coupling.
It cannot make every backend behave identically.
Backends differ in:
- storage architecture
- query language
- indexing
- cardinality limits
- retention
- sampling support
- dashboarding
- alerting
- trace-to-log navigation
- operational cost
- scaling characteristics
A query designed for one backend may need to be rewritten for another.
A dashboard may use backend-specific functions.
A sampling strategy may depend on collector or backend capabilities.
A backend may interpret or index attributes differently.
PORTABLE
telemetry generation
context propagation
common data model
OTLP transport
LESS PORTABLE
queries
dashboards
alerts
retention behaviour
backend operations
OpenTelemetry gives the platform room to change.
It does not abolish the consequences of choosing.
Instrumentation quality still determines telemetry quality
A standard SDK cannot rescue poor instrumentation.
A service may emit spans named:
handle
request
execute
Those spans reveal little.
It may attach:
customer_id
order_id
complete request body
to every metric, causing dangerous cardinality or privacy problems.
It may forget to propagate context through Kafka.
It may mark every business rejection as a system error.
It may create a separate span for every small function call until traces resemble a fluorescent hedgerow.
OpenTelemetry standardises the mechanism.
Teams still need an instrumentation design.
Useful questions include:
Which operations deserve spans?
Which events belong in logs?
Which behaviours need metrics?
Which attributes are bounded?
Which identifiers are sensitive?
Which business states matter during incidents?
The platform should provide conventions, examples and libraries rather than merely handing teams an SDK.
Metrics need careful attribute choices
A metric might record request count:
rpc.server.requests
Useful attributes could include:
service:
catalog
method:
GetProduct
result:
success
An attribute such as:
product_id
could create one time series per product.
An attribute such as:
customer_email
could create enormous cardinality while leaking personal information.
METRICS
describe populations
TRACES AND LOGS
can carry selected
per-operation context
OpenTelemetry does not remove cardinality limits.
Its common model makes it easier for a platform to establish consistent rules around them.
Logs require more than redirecting stdout
An application can write structured logs to standard output.
A collector can gather and export them.
That alone does not guarantee useful correlation.
A log becomes more valuable when it includes:
- service identity
- environment
- event name
- trace ID
- span ID
- deployment version
- relevant business-operation identity
For example:
{
"event": "payment_outcome_unknown",
"service": "payment-service",
"payment_id": "pay_91a2c",
"trace_id": "4f918c...",
"span_id": "f20a7e..."
}
The trace ID allows an operator to move from a failing span to the detailed application event.
TRACE
shows where the operation failed
│
▼
LOG
explains the domain consequence
OpenTelemetry can support this correlation.
The application still needs structured events with stable meaning.
Sampling belongs in the architecture
A busy platform may produce more traces than it can retain.
Sampling controls the volume.
Head sampling decides early:
sample this request?
yes or no
It is efficient.
It may discard a request that later becomes slow or fails.
Tail sampling waits for more of the trace before deciding.
It can retain:
errors
slow traces
selected services
rare operations
Tail sampling often belongs in a collector gateway because the gateway can observe spans from across the trace.
APPLICATIONS
│
▼
COLLECTOR GATEWAY
│
▼
TAIL-SAMPLING DECISION
│
▼
TRACE BACKEND
Sampling policy is not merely a backend setting.
It affects which operational stories survive.
The platform should document:
- what is sampled
- where the decision occurs
- which traces are prioritised
- how sampling affects investigations
Telemetry export must not control application success
The observability backend may become unavailable.
The business service should usually continue serving requests.
APPLICATION
│
├── complete business operation
│
└── export telemetry asynchronously
SDKs and collectors commonly batch telemetry and retry export.
Buffers must remain bounded.
When the telemetry pipeline cannot keep up, the system may need to drop data rather than consume unlimited application memory.
BACKEND UNAVAILABLE
│
▼
BUFFER FILLS
│
├── bounded retry
└── telemetry dropped
The trade-off should be visible.
A metric showing dropped spans is better than silent disappearance.
Audit or regulatory events may require a stronger delivery design than ordinary diagnostic traces.
Not every signal needs the same durability guarantee.
The collector needs its own observability
The collector is part of the observability system.
It can fail.
Possible failure modes include:
- rejected telemetry
- full queues
- memory exhaustion
- export timeouts
- malformed attributes
- backend throttling
- unavailable destinations
- excessive cardinality
- dropped batches
The collector should expose signals such as:
records received
records exported
records refused
records dropped
export failures
queue utilisation
processor latency
APPLICATION TELEMETRY
│
▼
COLLECTOR HEALTH
│
▼
BACKEND INGESTION
Without this visibility, missing telemetry can be mistaken for healthy behaviour.
A silent collector failure creates a particularly elegant operational paradox: the system designed to reveal failure has misplaced its own glasses.
Configuration should remain separate from instrumentation
Application instrumentation should not contain hard-coded backend details.
For example, the service code should not need:
exporter.NewSpecificVendorExporter(
"vendor.example",
"production-account",
)
A more portable arrangement uses deployment configuration:
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_SERVICE_NAME
OTEL_RESOURCE_ATTRIBUTES
sampling configuration
The same application artefact can run in development, staging and production.
Each environment can route telemetry differently.
ONE APPLICATION IMAGE
│
├── dev collector
├── staging collector
└── prod collector
This mirrors the wider build-once, promote-many principle.
Instrumentation remains part of the application.
Destination configuration remains part of the environment.
A platform should make instrumentation the easy path
A golden path for bfstore services could provide:
- OpenTelemetry SDK setup
- gRPC interceptors
- HTTP client instrumentation
- MySQL instrumentation
- Kafka context propagation
- standard resource attributes
- structured logging helpers
- collector endpoint configuration
- example dashboards
- instrumentation tests
A newly generated service should begin with:
request tracing
runtime metrics
trace-correlated logs
service identity
deployment metadata
The developer should add domain-specific instrumentation where it matters.
PLATFORM PROVIDES
technical telemetry foundations
APPLICATION TEAM PROVIDES
business meaning
This is a strong platform boundary.
The platform standardises the plumbing.
The service owner explains what successful work means.
A bfstore observability path
Suppose the catalog service receives:
GetProduct
The gRPC server instrumentation creates a server span.
The MySQL client instrumentation creates a child span.
The application adds a domain event when the product does not exist.
CatalogService/GetProduct
└── MySQL SELECT product
The service emits:
- a request-duration metric
- a request-result counter
- a structured not-found log
- a distributed trace
The SDK exports using OTLP:
CATALOG SERVICE
│
▼
OTLP
│
▼
BFSTORE COLLECTOR
The collector adds environment metadata:
project:
bfstore
environment:
prod
cluster:
workloads-prod
It routes the signals:
LOGS
│
▼
LOG STORAGE
METRICS
│
▼
METRICS STORAGE
TRACES
│
▼
TRACE STORAGE
Later, the trace backend changes.
The catalog code does not.
The platform updates the collector exporter and validates the new destination.
BEFORE
OTLP → collector → trace backend A
AFTER
OTLP → collector → trace backend B
Dashboards and queries may still require migration.
The instrumentation boundary remains stable.
Testing the portability claim
I would test an OpenTelemetry design with several questions.
API dependency
Does application code depend on
OpenTelemetry APIs or a backend SDK?
Transport
Can the application export using
a common protocol such as OTLP?
Routing
Can destinations change through
collector or environment configuration?
Propagation
Does trace context cross every
supported service boundary?
Vocabulary
Are common attributes aligned
with shared semantic conventions?
Domain meaning
Does custom instrumentation explain
business outcomes as well as technical calls?
Cardinality
Are metric and resource attributes
bounded and safe?
Privacy
Could secrets, payloads or personal data
enter spans, logs or baggage?
Failure isolation
Can the workload continue when
telemetry export fails?
Pipeline health
Can operators detect dropped,
rejected or delayed telemetry?
Migration
Which parts remain backend-specific,
including queries, dashboards and alerts?
These questions make “instrument once” a design claim that can be tested rather than a slogan printed on a collector-shaped mug.
The mental model I am keeping
My earlier model was:
OBSERVABILITY INTEGRATION
install the SDK
provided by the backend
The stronger model is:
APPLICATION BEHAVIOUR
│
▼
OTEL INSTRUMENTATION
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
LOGS METRICS TRACES
│ │ │
└────────────────┼────────────────┘
▼
OTLP
│
▼
COLLECTOR
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
PROCESS ROUTE EXPORT
│ │ │
└────────────────┼────────────────┘
▼
CHOSEN BACKENDS
The API answers:
How does application code
describe telemetry?
The SDK answers:
How is telemetry produced,
sampled and processed locally?
Context propagation answers:
How do several processes contribute
to one distributed trace?
Semantic conventions answer:
Which common vocabulary
describes standard operations?
OTLP answers:
How can telemetry travel
through a common protocol?
The collector answers:
Where should routing, enrichment,
sampling and export occur?
The backend answers:
How is telemetry stored,
queried and presented?
Platform conventions answer:
Which instrumentation choices
should remain consistent across services?
Application teams answer:
Which domain events and outcomes
give the telemetry meaning?
And portability answers:
Which parts can change without
rewriting every workload?
OpenTelemetry does not make observability storage irrelevant.
The backend still determines how telemetry is retained, queried, visualised and operated.
It does not make instrumentation automatic.
Teams still need to identify meaningful operations, propagate context and control sensitive attributes.
It does not make every backend equivalent.
Queries, dashboards, alerts and cost models remain different.
What it changes is the coupling.
The application no longer needs to begin its observability design by choosing a storage vendor’s private vocabulary.
It can begin with its own behaviour:
a request arrived
a dependency was called
a message was published
a payment became uncertain
an order completed
OpenTelemetry turns those observations into common signals.
OTLP carries them across a standard boundary.
The collector applies platform policy and sends them to the destinations chosen for the environment.
Instrument once does not mean decide once. It means preserve the freedom to change where telemetry goes without teaching every application a new observability language.
That is the promise I find useful.
Not backend independence as a fantasy, but backend flexibility as an architectural property.