Once two parts of a system need to communicate, the diagrams become deceptively simple.
service A ─────────► service B
One box talks to another box.
Architecture achieved.
Except the arrow leaves nearly every important question unanswered.
What does service A send?
Does A need an answer immediately?
Must B be available at that moment?
What does success mean?
What happens if B performs the work but A never receives the reply?
Can several systems receive the same information?
Can the receiver process the work later?
Can the sender safely try again?
The arrow is not the architecture.
The interaction contract is.
Terms such as these appear constantly:
API
HTTP
RPC
messaging
They are often compared as peers even though they describe different layers of a system.
A useful first separation is:
SEMANTIC INTENT
query
command
event
INTERACTION STYLE
request-response
one-way handoff
publish-subscribe
INTERFACE CONTRACT
operations
schemas
errors
compatibility
APPLICATION PROTOCOL
HTTP
gRPC protocol
AMQP
Kafka protocol
TRANSPORT
TCP
QUIC
local IPC
The boundaries are not perfectly rigid, but the layers answer different questions.
An API is an interface exposed by one piece of software for another to use.
RPC is an interaction style that presents remote operations as procedure or method calls.
Messaging is an interaction style in which systems communicate by producing and consuming messages, often through intermediary infrastructure.
An RPC service has an API.
A producer and consumer share a message contract that forms an API between them.
An HTTP endpoint exposes an API, but not every API uses HTTP.
The interesting decision is therefore not:
Which fashionable communication technology should I use?
It is:
What relationship should exist between the communicating systems?
The interaction contract comes first
An API defines what another software component is allowed to ask, provide or observe.
A programming library exposes an API:
product, err := catalog.FindProduct(ctx, productID)
An operating system exposes APIs through system calls:
read(fd, buffer, count);
A database driver exposes an API:
rows, err := db.QueryContext(ctx, query)
A remote service can expose an API over HTTP:
GET /products/42 HTTP/1.1
Host: api.example.com
A gRPC service can expose an API through a service definition:
service CatalogService {
rpc GetProduct(GetProductRequest)
returns (GetProductResponse);
}
A messaging boundary can expose a contract through an event schema:
message ProductPriceChanged {
string product_id = 1;
int64 previous_price_minor = 2;
int64 new_price_minor = 3;
}
These interfaces define some combination of:
available operations
accepted inputs
returned outputs
message shapes
error behaviour
compatibility expectations
The transport determines how information moves.
The API determines what the interaction means.
Once the interface crosses a process or machine boundary, the contract must also account for distributed failure.
A remote interaction creates uncertainty
A local function call stays inside one process:
caller
│
▼
function
│
▼
return value
A remote interaction crosses several boundaries:
caller process
│
▼
client library
│
▼
network
│
▼
server process
│
▼
application logic
The caller may fail.
The server may fail.
The network may delay or interrupt the exchange.
A proxy, gateway or broker may fail.
The request may arrive while the response does not.
The client may stop waiting while the server continues working.
The most important example is simple.
Suppose a client sends:
CreateOrder
The server:
- receives the request;
- creates the order;
- commits it;
- sends the response.
Then the connection fails before the response reaches the client.
The client observes:
timeout
The server state may be:
order created
A timeout tells the caller:
I did not receive a timely answer.
It does not prove:
The receiver did no work.
This ambiguity is central to remote communication.
It drives the need for:
deadlines
cancellation
retries
idempotency
operation identifiers
status queries
observability
A remote API is therefore partly a software interface and partly a distributed-systems agreement.
Cancellation is not rollback
A client can stop waiting for a request or cancel an RPC.
That cancellation may reach the server.
But it does not automatically undo work already performed.
client cancels
│
├── server may notice before starting
├── server may stop cooperatively
└── server may already have committed the effect
Cancellation often means:
The caller no longer needs or waits for this result.
It does not necessarily mean:
Reverse every side effect.
An API that supports cancellation still needs clear semantics for work that has already started or completed.
Convenient syntax should not become distributed amnesia.
Direct request-response
In a direct request-response interaction, a client contacts a logical service or endpoint and waits for an answer.
CLIENT
│
│ request
▼
SERVER
│
│ response
▼
CLIENT
The concrete server instance may be selected through DNS, service discovery, a gateway or a load balancer.
The caller still depends on the logical service participating during the interaction.
That creates temporal coupling:
The participants need to be available at overlapping times.
Sometimes that is exactly what the business operation requires.
The caller may need to know:
Does this product exist?
What is its current price?
Is this basket valid?
Was this payment authorisation accepted?
Waiting for a direct answer is not an architectural mistake when the workflow genuinely depends on that answer.
HTTP provides a general request-response protocol
An HTTP request contains:
method
target
headers
optional body
A response contains:
status code
headers
optional body
For example:
GET /products/42 HTTP/1.1
Host: shop.example
Accept: application/json
Response:
HTTP/1.1 200 OK
Content-Type: application/json
{
"id": "42",
"name": "Gopher reading chair",
"price": 24900
}
One common HTTP API style organises interactions around resources:
/products
/products/42
/customers/72
/orders/5001
Methods then carry standardised semantics:
GET /products/42
Retrieve a representation.
POST /orders
Submit information for processing, often creating a new subordinate resource.
PUT /customers/72/preferences
Replace the state represented at a known target.
DELETE /basket/items/103
Request removal of a resource or relationship.
These method properties matter when reasoning about retries:
SAFE
intended not to change application state
IDEMPOTENT
repeating the same request has the same intended effect
CACHEABLE
a response may be reused under defined conditions
GET is defined as safe and idempotent.
PUT and DELETE are idempotent by method semantics.
POST is not inherently idempotent, although a particular API can make a POST operation safely repeatable through an idempotency contract.
HTTP gives the interaction useful protocol semantics.
It does not decide the complete business meaning for me.
HTTP API does not mean REST API
These are not synonyms:
HTTP API
REST API
JSON API
HTTP is an application protocol.
JSON is one possible representation format.
REST is an architectural style with constraints extending beyond:
uses HTTP methods
An HTTP API might expose an operation-shaped route:
POST /orders/5001/cancel
or:
POST /calculate-shipping-quote
It is still an HTTP API.
I do not need to award every endpoint a REST purity certificate.
I need to ask whether the interface communicates its behaviour clearly and uses HTTP semantics deliberately.
An HTTP response may mean completed or accepted
Consider:
POST /reports
The server could complete the operation before replying:
request
│
▼
generate report
│
▼
store report
│
▼
return completed result
Or it could record responsibility for the work and process it later:
request
│
▼
record report job
│
▼
202 Accepted
│
▼
generate report later
A useful asynchronous HTTP contract often provides a way to observe progress:
HTTP/1.1 202 Accepted
Location: /report-jobs/8f31
The client can then query the job resource.
The response must state which milestone has been reached:
request received
request validated
work accepted
work completed
result persisted
A successful network response is not automatically proof that every desired downstream effect has finished.
RPC makes remote behaviour callable
RPC stands for remote procedure call.
It lets a client invoke an operation implemented by another process or machine through an interface resembling a procedure or method call.
A service definition might contain:
service InventoryService {
rpc ReserveStock(ReserveStockRequest)
returns (ReserveStockResponse);
}
Client code might look conceptually like:
response, err := inventoryClient.ReserveStock(
ctx,
&inventoryv1.ReserveStockRequest{
ProductId: "chair-42",
Quantity: 1,
},
)
The local-looking call hides several steps:
serialise request
│
▼
send over network
│
▼
server receives request
│
▼
invoke implementation
│
▼
serialise response
│
▼
send response
A remote call is not a local call.
It can fail because:
DNS resolution failed
connection failed
request timed out
server unavailable
response was lost
load balancer reset the connection
client cancelled
server completed after the client deadline
A good RPC interface makes these concerns visible through mechanisms such as:
context
deadline
cancellation
status code
metadata
retry policy
Some RPC frameworks also support client, server or bidirectional streaming. Streaming changes the shape of the exchange, but not the need to reason about remote timing, cancellation and partial failure.
RPC contracts and operation size
RPC interfaces commonly describe explicit capabilities:
service OrderService {
rpc CreateOrder(CreateOrderRequest)
returns (CreateOrderResponse);
rpc CancelOrder(CancelOrderRequest)
returns (CancelOrderResponse);
rpc GetOrder(GetOrderRequest)
returns (GetOrderResponse);
}
An interface definition language can describe the contract in a language-neutral form:
syntax = "proto3";
package inventory.v1;
service InventoryService {
rpc CheckAvailability(CheckAvailabilityRequest)
returns (CheckAvailabilityResponse);
}
message CheckAvailabilityRequest {
string product_id = 1;
uint32 requested_quantity = 2;
}
message CheckAvailabilityResponse {
bool available = 1;
uint32 available_quantity = 2;
}
Tools can generate:
client code
server interfaces
serialisation code
message types
The generated code defines structure.
It cannot decide what this means:
available = false
Possible meanings include:
product does not exist
product exists but stock is zero
requested quantity is too large
inventory service is degraded
The semantics still need human design.
RPC can also tempt me into a chatty interface:
GetCustomer
GetAddress
GetBasket
GetProduct
CheckInventory
GetShippingMethod
Each innocent-looking method is a network round trip with its own latency and failure probability.
A remote interface should be designed around useful network-sized operations, not copied from an in-process object model.
Machine-readable contracts are not exclusive to RPC. HTTP APIs can also be described using OpenAPI, supporting documentation, validation, client generation and testing.
The interaction style and the contract-description technology are separate decisions.
Brokered messaging changes the relationship
Direct communication usually looks like:
sender ─────────► receiver
Brokered messaging adds another role:
PRODUCER
│
│ publish message
▼
BROKER
│
│ retain, route or deliver
▼
CONSUMER
The producer sends to the messaging system rather than directly invoking a consumer instance.
Depending on the platform, the broker may provide some combination of:
routing
buffering
retention
delivery
consumer coordination
acknowledgement
The producer may not need to know:
which consumer instance processes the message
when processing begins
how many consumers are interested
“Messaging” is still a family of models rather than one universal set of guarantees.
A message contract includes meaning
Consider:
{
"order_id": "5001",
"customer_id": "72",
"occurred_at": "2021-06-27T14:32:00Z"
}
Valid JSON does not explain:
What kind of message is this?
Who produced it?
What does publication mean?
Can fields be absent?
Can it be delivered more than once?
Does order matter?
How long is it retained?
What should happen when processing fails?
A useful message contract may include:
message name
schema
semantic meaning
version
identifier
timestamp
source
correlation information
The message format defines structure.
The contract defines meaning.
Commands and events have different intent
A command asks a capability to perform work:
ReserveInventory
SendOrderConfirmation
GenerateInvoice
It expresses:
Please perform this action.
A command usually has one logical handler, even if several consumer instances compete to process it.
An event reports a fact that has already occurred:
OrderCreated
InventoryReserved
PaymentAuthorised
It expresses:
This state transition happened.
Several independent consumers may react:
OrderCreated
│
├──► notification
├──► analytics
└──► fulfilment
An event should not be a disguised instruction.
A message called:
OrderCreated
should not secretly mean:
please try to create an order
The fact should be durably true according to the producer’s contract before it is published as an event.
Events are normally immutable records. A later correction is another fact, not an edit that rewrites history for whichever consumers already observed the original message.
Queue-style work and publish-subscribe
One messaging pattern distributes work among competing consumers:
QUEUE
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
worker 1 worker 2 worker 3
Each delivery attempt is normally assigned to one consumer instance.
The same logical message may still be delivered again after failure.
This pattern fits:
background jobs
workload distribution
burst absorption
retryable processing
Another pattern distributes one fact to several independent consumers:
EVENT
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
consumer A consumer B consumer C
This can let new consumers react without changing the original producer.
It also creates responsibilities around:
schema evolution
consumer ownership
retention
replay
duplicates
ordering
Decoupling one dependency introduces another contract that still needs governance.
Messaging changes coupling rather than eliminating it
Durable intermediary storage can reduce temporal coupling:
producer publishes at 14:00
│
▼
broker retains message
│
▼
consumer starts at 14:10
│
▼
consumer processes message
The producer and consumer do not need to be available simultaneously.
But they remain coupled through:
message schema
message meaning
topic or queue name
routing keys
ordering assumptions
delivery policy
retention
There are several useful kinds of coupling:
LOCATION COUPLING
Must the sender know the receiver's address?
TEMPORAL COUPLING
Must both participate at the same time?
SCHEMA COUPLING
Must both understand the same shape?
SEMANTIC COUPLING
Must both agree on what the interaction means?
DEPLOYMENT COUPLING
Must they be upgraded together?
A broker is not a machine for turning unclear contracts into independent systems.
Backlogs store waiting and overload
Messaging can absorb a burst:
messages arrive quickly
│
▼
backlog grows
│
▼
consumers process at a sustainable rate
Asynchronous architecture does not remove waiting.
It relocates and stores it.
That only works while the system has enough:
storage
retention
consumer capacity
recovery time
A queue converts immediate overload into stored backlog only while those limits remain healthy.
The system needs to observe:
message age
consumer lag
failed deliveries
retry volume
poison messages
storage capacity
Direct systems need their own overload controls:
bounded concurrency
rate limits
deadlines
admission control
load shedding
Messaging systems may also need producer throttling and bounded retention.
No communication style creates infinite capacity.
Acknowledgement is not completion
Messaging introduces several different acknowledgements:
producer → broker acknowledgement
broker → consumer delivery
consumer → broker acknowledgement
consumer → business system completion
A producer acknowledgement might mean:
broker received the message in memory
broker stored it durably
a replica stored it
A consumer acknowledgement might mean:
consumer received the message
consumer finished local processing
consumer committed a database change
Neither automatically proves that every downstream business effect completed.
HTTP and RPC responses have the same problem in another form.
The communication contract must say which milestone success represents.
Two systems can otherwise both report success while meaning entirely different things.
Delivery guarantees need a boundary
Messaging systems are often described with three delivery models.
AT-MOST-ONCE
loss is possible
duplicate delivery is avoided
AT-LEAST-ONCE
redelivery is possible
consumers must handle duplicates
EXACTLY-ONCE
meaningful only within a clearly defined boundary
An exactly-once guarantee inside a broker, log or database transaction does not automatically make an arbitrary external effect happen exactly once.
For example:
consume message
│
▼
send email
│
▼
consumer crashes before acknowledgement
│
▼
message delivered again
The broker may be behaving correctly.
The user can still receive two emails.
Delivery guarantees must name the boundary they cover.
Business idempotency remains an application concern whenever effects cross that boundary.
Retries are new attempts
A retry is not a continuation of the first request.
It is another attempt.
The receiver may observe:
attempt 1
attempt 2
even if the caller experienced:
one operation that seemed slow
For a state-changing operation, a stable operation identifier can let the receiver recognise repeated attempts at the same logical work:
operation_id = checkout-7f82...
A useful contract might say:
same identifier + same request
│
└── return original outcome
same identifier + different request
│
└── reject as conflict
A retry policy also needs to decide:
which failures are retryable
how many attempts are allowed
what the overall deadline is
whether to use backoff and jitter
how retries are observed
Immediate retries from many clients can amplify an outage:
server overloaded
│
▼
many clients retry together
│
▼
server becomes more overloaded
Idempotency makes repetition safer.
It does not make unlimited repetition wise.
Messaging consumers need duplicate handling
A consumer may complete work and fail before acknowledging the message:
consumer receives message
│
▼
updates database
│
▼
fails before acknowledgement
│
▼
message delivered again
The consumer must determine whether the business effect already occurred.
That may require:
message identifier
processed-message record
natural business key
conditional update
idempotent destination
This is the same idempotency lesson cron taught me, now travelling through a broker with considerably better branding.
A dead-letter queue can prevent an endless retry loop.
It does not complete the missing business work.
Someone or something still needs an investigation and recovery policy.
Database changes and event publication form a dual-write problem
Suppose an order service must:
1. commit OrderCreated state
2. publish OrderCreated event
If the database commits and publication fails:
database state changed
event missing
If the event is published before the database transaction fails:
event describes a fact
that never committed
Two independent writes cannot be made atomic merely by placing them next to each other in source code.
One common approach is a transactional outbox:
single database transaction
│
├── update business state
└── insert outgoing-message record
│
▼
separate publisher
│
└── retries delivery
The local state change and the intention to publish are recorded together.
The publisher may still deliver more than once, so consumers still need duplicate handling.
The outbox addresses the missing-message gap.
It does not make every downstream side effect magically exactly once.
Ordering requires a precise boundary
Messaging can expose several different kinds of order:
delivery order
processing-start order
processing-completion order
effect-commit order
They are not automatically the same.
Even if a broker delivers:
OrderCreated
OrderCancelled
in that order, parallel consumers may complete the second message first.
Retries can also delay an earlier message while later work succeeds.
An architecture should define the narrowest useful ordering boundary:
per order
per customer
per partition
per queue
per connection
not guaranteed
A total system-wide order is expensive and often unnecessary.
The useful question is:
Which related messages require ordering, and at which stage must that order hold?
Query, command and event describe intent
Three semantic categories help separate meaning from transport.
Query
A query asks for information:
GetProduct
CheckInventory
FindCustomer
It should not intentionally change business state merely because the result was requested.
The caller usually needs a response.
Command
A command asks a capability to perform work:
CreateOrder
ReserveInventory
CancelShipment
It may be sent through HTTP, RPC or messaging.
The receiver can accept or reject it.
Event
An event reports a fact that has already occurred:
OrderCreated
InventoryReserved
ShipmentCancelled
Consumers react according to their own responsibilities.
These are semantic categories, not transport categories:
QUERY
└── HTTP GET or RPC
COMMAND
└── HTTP POST, RPC or message
EVENT
└── message, event stream or webhook
The transport carries the interaction.
The semantic category explains its intent.
Choosing by interaction requirements
A simplified comparison looks like this:
| Question | Direct HTTP API or RPC | Brokered messaging |
|---|---|---|
| Does the caller normally contact a known logical service? | Yes | Usually contacts broker or platform |
| Does the caller normally wait for a direct result? | Yes | Often no |
| Must the receiver be available during the interaction? | Usually yes | Not necessarily with durable buffering |
| Can several consumers receive the same information? | Requires explicit fan-out | Often supported naturally |
| Can work be buffered during a traffic burst? | Usually needs additional queueing | Common messaging capability |
| Is the final result immediately available to the caller? | Often | Usually not without request-reply |
| Is duplicate handling still required? | Yes | Yes |
| Are schemas and semantics still shared contracts? | Yes | Yes |
| Does the network still create uncertainty? | Yes | Yes |
This table is not a winner-selection machine.
It shows that the choices create different relationships.
Direct request-response often fits when:
the caller needs an immediate answer
the operation is naturally request-response
the target capability is known
the caller must decide from the result
Messaging often fits when:
the sender need not wait for final processing
work can happen later
traffic bursts should be buffered
several consumers need the same fact
producer and consumer lifecycles should be separated
Messaging is not a cure for slow request-response design.
If an application publishes a message and immediately displays:
Order completed
before any consumer has processed it, that is not resilience.
It is a false statement with a queue behind it.
The user-facing result must match the system milestone actually reached:
Order received
Checkout processing
Order completed
Those are different contracts.
One workflow can combine styles
A checkout might use several interaction styles:
CUSTOMER
│
│ HTTP request
▼
CHECKOUT API
│
│ RPC
▼
INVENTORY SERVICE
│
│ response
▼
CHECKOUT API
│
│ publish event
▼
MESSAGE BROKER
│
├──► NOTIFICATION
├──► ANALYTICS
└──► FULFILMENT
The direct call answers something required immediately:
Can the requested stock be reserved?
The event distributes a fact to work that need not complete inside the customer request:
OrderCreated
The architecture is not selecting one communication technology for the entire system.
It is selecting an interaction contract for each boundary.
RPC is not automatically internal.
HTTP is not automatically external.
Commands can travel through messaging.
Events can arrive through HTTP webhooks.
The boundary and its requirements should select the interaction, not a slogan copied from a conference slide.
Every boundary needs ownership
Whichever style I use, somebody must own:
the contract
versioning
compatibility rules
availability expectations
error semantics
delivery guarantees
observability
deprecation
For direct APIs, that might include:
endpoint or method definitions
request and response schemas
status codes
deadlines
For messaging, it might include:
topic or queue purpose
message schema
producer guarantees
consumer expectations
retention
ordering
duplicate policy
A communication boundary with no owner becomes shared infrastructure folklore.
Folklore is charming in forests.
It is less charming during an incident.
The mental model I am keeping
My original model was:
service A
│
▼
service B
The new model asks four questions before choosing technology:
What must the sender know?
What must it wait for?
What guarantee does it receive?
What happens when the interaction only half succeeds?
Then I can choose the appropriate layers:
semantic intent
│
▼
query, command or event
│
▼
interaction style
│
▼
direct response, handoff or publication
│
▼
interface and failure contract
│
▼
protocol and transport
API, HTTP, RPC and messaging do not make the network disappear.
They do not remove the need for deadlines, truthful acknowledgements, duplicate handling or ownership.
The useful design is not hidden in the arrow.
It is the contract that explains what the arrow means.
References and further reading
HTTP message model
RFC 7230: HTTP/1.1 Message Syntax and Routing Defines the HTTP/1.1 message structure, routing model, connection handling and request-response framing used in 2021.
RFC 7231: HTTP/1.1 Semantics and Content
Defines HTTP methods, response status codes, representations and request semantics, including safe and idempotent methods and 202 Accepted.
HTTP API contracts
OpenAPI Specification 3.0.3 Defines a machine-readable format for describing HTTP APIs, including paths, operations, parameters, request bodies, responses and reusable schemas.
Remote procedure calls
gRPC: Introduction Introduces gRPC’s service-definition model, Protocol Buffers contracts and generated client and server code.
gRPC: Core concepts, architecture and lifecycle Documents unary and streaming RPCs, metadata, deadlines, cancellation and the separate conclusions clients and servers may reach about an RPC’s outcome.
Messaging protocols and brokers
OASIS AMQP Version 1.0: Overview Introduces AMQP as a wire-level protocol for interoperable business messaging.
RabbitMQ: AMQP 0-9-1 Model Explained Explains producers, exchanges, queues, bindings and consumers in RabbitMQ’s AMQP 0-9-1 messaging model.
RabbitMQ Tutorials Provides practical examples of work queues, publish-subscribe, routing, topics and messaging-based request-reply.
Event streaming
Apache Kafka 2.8 Documentation The Kafka documentation current around this article’s publication period, covering topics, partitions, producers, consumers, consumer groups, retention and Kafka’s distributed log design.
Apache Kafka 2.8: Implementation Documents Kafka’s log and message-storage implementation.
Apache Kafka 2.8: Distribution Explains replication, consumer-group coordination and distributed processing behaviour in Kafka 2.8.