Distributed Communication Patterns#

Before any pattern makes sense you need one idea. A local function call has two outcomes: it returns, or it throws. A remote call has three: it returns, it fails, or you never find out. That third outcome does not exist inside a process, it cannot be eliminated, and every single pattern in this course is machinery for surviving it.

Notice what went wrong there. Nobody wrote a bug. The server did its job, the network dropped one packet out of billions, and the client did the textbook thing. The double charge is an emergent property of a correct client talking to a correct server over an imperfect wire.

• A timeout is not an errorAn HTTP 500 tells you the work failed. A timeout tells you nothing — the request may never have arrived, may have failed, or may have completely succeeded. Code that treats them identically is code that double-charges people.
• Slow is worse than downA dead service gives you a fast connection-refused, which is easy to handle. A service answering in 30 seconds holds your threads, your connections and your memory hostage, and takes your service down with it. Grey failure is the hard case.

What the network does to your numbers#

• In-process call ~1 ns
• Same-datacentre RPC ~0.5 ms
• Cross-region RPC ~70 ms
• 4G round trip ~180 ms

Those numbers are six orders of magnitude apart, and the spread is why a refactor that “just” moves a method into another service can make a page ten times slower. But latency is only half of it. The other half is arithmetic:

• Sync latency sum of the chain
• Sync availability product of the chain
• Async latency one broker write
• Async consistency eventual

This is the first and biggest fork in the road, and it is not really a technology choice. It is a choice about who waits. In a synchronous call the caller holds a connection open until the work is finished. In an asynchronous one the caller hands the work to a broker, gets an acknowledgement, and leaves.

• Choose synchronous when the caller needs the answerA price, an authorisation, a seat availability, a login. If the next line of the caller's code cannot run without the result, a queue only adds machinery around a wait you still have to do.
• Choose asynchronous when the caller needs an acknowledgementSend the email, update the search index, recalculate the recommendations, emit the audit record. Nobody is watching, so do not make a customer wait for it — and do not let its outage become yours.
• The sync trap“It is only one more call” is how a 200 ms endpoint becomes a 2 s endpoint over eighteen months, one reasonable pull request at a time.
• The async trapEventual consistency is a product problem, not just an engineering one. Somebody has to decide what the screen says while the order exists but is not yet paid for.

• REST + JSON universal
• gRPC + protobuf 4× smaller
• GraphQL one flexible endpoint

Once you have decided to make a synchronous call, you pick a style. All three of these are request/response over HTTP; they differ in who decides the shape of the response and how much the two ends must agree in advance.

The animation shows gRPC's four call shapes, because those are the ones people have not met. The encoding difference matters just as much, and it is where most of the performance argument lives:

syntax = "proto3";
package orders.v1;

service Orders {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc WatchOrder(GetOrderRequest) returns (stream OrderEvent);
}

message GetOrderRequest {
  string order_id = 1;
}

message Order {
  string order_id  = 1;
  string status    = 2;   // never renumber a tag
  int64  total_minor = 3;
  string currency  = 4;
  reserved 5;             // 5 was 'legacy_total', do not reuse
}

GET /v1/orders/A-4471

{
  "id": "A-4471",
  "status": "paid",
  "amount": 8210,
  "currency": "GBP",
  "_links": { "cancel": "/v1/orders/A-4471/cancellation" }
}

# GraphQL: the CLIENT chooses the fields, in one round trip

query {
  order(id: "A-4471") {
    status
    amount
    customer { name tier }      # would be a second REST call
    items { sku qty }           # would be a third
  }
}

• REST wins at the edgeCacheable by URL, debuggable with curl, understood by every proxy and every developer, and it survives a receiver who has never seen your schema. For a public API this is usually the end of the discussion.
• gRPC wins between your own servicesGenerated clients in every language, a schema that CI can check for breaking changes, HTTP/2 multiplexing, streaming, and 4× smaller payloads. Deadlines and cancellation are built into the protocol rather than bolted on.
• GraphQL wins for varied clientsWhen an iOS screen, an Android screen and a web dashboard all need different slices of the same graph, letting the client specify the slice beats maintaining three bespoke endpoints.
• And the costsgRPC is awkward from a browser (you need grpc-web or a gateway) and uncacheable. GraphQL moves query cost to runtime — an innocent query can fan out into an N+1 storm, so you need dataloaders, depth limits and cost analysis before you expose it publicly.

• Discovery a freshness problem
• Round robin counts turns
• Least connections counts work

“Call the orders service” hides two questions. Where is it? — its instances are ephemeral, rescheduled constantly, and their addresses change every deploy. And which one? — because there are eleven of them and they are not equally healthy.

Then you pick one. The default everyone inherits is round robin, and it fails in a specific, common way that is worth seeing once:

• No timeout wait forever
• Per-hop timeout wasted work
• Propagated deadline nothing wasted

A timeout is the only way to convert the third outcome into the second one: it turns “I do not know” into a definite failure you can act on. Every remote call needs one. But a timeout set independently at each hop produces a subtle, expensive bug.

# The caller sets ONE budget for the whole request tree.
async def handle_request(request):
    deadline = time.monotonic() + 3.0          # absolute, not a duration

    price = await get_price(request.sku, deadline)
    stock = await get_stock(request.sku, deadline)
    return render(price, stock)


async def get_price(sku, deadline):
    remaining = deadline - time.monotonic()
    if remaining <= 0.05:                      # not enough time to be useful
        raise DeadlineExceeded()               # fail now, do not start the call

    # Pass the remainder down; the next hop subtracts its own elapsed time.
    async with httpx.AsyncClient() as client:
        return await client.get(
            f"{PRICING}/price/{sku}",
            timeout=remaining,
            headers={"X-Request-Deadline-Ms": str(int(remaining * 1000))},
        )

// The caller sets ONE budget for the whole request tree.
async function handleRequest(req) {
  const deadline = Date.now() + 3000;          // absolute, not a duration

  const price = await getPrice(req.sku, deadline);
  const stock = await getStock(req.sku, deadline);
  return render(price, stock);
}

async function getPrice(sku, deadline) {
  const remaining = deadline - Date.now();
  if (remaining <= 50) throw new DeadlineExceeded(); // fail now, do not start

  // AbortController is how you actually cancel in-flight work in JS.
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), remaining);
  try {
    return await fetch(`${PRICING}/price/${sku}`, {
      signal: ac.signal,
      headers: { "X-Request-Deadline-Ms": String(remaining) },
    });
  } finally {
    clearTimeout(timer);
  }
}

• Immediate retry load amplifier
• Exponential still synchronised
• Full jitter smooth
• Retry budget ≤ 10% of traffic

Retrying is the obvious response to a failed call and the easiest way to turn a small problem into an outage. The difference between the two is entirely in when you retry.

Step through all three variants above. The first is a load amplifier aimed at the thing that is already struggling. The second fixes the volume and leaves every client synchronised. Only the third — adding randomness — produces the smooth ramp a recovering service can actually survive.

RETRYABLE = {408, 429, 500, 502, 503, 504}

async def call_with_retries(fn, *, attempts=4, base=0.2, cap=10.0):
    for attempt in range(attempts):
        try:
            return await fn()
        except HttpError as e:
            # Never retry a 400 or a 422 - the request itself is wrong,
            # and it will be exactly as wrong the second time.
            if e.status not in RETRYABLE or attempt == attempts - 1:
                raise
            # Honour the server if it told us when to come back.
            wait = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
        except (TimeoutError, ConnectionError):
            if attempt == attempts - 1:
                raise
            wait = random.uniform(0, min(cap, base * 2 ** attempt))

        await asyncio.sleep(wait)

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

async function callWithRetries(fn, { attempts = 4, base = 200, cap = 10_000 } = {}) {
  for (let attempt = 0; attempt < attempts; attempt += 1) {
    try {
      return await fn();
    } catch (err) {
      const last = attempt === attempts - 1;
      const retryable = err.status === undefined || RETRYABLE.has(err.status);
      // Never retry a 400 or a 422 - the request itself is wrong.
      if (last || !retryable) throw err;

      // Full jitter: a uniform sample from the whole backoff window.
      const window = Math.min(cap, base * 2 ** attempt);
      const wait = err.retryAfterMs ?? Math.random() * window;
      await new Promise((r) => setTimeout(r, wait));
    }
  }
}

The rule that people miss is that retries compose multiplicatively. Three layers each retrying three times is not three retries, it is twenty-seven:

• Safe to retry GET, PUT, DELETE
• Not safe POST charge, send
• Fix client-minted key

Retries and at-least-once delivery both mean the same thing: your handler will be called more than once with the same input, and not occasionally — routinely, under load, exactly when it matters. Idempotency is the property that makes that harmless, and it is the single highest-leverage idea in this course.

@app.post("/charges")
async def create_charge(body: ChargeRequest, idempotency_key: str = Header(...)):
    async with db.transaction():                      # ONE transaction
        try:
            # The unique index is the lock. Never "select then insert" -
            # two concurrent retries will both pass that check.
            await db.execute(
                "insert into idempotency_keys (key, request_hash) values ($1, $2)",
                idempotency_key, hash_of(body),
            )
        except UniqueViolation:
            stored = await db.fetchrow(
                "select response, request_hash from idempotency_keys where key = $1",
                idempotency_key,
            )
            # Same key, different body = a client bug worth shouting about.
            if stored["request_hash"] != hash_of(body):
                raise HTTPException(422, "idempotency key reused with a different body")
            return json.loads(stored["response"])     # replay the original answer

        charge = await ledger.charge(body.amount, body.currency, body.card_token)
        await db.execute(
            "update idempotency_keys set response = $1 where key = $2",
            json.dumps(charge), idempotency_key,
        )
        return charge

app.post("/charges", async (req, res) => {
  const key = req.header("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });

  await db.transaction(async (tx) => {              // ONE transaction
    try {
      // The unique index is the lock. Never "select then insert" -
      // two concurrent retries will both pass that check.
      await tx.query(
        "insert into idempotency_keys (key, request_hash) values ($1, $2)",
        [key, hashOf(req.body)],
      );
    } catch (err) {
      if (err.code !== UNIQUE_VIOLATION) throw err;
      const { rows } = await tx.query(
        "select response, request_hash from idempotency_keys where key = $1", [key],
      );
      // Same key, different body = a client bug worth shouting about.
      if (rows[0].request_hash !== hashOf(req.body)) {
        return res.status(422).json({ error: "key reused with a different body" });
      }
      return res.json(rows[0].response);             // replay the original answer
    }

    const charge = await ledger.charge(req.body);
    await tx.query("update idempotency_keys set response = $1 where key = $2",
      [charge, key]);
    res.json(charge);
  });
});

• Naturally idempotent — do this where you canPUT /orders/44 {status:"shipped"} sets an absolute state, so running it twice is identical to running it once. Prefer absolute assignments (set balance = 90) over relative ones (balance = balance - 10) whenever the domain allows.
• Needs an explicit keyAnything that appends, sends, charges or emits. POST /charges, POST /emails, append to ledger. There is no way to make these safe without remembering that you already did them.

• Breaker stop calling
• Bulkhead cap the damage
• Shedding reject early

Timeouts, retries and idempotency make a single call survivable. These two patterns stop a failing dependency from consuming the calling service itself. A circuit breaker decides whether to call at all; a bulkhead decides how much of you can be tied up when the answer is yes.

The breaker keeps you from calling a broken thing. The bulkhead keeps the calls you do make from eating every resource you have:

• Queue one consumer gets it
• Topic everyone gets a copy
• Log + replay

A broker gives you three things at once: it decouples in time (the receiver need not be up), it decouples in identity (the sender need not know who receives), and it absorbs load (a spike becomes a backlog instead of an error page). What you choose between is not really a product but a delivery semantic.

Now the same infrastructure with the opposite semantic. A queue splits work between consumers; a topic copies it to all of them:

The third shape is the log — Kafka, Pulsar, Kinesis. It looks like a topic but it does not delete what it delivers, and that one difference changes everything downstream of it:

• Choose a queuewhen work is disposable once done, consumers are interchangeable, and you want per-message retries and a dead letter queue for free. Job processing, emails, thumbnails.
• Choose a logwhen the history is the product: multiple independent readers, replay after a bug, ordering within a key, or feeding both a real-time consumer and a nightly analytics job from the same data.
• Log costsParallelism is capped by partition count, which you must choose up front. Rebalances produce duplicates. And there is no per-message retry — one poison record blocks its whole partition unless you handle it yourself.

• At most once may lose
• At least once may duplicate
• Exactly once only as an effect

Every broker's guarantee comes down to one decision in the consumer: do you acknowledge before or after doing the work? That is the whole thing. Everything else is marketing.

Ordering has the same character — a guarantee that is narrower than people assume, and silently lost by a default setting:

# 1. Same entity -> same key -> same partition -> ordered.
await producer.send(
    topic="order-events",
    key=order_id.encode(),          # NOT a random uuid, NOT round robin
    value=event.to_json(),
    headers=[("event-id", event.id.encode()), ("traceparent", ctx.encode())],
)


# 2. Belt and braces: reject stale events even if they do arrive in order.
async def handle(event):
    updated = await db.execute(
        """update orders
              set status = $1, version = $2
            where id = $3 and version < $2""",   # monotonic guard
        event.status, event.version, event.order_id,
    )
    if updated == 0:
        log.info("stale or duplicate event ignored", event_id=event.id)

// 1. Same entity -> same key -> same partition -> ordered.
await producer.send({
  topic: "order-events",
  messages: [{
    key: orderId,                   // NOT a random uuid, NOT round robin
    value: JSON.stringify(event),
    headers: { "event-id": event.id, traceparent: ctx },
  }],
});

// 2. Belt and braces: reject stale events even if they do arrive in order.
async function handle(event) {
  const { rowCount } = await db.query(
    `update orders
        set status = $1, version = $2
      where id = $3 and version < $2`,          // monotonic guard
    [event.status, event.version, event.orderId],
  );
  if (rowCount === 0) log.info({ eventId: event.id }, "stale or duplicate ignored");
}

• Poison message fails forever
• DLQ unblocks the stream
• Unbounded queue hidden overload

Retries assume failure is transient. Some failures are not — a malformed payload will fail identically forever, and while it is being retried it blocks everything behind it.

The other broker failure mode is quieter, and it is the one that kills processes at 3 a.m.: a producer that is simply faster than its consumer.

• Dual write no correct version
• Outbox one transaction
• CDC zero app changes

Here is the most common bug in event-driven systems, and it is in code that looks completely reasonable: save to the database, then publish an event. Two systems, two commits, and no transaction that spans them.

async def place_order(order):
    async with db.transaction():                  # ONE transaction, ONE system
        await db.execute(
            "insert into orders (id, customer_id, total_minor) values ($1, $2, $3)",
            order.id, order.customer_id, order.total_minor,
        )
        await db.execute(
            """insert into outbox (id, topic, partition_key, payload)
               values ($1, $2, $3, $4)""",
            uuid.uuid4(), "order-events", order.id, order.to_event_json(),
        )
    # No publish here. Nothing to fail. Nothing to retry. Nothing to get wrong.


# A separate process, restartable at any moment, publishes what committed.
async def relay():
    while True:
        rows = await db.fetch(
            "select * from outbox where published_at is null order by id limit 100"
        )
        for row in rows:
            await broker.publish(row["topic"], key=row["partition_key"],
                                 value=row["payload"], headers={"event-id": row["id"]})
            # A crash here republishes on restart - consumers dedupe on event-id.
            await db.execute("update outbox set published_at = now() where id = $1",
                             row["id"])
        await asyncio.sleep(0.2)

async function placeOrder(order) {
  await db.transaction(async (tx) => {          // ONE transaction, ONE system
    await tx.query(
      "insert into orders (id, customer_id, total_minor) values ($1, $2, $3)",
      [order.id, order.customerId, order.totalMinor],
    );
    await tx.query(
      `insert into outbox (id, topic, partition_key, payload)
       values ($1, $2, $3, $4)`,
      [randomUUID(), "order-events", order.id, order.toEventJson()],
    );
  });
  // No publish here. Nothing to fail. Nothing to retry. Nothing to get wrong.
}

// A separate process, restartable at any moment, publishes what committed.
async function relay() {
  for (;;) {
    const { rows } = await db.query(
      "select * from outbox where published_at is null order by id limit 100",
    );
    for (const row of rows) {
      await broker.publish(row.topic, {
        key: row.partition_key, value: row.payload, headers: { "event-id": row.id },
      });
      // A crash here republishes on restart - consumers dedupe on event-id.
      await db.query("update outbox set published_at = now() where id = $1", [row.id]);
    }
    await sleep(200);
  }
}

The outbox converts an impossible problem into a solved one plus a tolerable one: atomicity within a single database, and at-least-once publishing that consumers deduplicate. The alternative, if you cannot change the application at all, is to let the database's own log be the source of events:

• Steps local transactions
• Rollback does not exist
• Instead compensations

One business operation, four services, four databases. You cannot wrap that in a transaction, so you give up atomicity and replace it with a sequence of local transactions, each with a matching compensation that semantically offsets it.

• OrchestrationOne component owns the state machine and calls each step. Easy to see, easy to debug, easy to add a timeout or a manual intervention step. Use it for anything long, critical, or with more than three steps.
• ChoreographyEach service reacts to events and publishes its own. Maximum decoupling, and adding a participant needs no change anywhere else. Good for two or three stable steps.
• Orchestration's costThe orchestrator knows about everyone, so it becomes a place logic accumulates and a service every team must change.
• Choreography's costThe flow exists nowhere as a readable artefact. “Why did this order stall?” means correlating logs across five services, and cyclic subscriptions become loops nobody designed.

For contrast, the mechanism sagas replace — two-phase commit — is worth understanding precisely so you can explain why you did not use it:

• Polling simple, wasteful
• Long polling push over plain HTTP
• SSE server → client
• WebSocket both ways

Everything so far has been servers talking to servers. The last hop is different, because the client is behind a NAT, on a flaky radio, and cannot be called. So the connection must start from their side, and the question becomes how long you keep it open: a fresh request every few seconds (polling), one request the server holds until it has news (long polling), one response that never ends (server-sent events), or a connection that stops being HTTP and carries messages both ways (WebSockets).

• Strength — SSE is just HTTPIt carries your cookies, auth, CORS rules, tracing and HTTP/2 multiplexing unchanged, and EventSource reconnects by itself, sending Last-Event-ID so the server can resume.
• Strength — WebSockets are truly two-wayEither side sends whenever it likes with a few bytes of framing, which is what chat, collaborative editing and games need.
• Weakness — SSE is one-way textThe client still sends with ordinary requests, binary costs a base64 tax, and EventSource cannot set an Authorization header.
• Weakness — a WebSocket is a protocol you now ownAfter the 101 there are no status codes, retries, acks or per-request auth. Envelopes, acks, heartbeats, resume and reconnect are all yours to build.

The detail that separates a demo from a production stream is what happens on reconnect. Connections will drop — every deploy drops all of them at once — so each event needs an id, and the server needs a backlog it can replay from:

• Webhook server → server
• Retry window hours, not ms
• Long job 202 + status URL
• Reply matching correlation id

Two patterns for when the answer cannot come back on the connection that asked for it. A webhook is used when the receiver is another company's server: you make the HTTP call, to an endpoint you do not control and cannot debug. Asynchronous request-reply is used when the work outlasts any sane timeout: you return a handle immediately (202 Accepted plus a status URL) and deliver the result later — by the client polling, by a push, or by a webhook. Inside a system the same shape runs over queues, with a reply_to queue and a correlation id that matches each answer to its question.

• Strength — nobody holds a connectionThe caller is released in milliseconds, the work runs on workers that scale independently, and a deploy in the middle costs nothing because the job lives in a database.
• Strength — the server sets the paceRetry-After on every status response lets you slow every poller down during an incident without a client release.
• Weakness — a resource you now ownThe job handle must survive restarts, be idempotent to create, expose progress and errors, and expire explicitly with 410 Gone.
• Weakness — webhooks fail on their schedule, not yoursReceivers are down for hours, deliveries arrive out of order and more than once, and the callback URL is an SSRF vector you have to validate.

Everything above, compressed into the three decisions you actually make.

Pick a communication style#

Pick a reliability pattern#

The questions to ask of any design#

Where to go next#

1. 📘 The Distributed Communication Patterns Detailed Course — the same ground in 47 sections, plus transports and HTTP/3, polling, server-sent events, WebSockets and async request-reply in depth, service meshes, consensus and quorums, CQRS and event sourcing, scatter-gather and tail latency, distributed tracing, contract testing and a practice roadmap.
2. 📚 Browse both guides in the Distributed Communication Courses catalog.
3. 📖 AWS Builders' Library — Timeouts, retries, and backoff with jitter — the measured data behind section 5, from people running it at planetary scale.
4. 🧩 microservices.io pattern catalog — Chris Richardson's reference for saga, outbox, API gateway and their variants.
5. 🧠 Martin Fowler — What do you mean by “event-driven”? — the clearest short treatment of the four things people mean by that phrase.

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." — Leslie Lamport