Exactly-once is a lie. Here is what to do when the side effect is physical

Exactly-once delivery does not exist, and no queue vendor can sell it to you. What can exist is an exactly-once effect: a system where the message may arrive many times and the thing you cared about happens once. Getting there requires a deterministic identity and a durable record at every boundary where a message can repeat. If the effect is physical, you also need to accept a fourth outcome besides success, failure, and timeout: you do not know.

Printing is the worked example below, because printing is where the ambiguity is impossible to hide.

The impossibility is old and it is not going away

Two processes communicating over a lossy channel cannot reach certain agreement about a shared decision. That is the Two Generals' Problem, the first communication problem proven unsolvable, and every practical consequence follows from it. You get to pick your failure:

• At-most-once: send it, do not retry. You lose messages when the network drops.
• At-least-once: retry until acknowledged. You duplicate messages when the acknowledgment is what got lost.

Every system that claims exactly-once is doing at-least-once delivery plus deduplication somewhere. The useful question is not "is it exactly-once" but "where does the dedupe state live, and what happens when that state and the effect do not commit together."

Databases can fake it because the effect and the record share a transaction

Inside one database, exactly-once is cheap. INSERT ... ON CONFLICT DO NOTHING with a unique key gives you dedupe, and the dedupe record and the effect commit atomically. Either both happened or neither did. There is no window.

That property is doing all the work, and it disappears the moment the effect leaves the database. A printer cannot join your transaction. Neither can an SMS gateway, a payout rail, a door lock, or a robot arm. For all of those, the sequence is necessarily:

1. Record that you intend to act.
2. Act.
3. Record what happened.

A crash between 1 and 2 means you may print nothing while your records say you might have. A crash between 2 and 3 means you have printed and your records do not know. You cannot close that window, only make it small and make its contents recoverable.

Every repeatable boundary needs its own key and its own ledger

The common mistake is putting idempotency in one place, usually the public API, and assuming the rest of the path inherits it. It does not. Each hop has its own reason to repeat.

Boundary	Repeats because	Identity to key on	Ledger that decides
Caller to API	Client retry, connection reset, queue redelivery	The business object (this order's shipping label)	Job table, keyed by the caller's idempotency key
API to worker or dispatcher	Lease expiry, crashed worker, two servers racing	The job id	A lease with an owner and a deadline
Worker to device or provider	Reconnect, redelivery after restart, provider retry	The job id	A durable record on the device itself

Three boundaries, three ledgers. Removing any one of them reintroduces duplicates from a different direction.

Boundary one: key on the business object, not the request

An idempotency key should name the unique effect your business intends, not the network attempt that carried it. order-1042-shipping-label is a key. A fresh UUID per attempt is not, because every retry looks new.

RocketPrint is a remote printing API and desktop Station for software platforms: your backend POSTs a print job, and a small app beside the printer prints it, with no inbound networking, no drivers for raw label formats, and a durable job record. At the creation boundary we take an Idempotency-Key header. An identical request under the same key returns the original job and sets an Idempotency-Replayed response header. A different body under the same key returns 409.

That 409 is the part worth copying. Silently accepting the second payload would mean a changed shipping address quietly did nothing, which is worse than an error. A conflict tells the caller they have made one of two mistakes: they reused an identity by accident, or the document genuinely changed and needs a new intentional identity. More on key design in how to stop retries from printing duplicate shipping labels.

Boundary two: the dispatcher needs a lease, not good intentions

Once the job is saved, something has to pick it up and send it. If two application servers both scan for pending work, they will both find the same row, and the deduplication you built at the API boundary will not help, because there is only one job and it is about to be delivered twice.

The fix is a lease: a state the row enters when exactly one server claims it, recorded with an owner and a deadline. In our status list this is dispatching, and its whole purpose is that a job is never sent twice. The deadline matters as much as the claim: a worker that dies holding a lease must have it expire, or the job is stuck forever.

Boundary three: the device needs its own memory

The last hop is the one people forget. Suppose the job reaches the machine next to the printer, the print command succeeds, and the process dies before its acknowledgment reaches the server. On reconnect the server has a job it believes is undelivered, so it delivers it again. A stateless agent prints a second label.

So the agent keeps its own durable record of finished jobs across restarts, and a redelivered job it already knows about replays the recorded outcome instead of printing again. The server's ledger cannot do this job, because the server is precisely the party that does not know.

The worker loop

Here is the shape, with the pieces that matter and none of the framework.

type Attempt = { jobId: string | null; status: string | null };

async function ensureLabelPrinted(shipment: Shipment): Promise<Attempt> {
  // 1. Deterministic identity, created with the business event, stored beside it.
  const key = `shipment-${shipment.id}-label-v${shipment.labelVersion}`;

  // 2. If we already have a job id, never create another one.
  const existing = await db.printAttempts.find({ idempotencyKey: key });
  if (existing?.jobId) return pollToTerminal(existing.jobId);

  await db.printAttempts.upsert({ idempotencyKey: key, jobId: null });

  const res = await fetch("https://rocketprint.io/api/v1/print-jobs", {
    method: "POST",
    headers: {
      "X-API-Key": process.env.ROCKETPRINT_KEY!,
      "Idempotency-Key": key,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      printerId: shipment.printerId,
      contentType: "raw_base64",
      content: shipment.zplBase64,
      title: `Label ${shipment.id}`,
      expiresInSeconds: 900, // a stale label should expire, not print tomorrow
    }),
  });

  // 3. A conflict means the identity is being reused for different content.
  //    Do not retry it. It is a bug or an unversioned document change.
  if (res.status === 409) throw new ConflictingReprint(key);

  // 4. Rate limits tell you when to come back.
  if (res.status === 429) {
    await sleep(Number(res.headers.get("Retry-After") ?? 5) * 1000);
    return ensureLabelPrinted(shipment);
  }

  if (!res.ok) throw new TransientError(res.status); // safe: same key next time

  const job = await res.json();
  await db.printAttempts.update({ idempotencyKey: key, jobId: job.id });
  return pollToTerminal(job.id);
}

The key is computed from the shipment, so every retry of this function, including after a process restart, sends the same key. The ambiguous case, where the POST times out after the server had saved the job, needs no special handling: the next attempt sends the same key and gets the original job back with Idempotency-Replayed set.

Unknown is an outcome, and retrying it is the dangerous default

Here is where physical effects stop resembling messages. Our terminal states are completed, failed, canceled, expired, and unknown. The last one exists because sometimes an operation is interrupted with an unconfirmed outcome, and the truthful answer is that we do not know whether it printed.

The tempting policy is to treat unknown as failure and retry. That is the policy that produces two labels on one parcel. The correct policy for an unobservable physical effect is to stop and consult something outside the software: a person who can look at the printer, or a sensor if you have one. In our case unknown is a terminal status, but a later acknowledgment from Station can still settle what actually happened, so it is the one terminal state worth revisiting rather than acting on immediately.

If you take one design rule from this post, take that one. Every system with a physical side effect needs a state that means "a human or a sensor must look," and a policy that forbids automatic retry from that state. We wrote the full transition model up in state machines for side effects you cannot fully observe.

What we can and cannot promise

Being specific about this is the only way the rest of the post means anything.

We can promise that replaying the same instruction does not create a second job, and that redelivering a job the Station already recorded does not intentionally issue a second print.

We cannot promise that exactly one readable label physically emerged. completed means the print command succeeded through the operating system's print path. It is not proof that paper came out, that the ribbon had ink, or that the label was not printed across a fold.

We also do not ship webhooks today, which means the outcome path is polling or the Node client's waitFor helper. That is a real gap: it makes the loop above chattier than it should be. The status reference lists every state and its transitions, and the reliability page is where we keep the list of failures we cannot see.