← All field notes
Architecture

The outbound-only agent, the pattern behind Tailscale nodes, CI runners, and print stations

Reaching into a private network is a solved problem when the private side dials out. The tunnel is the easy part. The work is in the control plane: device identity, routing, queueing, liveness, and reconnects that do not duplicate work.

If you need to make something happen inside a network you do not control, do not ask for a firewall rule. Put a small agent on the private side and have it dial out to you. Every connection is then outbound from the customer's perspective, and you never need port forwarding, a VPN, or an inbound rule that some security review will spend three weeks on.

That part is well understood. The underrated part is that the tunnel is the trivial component. What makes an outbound-only system work or fail is the control plane: how you identify the agent, how you route work by identity, what happens to work while the agent is gone, and how you avoid doing it twice when the agent comes back.

You already run several of these

System Agent on the private side Dials out to What the vendor documents
Tailscale tailscaled on each node A coordination server that holds each node's public key and current whereabouts Peers traverse NAT rather than accepting inbound connections, so there are no public-facing open ports
GitHub Actions self-hosted runners The runner process GitHub, over an HTTPS long poll Only an outbound connection is required; GitHub never connects in to the runner
ngrok The ngrok agent The ngrok service The agent establishes long-lived TLS connections out, and receives inbound traffic back down them
Cloudflare Tunnel cloudflared Cloudflare's network Connecting out lets you block all inbound traffic to the origin
Print stations A desktop app beside the printer The print API A printer in a warehouse has no public address and should not

Managed database connectors that sit inside a customer VPC follow the same shape. The topology repeats: a long-lived outbound connection, a server that knows which agent is which, and work flowing backwards down a pipe the client opened.

RocketPrint is a remote printing API and desktop Station for software platforms: your backend POSTs a print job, and a small app beside the printer prints it, with no inbound networking, no drivers for raw label formats, and a durable job record. Station connects outbound only, over an encrypted WebSocket. That is the whole of our networking story, and it is the reason the security conversation with a customer's IT team is short. How a cloud printing API reaches a printer behind a firewall walks the same path with the packets.

The control plane is the product

Once the socket exists, you own four responsibilities, and each one is a place teams get surprised.

Responsibility one: authenticate the agent as a device, not as a user

The agent is not a person and should not hold a person's credential. A user's API key handed to a machine outlives the employee, gets copied when the machine is imaged, and cannot be revoked without breaking something else.

Enroll instead. The pattern that works: the agent displays a code, a human authorizes it in a browser while signed in, and the agent receives its own access token bound to that device. Ours does exactly that, and stations can be revoked individually from the console. Build revocation on day one, because you will need it the day a laptop with an enrolled agent walks out of a building.

Insist on two properties. The credential is device-bound, so copying a config file to another machine does not yield a working agent. And the agent's authority is narrow: our Station prints only jobs for printers it reported, on the account it signed in to. An agent that can be tricked into acting on another tenant's work is a tenancy bug wearing a networking costume. There is more on the threat model in our remote printing security review.

Responsibility two: route by identity, not by address

In an outbound world, addresses are meaningless: the agent's IP is a NAT translation that changes, and the hostname is whatever someone typed during setup. Resources therefore need stable ids assigned by you, plus a mapping from each resource to the agent currently reporting it. Our printer ids stay stable across renames, because the alternative is that someone renames "Label Printer 2" and every stored reference in a customer's database breaks silently. Design the identity so the agent is an implementation detail: the caller asks for printer prn_..., not for "whatever is attached to the machine that last connected from the Denver office."

Responsibility three: queue when the agent is absent, and expire the queue

An outbound agent is offline regularly: the machine sleeps, the shift ends, the Wi-Fi drops, someone reboots. Absence is normal traffic, not an incident, so accepted work has to wait somewhere durable, and waiting has to have a deadline. Jobs for an offline printer wait with us until they expire or are canceled, and they print when the computer reconnects. That produces a failure mode customers do meet: a machine off for hours will print everything queued for it the moment it comes back, unless you canceled first.

Two controls make that manageable, and both belong in the API rather than in a runbook. First, a per-job expiry so stale work dies on its own. Ours takes expiresInSeconds from 10 to 86,400, defaulting to 3,600, and a job not dispatched by its deadline becomes expired. Second, a cancel window with an honest boundary: you can cancel a job that is still pending, but not one already sent to the agent, because the effect may already have happened. What happens when a warehouse printer goes offline has the operational version of this.

Responsibility four: liveness is a claim with a timestamp

You cannot ask the agent whether it is alive, because your question travels the same broken path as its answer. What you have is evidence: the last time it said something, and whether the socket is still writable.

So expose presence as what it is. We report printers as online or offline with the station that reported them, and that presence is an inference from a connection, not a promise about the next second. Time out on the server side, since a client that has crashed will never tell you.

Four failure modes that only appear in production

Half-open sockets. The most expensive one. A TCP connection can be dead in one direction while both sides believe it is healthy. The agent thinks it is connected and waits forever. The server thinks it delivered work. Nothing errors. Application-level heartbeats in both directions are the only reliable detection, and the timeout must trigger a teardown, not a log line.

Reconnect storms. When your control plane restarts, every agent reconnects at once, and if they all retry on a fixed one-second interval you have built a denial of service against yourself out of your own client. Exponential backoff with randomized jitter, and a cap.

Duplicate delivery after reconnect. This is where an outbound-only design quietly becomes an at-least-once delivery system. The agent did the work, the acknowledgment was lost with the socket, and on reconnect the server redelivers. If the effect is physical, the agent must keep its own durable record of finished work across restarts and replay the recorded outcome instead of acting again. Ours does. On the server side, a lease means exactly one server can be delivering a given job: in our status list that state is dispatching, and it exists so a job is never sent twice.

Clock skew. The agent's clock is wrong. Treat its timestamps as reported values for display, and make decisions that matter, expiry included, on server time.

We shipped the half-open socket bug ourselves

It would be dishonest to list that first failure mode as though we had only read about it. Station 1.0.6 fixed socket reconnection, because a reconnect could leave a dead socket behind: the agent believed it had a live connection. The fix was to tear the old socket down safely and create a new one, rather than reusing an object whose state was no longer trustworthy.

The lesson generalizes past our code. A reconnect is not a retry of a connection, it is the construction of a new one, and any state associated with the old connection has to be explicitly discarded. If your reconnect logic reuses a socket, a channel object, or a subscription list, it can succeed in a way that leaves you with an agent that looks healthy and receives nothing. Update behavior around that is its own topic, covered in desktop agent auto-updates that do not break the shift.

Design rules, condensed

  • Dial out, always. No inbound rule, no VPN requirement.
  • Enroll the device with human authorization, issue a device-bound token, and build revocation on day one.
  • Scope the agent's authority to the resources it reported and the tenant it belongs to.
  • Give resources stable server-assigned ids that survive renames.
  • Queue work durably while the agent is away, and give every unit of work an expiry.
  • State the cancel boundary precisely, because after delivery you are guessing.
  • Heartbeat in both directions, with jitter, and time out server-side.
  • Keep a ledger at both ends, so redelivery after a reconnect is idempotent.
  • Decide expiry and ordering on server time, and treat reconnect as construction rather than reuse.

Two gaps on our side, since they change how you would build against this: we ship no webhooks today, so outcomes come from polling or the Node client's waitFor helper, and Station has no Linux build available for self-service installation. The Station documentation covers enrollment and troubleshooting, and the reliability page lists what we cannot see from our side.

END / outbound-only-agents-architecture-pattern

More field notes