"Give me /events, not webhooks." That sentiment hit the front page of Hacker News because it captures a frustration shared by every developer who has built a webhook consumer at scale. Webhooks are conceptually simple -- a provider sends an HTTP POST to your URL when something happens. In practice, they are a source of silent data loss, security vulnerabilities, and infrastructure complexity that scales worse than the systems they integrate.
+const bodyContent = `Trying to set up a persistent, stable webhook endpoint URL that doesn't change every time you restart? Stop patching URLs and replace the webhook with a persistent agent tunnel instead. A tunnel survives restarts, works from behind NAT, and streams events over an encrypted connection — no public HTTP endpoint to maintain. Webhooks are conceptually simple -- a provider sends an HTTP POST to your URL when something happens. In practice, they are a source of silent data loss, security vulnerabilities, and infrastructure complexity that scales worse than the systems they integrate.
-Nearly 20% of webhook event deliveries fail silently during peak loads. The provider's retry logic is a black box you do not control. Events arrive out of order. Your endpoint needs a routable IP address, which means code running behind NAT -- laptops, home servers, CI runners, most AI agents -- cannot receive webhooks at all. And the security model is inverted: you are exposing a public HTTP endpoint that anyone on the internet can POST to.
+"Give me /events, not webhooks." That sentiment hit the front page of Hacker News because it captures a frustration shared by every developer who has built a webhook consumer at scale. A meaningful share of webhook event deliveries fail silently during peak loads. The provider's retry logic is a black box you do not control. Events arrive out of order. Your endpoint needs a routable IP address, which means code running behind NAT -- laptops, home servers, CI runners, most AI agents -- cannot receive webhooks at all. And the security model is inverted: you are exposing a public HTTP endpoint that anyone on the internet can POST to.
This article examines why webhooks break down for agent-to-agent communication, why the common workarounds do not solve the fundamental problems, and how persistent encrypted tunnels with built-in event streaming provide a better model.
Webhooks turn your application into a distributed system. The moment you accept an incoming HTTP POST from an external provider, you inherit every hard problem in distributed computing: partial failure, message ordering, idempotency, and exactly-once delivery. Most teams do not realize this until they start losing events in production.
Webhook providers typically retry failed deliveries on an exponential backoff schedule. If your server returns a 500 or times out during a deployment window, the provider queues the retry. If your server is down for longer than the retry window -- which varies wildly between providers, from 30 minutes to 72 hours -- those events are gone. You have no way to know they existed. There is no consumer-side replay. There is no offset you can rewind to.
+Webhook providers typically retry failed deliveries on an exponential backoff schedule. If your server returns a 500 or times out during a deployment window, the provider queues the retry. If your server is down for longer than the retry window -- which varies wildly between providers -- those events are gone. You have no way to know they existed. There is no consumer-side replay. There is no offset you can rewind to.
-A production study of webhook delivery across major SaaS platforms found that nearly 20% of deliveries fail during peak loads. Not 20% of total events over the lifetime of the integration -- 20% during the hours when you need them most. Payment processing webhooks during Black Friday. CI/CD webhooks during a deploy. Agent task completion signals during a burst of parallel work.
+A production study of webhook delivery across major SaaS platforms found that a meaningful share of deliveries fail during peak loads -- not over the lifetime of the integration, but during the hours when you need them most. Payment processing webhooks during Black Friday. CI/CD webhooks during a deploy. Agent task completion signals during a burst of parallel work.
Webhooks require the consumer to expose a routable HTTP endpoint. This is trivial if you run in a cloud data center with a static IP. It is impossible if your code runs behind NAT. And most AI agents run behind NAT.
+Webhooks require the consumer to expose a routable HTTP endpoint. This is trivial if you run in a cloud data center with a static IP. It is impossible if your code runs behind NAT. And most AI agents run behind NAT.
Consider the deployment reality: an agent running on a developer laptop behind a home router. An agent running in a Docker container behind a corporate firewall. An agent running on a cloud VM with no public IP (which is the default on most cloud providers now, for good security reasons). None of these can receive webhooks without additional infrastructure.
@@ -35,18 +35,18 @@ const bodyContent = `"Give me /events, not webhooks." That sentiment hit the
To handle webhooks reliably, you need to build infrastructure that rivals the webhook provider itself. One engineering team documented what it takes to process a single webhook safely:
--"You would need 4 new services (SQS, S3, Publisher, Consumer) just to handle a single webhook safely."
+"You would need new services (SQS, S3, Publisher, Consumer) just to handle a single webhook safely."
An incoming webhook hits a lightweight receiver that immediately returns 200 OK. The receiver pushes the raw payload to a queue (SQS, RabbitMQ). A consumer reads from the queue with retry logic. Failed events go to a dead letter queue. A separate service monitors the dead letter queue and alerts. You need idempotency keys to handle duplicate deliveries. You need ordering logic if events must be processed sequentially. This is four to six services to reliably receive an HTTP POST.
+An incoming webhook hits a lightweight receiver that immediately returns 200 OK. The receiver pushes the raw payload to a queue (SQS, RabbitMQ). A consumer reads from the queue with retry logic. Failed events go to a dead letter queue. A separate service monitors the dead letter queue and alerts. You need idempotency keys to handle duplicate deliveries. You need ordering logic if events must be processed sequentially. This is a stack of services to reliably receive an HTTP POST.
The most common workaround for the "no public URL" problem is a tunneling service like ngrok. It creates a temporary public URL that tunnels traffic to your local machine. For development, this is convenient. For production agent communication, it introduces its own problems.
-The free tier of ngrok limits you to 20 connections per minute and assigns a new random subdomain every session. Your webhook URL changes every time you restart the tunnel -- roughly every 7 hours on the free tier. That means reconfiguring every webhook provider that points at your endpoint, which is a manual process for most SaaS integrations and completely impractical for agent-to-agent communication where peers discover each other dynamically.
+The free tier of ngrok rate-limits connections and assigns a new random subdomain every session. Your webhook URL changes every time you restart the tunnel -- frequently on the free tier. That means reconfiguring every webhook provider that points at your endpoint, which is a manual process for most SaaS integrations and completely impractical for agent-to-agent communication where peers discover each other dynamically.
Paid tiers fix the URL stability problem but introduce a dependency on a third-party service that sits in the data path. Every webhook payload passes through ngrok's servers in plaintext (unless you add your own TLS layer). For agent communication carrying sensitive data -- task results, model outputs, customer information -- this is an unacceptable trust model.
@@ -54,19 +54,19 @@ const bodyContent = `"Give me /events, not webhooks." That sentiment hit the
The webhook model is "push to a URL." The persistent tunnel model is "maintain a connection and stream events." This is a fundamental architectural difference, not a minor protocol variation.
+The webhook model is "push to a URL." The persistent tunnel model is "maintain a connection and stream events." This is a fundamental architectural difference, not a minor protocol variation.
In the webhook model, the producer decides when to send data and where to send it. The consumer is passive -- it sits and waits for POSTs. If the consumer is offline, events are lost (or queued on the producer side, which is the producer's problem, not yours). The consumer has no control over delivery timing, ordering, or backpressure.
In the persistent tunnel model, both sides maintain an active connection. The consumer subscribes to specific event topics. Events flow over the existing tunnel -- no new connection setup per event. If the consumer disconnects, it resubscribes when it reconnects. The connection itself handles encryption, NAT traversal, and peer authentication. There is no public URL because the consumer initiates the connection outward, through NAT, to a rendezvous point.
-Pilot Protocol implements this model with its event stream on port 1002. Agents connect to each other through encrypted UDP tunnels with automatic NAT traversal (STUN discovery, hole-punching, relay fallback). Once connected, they can publish and subscribe to topic-based event streams without any additional infrastructure.
+Pilot Protocol implements this model with its event stream on port 1002. Agents connect to each other through encrypted UDP tunnels with automatic NAT traversal (STUN discovery, hole-punching, relay fallback). Once connected, they can publish and subscribe to topic-based event streams without any additional infrastructure.
Port 1002 is Pilot's built-in pub/sub service. It supports topic-based routing with wildcard subscriptions, persistent connections, and encrypted transport. Here is how it works in practice.
@@ -218,7 +218,7 @@ publish("