The message arrived. The state is still wrong.

Aug 30, 2026

A pod restarts on a Tuesday morning.

It reconnects to the message broker, resumes from its last offset, drains the backlog and reports zero consumer lag. The local cache is serving traffic again. Every dashboard is green.

The cache is also wrong.

While the pod was away, the broker deleted part of its backlog. Or the subscription expired. Or a key moved to another cache owner while an invalidation was still in flight. The consumer is caught up with everything the messaging system still knows about, which is not the same thing as being caught up with the database.

That distinction is the subject of a seven-page HotOS paper by Atul Adya, Phil Bogle and Colin Meek: Understanding the limitations of pubsub systems.

Its argument is uncomfortable because pub/sub is one of the standard answers to almost every distributed-systems question. Put changes on Kafka. Subscribe to them. Keep the services loosely coupled. Scale the consumers. Retain enough history to survive an outage.

The paper asks a more useful question: what does the consumer actually know when the messages stop arriving?

Usually, less than we pretend.

The hidden database in the middle

A durable pub/sub system is two systems wearing one name.

The first distributes notifications. The second stores those notifications in a replicated log so disconnected consumers can return later. That second part is hard state: it has replication, retention, versions, transactions, compaction and recovery procedures. It is a database with a deliberately narrow API.

That would be fine if the log were the source of truth. Sometimes it is. Clicks, telemetry and immutable business events may be born in the log, and a stream-oriented store is a natural home for them.

But a large class of pub/sub deployments starts with state that already lives somewhere else. A row changes in PostgreSQL, change data capture turns it into an event, a broker stores another copy, and consumers use that copy to reconstruct their own view. There are now two histories: the transaction history decided by PostgreSQL and the delivery history decided by the broker.

The broker can guarantee ordering within a partition. It can guarantee at-least-once delivery for messages it still retains. It may even support transactions of its own. None of those guarantees alone proves that a consumer’s materialized state corresponds to a state that ever existed in PostgreSQL.

The guarantee is real. It is just at the wrong layer.

Retention makes decoupling conditional

Pub/sub is sold as decoupling producers from consumers. The producer keeps producing while the consumer is offline; the log absorbs the difference.

That works until the difference is larger than the retained log.

Retention cannot be infinite by default. Backlogs consume storage, compaction removes intermediate versions, subscriptions expire and operators eventually delete things that have been red for too long. Once required history is gone, a consumer needs a new snapshot of the source.

The awkward part is not that resynchronization exists. Any bounded system eventually needs it. The awkward part is that resynchronization is normally an application-specific emergency procedure outside the subscription contract.

A consumer can observe an empty poll, zero lag or the newest available offset. None answers the end-to-end question: “Does my state include every relevant source transaction through version V?”

Even where a broker exposes an out-of-range offset or another detectable gap, the application still has to invent the protocol that takes a source snapshot without racing concurrent changes, replaces local state, and resumes at the matching source position. The most important recovery path is left as integration code.

Loose coupling therefore has an expiry time. It may be six hours, seven days, or two minutes under enough pressure. After that, the producer and consumer are coupled again through a manual repair process nobody exercised during the happy path.

Delivery order is not source order

Consider a permissions database.

Transaction 40 removes Alice from the finance group. Transaction 41 grants the finance group access to a report. The source never contains a state where Alice can read that report.

If those changes travel through different partitions and are applied in reverse order, the replica can temporarily expose exactly that impossible state. Per-key versions prevent an older value from overwriting a newer value for the same row, but they do not preserve a transaction boundary spanning several keys. Tombstones solve resurrection bugs; they do not create a globally consistent snapshot.

Serializing every transaction preserves order and gives up scale. Partitioning gives scale and weakens the view. Periodically replacing the replica with a full snapshot eventually repairs it and admits that the event path is not sufficient on its own.

The source database already decided the order and atomicity. Reconstructing weaker versions of both in an intermediate log is expensive work to end up with a weaker claim.

Cache ownership makes it worse

Distributed caches add another race because the target is moving.

Suppose key x moves from pod A to pod B during rebalancing. Pod B learns that it owns x and reads its current value. An invalidation already in flight is still routed to pod A according to the old assignment. Pod A acknowledges it. Pod B never sees it and may keep the stale value indefinitely.

Leases can close parts of this race, at the cost of periods where nobody owns the range. Broadcasting every invalidation to every cache node avoids ownership mistakes and eventually stops scaling. TTLs make the wrong value disappear later and turn correctness into a timer.

These are not broker bugs. The broker does not know which key range a consumer currently owns, which source version its cache represents, or whether a handoff preserved complete knowledge. We are asking a topic subscription to implement a state-transfer protocol.

Storage plus watch

The paper’s alternative is to separate the two responsibilities explicitly:

The authoritative store owns durable state and the read path.
A separate watch layer owns notification about changes to that state.

Producers write to the store. Consumers can read a deliberately narrow view of it, rather than being given access to arbitrary internal tables. The watch layer distributes soft state: if its buffers disappear, correctness does not disappear with them because consumers can reconstruct from the authoritative store.

The interesting part is the watch contract. It has three signals rather than one:

onEvent says a mutation happened at a source version.
onProgress says all relevant mutations through a source version have been delivered for a key range.
onResync says the watch layer can no longer bridge the consumer’s cursor and the consumer must rebuild from a source snapshot.

An event changes local state. Progress establishes knowledge about local state. Resync admits lost knowledge.

That last pair is what a normal subscription is missing. Silence is ambiguous: nothing may have changed, the network may be partitioned, the consumer may be behind, or required history may already be gone. A progress statement turns silence into evidence. A resync statement turns an unrecoverable gap into a normal state transition rather than silent corruption.

The watch can also be scoped by key range instead of a static topic partition. A cache node asks for the range it owns from a known source version. If ownership changes, the new node establishes knowledge for the new range. This allows application sharding and broker sharding to evolve independently.

Kubernetes already made this model familiar. Controllers read desired state, watch for changes, and reconcile actual state. A watch may expire; the controller lists again and resumes. The notification is an optimization around authoritative state, not the only surviving account of it.

This is not “replace Kafka”

The paper makes a broad claim that storage plus watch can address pub/sub use cases. I think the useful product boundary is narrower.

If Kafka is where an event is born and its ordered history is the product, Kafka is the durable store in this model. If RabbitMQ is distributing commands where one worker must perform each job, a database watch is not automatically a better work queue. If the workload is high-volume telemetry ingestion or stream processing, introducing PostgreSQL because “storage plus watch” sounds cleaner would be architectural theatre.

The strongest case is where authoritative state already exists and consumers need a derived view of it: local caches, search indexes, authorization projections, read models, heterogeneous replicas and controllers reconciling desired with actual state.

In those systems, the question is not “Did I receive message 812?” It is “What source state can I safely claim to represent?”

That is the boundary I want to build around.

watchd

watchd is my attempt to turn the paper’s contract into a cloud-native tool.

The first source is PostgreSQL. Producers keep writing normal PostgreSQL transactions; they do not publish a second application event. watchd reads committed changes through logical replication, preserves transaction boundaries and hands batches to a local durability boundary. It acknowledges a PostgreSQL log position only after that boundary accepts the transaction. If the connection fails in between, replay is allowed. Silent loss is not.

The current reader also treats a missing replication slot as lost source history. It does not quietly create a new slot at the current database position and pretend the consumer is healthy. It reconnects after transient failures, bounds in-flight transaction memory, rejects unsupported state changes explicitly and exposes its last received and safely acknowledged source positions.

That is necessary plumbing, not the finished product.

The next hard boundary is snapshot coordination. A new consumer needs to read a PostgreSQL snapshot and begin consuming changes from the corresponding log position without a gap between the two. After that come the replay buffer and public watch API: events, progress and resync; then a Go SDK that turns those signals into a local projection whose freshness can be checked before serving it.

The intended deployment is a normal service that fits naturally beside applications in Kubernetes, but Kubernetes is not a requirement. A VM, a container platform or a local process can run the same daemon. The cloud-native part is the failure model: restarts, rescheduling and temporary partitions are expected, and a consumer must recover or become explicitly stale.

PostgreSQL is the starting point, not the definition. The core contract should eventually admit other authoritative or ingestion stores, but pretending to be generic before one adapter is correct would mostly produce interfaces with confident names and no evidence behind them.

What I want to learn

There are easier ways to build a side project than replication protocols, snapshot handoffs, bounded replay, dynamic ownership and claims about freshness.

That is why this one is interesting.

The engineering test is not whether watchd can move a row change from PostgreSQL to a Go process. LISTEN/NOTIFY can move a notification. Kafka can move an event. CDC connectors can move entire databases.

The test is whether watchd can tell a consumer, with evidence, one of two things:

your projection is complete through this source version;
it is not, and here is the recovery path.

Anything between those statements is where stale state hides.