Reproduction steps
- Run Fluxer self-hosted on Kubernetes with the Helm chart, with a single NATS pod.
- Hard power off the node NATS runs on. No clean shutdown, so the TCP connections are never closed.
- Let Kubernetes start NATS again on another node.
- Open a Text channel in the app.
Expected behaviour
The gateway notices the NATS connection is gone, reconnects and subscribes again, like the other services do. Messages keep loading.
Actual behaviour
The other services (messages, shards) logged nats: IO error, then NATS reconnected about a minute later. The gateway logged nothing about NATS and never reconnected. For hours afterwards, every API call to the gateway failed with no responders, so channels showed no messages (503). Voice audio still worked. The gateway pod stayed ready the whole time. Restarting the gateway fixed it.
I think the gateway's connection to the old NATS stayed half-open, so it never saw a disconnect. #3690 (similar thing i found) only resubscribes after a connection is reported closed. A PING/PONG check on the NATS connection would probably catch this.
Build information
Versions running on the Kubernetes Cluster:
Helm Chart Digest: 3b21662
fluxer-api: 2026.1108.202538
fluxer-gateway: 2026.1108.202517
fluxer-media-proxy: 2026.1108.202516
fluxer-push: 2026.1108.202538
fluxer-admin: 2026.1108.202528
fluxer-app-proxy-self-hosted: 2026.1108.202519
fluxer-static: 2026.1108.202537
fluxer-gifs: 2026.1108.202527
fluxer-messages: 2026.1108.202528
fluxer-snowflakes: 2026.1108.202600
fluxer-unfurl: 2026.1108.202602
fluxer-users: 2026.1108.202605
Platform
Self-hosting
Evidence
messages-shard: "client error: nats: IO error" 09:27:39
messages-shard: "NATS reconnected" 09:28:31
gateway: (no NATS log lines at all for 8 hours)
api: "[nats-rpc] no responders for subject" rpc.gateway.guild.get_auth_context
api: GET /v1/channels/:channel_id/messages status 503
1 comment
Comment by Rex