(Kubernetes) Gateway does not detect a dead NATS connection when the NATS server's host loses power

(#4781) Bug Open: Fix pending messaging self-hosting

Reproduction steps

  1. Run Fluxer self-hosted on Kubernetes with the Helm chart, with a single NATS pod.
  2. Hard power off the node NATS runs on. No clean shutdown, so the TCP connections are never closed.
  3. Let Kubernetes start NATS again on another node.
  4. Open a Text channel in the app.

Expected behaviour

The gateway notices the NATS connection is gone, reconnects and subscribes again, like the other services do. Messages keep loading.

Actual behaviour

The other services (messages, shards) logged nats: IO error, then NATS reconnected about a minute later. The gateway logged nothing about NATS and never reconnected. For hours afterwards, every API call to the gateway failed with no responders, so channels showed no messages (503). Voice audio still worked. The gateway pod stayed ready the whole time. Restarting the gateway fixed it. I think the gateway's connection to the old NATS stayed half-open, so it never saw a disconnect. #3690 (similar thing i found) only resubscribes after a connection is reported closed. A PING/PONG check on the NATS connection would probably catch this.

Build information

Versions running on the Kubernetes Cluster: Helm Chart Digest: 3b21662 fluxer-api: 2026.1108.202538 fluxer-gateway: 2026.1108.202517 fluxer-media-proxy: 2026.1108.202516 fluxer-push: 2026.1108.202538 fluxer-admin: 2026.1108.202528 fluxer-app-proxy-self-hosted: 2026.1108.202519 fluxer-static: 2026.1108.202537 fluxer-gifs: 2026.1108.202527 fluxer-messages: 2026.1108.202528 fluxer-snowflakes: 2026.1108.202600 fluxer-unfurl: 2026.1108.202602 fluxer-users: 2026.1108.202605

Platform

Self-hosting

Evidence

messages-shard: "client error: nats: IO error" 09:27:39 messages-shard: "NATS reconnected" 09:28:31 gateway: (no NATS log lines at all for 8 hours) api: "[nats-rpc] no responders for subject" rpc.gateway.guild.get_auth_context api: GET /v1/channels/:channel_id/messages status 503
  1. The pull request fluxer#4149 mentions this post Merged

1 comment

Sign in with Fluxer to comment and vote.
Comment by Rex
RexSystem 1 vote
Status changed from Needs triage to Fix pending
Fixed in fluxer#4149. The fix will be live after the next gateway deploy.