Edit history

(Kubernetes) Gateway does not detect a dead NATS connection when the NATS server's host loses power has not been edited, so there are no earlier versions.

Current version | Original by Domi
Show

(Kubernetes) Gateway does not detect a dead NATS connection when the NATS server's host loses power

Reproduction steps

  1. Run Fluxer self-hosted on Kubernetes with the Helm chart, with a single NATS pod.
  2. Hard power off the node NATS runs on. No clean shutdown, so the TCP connections are never closed.
  3. Let Kubernetes start NATS again on another node.
  4. Open a Text channel in the app.

Expected behaviour

The gateway notices the NATS connection is gone, reconnects and subscribes again, like the other services do. Messages keep loading.

Actual behaviour

The other services (messages, shards) logged nats: IO error, then NATS reconnected about a minute later. The gateway logged nothing about NATS and never reconnected. For hours afterwards, every API call to the gateway failed with no responders, so channels showed no messages (503). Voice audio still worked. The gateway pod stayed ready the whole time. Restarting the gateway fixed it. I think the gateway's connection to the old NATS stayed half-open, so it never saw a disconnect. #3690 (similar thing i found) only resubscribes after a connection is reported closed. A PING/PONG check on the NATS connection would probably catch this.

Build information

Versions running on the Kubernetes Cluster: Helm Chart Digest: 3b21662 fluxer-api: 2026.1108.202538 fluxer-gateway: 2026.1108.202517 fluxer-media-proxy: 2026.1108.202516 fluxer-push: 2026.1108.202538 fluxer-admin: 2026.1108.202528 fluxer-app-proxy-self-hosted: 2026.1108.202519 fluxer-static: 2026.1108.202537 fluxer-gifs: 2026.1108.202527 fluxer-messages: 2026.1108.202528 fluxer-snowflakes: 2026.1108.202600 fluxer-unfurl: 2026.1108.202602 fluxer-users: 2026.1108.202605

Platform

Self-hosting

Evidence

messages-shard: "client error: nats: IO error" 09:27:39 messages-shard: "NATS reconnected" 09:28:31 gateway: (no NATS log lines at all for 8 hours) api: "[nats-rpc] no responders for subject" rpc.gateway.guild.get_auth_context api: GET /v1/channels/:channel_id/messages status 503