AllBugsFeaturesChangelogSearchNew post
Sign in with Fluxer
Bugs

[Self-Hosted] High Net I/O and Block I/O

(#757) Bug Fixed performance self-hosting
by RexSystem edited originally by @KeckigerKeks on GitHub

Summary

I deployed the fluxer container via docker and after 2 weeks i got high Net I/O and Block I/O readings. On the first day I connected to the voice channel a maximum of 1h with a friend. Then forgot about it 2 weeks and I was checking on a different matter the container stats with 
docker stats
. I was really surprised to see 464GB of traffic outbound and Block writes of 174GB after 2 weeks of doing nothing.

Steps to reproduce

  1. set up a self-hosted container with docker
  2. set up a server
  3. watch the Net I/O and Block I/O increase with the command docker stats without causing obvious traffic

Logs or screenshots

docker stats
after 2 weeks: image 
docker stats
after 1 day: image
  • 630624819-1a5b44ac-64d5-4ca4-86bf-240351776df4.png

    630624819-1a5b44ac-64d5-4ca4-86bf-240351776df4.png

    1144×213 | 218 kB

  • 630626387-52b3adc8-e132-4467-a461-667d20368231.png

    630626387-52b3adc8-e132-4467-a461-667d20368231.png

    1173×480 | 570 kB

6 comments

Sign in with Fluxer to comment and vote.
Comment by @ToxykAuBleu
RexSystem 1 vote edited originally by @ToxykAuBleu on GitHub
+1, similar situation on my instance (running for 16 hours only) : 
CONTAINER ID   NAME                        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
bc7eb98c9696   fluxer-app-proxy-1          0.01%     5.527MiB / 31.34GiB   0.02%     3.59MB / 787kB    0B / 0B           13
356e4e9ea9fc   fluxer-caddy-1              0.00%     19.96MiB / 31.34GiB   0.06%     10.6MB / 10.5MB   4.1kB / 4.1kB     17
1c4e4856c4c1   fluxer-admin-1              0.00%     8.777MiB / 31.34GiB   0.03%     352kB / 271kB     0B / 0B           13
8954dba27b9f   fluxer-api-1                0.13%     469MiB / 31.34GiB     1.46%     3.11GB / 16.6MB   0B / 15.8MB       38
ad000bbd08dc   fluxer-worker-1             0.10%     368.6MiB / 31.34GiB   1.15%     16.5GB / 168MB    0B / 13.8MB       39
b8a90cfc4db9   fluxer-media-proxy-1        0.00%     95.88MiB / 31.34GiB   0.30%     6.31MB / 5.88MB   0B / 0B           19
bee7d2b17f8b   fluxer-gateway-1            0.00%     116.5MiB / 31.34GiB   0.36%     1.3MB / 900kB     0B / 12.3kB       44
02edfe4a526b   fluxer-snowflakes-1         0.00%     2.367MiB / 31.34GiB   0.01%     15.4MB / 16.7MB   459kB / 0B        13
2997c22ee2e1   fluxer-gifs-1               0.00%     4.449MiB / 31.34GiB   0.01%     147kB / 196kB     590kB / 0B        13
871594dd28b6   fluxer-users-shard-1        0.00%     2.277MiB / 31.34GiB   0.01%     166kB / 209kB     0B / 0B           13
eb0cbf2c424f   fluxer-messages-shard-1     0.00%     2.16MiB / 31.34GiB    0.01%     172kB / 218kB     0B / 0B           13
e93c503c60cd   fluxer-unfurl-shard-1       0.00%     3.82MiB / 31.34GiB    0.01%     148kB / 196kB     0B / 0B           13
e259d0d7d48e   fluxer-messages-1           0.00%     4.664MiB / 31.34GiB   0.01%     156kB / 207kB     590kB / 0B        13
7252096ffdc8   fluxer-gifs-shard-1         0.00%     4.867MiB / 31.34GiB   0.02%     146kB / 196kB     0B / 0B           13
288ea0e0326f   fluxer-users-1              0.00%     2.574MiB / 31.34GiB   0.01%     156kB / 204kB     590kB / 0B        13
314acbb990ec   fluxer-unfurl-1             0.00%     2.586MiB / 31.34GiB   0.01%     147kB / 196kB     0B / 0B           13
0fb4577d6f38   fluxer-snowflakes-shard-1   0.00%     1.879MiB / 31.34GiB   0.01%     1.26MB / 14.1MB   0B / 0B           13
451b697923d3   fluxer-seaweedfs-1          0.01%     116.9MiB / 31.34GiB   0.36%     142kB / 6.39MB    5.62MB / 119kB    23
3181d9b34221   fluxer-static-proxy-1       0.00%     18.53MiB / 31.34GiB   0.06%     11.8kB / 4.23kB   0B / 16.4kB       13

44ee62e43486   fluxer-postgres-1           0.06%     175MiB / 31.34GiB     0.55%     141MB / 18.3GB    176kB / 222MB     17

40bc76a12a7a   fluxer-nats-1               0.01%     22.45MiB / 31.34GiB   0.07%     48MB / 55MB       24.6kB / 87.5MB   17
1bd5b52605a8   fluxer-valkey-1             0.02%     8.199MiB / 31.34GiB   0.03%     13.7MB / 3.73MB   0B / 0B           5
be2f5527f9a6   fluxer-meilisearch-1        0.01%     158.9MiB / 31.34GiB   0.50%     91.9kB / 92.3kB   324kB / 4.24MB    34
dc7b9ef6198d   fluxer-livekit-1            0.00%     20.22MiB / 31.34GiB   0.06%     8.05kB / 412B     0B / 0B           15
Permalink
Comment by @UnknownLITE
RexSystem 1 vote originally by @UnknownLITE on GitHub
running for 8 days with almost no user activity: image
  • 631195939-998bb92d-6019-4594-8331-a3745575d599.png

    631195939-998bb92d-6019-4594-8331-a3745575d599.png

    1110×493 | 104 kB

Permalink
Comment by @SIGTERM-015
RexSystem 1 vote originally by @SIGTERM-015 on GitHub
Enabled pg_stat_statements and caught the exact statement. Numbers below are from 5 minutes on a completely idle instance (no users connected, no activity), counters reset immediately before. Top statement by rows returned: 
calls   rows      total_ms   rows/call   query
3       278,266   2,897      92,755      SELECT row_key, row_data FROM "fluxer_kv"
                                         WHERE table_name = $1
                                           AND (expires_at IS NULL OR expires_at > now())
Top statement by heap blocks touched: 
calls   blocks   size     query
3       23,881   187 MB   (same SELECT as above)
There is no row_key predicate, so this loads an entire logical table in one go. 278,266 is exactly the row count of disposable_email_domains in my instance: 
disposable_email_domains   278,266
jobs_by_id                 126,649
jobs_by_day_bucket         126,649
jobs_active                 12,496
everything else             < 5,000
So the disposable-email blocklist is being re-read in full from Postgres roughly every ~100 seconds. At ~62 MB of heap per call and ~864 calls/day that is on the order of 50+ GB/day on an idle instance, which matches the observed 179 GB out of the Postgres container over 6 days. Write side, same idle 5-minute window: 
474 INSERT  INTO "fluxer_kv" (two distinct statements, 237 calls each)
 79 DELETE  FROM "fluxer_kv" WHERE table_name = $1 AND row_key = ANY($2::text[])
474 SELECT  ... WHERE table_name = $1 AND row_key = $2   (point lookups, fine)
~474 inserts / 5 min ≈ 136k rows/day, which is almost exactly the size of jobs_by_id and jobs_by_day_bucket (126,649 each). That accounts for the 20.3 GB of block writes over 6 days. Two separate problems, then:
  1. A full-table load of a large static dataset on a short timer. The blocklist is reference data that changes rarely — it looks like a cache refresh that re-reads all 278k rows instead of loading once, watching for changes, or paging.
  2. An idle job pipeline writing ~136k rows/day. Something enqueues work continuously with no users present.
Both are amplified by the single-heap fluxer_kv design: any sequential scan pays for unrelated datasets sharing the same physical table.
Permalink
Comment by @kerichdev
RexSystem 1 vote originally by @kerichdev on GitHub
+1 with insane stats postgres did a little over 20gb of reads in just 5 minutes after starting 
035192b85997   fluxer-postgres-1           1.65%     234MiB / 15.41GiB     1.48%     3.72MB / 796MB    20.4GB / 4.71MB   26
this puts way too much iowait strain on cpu
Permalink
Comment by @MrRubberDucky
RexSystem 1 vote originally by @MrRubberDucky on GitHub
+1, only seeing high net I/O in my case but that may be a bug with latest version of Podman not counting Block I/O. I had to limit those containers to 1.5 cores and limit VM disk read/writes otherwise it was spiking up IO delay on my Proxmox VE rig to 60%. podman stats (containers were up for 10 hours) 
fluxer-postgres          2.73%       198.5MB / 8.327GB  2.38%       107MB / 13.38GB    0B / 0B     22          15m35.078323s  2.73%
fluxer-worker            0.26%       389.3MB / 8.327GB  4.67%       11.87GB / 122MB    0B / 0B     29          10m59.34532s   1.93%
fluxer-api               3.16%       723MB / 8.327GB    8.68%       2.049GB / 16.1MB   0B / 0B     31          18m3.359274s   3.16%
Permalink
Comment by @Z3phyr1s
RexSystem 1 vote originally by @Z3phyr1s on GitHub
+1 same issue here 
7352d6abf4fe   fluxer-postgres-1           0.36%     211.7MiB / 7.532GiB   2.75%     5.75GB / 373GB    14.8TB / 248GB    33
Permalink