RexSystem1 voteoriginally by @SIGTERM-015 on GitHub
Enabled pg_stat_statements and caught the exact statement. Numbers below are from 5 minutes on a completely idle instance (no users connected, no activity), counters reset immediately before.
Top statement by rows returned:
calls rows total_ms rows/call query
3 278,266 2,897 92,755 SELECT row_key, row_data FROM "fluxer_kv"
WHERE table_name = $1
AND (expires_at IS NULL OR expires_at > now())
There is no row_key predicate, so this loads an entire logical table in one go. 278,266 is exactly the row count of disposable_email_domains in my instance:
So the disposable-email blocklist is being re-read in full from Postgres roughly every ~100 seconds. At ~62 MB of heap per call and ~864 calls/day that is on the order of 50+ GB/day on an idle instance, which matches the observed 179 GB out of the Postgres container over 6 days.
Write side, same idle 5-minute window:
474 INSERT INTO "fluxer_kv" (two distinct statements, 237 calls each)
79 DELETE FROM "fluxer_kv" WHERE table_name = $1 AND row_key = ANY($2::text[])
474 SELECT ... WHERE table_name = $1 AND row_key = $2 (point lookups, fine)
~474 inserts / 5 min ≈ 136k rows/day, which is almost exactly the size of jobs_by_id and jobs_by_day_bucket (126,649 each). That accounts for the 20.3 GB of block writes over 6 days.
Two separate problems, then:
A full-table load of a large static dataset on a short timer. The blocklist is reference data that changes rarely — it looks like a cache refresh that re-reads all 278k rows instead of loading once, watching for changes, or paging.
An idle job pipeline writing ~136k rows/day. Something enqueues work continuously with no users present.
Both are amplified by the single-heap fluxer_kv design: any sequential scan pays for unrelated datasets sharing the same physical table.
Thread
Comment by @SIGTERM-015
pg_stat_statementsand caught the exact statement. Numbers below are from 5 minutes on a completely idle instance (no users connected, no activity), counters reset immediately before. Top statement by rows returned:row_keypredicate, so this loads an entire logical table in one go. 278,266 is exactly the row count ofdisposable_email_domainsin my instance:jobs_by_idandjobs_by_day_bucket(126,649 each). That accounts for the 20.3 GB of block writes over 6 days. Two separate problems, then:- A full-table load of a large static dataset on a short timer. The blocklist is reference data that changes rarely — it looks like a cache refresh that re-reads all 278k rows instead of loading once, watching for changes, or paging.
- An idle job pipeline writing ~136k rows/day. Something enqueues work continuously with no users present.
Both are amplified by the single-heapfluxer_kvdesign: any sequential scan pays for unrelated datasets sharing the same physical table.