Edit history

Earlier versions of Rexctl Backup Helper Container Can Produce Very Large JSON Log File, newest first.

Current version | Edited by Falco1h ago

Changes

EvidenceLLM was used for troubleshooting.> [!NOTE]LLM was used for troubleshooting.When a backup process was running, I ran this in a separate terminal window:

Show

Rexctl Backup Helper Container Can Produce Very Large JSON Log File

Reproduction steps

On a Linux Docker Host, confirm Docker's default logging driver. It should be set to json-file with no log rotation
rexctl backup location "insert fstab mounted NFS share here" (this might not matter)
rexctl backup
Backup starts, but VM runs out of usable disk space due to large log file.

Expected behaviour

Backup runs and the Fluxer instance is brought back up.

Actual behaviour

On limited drive space, my Fluxer VM produced a 15.2 GiB log file during the backup process that kept pushing its local virtual disk into full capacity. I expanded its virtual disk to aide with troubleshooting.

Build information

Fluxer / rexctl version: 2026.1025.24548 Source commit: 80a0e98c Runtime: Docker Engine on Linux Docker logging driver: json-file, without log rotation configured

Platform

Self-hosting

Evidence

Note

LLM was used for troubleshooting.

When a backup process was running, I ran this in a separate terminal window:

watch -n 10 '
df -h /
sudo du -xhd1 /var /home /tmp 2>/dev/null | sort -h | tail -15
'

Watch showed that /var/lib grew by +3 GB within 30 seconds. Closed out and then ran this with the backup job still running:

sudo find /var/lib/docker/containers \
  -type f -name '*-json.log' \
  -printf '%s %p\n' 2>/dev/null |
  sort -nr | head -10
docker ps -a --no-trunc \
  --format 'table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}'

The relevant output for this command was:

16304118684 /var/lib/docker/containers/98069b0a50cc528e3677bb41129a90dd513f4f0e5b9d0ca2bb21c84d995fa126/98069b0a50cc528e3677bb41129a90dd513f4f0e5b9d0ca2bb21c84d995fa126-json.log
824825 
CONTAINER ID                                                       IMAGE                                                                                                                    NAMES                                   STATUS
98069b0a50cc528e3677bb41129a90dd513f4f0e5b9d0ca2bb21c84d995fa126   postgres:16-alpine                                                                                                       fluxer-rexctl-helper-4fe1908025fabf4b   Up 3 minutes

When I stopped the backup process, helper goes away and disk space goes back to normal. LLM recommended the following fix:

sudo mkdir -p /etc/docker

if sudo test -e /etc/docker/daemon.json; then
  echo "Existing daemon.json found; not overwriting it."
  sudo cat /etc/docker/daemon.json
else
  sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}
EOF
  echo "Created /etc/docker/daemon.json"
fi

After restarting docker, I was able to use rexctl backup without issues.

Original by Falco1h ago