AllBugsFeaturesChangelogSearchNew post
Sign in with Fluxer
Bugs

[Self-hosted] Uploads fail with Access Denied when the SeaweedFS bucket is missing

(#670) Bug Fixed media self-hosting
by RexSystem edited originally by @Hacker6329 on GitHub

Summary

I'm trying to update my guild icon but i'm getting an internal error. I'm also having issues uploading files on channel, it says that the file has been moved or removed. I'm the owner of the guild.

Steps to reproduce

Guild Image Change

  1. Open Guild
  2. Attempt to change icon
  3. Click Save
  4. Internal Error

File Upload

  1. Open a channel
  2. Upload a file
  3. Add some text
  4. Click enter and get the error

Environment

Version: Latest Self-Hosted OS: Windows 11 Browser: Self-Hosted

Logs or screenshots

GUILD ICON CHANGE:
api-1  | {
    "level": "info",
    "time": "2026-06-24T16:05:52.802Z",
    "service": "fluxer-api",
    "env": "production",
    "method": "GET",
    "path": "/v1/channels/1519349858559852561/voice-debug-logging/session",
    "status": 200,
    "durationMs": 70,
    "msg": "Request completed"
}
api-1  | {
    "level": "error",
    "time": "2026-06-24T16:05:52.811Z",
    "service": "fluxer-api",
    "env": "production",
    "error": {
        "type": "AccessDenied",
        "message": "Access Denied.",
        "stack": "AccessDenied: Access Denied.\n    at AwsRestXmlProtocol.handleError (/usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+core@3.973.12/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1814:27)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async AwsRestXmlProtocol.deserializeResponse (/usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+core@3.23.4/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:314:13)\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+core@3.23.4/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:26:24\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:386:20\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+middleware-retry@4.4.35/node_modules/@smithy/middleware-retry/dist-cjs/index.js:254:46\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-flexible-checksums@3.972.10/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:241:24\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:63:28\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:90:20\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-logger@3.972.3/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26",
        "$fault": "client",
        "$metadata": {
            "httpStatusCode": 403,
            "requestId": "18BC1001E257221FE7B94B94",
            "attempts": 1,
            "totalRetryDelay": 0
        },
        "name": "AccessDenied",
        "Code": "AccessDenied",
        "Resource": "/fluxer/icons/1519349858559852544/d58362cb",
        "RequestId": "18BC1001E257221FE7B94B94",
        "Key": "icons/1519349858559852544/d58362cb",
        "BucketName": "fluxer"
    },
    "s3Key": "icons/1519349858559852544/d58362cb",
    "assetType": "icon",
    "entityType": "guild",
    "msg": "Asset upload to S3 failed"
}
api-1  | {
    "level": "error",
    "time": "2026-06-24T16:05:52.811Z",
    "service": "errors",
    "env": "production",
    "err": {
        "type": "Error",
        "message": "Failed to upload asset to S3: Access Denied.",
        "stack": "Error: Failed to upload asset to S3: Access Denied.\n    at EntityAssetService.uploadToS3 (/usr/src/app/fluxer_api/src/api/infrastructure/EntityAssetService.ts:477:10)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async EntityAssetService.prepareAssetUpload (/usr/src/app/fluxer_api/src/api/infrastructure/EntityAssetService.ts:170:3)\n    at async GuildOperationsService.updateGuild (/usr/src/app/fluxer_api/src/api/guild/services/data/GuildOperationsService.ts:460:26)\n    at async GuildService.updateGuild (/usr/src/app/fluxer_api/src/api/guild/services/GuildService.ts:240:54)\n    at async Object.handler (/usr/src/app/fluxer_api/src/api/guild/controllers/GuildBaseController.ts:207:20)\n    at async dispatch (file:///usr/src/app/fluxer_api/node_modules/.pnpm/hono@4.12.2/node_modules/hono/dist/compose.js:22:17)\n    at async handler (file:///usr/src/app/fluxer_api/node_modules/.pnpm/hono@4.12.2/node_modules/hono/dist/hono-base.js:118:39)\n    at async dispatch (file:///usr/src/app/fluxer_api/node_modules/.pnpm/hono@4.12.2/node_modules/hono/dist/compose.js:22:17)\n    at async Object.handler (/usr/src/app/fluxer_api/src/api/middleware/ResponseTypeMiddleware.ts:209:3)"
    },
    "msg": "Unhandled error occurred"
}
api-1  | {
    "level": "info",
    "time": "2026-06-24T16:05:52.812Z",
    "service": "fluxer-api",
    "env": "production",
    "method": "PATCH",
    "path": "/v1/guilds/1519349858559852544",
    "status": 500,
    "durationMs": 124,
    "msg": "Request completed"
}

FILE UPLOAD LOG:
api-1  | {
    "level": "error",
    "time": "2026-06-24T16:11:11.527Z",
    "service": "errors",
    "env": "production",
    "err": {
        "type": "S3ServiceException",
        "message": "We encountered an internal error, please try again.",
        "stack": "InternalError: We encountered an internal error, please try again.\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+core@3.973.12/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:69:67)\n    at AwsRestXmlProtocol.handleError (/usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+core@3.973.12/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1810:65)\n    at AwsRestXmlProtocol.deserializeResponse (/usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+core@3.23.4/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:314:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+core@3.23.4/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:26:24\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:386:20\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@smithy+middleware-retry@4.4.35/node_modules/@smithy/middleware-retry/dist-cjs/index.js:254:46\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:63:28\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.972.12/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:90:20\n    at async /usr/src/app/fluxer_api/node_modules/.pnpm/@aws-sdk+middleware-logger@3.972.3/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26",
        "$fault": "server",
        "$metadata": {
            "httpStatusCode": 500,
            "requestId": "18BC104C174BFF5D25DB8856",
            "attempts": 3,
            "totalRetryDelay": 150
        },
        "name": "InternalError",
        "Code": "InternalError",
        "Resource": "/fluxer/attachments/1519349858559852557/1519374340443865088/error.json",
        "RequestId": "18BC104C174BFF5D25DB8856",
        "Key": "attachments/1519349858559852557/1519374340443865088/error.json",
        "BucketName": "fluxer"
    },
    "msg": "Unhandled error occurred"
}
api-1  | {
    "level": "info",
    "time": "2026-06-24T16:11:11.527Z",
    "service": "fluxer-api",
    "env": "production",
    "method": "POST",
    "path": "/v1/channels/1519349858559852557/messages",
    "status": 500,
    "durationMs": 190,
    "msg": "Request completed"
}

Merged posts

These posts were merged into this one. Their comments are now part of the conversation below, marked with where they came from.

Merged from #677 When I attempt to change my profile avatar or banner, the save changes button loads for ~30/40 seconds before going back to normal

RexSystemoriginally by @adavi1995 on GitHub

Report details

Summary

I understand this may be setup error and not a bug - bug was the closest option. When I attempt to change my profile avatar or banner, the save changes button loads for ~30/40 seconds before going back to normal. I had a working instance, wiped it all and started over with the same config. When inspecting in a browser I have an http 500 error - I have attached the logs from seaweedfs

Steps to reproduce

  1. User settings
  2. Upload an image
  3. Save changes

Logs or screenshots

I0626 13:46:09.254822 filer.go:453 Registered IAM gRPC service on filer (unauthenticated; set jwt.filer_signing.key in security.toml to require admin Bearer token)
I0626 13:46:09.254886 grpc_client_server.go:158 gRPC also listening on Unix socket /tmp/seaweedfs-filer-grpc-18888.sock
I0626 13:46:09.445695 s3.go:290 S3 read filer buckets dir: /buckets
I0626 13:46:09.445704 s3.go:292 S3 read master addresses for discovery: [172.18.0.6:9333]
I0626 13:46:09.445710 s3.go:300 connected to filers [172.18.0.6:8888]
I0626 13:46:09.445717 s3.go:321 Starting S3 API Server with standard IAM
I0626 13:46:09.447043 filer_client.go:176 FilerClient: started filer discovery for group '' (refresh interval: 5m0s)
W0626 13:46:09.447093 s3_sse_s3.go:862 SSE-S3 KeyManager: neither s3.sse.kek.passphrase nor WEED_S3_SSE_KEK_PASSPHRASE is set; the KEK will be stored on the filer in plaintext. Set one to enable encrypted-at-rest KEK storage.
I0626 13:46:09.447254 s3_sse_s3.go:567 SSE-S3 KeyManager: No KEK configured. SSE-S3 encryption is disabled. Set s3.sse.kek or s3.sse.key in security.toml to enable it.
I0626 13:46:09.447514 master_grpc_server.go:455 + client .s3@172.18.0.6:18333
I0626 13:46:09.447726 masterclient.go:312 + filer@172.18.0.6:8888 noticed .s3 172.18.0.6:18333
E0626 13:46:09.447809 s3api_server.go:338 Failed to load IAM configuration: no signing key found for STS service; please provide 'signingKey' in IAM config, configure 'jwt.filer_signing.key' in security.toml, or ensure SSE-S3 is initialized
I0626 13:46:09.449692 grpc_client_server.go:158 gRPC also listening on Unix socket /tmp/seaweedfs-s3-grpc-18333.sock
I0626 13:46:09.449707 s3.go:499 Start Seaweed S3 API Server 30GB 4.34 c6cf5a5bd at http port 8333
I0626 13:46:09.449825 s3.go:541 Start Iceberg REST Catalog Server at http://172.18.0.6:8181
I0626 13:46:09.450026 filer_grpc_server_sub_meta.go:674 + local listener s3@@ clientId 699439012 clientEpoch 2
I0626 13:46:09.450034 filer_grpc_server_sub_meta.go:330  + s3@@ local subscribe /etc/ from {Time:2026-06-26 13:46:09.445727862 +0000 UTC Offset:-2 IsOffsetBased:false} clientId:699439012
I0626 13:46:09.578413 master_client.go:24 the cluster has 1 filer
I0626 13:46:09.578426 filer_client.go:429 FilerClient: removed 1 filer(s) no longer in group '': [172.18.0.6:8888]
I0626 13:46:09.578432 filer_client.go:432 FilerClient: discovered 1 new filer(s) in group '': [172.18.0.6:8888]
I0626 13:46:10.143490 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "" version 1782481570143468217: [172.18.0.6:8888.18888]
I0626 13:46:10.143666 masterclient.go:327 LockRing: filer@172.18.0.6:8888 received ring update v1782481570143468217: [172.18.0.6:8888.18888]
I0626 13:46:10.143678 masterclient.go:327 LockRing: s3@172.18.0.6:18333 received ring update v1782481570143468217: [172.18.0.6:8888.18888]
I0626 13:46:10.143699 filer.go:152 LockRing: applying master ring update v1782481570143468217: [172.18.0.6:8888]
I0626 13:46:10.309307 masterclient.go:356 updateVidMap ignore short heartbeat: volume_location:{leader:"172.18.0.6:9333.19333"}
I0626 13:46:10.309363 masterclient.go:289 .master masterClient failed to receive from 172.18.0.6:9333: EOF
I0626 13:46:10.987978 meta_aggregator.go:110 loopSubscribeToOneFiler read 172.18.0.6:8888 start from 2026-06-26 13:45:09.254096297 +0000 UTC 1782481509254096297
I0626 13:46:10.988321 meta_aggregator.go:236 subscribing remote 172.18.0.6:8888 meta change: 2026-06-26 13:45:09.254096297 +0000 UTC, clientId:2095494341
I0626 13:46:10.989070 filer_grpc_server_sub_meta.go:674 + local listener filer:172.18.0.6:8888.18888@@ clientId -2095494341 clientEpoch 1
I0626 13:46:10.989078 filer_grpc_server_sub_meta.go:330  + filer:172.18.0.6:8888.18888@@ local subscribe / from {Time:2026-06-26 13:45:09.254096297 +0000 UTC Offset:-2 IsOffsetBased:false} clientId:-2095494341
I0626 13:46:11.309673 masterclient.go:498 .master masterClient reconnection attempt #1
I0626 13:46:11.310444 master_grpc_server.go:455 + client .master@172.18.0.6:9333.19333
I0626 13:46:11.310595 masterclient.go:312 + master@172.18.0.6:9333 noticed .master 172.18.0.6:9333.19333
I0626 13:46:11.310635 masterclient.go:312 + filer@172.18.0.6:8888 noticed .master 172.18.0.6:9333.19333
I0626 13:46:49.451585 lock_client.go:238 Lock owner changed from  to 172.18.0.6:8888-s3-metrics
I0626 13:48:06.789320 master_grpc_server_volume.go:140 volume grow &{Option:{"collection":"fluxer","replication":{},"ttl":{"Count":0,"Unit":0},"version":3} Count:0 Force:false Reason:grpc assign}
I0626 13:48:06.789373 node.go:220 topo failed to pick 1 from  0 node candidates
I0626 13:48:06.789378 volume_growth.go:140 create 7 volume, created 0: Not enough data nodes found!
I0626 13:48:06.989892 master_grpc_server_volume.go:140 volume grow &{Option:{"collection":"fluxer","replication":{},"ttl":{"Count":0,"Unit":0},"version":3} Count:0 Force:false Reason:grpc assign}
I0626 13:48:06.989934 node.go:220 topo failed to pick 1 from  0 node candidates

10 comments

Sign in with Fluxer to comment and vote.
Comment by @Hacker6329
RexSystem 1 vote originally by @Hacker6329 on GitHub OP
I've found the issue: for some reason seaweedfs didn't have fluxer as bucket. I fixed it doing:
sudo docker compose exec seaweedfs sh weed sheel s3.bucket.list
If you don't see "fluxer" then type:
s3.bucket.create -name fluxer s3.bucket.list
Now you should see fluxer, if that works exit weed shell and restart with:
sudo docker compose down sudo docker compose up -d
Permalink
Comment by @JonaldJohnston
RexSystem 1 vote edited originally by @JonaldJohnston on GitHub
Seems to be an issue with seaweedfs-init. However, on my end, the issue only shows up when the data volume for seaweedfs is mounted somewhere else (in my case, a partition on an internal HDD). this is the output from docker compose logs -f seaweedfs-init: 
seaweedfs-init-1  | error: get filer configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address"
seaweedfs-init-1  | error: get filer configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address"
seaweedfs-init-1  | error: get filer configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address"
seaweedfs-init-1  | error: get filer configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address"
seaweedfs-init-1  | error: get filer configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address"
seaweedfs-init-1  | buckets ready
using this configuration for seaweedfs in docker-compose.yml: 
seaweedfs:
    image: chrislusf/seaweedfs:4.34
    restart: unless-stopped
    networks: [fluxer]
    command: ["server", "-s3", "-dir=/data"]
    volumes:
      - /mnt/data/fluxer/seaweedfs:/data:rw
also, in my particular case, all the buckets failed to be created, not just the fluxer bucket. Manually creating the buckets using @Hacker6329's method worked fine and I could upload icons and other media after. however, this issue does not show up if I leave the defaults and therefore the data volume gets located on my internal NVME instead. my very naive guess is that seaweedfs-init is starting up before seaweedfs is actually finished setting up completely and the slower r/w speed on the HDD is exacerbating the problem. Just a guess though 🫠.
Permalink
Comment by @Hacker6329
RexSystem 1 vote originally by @Hacker6329 on GitHub OP
Talking about the fluxer self-hosted instance, this issue has happened on a proxmox privileged lxc container with ubuntu 22.04. The guide was followed step-by-step. The container runs on an nvme, no old HDDs or secondary disk, all the space is in the same nvme disk. Don't know if this matters or not but fluxer instance is being served over cloudflared tunnel.
Permalink
Comment by @adavi1995
RexSystem 1 vote edited Merged from #677 originally by @adavi1995 on GitHub
Sounds similar to my issue: #670 I've suggested a temporary fix if thats your issue.
Thank you! I already had the fluxer bucket so I changed the volume for seaweedfs to a different location, then I exec'd and ran the commands and the fluxer bucket was missing. I was able to create it and restart my docker containers and image uploading is working again, at least profile specific uploading. I have not tested uploading via DM or chats. Closing this since it is now resolved.
Permalink
Comment by @DerP4si
RexSystem 1 vote originally by @DerP4si on GitHub
same for me, but in my case it says that the file has a wrong format. I tried jpg, png, jpeg... Don't know whats the problem. image
  • 614683998-ca77b8a4-7215-4117-89c4-d4509fd4c9ce.png

    614683998-ca77b8a4-7215-4117-89c4-d4509fd4c9ce.png

    539×195 | 17 kB

Permalink
Comment by @gzpr
RexSystem 1 vote edited originally by @gzpr on GitHub
On a fresh instance using default settings uploading media does not work at all for me. Changing my avatar used to work, but I just deleted all the docker volumes after pulling the latest images, and now that doesn't work either. Error messages I'm seeing when trying to change my avatar: 
seaweedfs-1         | I0630 23:20:38.927918 node.go:220 topo failed to pick 1 from  0 node candidates
seaweedfs-1         | I0630 23:20:38.927932 volume_growth.go:140 create 7 volume, created 0: Not enough data nodes found!
seaweedfs-1         | I0630 23:20:39.128866 master_grpc_server_volume.go:140 volume grow &{Option:{"collection":"fluxer","replication":{},"ttl":{"Count":0,"Unit":0},"version":3} Count:0 Force:false Reason:grpc as
Error messages when trying to upload media i.e. sending an image to myself via notes: 
seaweedfs-1         | I0630 23:05:36.009682 master_grpc_server_volume.go:140 volume grow &{Option:{"collection":"fluxer-uploads","replication":{},"ttl":{"Count":0,"Unit":0},"version":3} Count:0 Force:false Reason:grpc assign}
seaweedfs-1         | I0630 23:05:36.009765 node.go:220 topo failed to pick 1 from  0 node candidates
seaweedfs-1         | I0630 23:05:36.009779 volume_growth.go:140 create 7 volume, created 0: Not enough data nodes found!
seaweedfs-1         | I0630 23:05:36.198569 master_grpc_server_volume.go:96 volume grow request for dc:DefaultDataCenter rack:DefaultRack failed: only 0 volumes left, not enough for 2
seaweedfs-1         | I0630 23:05:36.210742 master_grpc_server_volume.go:140 volume grow &{Option:{"collection":"fluxer-uploads","replication":{},"ttl":{"Count":0,"Unit":0},"version":3} Count:0 Force:false Reason:grpc assign}
As far as I can see all my buckets are present: `fluxer size:0 chunk:0 fluxer-downloads size:0 chunk:0 fluxer-harvests size:0 chunk:0 fluxer-reports size:0 chunk:0 fluxer-uploads size:0 chunk:0` EDIT: Managed to get it to work by adding "-volume.max=100" to the command line under seaweedfs: section in the docker compose, deleting the seaweed volume, and then restarting everything(I also had to recreate the fluxer bucket per previously provided instructions) That said, I really have no idea how weedfs or any other object oriented storage really works for that matter and added the command solely because an LLM told me to, I don't know if this might have some other undesirable consequences. It will still error out after the first upload, but all subsequent uploads seem to work without issue after that.
Permalink
Comment by @mothdotmonster
RexSystem 1 vote originally by @mothdotmonster on GitHub
gzpr's workaround fixed it for me as well. Here's what the full seaweedfs block looks like now under services in the docker compose file for me, in case that helps anyone: 
seaweedfs:
    image: chrislusf/seaweedfs:4.34
    restart: unless-stopped
    networks: [fluxer]
    command: ["server", "-s3", "-dir=/data", "-volume.max=100"]
    volumes:
      - seaweedfs-data:/data
Then I ran docker compose down and docker compose up -d and all is now good. I also have no idea why this fixes it though, sorry.
Permalink
Comment by @DerP4si
RexSystem 1 vote originally by @DerP4si on GitHub
same for me, but in my case it says that the file has a wrong format. I tried jpg, png, jpeg... Don't know whats the problem. image
I fixed it... the problem was the media proxy container which throwed an error. After I fixed it, there was no problem with files anymore.
  • 614683998-ca77b8a4-7215-4117-89c4-d4509fd4c9ce.png

    614683998-ca77b8a4-7215-4117-89c4-d4509fd4c9ce.png

    539×195 | 17 kB

Permalink
Deleted comment
Removed by moderator Rex: Removed a general status note that was posted on many GitHub threads. It no longer applies here.
Comment by Hampus
HampusStaff 1 vote originally by @hampus-fluxer on GitHub
This should be fixed via #1654 and #2476, with #3076 for the second problem further down. An existing install needs the updated stack files, for example via sh install.sh --update. If your bucket is already missing or your volume slots are already used up, you may still need to create the bucket once by hand (as described above) or clear the SeaweedFS volume.
Permalink