Problem
Long-term Fluxer instances may / will eventually run into a storage problem which is going to be partly caused by storing thousands of uncompressed JPEGs, GIFs or/and PNGs on their servers. This is same problem that some Matrix instance administrators ran into back in the day (think they still do, though it's better nowadays), maybe even Discord considering they're compressing all attachments by default now, and I suspect it will also be an enormous problem later for federation where servers will need to exchange data, as a single instance could completely nuke somebody's bandwidth and storage by having them federate potentially gigabytes worth of uncompressed screenshots, memes or other stuff off somebody's else server.
My point with this is that storage is not infinite and bandwidth is expensive, so any means of saving space is very appreciated and even needed in some cases for long-term operation. Compression would mean that we can store way more images before needing to look into expanding storage or pruning old media to free up space.
Proposal
- Allow people to choose to compress their attachments before sending them.
This can be opt-in from the client, or enforced instance-wide where compression of static media would occur on the server, sparing the resources of the device for this task. For example Matrix clients such as FluffyChat ask users if they'd like to compress their attachment before sending it, some clients ask users if they'd like to send their attachment uncompressed instead. If instance enforces it globally, this toggle wouldn't be shown to end-users that don't have specific grant to be able to send uncompressed media. (that one is just more of my own whimsy though)
- Let instance administrators enforce it globally from admin panel
Any instance administrator would be able to apply instance-wide forced compression with a checkbox inside the admin panel. This checkbox would force any attachment uploaded by users to be compressed by server, or user themselves via client.
- Customizable compression quality and adjustable strength & limits based on user limits
For example 'Free' members could have their images be compressed in a harsher manner (60% quality) where 'subscribers' could get a little bit better compression (80% quality) for
x codec.
- Ability to set compression levels based on uploaded image file size
Something akin to current Media Expiry settings, except it's for image compression. Any upload above
X threshold in bytes would be treated to
Y quality settings, or preset. This would be overridden partially with user limits but it's intended to be used with global enforcement of compression, giving administrators more control over how it works and how harsh or light it is.
- Remember selection by saving it to client settings
Each user should get an option to choose to 'permanently opt-in' ex. 'Don't remind me again' which could always be changed / revoked in Settings again. This option would only really always keep compression checkbox checked, or maybe always send compressed without any extra input from the user, or keep the option completely hidden, always sending uncompressed. This is implying 1st option is a thing where user gets given a choice to do it or not in an instance that doesn't enforce image compression globally.
- Emojis and Stickers should also undergo compression regardless of current settings
Subjective but they will be used a lot, especially during federation. Discord compresses all emojis to
.webp with exception of gifs and stickers but that might've changed. I know that Discord used to compress
.gif into animated
.webp for emojis, no idea if they're still doing it or not.
- Ability to treat older uploads with compression straight from the admin panel
There should be a option in admin panel for administrators to request compression of old media inside the S3 buckets in case they enabled the setting on an already running instance & still want to benefit from the compression on older uploads. This could have a date filter from-to, or by file size ex. anything above 10MB gets compressed, maybe even anything above 10MB gets compressed in a harsher manner.
Notes
Discord to my knowledge by default compresses all media without any ability to toggle it off when one so desires so Fluxer could do something better here. Proposal is more of me just intensely brainstorming and thinking of how things are done right now and how they could be potentially implemented so it may be a bit over the top. It doesn't have to be
that complicated but it can be for those who wish to customize even the littlest things.
While setting restrictive File Size limits could work, it still doesn't really do much aside from making it annoying on the end-user and forcing them to do all the work themselves while
still not saving any space or solving the core problem at hand. Those 1MB attachments could still be compressed! That 512KB JPEG you just uploaded? Could be reduced to 50KB! Free savings for
arguably not much drop in quality.
Technically gifs could also be compressed to something like animated AVIF or animated webp but that one is a bit more complicated to pull off properly and can sometimes result in a gif just arriving completely nuked.
---
Example of space savings
Base file is
1,032,076 bytes so around ~1MB. It's a screenshot from Hytale. Let's compress it with
magick and see what we end up with.
.avif at 80% quality: 106,362 bytes / 106.4KB.webp at 75% quality: 67,544 bytes / 67.5KB.avif at 60% quality (comparable to .webp at 75%): 64,518 bytes / 64.5KB
Pretty big file size reduction after compressing it. This of course won't always be the case and savings may be minimal. Here's the screenshot in question on my disk, uploading would be pointless here as GitHub compresses attachments afaik.
obraz
---
I've looked around but no one mentioned this specifically, unless it's in some meta discussion that I'm not aware of.
3 comments
Comment by @HeyBanditoz
Comment by @mizarc
Comment by @snappyapple632