AllBugsFeaturesChangelogSearchNew post
Sign in with Fluxer
Features

Per-user third-party media storage solution

(#983) Feature Under consideration media self-hosting
by RexSystem edited originally by @lexifuzzpup on GitHub

Problem

When you send a piece of media, it has an expiry date, which for my friends, is a deal-breaker when moving to this application. I understand that it's for saving storage space and is an essential part of the early development of this application, but there should be another option for uploading files with an indefinite lifetime.

Proposed solution

Implement an option (in user settings) where the client can automatically upload to an alternative media bucket (self-hosted, perhaps another application for Fluxer) and send a link (or other embed) in place of native file hosting. Then only a file mirror should need to be hosted, so no server space would be used in serving the content.

Notes

Process

1) A user types a message and adds an attachment 2) When the attachment is sent, if the client has a custom configured media bucket, then their app makes a request to their personal backend 3) A public media link is returned from the personal backend, which is embedded in the message as a special type of file (or just as an embedded link)

Proposed MVP

A simple Docker container that, when run, serves as an alternative media bucket for Fluxer. A user then connects it to their Fluxer account, so files are uploaded there, instead of uploading to the main Fluxer servers. When other users load the sent embed/file, a request is made to the author's alternative media bucket (likely through a mirror, to protect IP addresses) and displayed on the other user's client. I suppose an example link could be, if I hosted an instance of this hypothetical application: 
https://fluxer.lexifur.me/media/1474100806723788885/128a2215-1ba3-4236-8dfd-068a357c43b3.mp4
^^^^^   ^^^^^^^^^^^^^^^^^       ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 TLS   media bucket domain         fluxer user id         randomly generated media UUID

User requirements

  • Some amount of actual storage
  • A self-hosted, dedicated server with Docker
  • A custom domain
  • SSL/TLS for their domain

Expansion

  • Multiple users can join an alternative media bucket, added via an Oauth integration
  • Servers/guilds can specify their own alternative media bucket, where media is stored for any user sending files within
  • The docker container can integrate with other remote file servers/protocols, including S3 buckets, FTP/FTPS/SFTP, SMB, Webdav, etc. and act as a broker to other storage solutions rather than a database, in the case of huge data

13 comments

Sign in with Fluxer to comment and vote.
Comment by @Jonesyandbeast
RexSystem 1 vote originally by @Jonesyandbeast on GitHub 1 reply
What if we also had an option in settings to have select file saving on the users computer. This may cause issues with storage, so we could spread media storage across members of the server. If a user's limit passes an arbitrary limit(say 5 gb) we could then start expiring media. This could work especially well if the most recent percent(5-10% of allocated storage) was duplicated across all users storage for offline access with the rest being selectively allocated among users. Let me know if there are any flaws in this logic or if anyone else has ideas for this type of storage solution.
Permalink
Comment by @lexifuzzpup
RexSystem 1 vote originally by @lexifuzzpup on GitHub OP
I think a decentralized storage system would be interesting, but not necessarily something that should be worked on right now. It would be a fun serverless implementation of the storage issue, but from my experience, they take a long time to implement and can be unreliable. I see some flaws and implementation detail issues with this idea:
  • It doesn't necessarily solve the file deletion issue, since they'll still be deleted above a threshold.
  • Fluxer has users who access the app using various devices, and not all devices are created equal. Some people might be running beefy computers, while some might be using a mobile phone from 2011, so how should media sharing be decided?
  • If a user is offline who holds media, how is that file retrieved?
  • Decentralization adds to the turbidity and uncertainty of communities; some files may be unavailable at times or may get corrupted. For example, someone being the sole possessor of a piece of media allows them to manipulate (albeit solvable with crypto signatures) or delete the original file without the author's knowledge.
However, I do see some pros over the Docker image idea:
  • Fluxer doesn't have to store media on the server, for users who have decentralized media buckets enabled.
  • If a torrent approach is used, then media can be delivered a lot faster to retrievers.
  • Users don't have to set up and administrate a Docker container.
Permalink
Comment by @peq42
RexSystem 1 vote originally by @peq42 on GitHub
I love the idea of self hosting media, this gets an upvote from me
Permalink
Comment by @omrih4
RexSystem 1 vote originally by @omrih4 on GitHub
Would it be possible with federation so media is uploaded to the users home server (ie a self hosted one), even when chatting on a community on, for example fluxer.app
Permalink
Comment by @Shanesan
RexSystem 1 vote originally by @Shanesan on GitHub 1 reply
Though I love this idea in concept, in practice I can't imagine a lay person to be owning a domain, let alone running a docker container. SSL? Can't see it. However what might work is supporting the dozen different Drive systems, like Proton, or Dropbox, or whatever to start, and then expanding beyond that to personal storage containers - or even better, locally hosted Drive systems users might already have, like Synology or Nextcloud, would be really cool. Another issue with local, individually hosted solutions is that if their computer is off (if they're hosting it there - lay people, right?) a lot of the context of the conversation has now 404'd.
Permalink
Comment by @lexifuzzpup
RexSystem 1 vote originally by @lexifuzzpup on GitHub OP
I agree that it's a major drawback. I was imagining sort of "the techy person of the friend group" setting up a server for a few people, but it might be easier to integrate with third-party services like that. Maybe, in the beginning, third-party services could be supported, and once development settles a bit, the self-hosted option for individuals and servers could be implemented. I see the idea of self-hosting storage and privacy to be on-brand for Fluxer, since the author has mentioned federation and E2EE chats in their roadmap.
Permalink
Comment by @JLUsr
RexSystem 1 vote originally by @JLUsr on GitHub
In general, having the option for users to define their own CDN (be it self-hosted or another service) would be stellar. If a service goes down, existing media can fallback on placeholders. The only concern I have is how media is handled during upload when a service defined by the user goes down temporarily. Do you have it alert the user and fallback on fluxers default CDN automatically or should the user receive a prompt asking what they would like to do? There needs to be a good balance of usability and control for the end-user, I just don't know what that balance looks like.
Servers/guilds can specify their own alternative media bucket, where media is stored for any user sending files within
I absolutely love this; however, I do think users should still be able to choose between their self-hosted/custom setup or the server/guilds media storage. If I am going to self-host my media I would like to clearly have a choice when joining a server/guild (or in a per-server settings page) to choose where my media is stored.
Permalink
Comment by @coldreindeer
RexSystem 1 vote originally by @coldreindeer on GitHub 1 reply
Am I the only one wondering if self-hosted servers will get to keep files permanently when self-hosting is eventually available? I think it would be good if there was a role permission (ie server owner or admins can use) to ignore message expiration dates. So you can set up rules channels for example with images that do not need to be re-posted after a few years. Or even a channel type that isn't for typical messages. Like most servers will probably have a rules channel and those messages should not have to be re-posted. While most messages are fine to expire some need to be permanent.
Permalink
Comment by @coldreindeer
RexSystem 1 vote originally by @coldreindeer on GitHub
In regards to ignoring the expiration, perhaps there would be a icon you can click on before sending a message to pin a message if you have a certain permission, somewhat like how pinning a message would work, but without pinning the message and instead just removing the expiration.
Permalink
Comment by @rogue-agent
RexSystem 1 vote originally by @rogue-agent on GitHub
I would hope for Fediverse media to be properly and seeminglessly embedded.
Permalink
Comment by @nekoame-git
RexSystem 1 vote originally by @nekoame-git on GitHub 1 reply
Agreed, I have the same idea: let the server backend automatically (or manually) upload media files that have exceeded a set expiration time to a self-hosted image hosting service (such as a NAS) via API, and automatically replace the original media file information with the corresponding image hosting link. This can significantly reduce server storage pressure while ensuring that media files do not become completely invalid. However, I think there is one issue: how to ensure the authenticity of the original media files is not compromised after replacement? Perhaps a warning message could be displayed near the replaced link after the change. This approach might be more suitable for private, entertainment-oriented self-hosted servers—at least, I would be happy to use it on my server.
Permalink
Comment by @rogue-agent
RexSystem 1 vote originally by @rogue-agent on GitHub
The api could give the main server the files MD5 for storage and then every time the file is being called from the user defined storage, the MD5 has to match.
Permalink