Self-hosting Fluxer: findings, bugs, and patches from a real deployment

(#1037) Feature Shipped self-hosting

7mo agoby RexSystem originally by @mgabor3141 on GitHub

Thread

RexSystem 1 vote 7mo ago originally by @Mar0xy on GitHub 2 replies

Gonna leave this here for anyone interested in setting up Cassandra as I had to experiment with this for 3 hours

Setup Guide

1. Add Service to Compose

Here we will use the bitnamilegacy image as all the other cassandra images require you to directly have a full on cassandra config to get them to work.

cassandra:
    image: bitnamilegacy/cassandra:latest
    container_name: cassandra
    ports:
      - '7000:7000'
      - '7199:7199'
      - '9042:9042'
    environment:
      CASSANDRA_DATACENTER: dc1
      CASSANDRA_ENABLE_RPC: true
      CASSANDRA_USER: username
      CASSANDRA_PASSWORD: password
      CASSANDRA_PASSWORD_SEEDER: yes
    volumes:
      - cassandra_data:/bitnami/cassandra

also add the volume to the volumes section as cassandra_data:

2. Configure cassandra in the Fluxer Config

"database": {
    "backend": "cassandra",
    "cassandra": {
      "hosts": ["cassandra"],
      "local_dc": "datacenter1",
      "username": "user",
      "password": "password",
      "keyspace": "fluxer"
    }
  },

3. Prepare Cassandra

Start just the cassandra instance and wait for it to be fully booted (takes about 30-40 seconds) then run the following commands:

docker compose exec -it cassandra cqlsh -u user -p password
CREATE KEYSPACE fluxer WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

this sets up the keyspace used in the config with basically the simplest of replication options

4. Run Migrations

Now we will get to the hard part as the migration script is not included in the docker image you will have to run all migrations yourself one by one in the shell by going through each file in fluxer_devops/cassandra/migrations

5. Start Fluxer

After all this is done fluxer should be able to start without issues

Caveats

Cassandra tends to eat a lot of RAM so make sure you either host it on a separate machine if possible or you have a beefy server otherwise it will Exited with code 137 while fluxer is up and you get the dreaded loading/splash screen

Benefits

You no longer have to deal with the KV SQLite file meaning you can actually easily modify the table entries and etc like adding visionary to a user by running this query

UPDATE fluxer.users SET premium_type = 2, premium_lifetime_sequence = visionarynumber, premium_since = toTimestamp(now()), premium_until = null, premium_billing_cycle = null, premium_will_cancel = false WHERE user_id = userid;

RAM usage improvement

OOTB Cassandra takes up as much RAM as it can (in my case it was 8GB) you can limit the usage by adding this into the compose: environments section on the cassandra service:

JAVA_TOOL_OPTIONS: -Xmn2G -XX:+UseG1GC -Xms2G -Xmx2G

new deploy section in the service:

    deploy:
      resources:
        limits:
          memory: "2GB"

This should limit cassandra to 2GB of ram with the downside of some API stuff taking a bit longer to load

Permalink

RexSystem 1 vote 7mo ago originally by @mgabor3141 on GitHub OP

Have you noticed meaningful difference over sqlite? How many users?

Permalink

RexSystem 1 vote 7mo ago originally by @Mar0xy on GitHub

There are no real differences between cassandra and sqlite besides the fact that cassandra is better for redundancy/replication/scaling and that it isn't KV based like mentioned previously which makes managing the DB directly much easier. In terms of users my instance only has me and a friend as well as 2 bot accounts and it has been working fine so far after I applied a few more changes like setting this env variable CASSANDRA_CFG_YAML_DISK_ACCESS_MODE: mmap_index_only

Permalink