Unable to change profile picture or send most images because of false NSFW flag

(#343) Bug Closed: Fixed media moderation

Summary

Fluxer tells me when i try to change my profile picture that its explicit content even when it isn't, and most images are flagged as inappropriate for some reason.

Steps to reproduce

  1. Go to Profile in settings
  2. try to change your profile picture
  3. click save

Logs or screenshots (optional)

image image

15 comments

Sign in with Fluxer to comment and vote.
Comment by @rillabel
RexSystem 1 vote originally by @rillabel on GitHub
Just came to report this one as well, not sure what convention is so leaving a comment here instead
Comment by @angahith
RexSystem 1 vote originally by @angahith on GitHub
Also here to report this issue, except it happens to me when uploading stickers, where it won't even say it's because it considers it an explicit image, and just returns POST /guilds/xxx/stickers [400] Happens with several completely innocent stickers, as well as one that contains the word "gay", which is kind of ridiculous if that's considered explicit. Edit: Oh boy, yeah, this filter is... bad. I did some more tests and it has a tendency of flagging pride flags as explicit. For the record, this is flagged as explicit: image
Comment by @GlenMerlin
RexSystem 1 vote originally by @GlenMerlin on GitHub
It flags these in the chat too and lest anyone thing it's homophobic These four images were all banned image image image image but this one wasn't image I have a feeling this is either just a really poor NSFW filter or the filter is dying and 1/5th of images are actually getting processed and the rest are failing and the client is showing the wrong error
Comment by @GlenMerlin
RexSystem 1 vote originally by @GlenMerlin on GitHub
leading me to believe filter is dying. Looking at all of these images the last one is the lowest resolution out of all of them (nearly half of everything else) and extremely low resolution profile pictures get through just fine. I think it might just be server load or some kind of compression breaking
Comment by @jilek-josef
RexSystem 1 vote originally by @jilek-josef on GitHub
did fluxer use The Hive as moderation platform? it's kinda known to be trash and marks furry as nsfw
Comment by @DW01
RexSystem 1 vote originally by @DW01 on GitHub
Can confirm that the filter is busted in some way. I did some informal testing by sending both NSFW text and images to Personal Notes. It didn't trigger on the text or two of the images (which were character reference sheets with very visible dicks), but did on the third image, which had the dick featured much more prominently. It also shouldn't be triggering on Personal Notes or in DMs at all, which points toward a poorly-configured or implemented channel filter, and those features obviously being implemented as channels. E: Did some perusing of the code, and it appears that they're using an Open Neural Network Exchange model to make an inference on whether an image contains NSFW material on the media proxy level (globally). This is certainly not fine-grained enough to allow it through in some cases but not others; it should live inside the server itself imo
Comment by @ppsmol
RexSystem 1 vote originally by @ppsmol on GitHub
Same issue, only pics that went through was flat color, it seems it flags any face as nsfw. Didn't even use furry pictures, just anime screenshots, illustrations etc.
Comment by @DW01
RexSystem 1 vote edited originally by @DW01 on GitHub
Same issue, only pics that went through was flat color, it seems it flags any face as nsfw. Didn't even use furry pictures, just anime screenshots, illustrations etc.
This isn't universally the case, seems some people are getting hit harder than others. I managed to upload a couple of large-format reference sheets (>2000px wide) with faces and dicks fine, but the filter got me on lineart featuring a prominent dick. Yeah, this is very inconsistent. It just nailed the full version of the image I use for my avatar on the platform (different than this one; GitHub falls over whenever I try to change it here lol), which is just a headshot I drew of my fursona showing an additional bit of one of his wings, but DIDN'T nail the circular crop of the exact same image I actually use for my avatar. @hampus-fluxer What's going on here? Seems like the model being used behind the scenes is flagging huge categories of images as NSFW that very clearly aren't even adjacent, but randomly letting actual NSFW through if it's high enough resolution, I think?
Comment by @DW01
RexSystem 1 vote originally by @DW01 on GitHub
With the latest update on the move to new infra, I was able to upload the two images that previously got blocked by the filter's model to Personal Notes! Haven't tested DMs yet, but this is a good sign for sure.
Comment by @TheFurryGamer2016
RexSystem 1 vote originally by @TheFurryGamer2016 on GitHub
While this is an issue on discord, resolution of this issue doesn't really seem possible with current technology. Keep this unresolved and remove image checks, as this is a thing we've wanted to avoid with discord as well.