This will happen if the host serving the avatar URL is temporarily unavailable and we'll cache the failure for a little while before it starts working again, matching these symptoms. Since remote websites' reachability is outside of our control, this is working as intended at this time. Thanks for the report!
Thread
Comment by Hampus