E2EE does prevent listening in but it looks like E2EE clients don't require other clients to have encryption. So you can talk to other clients but not listen in without the key. Might want to require e2ee clients to only output audio from other e2ee clients and ignore unencrypted audio.
I don't know much about livekit or e2ee setups but I did notice the server gateway is handing out the e2ee keys. Not sure if that's just a shared secret for the clients to then negotiate their own actual keys or how livekit does its PBKDF2 magic but might need a better audit on the e2ee voice long term.
Thread
Comment by @mcd1992