Inaccurate meme - the white and red RCAs in composite typically don't actually carry the left and right channels - usually, the white one is L+R, meaning both the left and right channels combined into one, and the red one is L-R, the difference between the right and left channels.
This is done so that a mono television, which will only have a yellow and white port, will still be able to hear both audio channels, as opposed to having to completely miss out on one of them
Most N64 games don't actually use the controller pak, instead using their own battery saves