Audio isn't send just over the ground pin, or the other pin. It's sent over both. If both weren't necessary, then why not just make audio connectors with one pin and audio cables with one wire?
The audio signal is represented by a voltage at the input. And the source of the signal manipulates that voltage by adding or removing electrons, what we usually call "current". And since electrons aren't just magically created, but must come from somewhere, where are they obtained?
The answer is the signal source doesn't create or destroy electrons: if it wants to increase the voltage then it takes electrons from the "audio" pin and deposits electrons in the ground pin. And to decrease the voltage it does the opposite.
So naturally, if you try to measure "just" the ground pin, you're going to see something. Only what you see is going to be highly dependent on how you've connected the test equipment.
The more likely reason your mixer can't totally eliminate the signal is probably just because mixers don't work like that. The faders only attenuate the signal, perhaps to the point where it's no longer audible. Or perhaps not, depending on how much amplification follows the mixer output, how good your ears are, and how quiet the room is.