In the article Measuring HF Balun Performance the author (Ron Skelton, W6WO) states:
The Common Mode Rejection Ratio (CMRR) of a balun is defined in professional literature as the ratio of wanted to un-wanted transmitted power
This sounds like a reasonable start: the claim is that the CMRR is a good measurement of balun performance.
He continues:
Measurements and calculations of CMRR [Common Mode Rejection Ratio] can be made by directly measuring impedance in the common-mode path, or by observing power flows using a power meter or oscilloscope
…but then goes on to say that this is subject to errors due to phase imbalances that would not be detected with simpler methods.
Any balun or line choke can be viewed as a 3-port device.
The unbalanced input would count as the one port, and each of the balanced outputs (referenced to the same ground as the input, afaict) is another port.
He then gives the equation:
CMRR dB =20*log (S21+S31)/(S21-S31)
This basically compares the difference in power transfer from the input to each of the outputs (relative to each other).
So to use this technique, one would need at least a 2-port VNA to measure first the signal transfer to the one output, and then the other. Then a calculated graph based on those two plots could be derived using the above formula.
I don't think this method covers all the things that can go wrong with baluns, however. For example, this measurement doesn't seem to tell you anything about the power handling capability of a balun, nor what happens if you exceed it.