how does slotted coax Balun eliminate common current? why the length of the slot should be quarter wavelength?
The slotted coax balun is very similar to the folded balun (aka Pawsey stub). I wrote about the folded balun in my article...
The references at the bottom of the article provide some good quotes to consider. There are two things to consider for your question: why and where the split balun is shorted back together.
Why the split balun is shorted
Balanis discusses the theory behind matching techniques including the 1/4 Wave Coaxial 1:1 Balun. In particular he describes the purpose of the electrical short of the coax center conductor to shield, is to maintain balance, thereby, ensuring no current flows back to the transceiver on the outside of the coax. As both halves of the split balun are fed by energy at the feedpoint 180 degrees out of phase, when the currents join at the shorted end, they are in balance and just act like a short between each other with little eagerness to flow current to the third conductor (the mast or feed system).
I've built several folded baluns for various tests and do find they put a hard stop on any common mode current that tries to flow back down the feed system and over essentially all frequencies (or at least frequencies well outside the antenna's operation band). It may well be erroneous to say all common mode current is stopped, but my measurements suggests certainly most is quelled by the, essentially, short across the end of the feed line.
Where the split balun is shorted
If we make the split balun portion much shorter than a quarter-wave at the desired operating frequency, we can see we are simply shorting the feedpoint of the antenna and nothing will work optimally. However, if we place this short exactly one quarter-wavelength back along the feed system, we can surmise the transformer action of a quarter-wave length of wire will convert the impedance at one end to a different impedance at the other. The classic example of a quarter-wave wire transformer are the radials and vertical of the classic quarter-wave monopole antenna. The high impedance present at the end of the radials (because they are in freespace) is converted to a low impedance 1/4, 3/4, 5/4, etc. wavelengths away from the tip. In this way, the radials present a low impedance (opposite their tip impedance) to the center of the antenna thereby drawing lots of shield current and leaving less to flow back down the antenna mast or feedline.
The split or slotted coaxial balun works the same way, but in reverse. In this case the location where the split balun joins is considered a direct short to each other and the feed system. This low impedance is converted through the 1/4 wavelength of conductor to a high impedance at the feed point. In this use, the high impedance presents a strong rejection to the relatively low impedance currents and keeps them from flowing back down the line thus keeping the currents on the intended antenna elements.
Balanis summarizes this much better:
“The parallel auxiliary line need not be made 1/4 wave in length to achieve the balance. It is made 1/4 wave to prevent the upsetting of the normal operation of the antenna.”