Do the radials in a quarter or 5/8 wavelength antennas connect to the mesh of the coaxial cable?
Yes.
Every antenna, at the feed point, has two conductors entering it and connected to something. When an antenna is fed by a transmission line, that line also has two conductors (shield and center conductor, in the case of coaxial cable) which are connected to the corresponding points of the antenna.
In the case of a common vertical antenna, the shield is always connected to the horizontal conductive structure underneath the vertical element, regardless of the form that structure takes. (The reverse would not work as well, because there are usually incidental indirect connections between the coax shield and ground (whether earth or chassis) at the transmitter's end of the feed line — this does not totally “short out” the antenna system — it forms an unintended loop antenna instead — but it certainly means the antenna is not functioning as designed. Lots of things will work as antennas by accident, including mis-installed antennas.)
If so, how does that compare to the ground plane in a magnetic antenna, that is, the roof of a car, that is not electrically connected to the mesh of the cable?
The magnetic antenna mount has a wide metal disk in its base, close against the roof. This forms a capacitor, and as you should know, capacitors can pass RF. Thus, the roof is electrically connected to the shield of the feed line, just not at DC.
The best capacitive coupling would be obtained by using a mount with a large surface area. The other factor is the thickness of the dielectric (i.e. your car's paint), but if you're willing to scrape paint off then you might as well install a permanent hole mount instead.
The other differences are that the radials are short and sparse whereas a good roof is contiguous and thick, but they are serving the same function anyway. see this question and answer for more information on how the size of a ground plane or radials affects an antenna.