Given a middling-power rig (up to 100 W) requiring a 50 Ω antenna, a few options for the coaxial cable with a 50 Ω impedance are
Apart from cost, what factors are relevant in deciding which coaxial cable to use?
Amateur Radio Stack Exchange is a question and answer site for amateur radio enthusiasts. It only takes a minute to sign up.Sign up to join this community
The most important thing about the cable is how much loss are you willing to accept in the feed line. Figure out how much cable you will need, and then determine what the loss on said cable will be. As I mentioned on my website, here's a few good rules of thumb:
Longer cable equals more loss. Higher frequency is also more loss. Here's a few websites that talk about what the loss is per length and frequency of each cable:
Your question is missing an important item; the frequency at which you're operating.
I work a lot of VHF+ contests, and all four of the varieties of coax you mention pose far too much loss of signal to be viable for those frequencies. I use LMR-400 Ultraflex for 6 meter, and 3/8" Andrews Heliax hardline for 2 meter, 222, and 440 bands.
http://www.w4rp.com/ref/coax.html is a great resource for tracking how much loss coax presents at various frequencies.
Another point you don't mention is where you're intending to use the coax. Tower applications require different coax considerations than a mobile installation, and SOTA requirements are different still.
In addition to characteristic impedance (such as 50 ohms) and cost, which you've already considered, the two main factors for coax feed line types are line loss at a particular frequency, and velocity factor. The line loss varies depending on operating frequency, so it depends on your application. For line loss, the main factors are what frequency bands you're going to use over the feedline, and how long the distance is between the transmitter and the antenna. For VHF and above, line loss is a much bigger consideration. Velocity factor comes into play if you are going to create matching stubs or if you are working with non-resonant antennas like the G5RV and need to avoid certain lengths.
Other factors include how flexible the line is (braided vs. solid wire), how much motion it will endure (rotors, portable operations), how waterproof it is (permanent or underground installation), how hard it is to work with (LMR-100 is notoriously hard to attach connectors to), etc.
Although you could get into a length debate about all the different properties and what is best, most people boil down to the one big decision: "what is the loss per foot or meter of this cable". To check out various cables, you should use a Loss Calculator (there are many if you search the net). These will let you put in the length of line, the type of line, etc and you can see the results for what you'll lose, power-wise, by sending a signal over it.
Another big component, is what does it have to go through? Yeah, RG-8 is awesome but you're probably not going to run it throughout your car. It's not flexible enough and would require some whopping big holes. (ok, most people would suffer the holes, but the flexibility of using it would be a problem). That's the reason that many antenna mounts come with something closer to RG-58 or worse. The cable is much easier to run in small spaces. It's a short distance, at least!