This is really a mixed question. It's not the mode that dictates the polarization. It's the situation in which that mode is used that dictates it.
FM is typically only used at VHF and above frequencies where we have enough bandwidth to use it. VHF/UHF being line of sight is the choice for local work either simplex or via a repeater. For obvious reasons, mobile antennas are all vertical. Therefore all the repeaters also use vertical antennas.
Since it's line of sight and you aren't bouncing off the ionosphere, the polarization of the signal stays vertical. Hence, working FM, vertical is the way to go.
That's not to say you can't work horizontally, but you'll give up a lot of strength. Of course you could setup a QSO with someone else also going horizontally and it would work just fine.
By contrast, SSB and CW work is mostly in the HF and low VHF range where vertical antennas are less likely to be a good idea. There are some folks that work VHF SSB, but that's pretty rare, at least in my area.
At HF frequencies, you are going to get some hop and the polarization gets randomized by the ionosphere on the hop. So, for HF work, the polarization you start with doesn't matter.