SSB phone requires good frequency stability to be intelligible. Anything over 10Hz of frequency difference between transmitter and receiver and the signal starts to sound funny; beyond 50-70 Hz of error, intelligibility starts to drop. Hitting that target is more difficult at higher frequencies than at low frequencies. In the HF bands, getting within 10Hz requires 1 part per million (order of magnitude) relative frequency accuracy. For VHF it's more like 0.1 ppm, UHF is 0.01 ppm, and microwave we start thinking about GPS-locked oscillators.
FM, on the other hand, is pretty insensitive to frequency errors. For an FM signal with a deviation of 5kHz, a frequency offset of 1kHz between transmitter and receiver is almost unnoticeable, and even 2 kHz doesn't impair communications too much. So for practical purposes, let's say FM is 50 times as forgiving.
Even base station and mobile VHF/UHF rigs are much more likely to have FM than SSB for this reason — it's simply much easier to get acceptable FM at those frequencies than acceptable SSB. "All-mode" rigs are usually (though not always) the more expensive ones.
HTs have the same problems to contend with plus they're generally lower-cost, need to be smaller and lighter, and generally use unregulated battery power that tends to droop over the course of a transmission. All of those negatively impact frequency regulation, and keeping a signal accurate to within a few tens of Hz under those conditions is simply impractical.