Is it feasible to determine what sort of modulation is used for a given signal by looking at the spectrum / FFT? Does software like this exist?
You can definitely identify many of them by eye, have a look at:
As for software that does it automatically, I'm not sure anything like that exists. If you find something, please let us know.