On some (most?) receivers, if the RF gain is turned up too high, intermodulation distortion increases due to increasing non-linearities (maybe even clipping, etc.)
I'd argue that's all receivers, even, since that's exactly what "turned up too high" would imply. You get a non-linear effect, and that always leads to intermodulation. If that doesn't happen (significantly), you can increase gain ;)
Is it possible, just by examining typical received over-the-air RF signals during normal operating (e.g. no testing via artificially injected 2-tone signals), to detect the amount of IM distortion in order to determine whether the front-end RF gain should be reduced (or attenuation should be added) to lessen the distortion?
Ah, that requires a bit of knowledge on the side of your signal. If you don't know anything about your signal, it's hard to tell whether it's distorted.
Either by listening to the demodulated audio or by examining the FFT spectrum?
Haha! Demodulated Audio means you know quite a lot about the signal, right?
So, the first place to look is simply at a time-domain amplitude statistic: if the amount of amplitudes that are really high is unusual for your type of signal. The classical stochastic property that reflects how much the signal amplitude distribution is not at its center of weight is kurtosis, i.e. the fourth central moment.
Sadly, that's not something you "see" in the spectrum. What you do see is the spectral widening of the channel.
Note how instead of the single central channel, the whole spectrum is filled with intermodulation products. This might be an extreme example, but it illustrates the issue well.
When dealing with modes with (unsuppressed) carrier, a typical sign of things that look a bit or ever more like a square wave and not like a perfect sine due to overdriving are lines in the spectrum at equal distances. Whenever I see a nice comb on a spectrum display, I'm pretty sure I see the leakage of a digital clock, or an overdriven sine wave.
There's other indications. If your noise floor pulses: it's kind of unlikely that natural noise sources do that, and unlikely that a lot of interferers decide to synchronously go up and down in power. More likely is that a strong signal in a strong moment leads to more intermodulation, and a weaker strong signal to less intermodulation.
If you know you're dealing with OFDM (e.g. in the FreeDV voice mode), then that's a spectrum with steep edges, and small sidelobes to either side. If your channel was flat, it'd be flat-top (but for a few sinc-induced ripples), but you're using OFDM because your channel isn't flat, so you'll have something that has a few trenches/hills, but still has a steep transition at the edges. If the spectrum starts looking symmetrically round, then the best explanation is usually nonlinearity.
If you have a digital system with constellation points of different power: If the outer points seem to suffer a larger amount of noise variance, that's nonlinearity that's worse for large amplitudes.
It's hard to pinpoint a "audio" effect for too high gain, because all modes are affected in different ways: FM is relatively robust (there's but one tone in your channel at any time, so not really much to intermodulate with, so your receiver's PLL will track that tone usually quite reliably), whereas AM will suffer intermodulation (which would probably sound like a really weak amplifier trying to drive a large speaker, or a small speaker being beaten to death by a monster amplifier).
