We'll have to lay down some rules for the comparison:
- For a fair comparison, the energy per bit in each scheme under consideration is constant. So, a "fair" alternative would be switching from 4-PSK to 8-PSK while maintaining the same transmitter power and halving the symbol rate.
- Likewise, the noise spectral density is constant.
- Forward error correction, interleaving, etc. are not considered, as it's assumed these methods would be applied with equal efficacy regardless of the underlying modulation.
- "Weak signal" means we wish to maximize the probability of correctly decoding a bit under noise-limited conditions.
- The comparison is in an additive white Gaussian noise (AWGN) channel, that is one where there's only white noise, and no multipath, etc.
- Frequency shift keying and phase shift keying are the only options.
- Implementation details (complexity, non-ideal properties of the receiver, etc) are irrelevant.
If we consider just binary phase shift keying or frequency shift keying, PSK wins handily. You can even halve the symbol rate and use 4-PSK, and get the same performance as 2-PSK in half the bandwidth. The bit error rates for binary PSK and coherently detected1 binary FSK are:
$$ P_{b(BPSK)} = Q \left( \sqrt{2E_b \over N_0} \right) \tag 1 $$
$$ P_{b(BFSK)} = Q \left( \sqrt{E_b \over N_0} \right) \tag 2 $$
Where $N_0$ is the noise spectral density, $E_b$ the energy per bit, and Q a scaled version of the error function:
$$ Q(x)=\operatorname {erfc} \left({\frac {x}{\sqrt {2}}}\right),\ x\geq 0 $$
Note the $2$ in the $2E_b$ factor of equation 1 means transmitter power goes twice as far for BPSK.
4-PSK has the same BER as 2-PSK: it is effectively 2 orthogonal 2-PSK signals. It's easy enough to understand intuitively, examining the constellation diagram:
Each symbol contains two bits. If we're left of the vertical axis, the first bit is a 0. Otherwise it's a 1. Since the first bit is determined only by the position left or right, the position up or down is irrelevant, or orthogonal.
Orthogonality is key. Taking this idea further to 8-PSK results in an increase in BER since the symbols are no longer orthogonal.²
By Splash - Own work, CC BY-SA 3.0, Link
Now back to FSK. With binary FSK there are two tones, each conveying a 1 or a 0. But as with PSK the order can be increased: if we have 4 tones, each maps to two bits. With eight possible tones, a single tone conveys 3 bits.
With PSK, as the order was increased the bandwidth went down and the error rate went up. With FSK the opposite happens. This is because unlike PSK, with FSK we can add more tones while retaining orthogonality, provided there's sufficient bandwidth to space them apart.
Why does this work? Imagine a receiver that looks at a single frequency and has to decide if a tone is present there or not. At some transmitter power and tone period this will work reliability, and each period communicates one bit. Say tone present = 1, and absent = 0.
Now if there are 16 such receivers, the detection of a tone present communicates not one bit but four (there are sixteen possible combinations of four bits). The more possible tones, the more bits are communicated per tone, with no additional transmitter power.
Or equivalently, we could slow the symbol rate but hold the bit rate constant, putting more time and thus energy into each tone and making it easier to detect. Conveniently, decreasing the symbol rate also decreases the minimum spacing between the tones to maintain orthogonality.
Atlanta RF provides a nice graph:
The weak-signal performance-enhancing properties of wide bandwidths are reflected in the (modified here to assume white Gaussian noise) Shannon-Hartley theorem:
$$ C = B \log_2 \left( 1+\frac{S}{N_0 B} \right) $$
Where:
- $C$ is the channel capacity, in bits per second
- $B$ is the bandwidth, in hertz
- $S$ is the signal power, in watts
- $N_0$ is the noise power spectral density, in watts per hertz
So while increasing the bandwidth does decrease the SNR term, it increases the $B$ term faster.
1: Coherent FSK detection is the most complex, and most accurate method of detection.
2: However, it does improve spectral efficiency. Many applications are bandwidth limited, so higher-order constellations are popular. For example, Wi-Fi uses at the higher data rates 64-QAM to cram the highest bit rate possible in the limited bandwidth available.