The constellation diagrams for a 4-ary modulation using PSK (qpsk),MSK ,QAM and OQPSK look alike. How to differentiate them?
In general, any modulation that makes sense to plot on a constellation diagram could be described as a kind of QAM. However, we generally use the term QAM only for modulations that use both degrees of freedom on the plot (amplitude and phase).
If the modulation varies the phase but not the amplitude — all the constellation points lie on a circle about the origin — then we call it PSK (phase-shift keying).
If the modulation varies the amplitude, but not the phase — all the constellation points lie on a ray from the origin — then we call it ASK (amplitude-shift keying).
So you can tell QAM from PSK because QAM displays multiple amplitudes as well as multiple phases, whereas PSK has exactly one amplitude. MSK and OQPSK are subtypes of PSK and you are right to think that you cannot tell them apart on a constellation diagram, but the question you show is not asking you to do so — only to distinguish between ASK, PSK, and QAM.
The distance of a point in the constellation diagram from the origin gives the amplitude while the angle each point makes with the horizontal axis gives the phase. PSK can't be the right option here because the amplitudes are different and we have the same phases with respect to the horizontal axis. However, in the case of PSK, we have the same amplitude and different phases(this depends on the order of modulation). For 8-PSK, we have a single amplitude and eight different phases. This means that the constellation points lie on a circle. This also eliminates option c.) This also eliminates QAM because QAM has multiple amplitudes as well as phases which is not the case here.