Your reasoning is not too far off.
Say you attach a signal generator to an antenna, and then probe the magnetic and electric fields at many places around this antenna in a test chamber. The ratio of the electric field strength to the magnetic field strength is called the field impedance.
For any antenna, several wavelengths away (in the far field), this ratio will be approximately 377 ohms (an ohm is a volt/ampere). This is a physical constant called the impedance of free space.
But closer to the antenna, the field impedance can vary by design. Let's compare a dipole that's small relative to wavelength, to a loop of similar size. Very close to the antenna, the field impedance for the dipole will be high, and low for the loop. And the intuitive explanation is as you expect: the loop is a short circuit, and the dipole an open circuit.
By reciprocity, the field strength measured when the antenna is transmitting is also proportional to its sensitivity when receiving. So very close to the antenna, the short dipole will be better at detecting electric fields, and the small loop will be better at detecting magnetic fields.
Paradoxically, at some intermediate distance which isn't very near the antenna, but is also not far enough away to be in the far field, the situation reverses. W8JI has this great graph:
If the vertical axis were logarithmic, these two antennas would each have a field impedance that's the mirror of the other. This is an example of duality.
You asked specifically if loops have reduced sensitivity to atmospheric noise, and the answer is no. Since such noise is in the far field of the antenna, a dipole and a loop will perform identically if all else (efficiency, polarization, feed arrangement, height above ground, etc) are equal.
However especially on HF, there can be quite a lot of noise in the near field of the antenna. Since the near field impedance of these antennas are quite different and complementary, it is often the case that a loop does not pick up the same noise sources that a dipole does.
You also mention a folded dipole, so I emphasize the above explanation works only for dipoles or loops which are small relative to wavelength. Folded dipoles are often a half-wavelength long, that is, the size of resonant ordinary dipoles. Other than the higher feedpoint impedance, they work just like ordinary dipoles. Just why this is merits a question of its own: How does a folded dipole work?