You've pretty much hit the nail on it's proverbial head:
Both are power supply architectures, and as such generally fulfill the role of providing a constant-as-necessary voltage source (or current source, for some applications, but we're most likely talking about voltage sources).
Now, you can imagine a linear power supply simply as a self-adjusting resistor. It adjusts its resistance exactly as much as to drop enough voltage so that the voltage between its output terminal and its ground terminal is constant.
Of course, that means that $V_{drop}\cdot I_{output} = P_{wasted}$, i.e. all that voltage drop is waste heat. You hence simply can't do that for overly large voltage drops or overly large currents, because things get hot. Things getting hot means you need beefier voltage regulator ICs and larger heat sinks. That's expensive. For large voltage drops, and security critical things, you simply can't do that, because no (affordable) semiconductors exist that support both the power and sufficient isolation.
Now, a switch-mode power supply works by having an energy storage from which it takes small portions of energy. In a typical topology, this means that you have your input voltage over a coil, which of course starts to build up a magnetic field. Then, you switch the in- or the output off and on again, to bring the voltage across that coil onto your target output voltage. The energy taken by the output in shape of current is taken from the coil's magnetic field, so that the amount of switching on and off needs to accomodate the target voltage and the amount of current drawn, and the fact that a higher input voltage can "recharge" the coil faster.
With a bit of cleverness, you can, that way, even step up voltage! Something that's impossible to do with linear power supplies.
If you look at the power wasted, there theoretically is zero: energy is just being converted from input electrical current to magnetic field and back to output current at a different voltage, and all these processes are theoretically lossless.
In practice, you of course get the resistances in your coil's wires and the losses in the coil's core as well as resistances in the switch (typically, a transistor), so things are maybe 80% to 98% efficient – but that's way, way more efficient then let's say converting 1A of current from 12 V to 5V:
With a linear supply $I_{out}=I_{in}$ (plus losses), so $P_{in}=V_{in}\cdot I_{in}=V_{in}\cdot I_{out} = V_{in}\frac{P_{out}}{V_{out}}=\frac{V_{in}}{V_{out}}P_{out}$, with a switch mode supply $P_{out}=P_{in}$, only! Which means that the linear supply scheme uses $\frac{12}5=2.4$ times as much power!
Regarding noise:
Of course, switching necessarily means that you switch a current on and off, and you'll see that on the output side of your coil, as ripple. Typically, a smoothing capacitor takes the job of reducing that ripple to a degree sufficient for the electronics supplied to work.
However, reducing doesn't mean getting totally rid of, and you'll need to design a smoothing stage that fits your need: if your need is just driving a microwave oven's clock display, ripple-rich (hence, noisy) output voltage doesn't matter to you at all. If you're trying to pick up some weak transmission on a frequency close to the frequency your switch mode supply works at (or one of its harmonics), you'll have a harder time.
In professional lab and SDR equipment, which still often is cost-, size- and power-restricted, you'll often actually find a cascade of power supplies; for example:
A set of switchers to step down the 6 to 9 V (which usually comes from a switch-mode "wall wart", converting grid to something safe for humans to touch) input to 5 V, 3.3 V and 1.8 V; the last voltage to be converted to 1.2 V through a low-drop linear supply for the high-speed digital logic and memories, 3.3 V for direct usage by the digital interface ICs, which don't care about noise nor very exact voltage overly much, and 5V to be regulated with another set of linear supplies for all the analog signal chain, with plenty of RC low pass filters in between the switching power supply, the linear regulator, and if small and constant current draw allows, between linear regulator and the device (amplifier, mixer, adjustable attenuators, microwave switches...).