Antenna efficiency can be defined as the ratio of the radiation resistance to the total resistance of the antenna (radiation resistance, plus ohmic resistance of the antenna, plus ground losses, etc), normalized to the feedpoint impedance. If radiation resistance is the only resistance in the antenna, then it's 100% efficient.
$$ \text{efficiency} = \frac{R_\text{radiation}}{R_\text{radiation} + R_\text{ohmic} + R_\text{ground}+R_\text{any other kind}} $$
Performing this calculation can be tricky since there are so many potential sources of loss which may not be thought of as a "resistance", but can be expressed as such. Dielectric losses are a good example.
Loading coils and matching networks are significant only to the extent that they introduce additional loss. Sources of loss might include:
- resistive losses in the coils
- magnetic hysteresis losses in the core material of inductors
- dielectric losses in capacitors
- unintentional near-field coupling to nearby lossy materials (mast, trees, etc)
Shortening the antenna tends to decrease the radiation resistance. This doesn't make the antenna less efficient itself: it just makes other losses more significant, since they are now a greater fraction of the effective total resistance of the antenna.
It doesn't matter much if you use loading coils or "some sort of matching network". Loading coils are a matching network. If you take the loading coils out, your matching network, if at the feedpoint near where the loading coils would have been, will be an inductance of similar value. What matters is the losses you introduce with whatever matching technique you use. You can make very good loading coils that introduce no significant losses (fat, low resistance wire around a low-loss core like air), or you can make very poor coils that turn your antenna into a heater (very thin wire around a lossy iron core).
Shortening a dipole has a very small effect on the radiation pattern. As the dipole becomes shorter, it becomes less like a half-wave dipole and more like a Hertzian dipole. Wikipedia provides a nice graphical comparison of radiation patterns, with the half-wave dipole as a solid line, and the Hertzian dipole as a dashed line:
As you can see, the half-wave dipole has a marginally higher directivity, but they are close enough that for most practical concerns they can be considered identical.
