Here is an easy way to visualize what's going on with those pesky side lobes.
We take the case of a centerfed resonant dipole with a wire a little longer than the dipole mounted behind it by some fraction of a wavelength. That wire is intended to reflect the output of the dipole in the "back" direction (by absorbing and immediately re-radiating it) so it adds to the output in the "forward" direction.
We find that by playing with the offset distance, we can make the reflected wave be in phase with the wave train being radiated by the dipole, and get perhaps +3dB of gain in the forward direction, on-axis.
But things are different as we move off-axis. If we model each element as a point source and sweep a receiving antenna through 180 degrees in front of the element pair, there will be certain angles in that sweep where the direct and reflected waves are 180 degrees out-of-phase, and cancel out. What remains is the on-axis "main lobe" and two off-axis "minor lobes", separated from the main lobe by those "nulls" where the cancellation happens.
By adding director elements in front of the active dipole, we can further concentrate the output of the antenna in the desired direction by squeezing the side lobes inwards, further strengthening the main lobe- but at the expense of adding even more "nulls" to the radiation pattern where the phases of the waves in the beam interfere and cancel each other out. So instead of two side lobes, now we have three or four flanking the main beam.
So we see that by trying to further and further "confine" the radiation from the dipole into a collimated beam that gets stronger and stronger, we obtain more and more side lobes squeezed together around the main lobe.