What is the function of this filter? Receiver preselector or transmitter anti-spurious emission? The judging criteria would be different based on the intended use.
You omitted the filter design detail, but it appears to be Buterworth or very low ripple Chebyshev. However, the VNA traces appear to be a Chebyshev filter with some significant passband ripple, at least in the way you adjusted the filter. So, the impedance fluctuation in the passband probably results from how you adjusted, but it is also somewhat expected.
No analog filter design maintains 50 ohms for the entire passband. To mitigate your concern and give some allowance for the component tolerances and adjustment errors, you should design a filter with a passband that is quite a bit wider than the actual range you use.
Your S11 looks ok. It looks good for receiver applications or low-power uses but less than excellent for a transmitter power line.
If the filter was for transmitter power line, I would recommend using a low-pass rather than bandpass. Bandpass will have greater insertion loss for any level of harmonic attenuation, and the high-pass portion of the band-pass is not particularly useful.
The insertion loss of this filter seems a bit high. Your filter might be adjusted for a too high Q point, and the inductors may not have enough Q values. In that case, one would expect a higher insertion loss and not very steep transition bands. That may be what we see on the traces.
This filter looks reasonable for applications like post-mix or multiplier cleanup purposes (but we don't need to do those things for 80m).
When asking a filter question again, please include your design objective (filter's function, transmitter's design outline, required spec, etc., to give some context) and the filter design parameter panel on the left side of the Marki design calculator. That will get more focused answer.
