Thomas,
Thanks again! To test I removed the amplifier after the first mixer to get 9dB attenuation. Results are the same so I am happy.
W.r.t the level 7 mixers. I did not know the importance of IP3 when I started with my build and second, they where comparable cheap. Did some investigation and found some cheap ADE-42MH mixer that match the required frequencies and are expected give a IP3 of 17dBm. Going above that will require some serious cash and I will wait with that till I have ironed out all the other unknown's, like how to put the SA in a box? and shield the mixers against 25MHz leakage from the LO XCO reference, for the rest the SA is clean with noise floor at -105dBm with resolution bandwith of 300khz/30kHz from 100kHz till 1.8GHz. I also need to build a narrower resolution filters (if the ADF4351 phase noise will allow) but I'm contemplating why not do all resolution filtering from 30kHz and below using a final IF at 80kHz into the audio in of a PC and the rest in DSP. My home build VNA is already using a I/Q amplitude/phase detector in audio DSP so it should be doable. But my current log detector has 120dB dynamic range and even with a 24 bit audio input I'm not sure if I can reach 120dB. Still so much to learn.