To make a first comparison, here is the output of my 8753ES at 1GHz, CW mode, 0dBm output, Trigger HOLD, measured on a Spectrum.
?
If you have a similar image, the PLL A11 would be correct.
You do not have an UNLOCK message, this means that you have enough signal to maintain synchronization.
?
If the signal going to the sampler R was attenuated, the average during an S11 measurement would give an average signal offset from the reference (when nothing is connected to port #1).
But looking at your image, test S11, nothing connected to port #1, apart from the noise everything seems correct.
?
Do a test from 30kHz to 250kHz, the A11 module is bypassed, if the problem persists, A11 is not the cause.
?
Either the problem comes from A7, or it is further away like on the IF DIGITAL.
?
Make measurements on the samplers separately, MEAS A, B and R. Do you have the noise problem on all samplers.
?
I will wait for your answer, and I will be able to make comparative measurements on my VNA 8753ES.