Thank you, Chuck, for clarifying that the calibrator output is 0V to requested level positive 1KHz.
Then it seems that I've done nothing wrong till now. But was defeated by LIMIT problem. I'll try checking the power supply ripple again making sure that everything is nominal. But the trimmed voltages stay where they are. I doubt that the ripple would be beyond the spec (the power supply and A5 board were re-capped).
Also have adjusted the vertical readout jitter. Minimizing it slightly affected the vertical gain (like you have stated before).
The only thing which I haven't done so far was the variable caps on attenuators for the channel 1/2.
I wouldn't think that the problem is with the hybrids, since the amplification and scaling is working.
Maxim