I'm afraid I should back up again a bit. I retested everything after the third converter. First, I disconneccted A15J601 (the 10.7 IF from the third converter). Once I re-ran the IF calibration, the noise floor was more normal.
Next, I injected a 10.7 MHz artificial IF from an 8648B downstream into J3A4 (your block diagram shows this one), starting at -120 dBm. As I raised the power toward -15 dBm, the spectrum analyzer showed a reasonable reading.
I believe the problem is in the RF section, not in the log amp section, so I went further.
The service manual includes a section to confirm the 3rd converter output, which should be here, below (Pg 618). The 3rd converter output, under test conditions, should be ~ -15 dBm (the max power I also injected in my test above). The amplitude at 10.7 MHz is actually 18 dBm.
This would saturate the input section, correct?
What I don't yet understand is how decreasing the resolution bandwidth results in a reasonable signal. I have to play a bit more on that one.
Thanks for the help on this. I'm learning a lot, and it is quite an interesting puzzle.