Your calculation is not fully correct as when sung a span of 120MHz (118MHz) your noise power is directed to the mixer via the 800MHz low pass filter so the math is 10*log(800E06) = -88 + 89= +1 dBm and on the brink to be too high. Using Ultra the bandwidth it is even higher
Follow up to my last post.? Below is a s21 plot of a Para Dynamics low pass TX filter that was done using a NanoVNA-H4.
One method to measure a filter with a tinySA is to use a Noise Generator (NG) but one has to be careful that the NG output is not too high because it might damage the tinySA.? I have a lab NG that outputs 100 mV RMS into a 50 ohm load? (-6.9 dBm) when set to a specific noise bandwidth. In the plot below? the tinySA measures -88 dBm/Hz with the trace marker set to Noise.? With 120 MHz BW the total noise power is 10*log(120E06) = -88 + 81= -7 dBm which is within the safe limit of the tinySA
There is another method to verify the total noise power with the tinySA and is recommended when the noise spectrum is not reasonably flat. ? The technique requires use of the Channel Power measurement tool.? Below a span of 114 MHz. divided into 3 bands yields measurements of -12.1, -13.2 and -13.8 dBm.? Added together on? power basis this is a total of -8.2 dBm which is very close to the calculations above.?
In the plot below the span is set from 2 MHz. to 120 Mhz. and the Trace Averaging is turned on to get a smooth trace. Note that the power measured by marker 1 is the total power measured with a resolution bandwidth of 600 kHz. The noise has a slight rolloff so the Trace Normalize is activated prior to measuring the filter.?
Next we insert the filter and make the response measurement.? It is clearly not showing the maximum attenuation of the filter that was obtained with the NanoVNA.? There is not enough dynamic range and increasing the NG level has risks of damage to the tinySA.
However if one has a RF or AWG signal generator with sweep capability there is a solution that can be used when the SA does not have a tracking generator.?? For this test a Siglent 2122x is set to sweep over the frequency range of 2 MHz to 120 MHz over 120 seconds.? This is slow enough that the tinySA can make a measurement in Max Hold mode.? After a sweep the normalize function is used to adjust for amplitude vs frequency variation of the generator.?? Then the filter is inserted and the measurement made (reset of Max Hold is required.) The filter measurement is much better and similar to that obtained with the NanoVNA.
Summary - These filter measurements were made with a tinySA Ultra but also applies to the tinySA.? Filter measurement is easier with a NanoVNA? but good results can be obtained with a tinySA using a noise or signal generator in swept frequency mode.