Be aware the tinySA is feeling the total power (summed power) at the mixer and not just the peak power at one single frequency so if the bandwidth is wide you have to reduce power further than the automatic attenuation will do.
Start by ensuring the peak of the input is below -40dBm (after the internal attenuator) and see if that makes a difference versus -55dBm or -25dBm
------------------------------------------
For more info on the tinySA go to https://tinysa.org/wiki/