It would help us if you could install the a new FW. V1.1-77 and run an additional test
Connect high and low with SMA cable
Set CAL OUTPUT to 30MHz
Start MEASURE/MORE/LINEAR
A green line should appear and the attenuation will start at 30dB and step by step decrease till zero
When completed (will take some time) with a good attenuator the screen should look like this
The line does not have to be perfectly flat.

Please post a picture of the screen with your tinySA
--
------------------------------------------
For more info on the tinySA go to https://tinysa.org/wiki/