Hi all again,
I did another thing instead. I replaced the LO with another signal generator. One with lower harmonics, 42dBc compared to 24dBc.
The result is very similar to before, so now I've verified what Erik said. There is no need to filter the LO.
After I took the screenshots I realized that the LO power in this case was 7dBm but with the module around 4.8dBm which might explain some differences.
All in all, I've learned something today.
Cheers
Marcus
