The root cause for the mismatch in ip3 turned out to be a wrong distribution of IF gain over the total chain. This reduced the 1dB compression point so a input of -10dBm resulted in -13dBM.
After correcrion the IP3 measured at the spec level of 10dBm.