in this case gate capacitance is not the issue as 2xBS170 parallel and one FDT86256 has almost equal capacitance.?
I'm not sure you're comparing like with like. The capacitances of the FDT86256 are quoted at vDS = 75 V, whereas those in the BS170 datasheet are quoted at vDS = 10 V. Everything in these devices is a curve and when it comes to capacitances, they are higher at lower voltages.?
?
The problem is gate driving voltage is low.
Anyway - yes, I think this is probably the main issue, yes... typical threshold voltage for a BS170 is 2.1 V but for a FDT86256 it is 3.5 V. Which means 5V isn't enough to allow sufficient drain current to get full power output. The capacitances make this all worse at higher frequencies.
A great misunderstanding about these inexpensive MOSFETs like BS170 or IRF510 is that they are switches. And that somehow by some magic, an expensive "real RF amplifier MOSFET" like a Mitsubishi?RD16HHF1 is somehow a more linear type of amplifier. Mostly marketing spin. Look at the transfer characteristics for each, and you'll see what I mean.?
The MOSFET is only like a "switch" if it is driven by a sufficiently large and square drive signal. The "threshold" voltage is not the voltage at which a MOSFET switches on. The drive signal gets less and less square as the frequency goes up due to the gate capacitance so everything gets worse, this is why high frequency bands tend to produce lower output. A higher level squarewave drive helps to counteract this, hence the improvement with 6V supply to the driver.