I failed to mention but I have tried it with Aluminium Nitride, Silpads (Grey and Blue) and Mica insulators, there was no difference at all. I should get a contact sensor to measure the rise of the heatsink temperature to know how these two are related.
I will further document this over the next few days.