Shortly before leaving Tektronix, I wrote a good app note on making floating measurements, related to SMPS. That was 20 years ago, I don't think Tek still has it on the web site. I will try to find the original file and post it. We made several real measurements in the lab for research in the paper. Floating instruments, isolators, and differential measurement techniques were compared. Without rehashing the entire note, a summary is that for most applications, differential measurements give the best results.
Subtracting channels is a differential mode of operating the scope. Compared with a true differential amplifier or probe, channel A minus channel B has much poorer CMRR at higher frequencies. The other major limitation is common mode range. The common mode is the voltage you are floating on. IT is limited to the full scale of the scope at the V/DIV setting you are using. This is a real limitation in SMPS work when you need to measure small voltages riding on top o flarge voltages. The common example in higher power inverters is measuring the upper gate drive in a push-pull drive stage. The upper gate only has a few volt (15 maximum) swing riding on a 350V swing. To keep the common mode (350 V swing) within the common mode range, you need to set both channels to 50 V/div. IT is difficult to see the gate drive with a 5 to 15 V swing with the scope set to 50 V/div.
Steve