I think I know why QMX transmitted without VOX. PTT was activated threw CAT (I didn't disconnect QMX from PC). And microphone was connected to QMX. So it received audio signal via air from PC and transmitted as usual SSB.
I am not sure how good PC receives signal from QMX. It can do it only threw USB because there is no dynamic in QMX. And on VARA monitor I see waterfall with corresponding signals. But I am not sure that they are strong enough taking that output signal threw USB is so weak.
I still do not know why my VOX doesn't work. Did you check VOX with the latest firmware?