Reporting back.
I needed to raise the audio gain in the QMX band configuration menu from 54 to 60 to get about 30dB noise floor on an empty waterfall (WSJT-X reports 32dB).
Now when I record on a busy waterfall (WSJT-X reports the loudest decoded signal at +6dB) the maximum absolute value is 569.
So the audio dynamic range is still around10 bits.
This is probably about right when the audio device is 16 bits, but for a 24-bit audio device it seems low.
That said, I'm getting 25 decodes per interval on 20m right now so in reality there's not much to complain about!
Thanks, Hans!
73, Mike KK7ER