First, I think that you are shorting one side of your balanced input to ground. Starting at "Transformer center taps connected to case", then through the case to the BNC shell, then to the ground on the VNA, then to the shell of CH0, then to one of the balanced input lines. So I don't think that any of your measurements mean anything.
If you want to measure the balanced input impedance, you could connect a floating 50 ohm load at the output BNC. You should expect to read 112.5 ohms, this is the impedance because the 6 to 4 turn ratio results in the turns ratio (1.5) squared (2.25) times 50 ohms.
To measure the through loss, you could make another 4 to 6 turn transformer and connect it between the CH0 and the balanced input. Any loss you measure needs to be divided between this balancing transformer and the device under test. You could connect two identical 4 to 6 turn transformers back to back, measure the loss and divide by two, then just subtract this test fixture loss from the overall loss you measure.
You might also want to evaluate the common mode impedance by connecting the CH0 center connector to the shorted together balanced input lines and connect the BNC to CH1, to provide both a ground return and a load on the output.
