Siggi,
In the given code snippet, the X register is 16 bit and the full 16 bit write is performed. The DAC port has two bytes: 0x87F for the MSByte and 0x880 for the LSByte. The X register is loaded with the 0x00AC absolute value.
As we look at the bits of the MSByte (0x87F), the bit 7 is the interrupt interval counter/timer reset control. The bits 4-6 are the channel code selects for the demultiplexer.
BTW, as you can see in some other examples, the DAC is accessed in parts. But I've seen a few places (could only find it by LA) where the DAC is addressed indirectly and the Tek programmers even use the stack pointer as the register to write the data into the DAC!!! The code is scarce indeed and was written different programmers.
And you are exactly to the point, IMHO, that if I follow the DAC calibration procedure, the DAC is railing indeed to the right side or even just a bit smaller. Can put the delta cursor over the 11th vertical division or a bit further right! Therefore I also assumed that there is something weird with the DAC and something is dragging the line.
I'll measure all the resistors and levels with 6.5 digit multimeters and re-calculate the values and voltages. Logically if this is not the resistors then either the caps or the DEMUXers and/or OPAMPS.
Even a small leakage in the de-muxer will create the voltage offset.
So the goal is check the DAC reference circuit in the first step, since now the digital part of the circuit behavior is known.