Martin -
I'm no 7854 expert, but I've been poking around in a few lately, and the manual has been my bedtime reading material, so I know just enough to be dangerous. That being said, a few thoughts:
From experience, the CPU can look like it's running, but be running through garbage, a problem I'm still trying to troubleshoot on a comatose 7854.
Do the button lights change when you press them?
Does it start working again after powering down & powering back up, or does it need time to cool down?
Being able to send a debounced _RESET is a handy trouble-shooting tool. I think I just brought TP1200-1 to ground.
The mode select buttons are read to the data bus through buffer U61 whenever the _FPR signal is active. This is triggered by the CPU's DBIN line OR'd with some address decoding logic.
Ya got a hard-copy of the manual? I can't imagine trying to troubleshoot these without one (or two!). There's a few folks on here that /really/ know their way around the 7854 - hopefully you can tease out enough details to pique their interest!
cheers,
Paul