Two possibilities come up in to me:
1 - Test the encoder using a scope while the unit is on.? That would be connect a probe to each of the two encoder (not switch) pins and rotate at a constant speed to verify that the pulse is there (note, it will be a zero pulse, in that the encoder is switching to ground). If you have a dual trace scope you can verify that the pulses are 90 degrees out of phase. Otherwise test each individually to verify operation.
2 - replace the Nano if you have a spare.
I have read of similar issues with others.? In at least one case the solution was to install pull up resistors as is used for the CW key input.
If it were my rig, and since I have spare Nanos, I would go that route first.? Some Nanos are better at sensing the encoder than others.
Just suggestions.? There could be more at play here.
73
Evan
AC9TU
