This is why I suggested the use of a choke at the feedpoint; you can use a few turns of the transmission line or a so-called "bazooka" or "sleeve" balun:
This will force the currents on both sides of the dipole to be identical and eliminate antenna currents from flowing along the feedline. You are changing these currents when you move your hand along the feedline, which is changing your impedance readings.
73, Don N2VGU