The VNA and you, when hold on hand, are part of antenna. You must change the installation, feeder route, ground if possible properly.
Any trick to allow a reliable measure if removed later for normal operation then the SWR will become unstable again.
Usually a line choke isolates the antenna feeder side from outside world (shack...)