Nico,
A couple of points to consider.
1. I can't tell from your PCB photo if you had tracks on both sides of the board in order to implemented a transmission line from the matching network to the helical antenna. This is required because a single track will make for an additional radiating element. Attached is the manufacturers layout for your antenna.
2. The edelay on most of the NanoVNA products is the round trip delay. You used 760 ps which is only one way. On my NanoVNA I have to enter 1.52 ns. To verify your edelay is correct short the end of 6 inches of RG174 and see if you are on the far left of the horizontal line on the Smith Chart. Adjust your edelay until you are.
3. The edelay method will be OK for Return Loss and SWR measurements. Your impedance measurement (R & X) results will be MUCH better if you calibrate right at the end of the cable with an open, short and 50 ohm SMD chip. Then solder to your board. The edelay at 900 MHz. will not give you good estimates of R and X because of slight loss in the cable and impedance bump at the SMA connection. Try this yourself by measuring a 100 ohm SMD resistor using both methods.
4. You need to use ferrites on the cable which are appropriate for the frequency of operation. I suggest a number of Fair Rite Mix 61 as shown in the attachment. A binocular core with holes big enough for a few turns would help reduce common mode current.
Roger Need
