The 'no trip' setting is a bit of a lottery - all the test designers can do is introduce a very low L-E current or a very short duration higher current, and hope it is not enough to operate the mechanism - which will depend on how much standing leakage there already is in the circuit (i.e how long and how much capacitnce to ground becuase of that - allow 100pF/m if you are not sure) and how ' frisky' the trip is -and the modern electronc ones, where there is something closer to a solenoid firing circuit, have less inertia than the fully mechanical models.

However, to measure Zs accurately you really need to draw enough fault current that the 230V drops by a measureable amount, as it is the change in voltage divided by the change in fault current, that is the estimate of Zs. There is a problem then with low Zs systems, that the natural uncertainty in the 230V or whatever, especially on supplies shared with loads that are periodically switching, comes to dominate over the quantity to be tested. The combination of readings of L-N and L-E voltages allows a better discrimination against that sort of background 'noise', and more importantly in this case, a lower test current gives the same 'signal to noise' ratio.
In summary, although I cannot see exactly what you are doing, I suspect nothing is actually wrong, other than an unfortunate combination of 'frisky' RCBOs and a meter that was really designed to test something clunkier.
Although you have managed this time anyway, in future if you are stuck and it is a killing matter to have an accurate Zs reading for the record you can measure Zs on the supply side of the RCD, and add the result of an r1/r2 type test,
or verify earthing with a wander lead test of r2 (- note that r1 can be verified by L-N loop) it is more work however.

Equally, downstream of an RCD, to know the exact value of Zs is less serious, as unless it is near open circuit the trip will operate.

-------------------------
regards Mike