It's the ratio that counts. It doesn't matter if r = 10√3 because x will be 10 and the ratio is 10√3/10 = √3. So use the simplest values of r and x, r = √3 and x = 1 and y = √(r2-x2) = √(3-1) = ±√2 (which is an irrational number, so it cannot be represented as a fraction).
If tan θ < 0, then θ is in quadrant II or IV. If sec θ > 0, then we're down to quadrant IV so y = -√2.