Plot vector a on the +X axis.
Next, recognize that the resultant must lie on the y-axis ("the resultant is at right angles to vector a").
There are 2 possible solutions: resultant in +y or -y
For either one, there will be a right triangle where the hypotenuse is vector b. Call the resultant "r". Using the Pythagorean Theorem, we have b^2 = a^2 + r^2
Enter the values:
10^2 = 6^2 + r^2
solving and rearranging, we have:
r^2 = 100 - 36 = 64
So: r = 8
For the angle, recognize that the vector b must be in the 2nd or 3rd quadrant. Therefor, its angle is 90 deg + the small angle in the triangle. For a 3:4:5 right triangle, the small angle is arctan (O / A) = arctan 0.75 = 36.9 deg.
Thus, the angle between vectors a and b is 126.9 deg (either + or -)
To see this clearly, DRAW A PICTURE