pi < theta < 3pi/2 refers to quadrant III (x and y are both negative).
sin theta = opposite/hypotenuse, so if hypotenuse is 1 opposite= -0.866.
cos theta = adjacent/hypotenuse
adjacent= square root of (hypotenuse^2 - opposite^2) = square root of (1- 0.75)= square root of 0.25= 0.5
So to recap, cos theta = adjacent/hypotenuse, and we just found that adjacent= 0.25. Hypotenuse=1.
Because we are in quadrant III you have to account for that when determining if your answer is positive or negative.