Oh no, the sine and cosine functions MUST be between -1 and 1,
so the statement of your problem is incorrect...
However, judging by the values given, the Pythagorean triple
of {5,12,13} will work just fine, so you can see how the problem
is solved.
So here is the RE-STATEMENT of the problem:
Given: cos A = 5/13 with A in quadrant 1 ;
sin B = -12/13 with B in quadrant 4;
The goal is to calculate sin(A-B)
The given information is consistent with the rule
of signs for trig functions on the unit circle.
Remember "Add Sugar To Coffee"..
A S T C
So ALL trig functions are positive in quadrant 1;
SINE is positive in quadrant 2;
Tangent is positive in quadrant 3;
Cosine is positive in quadrant 4; <--- given sine is negative, so we're good
Regarding angle A:
cosine is 5/13 --> adjacent = 5 and hypotenuse = 13;
then opposite is 12
So then sin(A) = 12/13, cos(A) = 5/13, and tan(A) = 12/5
Regarding angle B:
sine is -12/13 --> opposite is -12 and hypotenuse =13;
then adjacent is -5
So then sin(B) = -12/13, cos(B) = -5/13, tan(B) = 12/5
By angle subtraction formula, sin(A-B) = sinA cos B + sin B cos A
= (12/13)(-5/13) + (-12/13)(5/13)
= -60/169 + -60/169
= -120/169