A good first step for any vector problem is to draw a picture, drawing the vectors to scale and at the proper angles. This helps with visualizing the problem/solution. Draw the first vector starting at the origin; the second vector will be drawn starting at the head of the first vector. Since the second vector is subtracted, also draw (use a dashed or different colored line) in a direction 180 degrees from the original direction.
At this point you will also define your positive and negative directions in the horizontal and vertical. It is traditional to define East as positive horizontal and North as positive vertical, so this is what I will do.
Second step is to break each vector into its horizontal and vertical components. I am not 100% sure what your notation (N37W) means - does it mean the angle is 37 degrees north of west or 37 degrees west of north? Which is correct affects the way the trig functions are applied.
Vector 1: 1537 km (N37W)
Angle definition horizontal component vertical component
37 degrees north of west 1537 * cos 37 = 1227.5km (w) 1537 * sin 37 = 958.5 (N)
37 degrees west of north 1537 * sin 37 = 958.5 (w) 1537 * cos 37 = 1227.5km (N)
Vector 2: 853 km (W) - since this vector is only to the west, the vertical component is 0
Third step: Perform the specified operations on the horizontal and vertical components SEPARATELY. At this point remember to apply the correct sign based on the coordinate system defined in step 1. In this example, I will be using the assumption that the specified angle is 37 degrees north of west. If this is not correct, you can repeat the step using the other components.
Horizontal component: X = -1228 - (-853) = - 375 km. Since the result came out negative and west is negative, the horizontal component is: 375 km (W)
Vertical component: Y = 959 - 0 = 959 km (N)
step 4: Find the magnitude and angle of the resultant vector using the Pythagorean theorem and inverse tangent function:
R = √(X2 + Y2) = √[(375)2 + (959)2] = 1030 km
Direction: NOTE: I am finding the angle wrt the West direction, so use the absolute value of the horizontal component.
Θ = tan-1(Y/X) = tan-1(959/375) = 69 degrees north of west.
If the original angle was wrt the north direction, then there are two options from this point - solve the equation as written and then subtract from 90 degrees to get the angle wrt the north direction OR use Θ = tan-1(X/Y)