Draw a figure!
At 4PM A has sailed 100 km east and is 10 km west of B's course.
B has sailed 80 km north.
The distance between them is sqrt(6500)
The equation you need is z (dz/dt)=x(dx/dt)+y(dy/dt) where z is the distance between them; x is the east west distance and y is the north direction.
Plug in the numbers and solve for dz/dt.