What you are being asked here is what is the shortest distance between point R and the line between A and B.
The answer is the perpendicular distance between the point and the line.
To answer the problem:
You need the slope of the line.
The slope of the perpendicular will be the negative reciprocal of this slope.
Now use the point slope form to get the equation of the line through R and perpendicular to AB.
I think you should be able to answer this problem with that information.