This becomes a little easier to solve when you realize that the line from P will be perpendicular to the line AB.
You can use a dot product to find the distance of a vector from the start of the line to P, 'projected' onto the original vector.
public Vector3 NearestPointOnLine(Vector3 startPoint, Vector3 endPoint, Vector3 pnt)
{
// make a Vector3 representing the line
Vector3 line = (endPoint - startPoint);
// get the length of line to use in clamping the result to be ON line
Vector3 len = line.magnitude;
// make the length of line equal 1
line.Normalize();
// make a vector from the start of line to pnt
Vector3 v = pnt - startPoint;
// using the Dot Product gives us the length v 'projected' onto line
float d = Vector3.Dot(v, line);
// clamping this value makes sure it doesn't accidentally run past 'end'
// if you want to allow results off the end of the line, leave this off.
d = Mathf.Clamp(d, 0f, len);
// return a vector along the original line d units
return startPoint + line * d;
}