This is a center of mass problem. Since we ignore any clinging and pulling from the water, there's nothing to change the center of mass of the boat, so if one person moves forward in the boat, the boat must move back a compensating amount.
How much does the boat move if m1 and m2 change places? The answer will turn out to be L(m2-m1)/M but let's see if we can understand why. Let's first try with some real numbers. Let's have L=10 m, M=100 kg, m1=20 kg and m2=30 kg. We'll also stick the center of the boat on an xy axis at x=0, which puts m1 at -L/2 and m2 at +L/2
The center of mass formula says CM=(-m1*L/2 + m2*L/2 + M*0)/(m1+m2+M). With our numbers, CM = (-20*5+30*5+0)/150 = 1/3 m. That makes sense, the heavier person is on the right so the CM is to the right of center. Now the masses switch place, which moves the CM to -1/3 m (try it!).
But wait, the CM can't change! That means the boat must move to the right just enough so the CM is still 1/3. If x is the distance the boat moves, then (100x + 5*20 -5*30)/150 = 1/3. If you solve that equation, you find x=1. The boat moves one meter to the right.
But we didn't want a number, we wanted a formula. We've got to go back and do it all over again keeping the letters as they were. We get that the old center of mass was L/2*(m2-m1)/(m1+m2+M). The new center of mass is (Mx + L/2*m1 - L/2*m2)/(m1+m2+M). Set those 2 equal and solve the resulting equation for x and you should find x = L(m2-m1)/M. That's the distance the boat moves. It's to the right if m2>m1 and to the left if m2<m1. (What happens if m2=m1?)
Post a follow up if you need any more help.
