I don't know the steps to this problem.
6x2a + 5xy - 4a2y
This is a good problem. I would suggest you go about it by finding all prime factors of each monomial.
6x2a ----> 3*2*x*x*a
5xy ------> 5*x*y
-4a2y ---> -2*2*a*a*y
Now note that there is no one prime factor that is common in all three monomials listed above. Instead, you should look at any two monomials and see what is common in them. I will choose the two monomials that have the most commonalities.
6x2a - 4a2y + 5xy
2a(3x2 - 2ay) + 5xy
You can leave your answer as such. If you want to continue further, you can complete the square for the expression inside the parenthesis by first dividing out 3 and then commencing.