You're right on the homogeneous part. e^x is already included in the homogenous part, so let's try your second guess yp = Axe^x. To find A, we plug yp into the differential equation. Since yp' = A(xe^x+e^x),
yp'' = A(xe^x+2e^x). Plugging into yp''-5yp' +4yp = 3e^x, we get
A(xe^x+2e^x)-5A(xe^x+e^x)+4Axe^x = 3e^x,
and grouping terms we get
(A-5A+4A)xe^x+(2A-5A)e^x = 3e^x
-3Ae^x = 3e^x.
so A must be -1.
Thus the general solution is yp plus the homogenous solution.