Earlier I posted a response to this question. I've since realized that response had some flawed assumptions, so I deleted it, not wanting to give you false information. After another think, here's how I would approach this problem.
Let's say our solution is of the form:
y = g(x)*e^(2x) + h(x)*e^(-2x), where g and h are polynomials. We want to ask what the highest order of these polynomials are, to satisfy our ODE. Also we should consider how many constants of integration are needed to uniquely determine g or h. Here are some useful rule of derivatives of functions:
∂^(n) = (d/dx)^n (notation for nth derivative resp to x)
g_n = ∂^(n)( g ) (notation for nth derivative of g, same for h)
h_n = ∂^(n)( h )
∂^(n)(e^(2x)) = (2)^2 * e^(2x)
∂^(n)(e^(-2x)) = (-2)^2 * e^(2x)
∂^(2)(g*f) = g_2*f + 2g_1*f_1 + g*f_2
∂^(4)(g*f) = g_4*f + 4g_3*f_1 + 6g_2*f_2 + 4g_1*h_3 + g*f_4
With these properties, we can show that
g'''' + 8g''' + 16g'' = 7x^2
and
h'''' + 8h''' + 16h'' = -x
These are both 4th order ODEs, but can each be reduced to 2nd order ODEs, by integration. Therefor they are each uniquely defined with only 2 conditions. This means that for each equation we need altogether 3 constants of integration (2 to deal with the uniqueness, and 1 because of the right hand side of the ODE). Let's call the highest power of x in g, the order of g. What is the lowest order of g such that we are still left with a power of x^2 after taking the derivatives on the left side. The answer is order 4. If we had just a 3rd order polynomial, then after differentiating we could never "reach" the 7x^2 term. Thus the highest power in g is x^4. A similar argument can show that the highest power of x in h is x^3.
Keep in mind that both functions, g and h must have 3 coefficients to uniquely determine them, thus the correct answer is b.
