This requires integration by parts, which allows us to solve the integral of the product of two functions.
∫ x2cosx dx
This integral has two functions being multiplied together, x2 and cosx. One of these we will assign to be itself, or u, and the other will be the derivative of a function v, or dv. We need to figure out which is which in order to continue.
A neat trick I use is the acronym LIPET, standing for Logarithmic, Inverse trigonometric, Polynomial, Exponential, Trigonometric. The order of this acronym tells you which function is the u and which function is the dv. We have x2, which is a polynomial function, and cosx, which is a trigonometric. P comes before T in the acronym, so x2 is the u and cosx is the dv.
The rule for integration by parts is: ∫ u dv = uv - ∫ v du. We now know u and dv, so we still need to find du and v, by differentiating u and integrating dv. This is why we use the LIPET to assign u and dv, since it allows us to find functions that we can easily differentiate and integrate.
u = x2 v = sinx
du = 2x dx dv = cosx dx
uv - ∫ v du = (x2)(sinx) - ∫ sinx(2x dx) = x2sinx - ∫ 2xsinx dx + C
Based on your question, the f(x) you're looking for is x2sinx + C.
If you were to continue to solve this integral, you would need to use integration by parts again for ∫ 2xsinx dx. Again, 2x is a polynomial, and sinx is trigonometric, so 2x would be the u and sinx would be the dv.