When you're looking for extrema on a closed interval, you must consider any critical points that lie in the interval as well as the end points of the interval. If you're looking on an open interval, then there are no endpoints, and you only need to consider critical values that lie in the interval you're interested in. In this case, once you've found critical values, any that are not between 0 and 5 you simply ignore.
To find the derivative of this function you can approach it two ways. You can either multiply all your factors together, and use the power rule on the resulting polynomial as DI did, or you can use the product rule, which extends easily to products of more than two factors.
Your basic power rule is:
d/dx(ab) = a'b + ab'
If you have more than two factors, you follow the same idea. One term per factor, and each term has the derivative of exactly one of the factors:
d/dx(abc) = a'bc + ab'c + abc'
Taking the derivative of this function using the power rule goes as follows:
V(x) = x(10-2x)(16-2x)
To make life easier, I'll factor a 2 out of both of the later terms
V(x) = 4x(5-x)(8-x)
V'(x) = (4x)'(5-x)(8-x) + (4x)(5-x)'(8-x) + (4x)(5-x)(8-x)'
V'(x) = 4(5-x)(8-x) + 4x(-1)(8-x) + 4x(5-x)(-1)
V'(x) = 4(40-13x+x2) - 4x(8-x) - 4x(5-x)
V'(x) = 160 - 52x + 4x2 - 32x + 4x2 - 20x + 4x2
V'(x) = 160 - 104x + 12x2
V'(x) = 4(3x2 - 26x + 40)
V'(x) = 4(3x-20)(x-2)
Critical values are 20/3 and 2. 20/3 > 5, which is not in your interval, so x=2 is the only critical value you're interested in. V(2) = 4(2)(3)(6) = 144 is the maximum volume of this box.