Amanda,
This is a straight plug in problem. Generally, the dot product of two vectors equals the product of their magnitudes and the cosine of the angle between them. Therefore the answer to your question is
8*5*cos 105 degrees. But cos 105 = cos (60+45)=cos60cos45-sin60sin45=(1/2)(sqrt2/2)-sqrt3/2)(sqrt2/2)
=(sqrt2-sqrt6)/4. So the final answer is 40*(sqrt2-sqrt6)/4=10*(sqrt2-sqrt6). Our value for cos 105 degrees passes a sanity check: 105 degrees is in the second quadrant, so its cosine is expected to be, and is, negative.