It depends on the situation. Unless there is a very good reason, it is better to decline a draw offer. A good reason would be if you are in a lost position (very unlikely that your opponent would offer you a draw in such a case, has happened to me only once in my long tournament career).
Personally, I have lost only one game after declining a draw, won tens of games, and in a few cases the draw was inevitable (for example a player repeats position a second time, offers a draw, and even if you decline, the draw would be automatic after the third repetition.
If the situation is unclear with equal chances, it is better to decline. Better to learn a lesson after a tough fight that did not go your way, than get the easy half point of a draw.
Never accept a draw out of fear that you might blunder during the remaining of the game. And if you lose a game where you declined a draw, at least you can be proud that you tried your best.