Assuming that I had answered a question and didn't get any votes(up or down) for a month or some weeks, should I worry about delete it? I mean, should I assume that it is a bad answer as well if it was down voted?
I'm questioning it because I can see that there are a lot questions on SU with a high number of useless answers that didn't got any votes and are just "accumulating garbage" on SU and I would just like to create a discussion about it.
Taking your question title literal, I would say no! or else I would have to delete 200 of my answers...
No votes doesn't mean the answer is bad, it could've been helpful to a one-time visitor who doesn't have a way to vote on your answer. I would only really remove my answers when they accumulate down-votes over time, because in that case they do result in noise.