There is a guy who lost it completely. He has 1 rep (seems it doesn't go any lower than 1).
His last 'answers' on superuser are extremely offensive and do not add any value to the site. Why is he allowed to downvote still?
He doesn't have enough reputation to downvote either questions or answers, so any downvotes are coming from another user.
Please see here in the Help Center which explains that a user requires 125 reputation on the site to cast downvotes on either questions or answers.
His comments will almost certainly result in him being banned (if that hasn't already been taken care of), but any downvotes you receive are not from this specific user. Going forward, you should not assume that just because you receive offensive comments from a user that they are also downvoting you (or downvoting the question/answer they're commenting on). There is no way for you to know who exactly downvoted any particular question or answer. The only way to rule out whether it was a specific user is to check their rep: if they have less than 125 rep, it wasn't them, for certain.