This has been suggested before and status-declined. The Team's position is that the main focus of the site is questions and answers, so they usually deny comment-related feature requests.
I believe there could be a usage scenario for this, but don't personally feel that we're missing out on much by not having downvotes on comments. Also, downvotes on comments might lead to even more bickering and arguing, which we try to avoid. The current comment system seems to work for what it was designed to do.