A good number of arguements for and against question downvotes being free are herefree are here, and i'd like to not re-visit these arguments.
Rather than that, i'd like to show what my voting patterns are like.
So, roughly I upvote 5 times as often as i downvote (and i sure as hell need to vote more)
However, the detailed statistics paint a slightly different picture.
86 of these 146 votes were for things that were deleted or closed.
On the other hand i have 240 moderator flags of which 228 are helpful, and 830 revisions (a good chunk of it might be retags, admittedly).
High rep users have better things to do than downvote everything!
A average session on SU involves scanning the front page for interesting things, looking at bounties, looking through the review tab (Cause the only way i can get flags is from answers that should be comments, and spammers.)
We do have things linked on the VTC chat page, but thats public and has been open for a while - and well, you're free to join in and ask why. Editing, closing, getting things merged and other janitorial things are done a lot more than downvotes. Downvotes are boring, I'd rather sort out a bad answer by posting a better answer, and bad questions through salvage or closure. In many cases we actually try our best to get the OP to adjust the question, or even get questions reviewed and reopened over meta.
I have no idea why people seem to think there's some kind of downvote cabal keeping people down. Post good answers (or good questions! I've been known to give bounties for questions i find useful), and get rep. Don't forget 1 upvote = 5 downvotes, so a solitary downvote means very little if the answer is good.