Down voting on an answer costs 1 rep irrelevant of whether you leave a comment or not. You are under no obligation to leave a comment. The losing of rep when down voting is to prevent abuse of the voting system.