Another idea: why not downvote any "extra" answers to duplicate, closed questions? If the only thing certain users respond to is votes, then maybe a few downvotes are in order? Or, perhaps disallow voting on closed questions entirely?
After looking at the examples Diago cited, I am not sure there is a problem.
- This answer on a question closed as subjective by the community caused some interesting comments.
- This question which is not computer related has 1 answer. Common pattern here: Vote to close and answer question.
- This exact duplicate generated 1 high rep answer.
- This question should have been closed outright.
- This is another multiple duplicate question.
None of these closed questions resulted in any significant rep, and the answers did not seem harmful.
The one exception is the cut-and-pasted duplicate answer on #5. In this case, I think the (answer to the) closed question should be deleted, primarily because of the blatant cut-and-paste rep gain, which means any rep gained would be removed on recalc as well.