In fact that's (almost) how it works.

If a question has four close votes and a moderator decides to leave it open through the queue, it will not show up on the queue anymore.

Note that this doesn't prevent users from voting on those questions when they see them on the regular site or through the 10k Tools.