The bounty process is set up in the way it is and it will probably not be changed any time soon.
If there is no new answer to the question you put a bounty on, then you will get the bounty refunded. If someone posts an answer and it receives answers, then the author of that answer will receive half the bounty or the full bounty if you manually give it to them for their answer.