Is there a way to stop people from automatically using these tags together, provide some kind of warning, something?
Not currently. If using two tags together leads to ambiguous tagging or is otherwise very bad, I'd maybe write a feature-request to address this and point out why it would be useful to have.
As it is, the mac tag is pretty clear. It relates to hardware. It is just that people seem to be ignoring the wiki.
Yes. As far as I can tell, most users don't read the tag wiki excerpt anyway. It's right there:
It's good we already have the MAC address confusion in there, but I don't see a way to fit another sentence like "Use the osx tag for questions about the operating system". That would be too much clutter.
We can't cover any possible case of confusion, and even the way Apple sells their products is ambiguous with regards to how they use the term "Mac". Then there's Microsoft. They only sell "Mac" products like office-for-mac and not "Office for OS X".
That being said, I don't think it's a huge problem to have osx and mac together. I believe that anybody who is interested in Apple questions on Super User will favorite both tags, since with Apple products that are on-topic here (i.e. Macs), using OS X is almost exclusively implied.
However, if you see a post where the usage is wrong, I think the best we can do is to remove the tag, replace it with osx if necessary, and leave a custom message. If (and only if) you do this, the user will be informed of the edit. Sooner or later, they will probably learn.