I posted an answer on ( Does Virtualbox support DMG files on non-Mac OSes? ), it was deleted, and cannot be undeleted. The reason being that it was declared as not an answer, but it is.
The OP's question said they were unable to mount dmg files with VirtualBOx on a non-OSX host. They said according to some posts they've found they believe it is a supported function, and then asked
Can someone who used DMG files with Virtualbox on Windows confirm or deny this?
So I took first hand test results with VirtualBox on Windows 8 that showed DMG is listed as supported when you browse for media, but that I too could not mount one. I also noted that our error messages were not the same, which I believe was a helpful note given that the OP has since discovered the error was caused by the dmg file they were using; they have successfully mounted a different dmg.
I'm not saying my answer is the worlds greatest. Obviously a quote from the VirtualBox manual confirming support of dmg would be better than first hand test results, but I couldn't find a quote so I provided the next best thing.
I believe the reason my answer was flagged, was that people mistook my comment about the error messages to mean that I wanted to get help solving a problem, when I was merely trying to point out that dmg mount failure can be caused by more than one thing. If I had had any information allowing me to elaborate on the errors I would have, but I didn't so I left it at that. Instead of telling me my answer is not an answer I would suggest people focus their energy on improving it by perhaps adding in what the OP, and I's errors mean.
For the record there are currently 3 answers on the page. Mine (which actually addresses what the OP asked "Is DMG supported"), the OP's (Saying the error was due to a bad dmg file), and another user (Suggesting a work around would be to convert the dmg to another format).
My answer should be undeleted, because while it does not addrress the later developed issue of what the error message means, it DOES answer the original question of DMG support on non-OSX hosts.