For the record, the comment was:
not a solution. just information.
As long as comments aren't abusive, there's no reason to flag them. I don't see this as abuse; the user is merely expressing their opinion.
The correct way to handle an answer that you really feel is not an answer is to do any combination of the following: downvote; comment; if you feel strongly that it's obviously not an answer then flag the answer. Just commenting isn't likely to cause anything to happen, so you'd only comment if you don't really care if anything happens.
Comments are "ephemeral"; they can be cleaned up, deleted at any time, for any reason by a mod, even if the comments are constructive and helpful (just to reduce the noise on an answer, for instance). Comments are designed to be a "temporary work area" where people can collaborate on how to improve a question/answer. They only deserve speedy and prioritized deletion if they are somehow harming the site.
Obviously the mod disagreed with the comment that your answer is not an answer, or the mod would've deleted your answer when reviewing the whole situation. I assume they didn't delete the comment because the user has the right to their opinion, and the comments on your answer aren't noisy enough to warrant speedy deletion.
In conclusion,
should I have not bothered flagging the comment at all?
Yes, you should not have bothered. But don't worry about one single flag decline, as it really doesn't make a dent.
On the other hand, getting flags declined on a regular basis (with a very small percentage of helpful flags) is bad for your ability to conveniently reach mods, because users with a large percentage of declined flags have their flags "de-prioritized" (as I understand it) somehow indicating to mods that they're likely bogus. I imagine extreme abusers of the flag system are either suspended/banned or their flags are ignored entirely.