There are various reasons why this question would be off topic.
It's about software (codec) development.
And even if it were on topic: It's unclear what's being asked. It might be clear what they want—like asking "How do I drive a car?". You should interpret this option as "this question is too vague and unspecific". It wouldn't be possible to give a reasonably scoped answer without much guesswork.
It's too broad, as entire books could be written on codec design.
By the way, the old option read "not a real question". So while in fact there could have been a question, it should be phrased in such a way that it is useful for Stack Exchange sites.
You should not flag anything as very low quality unless the post cannot be salvaged through editing and needs to be deleted right away. In almost any case, it's preferred to flag the question using a close option if it applies, or to fix it otherwise. If it can't be closed but can also not be edited into better shape, then flag as very low quality, but use this flag sparingly please.