I just wanted to point out a pattern/bias I've been seeing regarding reviews that may be harming, if not defeating the purpose of the audit system.
Many audits exist that require a downvote/flag for a question whose score is only -1 and not deleted.
Context: One person found the question low quality enough to vote it down (yet not low quality enough to delete or close) - this is enough to hold it up as an example to all of crap content?
It seems (I can't provide numbers, only my own experience) that negative audits (reviewer must downvote or flag to pass) are much more prevalent than positive audits (reviewer must upvote to pass).
Context: Introduces an unconscious bias: downvote first.
Positive audits can be distinguished simply by their length
Context: Introduces an unconscious bias: If a review Q/A is very long, it should probably be upvoted first.
Audit questions frequently appear that have a blank user profile (in other words, only a username, no score or even the default anonymous user image)
Context: Audit questions are trivially distinguishable from nonaudit questions, meaning the reviewer does not even need to read the question to know the action they must take.
All of these things taken together leads to a very simple way to game the review audits without even reading the question, which looks something like this in pseudocode:
if (contentLength == short) {
voteDown;
if (!auditPassOrFail) {
retractVote;
processNormally;
}
} else if (contentLength == long) {
voteUp;
}
If the whole idea of review audits is to ensure reviewers are paying attention, then these issues compromise the integrity and point of the system. I would recommend the following changes in response:
Insert more positive audits
- A significant number of these should be short-form content.
- Length does not necessarily equal quality.
- Failing to recognize good content is arguably a greater evil than failing to recognize bad content
Conversely, insert more negative audits that are long-form
Ensure that audits look like normal questions
- Realistic vote counts
- User avatars should be present, even if they're the automatically generated ones.
Revisit whether a single downvote is consensus that a question deserves "this is an example of crap" status
Allowing vote retraction directly in reviews is probably a bad idea
- This question is short? Click the down arrow first. No audit message? Undo the vote and look at the content.
Just clicking the vote arrow confirms the action in an audit - it should probably require one to click the "done" button.
- Vote the wrong way, and you flunk the audit immediately. Given how malleable votes usually are (you can unclick the button for quite some time afterwards), this is quite counter-intuitive.
This balances out the biases, and helps the stated goal of the audit system: ensuring that reviewers are paying attention.
Any thoughts?
