12

I just wanted to point out a pattern/bias I've been seeing regarding reviews that may be harming, if not defeating the purpose of the audit system.


Many audits exist that require a downvote/flag for a question whose score is only -1 and not deleted.

Context: One person found the question low quality enough to vote it down (yet not low quality enough to delete or close) - this is enough to hold it up as an example to all of crap content?


It seems (I can't provide numbers, only my own experience) that negative audits (reviewer must downvote or flag to pass) are much more prevalent than positive audits (reviewer must upvote to pass).

Context: Introduces an unconscious bias: downvote first.


Positive audits can be distinguished simply by their length

Context: Introduces an unconscious bias: If a review Q/A is very long, it should probably be upvoted first.


Audit questions frequently appear that have a blank user profile (in other words, only a username, no score or even the default anonymous user image)

Context: Audit questions are trivially distinguishable from nonaudit questions, meaning the reviewer does not even need to read the question to know the action they must take.


All of these things taken together leads to a very simple way to game the review audits without even reading the question, which looks something like this in pseudocode:

if (contentLength == short) {
  voteDown;
  if (!auditPassOrFail) {
    retractVote;
    processNormally;
  }
} else if (contentLength == long) {
  voteUp;
}

If the whole idea of review audits is to ensure reviewers are paying attention, then these issues compromise the integrity and point of the system. I would recommend the following changes in response:

  • Insert more positive audits

    • A significant number of these should be short-form content.
    • Length does not necessarily equal quality.
    • Failing to recognize good content is arguably a greater evil than failing to recognize bad content
  • Conversely, insert more negative audits that are long-form

  • Ensure that audits look like normal questions

    • Realistic vote counts
    • User avatars should be present, even if they're the automatically generated ones.
  • Revisit whether a single downvote is consensus that a question deserves "this is an example of crap" status

  • Allowing vote retraction directly in reviews is probably a bad idea

    • This question is short? Click the down arrow first. No audit message? Undo the vote and look at the content.
  • Just clicking the vote arrow confirms the action in an audit - it should probably require one to click the "done" button.

    • Vote the wrong way, and you flunk the audit immediately. Given how malleable votes usually are (you can unclick the button for quite some time afterwards), this is quite counter-intuitive.

This balances out the biases, and helps the stated goal of the audit system: ensuring that reviewers are paying attention.

Any thoughts?

  • 2
    The problem is this. The majority of stuff in the review queue should be downvoted, because it being in the review queue, suggests that somebody flagged it as a quality problem. Good answers and questions don't get reviewed. Stuff added by people with less then 100 reputation is another story, there is a reason that content, gets its own queue. You are right, the audit system has its quirks, but I don't agree it creates a bias. – Ramhound Mar 22 '16 at 16:07
  • I'm not sure what you're saying about the solitary downvote.  ISTM that most of the "bad" (you-should-downvote-this) audits are posts that, in real life, have been deleted.  Displaying it to the reviewer as a deleted post would obviously defeat the purpose; displaying it with a score of -5 would be just as bad.  They're displayed with low scores so you have to think about them (or game the system), rather than just do a group-think pile-on. – Scott Mar 26 '16 at 7:07
8

Audits are never really subtle. The scary thing is, some people still fail them! This doesn't even need to be malicious; when performing repetitive work, it's all too easy to succumb to fatigue and let your muscle memory carry you forward while your mind goes to sleep. This is somewhat less dangerous when reviewing than when, say, operating a motor vehicle... But it's still problematic for the site.

So we have these (fairly obvious) audits. If you notice them, great! Pass and move on. If ever you don't, it's time to take a break and do something more stimulating for a while.

  • 1
    I never really considered that this was intentional, but it makes sense when you put it that way. I think I've got 4 failures on record, and every single one of them was an example of either "Eh.. I'm not sure, I'll say no action" or not paying enough attention and clicking the wrong vote button. I still think it would be nice if you had to hit "Done" before the audit is passed or failed though. – Mikey T.K. Mar 22 '16 at 16:13
  • 4
    If you are not sure, hit skip, instead of no action. This way the correct, since you don't know what the correct action is, somebody else who does can handle it. – Ramhound Mar 22 '16 at 16:35
  • 2
    I'm not sure if it's just my connection or something, but the audits for me seem to take a few seconds to load where as a legit review takes a second tops. I immediately realise I'm being tested by that delay. – Jonno Mar 22 '16 at 16:48
  • @Jonno it's not just you – Sathyajith Bhat Mar 22 '16 at 18:39
  • 2
    @Jonno That's definitely a phenomenon in the Suggested Edits queue. I haven't noticed it in the others, but it might be there. – Ben N Mar 23 '16 at 1:23
  • I think the slow thing happens with all queu's. If it is damn slow I will click reject, enter as comment 'reason: review question' and then read the question before confirming. – Hennes Mar 25 '16 at 13:13
  • 2
    @Hennes How awful. If Superuser changes behavior, and slows on a good edit, you could reject a quality edit. Even worse: your description of "review question" may look like a command, as if review is a verb and not an adjective. However, your job was to review the question. It seems you're rejecting an improvement with a comment that whoever reads your description should perform the work that you volunteered to take on. If you paste in a standard reply of "reason: this is a review question", that would at least be more clear about your reasoning. – TOOGAM Mar 26 '16 at 16:07
  • 1
    The ` and then read the question before confirming` part is essential. But the first part is almost reflex memory. Slow? Must be a review. Type in reject and then read to make sure it actually is a review. – Hennes Mar 26 '16 at 16:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .