Should reviews limit be increased?

Question

The number of posts to be reviewed does not stop growing. Where once it was possible to clear these posts to zero, today that number has passed 700, and it was lately standing even at more than 800.

However, when I try to do my part and vote on the posts in question, I am quickly blocked by the limit of 20 reviews per 12 hours (even though the voting message says "24 votes"). It also doesn't help that I am periodically presented with "test" questions.

I do not understand where these limits of 20-per-12 came from, but I believe that they are at the root of the problem of increasing number of untreated posts in the Close queue.

My suggestion is to increase the number of reviews above 20, for example to 50 as I believe is the case now on Stack Overflow.

I would also suggest a further increase in that limit for users that have achieved the privilege level of trusted user, perhaps even making it unlimited, and also to not post test questions to such users.

The other limit of 12 hours is in my opinion less useful to change, as for many users there are normally only certain hours of the day when it is possible to contribute here. But perhaps making it 6 or even 3 hours would do just as well.

[FOLLOW UP]

It has been answered by @Shog9 that the number of reviews is limited to 20 in order to democratically divide reviews among as many reviewers as possible.

I have been tracking reviews for the last week, and I can state that this is not what's happening. There is a small number of active reviewers that are engaged in reducing the Close queue. When these dedicated users are less active, the queue grows. When most of them are active at the same time, the queue shrinks.

My conclusion is still the same - the limit of 20 reviews blocks these community-minded users from finishing the entire queue. As @Shog9 has said, some posts in the queue have been there for more than a year.

Answer 1

There are enough people who could be reviewing these to drop the queue to nothing very quickly, if they were inclined to do so. It's important to encourage as many reviewers as possible; when a small handful of users are effectively acting as gatekeepers for every question that arrives on the site, it becomes very difficult for the majority of active users to have an influence on what is being asked and answered here - it's better to have 50 people here doing 20 reviews each day, not 20 people doing 50 reviews.

And that's not even getting into how easy it is to become careless when you're reviewing dozens of posts in one sitting.

In fact, it's not altogether a bad thing if the queue never reaches 0. There's a certain urgency that arises when a queue hovers close to empty, and that's not always a good thing! Getting suggested edits or new-user posts reviewed quickly is helpful; these tend to be quite time-sensitive. Closing can be, but while a suggested edit may become worthless in a matter of minutes, many poor questions can still be closed a day or two later with no ill effects. Given it takes 4 days before close votes even start to age away, a question that doesn't need to be closed doesn't really suffer by staying in the queue for that long either.

The big problem here is that some of those pending review tasks are a year old!

See, the queue prioritizes new stuff first - tasks that are newly-created stand a better chance of being shown to reviewers than old tasks do. Tasks that've already been reviewed also get prioritized - so y'all are actually doing a pretty good job of keeping up with tasks that showed up in the past few weeks (pretty much since we added that indicator at the top for 10K users).

So what can you do about those old questions lurking in the back of the queue? Well, a good chunk of them probably don't matter - that's why they're lurking, because no one is looking at them. But some of them could use a bit of attention... And this is where a recently-introduced change to the closing system can help you out: if you have a gold badge in a given tag, filter the close queue down to just duplicate reviews in just that tag, and you'll be able to single-handedly clear out up to 20 every day:

• Possible duplicates in the windows tag
• Possible duplicates in the linux tag
• Possible duplicates in the osx tag

Incidentally, I love filtering by tag - it's the only thing that makes reviewing close votes bearable. So I went and turned on another new feature:

That's m0sa's new filtering UI - you get quick links from /review directly to the top tags and close reasons. I like it a lot, and hope y'all will as well.

There's one other problem: y'all get 20 reviews, and only 24 votes - so if you review 20 posts and all 20 should be closed, that only leaves 4 votes for new questions that you encounter on the site. That paltry surplus leaves some of you in the position of having to ration your votes - not a good way to encourage folks to review (unless you wait until just prior to the new day). So I'm raising the number of close votes per person here to 50 per day. That's enough to review judiciously, close new questions if necessary, and still have plenty of votes in reserve in case something terrible happens.

The number of reviews you can do each day remains at 20 - it'll automatically increase to 40 if a queue ever goes over 1000.

Answer 2

I think this is a great idea but with a caveat. New reviewers have a less limited ability as opposed to tried and tested reviewers. This can be tested based on the badges relative the reviewing queue that's used.

• < Bronze - 20 reviews
• Bronze - 25 reviews
• Silver - 35 reviews
• Gold - 50 reviews

This would help to give more power to those that actively use the system for good, but discourage the misuse of the power for newer reviewers.

Answer 3

I do not understand where these limits of 20-per-12 came from, but I believe that they are at the root of the problem of increasing number of untreated posts in the Close queue.

Perhaps it's to encourage a wide number of people to participate with a reasonable load rather than a few working a large load on it. I can see a case that, for a democratized, self-maintaining site, the more diversity of people that can look at these, the fairer and better for the site.

Of course, maybe the limits in and of themselves aren't having that effect.

I would also suggest a further increase in that limit for users that have achieved the privilege level of trusted user, perhaps even making it unlimited, and also to not post test questions to such users.

I think the test questions are necessary to prevent the temptation of not reading them carefully. By having that in place, it's an assurance that the review process has integrity and can be trusted, being friendlier and fairer to new users. Maybe trusted users should have less test questions but I disagree with eliminating them entirely.

I agree that trusted users should have a higher limit. I think a limit should exist, though.

Answer 4

This suggestion is a non-starter. Let's say the review limit is raised and reviewers take advantage of it. In a short period of time, the already not overwhelming stack of close votes will go to 0. Then what? When is the higher limit going to come in play again?

So, really, what's the point? Is the current stack of ~600 questions awaiting close votes such a big problem that it requires a policy change? Considering the stack has been roughly the same size for months (and actually shrinking over the past few weeks, not increasing as you claim), the system is pretty close to an equilibrium point. To me, that's a sign of the current policy working as it should.

I suggest you resolve to meeting your review limit each day if the number of question awaiting review is a hindrance, and maybe encourage others to do the same.

Answer 5

@Shog9, wrote, 11 days ago:

The number of reviews you can do each day remains at 20 - it'll automatically increase to 40 if a queue ever goes over 1000.

That's the type of system, I think is, intelligent: one that adapts limits based on real conditions. It automatically allows more input when not enough people are processing a queue, but refines the limits when many people are contributing to ensure that decision making is decentralized & spread out.

With enough heuristics in place, the system moves more towards sharing characteristics in common with AI systems. AI systems are the way to go to handle larger volumes consistently and objectively. As they improve they can even handle more "edge cases".