26

The number of posts to be reviewed does not stop growing. Where once it was possible to clear these posts to zero, today that number has passed 700, and it was lately standing even at more than 800.

However, when I try to do my part and vote on the posts in question, I am quickly blocked by the limit of 20 reviews per 12 hours (even though the voting message says "24 votes"). It also doesn't help that I am periodically presented with "test" questions.

I do not understand where these limits of 20-per-12 came from, but I believe that they are at the root of the problem of increasing number of untreated posts in the Close queue.

My suggestion is to increase the number of reviews above 20, for example to 50 as I believe is the case now on Stack Overflow.

I would also suggest a further increase in that limit for users that have achieved the privilege level of trusted user, perhaps even making it unlimited, and also to not post test questions to such users.

The other limit of 12 hours is in my opinion less useful to change, as for many users there are normally only certain hours of the day when it is possible to contribute here. But perhaps making it 6 or even 3 hours would do just as well.

[FOLLOW UP]

It has been answered by @Shog9 that the number of reviews is limited to 20 in order to democratically divide reviews among as many reviewers as possible.

I have been tracking reviews for the last week, and I can state that this is not what's happening. There is a small number of active reviewers that are engaged in reducing the Close queue. When these dedicated users are less active, the queue grows. When most of them are active at the same time, the queue shrinks.

My conclusion is still the same - the limit of 20 reviews blocks these community-minded users from finishing the entire queue. As @Shog9 has said, some posts in the queue have been there for more than a year.

13
  • Quick question: I do a lot of close voting almost every day, and I rarely, if ever, see you listed among the recent reviewers. Are you privy to moderation tools because of high rep, and that's why you're never listed among recent reviewers or what?
    – Excellll
    May 15, 2014 at 14:10
  • @Excellll: One gets access to moderator tools from 10,000 rep. I have only started seriously doing reviews lately because of their large number but get blocked every day by that limit of 20, which is frustrating.
    – harrymc
    May 15, 2014 at 15:17
  • I agree with the premise, however (unfortunately), considering the state of the close queue at SO, I can't imagine this hasn't been discussed already. May 15, 2014 at 16:54
  • @Raystafarian: I have searched but haven't found such posts. Even if it was already discussed, the situation now is quite different : SU has succeeded too much ...
    – harrymc
    May 15, 2014 at 17:11
  • Well, you can filter your close vote queue by your gold badge tags and close duplicates with one vote, for now. Otherwise, I agree. May 15, 2014 at 17:22
  • @Raystafarian: I have tried that and it doesn't seem to work.
    – harrymc
    May 15, 2014 at 18:20
  • 5
    I think the amount of close votes you can issue should be based on not only your reputation but perhaps the quality of those votes you issued. For example if you use a close vote and the question is closed you get that vote back right away otherwise you wait the 6-12 hours you normally would wait. This prevents somebody from trolling the community who happens to have 1,000 reputation.
    – Ramhound
    May 15, 2014 at 18:21
  • @Ramhound: (One needs 3,000 rep for that.) What I'm trying here is only to discuss getting some limits changed. Please post ideas for new mechanisms as separate posts.
    – harrymc
    May 15, 2014 at 19:26
  • 4
    I just finished reviewing a very complex question. After long analysis I decided to close it, then was told this was only a test and I passed! Stop wasting my time, please!
    – harrymc
    May 16, 2014 at 5:24
  • 1
    It also doesn't help that I am periodically presented with childish "test" questions and waste my time on analyzing these dummies. --- If only I could give more than 1 * +1... This question is so patronizing it is totally off putting.
    – Dave
    May 16, 2014 at 12:43
  • Also, are there any stats on how many questions which are typically voted on ARE closed (or are not closed)? I mean is it worth considering increasting the points you need before casting votes (I think SU is 2k, but SO is 3k)
    – Dave
    May 16, 2014 at 14:33
  • Both SU and SO are 3K, @Dave (along with every other site that isn't in beta).
    – Shog9
    May 19, 2014 at 22:04
  • 2
    The time until next review is not random, it is always on 0:00 UTC. You can view the server's time by using the achievements menu and look at the upper right. May 20, 2014 at 2:03

5 Answers 5

9

There are enough people who could be reviewing these to drop the queue to nothing very quickly, if they were inclined to do so. It's important to encourage as many reviewers as possible; when a small handful of users are effectively acting as gatekeepers for every question that arrives on the site, it becomes very difficult for the majority of active users to have an influence on what is being asked and answered here - it's better to have 50 people here doing 20 reviews each day, not 20 people doing 50 reviews.

And that's not even getting into how easy it is to become careless when you're reviewing dozens of posts in one sitting.

In fact, it's not altogether a bad thing if the queue never reaches 0. There's a certain urgency that arises when a queue hovers close to empty, and that's not always a good thing! Getting suggested edits or new-user posts reviewed quickly is helpful; these tend to be quite time-sensitive. Closing can be, but while a suggested edit may become worthless in a matter of minutes, many poor questions can still be closed a day or two later with no ill effects. Given it takes 4 days before close votes even start to age away, a question that doesn't need to be closed doesn't really suffer by staying in the queue for that long either.

The big problem here is that some of those pending review tasks are a year old!

See, the queue prioritizes new stuff first - tasks that are newly-created stand a better chance of being shown to reviewers than old tasks do. Tasks that've already been reviewed also get prioritized - so y'all are actually doing a pretty good job of keeping up with tasks that showed up in the past few weeks (pretty much since we added that indicator at the top for 10K users).

So what can you do about those old questions lurking in the back of the queue? Well, a good chunk of them probably don't matter - that's why they're lurking, because no one is looking at them. But some of them could use a bit of attention... And this is where a recently-introduced change to the closing system can help you out: if you have a gold badge in a given tag, filter the close queue down to just duplicate reviews in just that tag, and you'll be able to single-handedly clear out up to 20 every day:

Incidentally, I love filtering by tag - it's the only thing that makes reviewing close votes bearable. So I went and turned on another new feature:

quick filter links

That's m0sa's new filtering UI - you get quick links from /review directly to the top tags and close reasons. I like it a lot, and hope y'all will as well.

There's one other problem: y'all get 20 reviews, and only 24 votes - so if you review 20 posts and all 20 should be closed, that only leaves 4 votes for new questions that you encounter on the site. That paltry surplus leaves some of you in the position of having to ration your votes - not a good way to encourage folks to review (unless you wait until just prior to the new day). So I'm raising the number of close votes per person here to 50 per day. That's enough to review judiciously, close new questions if necessary, and still have plenty of votes in reserve in case something terrible happens.

The number of reviews you can do each day remains at 20 - it'll automatically increase to 40 if a queue ever goes over 1000.

15
  • Cool! Except there's only 14 Super Users with Gold Tag Badges and not all of those are consistent reviewers. May 19, 2014 at 23:18
  • Only takes one of them to make a big difference, @KronoS. Which is kinda the point.
    – Shog9
    May 19, 2014 at 23:22
  • I guess that makes sense. May 19, 2014 at 23:23
  • Actually now that I think about it more and more I do like that Super Power and how limited the number of users would have it. May 19, 2014 at 23:24
  • 5
    This is a big step forward, but is still a bit skewed : If I open the post not thru the review system I can close 50, but only 20 thru reviewing, so I can still review 50 this way and bypass the 20 limit. So why not make reviews also 50? One more idea: Sort the Close queue first by number of close votes descending and then by date. This way, closing will not be postponed. Another remark: Duplicate votes are relatively rare, so maybe something should be done for the other types, for example setting one gold-vote to count as two votes, to speed up closing.
    – harrymc
    May 20, 2014 at 6:08
  • Also, the duplicate filter has some issues. See this screen capture.
    – harrymc
    May 20, 2014 at 8:39
  • @harrymc what do you mean by a gold vote? A gold review vote or gold tag vote? May 20, 2014 at 16:11
  • @KronoS: I meant a gold tag vote. I also up-voted your answer. In the same direction you proposed, maybe we could have something like 1 gold vote = 3 votes and 1 silver = 2 votes.
    – harrymc
    May 20, 2014 at 19:20
  • @harrymc About the number of reviews: It's because SE wants you to be able to salvage or edit the questions, instead of always clicking 'Close' on every question. I asked this myself, as I had 25 Reviews, but still had close votes left (post from over a year ago) May 21, 2014 at 19:42
  • 1
    That's not quite true, @CanadianLuke - if it was, we'd give you more reviews than close votes. What we don't want to happen is you blowing all of your close votes in /review and then having to flag problematic questions you run across. It is a good idea to salvage useful questions if possible, but in many cases it simply isn't.
    – Shog9
    May 21, 2014 at 19:45
  • Ahhh, OK. Misinterpreted then May 21, 2014 at 20:01
  • @Shog9: I really don't think that a high-reputation user needs help in managing his vote count. Reviewing 50 posts in one day is really an exploit (might be worth a badge). I would suggest setting both limits to 50 and let the reviewers manage their own votes. 20 is too low, especially since not everybody does reviews every day. Also, seriously reviewing a complicated post just to find out it was only a test is really discouraging.
    – harrymc
    May 21, 2014 at 20:20
  • 2
    As I stated in my answer, the reason for the low review cap is simply to give a broad cross-section of reviewers a shot at reviewing. The higher vote cap is to avoid penalizing reviewers for reviewing. @harrymc
    – Shog9
    May 21, 2014 at 20:24
  • @Shog9: It's illogical to suppose that a low review limit is an encouraging factor for reviewers. Just the opposite (at least in my case). I think it's more important that close votes be treated than waiting for more reviewers to arrive. This just places a greater burden on the mediators rather than spreading the load.
    – harrymc
    May 21, 2014 at 20:36
  • @Shog9: Please read my addition to my post.
    – harrymc
    May 27, 2014 at 15:22
4

I think this is a great idea but with a caveat. New reviewers have a less limited ability as opposed to tried and tested reviewers. This can be tested based on the badges relative the reviewing queue that's used.

  • < Bronze - 20 reviews
  • Bronze - 25 reviews
  • Silver - 35 reviews
  • Gold - 50 reviews

This would help to give more power to those that actively use the system for good, but discourage the misuse of the power for newer reviewers.

3

I do not understand where these limits of 20-per-12 came from, but I believe that they are at the root of the problem of increasing number of untreated posts in the Close queue.

Perhaps it's to encourage a wide number of people to participate with a reasonable load rather than a few working a large load on it. I can see a case that, for a democratized, self-maintaining site, the more diversity of people that can look at these, the fairer and better for the site.

Of course, maybe the limits in and of themselves aren't having that effect.

I would also suggest a further increase in that limit for users that have achieved the privilege level of trusted user, perhaps even making it unlimited, and also to not post test questions to such users.

I think the test questions are necessary to prevent the temptation of not reading them carefully. By having that in place, it's an assurance that the review process has integrity and can be trusted, being friendlier and fairer to new users. Maybe trusted users should have less test questions but I disagree with eliminating them entirely.

I agree that trusted users should have a higher limit. I think a limit should exist, though.

3
  • I agree that the limits help keep the review process democratic, as opposed to oligarchic.
    – Excellll
    May 17, 2014 at 19:55
  • 10
    Perhaps it's to encourage a wide number of people to participate with a reasonable load rather than a few working a large load on it. Well it's not doing that. All it's doing is making the same people work a large load on it over a large period of time.
    – Jon
    May 18, 2014 at 0:45
  • 1
    @Excellll: SU is already "oligarchic", based on reputation, and mediators are elected, but for life. There no credible alternative to this system, although maybe it is time for it to be tweaked.
    – harrymc
    May 18, 2014 at 8:00
1

This suggestion is a non-starter. Let's say the review limit is raised and reviewers take advantage of it. In a short period of time, the already not overwhelming stack of close votes will go to 0. Then what? When is the higher limit going to come in play again?

So, really, what's the point? Is the current stack of ~600 questions awaiting close votes such a big problem that it requires a policy change? Considering the stack has been roughly the same size for months (and actually shrinking over the past few weeks, not increasing as you claim), the system is pretty close to an equilibrium point. To me, that's a sign of the current policy working as it should.

I suggest you resolve to meeting your review limit each day if the number of question awaiting review is a hindrance, and maybe encourage others to do the same.

3
  • Don't you think that asking reviewers to come back every day in order to decimate the Close queue is a bit too demanding?
    – harrymc
    May 18, 2014 at 16:02
  • No, no one user in particular is being tasked with clearing the review queue. You don't have to do it every day. If you choose to, great, but there's no expectation of you to do that.
    – Excellll
    May 18, 2014 at 17:22
  • 2
    Actually there are users that probably feel themselves tasked with this and are working pretty hard: I suppose that the 20-per-12 limit doesn't apply to mediators. Increasing these limits would let them get more help.
    – harrymc
    May 18, 2014 at 20:49
1

@Shog9, wrote, 11 days ago:

The number of reviews you can do each day remains at 20 - it'll automatically increase to 40 if a queue ever goes over 1000.

That's the type of system, I think is, intelligent: one that adapts limits based on real conditions. It automatically allows more input when not enough people are processing a queue, but refines the limits when many people are contributing to ensure that decision making is decentralized & spread out.

With enough heuristics in place, the system moves more towards sharing characteristics in common with AI systems. AI systems are the way to go to handle larger volumes consistently and objectively. As they improve they can even handle more "edge cases".

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .