Is 2 stars reject or approve? Andrew's answer in October's AMA
In the October AMA I asked this:
Q: DGaleano - Simple yes or not opr question. If I vote 2 stars to a candidate and it gets approved....is it an agreement?
And AK answer was:
A: It’s not a simple yes or no question and one I can’t answer to avoid people gaming the system.
I think that was a yes. What do you think?
Post edited by DGaleano on
1
Comments
Probably depends on where you give 2 stars.
I asked a similar question once (outcome if all reviewers voted 2* on every section) which got the answer that a 2* accessibility rating would mean a reject, regardless of the other ratings.
3 star is / was clarrifed as being neutral, so by that logic lower is a reject.
I'd prefer not to assume that so that is why I asked in the AMA and here I'm only concerned about the meaning of Andrew's answer.
Your reasoning is right but many times we have seen "right" is not the way that NIA thinks.
Yes, I understand 2* in accessibility is reject, but then my question in the AMA is "do you get an agreement or not?"
You voted 2* in "should this be a portal" so, did you approved or rejected? What do you think about Andrews answer to that.?
My question is not about the portal being approved or rejected but about the agreement you get as a reviewer which in the end is what counts for your medal and it is what really tells if you are a good reviewer or not.
the real question is, if you vote 2 and it is approved do you get an agreement, but if it is rejected do you get an agreement?
If I was writing a system that measured "Agreement", the first thing I'd try is comparing your score to the average score, and setting a threshold for how far out you can be.
This would give a system that doesn't give a yes/no answer to your question, and is gameable; imagine I've set a threshold of 1.75 stars. between your vote and the average. If you award a 2 and everyone else says 5, it's approved and you don't get an agreement. You award a 2 and the others give it a mixture of 3s and 4s, it's approved but you do get an agreement. Clearly you can game this system by never using 1 or 5, so you'd actually need to have a more complex system, perhaps one that reduces the threshold for people who only use 2,3 or 4.
Interesting!!
Then your system would be a fair system.
I'm not sure the current system is like that.
I'm pretty sure the system isn't like that, or I wouldn't have posted how to game it... but I think it shows that the answer you got was an honest one. I think you got the reply "It’s not a simple yes or no question and one I can’t answer to avoid people gaming the system." because really is "... not a simple yes or no question and one [he] can’t answer to avoid people gaming the system."
@Rostwold I now see your point.
Thanks a lot for your insight. That was really helpful.
I was not as open minded as I should have been. Your observation made me realized that.
Thanks again
Your title doesn't really match the question. Only 1 star is explicit rejection. A 2 star is very low approval.
If that was the case wouldn't Krug have said that?
It's how the OPR guide was phrased. 2-5 were "passing" ratings. No idea what sort of overall threshold would be needed to make the cut though. That said, the question was about a specific rating affecting Wayfarer agreements. If he were to spell out how it's all set up, people would just rate for agreements rather than accurately like some are already doing.
That said, I'm easily maintaining great status without giving 2 craps about popular opinions other than interpretation of guidelines and AMAs. So clearly the 1 star addicts aren't messing up my agreements too much.
i always thought its a vote system, if the poi doesnt pass the threshold dont go in, and i bet if the threshold is 30 stars and everybody voted 2 starts is an agreement, if it gets approved and you voted 2 stars but many voted 5 stars and it got 50+ stars is not an agreement for you but for the others because you didnt vote according to the majority
that is how i see it, overall i can be wrong i can be right, doesnt matter in the end
It's hard to think of 2 as a 'passing' rating. Will it pass even if every reviewer ends up voting 2...?
Though I suppose it might be like US academic grading. One star is F (fail), while two is D: a barely passing grade. But then where's the cut-off point? If the average of 2 were enough to pass, then I'm sure there'd be far less rejections.
Ah well, better not overthink it. I'll just keep voting as I always have. Haven't had any trouble maintaining great rating, so no need to change it.
My question is about Andrew's answer regarding agreements. What do you think?
They will never provide much info on such a topic.
Personally I always saw 2-4 as reject/neutral/Yes
With the agreements being awarded per result except for 3s possibly getting an agreement if majority vote 3 regardless of which way it goes. Or simple it's a NO comment and you miss out
The AMA question was if you get an agreement if a 2 star rated portal passes. Your title asked if 2 star ratings are approving or rejecting. Two different questions.
As for AMA, my guess, regardless of how many stars, is that agreements are based on your rating vs the majority rating.
I think in a previous AMA, Andrew Krug answered that if a portal had an average rating of 3-stars in every category, it would be rejected. Does anyone remember which AMA I'm possibly thinking of?
I asked that question (if everything gets 2*, will that end up being rejected)
His answer was: an average 2* rating for safely accessible would end up in a reject
I will update this comment with the full Q/A
Edit: it was the February 2019 AMA
Q: There still is a lot of confusion
about the star ratings in OPR, hypothetically, if every
reviewer rates every section 2*, would that portal bea acceptedor rejected?
A: I was told that, "If all reviewers rated all items 2*
it would be rejected. Not all answers are considered
equally so just the low safety rating would be a
good reason for a rejection."
Sorry about the formatting, on phone so can't edit it properly