Page 1 of 1

Problem with Metacritic

Posted: February 3rd, 2017, 5:09 pm
by Steve
I'm one of those people who hate Metacritic but often use them as a reference just to get a general idea of review trends and as links to other sites. Of course, it's also a reference for media other than video games. I've always felt their weighted metascore was not a good indication of true review averages with sometimes a single negative review bringing down an otherwise well-reviewed game (Uncharted 4, Quantum Break).

Even though it is a new movie, I think I may have the proof I need to convince myself this website is a joke. I wasn't even checking movies but on the sites homepage, Rings is shown with a metascore of 19. While I had no plans to see this movie I was certainly curious on the scores so I checked out all 11 reviews. Here are the scores:
40, 30, 30, 25, 25, 25, 25, 20, 20, 20, and 16 (The score of 16 being from Indiewire)

I'm not sure of their weighting formula but it would be hard to convince me that the metascore is 19 when only a single review is under 19 - and barely under 19. Again, I understand there is a formula used and I understand some reviews carry more weight than others but to me, this metascore should at least be in the low 20's. It certainly doesn't change the overwhelming negative average that much but to me it shows that things like this probably happen more often than not. And when these scores have an impact on sales and public opinion, I think there is a problem.

Re: Problem with Metacritic

Posted: February 3rd, 2017, 5:28 pm
by scotland
That's interesting, isn't it.

The only sub-20 review is by Eric Kohn submitting for Indiewire, compared to reviews from the New York Times and Variety, which you would hope would be weighed more heavily than Eric Kohn of IndieWire. So if they are using the scores as listed the only way this works as a weighted average is if Metacritic gives Indiewire a weight of 20 and all the rest have a weight of 1. So there you, Metacritic's secret sauce, Eric Kohn's opinion.

Re: Problem with Metacritic

Posted: February 4th, 2017, 4:18 am
by David
Looking at it now it says it's still a 19 with 13 reviews. One of the new reviews was a 0 from Peter Travers of Rolling Stone. If I was to guess I'd say they updated the Metacritic score without yet linking to the review, seeing as how Travers is a bigger name reviewer and the Metacritic score didn't change with his 0 added in. The average of all the reviews is 25.xx, so clearly Rolling Stone's review and a few of the other 20s wight more heavily than not.

I think there are problems with Metacritic, but I don't understand why so many people hate the site (and usually still use it as a reference funny enough). The biggest I think is that their weighted system isn't transparent, and obviously heavily favors certain people/websites as your example shows. I'd rather see them give a straight average of every score, and then separately list "well respected reviews" like Rotten Tomatoes does. Not to mention the review scales can vary greatly between each site, more so for games it feels like. On a 5 point scale 3/5 may be decent to good, whereas on a 10 point scale a 6/10 is probably seen as less than good.

I don't see the issue with Uncharted 4 and Quantum Break's score though. I remember seeing people freaking out over The Washington Post's 40 score for Uncharted score and how it dropped from a 94 to a 93, but I'm not sure how it overly affected its score any more than an outlier normally would in a straight average. Would you rather have the score not included because it's an outlier? That would be more dishonest than including it I would think. Quantum Break's score looks fair as well at a quick glance, with a lot of mediocre 60-70 scores and a handful of 40s. I think Metacritic is at its best as a place where you can get a quick overview, and quick access to many website's reviews in one spot without having to go to each site and find the review manually.

Re: Problem with Metacritic

Posted: November 30th, 2018, 11:15 am
by Steve
Another movie I'm not planning on seeing but was interested in how The Possession of Hannah Grace was reviewed. Of course, the first thing I see is a metascore of 46 based on 4 critics. So, I check all 4 reviews and each one is 50. How is 46 even a weighted average? I'm not sure how often it happens but to think people's bonuses are at least partially determined by this score is disturbing. What is more likely happening is that sales are affected by people who just rely on these scores. Just like an item being priced $9.99 instead of $10.00, scores like 69 can have an impact on perception vs. a score like 70. Even 89 vs. 90. I'd love to know if there is some 'science' behind weighted averages that I don't know about.