The Most and Least Consistent Videogame Publishers

The scope of this report is limited to the following platforms; PC, PS3, Xbox360, PSP, Wii, and the DS. We included games from our data catalog from 2000 to 2008 for records where a Metacritic score was available.  This was a straightforward method to include only game titles where at least four or more individual reviews were applied.

Including all games titles, regardless of the number of reviews, is problematic. For example, there are titles where only one review was posted.  One or two reviews for a game did not meet what we considered to be an acceptable ”consensus”, and were therefore omitted.

To mitigate concerns regarding the weighting influence of Metacritic scores, all weights were eliminated and new scores were applied by averaging all reviews that attributed to the related game title.  This process encompassed 274 Media Outlets, 389 Publishers, 4,259 games, and 87,329 individual reviews*.   The average number of contributing outlets per game title was 20.5.

The non-weighted average scores were applied to a standard grading scale with an associated grade value:

Grade Scores were then totaled for each publisher.  Those with the highest values are assumed to be the most consistent Publishers, while those with the lowest total, the least consistent.  Using this method, publishers who may have created stellar titles, would also be penalized for each low scoring game and given no credit for average games.

For reference, we did consider two other methods that were dismissed:

• Comparing the highest percentage of positive scores by publisher yielded publishers releasing only a few titles and had almost no consideration for larger publishers.

• Applying the grade scale to Metacritics own favorability index yielded erratic results as explained previously in our 2008 scoring summary. We lost any correlation between top performers and their related average scores. Specifically we noted too many high ranking publishers with low average scores.

In addition, there are several variations of publisher names depending on factors such as territory, partnerships, etc.  These were all normalized and rolled up into the master publisher.  For example, Sega, regardless of territory would be represented simply as Sega.  Electronic Arts, EA Sports, Electronic Arts / Warner Bros were represented as Electronic Arts.  The only exception here is in regard to T2, 2K, and Rockstar which we treated as individual publishers rather than rolling all properties under one umbrella.

In regard to the normalization of scores, i.e. conversion of different scoring scales to a 100 scale, we rely on the native Metacritic conversion.  This is a potential concern regardless of the datasource and is in no way unique to Metacritic.  We make the assumption that this conversion is fair regardless of game or publisher.  In other words, any possible errors or interpretation of conversion benefits or penalizes each publisher equitably.

Finally, it is imperative to keep these results in perspective as they do not reflect trends towards an increase or decrease in quality trends.  For example, a publisher on the “Most Consistent List” may in fact be trending towards a decline in quality whereas a publisher on the “Least Consistent List” may trend towards an increase in quality.    It is also entirely plausible that the publisher noted, may be a standout leader in a particular genre or platform but performing poorly in another.  These variations and possible trends are not presented as it was not inclusive of the requested scope.

*Our first draft of the report applied the native Metacritic scores.  We took the extra steps to eliminate the weights in the final report as to address any concern over the weighting impact.   In the end, the weighting influence was negligable and the results, almost verbatim.   Typically, weighting deviations are most noticable as you examine narrow segments of data, or plot data over time.  For example, a historical summary for a specific platform by year or quarter may experience such issues.