For Grins… The Developer Summary

It was interesting to review the feedback and comments from the related coverage associated from our previous post “The Most and Least Consistent Videogame Publishers.”   One report that many wanted to see was a developer specific score card which I thought might be interesting as well.  I had some time, so I thought I’d put something together.

The method was pretty much the same as we used for the publisher report, except I didn’t roll up or consolidate developers and all platforms were included.  No pretty charts or graphs here, just an excel file containing the master developer summary, and tabs for breakdowns by ESRB and Platform. 

Here’s the scope:

• Date Range:  2000 to 2008
• Platforms:  All (Dreamcast, DS, Gamecube, GBA, N64, nGage, PC, PlayStation, PS2,PS3, PSP, Wii, Xbox, Xbox 360)
• Developers:  All  (1742 devs)
• Game Titles: 8,411
• Media Reviews:  168,220 (est)
• Source:  Metacritic 
• Score Type:  Non-weighted game averages were calculated to replace the weighted Metacritic score.
• Filter:  Metacritic Score required (games with four or more reviews applied)
• Method:  ABCDF (+2,+1,0,-1,-2) Gradescale

Right Click on the Image Above, and “Save Target As.”

If you want to see the best in each of the categories, look for the developers with the highest Grade Score or the lowest “Rank” number.  Each tab in the spreadsheet lists all the score card results by ESRB and Platform, so you’ll be able find how your favorite developer ranks. 

Biggest disclaimer here…  I’m sure that out of the 1,700 or so developers, some of the applicable studio detail may not be intact.  For example, the Sony Developers and associated studio regions may not be entirely accurate.  Although I did normalize the developer names to address spelling anomolies, that is about all I did.  There may be some devs that were rolled up under master studios (ex:  EA Sports) where accurate studio/dev breakdowns didn’t exist.

Finally, I make no determination of what this summary does, or does not represent.  I’m simply presenting the score card summary. 

Cheers,

Tim

The Most and Least Consistent Videogame Publishers

The scope of this report is limited to the following platforms; PC, PS3, Xbox360, PSP, Wii, and the DS. We included games from our data catalog from 2000 to 2008 for records where a Metacritic score was available.  This was a straightforward method to include only game titles where at least four or more individual reviews were applied.

Including all games titles, regardless of the number of reviews, is problematic. For example, there are titles where only one review was posted.  One or two reviews for a game did not meet what we considered to be an acceptable ”consensus”, and were therefore omitted.

To mitigate concerns regarding the weighting influence of Metacritic scores, all weights were eliminated and new scores were applied by averaging all reviews that attributed to the related game title.  This process encompassed 274 Media Outlets, 389 Publishers, 4,259 games, and 87,329 individual reviews*.   The average number of contributing outlets per game title was 20.5.

The non-weighted average scores were applied to a standard grading scale with an associated grade value:

Grade Scores were then totaled for each publisher.  Those with the highest values are assumed to be the most consistent Publishers, while those with the lowest total, the least consistent.  Using this method, publishers who may have created stellar titles, would also be penalized for each low scoring game and given no credit for average games.

For reference, we did consider two other methods that were dismissed:

• Comparing the highest percentage of positive scores by publisher yielded publishers releasing only a few titles and had almost no consideration for larger publishers.

• Applying the grade scale to Metacritics own favorability index yielded erratic results as explained previously in our 2008 scoring summary. We lost any correlation between top performers and their related average scores. Specifically we noted too many high ranking publishers with low average scores.

In addition, there are several variations of publisher names depending on factors such as territory, partnerships, etc.  These were all normalized and rolled up into the master publisher.  For example, Sega, regardless of territory would be represented simply as Sega.  Electronic Arts, EA Sports, Electronic Arts / Warner Bros were represented as Electronic Arts.  The only exception here is in regard to T2, 2K, and Rockstar which we treated as individual publishers rather than rolling all properties under one umbrella.

In regard to the normalization of scores, i.e. conversion of different scoring scales to a 100 scale, we rely on the native Metacritic conversion.  This is a potential concern regardless of the datasource and is in no way unique to Metacritic.  We make the assumption that this conversion is fair regardless of game or publisher.  In other words, any possible errors or interpretation of conversion benefits or penalizes each publisher equitably.

Finally, it is imperative to keep these results in perspective as they do not reflect trends towards an increase or decrease in quality trends.  For example, a publisher on the “Most Consistent List” may in fact be trending towards a decline in quality whereas a publisher on the “Least Consistent List” may trend towards an increase in quality.    It is also entirely plausible that the publisher noted, may be a standout leader in a particular genre or platform but performing poorly in another.  These variations and possible trends are not presented as it was not inclusive of the requested scope.

*Our first draft of the report applied the native Metacritic scores.  We took the extra steps to eliminate the weights in the final report as to address any concern over the weighting impact.   In the end, the weighting influence was negligable and the results, almost verbatim.   Typically, weighting deviations are most noticable as you examine narrow segments of data, or plot data over time.  For example, a historical summary for a specific platform by year or quarter may experience such issues.