Archive March » GameQuarry

Archive for March, 2009

For Grins… The Developer Summary

It was interesting to review the feedback and comments from the related coverage associated from our previous post “The Most and Least Consistent Videogame Publishers.” One report that many wanted to see was a developer specific score card which I thought might be interesting as well. I had some time, so I thought I’d put something together.

The method was pretty much the same as we used for the publisher report, except I didn’t roll up or consolidate developers and all platforms were included. No pretty charts or graphs here, just an excel file containing the master developer summary, and tabs for breakdowns by ESRB and Platform.

Here’s the scope:

Date Range: 2000 to 2008
Platforms: All (Dreamcast, DS, Gamecube, GBA, N64, nGage, PC, PlayStation, PS2,PS3, PSP, Wii, Xbox, Xbox 360)
Developers: All (1742 devs)
Game Titles: 8,411
Media Reviews: 168,220 (est)
Source: Metacritic
Score Type: Non-weighted game averages were calculated to replace the weighted Metacritic score.
Filter: Metacritic Score required (games with four or more reviews applied)
Method: ABCDF (+2,+1,0,-1,-2) Gradescale

Developer Score Card

Right Click on the Image Above, and “Save Target As.”

If you want to see the best in each of the categories, look for the developers with the highest Grade Score or the lowest “Rank” number. Each tab in the spreadsheet lists all the score card results by ESRB and Platform, so you’ll be able find how your favorite developer ranks.

Biggest disclaimer here… I’m sure that out of the 1,700 or so developers, some of the applicable studio detail may not be intact. For example, the Sony Developers and associated studio regions may not be entirely accurate. Although I did normalize the developer names to address spelling anomolies, that is about all I did. There may be some devs that were rolled up under master studios (ex: EA Sports) where accurate studio/dev breakdowns didn’t exist.

Finally, I make no determination of what this summary does, or does not represent. I’m simply presenting the score card summary.

Cheers,

Tim

Monday, March 16th, 2009 Historical 2 Comments

The Most and Least Consistent Videogame Publishers

The scope of this report is limited to the following platforms; PC, PS3, Xbox360, PSP, Wii, and the DS. We included games from our data catalog from 2000 to 2008 for records where a Metacritic score was available. This was a straightforward method to include only game titles where at least four or more individual reviews were applied.

Including all games titles, regardless of the number of reviews, is problematic. For example, there are titles where only one review was posted. One or two reviews for a game did not meet what we considered to be an acceptable ”consensus”, and were therefore omitted.

To mitigate concerns regarding the weighting influence of Metacritic scores, all weights were eliminated and new scores were applied by averaging all reviews that attributed to the related game title. This process encompassed 274 Media Outlets, 389 Publishers, 4,259 games, and 87,329 individual reviews*. The average number of contributing outlets per game title was 20.5.

The non-weighted average scores were applied to a standard grading scale with an associated grade value:

Game Grading Scale

Grade Scores were then totaled for each publisher. Those with the highest values are assumed to be the most consistent Publishers, while those with the lowest total, the least consistent. Using this method, publishers who may have created stellar titles, would also be penalized for each low scoring game and given no credit for average games.

For reference, we did consider two other methods that were dismissed:

• Comparing the highest percentage of positive scores by publisher yielded publishers releasing only a few titles and had almost no consideration for larger publishers.

• Applying the grade scale to Metacritics own favorability index yielded erratic results as explained previously in our 2008 scoring summary. We lost any correlation between top performers and their related average scores. Specifically we noted too many high ranking publishers with low average scores.

In addition, there are several variations of publisher names depending on factors such as territory, partnerships, etc. These were all normalized and rolled up into the master publisher. For example, Sega, regardless of territory would be represented simply as Sega. Electronic Arts, EA Sports, Electronic Arts / Warner Bros were represented as Electronic Arts. The only exception here is in regard to T2, 2K, and Rockstar which we treated as individual publishers rather than rolling all properties under one umbrella.

In regard to the normalization of scores, i.e. conversion of different scoring scales to a 100 scale, we rely on the native Metacritic conversion. This is a potential concern regardless of the datasource and is in no way unique to Metacritic. We make the assumption that this conversion is fair regardless of game or publisher. In other words, any possible errors or interpretation of conversion benefits or penalizes each publisher equitably.

Finally, it is imperative to keep these results in perspective as they do not reflect trends towards an increase or decrease in quality trends. For example, a publisher on the “Most Consistent List” may in fact be trending towards a decline in quality whereas a publisher on the “Least Consistent List” may trend towards an increase in quality. It is also entirely plausible that the publisher noted, may be a standout leader in a particular genre or platform but performing poorly in another. These variations and possible trends are not presented as it was not inclusive of the requested scope.

Most Consistent Publishers

Least Consistent Publishers

*Our first draft of the report applied the native Metacritic scores. We took the extra steps to eliminate the weights in the final report as to address any concern over the weighting impact. In the end, the weighting influence was negligable and the results, almost verbatim. Typically, weighting deviations are most noticable as you examine narrow segments of data, or plot data over time. For example, a historical summary for a specific platform by year or quarter may experience such issues.

Wednesday, March 4th, 2009 Historical 13 Comments

The 2008 Metacritic Summary Guide

We have presented what is likely the most comprehensive summary of Metacritic scoring summary for 2008, but the we realize the amount of detail may be duanting and a bit tough to navigate. This section should help you find what you need.

Game Titles with notable weighting swings (2008)

This section provides details regarding games released in 2008 where Metacritics Weighting system had a noteable impact to the Metacritic Score.

Metacritic 2008 Platform Summary

The platform summary is helpful for those who are looking for detail regarding the volume of titles by platform or highest scoring platforms. Also in this section is our determination of the most consistent platform in regard to favorable scores.

Metacritic 2008 ESRB Summary

Curious about the volume and average scores as it relates to ESRB? This section is formatted in the same manner as the platform summary and details volume and highest scores by each ESRB Category as well as a look at favorability.

Metacritic 2008 Genre Summary

There is a wealth of interesting data here. In the genre summary, we’ll take a peek at Genre distribution and favorability overall, as well as digging in to both ESRB and Platform distribution by Genre.

Developers: Intro & ESRB Summary - Part One

The first part in our Developer series provides some interesting detail in regard to developer performance overall, and as it relates to their performance by ESRB Category.

Developers: Genre - Part Two

As an extention of our genre summary, we take a look at the both the volume and performance of developers by Genre.

Developers: Platforms & Consistent Peformers - Part Three

Developer volume and high score distribution is reviewed. We also make our own determination regarding the most consistent developers for 2008.

Publishers: Intro & ESRB Summary - Part One

Following the same format as we did with developers, we’ll take a look at the publisher volume and related scoring detail. The ESRB section will give you some insight regarding publisher performance by ESRB category.

Publishers: Genre Summary - Part Two

For those of you who want to know about how the various publishers perform by genre, this section is for you. Explored are both volume and highest scorers by genre.

Publishers: Platforms & Consistent Peformers - Part Three

Our final publisher section explores the performance by platform. Also presented is our determination of the most consistent publisher for 2008.

Media Outlet Summary

Who ever does this? Well, we do. In the media outlet summary we provide some context behind the reviewers that contribute to the Metacritic scores. Who contributed the most reviews? Who are favorable reviewers? Who are the most critical? Interesting stuff.

Historical Reference for 2008 Summary

The Historical reference provides and overview of game volume and average Metacritic scores from 2000 to 2008. Also provided are historical comparisons by platform and ESRB.

Metacritic Weighting Video Report

We have a few video reports you may find helpful. This video in particular is perhaps the most insightful as it expolores Metacritics Weighting System and the related impact to the index scores.

Monday, March 2nd, 2009 2008 Scoring Analysis No Comments

Historical Reference for 2008 Summary

This section provides some historical context you may find helpful. I have focused largely on overall game volume, and breakdowns for ESRB and Platforms, i.e. volume and average Metacritic scores by category.

Game Volume by Year

Average Metacritic Scores by Year

ESRB Volume by Year

Metacritic Averages by ESRB

Platform Volume by Year

Average Metacritic Scores by Platform

Monday, March 2nd, 2009 2008 Scoring Analysis, Historical No Comments

Media Outlet Summary

In the Media Outlet Summary we’ll look at those who contributed to each Metacritic Score for 2008. This will be the only section where we don’t actually use Metacritic scores, rather, the non-weighted average scores. Basically, this means that we’ll be going through each game title and averaging the scores manually, so there will be no weighting influence. For more information about the weighting system, please refer to our video report.

This following detail is a demonstration of what we refer to as a “deep dive.” It requires that we take notice not only of basic game and scoring meta data for each title, but recognize the outlet data as well. In a normal “shallow dive” we’d be looking at about 1,400 records for 2008, but in this case, we’ll be recognizing over 25,000 (each media outlet review).

We could use this method to determine possible outlet bias towards a particular platform, genre, developer, or publisher. We could furthermore determine outlet comparisons based on the outlet type. For example, Game Centric outlets vs. Consumer Outlets, or by region, i.e. North America vs. International.

For reference, there were 177 Media Outlets Contributing to Metacritic results for 2008. Our first table shows the top contributing media outlets for 2008, their average score, the average non-weighted score for the titles in which they contributed, and the variance between the two. A positive variance means they are more favorable to the average score, while a negative number notes they are more critical than average.

Top Media Outlets By Volume

The distribution of those media outlets that are more/less favorable than the average are represented by the pie chart below:

Media Outlet Favorability Distribution

These next two tables represent the most favorable and most critical outlets overall. Again, the “Var” column notes how many points higher/lower the outlet is for the average non-weighted score. For example, in the table below, Gamers Europe had 1 review with a score of 36 while the average of all the other contributing outlets was 65.

Most Favorable Outlets for 2008

Most Critical Outlets for 2008

Sunday, March 1st, 2009 2008 Scoring Analysis 1 Comment

Publishers: Platforms & Consistent Peformers - Part Three

As with the first two parts of our publisher summary, we’ll be continuing the table theme as we did with the ESRB’s and Genres. If you simply want to know who the most consistent publishers are, then scroll to the bottom.

These scores, averages, and counts are unique to the specific platform, and may not reflect the developers overall counts, or averages. Each Platform is represented by two tables, one displaying the most popular (frequency) of developers, and the second notes the highest MC scores in that Platform category by developer.

For reference, the green highlighted cells represent the highest value in that column, whereas the orange represents the lowest. For example, “Square Enix” has the highest value in the category, “Best MC Score” while the lowest score of the featured publishers is “Activision.”

Most Popular Publishers for Platform 'DS'

Highest Scoring Developers for Platform 'DS'

Most Popular Publishers for Platform 'PC'

Highest Scoring Publishers for Platform 'PC'

Most Popular Developers for Platform 'PS2'

Highest Scoring Publishers for Platform 'PS2'

Most Popular Publishers for Platform 'PS3'

Highest Scoring Publishers for Platform 'PS3'

Most Popular Publishers for Platform 'PSP'

Highest Scoring Publishers for Platform 'PSP'

Most Popular Publishers for Platform 'Wii'

Highest Scoring Publishers for Platform 'Wii'

Most Popular Publishers for Platform 'Xbox 360'

Highest Scoring Publishers for Platform 'Xbox 360'

Finally, we’ll wrap up this segment of the summary by trying to figure out who the most consistent publishers are. For reference, we’ll be applying the same method we did for platforms, ESRB, and Developers by using the following scale:

Game Grading Scale

The overall total will give us the most consistent publishers for 2008. And the winners are:

Most Consistent Publishers of 2008

The final two sections in the summary will provide some historic perspective, and we’ll also dive into some metrics regarding the media outlets that contribute to the Metacritic scores.

Sunday, March 1st, 2009 2008 Scoring Analysis No Comments

Publishers: Genre Summary - Part Two

These scores, averages, and counts are unique to the specific genre, and may not reflect the publishers overall counts, or averages. Each Genre is represented by two tables, one displaying the most popular (frequency) of developers, and the second notes the highest Metacritic scores in that Genre by developer.

If you have any questions in regard to how I came about this selection of Genres, or how they were aggregated, please review the Genre Summary. As with all the previous publisher summary, the counts do not include titles where no Metacritic Score was available (see the intro here).

For reference, the green highlighted cells represent the highest value in that column, whereas the orange represents the lowest. For example, THQ has the highest volume of titles, Rockstar with the “Best MC Score” and “Avg MC Score” while the lowest score goes to THQ (orange).

Most Popular Publishers for Genre 'Action Adventure'

Highest Scoring Publishers for Genre 'Action Adventure'

Most Popular Developers for Genre 'Action'

Highest Scoring Publishers for Genre 'Action'

Most Popular Publishers for Genre 'Adventure'

Highest Scoring Publishers for Genre 'Adventure'

Most Popular Publishers for Genre 'Driving'

Highest Scoring Developers for Genre 'Driving'

Most Popular Publishers for Genre 'Sports'

Highest Scoring Publishers for Genre 'Sports'

Most Popular Publishers for Genre 'Strategy'

Highest Scoring Publishers for Genre 'Strategy'

Most Popular Publishers for Genre 'Music'

Highest Scoring Publishers for Genre 'Music'

Most Popular Publishers for Genre 'Puzzle'

Highest Scoring Publishers for Genre 'Puzzle'

Most Popular Publishers for Genre 'Shooters'

Highest Scoring Publishers for Genre 'Shooters'

Publishers: Intro & ESRB Summary - Part One

As we did with the developer summary, the publisher detail will be in three parts. For reference, the counts shown in the following tables reflect game titles for which Metacritic had a score associated. Metacritic requires that each title have at least four media reviews before a score is assigned. For more details, please view our platform summary.

The publisher data in the next few sections is somewhat normalized. Publishers are often noted differently depending of branch, region, or if they were in a partnership with another party. For example, “Sega” and “Sega Europe” are all represented as “Sega.” We also consolidated the various 2K properties, but did not rollup Rockstar under the 2k umbrella.

The green highlighted areas you’ll see represent the highest values for a column, and the orange highlight for the lowest. For example, the Top 10 Publisher list below notes that Electronic Arts had the highest number of titles, Sony with the best Metacritic Scores, and Activision with worst score. The “Worst” scores are not the overall “worst”, yet just the lowest score in the table and/or column.

Top 10 Game Publishers by Volume

Now we’ll take the top publishers see how their scores are distributed in Metacritics Favorability Index.

Favorability Distribution for Top 5 Publishers

The table below illustrates the percentage distribution of games scored that fall within simple “positive”, “negative”, and “nuetral” categories. This should be helpful to identify basic quality performance.

Top Developers by Volume / General Favorability Distribution

The following are the top developers who scored the most titles falling into the favorability category “Universal Acclaim.”

Publishers with Most "Universal Acclaim" titles

This next table is a summary of highest Metacritic Scores and the associated publishers.

Publishers with the Highest Metacritic Scores

Next we’ll take a look at a series of tables related to the appropriate ESRB and see who publishes the most games per category as well as the best rated games. Note that the scores for best, worst, as well as the averages are a subset of that particular ESRB category. These scores may not reflect that publishers overall best, worst, etc. For example, the Average Rating, is actually the average for that publisher in the noted ESRB category.