Archive February » GameQuarry

Archive for February, 2009

Developers: Platforms & Consistent Peformers - Part Three

As with the first two parts of our developer summary, we’ll be continuing the table theme as we did with the ESRB’s and Genres. If you simply want to know who the most consistent developers are, then scroll to the bottom.

These scores, averages, and counts are unique to the specific platform, and may not reflect the developers overall counts, or averages. Each Platform is represented by two tables, one displaying the most popular (frequency) of developers, and the second notes the highest MC scores in that Platform category by developer.

For reference, the green highlighted cells represent the highest value in that column, whereas the orange represents the lowest. For example, “Square Enix” has the highest value in the category, “Best MC Score” while the lowest score of the featured developers is “Griptonite Games.”

Developers: Genre - Part Two

Not unlike the detail in Part 1 for our developer summary, we’ll be continuing the table theme as we did with the ESRB’s.

These scores, averages, and counts are unique to the specific genre, and may not reflect the developers overall counts, or averages. Each Genre is represented by two tables, one displaying the most popular (frequency) of developers, and the second notes the highest MC scores in that Genre by developer.

If you have any questions in regard to how I came about this selection of Genres, or how they were aggregated, please review the Genre Summary. For reference, the green highlighted cells represent the highest value in that column, whereas the orange represents the lowest. For example, Ubisoft Montreal has the highest values in the categories, “Best MC Score” and “Avg MC Score” while the lowest score goes to Heavy Iron Studios.

Most Popular Developers for Genre 'Action Adventure'

Best Scoring Developers for Genre 'Action Adventure'

Developers: Intro & ESRB Summary - Part One

Of the 1,650+ titles comprised in the summary for 2008, there are well over 300 developers. That pretty much limits us on what we can do with the data. For the most part, we’ll be displaying tables that identify the developers with a higher frequency of titles released per category, i.e. genere, platform, etc. We’ll also note which developers scored the highest Metacritic score in each category.

The green highlighted areas you’ll see represent the highest values for a column, and the orange highlight for the lowest. For example, TellTale Games had the highest Metacritic score above all others, whereas Konami had the lowest posted Metacritic Score, etc.

Top 10 Developers by Number of Titles

Now we’ll take the top developers see how their scores are distributed in Metacritics Favorability Index.

Top 5 Developers Favorability Index

The following are the top developers who scored the most titles falling into the favorability category “Universal Acclaim”

Top Developers Creating Titles considered 'Universal Acclaim'

Below is a list of the top 10 developers scoring the highest overall Metacritic Ratings.

Top 10 Developers by Highest MC Scores

Next we’ll take a look at a series of tables related to the appropriate ESRB and see who develops the most games per category as well as the best rated games. Note that the scores for best, worst, as well as the averages are a subset of that particular ESRB category. These scores may not reflect that developers overall best, worst, etc. For example, the Average Rating, is actually the average for that developer in the noted ESRB category.

An alternative scoring index for the videogame industry

By Tim Sweezey
GameQuarry.com

Whether a Videogame publisher, developer, investor, journalist, or seasoned gamer, all recognize Metacritic as the defacto industry scoring index. Nobody has to like it, or love it, as frankly, it can’t possibly be all things to everyone. Everybody wants and needs a scoring index that reflects their own interests and needs. And there lies the rub.

For those who are critical of Metacritic it’s often a reflection of “Mashups” in general, i.e. an almagamation of average scores from a plethora of media sources. Sometimes games are scored on a scales ranging from 1-4, 1-10, and 1-100, or simple letter grades. Add to that, some media outlets consider a “5″ an average score on a scale of 1-10, while others adopt a more common gradeschool approach where “7″ or “70″ or “C” represents an average score. The bottom line is that there are enough moving parts in a mashup site for anyone to find fault.

What you probably dont’ know about Metacritic and Mashups

In most cases, it makes little difference which index you use. If you took the two most popular mashup sites in the game industry, Metacritic.com and Gamerankings.com, the average difference for any given game title is only 1.2 points. And that minimal point spread is entirely relative. For example, Metacritic may convert letter grades differently than others, and they do apply their own weight system giving media outlets different importance ranks, but since the same rules apply to each game, it’s all equitable. In general, so long as a mashup site uses about 20 or so sources, all these scoring nuances mean little.

Obviously, one of the reasons Metacritic is such a target for critique is because it is the standard. Everyone in the industry speaks the same language when comparing quality as they all use the same index. Don’t fault Metacritic for being the standard, as they never offered themselves up for the role. Their focus is providing review indexes for consumers whether it’s videogames, movies, music, or books, which they do very well. They know the role they play within the industry, yet their methods aren’t influenced by that, and frankly, I find that commendable.

The truth is that no existing game index is appropriate for the videogame industry. The problem is that within the industry, there are different applications for these indexes, that no one mashup can currently accomodate.

The Public Relations and Marketing Index

If you are in Public Relations or Marketing, Metacritics weighting system may be an attractive concept because in general, it recognizes that different media outlets have a different importance. The problem is, those in PR or Marketing may not agree with the media outlets selected for the sample, or perhaps have issues with the manner in which the weights are applied.

Think about it from a publicists perspective. There are, and always will be, media outlets with a larger reach or higher relevancy than others, so reflecting a weight is entirely appropriate. Consider for a moment that you are working on a game title that doesn’t appeal to a hard core gamers, but still has mass appeal to a more casual audience. Weighting a consumer outlet such as the “New York Times” may be more appropriate than weighting a game-centric trade media outlet. Conversely, if you represent a multiplayer shooter, you want your weights applied to the game-centric trade outlets and not necessarily for those that are consumer oriented.

The Developers Index

Deveopers may need something even different. Assume for a moment that your bonus or performance review is at least somewhat dedicated to the quality of the games you produce. So, what is the quality indicator here? You could use a mashup index score to determine quality, but that might not be appropriate. In this case, we want consensus, which has zero room for any weight applications. What if you created a game with with an average score of 80, yet the Metacritic score was only 76 because of the weights applied? You may be penalized, at least in part, by some arbitrary weight that your management couldn’t begin to explain. At the same time, that weight could work to your favor as well. But then, maybe your company gave bonuses when it wasn’t really appropriate.

The Forecasting Index

Then you need another index for sales or forecasting. That’s right, if you have a game that you feel is going to be every bit as worthy of a ‘80′ index score and you want to use this as part of your forecast formula, then you need an appropriate index for the task. What you’ll likely need is to forecast on a consumer vs. game-centric media outlet index, as well as one that gives consideration to region/territory. Gamer-centric trade publications may be about to salivate over the new multiplayer shooter, but there are some countries where single player games are preferred simply because of the infrastructure available to them. In those cases, a heavier weight may apply to single player vs. multiplayer.

The New Index

The videogame industry would be well served to adopt a standard tiered scored index. They can start by coming up with a standard list of media outlets for not just North America, but an International list. Yes, just like Metacritic, although much more complete. In fact, they can learn a great deal from Metacritic. This media list isn’t nearly as daunting as it sounds, as every major publisher could have their preference of outlets reflected. I guarantee, any PR or Marketing team would be happy to provide their list. When it comes to scores, the more the merrier as the data sample just gains integrity.

From there, the segmentation of the various sub-indexes for PR and Marketing, Sales, Development, really don’t require consensus. It’s easy to tell which outlets are consumer vs. game-centric, and which are North American outlets vs. International. That just leaves the weighting system to deal with. In the end, what you end of up with is something that looks like this:

Master Index (all outlets non-weighted)
Regional Consumer Indexes (Weighted)
Regional Game Trade Indexes (Weighted)

So long is the method is entirely public and transparent, everyone should be happy. The overall quality index for the development team is based on the master index, and sales, marketing, and PR can use their respective regional indexes. In addition, the benefit expands to the consumer as seasoned gamers can better relate on the Game Trade Index, while the more casual gamers may find the Consumer Index more appropriate.

Finally, if your business model and/or compensation structure is based, even in part, on a set of data, you better make sure it’s appropriate data. Using consumer based mashups for PR, Marketing, Sales, and Development, is like like trying to hammer a nail in the wall with stapler. You might be able to hang that picture, but it’s going to come at a price.

This article is free to republish or reprint as long as the credit is given to gamequarry.com and the respective author.Tim Sweezey is the Vice President of Artisans Media Group, and the Senior Scoring Analyst for GameQuarry.com. Since having worked for Ziff-Davis Publishing as a Director of Information Technology, the Senior Technology Manager at both ZDTV and TechTV, Tim has been active within the Videogame Industry since 2003 specializing in Public Relations Technology Solutions and Analysis.

Sunday, February 15th, 2009 Articles No Comments

Metacritic 2008 Genre Summary

This next section was tricky in that each title can have multiple Metacritic genres assigned. Of the 1,654 titles from 2008, there are a total of well over 60 different genres with titles comprised of one to three genres each. I would have been generating charts for days, and I think the overall meaning would have been lost or at least diluted. Rather than attempt to assign only one genre per game title, I applied a much different approach which yeilded the summary information I was looking to achieve. The only occasion where I “changed” the genre was in regard to “Racing” and “Driving” which I rolled into one category.

In the end, what I decided to do was to categorize genre’s into basic buckets/categories. For example, the category of “shooter” is comprised of any title regardless of “First Person Shooter” or “Third Person” shooter. The same applies to all “Sports” so you won’t see specific sub-categories, i.e. baseball, basketball, football, etc. This is also true of the “strategy” category as the following summary doesn’t distinguish between “Turn Based Strategy”, “Real Time Strategy.”

The ability to assign multiple custom genre categories still exists as in the case “King’s Bounty: The Legend” by Atari which has both “Role-Playing Game” and “Turn-Based Strategy” genres applied. In this case this title would be part of the analysis of the “Strategy” category, as well as the “RPG” category. The downside here is that many genres such as “Wrestling” simply won’t be included in this review unless they also fall into one of the custom categories I created.

Finally, there are separate categories for “Action Adventure” and “Action” and “Adventure.” I did this because “Action Adventure” is not a specific genre category in Metacritics Index. They would apply “Action” and “Adventure” as separate genre categorys. So why didn’t I just stick with “Action Adventure” and not show results for other titles that might fall into discreet “Action” or “Adventure” categories? Well, both of these are common genres, even if they don’t alway appear together.

The table below breaks down the number of titles which fall into the specified genre category for each platform. Each row (genre) will also display at least one “Orange” and “Green” highlight which simply notes the lowest/highest number of titles for that genre. For example, the “PS3″ had the most titles for the genre ‘Driving/Racing’ while the “PS2″ had the least. You can use they key to identify which are the most and least popular genres for any given platform.

Number of Titles (scored) by Platform Genre

This next table displays the average Metacritic score for each Genre by Platform. As is the case above, I have also applied a “green” highlight for the highest average score and an “orange” highlight for the lowest average score. You can start to see some patterns here in regard to which platforms score the best for each genre.

You want to be careful here as some of this can surprizing/misleading. For example, you’ll notice the DS is noted as having the best average score for the “Shooter” genres, although it only has five game titles in that category. With that said, the takeaway here is in regard to general viability. The “DS” is in fact a viable platform for the “Shooter” genre. Likewise, you can also tell that the “PS3″ and “PSP” Platforms show a higher MC average than other platforms.

Average Scores by Genre / Platform

What we did here was take the highest Metacritic Score noted for each Genre/Platform. The cells highlighted in “green” note highest score for that particular genre. For example, the “PS3″ and “Xbox 360″ both had the highest Metacritic scores for the “Driving/Racing” genre. The platform with the largest frequency of high scores was the “Xbox 360″ while both the “PSP” and “DS” were associated with the lowest frequency of high scores.

Highest MC Scores by Genre / Platform

Much like the table above, the following notes the “lowest” Metacritic scores recorded for each Genre/Platform. The “Orange” highlight represents the lowest of the low scores for each genre. For example, the “Wii” had the lowest Metacritic score for the “Driving/Racing” genre. You’ll notice that the “Wii” has the highest frequency of low Metacritic average scores than any other platform.

Lowest MC Scores by Genre / Platform

This next table is pretty telling in it’s simplicity. It notes whether each platform is above, or below the average for the respective genre. The “Green” notes “Above Average”, “Orange” notes “Below Average” and “Light Blue” is “Average.”

You can tell that both the “PS3″ platform is associated with more “above average” scores than other platforms which is consistent with our platform summary. Conversely, the “DS” is associated with more “below average” scores.

Above & Below Avg by Genre / Platform

We’ll touch more on both Genres and Platforms as we present the detail for Publishers.

Sunday, February 15th, 2009 2008 Scoring Analysis 2 Comments

Metacritic 2008 ESRB Summary

I think the following falls into the category “More than you ever want to know,” but I wanted to make sure I represented this category correctly. One thing that I need to note is in regard to the RP (Rating Pending) category, which shouldn’t exist at all. What this means is that here are titles released and rated where the Metacritic genre wasn’t updated.

Total Number of Titles by ESRB Rating

Average Score by ESRB Rating

The following displays how many titles have been scored, vs. how many titles remain unscored. Again, this is due to Metacritic requiring four outlet reviews before an MC Score is assigned.

Scored vs. Unscored by ESRB

Just as we did in the platform summary, the following uses Metacritics Favorability Index with counts for each appropriate ESRB Category:

Metacritic Favorability Distribution by ESRB

Here, we took the the top two positive categories “Universal Acclaim” and “Generally Favorable Reviews” and rolled them up into a “Positive Index.” Similarly, a “Negative Index” was created for titles that fall into the “Generally Unfavorable Reviews” and “Overwhelming Dislike.”

Custom Favorability Index by ESRB

The following correlates the number of titles per ESRB category with Metacritics Favorability Index. You’ll find one chart per ESRB Category:

Favorabililty Distribution for ESRB 'E'

Favorabililty Distribution for ESRB 'E10+'

Favorabililty Distribution for ESRB 'T'

Favorability Distribution for ESRB 'M'

Finally, we’ll use the grade scale we applied for the platforms to determine which ESRB category is the most consistent in regard to quality. Again, here’s the scale:

Game Grading Scale

After applying the appropriate grade scale to each game title, we totalled the results for each ESRB and came up with the following consistency index ranked in order of ‘most consistent.’

ESRB Consistency Rank

The ESRB ranking ‘M’ for Mature was the most consistent in regard to game quality. This is also supported in other reports found within this 2008 summary.

Friday, February 13th, 2009 2008 Scoring Analysis No Comments

Metacritic 2008 Platform Summary

Here’s the first set of reports in the series for our 2008 recap which will focus on Platforms. Our first chart simply breaks down the distribution of Game Titles by Platform:

Number of Titles by Platform

The next chart denotes the average Metacritic score for each of the platforms:

Average Metacritic Score by Platform

As you’ll see from the chart below, there is a large percentage of game titles where Metacritic scores didn’t exist. This was largely due to Metacritics requirement that at least four media outlet reviews are required before a rating is applied, and the game title simply didn’t have the requisite reviews.

Also, the platform counts here reflect all “scored” and “non-scored” titles, while the MC Scoring Averages dismiss those without scores. In later sections of the summary, we’ll focus only on platform counts where scores have been applied.

Snapshot of Scored Titles Vs. Non-Scored Titles

What this report does is to take all the scoring data from 2008 and correlate it with Metcritics own favorability table which is located here. You can see from the following table how many titles fall within each favorability category:

Metacritic Favorability by Platform for 2008

To view which platforms performed the best, and which basically sucked overall, the following chart should be a good fit. I’ve taken the the top two positive categories “Universal Acclaim” and “Generally Favorable Reviews” and rolled them up into a “Positive Index.” Similarly, I created a “Negative Index” for titles that fall into the “Generally Unfavorable Reviews” and “Overwhelming Dislike.”

Custom Favorability Summary (above avg, avg, below avg)

You can see from the results above, that Xbox 360 titles had more positive scored titles than any other platform. Conversely, the Wii had more titles scoring poorly than any other platform.

Continuing with the quality theme here, we’ll try to figure out which of the following platforms score more consistently (positive) than others. To do this, we’ll use our own grade scale with associated values for each grade:

Game Grading Scale

We’ll take each game from 2008 (scored of course) and apply the appropriate points. In the end, we’ll take the best of the total points to determine which are the most consistent platform in regard to quality. Here’s the list in order of most consistent:

Most Consistent Platforms / Quality

If you are at all skeptical about my conclusions, you’ll see more data in our reports that substantiates that the PS3 is the most consistent platform in regard to overall game quality.

Well, that wraps up the platform summary for 2008, which just happens to be the easiest. If you think I’ve missed something of interest here, please let me know. Also, if you are looking for detail that would illustrate publisher performance by platform, that will come in a later report.

Thursday, February 12th, 2009 2008 Scoring Analysis 1 Comment

Game Titles with notable weighting swings (2008)

This swing report notes game titles where the Metacritic score, and the actual average of outlet scores varies by more than two points (due to Metacritics weighting).

Most of the games below reflect titles with an overall low quality, but there are some exception. Here are some examples:

NHL 09 scored a 73 average, yet only a 66 Metacritic Score (weighting penalty)
Dance Dance Revolution Universe 3 scored a 70 average, while the Metacritic Score was 65 (weighting penalty)
EverQuest II: The Shadow Oddessy scored an 83 average, while the Metacritic Score was 78 (weighting penalty)
Personal Trainer: Cooking scored an average of 78, while the MC Score was 81. (weighting bonus)

Game Title	Platform	MC	Avg	Swing
Backyard Baseball ‘09	Wii	37	48	11
Rapala Fishing Frenzy 2009	Xbox 360	41	49	8
Plattchen: Twist ‘n Paint	Wii	52	45	7
Madagascar: Escape 2 Africa	PS2	56	63	7
Travel Games for Dummies	DS	62	69	7
NHL 09	PC	66	73	7
George of the Jungle and the Search for the Secret	DS	44	38	6
Little League World Series 2008	DS	61	67	6
El Tigre: The Adventures of Manny Rivera	PS2	62	56	6
1 vs. 100	DS	37	32	5
Code of Honor: The French Foreign Legion	PC	37	42	5
Midnight Bowling	Wii	37	32	5
Major League Eating: The Game	Wii	45	50	5
Family Table Tennis	Wii	48	53	5
Hell’s Kitchen	DS	48	53	5
Top Trumps: Doctor Who	DS	53	58	5
Space Chimps	Xbox 360	55	50	5
LOL	DS	56	51	5
Madagascar: Escape 2 Africa	PC	56	61	5
Zenses Rainforest	DS	59	54	5
Saga	PC	61	66	5
My Weight Loss Coach	DS	63	58	5
Dance Dance Revolution Universe 3	Xbox 360	65	70	5
EverQuest II: The Shadow Odyssey	PC	78	83	5
Incredible Maze, The	Wii	28	32	4
Jumper: Griffin’s Story	Wii	28	24	4
Rapala Fishing Frenzy 2009	PS3	31	35	4
Falling Stars	PS2	32	36	4
Carnival Games: Mini-Golf	Wii	43	47	4
Tron	Xbox 360	43	39	4
Iron Man	PSP	44	40	4
Crash Bandicoot: Mind over Mutant	DS	45	49	4
Guild 2: Venice, The	PC	49	53	4
NovaStrike	PS3	49	53	4
Bratz: Super Babyz	DS	50	54	4
Simon the Sorcerer 4: Chaos Happens	PC	50	54	4
We Cheer	Wii	51	55	4
Pit Crew Panic!	Wii	52	56	4
Dash of Destruction	Xbox 360	53	49	4
Spore: Creepy & Cute Parts Pack	PC	53	57	4
Disney Sing It	PS3	54	58	4
Lineage II: The Chaotic Throne- The Kamael	PC	55	59	4
Twin Strike: Operation Thunder	Wii	55	51	4
Critter Round-Up	Wii	56	52	4
TNA iMPACT!	PS2	58	62	4
Fading Shadows	PSP	62	66	4
Murder in the Abbey	PC	62	66	4
Eternal Poison	PS2	65	69	4
Ferrari Challenge Trofeo Pirelli	Wii	65	61	4
Calvin Tucker’s Redneck Jamboree	Wii	29	26	3
Ten Pin Alley 2	Wii	30	33	3
Women’s Volleyball Championship	PS2	30	33	3
Toy Shop	DS	31	34	3
Iron Man	PC	32	35	3
Incredible Hulk, The	PC	33	36	3
Ford Racing Off Road	PS2	37	40	3
Ford Racing Off Road	Wii	37	34	3
High School Musical 2: Work This Out!	DS	38	35	3
Rock Revolution	PS3	39	42	3
King of Clubs	PSP	40	43	3
Iron Chef America: Supreme Cuisine	Wii	42	45	3
Crash Time	Xbox 360	44	47	3
Garfield's Fun Fest	DS	44	47	3
Niki - Rock ‘n’ Ball	Wii	44	47	3
Gothic 3: Forsaken Gods	PC	45	48	3
G1 Jockey 4 2008	PS3	46	49	3
Operation Darkness	Xbox 360	46	49	3
Alone in the Dark	PS2	47	50	3
Legendary	PC	48	51	3
Art of Murder: FBI Confidential	PC	50	53	3
Battlezone	Xbox 360	50	47	3
Celebrity Sports Showdown	Wii	50	53	3
Happy Tree Friends False Alarm	Xbox 360	51	48	3
Spider-Man: Web of Shadows	PC	51	54	3
Great War Nations: The Spartans	PC	52	49	3
Brain Assist	DS	53	50	3
Disney Sing It	Xbox 360	53	56	3
Ninja Reflex	DS	54	51	3
Alone in the Dark	PC	55	58	3
Sonic Riders: Zero Gravity	PS2	56	59	3
Hell’s Kitchen	Wii	57	54	3
Madagascar: Escape 2 Africa	PS3	58	61	3
Need for Speed Undercover	PS3	59	62	3
Helix	Wii	60	63	3
Crusaders: Thy Kingdom Come	PC	61	64	3
Dokapon Kingdom	PS2	61	64	3
Bolt	PS3	62	65	3
Dinosaur King	DS	62	59	3
Madagascar: Escape 2 Africa	Wii	62	65	3
Madagascar: Escape 2 Africa	Xbox 360	62	65	3
Wits & Wagers	Xbox 360	63	60	3
SingStar ABBA	PS2	64	67	3
Hinterland	PC	65	68	3
Rhapsody: A Musical Adventure	DS	65	68	3
Spore Creatures	DS	65	62	3
Brain Challenge	DS	68	65	3
Crosswords DS	DS	69	66	3
Nostradamus: The Last Prophecy	PC	70	67	3
Puzzler Collection	DS	71	68	3
Strong Bad’s Cool Game for Attractive People Episode 3: Baddest of the Bands	Wii	79	76	3
Personal Trainer: Cooking	DS	81	78	3

Thursday, February 12th, 2009 2008 Scoring Analysis 1 Comment

Part 3: Metacritic Weighting

Honestly, I really enjoyed working on this report. The video provides some insight in regard to the impact of the weighting system. We’ll demonstrate how large the point swings can be, and even examine how the weights have changed between 2000 and 2008.

Get the Flash Player to see this player.

Tuesday, February 10th, 2009 Metacritic Uncovered 1 Comment

Almost There!

We are well into our scoring analysis for 2008. It looks like the first video report will be using non-weighted scores referenced with the metacritic favorability chart:

Metacritic Favorability Table

At the moment we have averaged all the media outlet scores and are currently referencing the results with the table above. If you have any particular requests for this round, please let me know.

–Tim

Tuesday, February 10th, 2009 2008 Scoring Analysis No Comments

About GameQuarry.com

Archive for February, 2009

Developers: Platforms & Consistent Peformers - Part Three

Developers: Genre - Part Two

Developers: Intro & ESRB Summary - Part One

An alternative scoring index for the videogame industry

Metacritic 2008 Genre Summary

Metacritic 2008 ESRB Summary

Metacritic 2008 Platform Summary

Game Titles with notable weighting swings (2008)

Part 3: Metacritic Weighting

Almost There!

Pages

Search this Site

Archive

Links

Meta