An alternative scoring index for the videogame industry

By Tim Sweezey
GameQuarry.com

Whether a Videogame publisher, developer, investor, journalist, or seasoned gamer, all recognize Metacritic as the defacto industry scoring index.  Nobody has to like it, or love it, as frankly, it can’t possibly be all things to everyone.  Everybody wants and needs a scoring index that reflects their own interests and needs.  And there lies the rub.

For those who are critical of Metacritic it’s often a reflection of “Mashups” in general, i.e. an almagamation of average scores from a plethora of media sources.   Sometimes games are scored on a scales ranging from 1-4, 1-10, and 1-100, or simple letter grades.   Add to that, some media outlets consider a “5″ an average score on a scale of 1-10, while others adopt a more common gradeschool approach where “7″ or “70″ or “C” represents an average score.   The bottom line is that there are enough moving parts in a mashup site for anyone to find fault. 

What you probably dont’ know about Metacritic and Mashups

In most cases, it makes little difference which index you use.  If you took the two most popular mashup sites in the game industry, Metacritic.com and Gamerankings.com, the average difference for any given game title is only 1.2 points.  And that minimal point spread is entirely relative.  For example, Metacritic may convert letter grades differently than others, and they do apply their own weight system giving media outlets different importance ranks, but since the same rules apply to each game, it’s all equitable.  In general, so long as a mashup site uses about 20 or so sources, all these scoring nuances mean little.  

Obviously, one of the reasons Metacritic is such a target for critique is because it is the standard.  Everyone in the industry speaks the same language when comparing quality as they all use the same index.  Don’t fault Metacritic for being the standard, as they never offered themselves up for the role.  Their focus is providing review indexes for consumers whether it’s videogames, movies, music, or books, which they do very well.  They know the role they play within the industry, yet their methods aren’t influenced by that, and frankly, I find that commendable.

The truth is that no existing game index is appropriate for the videogame industry.  The problem is that within the industry, there are different applications for these indexes, that no one mashup can currently accomodate. 

The Public Relations and Marketing Index

If you are in Public Relations or Marketing, Metacritics weighting system may be an attractive concept because in general, it recognizes that different media outlets have a different importance.  The problem is, those in PR or Marketing may not agree with the media outlets selected for the sample, or perhaps have issues with the manner in which the weights are applied. 

Think about it from a publicists perspective.  There are, and always will be, media outlets with a larger reach or higher relevancy than others, so reflecting a weight is entirely appropriate.   Consider for a moment that you are working on a game title that doesn’t appeal to a hard core gamers, but still has mass appeal to a more casual audience.  Weighting a consumer outlet such as the “New York Times” may be more appropriate than weighting a game-centric trade media outlet.  Conversely, if you represent a multiplayer shooter, you want your weights applied to the game-centric trade outlets and not necessarily for those that are consumer oriented.

The Developers Index

Deveopers may need something even different.  Assume for a moment that your bonus or performance review is at least somewhat dedicated to the quality of the games you produce.  So, what is the quality indicator here?  You could use a mashup index score to determine quality, but that might not be appropriate.  In this case, we want consensus, which has zero room for any weight applications.  What if you created a game with with an average score of 80, yet the Metacritic score was only 76 because of the weights applied?  You may be penalized, at least in part, by some arbitrary weight that your management couldn’t begin to explain.  At the same time, that weight could work to your favor as well.  But then, maybe your company gave bonuses when it wasn’t really appropriate.

The Forecasting Index

Then you need another index for sales or forecasting.  That’s right, if you have a game that you feel is going to be every bit as worthy of a ‘80′ index score and you want to use this as part of your forecast formula, then you need an appropriate index for the task.  What you’ll likely need is to forecast on a consumer vs. game-centric media outlet index, as well as one that gives consideration to region/territory.  Gamer-centric trade publications may be about to salivate over the new multiplayer shooter, but there are some countries where single player games are preferred simply because of the infrastructure available to them.  In those cases, a heavier weight may apply to single player vs. multiplayer.

The New Index

The videogame industry would be well served to adopt a standard tiered scored index.   They can start by coming up with a standard list of media outlets for not just North America, but an International list.   Yes, just like Metacritic, although much more complete.   In fact, they can learn a great deal from Metacritic.  This media list isn’t nearly as daunting as it sounds, as every major publisher could have their preference of outlets reflected.  I guarantee, any PR or Marketing team would be happy to provide their list.  When it comes to scores, the more the merrier as the data sample just gains integrity.

From there, the segmentation of the various sub-indexes for PR and Marketing, Sales, Development, really don’t require consensus.   It’s easy to tell which outlets are consumer vs. game-centric, and which are North American outlets vs. International.  That just leaves the weighting system to deal with.  In the end, what you end of up with is something that looks like this:

• Master Index (all outlets non-weighted)
• Regional Consumer Indexes (Weighted)
• Regional Game Trade Indexes (Weighted)

So long is the method is entirely public and transparent, everyone should be happy.  The overall quality index for the development team is based on the master index, and sales, marketing, and PR can use their respective regional indexes.  In addition, the benefit expands to the consumer as seasoned gamers can better relate on the Game Trade Index, while the more casual gamers may find the Consumer Index more appropriate.

Finally, if your business model and/or compensation structure is based, even in part, on a set of data, you better make sure it’s appropriate data.  Using consumer based mashups for PR, Marketing, Sales, and Development, is like like trying to hammer a nail in the wall with stapler.   You might be able to hang that picture, but it’s going to come at a price.

This article is free to republish or reprint as long as the credit is given to gamequarry.com and the respective author.Tim Sweezey is the Vice President of Artisans Media Group, and the Senior Scoring Analyst for GameQuarry.com. Since having worked for Ziff-Davis Publishing as a Director of Information Technology, the Senior Technology Manager at both ZDTV and TechTV, Tim has been active within the Videogame Industry since 2003 specializing in Public Relations Technology Solutions and Analysis.