The irrelevance of awards

33 years ago, Chris Crawford explained why one should never take awards very seriously in The Journal of Computer Game Design, Volume 1, Number 5.

This is the time of year for annual awards.  The pundits gather their wits, their votes, and their courage, and they select the best products of 1987.  There’s nothing intrinsically wrong with the idea;  nowadays, so many products are released that an annual review of the best products provides needed perspective.  Sadly, though, the execution falls short of the ideal.

Consider the annual awards of MacUser magazine.  I noticed in their 1986 awards an apparent bias in favor of products that appeared late in the year.  I therefore compiled data on the products that won awards and compared it with the review dates of those products.  I then carried out a statistical test of the hypothessis that awards are given with no regard for release date.  The hypothesis was rejected at better than the .5% confidence level.  For you non-statisticians, it means that the awards were grossly biased in favor of recently released products.

Then there is the awards system sponsored by the Software Publisher’s Association.  The problem with the SPA awards arises from the fact that the software industry executives who vote for the awards don’t play many games themselves, and so they have no independent basis for making a decision.  Determined publishers will send free copies of their games to every elector to jog their memories.  It’s not quite the same thing as buying the election, but more than one publisher has mentioned to me the high cost of winning an SPA award.

I don’t mean to pick on MacUser or the SPA — they’re both solid organizations working to improve their systems.  I could have picked apart any of the awards systems.  These were the most convenient targets.  My point is that none of the various awards can stand up to close inspection.  

There is a fundamental reason why awards systems are such a mess:  there are just too many games coming out each year for any one person to make a fair determination.  Consider this:  each year, several dozen games companies release several hundred games to the marketplace.  It takes at least five hours of playtime to evaluate a game well enough to determine its worthiness for an award.  It would therefore require nearly a full-time worker just to play all the games.  Who’s got that kind of time to throw around?

The brutal truth is this: most of the people who cast votes for the “best game of the year” have played only a fraction of the hundreds of games out there.  Few of them have spent much time with the games for which they vote.  In short, it’s a crock.  Most of the electors for the Academy Awards have seen the movies for which they vote.  Most of the electors for book awards have read the books under consideration.  Is it too much to ask the same for games?  Apparently it is.

So what should we game designers do about it?  We certainly should not institute our own system of awards.  Let’s face it, we’re not much better equipped to select the best games of the year.  We probably spend more time playing games than most people, but I doubt that any of us are catholic enough in our game-playing to do justice to an annual award.

So I suppose the best we can do is behave graciously when we hear that we have received an award (or, more precisely, our publisher has received the award), thank everybody in sight, take the plaque or doodad home, mount it on the wall, and forget about it.  Lord help us all if we start taking these things seriously. 

That being said, the CGW games of the year were, for the most part, well worth playing. But then, as a former CGW contributor, I can testify that the editors, writers, and reviewers of CGW actually played the games all the way through.