Market Research: Big Business Does Not Necessarily Mean Being Right

trustrumMarketing2 Comments

A Dissection of the Wizards of the Coast Market Research Shows Us How Wrong Even Market Leaders Can Be

Wizards of the Coast Market Research Survey

WotC Logo is a Registered Trademark of Wizards of the Coast

Market research is something that many industries look towards to help guide them through their market in terms of product development and how best to interact with their customers.

When conducted properly and carefully, market research can grant extremely valuable insight into one’s market that allows a company to target its products better, increasing sales and customer satisfaction. This is why large corporations will conduct market research regarding something as seemingly simple and innocuous as a change in packaging or colour.

For the most part, however, the role-playing game industry is absent of any market research — especially of any of notable value — because it is a difficult thing to undertake properly, using a third party agency, and most small press publishers (who make up most of the industry) have no idea how to put it together on their own.

There is, however, one instance of market research in the role-playing game industry that people can look towards (if they must.)

Just when I think it’s going to disappear into obscurity (appropriately so), the much-cited, often lamented (by me, at least) Wizards of the Coast market research survey results of 2000 has once again shown up in a discussion I recently participated in. This time it was during a private talk between myself and another publisher while having a friendly talk about where the role-playing game industry has come from over the past decade or so, and the direction it is now moving towards. This conversation saw this other publisher citing the Wizards of the Coast market research survey (again, gah! DIE! DIE! DIE!) as a source about what the market used to be like.

Whenever I see the Wizards of the Coast market research survey results pop up in such conversations, I’m compelled to point out how unreliable this particular study is and then need to explain why. Frankly, I’m tired of having to type the same thing over and over so I’m going to summarize the issues here so they can merely be referenced in the future, especially since the previous place I dissected it upon (the old Misfit Studios message board) may be coming down soon.

First, let me state my qualifications for what I’m about to say. The market research industry was once my primary source of income. I worked in the industry, for various companies in various positions at various levels of involvement, for over a decade. Now, most people will say “that’s great and all, Steve, but Ryan Dancey, who was behind this market research, was president of Wizards of the Coast, a larger company, so obviously they know best.”

My response is a rather emphatic “no, quite the contrary.”

If there’s one thing working in market research teaches, it’s that executives, especially in larger companies, know pretty much zero about actual market research methodology. When they hand in a desired survey/research project to a market research company, the latter is inevitably going to point out frequent biases and discrepancies that are going to cause issues with proper data gathering and analysis, usually brought on by the fact that the company has its own pet theories and the like that slips into their project’s design, resulting in questions that, by form or content, shall almost certainly illicit biased responses that support and justify the pet theory rather than being designed with an unbiased, entirely open-ended purpose in mind (by intention or otherwise.)

Often, market research that is designed to reinforce a supposition rather than seek the truth is the result of the client’s ignorance of how to construct a proper research project, but sometimes the market research is actually meant to be biased so that those conducting it can have some data to present that supposedly backs up their opinions.

As I once again dissect the Wizards of the Coast market research survey, you’ll begin to see some of what I mean and why I think this particular survey was the result of either incompetence or the need for someone to have supporting “evidence” on hand should someone contest their opinion. Would Ryan Dancey do something like that? That’s for you to judge.

Keeping that in mind, I’ll state my observations are presented under the caveat that Wizards of the Cost states they’ve kept back some information, including the totality of their methodology, so I can only comment to the extent of what’s been released. That being said, let’s dive in.

The Wizards of the Coast Market Research Survey Dissection

You can find the Wizards of the Coast market research posted results here.

Adventure Game Industry Market Research Summary (RPGs) V1.0
Release Date: February 07, 2000

Summary prepared by: Ryan S. Dancey,
Vice President, Wizards of the Coast;
Brand Manager, Dungeons & Dragons

Permissions: This file is Copyright 2000, Wizards of the Coast. This file may be freely redistributed or quoted in whole or part, provided that this attribution remains intact.

Methodology: Wizards of the Coast regularly surveys various aspects of the adventure gaming channel; distributors, retailers and consumers to better understand their preferences, concerns, and needs. That data is regularly reviewed and distributed internally to senior management. The contents of this file are excerpts from those sources; the source materials themselves are confidential internal documents and are not available to the public. You have my assurances that to the best of my ability, the information presented in this document represents a fair and accurate representation of the data. Sources: The primary source is a market segmentation study conducted in the summer of 1999. No confidential information provided by non-Wizards companies was used in the preparation of this report.

Exclusions: The internal information gathered by Wizards is considered an important competitive advantage. Therefore, not all the information available to Wizards is incorporated in this document, and there may be areas where substantial, significant information is purposefully not included. An effort has been made to ensure that the absence of any portion of this confidential information would not render the material provided herein inaccurate or invalid.

Pokemon Effect: As this study was conducted just as the Pokemon TCG phenomenon was gathering speed. For this, and several internal reasons, I have elected not to present information on the TCG component of the industry
at this time.

So, here’s some legal jargon and the methodology statement. Moving on …

Updates: From time to time, I intend to revise and update this file to reflect our ongoing efforts to understand the industry. When an update occurs, the version number of the document will be changed, as will the “release date”. Interested parties can write to me at ryand@frpg.com to request an up to date copy of this document.

Although not related to my issues with the validity of the market research results presented, keep this part in mind when considering for yourself what parts to believe. This is an old survey that is often quoted by people to prove points in today’s market. Even if this was the ideal research project, conducted in the utmost perfection in 2000, the relevancy now has greatly deteriorated on a quarterly basis, let alone annually.

14 years is a LONG time when talking about the relevancy of ANY market research results. People need to stop talking about the Wizards of the Coast market research survey like it’s still relevant — as if it was ever accurate to begin with.

Also, to my knowledge, the posted market research results were never updated, as Ryan Dancey claims was his intention. Moving on …

Section 1: The Segmentation Study

Since so much of this data is derived from the ’99 Segmentation Study, it is important that the reader understand how this data was gathered.

For the purpose of the 1999 study, the following methodology was employed:

A two phase approach was used to determine information about trading card games (TCGs), role playing games (RPGs) and miniatures wargames (MWG) in the general US population between the ages of 12 and 35. For the rest of this document, this group is referred to as “the marketplace” or “the market”, or “the consumers”.

This age bracket was arbitrarily chosen on the basis of internal analysis regarding the probable target customers for the company’s products. We know for certain that there are lots of gamers older than 35, especially for games like Dungeons & Dragons; however, we wanted to keep the study to a manageable size and profile. Perhaps in a few years a more detailed study will be done of the entire population.

To be clear, choosing an age range for one’s study is common in research, but it is usually derived from more carefully defined data. The admission that it’s an arbitrary age range (wide though it may be) leaves a great potential for skewed data, especially in a market most publishers agree has an aging consumer base and suffers from difficulty attracting consumers in the lower (younger) age of the scale in comparable numbers.

It isn’t 100% certain this is going to be a problem, especially when one sees how problematic the rest of the Wizards of the Coast market research survey is, but it does open a wide door for possibly flawed results. The way the age range is selected on a guess cuts out a lot of people who could still be gaming after joining during the hobby’s early days, let alone older people who came to the hobby late and whose spending habits will vary from those of younger adults, etc.

Information from more than 65,000 people was gathered from a questionnaire sent to more than 20,000 households via a post card survey. This survey was used as a “screener” to create a general profile of the game playing population in the target age range, for the purposes of extrapolating trends to the general population.

There are several problems here.

1) Earlier Ryan Dancey stated the data is gathered from “distributors, retailers and consumers” … although the above statement seems to indicate the data presented here excludes the distributors and retailers — it’s not exactly clear. If the data is solely consumer based, then the statement regarding distributors and retailers is wholly irrelevant and confusing. If the data is indeed from mixed sources, then we’ve no idea how the “households” break down between retailers, distributors, and consumers. (Personally I believe it is the former issue and not the latter, but I can’t be 100% certain based on the way it’s presented.)

2) There are major issues to be found in gathering 65,000 data samples from 20,000 “households.” (First, a side comment: Although I’m sure these numbers are rounded off — normally a HUGE no-no when presenting an analysis like this — rather than being EXACTLY 65,000 and EXACTLY 20,000, they are the only numbers we have to go with, so I’ll speak in the same terms as the research without it really affecting what I have to say.) This means that, on average, 3.25 people were surveyed per household.

To accurately present a typical consumer, only one person per household should be sampled in most consumer research of this sort. There are exceptions, especially of the sort where certain group dynamics are relevant, but most of the data presented here is not of such nature. If the survey was asking about what goes on during a game session, for instance, asking 3.25 people in a house may prove relevant, as it’s likely those people game together.

Asking people in the same household to answer questions about buying habits as part of the same research creates flawed results because budgetary concerns and purchasing interests are more likely to overlap from one respondent to the next in cases of the respondents defining their purchase habits from a combined cash flow (such as a married couple with a shared bank account and budget, for example, or siblings or roommates who share books rather than each buying their own.)

To explain what I mean, consider this example:

Car Manufacturer A (CmA) wants to do research on who buys its new mini-van and why. To do so, it contacts households and asks them about their reasons for buying their particular mini-van (not necessarily the one that CmA manufactures) to get an idea of what people want.

If they call and get the wife, taking her information, they have completed one result and now know one data set for their project.

If they were then to ask to speak to the husband and asked him the same questions, they would then introduce a second data set into the results that should be identical to that which had been provided by the wife.

Because the results will be reduced to mere numbers for the sake of statistical analysis, it will appear as though there are two people, with no relationship, who own the same mini-van for the same reasons, etc. This weights the results towards households rather than general trends and individuals (and also allows the results to cater to pet theories by asking to speak to more people from households that present data that support preconceived notions), and misrepresents the sort of accuracy you’d get by limiting your results to one person per house.

Even if the husband and wife provide different answers regarding what was important to them in their decision to buy this particular mini-van, there is still skewing going on because the husband and wife will still share much of the same demographic information (e.g., where they live, when they bought the mini-van, the combined income of their household, what occupations are in the household, and so on.) This corrupts the results by weighing those demographics that are affected.

So, when looking at how the Wizards of the Coast market research survey accumulated its data, we see an instance of 3.25 results per house, which is worse than the husband / wife pairing in our mini-van example. Although it’s entirely possible (even likely) that those 3.25 people own different games (say, Bob likes Werewolf whereas his sister, Jane, doesn’t and only owns DnD 4e, which Bob hates), there is an incredibly noteworthy chance that purchasing trends will be affected.

Think of your own gaming group: if one of the other regulars owns all the splat books for your game of choice, the chances of your own need to buy them likely decreases. (This is supported by publishers reporting that splat book sales never reach the amount of sales shown by core books.) So, if each of the 3.25 people in each gaming household are looking towards the game collections of the other 2.25, it is safe to say that there are some decisions being made along the lines of “I don’t need to get that book because he/she already has it and I can easily borrow it from them.” Even if we give the benefit of the doubt and say this won’t happen often (which is generous considering all gamers know how often books get shared around in a group), even just a little of this activity corrupts the data.

If the results were gathered on the basis of one result per household, as is the normal practice in consumer market research, we wouldn’t see the data sets overlapping, nor would we see the issue of intermingling purchasing habits within a household affecting the gathered information. Instead, we’d have one person speaking on behalf of those 3.25 people.

This “screener” accurately represents the US population as a whole; it is a snapshot of the entire nation and is used to extrapolate trends from more focused surveys to the larger market.

How is it an accurate representation of the US population?

Aside from the whole “3.25 respondents per household” issue I’ve already brought up, how was the sampling weighted versus population densities and regions?

How many were from City/State A and how many were from City/State B, for instance?

How many are urban and how many are rural respondents?

How does the consumer sampling correlate to the concentration of consumer outlets for purchasing gaming materials? (In other words, how do we know how much of the data was taken from people with easy access to gaming materials and gaming groups versus those people who don’t have quick, easy access to the ability to purchase game materials or get together with fellow gamers?)

We already know that the respondents are improperly weighted to 3.25 respondents per household. This market research faux paus is further weighted by location as a result, so any flaws in the rest of the market research don’t just skew the data by an instance of 1 per error, as would be the case on a one person/household survey, but rather by an instance of 3.25 each time such a discrepancy enters the data because the location data the information was gathered from isn’t balanced.

This weighted effect every mistake made in the market research survey’s design, or in how a particular household answers it, is yet another reason why the data should have been gathered from one person per household.

Period.

I actually used to be in this situation when I first started Misfit Studios. I lived with two other guys in a house, and all three of us were gamers (they were both in my gaming group at the time, in fact.) Yet, I as always Gamemaster and I bought most of the books (as is often the case with the guy who gets stuck Gamemastering all the time.) If our house had participated in that survey (even though we were shy of their population sampling by .25), we would have been a shining example of how this sampling method is faulty. Our answers would have been heavily skewed by our household’s purchasing habits and how that data was then weighted across the sampling.

It’s also worth noting that I have a strong suspicion the “post cards” that were used to conduct the survey were those Wizards of the Coast inserted into many of their game books because, even as a pre-screening point of contact, such a method means there was absolutely no control over how the sample was initially gathered, by person (meaning there’s nothing to stop someone from sending in more than one post card) or region (meaning the post cards are merely accepted based on who bothers to send them in rather than a properly weighted representation coming in from various regions of the US.)

If such post cards are the mechanism used for this market research survey, it would also mean the general population wouldn’t be drawn from for the sample, but rather only Wizards of the Coast customers who purchased their products with said postcards in them.

There’s no actual proof this is where the market research survey’s data originated from, other than stating that “post cards” were used for the questionnaires, but I thought I’d throw this out as something you should consider on your own.

A follow up survey was completed by about a thousand respondents from the “screener”. The follow up survey is an extensive document with more than 100 questions. The particular individuals chosen to participate in this expanded survey represent the population, as determined by the screener. In other words, the small detailed survey group can be reasonably extrapolated to the larger screener group, and the larger screener group can be logically extrapolated to the public in general. This is a common, standard, and accepted methodology within the market research field.

Clarification: It’s a common and accepted methodology IF the data a secondary screener is drawing its respondents from can be considered valid to begin with. As I’ve pointed out, there were a lot of problems with the original 65,000 sampling, so the ability to say the secondary grouping of 1,000 respondents is an accurate representation of the US market is compromised to a great degree.

It’s also important to note that no information is provided on how the 65,000 sample was distilled to 1,000. Without that information, we have to fully trust in the fact that the process was done correctly. Considering my issues with the survey to this point, I’m not really willing to put my faith in that.

The data from the detailed survey was collated and prepared by the Wizards market Research Department, in conjunction with an external consulting firm.

We believe that the data is a fair and accurate representation of the hobby game consumer profile and that it does statistically correlate with the population as a whole in the US for the target age bracket.

As I stated earlier, in most instances a company’s internal market research department usually knows very little about how to conduct proper market research. Truly, it would make you weep to hear some of the experiences I’ve had over the years about how badly companies that are so big that their decisions radically affect global markets screw up their research and shift their business policies based on their results. So, that’s why companies that are solely dedicated to market research are brought in to lend a hand, but my confidence that the one hired for the Wizards of the Coast market research survey did its job properly is not at all high based on the errors regarding rudimentary methodology seen thus far.

Such internal market research departments are almost always used to prepare the market research preliminary screeners on the client company’s end, and the external market research company they hire then points out concerns, such as questions that will likely provide skewed results or will appear confusing to respondents (and thus likely result in flawed answers.)

Internal market research departments are also usually the ones who take the results provided by the external market research company and package them for internal analysis, use, and presentation within the client company. Internal market research departments don’t do the field work, and it is from such departments in the client company that errors or flaws in the research are most likely to be introduced either by their direct involvement or by how they demand the external market research company prepare and field their screener.

Defining the Terminology used in the Wizards of the Coast Market Research Survey

Section 2: Basic Terms

As a part of the detailed survey, the following terms and examples were provided to the respondents:

Term Example

(*)Paper RPGs Dungeons & Dragons
Card Games Bridge, Solitaire,
Uno, Poker
Trading Card Games Magic, Pokemon
Word/ knowledge Scrabble,
Trivial Pursuit
Puzzle computer games Tetris
Non-competitive problem solving Sim City, Myst
Puzzle table games Jenga, Dominoes
Class board games Chess, Monopoly, Go
Action/Shooter/Arcade Doom, Mortal Kombat
Miniatures table-top fantasy/sci-fi Warhammer
Games that use miniatures Battletech
War games Historical
Simulations Flight/car
Simulators
Strategy games Risk, Civilization
Social/party games Charades,
Pictionary
Strategic sport simulations Madden, MLB
Other non-sport games N/A

Specific questions were also designed to separate users of “computer Role Playing Games” vs. “paper Role Playing Games”.

(*) For my own purposes, I choose to use the term “Tabletop RPGs” in this document; the term “paper RPGs” was used in the study. The terms are synonyms; my choice is simply personal. I believe that in the fairly near future “paper” RPGs will hybridize with computer assistance – not becoming “computer RPGs” as that term is commonly understood, but not being games played simply with paper anymore either. Consider this a “forward looking” terminology.

Ryan Dancey’s irrelevant side commentary introduces doubt in the unbiased nature of the survey and its results. We’ve seen what happened of his prediction about hybridizing with computer assistance with the often-lamented Dungeons and Dragons 4e, illustrating that just because you predict something will happen doesn’t mean it will be successful at doing so.

Again we see an issue with Ryan Dancey’s preferences coming into play with the survey results. Is what Ryan Dancey prefers relevant to the survey, which should be a presentation of hard data?

Did this preference influence how the survey was designed?

Was it biased at all towards his belief regarding paper/computer hybrids?

We simply don’t know (although we’ll see how it affects his presentation of the results.) We can’t see the original survey so we have to hope that it didn’t and that all we’re seeing is hard data, but we just have to take this on faith.

In other words: Ryan Dancey should not be stating his personal opinions in this presentation market research survey results (especially when so much is being held back from the public, preventing independent verification or contrary analysis), market research data that is supposed to present material wholly independent of the opinions of him and every other person at Wizards of the Coast.

The term “D&D” is used herein to describe all flavors and types of D&D play; from old “white box” players up to people playtesting 3rd Edition.

Okay, here’s another point of concern. If playtesters of 3.0 were included in the dataset, there is the possibility of a brand bias being used because we don’t know if these playtesters just so happened to be contacted during random sampling, if Wizards of the Coast already had contact information on these people, or if Wizards of the Coast dropped a message in the playtest email list (or whatever) along the lines of “hey, we’re running a survey. Any of you guys and gals want to participate?”

Again, there’s no proof the latter happened but I am concerned with the possibilities brought up by the above statement.

So, why would this even be a problem?

Because it would mean that this survey isn’t based on a sampling of gamers at large. It would mean a sample skewing towards Wizards of the Coast’s own customers (granted this happens anyway considering their market share, but it certainly doesn’t need any more help in this regard) because it’s highly doubtful other publishers granted Wizards of the Coast equal access to its own playtesting groups so that Wizards of the Coast could make the same offer to them. In short, it would increase the chance of Wizards of the Coast customers carrying more weight in the final data gathered.

Also, think about this for the moment: is there anyone who plays any iterations of the Dungeons & Dragons game who can honestly say that their fellow Dungeons & Dragons players would agree they should all be grouped together collectively? That there are no notable differences — in play styles, purchasing habits, etc. — between Dungeons & Dragons fans? No differences of opinions that are so strongly felt that they lead to massive arguments and toxic feelings?

“Which version of D&D is best” is one of the most contested, re-occurring, and doomed-to-flaming discussions on any role-playing game Internet chat room, message board, or the like. The fact that Ryan Dancey and Wizards of the Coast grouped all such people together for the purpose of their market research survey may lead to some unreliable results (and by may, I actually mean will.)

Wizards of the Coast Market Research Survey Demographics

Section 3: Basic Demographics

The study provides the following information about the basic demographics of the tabletop RPG marketplace:

Size: 6% play or have played TRPGs (~ 5.5 million people)
3% play monthly (~ 2.25 million people)

Gender: 19% are female (monthly players)

Crossover: 17% of the total play MWGs monthly
46% of the total play computer RPGs monthly
26% of the total play TCGs monthly

The study provides the following information about the basic demographics of the computer RPG marketplace:

Size: 8% play or have played CRPGs (~7.3 million people)
5% play monthly (~4.5 million people)

Gender: 21% are female

Crossover: 33% of the total play tabletop RPGs monthly
21% of the total play TCGs monthly
13% of the total play MWGs monthly

The study provides the following information about the basic demographics of the MWG marketplace:

Size: 4% play or have played MWGs (~3.7 million people)
2% play monthly (~1.8 million people)

Gender: 21% are female

Crossover: 37% play tabletop RPGs
40% play computer RPGs
29% play TCGs

The age breakdown of players within the marketplace is:

Age TRPG MWG CRPG All Gamers(*)
12-15 23% 27% 23% 11%
16-18 18% 17% 16% 7%
19-24 25% 24% 23% 13%
25-35 34% 32% 37% 29%

(*) “All Gamers” means people in the study population who reported playing >any< of the game types monthly, not just TCGs, RPGs, MWGs or CRPGs.

And here’s where we see my previously mentioned issues with sampling 3.25 people per household rearing their heads.

If those 3.25 people are all playing tabletop games together, for instance, but don’t play CCGs because one of them refuses to play anything even resembling Pokemon (which is where I stand), that skews the data towards tabletop gamers and away from CCGs rather than showing a more “pure” result.

The 2000 US census also shows 50.9% of the American population is comprised of women, but we have no idea how many of the original 65,000 — or final 1,000 — respondents are women. To be an “accurate snapshot” of the American market, 50.9% of the respondents in both groups would need to be women. The way the data is presented indicates that, most likely, this was not so and results by gender were “gathered as they fall,” meaning information was taken as it came in and looked at afterwards, pointing at the incidents of response by gender as indicators of how the market broke down by gender.

This is problematic because a study looking to see what percent of the market is female should have been done separately from one looking to examine the overall market’s purchasing trends. Looking to find a gender demographic breakdown within a market should employ a “gathered as they fall,” open methodology using weighted, random sampling whereas examining market trends should use weighted gender requirements (meaning that, if a previous market research survey showed 21% of the market was female then 21% of the answers should be from females, even if they are harder to find and male respondents need to be turned away if data is already completed for their 79% of the research.)

Methodologies necessitated by these two results are almost certainly in conflict if the results are to be considered reliable, and so combining them into one study is almost certainly going to provide a self-fulfilling result.

The most obvious way I could see the Wizards of the Coast market research survey getting around this conflict is using the 65,000 original sampling to find out how much of the market is of a particular gender to determine how many males and females should be included in the second group of 1,000. This is, again, affected by the problems already mentioned regarding how the sampling was undertaken, along with the fact that this would be a guess on my part because Ryan Dancey doesn’t differentiate between the information gathered from the original 65,000 and that gathered from the 1,000 to follow.

Yet again, all we can do is hope for the best and trust that Wizards of the Coast but the trend of market research clients everywhere by refusing to stack the deck.

*cough*

Wizards of the Coast Market Research Survey Conclusions

Here is where we begin to see some data and what Ryan Dancey thinks it means.

Conclusions:

1. Few “General Gamers”:
The first, most notable conclusion we can draw from this information is that the mythical “hobby gamer” who plays TRPGs, CRPGs, MWGs and TCGs comprises a very, very small portion of the total market. A minority of gamers play more than one category of hobby game; very few play all three. The largest overlap, though still a minority, is with CRPGs and TRPGs.

Let’s return to my previous concerns regarding the hypothetical Bob who, along with the other 1.25 people in his household, likes TRPGs, CRPGs, MWGs, and TCGs, but his sister Jane only likes CRPGs and TRPGs and so, collectively as a group, this is all they can agree on to play together. We see in this example how sampling more than one person per household influences the results because, despite contrary interests, Bob and Jane would likely answer that they play CRPGs and TRPGs, and not MWGs and TCGs because that’s all the 3.25 of them can agree on as a group brought together by circumstances, even though the resulting data will portray them as individuals without relations of any kind expressing the merit of their personal preferences.

It would be different if the information stated what these people BOUGHT or were INTERESTED in, as we know gamers also collect game materials they may not necessarily get the chance to play but still find interest in, but note how the data states what people PLAY. If the question was phrased the same, some people may assume that “play” or “have played” really meant “what do you LIKE” while others took it literally and excluded their interests and restricted their answers to what they actively played at that time.

Adding a reasonable time frame would be a good idea, as well — say, along the lines of “which games do you play now or have played in the past 5 years?” For example, if someone asks me “Steve, what games do you play?” I’d likely think in a context of recently (or even currently), and would exclude games I own but haven’t touched in a few years, even if the completionist in me is still buying products in those game lines. (I haven’t played Deathwatch in years, for instance, but still get new product because I love to read it.)

I don’t imagine I’d be the only person who would think this way if asked this question, whereas other people would think in a context of what games they’ve played EVER or what games they haven’t played but would like to. I mean, I played Red Box DnD in Grade 5, so yeah, technically I’ve previously played it, but I haven’t owned it or played it in going on 30 years, so why would it be relevant regarding my current playing or purchasing habits?

Considering the wide possibility of gamer interpretation of such a question, a time frame or similar context would be necessary for clarity’s sake.

This is a good example of what I was talking about when I mentioned the sort of thing a market research company should catch and point out to a client because the language the client uses within it’s own industry (Wizards of the Coast may commonly use the word “play” as a synonym for “interest” around the office, for example, which isn’t necessarily true for the people surveyed) isn’t necessarily the same language/turn of phrase the respondent is going to view it in.

In either scenario — a)the issue with responses from 3.25 people/household representing collective play habits rather than that of individuals versus b) confusion over use of the term “play” — there are problems present that can affect the data. The chance of the respondents navigating these two issues on their own (considering the don’t realize the problems even exist) to result in data without bias or errors is incredibly unlikely, to say the least.

This is an exciting conclusion, because it indicates that a company can successfully create brand in one of the three hobby categories, and extend that brand into the other two without significantly cannibalizing sales. In other words, the people who buy the RPG are not likely to be the ones buying the MWG or the TCG.

And here’s where we see the sort of conclusion a client company will draw from market research results.

Considering all the problems brought up at this point, I hope you see how it can be bad for a company to make conclusions like the above based on data that has problems with how it was gathered. Sure, there’s a chance the conclusion may still have some validity, but that doesn’t mean it’s necessarily so. As they say, even a broken clock is right twice a day.

2. There are “Women in Gaming”
Second, it is clear that female gamers constitute a significant portion of the hobby gaming audience; essentially a fifth of the total market. This represents a total population of several million active female hobby gamers.

However, females, as a group, spend less than males on the hobby.

I don’t think anyone disputes there are women in gaming. I think most people would be shocked to find out that 1 in 5 gamers are women, however. As I stated before, this market research data is likely flawed based on equally flawed data gathering methods regarding defining gender demographics for the purpose of accurate market research analysis and conclusions.

3. Adventure Gaming is an adult hobby
More than half the market for hobby games is older than 19. There is a substantial “dip” in incidence of play from 16-18. This lends credence to the theory that most people are introduced to hobby gaming before high-school and play quite a bit, then leave the hobby until they reach college, and during college they return to the hobby in significant numbers.

There are BIG problems with this part of the study.

The sampling was, by the indication provided by the Wizards of the Coast market research survey, arbitrarily taken from people ages 12 to 35. We don’t know if the 65,000/1,000 respondents were weighted equally across this age range or selected at random. We need to know how the sample was broken down by age to believe any of these results, especially if we don’t know if the age and gender results were using cross-referencing requirements (meaning their 21% females were used to ensure each age group was comprised of 21% females.)

Here’s another example to see why this is so important: Let’s say we’re looking into a parking lot of 100 automobiles, broken down into groups of 10 cars of 10 different colors, one of which is red. There is no color with more cars than any of the other nine colors, correct? But we can look at those cars and form the conclusion:

90% of the cars in the lot are not red, so clearly red is not a popular color.”

The first part of the sentence is objectively true — it’s a statistical fact — but the latter part is a subjective statement that ignores the remaining data that was not presented as part of this market research (meaning it ignores the fact that all the other cars of different colors can just as easily and factually replace “red” in the above statement.)

We don’t know how the colors were chosen, nor do we know if the 10 cars/10 colors ratio developed on its own or was artificially enforced (meaning an 11th car of any one color was turned away so there will always be 10 cars each of 10 different colors.) We don’t know, so we can’t really relate what we’re seeing in this parking lot to the overall consumer market for cars and the colors people choose to buy them in.

The same goes for the age breakdown in this market research. Clearly, when talking about a market research sampling of people aged 12 to 35-year, the group of 19 to 35 is a much bigger range than the 12 to 18 group. So, if the market research sample was divided evenly amongst the ages rather than using the “gathered as it falls” method, there’s obviously a chance for results that skew towards the larger age bracket.

I also have a problem with employing opinions about “going to college.” How do we know they’re going to college and not just joining the job market, and thus gaining more liquid income?

Although the results indicate many of the respondents have post secondary educations, we again don’t know if this is a natural result found during random sampling or is the result of flawed sampling methods (such as more answers coming in from college towns than not, and not being filtered by qualification demographics, for instance.)

It may also indicate that the existing group of players is aging and not being refreshed by younger players at the same rate as in previous years.

A statement made in the absence of any data regarding “previous years.”

Section 4: The Role of Computers

There is an intense, ongoing discussion between publishers and customers about the use of computers and the interaction between computer game play and adventure game play. The market research study presented some revealing insights into this ongoing debate.

Internet Gaming: 51% of the TRPG players report that they have ever played a game on the internet. 28% report that they play an internet game monthly.

% Who want to buy software to help manage game and speed up combat: 52%
% Who want to play D&D over the internet with others: 50%
% Who read newsgroups, mailing lists and web sites: 37%
% Who currently play with computer assistance: 42%

What computer do gamers use?
Wintel Platform: 63%
Macintosh Platform: 9%

(The question was essentially “What platform have you used in the last month”, and “none” was an option, probably accounting for the missing percentage.)

What’s sitting at home?
Wintel Platform: 54%
Macintosh Platform: 7%

Three quarters of the sample use the Internet at least once a week, but only two thirds have access from home.

“Who plays electronic games?”
Computer Console/Handheld Both
Average Age: 26 23 20
Education
% 6th-8th: 5 20 27
% 9th-12th: 23 52 37
% College: 53 26 31
% Post Grad: 20 2 5
Marital Status
% Single: 52 65 76
% Partnered: 46 29 22

Games electronic gamers play monthly:
Computer Console/Handheld Both
% TRPGs: 72 54 57
% CRPGs: 44 21 50
% Puzzle Comp: 39 41 49
% Classic Board: 39 48 44
% Action/Shooter: 32 55 61
% Simulations: 25 36 40
% Strategy Games: 26 26 32

One conclusion we draw from this data is that people who play electronic games still find time to play TRPGs; it appears that these two pursuits are “complementary” or “noncompetitive” outside the scope of the macroeconomic “disposable income” competition.

People really need to stop referencing the information on technology presented in the Wizards of the Coast market research survey.

This data suffers from many of the problems I’ve already mentioned. I will comment on this part because it’s one that is often quoted in discussions regarding this survey’s relevancy.

The computer industry, especially with regards to role-playing games, has changed dramatically since 2000. Indeed, anything to do with electronics and computers has changed dramatically since then, especially with the advent of iPods, cheaper and more portable laptops, and and then tablets. Although it’s possible these numbers haven’t changed, it’s incredibly unlikely, especially when one looks at the proliferation of online role-playing games and online gaming communities.

The Big Section: Tabletop Role-Playing Game Market Research Data

Section 5: Tabletop RPG Business

We asked questions of people who play TRPGs to get a better and more detailed picture of that category. This section explores some of that data.

The market research study provides some useful information on the games TRPG players play when they’re not role playing:

51% play a non-TCG card game monthly
43% play a puzzle computer game monthly
43% play a classic board game monthly
58% play an “action/shooter” computer game monthly
41% play a “simulation” computer game monthly

The >least< played game types were:
26% play a TCG monthly
24% play a puzzle table game monthly
17% play a MWG monthly
17% play a social/party game monthly

When asked how likely a person was to be the DM/GM, the responses were:
2+ Sessions as DM/GM: 47%
Don’t DM/GM: 41%

When asked to describe a variety of past game experiences, the market provided the following data:
Question: Result

Used detailed tables & charts: 76%
Included Miniatures: 56%
Used “rules light” system: 58%
Diceless: 33%
Combat Oriented: 86% (*)
Live Action: 49%
House Rules: 80%
(*) Looked at in reverse, this interesting answer tells us that 14% of the gamers who play an RPG >have never played< a combat oriented RPG.

Again, aside from previously mentioned problems we need to keep in mind there are a lot of differences between 2000 and 2014, especially when one considers the changes brought on by the introduction of the Open Gaming License that remain today, nearly 15 years later, something that caused massive changes throughout the entire industry and market.

Of the people who reported playing a TRPG, we further screened for people who played D&D and asked those individuals some more detailed questions.

This data comes from people who have played D&D, not necessarily those who play monthly.

Here is an obvious problem. It shouldn’t surprise anyone that Wizards of the Coast is looking out for its own branding (who wouldn’t in their own market research survey?), but combining such an interest in a market research survey meant to also find general information about the market will run into trouble.

For instance, because the Wizards of the Coast market research survey also wants to gather info on DnD, how do we know they didn’t say something like “500 of our 1,000 people in the second group need to play DnD” when designing their market research because they wanted some brand focus, as is normal? This would mean people with otherwise valid and relevant data were rejected from the final 1,000 — which was also used for general industry results — on the basis of them wanting to get DnD specific data for the introduction of DnD 3.0 and the Open Game License.

This would mean the data gathered from the sampling wouldn’t be “people who game in general,” but rather largely “DnD gamers who may also play other games.”

The distinction is incredibly important.

Just how and where the lines of bias are drawn cannot be seen, considering the lack of clarification in presentation of the Wizards of the Coast market research survey results, but the fact that ANY branding-specific data gathering was combined with general data gathering , by definition, a bias to any data gathered that leans towards Wizards of the Coast, even if we consider the obvious existing Wizards of the Coast branding presence in the market at large.

Age: <12 12-15 16-18 19-24 25-25
Learned D&D: 23% 41% 15% 12% 9%

————–

One conclusion we drew from the data was that if a player had played longer than one year, the chances they would play another year were greater than if they had not yet been playing for a full year. In fact, the longer a person plays, the higher the chance they will stay in the game; in other words, players are >less< likely to quit playing D&D the longer they play, not >more< likely.

<=1 Year >1-5 Years >5 Years
Expect another Year: 40% 75% 88%

We asked what the frequency of play was:
Total D&D <=1 Year >1-5 Years >5 Years
Monthly: 7.2 4.9 13.2 5.9

So we see that the longer a player is in the game, the fewer times per month they play after the 5th year. Once the “acquisition” period (1st year) has passed, frequency of play accelerates tremendously, then drops. One explanation for this fact may be that since acquisition happens most often at age 15 or less, “new players” may have a lot of time available for gaming, but as they age, they have less time per month to play.

We looked at a few other questions based on how long a person had been playing the game:

[ if this chart gets mangled in the formatting, it has three columns of data ]
Typical 4 or More Average Sessions
Session Gamers In before Restart
5+ Hours Group (New Characters)
Total 28% 62% 15.4
<=1 Year 10% 48% 8.8
>1-5 Years 14% 60% 12.9
(*)>5 Years 42% 71% 19.6

(*) Remember that frequency of play is down sharply for these gamers)

This data tells us that the longer a person plays the game, the longer the game sessions get, the more people play in the game, and the longer the game progresses before a character restart. In fact, if you look at the >5 year group, you realize that the big jump in long sessions and in average sessions before a restart means that the 5+ year gamers are playing the same characters, on average, vastly longer than anyone else.

One conclusion might be that it takes 5 years for a player to really master the system and really figure out what kind of character that player likes to play.

All the previously mentioned problems aside, many people quoting the Wizards of the Coast market research survey usually don’t notice that it’s DnD oriented and that it presents its age-specific trends as industry-wide trends. Just because DnD may appeal to this or that age group doesn’t meant the same can be said of other games, which are likely to employ different themes, genres, and game mechanics. This is a flaw of the reader, however, and not Wizards of the Coast (well, beyond concerns already discussed), as the latter has clearly stated they’re talking about DnD.

The following financial figures are for TRPG players in general (D&D information, where available, is provided as well)

This data seems to validate the theory that young gamers, while very active, don’t spend a lot of money. (The following data is reported by for RPG expenditures) The big dollars come from adults…

Total spending by age:
12-17: $297
18-24: $850
25-25: $2,213

And, the longer they stay in the category, the greater their total outlays…
Play <=1 Year: $116
Play 1-5 Years: $562
Play >5 Years: $2,502

And if they can be induced to become a DM/GM, expenditures skyrocket.
Will DM/GM: $2,048
Will not DM/GM: $401

Some breakouts for the D&D population in particular…

Total D&D spending by age:
12-17: $164
18-24: $443
25-35: $1,642

Monthly D&D spending by age:
12-17: $10
18-24: $12
25-35: $14

Total D&D spending by time in game:
<=1 Year: $123
1-5 Years: $338
>5 Years: 1,756

Monthly D&D spending by time in game:
<=1 Year: $7
1-5 Years: $22
5 Years: $16

(Interesting note: Monthly spending in the first five years after adoption of the game is higher than the spending beyond that point – though the older, longer gamer plays the game more, they spend less. This may relate to the frequency of a character/game restart.)

D&D DM willingness effect on expenditures:
Will DM: $1,444 total / $21 monthly
Will not DM: $187 total / $7 monthly

(Interesting note here: Even people who don’t DM buy a heck of a lot more than just a PHB…)

Effect of miniatures addition to RPG mix:

Few miniatures owned/used: $139 total RPG spending

Many minis owned/used: $4,413 total RPG spending

This is a good example of how even information that has been gathered improperly has the possibility of being right, usually because these particular trends being examined in the Wizards of the Coast market research survey are so pervasive that they are going to stand out no matter how the sampling was done. For instance, it shouldn’t surprise anyone that the DM, being the person responsible for preparing and running the game, spends more money on game material than those people who refuse to DM (and thus only need to buy what is needed for their specific characters.)

As an aside, this section contains another market research no-no. “DM” is a term that is very tightly branded to DnD (indeed, it is a trademark of Wizards of the Coast.) As such, it shouldn’t be used in an aspect of the Wizards of the Coast market research survey that is meant to gather general industry section that. Doing so will almost certainly subconsciously bias responses due to this branding association and legal distinction that most consumers in the market is aware of.

The general term, “Gamemaster,” would have been far more appropriate. It may seem a minor thing, but it really can cause problems with data gathering for general market research purposes because it compromises any sense of neutrality.

This section also demonstrates why properly represented market research doesn’t mix it’s data on various points in the same section. The separation between DnD and general tabletop RPGs bleed into each other, which has resulted in confusion and people citing it incorrectly.

We found that players who were ‘lapsed’ – reported that they had played TRPGs but were not currently doing so; had spent more money than the current players, and had played more different games monthly – but interestingly, they had spent less money, on average, on D&D than players who were “current”.

(Current/Lapsed)
Mean RPG Spending Mean Total D&D Number
Spending RPGs Played
$1,273 / $1,667 $895 / $599 2.2 / 3.3

How does the information regarding spending by age cross over with the spending trends by current and lapsed gamers?

This is an incredibly important thing not to have pointed out because where one is in one’s life, and thus the time and money to spend on games, is highly tied to one’s age and career. It also makes the information regarding current/lapsed gamers rather difficult to do anything about because one can’t track the trends regarding customer entry and exiting of the market.

(The Wizards of the Coast market research survey isn’t really doing anything wrong here, but the picture being presented is incredibly general and not as useful as it could be. I’d like to think this is because Wizards of the Coast saw value in retaining this information for themselves alone and not because they didn’t thin to analyze it in the first place.)

One conclusion that could be drawn from this data is that gamers who don’t like D&D will spend a lot of money and try a lot of systems to find something they do like before they quit. Gamers who like D&D will spend less money and try fewer systems, but will spend more on D&D than those who don’t.

Point of interest: this is a conclusion we can draw only if we accept there were no flaws in how the data was gathered, and that the use of the survey to gather both general gaming info and DnD-specific info doesn’t create a bias. This conclusion can only be trusted if it is a natural result of the general data (such as asking “what games do you like?” and finding lots of DnD results) rather than being influenced in any way by the inclusion of DnD related questions that intermingle with general gaming questions.

When asked why a gamer lapsed, the answers (multiple choices allowed) were:

Got too busy with other things: 79%
Too few people to play with: 63%
Not enough time to play: 55%
Found a game I liked better: 38%
Unhappy with the game and the rules: 38%
Cost too much money: 32%
Burnt out from frequent play: 29%

Getting back to the people still playing the games, when asked what games TRPG players play monthly, the answers (multiple choices allowed) were:

D&D: 66%
Vampire: The Masquerade: 25%
Star Wars: 21%
Palladium: 16%
Werewolf: The Apocalypse: 15%
Shadowrun: 15%
Star Trek: 12%
Call of Cthulu: 8%
Legend of the Five Rings: 8%
Deadlands: 5%
Alternity: 4%
GURPS: 3%

Another instance of a breakdown by age would be incredibly relevant. A lack of an “other” option is also important to note. Anyway, on to more direct observations:

Issue #1: Palladium is a publisher, not a game, so which Palladium game?

Issue #2: Which version of Star Trek, considering the license has been owned by more than one company?

Issue #3: considering the % total is higher than 100%, we know this question allowed more than one answer per respondent, so why then was “other” not an option? Obviously there are more games out there than just those presented, so we’ve no idea if it’s simply a matter of an “other” option being available but nobody filling it out (HIGHLY unlikely) or the Wizards of the Coast market research survey biasing the data by only providing the above options as possible answers.

So, the confusion between publishers and games, along with the restrictions of limiting the possible answers, makes this information entirely worthless other than to see what we already know: regardless of how badly the data is messed up, we can still see that DnD was the industry’s major player in 1999/2000 (and who really needed this flawed question to tell us that?)

When asked to describe aspects of their games, on a scale from 1 to 5, answers were:   (normally/rarely)

Create Own Adventures: 42% / 11%
Create Own Campaign Material: 29% / 17%
Replay Adventures: 18% / 35%
Use adventures from magazines: 21% / 40%
Follow official D&D Rules 33% / 17%

Confusing point: Is this portion of the Wizards of the Coast market research survey still talking about the market in general or, by nature of the last line of data, is it talking about DnD specifically? If the former, the inclusion of DnD creates a bias in the general data. If the latter, we’ve no idea if this is true of games other than DnD, games that may be designed in such a way that engenders them to entirely different trends than those mentioned above.

When we asked RPG purchasers how many had purchased D&D at a particular retail type, the answers were:

(*)Hobby/game shops: 36%
Book Stores: 27%
Comic book stores: 18%
Specialty toy and game: 17%
Large toy store chains: 15%
Conventions: 4%

Again, there’s a noteworthy absence of “other.” What about gifts? What about such things as department stores, online purchasing, and mail order? The latter two are especially more relevant now than in 2000, which again shows us how dated this market research survey is, flawed or not.

We also have to note that this section of the market research survey largely only speaks to DnD purchasing habits and doesn’t address gaming in general, as many publishers don’t have the distribution resources to sell in places like comic book or toy stores, go to conventions, or deal with the hassle that is the book trade.

My Final Notes on the Wizards of the Coast Market Research Survey

Many of my issues with this market research survey and the presented results and conclusions of Ryan Dancey are born from the highly suspect methods used to sample the general populace in order to find how many of them are gamers, which then leads to how those gamers are further distilled by household, region, age, and gender, which is further corrupted by the inclusion of specific branding issues.

I think most of these problems in the market research survey were caused by sloppy methodology necessitated by WotC trying to cast its net too wide when looking for results they had their sights set upon. Even using better pre-screening methods, there are at least two separate surveys here (general gaming versus DnD branding), although both would have been better served by better preparation.

And, it needs to be restated, even though some of these issues would still exist otherwise, some of my concerns with the presented market research survey are obviously rooted in a lack of explanation and extrapolation on WotC’s part due to their concerns regarding keeping some of the more valuable data to themselves.

It certainly doesn’t help that Ryan Dancey interjects his opinions and provides his preferred conclusions throughout the market research survey presentation rather than just saying “here’s the data, draw your own conclusions.” By providing his own opinion and listing just one conclusion instead of various possibilities, he’s influencing how third parties will look at the market research survey data, especially if those third parties aren’t themselves experienced researchers and are able to see where Ryan Dancey was doing wrong

It’s also worth noting that in the 14 years since the Wizards of the Coast market research survey results were published and Ryan Dancey commented on them, Ryan Dancey’s career has pursued a trajectory that seems to have tried to fulfill his predictions, but those predictions never manifested for the industry at large. The failure of Dungeons & Dragons 4e to meld online tools with tabletop games showed that.

And lastly, let me close by reiterating: the information gathered by the Wizards of the Coast market research survey was presented in 2000 and gathered in 1999, so stop citing it as at all relevant to the modern market!

2 Comments on “Market Research: Big Business Does Not Necessarily Mean Being Right”

  1. Kate

    Well, do you know about any other survey which was conducted and its data are relevant to the modern market.??

    1. Steven Trustrum

      Market research is generally too expensive for most RPG publishers, and what is done amongst larger publishers (e.g., Hasbro) is not typically released to the public. Most research that anyone conducts will be along the lines of posting something on one’s message boards along the lines of “hey, customers, we’ve got this short survey we’d like you to ask” that they put together themselves rather than using a professional market research company.

Leave a Reply

Your email address will not be published. Required fields are marked *