The most popular research tweet on the Twitter #MRX hashtag in the past fortnight was Tom Ewing’s tweet of this screencap:
Cor, I can see why Apple don’t make a big thing about their market research – pages of grids, stone age stuff. #mrx pic.twitter.com/uV1rbPDTSq
— To Mewing! (@tomewing) May 1, 2014
Some folks on Twitter asked Tom what was wrong with this survey question. Here’s my take on some of the problems with this question, ranging from strategic mistakes to tactical missteps:
- Consumers often don’t know why they made a particular purchase – First, and most importantly, this question assumes respondents are conscious of the reasons they made the purchase. In reality, any purchase is as much an emotional decision as it is a rational one. The reasons we think we made a decision are often not the reasons we did (see “The influence of in-store music on wine selections“).
- The wording is not from the consumer point of view – The phrases “iOS 7″, “A6 chip”, “1080p HD video recording” and “LTE wireless” are not the sort of things you will hear in casual conversation. A good survey researcher speaks the language of the consumer, not the jargon of the engineering staff. Consumers are interested in benefits, not features: they want to be able to run the latest apps (the benefit of iOS 7), they want fast response (the benefit of the A6 chip), they want high resolution video (the 1080 pixel recording), and so on. I considered buying the iPhone 5c last month but didn’t because it didn’t feel “sturdy” enough – that’s the type of language consumers will use to think about their rationalizations. Apple should have run a pilot survey asking 100 consumers the open-ended question, “Why did you buy the iPhone 5c instead of a different phone?” and then developed a list of the most common reasons, in consumers’ actual words.
- The list of items seems incomplete – Perhaps the matrix was longer than the screen capture shows (heaven forbid!) but lots of common reasons why people might have chosen this particular device are missing: the Apple brand, price, being stuck in a contract with a carrier, wanting to continue to use existing Apple content (iTunes songs and shows, apps, etc.), and so on.
- Matrix questions produce less accurate results – A grid of questions shows less variance between items, as respondent think about each row less than if each item was an independent question. Plus, respondents hate them! – subjecting your own consumers to them is foolish. (See Grid-Cheating Panelists on Trial.) Given the lack of variance between results for items, whenever one of my clients forces me to use a matrix like this, I try to follow it up with, “What feature or attribute was the single most important to you in making your decision to purchase?”
- The 5-point bipolar scale is suboptimal – The most reliable bipolar scale (a scale with opposites, in this case, importance and unimportance) has 7 fully labeled points, but bipolar scales are more cognitively difficult for respondents. A 5-point unipolar scale would be more reliable: “Not at all important, slightly important, moderately important, very important, extremely important.”
To the anonymous Apple staffer who wrote this, please take a course in questionnaire design from the University of Georgia, Research Rockstar or Burke Institute. And don’t feel bad: these are common mistakes. (And I’ve made them all.)
And, one last item: Tom Ewing is an incredibly talented blogger. It’s ironic that his tweet about an Apple survey was more popular than any of his many great posts. You can check them out at Freaky Trigger.
Author Notes:
Jeffrey Henning
