Blog



Issues with Grid Questions

Grid or matrix questions are a concise technique for visually representing a series of items to be rated on a common scale.



Unfortunately, grid questions raise a number of issues:

  • Most importantly, participants dislike them, and they lead to poorer quality data. Only 72% said “I liked this survey” for a questionnaire using a traditional grid, vs. 80% for the same questions administered in an item-by-item format (though showing multiple items per page). The difference is starker because 6% dropped out of the traditional grid, vs. 1% for the item-by-item format. The traditional grid also had the worst correlation between opposing statements, -0.342 vs. 0.407. (Source: “Grappling with Grids” by Jené Grandmont, Lynn Goetzinger, Bob Graff, and Kara Dorbecker.)
  • Participants can be prone to “straightline” their answers to matrix questions, selecting the same choice for each row, in a virtual straight line down the grid.
  • In an SSI case study, participants completed the grid almost 20 times faster than the battery of regular questions: they took half a second per item in a grid but 8 to 10 seconds per item-by-item question. Faster speed of completion may lead to errors.
  • Interestingly, Vidal Diaz de Rada and Juan Antonio Domínguez found that participants completing grids in mail surveys showed greater acqueisence, more extreme response categories, and more rows left unanswered (“The quality of responses to grid questions as used in web questionnaires”, 2013).
  • According to research conducted by Knowledge Networks, participants may become confused about which item they are rating: “It seems that on a grid it is much easier to ‘miss’ the meaning of the items, resulting in more inconsistencies than when evaluating one item per screen.” (“Presentation of a Single Item Versus a Grid”, Mario Callegaro, et al)
  • According to “‘Up Means Good’: The Effect of Screen Position on Evaluative Ratings in Web Surveys”, by Roger Tourangeau, Mick P. Couper, and Frederick G. Conrad in 2013, items rated on a common scale received more favorable ratings if they occurred at the top of the list and more negative ratings if they occurred at the bottom. In an additional paper, “The Design of Grids in Web Surveys”, the authors find that dividing grids up into separate questions reduces missing data.

The simple way to address these issues is to ask the questions separately instead (called “item-by-item format”). Splitting the matrix into its component questions has greater predictive validity according to SSI’s report “Grid Test Summary” (March 2009), perhaps precisely because doing so does take participants more time to answer each question. In “An experiment comparing grids and item-by-item formats in web surveys completed through PCs and smartphones,” Melanie Revillaa, Daniele Toninelli, and Carlos Ochoa replicated this work, finding longer completion times for the item-by-item format and concluding that using this format improves the comparability of responses collected from smartphones vs. computers.

Item-by-item presentation is often done automatically when rendering surveys for the web, depending on the survey tool. For instance, the above grid question is displayed as the following screens on a mobile phone:


For times where the grid cannot be avoided, make sure to repeat column headings regularly, to minimize confusion when selecting a radio button:


In the Journal of Market research, David Gal and Derek D. Rucker, authors of the article "Answering the Unasked Question: Response Substitution in Consumer Surveys", suggest a newly observed participant behavior that can reduce the quality of survey data. "We propose that participants' answers to questions might sometimes reflect attitudes that participants want to convey, but that the researcher has not asked about, a phenomenon we term response substitution." While the authors didn't discuss the impact on grid questions, longer grid questions may be less prone to response substitution than shorter questions, as participants assume that one of the rows will cover the topic important to them; research has already shown that longer grids (10 items vs. 5 items) reduce the halo effect

Finally, grids consisting of agreement scales are best rewritten to use construct-specific scales.

 

Recent Headlines

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. PDI Releases New C-Store Shopper Report Jeffrey Henning 26-Jun-2019
  4. Issues with Grid Questions Jeffrey Henning 24-Jun-2019


White Papers


Consumer Research

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  4. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019


From Jeffrey Henning's Blog

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Asking Too Much Jeffrey Henning 07-Jun-2019
  3. Helping You Through the Entire Survey Research Process Jeffrey Henning 02-Mar-2019
  4. Market Research Conference Calendar - 2019 Jeffrey Henning 12-Dec-2018