Surveying customers is an outstanding way to get feedback. Doing it effectively, though, is as much a science as it is an art. We at Transmitive are launching a massive survey to understand the habits and preferences of our prospective customers, so my next series of posts will discuss survey design and execution.
 
To kick things off, it’s worth reviewing a few things that can go awry. The most obvious one, of course, is improper wording of survey questions. Respondents may be answering a different question than the one you intended. A confusing question may prompt the respondent to tick a random box and move on. And, most insidiously, you may be introducing subtle bias into responses based on how ask a question.
 
Although you may be familiar with examples of biasing questions, here are a few interesting lessons from across the web:

• Leading questions. From a 1978 study [1], the same question was asked two ways: “‘Are you in favor of giving special priority to buses in the rush hour or not?’ Another roughly equally sized group was asked ‘Are you in favor of giving special priority to buses in the rush hour or should cars have just as much priority as buses?’” Across 1,000 respondents, 69% responded ‘yes’ to the first question, and 55% responded ‘yes’ to the second. The second question was much more likely to bias car-drivers because it made clear that giving buses priority came at the expense of cars.

• Introducing a new concept. From a 1990 study [2], 93% of people agreed that “Women should get equal pay for equal work.” When a new concept was introduced into the question, the results changed substantially — presented with the statement “Women should get equal pay for equal work, even if this results in higher unemployment,” only 78% of people agreed. Clearly, the addition of a negative (and seemingly extraneous) concept changed respondents’ responses significantly.

• Question ordering. From a studentreader.com blog post, the question “Do you think the U.S. should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?” resulted in agreement from only 36% of respondents. When the following question preceded the above, however, the results practically inverted.  Asked first: “Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?,” and then asked whether the U.S. should allow Communist newspaper reporters, 73% agreed that the U.S. should allow Communist newspaper reporters to send news back to their papers.
  

• Double barreled questions. Although the examples aren’t as interesting, double-barreled questions are a big mistake. For example, don’t ask “Are you happy with your pay, benefits, and work/life balance at your job?” unless you really want to ask about three different things at once.  Otherwise, the responses will be more complex and far less useful than intended.
  
  
• Word ambiguity. The same word means different things to different people. One specific example that comes up in survey literature concerns frequency descriptors. The EvalThoughts blog [4] conducted an interesting (and very informal) survey of how people interpret various frequency descriptors in relation to the question: “On average, how often do you withdraw cash from your bank account each month?” Here are the results with [mean, standard deviation]: “never” [0.00, 0.00], “once” [1.10, 0.28], “a couple of times” [2.60, 1.04], “a few times” [4.73, 2.19], “several times” [9.13, 4.24].
  
  This is a powerful example because it quantifies exactly how subjective certain words are. Many grammar fanatics, for example, might think that a “couple” only refers to two things, even though the data show otherwise. Most any other category of words—emotions, signifiers of importance, etc.—will exhibit similarly ambiguous word meaning.

So, in conclusion, choose your words carefully! Best of luck. If you have other lessons, please pass them along.

Sources
[1] Kalton G; Collins M & Brook L (1978). Experiments in wording opinion questions. Applied
Statistics, 27 (2), 149-161.
[2] Gendall P & Hoek J (1990). A Question of Wording. Marketing Bulletin, 1, 25-36, Article 5.
[3] http://studentreader.com/category/political-science/public-opinion/ 
[4] http://www.evalthoughts.com/2009/10/surveying-bias-in-survey-response.html