People ask questions in order to learn what they wouldn’t otherwise know. When a respondent answers a question on a survey they must rely on their comprehension to interpret what that question means. Then during a process of memory retrieval they access relevant information for content to be included in their answer. Judgment is used for comparing and evaluating ideas in a way that allows information to be synthesized. During response selection an answer is selected and placed in the required response format. All stages of the cognitive process just described rely on the first stage which is question interpretation. For this reason questions used in surveys must be refined to minimize interpretation problems and thus reduce measurement error. If the respondent is unable to understand a question or fails to comprehend the question the way it was intended then the data is neither valid nor reliable. A question must use language in a way that makes the intended observer’s meaning behind that question obvious. In order to do this a survey must reflect an understanding of the population being sampled. Before administering a survey you need to ask yourself whether your questions measure what you intend them to measure. Is it valid? Do your respondents interpret your questions the same way. Is it reliable? In order to ensure that the respondent is answering the question we think they are answering it’s important to make questions clear and precise.
Level of Wording
Question wording can affect the answers that are received. The wording used in questions should be appropriate to the level of education and characteristics of the respondents. A researcher must be aware of the respondent’s understanding of difficult words, colloquialisms, jargon, technical terms, and language proficiency when settling on an appropriate level of wording. They also must be aware of words or phrases that are gender or ethnic sensitive. If the language used is not the first language of all your respondents then the level of language proficiency of all your respondents must be carefully considered. Level of wording is important because a respondent that is embarrassed to admit that they do not know something is likely to give any answer rather than admit they do not know. You may also see a lot of neutral response choices such as “do not know” or “no opinion.” Also, it can lead to a high rate of refusal to complete survey rate. In general you should use wording that is simple, direct, and to the point. A question that would not be appropriate to the level of wording of your average survey respondent could be:
“Did the attorney deliberately obfuscate the truth by inundating the jury with statements that were verbose and declamatory?”
An understanding of the population you are administering the survey to must be worked out in advance. If you are administering a survey to attorneys you would use a different level of wording than you would if you were asking the same questions to a group of layman.
If there is more than one way to interpret a question than that question is ambiguous. When a question is vague it has an imprecise range of application. If a question uses vague wording or is ambiguous it can lead respondents to interpret a question in a variety of ways. An example of an ambiguous question might be:
“Who do you buy groceries for?”
It is not clear whether either you or any household pets are included in this question. Is this asking who you usually buy groceries for? Does this include anything you might pick up for your parents that you no longer live with? A better way to ask this question would be:
“What is the total number of household occupants that you purchased groceries for in the past year?”
Ambiguity can also arise if a respondent is unclear about what type of question is being asked, the motive for the question being asked, the purpose for a question being asked, and when inferring motives behind a question that have no truth in reality.
A question is confusing if it leads to a feeling of uncertainty in the respondent when the intent or meaning is not clear. Here is an example:
“Does it seem possible or impossible that the Cubs will win the World Series, or do you feel they will always find a way to lose?”
1_ It seems possible
2_ It seems impossible
Here the respondent may feel trapped into either agreeing that the Cubs will find a way to lose or saying that it is impossible for the Cubs to win the World Series. A clearer way to ask this question would be:
“Does it seem possible that the Cubs will ever have a chance to win the World Series?”
1_It is possible that it could happen.
2_It is not possible that could happen.
Working Memory Overload
In order for a respondent to process a question it can not impose a cognitive burden on their working memory. Working memory is short term memory used to hold information while the respondent is comprehending a question. This can often happen when a question attempts to be too precise, by injecting complex syntax and embedded clauses, in an attempt to clearly define a question. Earlier we talked about vagueness and ambiguity. Couldn’t you make our earlier question more precise by rewording it as:
“Including yourself, any pets you own, but not including any relatives or friends that do not live in your household for at least 320 days a year, how many household occupants did you buy groceries, by groceries we mean any item of food bought at the grocery store, for in the last year?”
The problem with asking questions like this is that you are going to tax your respondents memory and some of them will not even get what you are trying to say. Individuals differ in how much their working memory can handle. Questions that pose a burden to some may not pose a burden to others. Avoid complex syntax in your survey questions. This means that its grammatical composition can not be dense, structurally ambiguous, or not well formed syntactically. Memory overload can occur with extremely long sentences, using too many logical operators (such as or, and, if-else, then), and quantifiers. In order to make responses more precise it is better to ask a series of simpler questions. The two major consequences of overworking memory are that items drop out of working memory or cognitive processing can slow down. The respondent may simply drop some of the intended meaning of the question and they could spend a lot of time coming up with an incomplete interpretation of the question.
Double Barreled Questions
When a question introduces two or more issues with the expectation of a single response it is a double barreled question. This inadvertently confuses the respondent and limits the useful data that you can collect. The respondent is forced to answer two issues with a single response which they may agree with and disagree with respectively as in the following question:
“Is the big city a great place to experience culture and raise a family?”
A person may very well feel that a big city is a great place to experience culture because of its diversity but that doesn’t mean that they want to raise their family there for a variety of reasons.
A presupposition is anything that is assumed to exist or be true and every question has a number of nouns or propositions that are presumed to be true. Questions that contain a false presupposition are known as leading questions. Under certain circumstances false presuppositions can cause respondents to make incorrect inferences about what occurred and to misremember events. Imagine a cop asking the following question:
“At what point did you quit speeding when you saw the truck swerving onto oncoming traffic?”
Pretend you weren’t speeding. There is evidence to suggest that asking leading questions like this can cause respondents to misremember what really happened.
In order to obtain accurate results a survey question should be neutral in order to welcome as many points of views as possible. A loaded question contains an incriminating assumption if the respondent accepts it to be true. An example of this can be found here:
“Do you believe that we should redistribute wealth by allowing tax breaks to expire?”
The loaded word here is “redistribute” as it is often associated with socialism. The question is asked in such a way that a respondent may not feel comfortable giving an honest answer. Injecting emotional language in surveys can bias survey results.
If you have a clear understanding of each survey question’s intent you can correct flaws in the survey instrument at the outset by pretesting your questions. Common methods used in pretesting include expert reviews, forms appraisal, cognitive interviews, and the use of focus groups.
Keep the focus of each question on the specific attribute or phenomenon that you are measuring. Collecting quality responses is highly dependent on the clarity of questions. Keep your syntax simple so that your respondent is not easily confused. Although you want your question to be precise you need to avoid using long, dependent clauses or compound, complex sentences as they can overwork your respondent’s memory and cause them to answer questions incorrectly. Just as you should not ask a question that is relevant to the research you are conducting you should not use a word that is not relevant to the question you are asking. Keep it brief. The longer a question is the more likely the respondent is to misinterpret it. Focus on asking one question at a time. Brevity can not come at the expense of the intended meaning of the question. You are simply being succinct or in other words choosing the shortest way to pose a question. Pretesting can allow you to further refine your questions so that the meaning you intend is not lost on your respondent.