Interpreting questions can be difficult
People ask questions in order to learn what they wouldn’t otherwise know. When someone answers a question on a survey they have to be capable of interpreting questions themselves.
Then, during a process of memory retrieval, they access relevant information for content to be included in their answer. Judgment is used for comparing and evaluating ideas in a way that lets information be unified.
During response selection, an answer is selected and placed in the required response format.
If the respondent is unable to understand a question or fails to comprehend the question the way it was intended then the data is neither valid nor reliable.
For this reason, questions used in surveys must be refined to minimize problems interpreting questions and thus reduce measurement error.
A question needs to use language in a way that makes its meaning obvious. To do this a survey must reflect an understanding of the sample population.
Before administering a survey you need to ask yourself whether your questions measure what you intend them to measure.
- Is the question valid? Do your respondents interpret your questions the same way?
- Is the question reliable? In order to ensure that the respondent is interpreting questions the way we need them to for a reliable answer, they have to be clear and precise.
Level of Wording
How a question is worded can affect the answers that are received. The wording used in questions should be appropriate to the level of education and characteristics of the respondents.
When settling on an appropriate level of wording, you as a researcher must be aware of the respondent’s understanding of:
- Difficult words
- Technical terms
- Language proficiency
You also need to be aware of words or phrases that are gender or ethnic sensitive. If the language used is not the first language of all your respondents then the level of language proficiency of all your respondents must be carefully considered.
How you word questions is important because a respondent that is embarrassed to admit that they do not know something is likely to give any answer rather than admit they do not know.
What you DON’T want to happen.
If people have trouble reading and interpreting questions you give them, you may also see a lot of neutral response choices such as:
- Do not know
- No opinion
Those aren’t really useful to you, are they?
That lack of understanding can lead to a high rate of refusal to complete survey rate. In general, you should use wording that is simple, direct, and to the point. A question that would not be appropriate to the level of wording of your average survey respondent could be:
“Did the attorney deliberately obfuscate the truth by inundating the jury with statements that were verbose and declamatory?”
You need to understand the audience you plan to survey in advance. If you are giving a survey to attorneys you would use a different level of wording than you would if you were asking the same questions to a group of layman.
If there is more than one way to interpret a question, it’s an ambiguous question. If a question uses vague wording or is ambiguous it can lead respondents to interpret a question in a variety of ways. An example of an ambiguous question might be:
“Who do you buy groceries for?”
This kind of question could lead to more questions when trying to answer it because it’s not clear enough.
I’m don’t know what to do with these…
- Are either you or any household pets are included in this question?
- Is this asking who you usually buy groceries for?
- Does this include anything you might pick up for your parents that you no longer live with?
A better way to ask this question would be:
“What is the total number of household occupants that you purchased groceries for in the past year?”
Ambiguity can also arise if a respondent is unclear about:
- What type of question is being asked
- The motive for the question being asked
A question is confusing if it leads to a feeling of uncertainty in the respondent when the intent or meaning is not clear. Here is an example:
“Does it seem possible or impossible that the Cubs will win the World Series, or do you feel they will always find a way to lose?”
- It seems possible
- It seems impossible
Here the respondent may feel trapped into either agreeing that the Cubs will find a way to lose or saying that it is impossible for the Cubs to win the World Series.
A clearer way to ask this question would be:
“Does it seem possible that the Cubs will ever have a chance to win the World Series?”
- It is possible that it could happen.
- It is not possible that could happen.
Working Memory Overload
In order for someone to process a question, it can not impose a cognitive burden on their working memory.
Working memory is short term memory used to hold information while the respondent is interpreting a question. This can often happen when a question attempts to be too precise, by injecting complex syntax and embedded clauses, in an attempt to clearly define a question.
Unfortunately, we aren’t equipped with backup memory disks in the brain.
Earlier we talked about vagueness and ambiguity. Couldn’t you make our earlier question more precise by rewording it as:
“Including yourself, any pets you own, but not including any relatives or friends that do not live in your household for at least 320 days a year, how many household occupants did you buy groceries, by groceries we mean any item of food bought at the grocery store, for in the last year?”
The problem with asking questions like this is that you are going to tax your respondent’s memory and some of them will not even get what you are trying to say.
Individuals differ in how much their working memory can handle. Questions that are hard for some may not be hard for others. Avoid complex syntax in your survey questions.
This means that its grammatical composition cannot be:
- Structurally ambiguous
- Not well-formed syntactically
Memory overload can occur with:
- Extremely long sentences
- Using too many logical operators (such as or, and, if-else, then), and quantifiers.
In order to make responses more precise, it’s better to ask a series of simpler questions. The two major consequences of overworking memory are that items drop out of working memory or cognitive processing can slow down.
The respondent may simply drop some of the intended meaning of the question and they could spend a lot of time coming up with an incomplete interpretation of the question.
When a question introduces two or more issues with the expectation of a single response it’s a double-barreled question.
This inadvertently confuses the respondent and limits the useful data that you can collect. The respondent is forced to answer two issues with a single response which they may agree with and disagree with respectively, like in the following question:
“Is the big city a great place to experience a culture and raise a family?”
A person may very well feel that a big city is a great place to experience culture because of its diversity but that doesn’t mean that they want to raise their family there for a variety of reasons.
A presupposition is anything that is assumed to exist or be true beforehand at the beginning of a line of argument or course of action.
Questions that contain a false presupposition are known as leading questions. Under certain circumstances, false presuppositions can cause respondents to make incorrect inferences about what occurred and to misremember events.
Imagine a cop asking the following question:
“At what point did you quit speeding when you saw the truck swerving onto oncoming traffic?”
Pretend you weren’t speeding. There is evidence to suggest that asking leading questions like this can cause respondents to misremember what really happened.
In order to obtain accurate results a survey question should be neutral in order to welcome as many points of views as possible.
A loaded question contains an incriminating assumption if the respondent accepts it to be true. An example of this is:
“Do you believe that we should redistribute wealth by allowing tax breaks to expire?”
The loaded word here is “redistribute” as it is often associated with socialism. The question is asked in such a way that a respondent may not feel comfortable giving an honest answer. Using emotional language in surveys can bias survey results.
If you have a clear understanding of each survey question’s intent you can correct flaws in the survey instrument at the outset by pretesting your questions.
Common methods used in pretesting include:
- Expert reviews
- Forms appraisal
- Cognitive interviews
- Focus groups
Here are a few tips to keep in mind for pretesting.
- Keep the focus of each question on the specific attribute or phenomenon that you are measuring. Collecting quality responses is highly dependent on the clarity of questions.
- Keep your syntax simple so that your respondent is not easily confused. Although you want your question to be precise you need to avoid using long, dependent clauses or compound, complex sentences as they can overwork your respondent’s memory and cause them to answer questions incorrectly.Just as you should not ask a question that is relevant to the research you are conducting you should not use a word that is not relevant to the question you are asking.
- Keep it brief. The longer a question is the more likely the respondent is to misinterpret it. Focus on asking one question at a time. Brevity cannot come at the expense of the intended meaning of the question. You are simply being succinct or in other words, choosing the shortest way to pose a question.
Pretesting can allow you to further refine your questions so that the meaning you intend is not lost on your respondent.