The psychology of survey response

When designing a questionnaire, we always have the respondent experience in mind. This is why this article has to start with the psychology of survey response.

There is a lot of useful literature available, but we go by the model developed by Roger Tourangeau (2000) in The psychology of survey response. It is a general-purpose tool that helps to easily assess the quality of a questionnaire.

According to this model, the human mind has to complete four steps in order to give a suitable answer, whenever a question is asked.

Let’s explain these steps by using an example: “How many Aspirin tablets do you take per month?”

1. Comprehend the question by identifying cues: You add context to the question by adding “Aspirin”, so it is possible to deduce the meaning. Without the context, the respondent could be unsure of whether it is an electronic tablet or a medical tablet the question is referring to. However, from the context: the expression “taking a tablet per month” refers to medication.
2. Retrieve relevant information from memory: Once the respondent has understood the question, they will start to retrieve relevant information from their memory. If this is too hard to remember exactly when the last time was, they might use another approach and estimate how often they buy a package of Aspirin on average.
3. Integrate available information into judgment: Let’s say the respondent comes to the conclusion that they buy one package with 20 tablets per year. That would mean, that the respondent takes about two tablets per month on average, which seems to be plausible to them.
4. Map judgment on response and give an answer: Depending on the question type, the respondent will now enter “2” into the open text field or select the right answer among the options given.

With the help of this model, you can try to anticipate the mental processes of your respondents for each and every question in your questionnaire. It really doesn’t matter whether it is an open-ended question, a single select, a multiple choice or any other question type.

Survey fatigue

It’s needless to say, that completing the four above steps will require concentration and eventually tire the respondents. The more questions you’ll ask in a row, the more you will use up the respondent’s attentiveness and concentration. In simple terms, survey fatigue is the product of the required mental effort and the length of the questionnaire – so you should either keep your questionnaire short or simple, or even better; short and simple.

And that leads us to the importance of data quality. Everyone has an individual limit of cognitive burden they can cope with. If the level of fatigue exceeds the mental capacity, the brain will start looking for shortcuts and heuristics to reduce the cognitive burden. You could, for example, put less effort into the retrieval of information and use only the first thing that comes into your mind. Or you could simply select the “Don’t know”-answering option. In any case, this so-called satisficing behavior leads to less accurate answers and, in its most extreme forms, to non-sense data.

In summary, good questionnaires preserve the respondents’ attentiveness and concentration by reducing the cognitive burden of participating. Consequently, they help to achieve superior data quality. This is exactly what we are striving for when designing online questionnaires.

What can you do to improve your questionnaire?

If you want to improve any given questionnaire, you can use two strategies: reducing the respondents’ mental effort for each single question or shortening the questionnaire in total.

Reducing the mental effort

Let’s start with the first strategy. In order to reduce the mental effort, you should optimise each of your questions for each of the four steps.

1. Use a neutral and comprehensible language. Be clear and unambiguous in your wording and don’t use interlaced questions or double negations.
2. Give cues to facilitate the retrieval of relevant information. These can be texts, illustrations or photos, but also previous answers from the respondent (“In a previous question, you’ve said that you don’t like the product design. What can we improve about it?”)
3. Don’t influence the respondent’s judgment. Possibly the hardest task, because every detail may have an impact. At least try to be cautious and avoid any active influence on the respondent (e.g. leading questions).
4. Provide a comprehensive set of answering options. Give a disjoint and comprehensive list of answering options or allow for honest feedback.

These improvements are pretty obvious! But there is another thing you should do: optimize the usability of your questionnaire. If the usability of your questionnaire is not intuitive, the respondent will need to put mental effort in filling it in. And that will of course harm the data quality of your survey.

Just preserving the tried and tested approaches from the past is not enough. Because you have so far never changed the layout of your questionnaire, doesn’t mean the data you get out of it is still high quality. The way people interact with the internet has changed and your questionnaire should always reflect the new usage patterns.

This is why we constantly update the layout and functionality of our survey engine. Among the many things we consider are the latest technical standards, new devices, and their specifications, and changes in usage behavior.

Shortening questionnaires

Let’s have a look at the second strategy for improving a questionnaire: shortening it, in order to reduce the duration of the mental effort.

Unfortunately, very often online questionnaires are too lengthy. The absence of a costly interviewer makes it easy to add irrelevant questions, just for the sake of collecting more data once you’re already at it. This hasn’t really been considered a problem for a long time, but mobile questionnaires recently brought new life to this discussion.

We know that data quality usually decreases after 20 minutes (slightly depending on the topic and other factors) and that there is no effective remedy against it. Sometimes we get asked to raise the incentive. While this may motivate the respondent to complete a lengthy questionnaire, it is not suitable for boosting the mental capacity or reducing the cognitive burden. So you may preserve the willingness to participate, but incentives do not improve data quality.

So first of all, you should get rid of all irrelevant questions and battery items. That is probably the hardest task, but as a rule of thumb, you should only collect data that you will really need to answer your research question. Secondly, shorten the question texts themselves (Hint: A maximum of 140 characters like Twitter would be ideal.). If the texts are too long, respondents may not read them thoroughly and just guess the question by reading the answering options or looking for keywords.

Once you have shortened your questionnaire to a reasonable length, you should optimise the order of the questions. Assuming that the respondents’ attentiveness and concentration is still good at the beginning and decreases towards the end, the most important questions should be asked as early as possible. Conversely, put questions about the demographic background towards the end of the questionnaire, as respondents will be able to answer them correctly, even if they feel a certain fatigue.

The case of mobile surveys

And that leads us to the last point. Do not exclude mobile users, just because the usability of your questionnaire is poor or the survey is too long, as this will bias the results of your study.

At first sight, mobile users have to put a higher effort in completing a questionnaire: their screen is considerably smaller and might be in situations that partially consume their attention. The surprising truth is that mobile research very often goes hand in hand with excellent data quality, because researchers actively design for an optimized survey experience. Following the principles above simply seems to be more plausible, when bringing the limitations of a mobile device to mind.

Frankly spoken: There is no such a thing as mobile-friendly. Either a questionnaire is user friendly, or it isn’t.

Bottom line

This article explores the art of designing online questionnaires with a focus on the psychology of survey response. It introduces a four-step model for understanding how respondents process questions and highlights the importance of minimizing cognitive burden to maintain data quality. The strategies for improvement include optimizing question clarity, cues for information retrieval, avoiding influence on judgment, and providing comprehensive answer options. Usability and layout updates for modern internet usage patterns are also emphasized. Additionally, the significance of shortening questionnaires by eliminating irrelevant questions and optimizing question length and order is highly important. Ultimately, the article underscores the importance of survey design in preserving data quality and respondent engagement.