In the previous papers in this mini-series (Ellis, 2025a 2025b, 2025c), we have been considering the use of questionnaires as a means of collecting data for both service improvement and research. We have seen how questionnaires are usually used for collecting quantitative data but may also be used for collecting qualitative data in some instances.
We have identified how questionnaires can be a relatively inexpensive way of collecting large amounts of information, but that they need to be well-designed in order to ensure that information is of use. This means ensuring that the purpose of the questionnaire is well-established before it is written and that the abilities of the respondents also need to be fully considered. We further identified that in order to manage large amounts of information, to make questionnaires accessible to respondents and for clarity, questions need to be designed so they can be answered in a variety of ways.
In the previous paper (Ellis, 2025c), we identified that it is important to consider the order in which questions are asked so as to engage the respondent at the right time with the process of answering. We also identified why it is important to check that questions and questionnaires actually ask what the creator set out for them to ask; that is, to check they are valid.
In this paper, we will consider some of the issues of bias that occur when designing and using questionnaire as a means of data collection.
General design bias
“Bias is a systematic mistake in the planning, execution, or analysis of a study that results in inaccurate conclusions” (Critical Appraisal Skills Programme, ND). As identified, bias can occur at any stage of the research process. In the questionnaire design, this may occur because, for example, the person setting the questions is, even subconsciously, looking for a specific outcome from the questionnaire. This may in turn lead them to framing the questions or structuring the questionnaire in a way that favours that outcome.
Avoiding this form of bias requires the researcher to frame questions in a manner that is not leading or suggestive, for example, rather than asking “How good was the service your received today?” ask “How would you rate the service you received today?” Framing the question in a neutral manner avoids issues of suggesting to the respondent that the response should be, in this case, a positive one.
Similarly, using jargon, technical terms or wording questions so that they require interpretation may exclude some sections of a population from completing the questionnaire, for example, people who do not have technical knowledge or whose reading ability is below the average. Systematically excluding people from the research in this way means the results may not actually represent the views of the population they set out to study; that is, they are not generalisable (Ellis, 2025d).
As well as providing leading questions or excluding people through the use of jargon, the order in which questions appear in a questionnaire can influence the ways in which people respond to them. This order–effect bias is known to affect the ways in which people respond to questions throughout a survey (Oldendick, 2008). For example, key questions early on in a questionnaire may anchor the way people respond later in the questionnaire. This means asking closed questions at the start of a questionnaire, for example, “is pain the biggest issue you have with your wound?”, is likely to cause people to identify pain as the biggest challenge facing wound management when asked an open question, such as “what is the biggest challenge facing wound management?” later in the questionnaire.
As well as the order of questions and the use of leading questions, the order of possible responses can impact the way people respond to questions, for example, when using Likert scales. According to Taherdoost (2024), the order of responses can have an impact on the way in which people respond to an attitudinal survey. This so-called “primacy effect,” can be seen where people are more likely to, for example, choose the option “strongly agree” when it is the first option in a list compared to when it is the last one.
Randomising the order of questions, and potential answers, when using scales, can help mitigate this form of bias in questionnaires. Randomisation ensures that conscious and unconscious bias are mitigated but can mean a questionnaire may appear a disjointed and can be more difficult to answer.
Respondent bias
Respondent bias in questionnaires may also arise because of the way respondents feel they should answer a question. These sorts of issues often arise when the questionnaire is set by someone the respondent is dependent on or who they do not want to be seen to offend. This includes the sorts of surveys and questionnaires undertaken in clinic settings. In such situations, sometimes respondents will intentionally alter their responses because they want to control how they appear to the person who gave them the questionnaire (Brandner and Hood, 2021).
One way of managing respondent bias is checking that people are reading and responding to the question as set in a consistent manner as opposed to randomly or without reading and understanding the question. A tried and tested technique to check this is to ask the same question more than once within the questionnaire and check if the answers tally.
Similarly, it is also useful for really important questions to pose the same question twice by rephrasing it the opposite way to the first time. For example, a simple yes/no answer to “Was your pain control acceptable during your latest dressing change?” may become “Did you experience an unacceptable level of pain during your latest dressing change?”. If someone answers “yes” to question 1 and later in the survey answers “yes” to question 2, then their answers are inconsistent and should be disregarded. One would expect to see an answer of “yes” for question 1 and “no” for question 2, or vice versa. Using this approach enables the question setter to see if the answer the respondent is giving is valid – that is, answers the question in the way they mean to answer the question.
When used with Likert scales this validation method can be quite useful – especially where the scoring is also reversed when a positive question is later repeated as a negative. Reversing the scoring when reversing the question means that the level of respondent bias can be ascertained.
Selection bias
There are various forms that selection bias can take when applying questionnaires. All forms of selection bias may mean that the study population is not representative of the population from which they are drawn. This affects the validity of the study (Arias et al., 2023).
One form of selection bias occurs when the people applying the questionnaire are not careful in ensuring that the intended respondents to the questionnaire have the means to access it, Smith and Noble (2025) give the example of setting a questionnaire, which can only be accessed online, for people who have limited access to the internet.
Another form of selection bias occur when questionnaires are only handed to people who are likely to provide positive answers. As with other biases, this can be done deliberately or subconsciously, either way the findings of the survey will not reflect the reality for the population the study is to be about.
There are two ways around this. The first is to target everyone form the population, for example, everyone who attends a wound care clinic over a period of time regardless of any characteristics. The second is to ensure that the recipients of the questionnaire are chosen randomly so that no human biases come into play.
Conclusion
The risk of bias in the design application and analysis of questionnaires is a pervasive issue. If sources of bias are not considered, a questionnaire can be of little or no value as a source of service improvement or in research. As with other aspects of questionnaire design, the best approach to managing bias is to be clear about what the questionnaire is aiming to do and the characteristics of the people to who it is to be applied. It is also imperative that sources of bias are considered and designed out.
In the subsequent paper in this mini-series on questionnaires, we will consider issues of reliability, ethics and the ways in which questionnaires can be applied.
