In the previous papers in this series (Ellis, 2025a; 2025b), we started to consider the nature and use of questionnaires as a means of collecting information both for research and in evaluating practice. We identified questionnaires are by their nature mainly quantitative, but that they can be used for collecting some qualitative information.

We identified how, when compared to other forms of research, questionnaires can be a relatively quick and cheap way to collect information for whatever purpose. We also identified the questions within a questionnaire need to be carefully constructed to avoid ambiguity and to enable respondents to answer in the ways they want. 

We identified how objective setting is a good way to frame what is wanted from a questionnaire before it is designed to help focus the content of the questionnaire and the way questions are set. We saw how it is important to ask questions within the questionnaire using terminology which is familiar to the potential respondents or risk them not being able to engage with it. We further saw how questions can be created such that the answers they collect can be free text, use rating scales (such as Likert), require simple yes/no answers or ask respondents to arrange things in order of importance to them. 

In this paper, we will consider the order of questions within a questionnaire, as well as how one might check if the questions one is asking make sense. We will also consider the nature of validity in questionnaire design. 

Why ordering of questions is important

It is at least as important to consider the order of questions within a questionnaire as it is to consider the questions themselves. While this is not so important in simple and short questionnaires, in longer and more complex questionnaires, getting the order and form of questions wrong can introduce significant bias; bias being the introduction of systematic errors affecting the validity and reliability of the research. We will consider more about bias in the next and final paper in this miniseries. 

There are some well-known effects on the reliability and validity of a questionnaire which might arise from the order in which the questions are presented, these include: 

• Primacy effect: Respondents might assign greater importance to any information given early on in the questionnaire thus impacting their later responses (Yan and Keusch, 2015). This is similar to anchoring, where an early piece of information shared within the questionnaire sets the tone for the ways in respondents answer later questions — how they are anchored
• Context (Carryover) effect: Earlier questions and the answers given may influence how respondents approach and consider later ones. This means the respondent may generate their own context for the questions changing their meaning.
• Recency effect: Respondents may give undue importance to the last question or option encountered, skewing their responses.
• Priming is very similar in that it refers to how someone is influenced by an idea, often  unrelated to the actual question, but which was introduced earlier in the questionnaire.  For example, if a question is asked about the use of honey-based topical applications and is then followed by a question asking about the most important innovations in wound care in the last 20 years, people may be more inclined to include honey-based topical applications in their answer than they otherwise would be. 

It is clear, therefore, that it is important to give consideration to the order and content of the questions in a questionnaire or there is a risk of inadvertently introducing bias.  

The order of questions

There is a plethora of advice about how questions should be ordered in any questionnaire. When designing a questionnaire, as opposed to using a pre-validated one (see later), there are some simple rules which apply in almost all cases, these are:

• Collect the simple data first (Ellis, 2025c). This includes collecting information such as demographic data which does not require a lot of thought and which won’t influence answers to later questions. Starting a questionnaire with complex questions can lead to respondents becoming uncomfortable and abandoning the questionnaire early.
• Grouping questions logically helps. If, for example, you want to understand people’s experiences of a wound care clinic, then ask questions about the environment together, about the staff together and about the processes together and so on. This enables the respondents to focus on that aspect of their experience and avoids confusion.
• Funnel the questionnaire. This means ask broad questions early and more complex or focussed ones later (Taherdoost, 2022). Later in the survey, people will be invested in the process and are more likely to continue. They are also warmed up to the themes and better prepared to answer the more taxing questions.
• Similarly, it is often advisable to place sensitive questions towards the end of the questionnaire because at this point people are warmed up to the topic and have invested time in it, meaning they are less likely to abandon it (Taherdoost, 2022).
• Where there are concerns about issues such as primacy and recency, it can be useful to randomly order the questions, or answer options where appropriate, in the questionnaire. This means answers are not chosen or questions answered because of where they are placed. That all said it is important to balance this against considerations of funnelling or pyramiding.
• Pyramiding refers to the opposite of funnelling.  That is where the sensitive questions are asked early and the more general ones later. This can work for surveys that are focused on attaining focussed, as well as broad answers to a topic and where respondent are likely to be invested in the research early.
• In other instances, using transitions helps respondents focus on new topic areas and provides clarity as a questionnaire progresses.  For example, when undertaking a survey about a wound care clinic one might make statements such as “now we are going to ask your opinion about the clinic environment” and “next we will be asking questions about your interactions with clinic staff”.  

Often the best way to know how to approach questionnaire design is to look at what others have done, paying special attention to the elements of the discussion around the strengths and weaknesses of the questionnaire design.  This should lead to the design of a questionnaire which may be used in pilot form enabling testing not only of the ordering of questions, but also important issues, such as individual question clarity (Tsang et al, 2017). 

Validity

The basic concept of validity within a questionnaire refers to whether or not the questions, and hence the questionnaire, actually ask what the question setter believes they have asked. There are a number of forms of validity which relate to questionnaires, we will consider the important ones here.

Face validity refers to whether the questionnaire looks, including to non-experts, to include the questions needed to get the information the question setters have set out to gather. It also considers issues such as the format, readability and clarity of the questions to the intended audience (Ranganathan et al, 2024). 

Content validity is about whether the questionnaire covers the elements of the topic it needs to cover. Unlike face validity, content validity needs to be evaluated by subject experts, e.g. tissue viability or wound care nurses, who understand the breadth and depth of the topic. In common with face validity, however, this is a subjective measure relying, as it does, on people’s opinions. 

Criterion (concurrent) validity measures how well the findings of a questionnaire agree with those of the gold standard questionnaire when applied concurrently. For example, a new short quality of life with a chronic wound questionnaire when compared to, for example, the validated Wound-QoL-14 (Janke et al, 2024).

Testing the validity of a questionnaire can be undertaken by:

• Getting non experts to review the questionnaire for content and the understandability of the questions.
• Getting experts to ensure the questionnaire covers key concepts and that the questions make sense.
• Comparing the answers gained in the questionnaire to any existing gold standard assessment (Ranganathan et al, 2024). 
• Pilot testing of the questionnaire and gathering feedback on content and clarity.
• Using statistics to test issues such as its internal consistency and reliability (see next paper).

It is advisable in most instances to use pre validated tools for data collection. These tools, often questionnaires, have been tried and tested, in different groups of people, to ensure they collect the information they set out to collect. They can save the researcher time and effort, produce valid results and many are free to access and use.  

Krug and Wensing (2023) wisely point out however, that when a validated tool is used in a population it was not previously validated for use with, it requires new proof of validity. So, for example, a wound care quality of life tool which is validated in Europe may not be valid for use is Asia. 

Conclusions

In this paper, we have identified issues a question setter might need to address when considering the order of questions within a questionnaire and why these are important. We have identified some elements of validity as they apply to questionnaire design and what a researcher might need to do to ensure they are collecting answers to questions they are concerned with.  

In later papers in this series on questionnaire design, we will consider issues of reliability as well as how questionnaires might be applied and how a sample of respondents might be chosen. We will also consider the ethical issues associated with questionnaires and how these can be addressed.