In the previous papers in this mini-series (Ellis, 2025a, 2025b, 2025c, 2026), we considered the use of questionnaires as a means of collecting data for service improvement and research purposes. We identified that questionnaires are useful for collecting mainly quantitative data, but also can be used for some forms of qualitative data collection and that they are popular because they are generally quite cheap to undertake.
That said, we have also seen how questionnaires require some thought in their design and that healthcare professionals who are aiming to use a questionnaire need to consider how questions are written, the order in which they appear, the options for answering a question and the way in which the questionnaire is applied. In the past paper in this series, Ellis (2026), we looked in some detail at the issues of bias and how this might be avoided in questionnaire design.
In this the final paper in this mini-series, we will consider issues of what is termed reliability and give consideration to the ethical aspects of writing and applying a questionnaire in a health or social care settings.
Reliability
Reliability is concerned with the consistency of a questionnaire; or indeed any research data collection method. A reliable questionnaire is one which produces similar results when used in similar circumstances with similar people (Philip, 2024); by people this means the people answering the questions and, where appropriate, those applying them.
In this sense, reliability is about dependability. If a questionnaire asks about pain during dressing changes, for example, then people whose experiences of pain are similar should broadly provide similar responses. If they do not, then the differences in answers may reflect weaknesses in the questionnaire rather than real differences in experience. These weaknesses may be reflected in the ways the questions are asked, which may be ambiguous or otherwise confusing
Reliability is related to validity but it is not the same thing. A questionnaire may be reliable without being valid. For example, a set of questions may consistently measure satisfaction with the friendliness of staff, but if the study is intended to measure the effectiveness of wound care education, then the questionnaire is not valid for that purpose – remember validity asks whether the questions being asked actually mean what they intend to mean to the people answering them.
Therefore, reliability is necessary, but not sufficient, for a useful questionnaire. The researcher needs to know both that the questionnaire is asking the right questions and that it is doing so consistently.
There are several ways in which reliability can be considered. Test-retest reliability refers to the extent to which the same questionnaire produces similar answers when given to the same people on more than one occasion, assuming that the thing being measured has not changed (Ruel et al, 2016). This may be useful when developing a questionnaire about professional confidence, knowledge or attitudes. If a tissue viability nurse completes a questionnaire about confidence in applying compression therapy on Monday and again the following Monday, and there has been no new training or clinical experience in between, then large changes in response may suggest that the questionnaire is unstable.
Internal consistency is another important aspect of reliability. This concerns whether items within a questionnaire that are intended to measure the same concept appear to work together (Sudweeks, 2018). For example, a questionnaire measuring confidence in wound assessment might include questions about recognising infection, measuring wound size, selecting dressings and documenting findings. If all of these are intended to contribute to an overall confidence score, then the researcher needs to know that they are sufficiently related to one another. If they are not, then it may be inappropriate to add the scores together and report them as if they represent a single idea.
Reliability may also be affected by the wording, layout and administration of the questionnaire. Ambiguous wording, double-barrelled questions, unclear response options or changes in the way the questionnaire is introduced can all alter responses (Ellis, 2025d). For example, asking: “Were the staff kind and efficient?” combines two issues in one question. A respondent may feel staff were kind and not efficient, or efficient and not kind. Therefore, the answer becomes unreliable because different respondents may interpret the same question in different ways.
Piloting is one of the most useful ways of improving reliability. A pilot does not need to be large, but it should involve people who are similar to those who will complete the final questionnaire. In healthcare this may mean piloting a patient questionnaire with people who have similar levels of literacy, similar clinical experiences and similar access needs to the intended respondents. It may also mean asking healthcare professionals to comment on whether the wording is clinically meaningful and whether the questions are likely to be interpreted consistently.
Therefore, reliability is not simply a statistical matter. It is also a practical and professional issue. A questionnaire which is rushed, poorly explained or difficult to complete is unlikely to produce dependable data, even if it appears well designed on paper.
Ethical issues
Questionnaires often appear ethically straightforward because they are usually short, inexpensive and relatively unobtrusive. However, the fact that a questionnaire is simple to administer does not mean that it is ethical neutral. Questionnaires ask people to give information about themselves, their work, their health, their experiences or their opinions. As such, and especially in health and social care settings, this information may be sensitive and may be provided by people who are unwell, anxious, dependent on services or concerned about the consequences of giving honest answers. This means even the most innocuous questionnaire has ethical elements to it.
The first ethical consideration is consent (Hammer, 2017). People should know why they are being asked to complete the questionnaire, who is collecting the information, how the information will be used and that they have a choice about taking part – that is, they can choose not to! This applies whether the questionnaire is being used for formal research, service evaluation or local quality improvement. In some circumstances, a brief written explanation at the start of the questionnaire may be sufficient. In others, especially where sensitive topics are involved, more detailed participant information may be required. There are no hard and fast rules about this; a lot of this is down to personal judgement or that of the presiding research ethics committee.
A second issue is voluntariness. This is particularly important when questionnaires are distributed by healthcare professionals to patients or by managers to staff (Ellis, 2025d). Patients may worry that refusing to complete a questionnaire or giving negative answers could affect the care they receive, while staff may worry that critical answers could influence how colleagues or employers view them. For this reason, questionnaires should be introduced in a way which makes it clear participation is optional and care, employment or professional relationships will not be affected by the decision to take part or not. This is something often captured in any participant information sheet that accompanies the questionnaire.
Confidentiality and anonymity also require careful thought both in terms of research and local quality improvement questionnaires. A questionnaire may be anonymous if no one, including the researcher, can identify who completed it. It may be confidential if the researcher can identify respondents, but agrees not to disclose their identities. These are not the same thing and should not be confused. For example, an online staff questionnaire which collects email addresses is not anonymous, even if names are not requested. Similarly, a questionnaire completed in a small specialist clinic may be indirectly identifiable if it asks about age, diagnosis, length of treatment and location. Considerable thought needs to be given to the inclusion of data within the questionnaire that may identify individuals and whether it is necessary for the purposes for which the information is being collected.
It is widely known that when people believe that a questionnaire may not be anonymous, for example if it is administered face to face, then the answers people give will tend towards those which are more socially desirable and, therefore, may not be a true representation of what they think or believe (Leon et al, 2021).
Data protection is closely related to confidentiality. Healthcare professionals should consider what information is necessary, how long it will be kept, where it will be stored and who will have access to it. Collecting unnecessary personal information increases risk without improving the quality of the study. For example, if the purpose of a questionnaire is to understand general satisfaction with a dressing clinic, it is unlikely that the respondent’s full name or NHS number is needed. The principle should therefore be to collect the minimum information required to answer the question being asked.
The content of the questionnaire may also raise ethical concerns. Questions about pain, poor care, emotional distress, safeguarding concerns, professional errors or dissatisfaction with services may be important, but they may also cause discomfort. This does not mean that such questions should never be asked. It does mean that they should only be asked when they are necessary, worded sensitively and accompanied by appropriate information about what respondents can do if completing the questionnaire causes concern, anxiety or distress. In many research settings it is the responsibility of the researcher to provide or to signpost support.
Similarly, researching sensitive topics can create scenarios where the researcher is under a professional obligation to investigate or report what they are being told. This means that something about the potential need to whistle blow for the greater good (Varkey, 2020) or for the protection of an at risk individual needs to be written into any participant information leaflet accompanying the questionnaire.
Finally, there is an ethical responsibility to use the findings honestly. It is unethical to invite people to provide information and then ignore the results, selectively report only favourable findings or present weak data as if they are conclusive. If questionnaires are used in service improvement, respondents may reasonably expect that their views will be taken seriously. This does not mean that every suggestion can be acted upon, but it does mean that the findings should be analysed fairly, reported accurately and used proportionately to improve the quality of the service provision.
Conclusion
We have seen how a meaningful questionnaire needs to be reliable, ethically sound and applied in a way which is appropriate to its purpose and to the population it is focused on surveying. Reliability helps ensure that the data collected are consistent while ethical research practice ensures respondents are respected and their rights protected.
For health and social care professionals, the key message is that questionnaires should be treated as research and evaluation instruments rather than as simple forms. They require planning, testing, explanation and honest analysis and reporting/application to service improvement.
Over the course of this miniseries, we have seen how when used well, questionnaires can provide useful information about patient experience, professional practice, service quality and clinical outcomes. Used poorly, questionnaires produce misleading findings and may even undermine trust. The value of a questionnaire therefore lies not only in the questions it asks, but in the care with which it is designed, applied and acted upon.
