10 Maximising response rates

In this chapter

By the end of this chapter, you should be able to:-

• define unit and item response rates;
• state why high response rates are important;
• identify potential reasons for unit and item non-response;
• describe and apply measures to enhance unit and item response rates.

Definition and analysis of unit and item response

As already noted, maximising response rates and minimising non-response bias are important factors in ensuring the quality of survey data.  Two forms of non-response need to be considered separately for some purposes. The term unit non-response refers to situations where a case (unit or respondent) selected for the survey sample fails to respond at all. The term item non-response refers to situations where the unit responds, but fails to provide any, or satisfactory, answers to one or more questions of parts of questions (items).

Response rates may be defined as follows:-

A quoted response rate should be a measure of the success of the survey in representing the target population. The number eligible to respond should include all whom it was ideally intended to cover, including: those who could not be contacted (possibly because of sampling frame defects); those who were inaccessible, unavailable or too ill to take part; and those who declined to take part, as well as those who responded. Only  those who produced usable responses should be counted as respondents. Those who were contacted and found to be ineligible according to study inclusion / exclusion criteria should be excluded from both numerator and denominator. In the case of postal surveys, there may be no direct way of knowing how many of the cases that failed to respond were ineligible and it may therefore be necessary to estimate the number of ineligibles in the denominator. The assumptions made in arriving at such estimates should be stated.  It should not be forgotten that, in addition to unit non-response, there may be sampling frame coverage deficiencies (sampling is covered in Chapter 12). However, these should be considered separately from response/non-response.

Reasons for unit non-response

The principal reasons for unit  non-response are:-

• sample member unable to respond – for example, too infirm, cognitively impaired, blind, illiterate, non-competent in language of survey;
• incomplete/inaccurate address, or sample member no longer at contact address – this is related to the quality of the sampling frame;
• refusals – in self-completion surveys, these may be explicit (for example, those who ring up and say they don’t wish to participate or send a letter to that effect),  or implicit (for example, those who return the questionnaire blank or simply do not return it at all);
• away from home for duration of survey;
• not at home when called on – this is not usually a major problem in postal surveys unless using recorded delivery, but may be a significant source of non-response in interviewer-administered surveys (in such surveys, the contact protocol should involve repeated attempts to make contact, on different days of the week and at different times of the day).

Bias in results due to unit non-response (i.e. non-response bias) must be clearly distinguished from response bias (a form of measurement error), which occurs when a response is received but is unsatisfactory or invalid for the purposes of the survey (e.g. social desirability bias – see Chapter 6). 

In survey research (both interviewer-administered surveys and self-completion questionnaires), non-respondents often differ significantly from those who do respond, in respect of a range of characteristics.  The characteristics concerned are in turn often likely to be associated with what the survey is trying to measure. For example, heavy drinkers are likely to be unavailable (because often away from home) or unmotivated to respond to a survey of alcohol consumption. Since heavy drinkers are often younger men, a deficiency of younger men in the achieved sample may indicate likely non-response bias.

Empirical evidence shows that respondents to postal surveys, particularly those returning their questionnaires early, are likely to be more interested in the survey topic, to make more favourable reports and to have been more successful in various ways than non-respondents.  The exact respondent characteristics that drive a tendency not to respond are harder to establish. Kanuk and Berenson (1975) concluded that the only consistent and widespread finding was that respondents to postal surveys tend to be better educated and therefore to have greater facility in writing. In respect of interview surveys, Goyder (1987) demonstrated that survey response rates (given that initial contact has been made) tend to be positively correlated with socio-economic status and negatively correlated with age. In health surveys of the general population and of specific patient groups, it has been shown that non-respondents to postal questionnaires are more likely to be in semi-skilled or unskilled manual occupations and to be from ethnic minorities, while respondents tend to be younger, have high levels of educational attainment, and have better health status (Cartwright, 1986); this latter finding is of particular relevance when the aim is to measure health status in the underlying population, since estimates derived from survey respondents are likely to be upwardly biased. 

These considerations, rooted in the nature of the target population, should be taken into account when assessing whether (say) a postal approach is likely to result in low response rates and consequent non-response bias. The characteristics of the target population cannot be changed, but evidence such as that just quoted provides strong arguments for keeping questionnaires short and simple and making them attractive and as easy as possible to fill in. These latter are factors within the survey designer’s control. Another thing that can be done is to keep achieved response numbers up to target by allowing in advance for losses due to non-response (see below).

A question often asked by students is  ‘what is the minimum “acceptable” rate of unit response?’. There can be no answer to this that fits all cases. For example, if a key survey aim is to provide an estimate of the number of seriously disabled people living in a local authority area, the fact  that both non-response and disability are known to be associated with being elderly should warn us that even a relatively low rate of non-response may cause serious bias. On the other hand, the same response rate to a survey that asks people whether they have ever visited certain European countries might be considered satisfactory, since there is unlikely to be any close link between factors influencing response and which countries a person has visited. Unfortunately it is more often than not a plausible hypothesis that there is an association between propensity to respond and what the survey is intended to measure. Therefore a response rate below (say) 50% is always a cause for concern, while even response rates as high as 80% may, in some cases (see the example of disability just given) still leave scope for serious non-response bias. For these reasons careful thought is always needed in setting response targets and applying all possible means of obtaining acceptable response from key groups.

Compensating for unit non-response

A 100% response rate is rarely (if ever) achieved. To achieve the numbers required in the responding (achieved) sample, the survey researcher therefore needs to estimate likely response rates and to over-sample accordingly.  For example, if sample size calculations (as discussed in Chapter 12) indicate that an achieved sample of 500 is required, and a response rate of 60% is expected (from experience in previous similar surveys, and / or from a rehearsal pilot), it will be necessary to send questionnaires to \(500 \div 0.6 \approx 833\) individuals. 

This will help to ensure that the achieved sample is big enough, but will do nothing to remove non-response bias. 

When a postal survey is addressed to members of the private household population and uses the Postcode Address File (Small Users) (PAF) as the sampling frame, it is also necessary to allow for the fact that, on average, about 88% of PAF addresses are domestic residences, the rest being small businesses, empty etc.  Failure to do this does not cause bias, but will cause the target achieved sample size to be undershot, even if the rate of response from eligible addresses has been accurately estimated. Therefore, using the numbers calculated above, it would be necessary to select \(833 \div 0.88 \approx 947\) addresses.

As well as over-sampling in anticipation of such losses, it is also important to address non-response bias, though the number of ways of doing this is usually limited. One way is to use statistical information that may be available or inferred about the total survey population, so as to compare the distribution of respondents on relevant variables with that of the population. This may give pointers to a likely source and direction of bias (for example, where respondents are found to be older, on average, than the known average age of the target population). If there is significant non-response bias, it may be necessary to adjust or weight responses to take account of this, but weighting generally does not remove all bias (because it essentially involves ‘replacing’ non-respondents with respondents).

A number of factors influence response rates (McColl et al., 2001; Edwards et al, 2002; 2009) – see Tables 10 and 11  Some are structural and largely beyond the control of the survey researcher.  The salience of the topic (how relevant and interesting it appears to be to the respondents) is a major determinant – the more salient, the better the response rate; thus for example, surveys on topics of personal relevance such as health generally get better response rates than those on more general topics. As a general rule, longer questionnaires get poorer response rates, but the strength of this relationship may depend on the topic of the survey.

Sponsorship and endorsement (the identity of the organisation(s) commissioning and carrying out the survey) are also important (McColl et al. 2001; Edwards et al., 2002, 2007).  Response rates are generally higher for surveys by public and voluntary sector organisations such as national or local government, charities etc and by academic departments than they are for surveys conducted by commercial and market research organisations.  In a survey of health service professional staff, an endorsement and appeal for co-operation in the covering letter from (say) a Chief Medical Officer is likely to raise response (unless the topic happens to be a contentious one between CMO and staff).

The population under investigation is also a determinant – surveys of specialist populations such as employees, patients and students generally have better response rates than those of general populations.

There are a number of other factors that may affect rates of response and can be manipulated by the survey researcher.  Table 10 provides a summary of suggested means of maximising response rates. Brown and colleagues (1989) drew on these suggestions to produce a task-analysis model of respondent decision-making, summarised in Table 11, which is useful in designing and implementing questionnaire surveys. Research evidence for the effectiveness of some of the proposed methods (e.g. personalisation of signature) is equivocal.  Two relatively recent literature reviews (McColl et al., 2001; Edwards et al., 2002; 2009) of studies in which one or more factors postulated to affect response rates to postal surveys, and open to manipulation in a survey, were manipulated in randomised controlled experiments concluded as follows:

Timing of survey

• Response rates do not appear to be affected by the day of posting (McColl et al., 2001).
• The month of posting may affect response rates (December and the peak summer holiday period may result in lower response rates), but this effect may be topic-specific (McColl et al., 2001).

Number and relative timing of contacts

• Response rates can be increased through multiple contacts with respondents (McColl et al., 2001; Edwards et al., 2002; 2009).
• Both pre-notification and follow-up contacts are effective in stimulating response rates (McColl et al., 2001; Edwards et al., 2002; 2009); if only one of these strategies is to be used, follow-up may be more powerful (McColl et al., 2001).

Pre-notification contacts

• Pre-notification is effective in increasing response rates (McColl et al., 2001; Edwards et al., 2002; 2009).
• Pre-notification by letter may be more effective than pre-notification by telephone (McColl et al., 2001).
• High involvement methods of pre-notification (e.g. foot-in-door approaches in which the respondent is first asked to comply with a simple task) have not been shown conclusively to improve response rates over simple pre-notification; such high involvement approaches are really only feasible where telephone or personal approaches to pre-notification are made (McColl et al., 2001).

Follow-up contacts (reminders)

• Follow-up contacts are highly effective in increasing response rates McColl et al., 2001; Edwards et al., 2002; 2009).
• There is no conclusive evidence that a threat of further follow-ups made in a reminder letter enhances response rates in all circumstances McColl et al., 2001; Edwards et al., 2002; 2009).
• Inclusion of a replacement questionnaire is effective McColl et al., 2001; Edwards et al., 2002; 2009); there is some evidence to suggest that it may be more appropriate to include the duplicate questionnaire with a second rather than a first reminder (McColl et al., 2001).
• There is no conclusive evidence that special mailing techniques for final reminders are superior to standard mailing (McColl et al., 2001). Postcard reminders appear to be as effective as letters and are generally cheaper (McColl et al., 2001); however, there may be concerns of confidentiality in health surveys.

Nature of contact and covering letter

• In the one study identified by Edwards and colleagues (2002; 2009) in which a request for an explanation about not participating was manipulated, a significantly higher response rate was obtained when an explanation for non-participation was requested. (Note however that research ethics committees may point out that participation in research is voluntary, and that non-respondents need not give a reason for their decision.)
• Explicitly giving a choice to opt out of the survey resulted in a significantly lower response rate (Edwards et al., 2002; 2009)
• Traditional style letters are more effective than novel approaches (McColl et al., 2001).
• There is little conclusive evidence that the characteristics of the signatory affect response rates (McColl et al., 2001; Edwards et al., 2002a, 2002b).
• Response rates do not appear to be positively related to hand-written signatures (McColl et al., 2001)
• There is some evidence from the review by Edwards and colleagues (2002; 2009) to suggest that coloured ink results in higher response rates than blue or black ink, but whether this relates to the body of the letter or the signature is unclear.
• No one type of appeal in the covering letter offers a consistent advantage (McColl et al., 2001; Edwards et al., 2002a, 2002b); rather the nature of the appeal should be matched to the anticipated motivations of the recipients.

Postage rates and types

• The review by Edwards and colleagues (2002; 2009) concluded that recorded delivery resulted in higher response rates than standard delivery; however, the cost-benefit relationship was not examined.
• Findings both from primary studies identified by McColl and colleagues (2001) and from previous reviews showed no consistent advantage of class of mail.  However, the most recent review by Edwards and colleagues (2002; 2009) indicated that first class outward mail offered a small advantage over other classes of mail  Whether such a small advantage is cost effective is unclear.
• Likewise, findings both from primary studies identified by McColl and colleagues (2001) and from previous reviews showed no consistent advantage of class of mail.  However, the most recent review by Edwards and colleagues (2002; 2009) indicated that  stamped returned envelope increased response rates relative to franked or business reply envelopes.  Note however, that neither this review nor the majority of primary studies examined the cost-effectiveness of using stamps (where the cost is incurred regardless of whether the questionnaire is returned) versus using business reply-paid envelopes (for which a charge is made only if the envelope is returned).
• No other statistically significant and consistent effects of postage style (including: questionnaire sent to work versus home address; colour of envelope; pre-paid return envelope versus not pre-paid; stamped versus franked outward envelope; commemorative versus ordinary stamp) were found in either of the most recent reviews (McColl et al., 2001; Edwards et al., 2002;  2007).

Confidentiality / anonymity

• Assurances of complete anonymity do not significantly improve response rates and may indeed have a detrimental effect (McColl et al., 2001).  A fully anonymous questionnaire (i.e. one in which there is no identifier whatsoever, as opposed to a confidential questionnaire, in which a unique identifier is used) makes it difficult to send out reminders.

Personalisation

• Edwards and colleagues (2002; 2009) found a significant effect of personalisation
• It appears (t al., 2001), however, that personalisation may interact with such factors as the nature of the appeal made in the covering letter and assurances of confidentiality.

Time cues and deadlines

• A short time cue can be effective in stimulating responses (McColl et al., 2001).
• Specification of a deadline for responding may increase the speed of response (and thereby reduce the number of reminders needed), but may have no effect on overall response rates (McColl et al., 2001; Edwards et al., 2002; 2009).

Sponsorship

• A university as sponsor or source may result in higher response rates (Edwards et al., 2002; 2009)
• However, the impact of sponsorship appears to be situation- and location-specific (McColl et al., 2001).

Saliency

• A salient (interesting, relevant and current) topic is effective in enhancing response rates (McColl et al., 2001; Edwards et al. 2002; 2009).

Incentives

• Incentives are generally a highly effective means of increasing responses (McColl et al., 2001; Edwards et al. 2002; 2009).
• Financial incentives are likely to be more effective than non-monetary incentives of similar value (McColl et al., 2001; Edwards et al. 2002; 2009).
• Enclosed incentives (non-conditional) are more effective than promised incentives (McColl et al., 2001; Edwards et al. 2002; 2009).

Feedback of results

• Offering feedback of survey results is generally not effective in stimulating response (McColl et al., 2001).

Miscellaneous

• Personal drop-off of questionnaires for self-completion may offer some advantages but the cost-effectiveness of this approach may be situation-specific (McColl et al., 2001).

Despite mixed findings, what is apparent is that there is no single method of enhancing response rates which is applicable in all settings. Indeed, manipulation of a single factor is unlikely to prove fruitful.  Instead the researcher should consider the total ‘package’ of questionnaire wording, questionnaire appearance, general motivational factors (anonymity / confidentiality; personalisation; nature of appeal; other aspects of covering letter; sponsorship; saliency); mechanical and perceptual factors (timing of survey; number, timing and method of contacts; postage rates and types); financial and other incentives.  The choice of techniques should also be informed by consideration of the likely barriers and motivational factors for each particular survey topic and study population. In assessing potential methods, the researcher should consider not only the likely impact on response rates, but also the potential for non-response and sample composition biases, response bias and item non-response effects, as well as implications for resources of time, money, personnel and materials (most of these factors were ignored in the primary studies reviewed by McColl and colleagues (2001) and by Edwards and colleagues (2002; 2009)). Be aware that the marginal benefits of intensive approaches to enhancing response rates (e.g. stamped return envelopes, recorded delivery, incentives) may be outweighed by the marginal costs; to date the focus of strategies to enhance response rates has been on effectiveness but not cost-effectiveness.

Table 10 Means of achieving objectives in maximising response rates (after Dillman, 1978)

Table 11 Task-analysis model of respondent decision-making (after Brown, Decker, et al., 1989)

Reasons for item non-response

Item response (failure to answer particular questions) is similar in its general effects to unit non-response. It is not generally as damaging as unit non-response (unless it affects key survey questions), but if on a serious scale it can be a great nuisance and can confuse the results of a survey. It is also a warning that some respondents who did answer a question (say, a question on income) may have had similar adverse reactions as deliberate unit non-respondents (see below) and that the quality of the obtained data may be suspect.  Reasons for item non-response may be classified as:-

• accidental item non-response – as a result of:  poor questionnaire instructions/routing; pages stuck together; respondent being distracted;
• deliberate item non-response – including: refusal to respond (for example, because of amount of information required or nature of information required); inability to answer (for example, because instructions on how to respond are unclear; the question is too complex; the respondent does not have the necessary information; the respondent’s position is not catered for).

As with unit non-response, high item response rates are important because:-

• the higher the response rate, the less likely there is to be non-response bias;
• the higher the response rate (i.e. the larger the achieved sample), the more precise are inferences drawn from the sample about the underlying population.
• widely varying rates of response to different questions can cause serious technical and interpretational problems when analysing and reporting the results of the survey.

Compensating for item non-response

The fact that some information about the characteristics and situation of those who fail to respond to particular questions is available from the rest of their questionnaire responses in principle gives the researcher more chance of repairing the effects. In some cases there may appear to be a high probability, from the evidence of other information on the questionnaire, that the correct response would have been “No”” or “0”. Such a judgement might be made, for example, where the respondent has written in “I have no income other than my Old Age Pension” and then ignored further questions about income; or where the respondent had failed to give the amount of the pension, but it could be estimated by reference to the pension regulations. On very large surveys, methods of imputing missing responses by statistically matching the respondent with other similar respondents are sometimes used.  Similarly, for some established instruments (e.g. some of those to measure health status / quality of life), algorithms have been defined to impute values to missing data based on the pattern of responses for that individual to other items measuring the same construct.  Realistically, however, in most survey situations, little can be done with missing responses other than to flag them when editing the data as “missing”, so that outcome for all items on the questionnaire is accounted for.

A related problem is that of apparently inappropriate or inconsistent response – for example, when a respondent’s answer to an earlier question was such that a subsequent question should have been skipped but that question has in fact been answered (e.g. someone who claims to be a non-smoker, but gives details of the number of cigarettes smoked).  In interviewer-administered surveys, it is possible to check apparently inconsistent responses if they are noticed during the interview itself (the use of computer assisted interview techniques can be of particular value here since the computer can be programmed to flag any inconsistencies in responses).  In self-completion questionnaires, however, it is often impossible to ascertain which is the ‘correct’ answer, and it may be necessary to recode the responses to both items as missing data (issues of coding are discussed in Chapter 13).

Summary of key points

• High unit and item response rates are desirable in the interests of minimising non-response bias and increasing the precision of parameter estimates.
• In all types of survey, non-respondents tend to differ in a systematic way from respondents, resulting in non-response bias.
• A high response rate will not necessarily guarantee a lack of non-response bias, and what constitutes an ‘acceptable’ response rate may vary from survey to survey; however, response rates below 50% are always a cause for concern.
• It is desirable to identify sources of unit non-response, but this is often difficult in postal and other self-completion surveys.
• In sampling, it is necessary to anticipate and compensate for non-response.
• Theories of respondent behaviour and empirical evidence show that unit response rates are affected both by structural factors, largely beyond the control of the survey researcher (including topic, sponsorship and study population) and factors which the researcher can control and manipulate (including timing of survey, number, timing and content of contacts, postage rates and types, and use of incentives).
• Reasons for item non-response may be sub-divided into accidental and deliberate.  Good question wording, appropriate response categories and clear instructions can help to minimise the risk of item non-response.  In some cases, data coding and imputation procedures may be used to compensate for item non-response, but often little can be done about this aspect of data quality.

Further reading

Mangione TW. Mail surveys - improving the quality. Thousand Oaks: Sage Publications, 1995. (Applied Social Research Methods Series, Volume 40) (Chapters 6 and 7).

Nakash RA,  Hutton JL,  Jorstad-Stein EC, Gates S and Lamb SE. Maximising response to postal questionnaires--a systematic review of randomised trials in health research. BMC Medical Research Methodology 2006; 6; 5

• There are a number of online calculators for working out survey response rates.  Make sure that the one you choose uses the definition of response rate that you consider appropriate.  See for example  http://www.quantitativeskills.com/sisa/calculations/resprhlp.htm
• The American Association of Public Opinion Research has an interesting document on its web site entitled Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.  It should be recognised that the categories recommended in that document are those that are used in US surveys and polls.  Nonetheless, it is instructive to consider the different categories of non-response.  See https://aapor.org/standards-and-ethics/standard-definitions/
• Val Angus and colleagues have a useful paper in the public access BioMed Central journal Health Services Research on the impact of a two-stage contact process (now required in many surveys of NHS patients) on postal questionnaire response rates https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-3-21

Current guidance and methodological literature for 2026–27:

• American Association for Public Opinion Research. Standard Definitions. 10th edition, 2023.
• American Association for Public Opinion Research. Response Rates Calculator. current calculator and guidance.

References

Brown TL, Decker DJ and Connelly NA.  Response to mail surveys on resource-based recreation topics: A behavioral model and an empirical analysis.  Leisure Sciences 1989;11; 9-110.

Cartwright A. Who responds to postal questionnaires? Journal of Epidemiology and Community Health 1986;40;267-273.

Dillman DA.  Mail and telephone surveys: The total design method,  New York:  John Wiley and Sons, Inc, 1978.

Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I.  Increasing response rates to postal questionnaires: systematic review.  British Medical Journal 2002;324;1183-1192. https://pubmed.ncbi.nlm.nih.gov/12016181/

Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I and Cooper R. Methods to influence response to postal questionnaires (Cochrane Methodology Review). In: The Cochrane Library, 2009. Oxford: Update Software. https://pubmed.ncbi.nlm.nih.gov/19588449/  

Goyder J. The silent minority: Nonrespondents on sample surveys, Cambridge: Polity Press, 1987.

Kanuk L and Berenson C. Mail surveys and response rates: A literature review. Journal of Marketing Research 1975;12;440-453.

Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, Harvey E, Garratt A and Bond J. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment 2001, 5(31) Downloadable from  https://www.journalslibrary.nihr.ac.uk/hta/hta5310/#/abstract