05 Identifying the variables for survey research

In this chapter

By the end of this chapter, you should be able to:-

• describe different methods for identifying the information to be gathered in a survey;
• list the different types of variables that may be included;
• apply a range of techniques for identifying and selecting relevant variables;
• distinguish between conceptual and operational definitions of variables;
• identify different levels of measurement and the implications for data collection and analysis;
• identify sources of survey questions.

Approaches to determining questionnaire content

In Chapters 1 and 2, we have already emphasised the need to ensure that we are indeed measuring what we intend to measure (the principle of validity), and that the information we collect should be determined by our research aims and objectives. There is no single way of identifying data requirements; rather a combination of  the following is likely to be required:-

• Literature review  – we need to determine the current state of knowledge on the topic of interest, and to be aware of existing theories and previous empirical findings.  For example, for a survey of patient satisfaction, the knowledge that other researchers have found a positive correlation between age and level of satisfaction would suggest that we need to collect data on age to test whether this observed relationship holds true in our population and whether any differences in satisfaction that we find, compared with other studies, could be explained by age differences.
• Brainstorming – a  ‘brainstorming’ session by the sponsor, survey team and others interested in the topic can be a useful source of ideas for items that might be included in the questionnaire.  However, it is important to guard against including items or questions on the grounds that ‘it might be interesting to know…’; there should be a clear picture of where each piece of data fits into the overall plan of analysis, and contributes to addressing the research aims and objectives.
• Qualitative research – particularly in researching ‘new’ topics or populations, qualitative research, in the form of unstructured interviews or focus groups with members of the target population, can be a very fruitful way of identifying relevant issues and concepts and of ensuring adequate face and content validity.  For example, in developing measures of health status and quality of life, it is only by asking people with the condition in question about the impact it has on their lives that we can be sure that we are measuring what is important to them, rather than to health professionals. Talking to members of the target audience also allows the researcher to become familiar with the typical vocabulary of the subjects of the survey, thereby allowing the questions to be worded in an appropriate way.
• Review of concepts and attributes to be measured – the usefulness of results depends not only on successfully measuring outcome variables (for example ‘quality of life’ or ‘satisfaction with services’), but also on being able to compare key sub-groups and to provide classificatory detail about sample members which is likely to affect outcomes. We must ensure that appropriate questions are included to measure not only all the relevant outcomes, but also the concepts and variables needed to classify sample members in relevant ways.

Determining the variables to be included

As a very crude rule, there is an inverse relationship between the quantity of data collected in a survey and the quality of those data.  One reason for this is that there is a limit to the time and effort that respondents are prepared to devote and, if overloaded, they will start looking for short cuts, to the detriment of data quality. Collecting data which are not subsequently analysed is a waste of resources and is ultimately unethical.  However, surveys can also founder if some data items vital to the research aims and objectives are not collected. 

In identifying what variables to include, the survey researcher needs to think about:-

• Dependent variables  – predicted, ‘outcome’ variables (e.g. rates of accident at work).
• Independent variables – predictor, ‘input’ variables (e.g. age & gender of worker).
• Confounding variables / factors  – these are associated with both independent and dependent variables; confounding variables may obscure the true relationship between an independent and dependent variable (e.g. occupation could be a confounding factor in examining the relationship between gender and accidents at work, if women work in less dangerous occupations and sectors).
• Modifier variables  – these specify particular conditions for a relationship between an independent and dependent variable; a modifier variable may ‘modify’ the relationship between an independent and dependent variable (e.g. an observed relationship between age and accidents at work may be modified by length of employment – younger workers may be more prone to accidents, but only if they are ‘new’ to the job).
• Intervening variables – a variable may intervene such that there may be an indirect relationship between the independent and dependent variables (e.g. older workers may be observed to be less prone to accidents because they are more likely to use safety equipment).  Note that it is often quite difficult to distinguish between intervening and confounding variables.
• Universal variables – those variables which are so often of relevance (generally as independent variables) in studies of groups or populations that their inclusion should be automatically considered (e.g. gender, age, ethnic group, marital status, socio-economic status).
• Time-defining variables – these are particularly relevant in longitudinal or panel surveys (e.g. values of a measure at Time 1, Time 2 etc), but date of response may be of interest even in cross-sectional studies.
• Population-defining variables – variables which characterise the study sample, and allow the comparisons of respondents and non-respondents (as noted above, demographic and socio-economic characteristics are so often of relevance to merit consideration for inclusion in any survey).

To ensure that adequate but not excessive quantities of data are collected, a number of techniques can be used to highlight data requirements:-

• Inventory of variables – using the approaches described in the previous section, draw up a list of potential items for inclusion.  This should include dependent and independent variables and measures of intervening, modifying and confounding factors.

• Construction of hypothetical path diagrams – since the relationships between independent and dependent variables are likely to be highly complex, with many possible intervening, modifying and confounding factors, path diagrams, in which the nature of these complex relationships is shown by arrows, can help to clarify what is going on.  At this stage, a decision might be taken to omit some variables, perhaps because the anticipated relationship appears to be so complicated that it would be too difficult to disentangle within the design limitations of the study.
• Tabular representation of hypotheses – constructing path diagrams helps to specify in tabular form the hypotheses to be tested.  In a tabular representation, the dependent or outcome variables are usually shown as columns and the independent variables as rows.  A plus sign in the table indicates a hypothesised positive relationship between independent and dependent variables; a negative sign indicates a negative relationship; a zero indicates a hypothesis of no association; a question mark indicates that we have no prior ideas about what to expect. If we find that, for certain variables, most of the entries are zeros or question marks, we might consider dropping those variables on the grounds that they are not central to our analyses.
• Construction of dummy tables – a further refinement of representing hypotheses in tabular form is to draw up dummy tables just as they will appear in the survey report, but without any figures in them.  This can help to ensure that data are collected on all relevant variables and in such a way as to allow the statistics of interest to be extracted or calculated (for example, if we indicate that a table of mean age is to be displayed, this will remind us to collect disaggregated data, rather than simply recording age-group).

Defining variables

Once we have identified which data we wish to collect, we need to come up with definitions of the associated data items.  We can distinguish between conceptual and operational definitions of variables / data items.

• Conceptual definition – the data item as we conceive it, a definition of the characteristic we would like to measure (e.g. daily alcohol consumption; occupational status)

• Operational definition – a ‘working’ definition, a definition of the characteristic we will actually measure (e.g. amount of alcohol drunk yesterday treated as a measure of daily consumption. How do we operationalise ‘occupation’ for those who are not gainfully employed at present?)

There are a number of inherent risks in ‘translating’ conceptual definitions into operational definitions that can be encompassed in questions. 

First, how can we ensure that the operational definition of the researcher matches that of the respondent?  Can we pose the question, and accompanying response categories and instructions on how to answer, in such a way as to convey our meaning clearly and unambiguously? Can we define and quantify events or behaviour in a way that is sufficiently precise, complete and consistent, yet is comprehensible to each respondent, means the same to all respondents and is conducive to reliable response. An example where the conceptual and operational definition of events is problematic is ‘accidents’. An example where problems of defining concepts, problems of defining countable units and problems of survey enumeration all interact is ‘physical exercise of type and intensity likely to improve cardio-vascular health’. Other problems of devising a satisfactory operational definition of an event or behaviour of interest are illustrated in Table 3.

Second, even if we can convey our meaning and operational definition clearly to the respondent, we may still end up substituting what we can measure (a proxy or surrogate data item) for what we really would like to measure (e.g. past behaviour as a surrogate for future behaviour; behaviour on a particular occasion as a proxy for habitual behaviour).  Both of these risks provide threats to our aim of collecting data that are valid, reliable and free from bias. We will return to the challenges of wording questions so that the operational and conceptual definitions match as closely as possible, and are adequately conveyed to the respondent, in Chapter 6.

Table 3 Examples of definitional and quantification problems encountered in surveys of behaviour and events

Types of data

As well as distinguishing between different types of variables, it is helpful to think about types of data and levels of measurement.

With respect to types of data, we can distinguish between:

• Binary / dichotomous variables – only two possible values are allowed (e.g. Yes / No; True / False; Male / Female)
• Categorical variables – more than two values are possible, but the categories are discrete, finite in number and often unordered (e.g. Single / Married or co-habiting / Divorced or separated / Widowed; Agree strongly / Agree / Uncertain / Disagree / Disagree strongly – this latter set of categories is in fact ordered)
• Continuous variables – infinite number of values possible (e.g. age).  Continuous data may be Normally or non-Normally distributed.

Classically, we may also make distinctions between four different levels of measurement.  These are distinguished by properties of ordering and distance.  The four levels are:-

• Nominal – objects are either the same or they are different; a typical application is classification (by gender, area etc);
• Ordinal – objects are assessed to be greater or smaller, better or worse etc, in other words there is some comparative component in terms of order; typical examples of ordinal level data are rankings, preferences and many classifications of status;
• Interval – as well as there being an ordering of ranks, the distance or interval between  adjacent ranks of a scale are equal; typical examples are attitudes measured on a Likert scale (e.g. strongly disagree – disagree – agree – strongly agree)
• Ratio – there is a meaningful zero, so comparisons of absolute magnitude are possible e.g. A costs twice as much as B, C is half the size of D; income, height and weight are all examples of ratio level data (but temperature in degrees Fahrenheit or Celsius is not – why not?)

Table 4 provides a summary of the properties of each level of measurement.  The appropriate statistical techniques are determined by the level of measurement, as summarised in Table 5; techniques appropriate to data collected at a lower level of measurement can be applied to data collected at a higher level, but the opposite does not hold true!

Note that higher level data can be ‘transformed’ to lower level data by grouping procedures (e.g. collapsing age in years into five-year age groups, thereby reducing ratio level data to interval level data).  Collapsing data in this way may mean that certain statistical techniques are no longer appropriate.  Note also that while it is always possible to collapse or aggregate data, it is not possible to disaggregate data collected at nominal, ordinal or interval level.

Table 4 Properties of levels of data

Table 5 Statistical techniques appropriate to different levels of data

+ denotes that statistics applicable to a ‘lower’ level of measurement are also appropriate – for example, that both the mode and the median may be used in respect of ordinal data, the mean, median and mode may all be used for interval and ratio data.

Choosing a suitable scale of measurement

There is always potential for conflict between the needs of the survey researcher (for completeness and detail) and of the survey respondent (to be able to respond easily).  The choice of level and scale of measurement also determines what statistical techniques can be used, as shown in Table 5 above.  In choosing the scale and level of measurement, it is important to keep a number of principles in mind:

• Appropriateness – the scale and level of measurement must be appropriate and relevant in terms of the research questions, and the conceptual and operational definitions of the data item.  It should yield sufficient detail for the study, but excess detail should be avoided; for example, it may be adequate to group ‘divorced’, ‘separated’ and ‘widowed’ together as a category distinct from ‘currently married or living as married’ and ‘single (never married)’.
• Practicability – the scale and level of measurement should be geared to the context and data collection method; for example, trying to collect disaggregated data on exact income may not be acceptable to respondents, though they may be willing to report income group.
• Sufficient power – the level of measurement should be such that the desired statistical analyses can be carried out (see table 5 above).
• Clearly defined categories – if necessary, operational definitions for the categories must be specified; for example, does ‘third level education’ include sub-degree courses such as Higher National Diplomas and/or vocational or professional training (e.g. nursing) undertaken post-secondary school?
• Sufficient categories – compression of responses into too few categories may lead to a loss of discriminatory power.  But avoid spurious distinctions that are, for practical purposes, not relevant or meaningful (see appropiateness above).
• Collective exhaustiveness – every possible value / response must be catered for.
• Mutually exclusivity – each response must fit unambiguously into a category.

Sources of survey questions

In Chapter 6, we discuss development and wording of questions and their associated response categories.  Each survey needs to be carefully tailored to its purpose and to the type of respondent to whom it is addressed. However, many social surveys will be addressing issues and concepts which have been the subject of previous research and surveys and it is important to avoid ‘re-inventing the wheel’. Apart from the advantages of drawing upon the expertise and experience of others, developing and refining new questions, and ensuring that they are valid and reliable, is time-consuming and expensive. In many circumstances, use can be made of existing well-validated questions or even whole questionnaires. On the other hand, the existence in the literature of some set of questions which has a relevant-sounding label does not guarantee that it is appropriate to this population and this study.

The UK Data Service

The UK Data Service provides a suite of online research resources with a specific focus on survey methods and can be used to locate, and view in context, survey questions as they were used in the data collection process. The resource can be used to assist with the design of new questionnaires, the search for questions/variables for secondary analysis, and the teaching of survey research methods. See  http://discover.ukdataservice.ac.uk/variables and

http://ukdataservice.ac.uk/get-data/key-data.aspx

ONS harmonised questions

For some years, the UK Office of National Statistics has been leading an initiative to standardise “… concepts, definitions and questions (known as 'inputs'), as well as the way in which the results are released ('outputs').” This is known as harmonisation. Details of the harmonised concepts, classifications and questions – which relate in the main to socio-demographics – may be found at

https://www.ons.gov.uk/methodology/classificationsandstandards/harmonisationwithinthegss

Questions from US polls and surveys

The Odum Institute at the University of North Carolina provides access to a comprehensive database of questions from US public opinion polls and surveys.  It can be searched, using a very simple interface, for questions on a myriad of topics.  Access it at http://odum.unc.edu/archive/ 

The U.S. National Center for Health Statistics publishes current surveys and data-collection systems for epidemiological and health-services research; the NCHS surveys overview links to individual programmes and documentation.

Core outcomes

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative brings together people interested in the development and application of agreed standardised sets of outcomes, known as ‘core outcome sets’. These sets represent the minimum that should be measured and reported in all clinical trials of a specific condition, and are also suitable for use in clinical audit or research other than randomised trials.  The database of measures is being added to on an ongoing basis – see http://www.comet-initiative.org/

Questions on health status and quality of life

Many surveys conducted in the area of public health or health services research have as their key outcome variables to be measured the ’general health status’ or ‘quality of life’ of respondents. As used in everyday discourse, neither of these concepts is very well-defined and neither is easy to measure in ways which meet the criteria of validity, reliability, lack of bias, discriminatory power and responsiveness to change over time.

There is nevertheless a wealth of instruments, with established validity and reliability, for measuring health status, both in general populations and in specific disease- and age-groups.  Rather than trying to develop and test a new set of questions, the survey researcher should first review these existing instruments and see whether any are appropriate to the aims and objectives of the planned survey.  Comprehensive reviews of instruments for measuring health status and quality of life have been produced by McDowell and Newell (1987), Wilkin and colleagues (1992) and Bowling (1994,1997). 

The MAPI Research Institute has constructed an electronic database (PROQOLID) with details of instruments to measure health status and quality of life, both in general populations and in specific disease groups.  Two levels of access are available.  There is free access to basic details of the instruments; access to further details requires payment of a membership fee.  The database is at https://eprovide.mapi-trust.org/  .

Of course, instruments should not be chosen unthinkingly – they need to be evaluated against explicit criteria to ensure that they are appropriate to the survey’s aims and objectives.  It is also important to remember that validity and reliability may need to be re-established if an instrument is used in a setting other than that for which it was developed (for example, if an instrument originally intended for application in a secondary care setting is to be used in primary care).  Further details of criteria by which instruments should be assessed are provided by Bentzen and colleagues (1998).

Questions on patient satisfaction

With increasing emphasis on patient and consumer involvement in health care, a number of questionnaires and related tools to measure patient satisfaction have been developed and validated in recent years.  In the UK, there has been particular emphasis on the development of instruments to measure satisfaction with primary care services.  The National Centre for Primary Care Research and Development developed the General Practice Assessment  Questionnaire (GPAS), which had the advantage of having national benchmark data, based on responses of almost 30,000 patients.  However, with the advent of the new GP contract, GPAS was superseded by a new questionnaire, GPAQ.  Since 2009, GPAQ has in turn been superseded in the Quality and Outcomes Framework by the GP Patient Survey – see http://www.gp-patient.co.uk/. 

Particular care is needed in selecting and adapting satisfaction questionnaires.  Remember that the organisation and delivery of care varies significantly from country to country.  Concepts and vocabulary appropriate in the US may not be relevant in the UK.

Summary of key points

• In determining the focus and content of a survey, one or more of the following techniques may be useful; literature review; brainstorming; qualitative research; review of concepts and attributes to be measured.
• In identifying the variables on which data are to collected, it is useful to identify and distinguish between: dependent variables; independent variables; confounding, modifying and intervening factors; ‘universal’ variables; time-defining and population-defining variables.
• Techniques to aid the collection of adequate but not excessive data include: compilation of an inventory of variables; construction of hypothetical path diagrams; drawing up of tabular representations of hypotheses; construction of dummy tables of results.
• It is important to distinguish between conceptual and operational definitions of variables; care must be taken in operationalising concepts.
• Variables may be classified as binary, categorical or continuous.
• Levels of measurement – nominal, ordinal, interval, ratio – are defined by properties of ordering and distance; the level of measurement has implications for statistical analysis.
• In choosing a scale of measurement, principles of appropriateness, practicability, power and category definition need to be taken into account.
• For some surveys, the time-consuming and costly process of developing and refining questions can be avoided by using already validated questions and scales. However, care must be taken in selection and application of these items and measures.

Further reading

• The section on Measurement in the Research Methods Knowledge Database http://www.socialresearchmethods.net/kb/measure.htm is good on levels of measurement – see http://www.socialresearchmethods.net/kb/measlevl.htm.
• The ONS provides detailed resources on harmonised questions at https://www.ons.gov.uk/methodology/classificationsandstandards/harmonisationwithinthegss
• A series of papers on measuring a range of key constructs, including ethnicity, income, physical activity in older people and alcohol consumption in youth can be found at https://ukdataservice.ac.uk/search/ (enter ‘thematic guides’ into the search box)
• A similar resource for US federal surveys may be found at Q-Bank - https://wwwn.cdc.gov/qbank/ 

Current guidance and methodological literature for 2026–27:

• Government Statistical Service. Harmonisation methodology and survey development toolkit. current guidance.
• DDI Alliance. DDI Common Core. Version 1, 2025.

References

Bentzen N, Christiansen T, and Meadows K. Selection and cross-cultural adaptation of health outcome measures. European Journal of General Practice 1998;4;27-33.

Bowling, A. Measuring disease: a review of disease specific quality of life measurement scales. Buckingham: Open University Press, 1994.

Bowling A.  Measuring health: a review of quality of life measurement scales (2^nd edition).  Milton Keynes: Open University Press, 1997.

McDowell I and Newall C.  Measuring health: a guide to rating scales and questionnaires.  Oxford: Oxford University Press, 1987.

Wilkin D, Hallam L and Doggett AM.  Measures of need and outcome in primary health care.  Oxford: Oxford Medical Publications, 1992.

Zeinio RN. Data collection techniques: mail questionnaires.  American Journal of Hospital Pharmacy  1980;37:1113-1119.