07 AI-assisted survey instrument design

Legitimate assistance and firm limits

Lower-risk uses include brainstorming constructs and candidate items, identifying jargon, checking reading burden, generating alternative phrasings, suggesting response options, mapping questionnaire logic, drafting test cases and comparing a draft against a documented style guide. AI may support translation or cognitive-testing preparation by producing alternatives and probes, but qualified translators, bilingual reviewers and target-language participants must evaluate them.

Do not ask a model to produce a ‘validated questionnaire’ from a short prompt. It may invent scales or citations, alter construct boundaries, introduce double-barrelled or leading wording, omit important subdomains and reproduce cultural or demographic bias from its training data. Small changes to prompts, model versions or provider systems can change outputs. AI-generated respondent data are not a substitute for sampled human responses; current evidence shows instability and reduced variation that undermines statistical inference.

Pitfalls and guardrails

• Hallucination and provenance: verify every claimed source against the original publication. Record which wording came from existing instruments, human authors and AI suggestions.
• Construct drift: maintain a construct map and item rationale. Reject fluent alternatives that shift time period, population, intensity or conceptual domain.
• Leading and cultural bias: review assumptions, polarity, examples, idioms and response options with diverse experts and members of the target population.
• Confidentiality and data protection: do not paste participant data, unpublished instruments or sensitive project information into an unapproved service. Check retention, training use, sub-processors, international transfers and contractual terms.
• Intellectual property: check whether suggested wording reproduces protected instruments and whether the tool’s terms permit the intended use. Do not infer licence from model output.
• Reproducibility: retain the provider, model/version, date, system settings, full prompts, supplied context, outputs, selections and human edits. Re-test after material model changes.
• Accessibility: apply plain-language and inclusive-design review, then test with assistive technologies and disabled participants. AI readability scores alone are insufficient.

A practical human-in-the-loop workflow

1. Write the research questions, construct definition, target population, mode and evidence standards without AI.
2. Inventory relevant validated measures and confirm licences. Prepare a data-minimised, approved workspace.
3. Give the model a bounded task, for example flag jargon or generate alternatives for one human-drafted item, rather than requesting a complete instrument.
4. Require citations where relevant, then verify each one independently. Compare suggestions against the construct map and questionnaire specification.
5. Use expert appraisal to remove biased, leading, inaccessible, culturally narrow or unsupported material. Keep the rejected as well as accepted outputs in the audit trail.
6. Conduct cognitive interviews with the target population, test translations and accessibility, and pilot the programmed instrument in every intended mode.
7. Revise from empirical findings. A human instrument owner signs off the final wording and documents AI use, model/version, prompts, review and limitations.

Evaluation and accountability

Pre-specify how the AI-assisted process will be judged. Relevant evidence may include content validity, cognitive-interview findings, test–retest reliability, differential item functioning, subgroup error analysis, translation equivalence and stability across repeated runs. Keep evaluation material separate from examples supplied to the model. If AI interacts with respondents, codes open text or changes routing, treat it as part of the measurement system and evaluate the entire system.

Assign a named human owner and a decision point before any instrument, classification or claim is released. Tell respondents when AI materially affects their interaction or data handling. Report the purpose, provider, model/version, date, prompts or rules, data supplied, validation, human review and known limitations sufficiently for another team to understand what was done.

Summary of key points

• Use AI for bounded assistance, never as evidence that an instrument is valid.
• Make hallucination, bias, privacy, intellectual-property and reproducibility checks explicit.
• Require expert review, cognitive interviewing, accessibility testing and piloting with the target population.
• Keep a versioned audit trail and accountable human approval.

Further reading

• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. 2024.
• Bisbee, J. et al. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 2024.
• U.S. Census Bureau. Leveraging Generative AI for Quality Assurance in Survey Questionnaire Development and Instrument Programming. Federal CASIC Workshops, 2026.

References

• American Association for Public Opinion Research. Responsible AI Integration in Survey Research. 2026.
• Information Commissioner’s Office. Guidance on AI and data protection.
• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023.
• American Association for Public Opinion Research. Transparency Initiative disclosure elements. Current guidance.