GeroStats

Survey Methods Handbook · Chapter 14 of 15

14 Big data and web-scraped data for survey research

In this chapter…

This chapter places web scraping, APIs and digital traces within the survey lifecycle rather than treating them as a substitute for survey design.

By the end of this chapter, you should be able to…

  • decide when web data can complement a survey and when not to collect them
  • address access terms, rights, privacy, ethics and secure acquisition
  • diagnose coverage and processing error and validate web measures against survey evidence

Web pages, platform APIs, digital traces, transactions and sensor streams can complement survey data, help construct frames, describe context or validate selected measures. They are not automatically representative because they are large, and they do not replace the conceptual work of defining a population and a construct. Treat web data as a designed acquisition with its own coverage, measurement, legal, ethical and processing errors.

Decide whether web data are appropriate

Specify the research question and the role of the web data before collecting anything: exploration, frame construction, contextual measurement, linkage, calibration or validation. Define the target population separately from the users, pages or events visible on a platform. Ask what behaviour produces a trace, who cannot produce it, how ranking and moderation shape visibility, and whether the same construct could be measured more directly or with less intrusion.

Do not scrape when access would defeat controls, create disproportionate privacy risk, breach an applicable agreement, overload a service, expose vulnerable people, or collect data that cannot be validated for the intended inference. Public visibility is not the same as informed consent, ethical acceptability or unrestricted reuse.

Access, law and ethics

Prefer a documented official API or licensed data service when it provides the required fields and stable access. For scraping, examine the website or platform terms, API conditions, copyright and database rights, privacy notices and applicable data-protection law. The robots exclusion protocol in RFC 9309 is a technical signal from a service to automated clients; it is neither blanket legal permission nor a complete ethical assessment.

Ethical review should consider consent, reasonable expectations, identifiability, group harm, sensitive inference, vulnerable populations and whether quoting content makes a person searchable. Establish a lawful basis where personal data are processed, minimise fields, apply access controls and set deletion rules. Pseudonyms and public usernames do not necessarily anonymise data. Document how platform changes or user deletion requests will be handled.

Web-data acquisition and validation pipeline. Collection proceeds only after legal, ethical and methodological gates are satisfied.
  1. SpecifyConstruct, target population and role alongside the survey
  2. AuthoriseAPI or access route, terms, rights, privacy and ethics review
  3. Acquire gentlyRate limits, retries, secure raw capture and collection log
  4. Diagnose errorCoverage, bots, duplicates, missingness and temporal change
  5. Validate and linkCompare with survey measures, calibrate cautiously and report provenance

Build a reproducible acquisition

Store the acquisition code, query terms, API version, endpoint, field definitions, time zone, collection window, request parameters and response schema. Respect rate limits; use back-off, caching and a descriptive user agent where appropriate. Keep immutable raw captures separately from derived data, with checksums, access logs and a processing manifest. Record failures and missing pages rather than treating absence as a genuine zero.

Web sources are temporally unstable. Interfaces, rankings, moderation, user behaviour and API fields can change without notice; historical results may not be reproducible from a later collection. Timestamp every request, version code and dependencies, retain permitted snapshots, and report gaps, deletions and platform incidents. Provenance should connect each analytical value to its source, acquisition and transformation.

Coverage, bots and measurement

A website’s users are rarely a sampling frame for the population of interest. Participation, visibility and data availability may vary with age, language, geography, access, platform policy and algorithmic ranking. Automated accounts, coordinated activity, duplicates and shared accounts complicate the unit of analysis. Detection rules are uncertain: analyse sensitivity to plausible classifications and do not equate ‘bot-like’ with invalid without evidence.

Search terms and classifiers introduce query and processing error. Define inclusion and exclusion rules before analysis; test recall and precision on human-reviewed material; examine subgroup performance; and keep a record of manual exclusions. Missing fields may mean not applicable, withheld, deleted, failed to load or never collected. Preserve these distinctions rather than collapsing them into one missing value.

Linkage, calibration and validation

Link web data to survey responses only with an appropriate lawful basis, participant information and secure separation of identifiers. Evaluate linkage error, unequal consent and whether linked respondents differ from the full sample. Do not use a high-dimensional trace as a proxy for an attitude without evidence that it measures the intended construct.

Compare web measures with survey measures collected in a relevant time window. Use hold-out samples, known benchmarks, convergent and discriminant evidence, and sensitivity analyses. Calibration or weighting may reduce known differences but cannot create coverage for unobserved groups or repair an ill-defined construct. Report the population each estimate actually represents.

Summary of key points

  • Place web data within the survey lifecycle and define their role before collection.
  • Check access terms, rights, privacy, ethics and reasonable expectations; robots.txt is only a technical signal.
  • Record provenance, time, code, schema, failures and transformations so the acquisition can be audited.
  • Diagnose coverage, bots, duplicates, missingness and platform change, then validate against relevant survey evidence.

Further reading

References