Asset Publisher

Collage: Stafeeva, stock.adobe.com (people); KAS

Facts and Findings

Online, offline or both?

Practical test of survey methods – which survey method delivers trustworthy, representative results?

Survey data that is supposedly representative of the population is published almost daily. However, its quality is increasingly being questioned. These doubts are also linked to an accelerating trend towards online surveys. So-called mixed-mode samples from online and telephone data are also increasingly entering the survey market. But which survey method delivers reliable results? What are the opportunities and problems of internet-based methods? What role do data weighting and accessibility play? Analyses of telephone, online and mixed-mode samples provide answers.

Asset Publisher

This paper uses a unique experiment to compare different survey methods. Two pure online samples, three mixed-mode samples and one pure telephone sample from different institutes are analyzed. The questions are identical. The analysis reinforces doubts about the quality of many online surveys. In order to obtain data representative of the population, there is no way around a random selection of respondents.

Some key findings of our study are:

  • The three survey methods - telephone, online and mixed-mode - differ in their quality due to the sampling method. While telephone methods rely on purely random samples, online surveys with non-random samples only reach people who show an increased willingness to participate.
  • The results of non-random samples vary unusually widely and therefore provide unreliable results.
  • Weightings are no guarantee for representativeness. They cannot compensate for missing groups ("offliners"/older people). Unknown deviations are not corrected by weightings but may be amplified.
  • Mixed-mode approaches can solve the socio-demographic problems of the telephone method but have qualitative shortcomings due to the non-random sampling in the online part.

Read the entire study “Online, offline or both? Practical test of survey methods” here as PDF.

Please note, to date the study is only available in German.

Asset Publisher

Contact

Dr. Sabine Pokorny

Dr

Electoral and Social Research

sabine.pokorny@kas.de +49 30 26996-3544
Contact

Dominik Hirndorf

Dominik Hirndorf

Electoral and Social Research

dominik.hirndorf@kas.de +49 30 26996-3858

comment-portlet

Asset Publisher

About this series

The series informs in a concentrated form about important positions of the Konrad-Adenauer-Stiftung on current topics. The individual issues present key findings and recommendations, offer brief analyses, explain the Foundation's further plans and name KAS contact persons.

 

Dr. Kristin Wesemann

Dr

Head of Strategy and Planning

kristin.wesemann@kas.de +49 30 26996-3803

Sophie Steybe

Referentin Publikationen

sophie.steybe@kas.de +49 30 26996-3726