This paper uses a unique experiment to compare different survey methods. Two pure online samples, three mixed-mode samples and one pure telephone sample from different institutes are analyzed. The questions are identical. The analysis reinforces doubts about the quality of many online surveys. In order to obtain data representative of the population, there is no way around a random selection of respondents.

Some key findings of our study are:

• The three survey methods - telephone, online and mixed-mode - differ in their quality due to the sampling method. While telephone methods rely on purely random samples, online surveys with non-random samples only reach people who show an increased willingness to participate.
• The results of non-random samples vary unusually widely and therefore provide unreliable results.
• Weightings are no guarantee for representativeness. They cannot compensate for missing groups ("offliners"/older people). Unknown deviations are not corrected by weightings but may be amplified.
• Mixed-mode approaches can solve the socio-demographic problems of the telephone method but have qualitative shortcomings due to the non-random sampling in the online part.

Read the entire study “Online, offline or both? Practical test of survey methods” here as PDF.

Please note, to date the study is only available in German.