Web surveys: does sampling matter?

At present there seems to be  lot of archive sector surveys on social media. This has got me thinking about sampling and whether the data the sector is collecting is representative and accurate.

What is sampling?

In an ideal world we would survey everyone in our target population. However, this is usually not practical or affordable. Instead we use sampling methods to survey a selection of our target population. However, it is important our sampling method ensures that this sample is representative.

Sampling methods can be be grouped into two types: probability or non-probability. In probability sampling everyone in the target population has the same chance of being selected. The selection of people in non-probability sampling is more arbitrary.

This short video from the University of Sheffield explains this well.

Convenience samples

Convenience sampling involves getting participants wherever you can find them and wherever is convenient. Web and surveys are usually a form convenience sampling. The survey is sent out to various networks by email and publicised on social media. Individuals choose whether to participate in the survey.

Convenience surveys are easy and cheap. They are sometimes used in pilot studies – to obtain basic data without the complexities of using a probability sample. However, they will not usually produce representative results and cannot speak for the entire target population.

“The result is a proliferation of poorly conducted ‘censuses’ and surveys based on large convenience samples that are likely to yield less accurate information than a wellconducted survey of a smaller sample.”  (Fricker, 2017)

Some believe that a process called weighting can be used to compensate for sampling issues in web surveys.

A practical example: I want to survey people who work in archives in the UK

We could randomly choose a percentage of people from a list of those who work in the sector e.g. members of the Archives and Records Association. This is called a a simple random sampling.

However, not everyone who works in archives is a member of the Archives and Records Association. Could this bias our sample? An alternative could be some type of cluster sampling approach where firstly archive services are randomly selected and then the staff in this service are sampled. This video explains cluster sampling in more detail.

Does it matter?

Web and social media surveys are cost effective and easy to administer. Perhaps I should not worry too much? However, if we really care about getting representative and quality data for the archives sector we should think carefully about what non probability sampling methods are available before resorting to a web surveys.

Further reading

  • Czaja, R., Blair, J., & Blair, E. (2014). Designing surveys: A guide to decisions and procedures.
  • Fricker, R. D (2017). Sampling Methods for Online Surveys in Fielding, N., Lee, R. M., & Blank, G. (2017;2016;). The SAGE handbook of online research methods (Second ed.). London: SAGE Publications Ltd

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s