Sampling Strategies and Data Collection

If you want to share information about this web page...

• LinkedIn
• X
• WhatsApp
• Reddit
• Facebook

In all empirical studies (where conclusions are drawn by studying reality), some form of data collection is conducted. The collected data is analyzed, and conclusions are then drawn. When collecting data, you must decide from where and how it will be gathered. You need to choose which individuals will answer surveys, be examined, etc. How do you best select the individuals to be studied/interviewed, and how do you carry it out? How does the sample affect the results? Can mistakes be made? By reading this page, you will gain a better understanding of what you need to consider regarding sampling and data collection. The page first discusses the situation in studies that focus on numbers (statistics) and then on studies that do not use numbers (qualitative methods).

You will understand this page best if you have read the pages Study design and Introduction to qualitative methods.

Let’s assume you want to know the prevalence of overweight individuals in a country. How do you proceed? In theory, you could imagine examining all the country’s inhabitants for overweight prevalence. However, for an entire population, this is often not feasible due to high costs and time constraints. If it involved a smaller group, such as all the people in a small town, it might be possible to examine every individual. Studying all individuals in the population you want to make statements about is called a census. In rare cases, a census can be conducted. In most cases, however, a sample survey is done, meaning only a portion of all individuals is examined. Those selected to be examined are called a sample. If the sample is selected correctly, it can be considered a miniature version of the larger population you want to discuss. The results from the sample then represent all individuals in what is called the underlying population. Proper sampling is more critical in epidemiological studies compared to experimental ones.

Data collection in a quantitative approach

Data collection begins long before the project is launched. It is an important part of the study planning. Planning the data collection involves several decisions that must be made well in advance of the actual data collection:

1. Determine which population you ultimately want to make statements about. This is called the “population of interest.”
2. Decide where you can find a sample of this population. This is called the “sampling frame.”
3. Decide on the appropriate sampling method for the sample.
4. Determine the inclusion and exclusion criteria.
5. Decide which data (which variables) should be collected.
6. Estimate the sample size needed to answer your questions.
7. Plan the logistics around data collection.
8. Finally, collect the data.

Points 1-7 above should be described in the study protocol and be included in the materials for the ethics review board to consider your project. Much of the work around data collection is done well before the actual data collection takes place.

Determining the population of interest

Most studies are sample surveys, so the participants in your study can be seen as a sample taken from an underlying population. The results from your project can then be used to draw conclusions about the underlying population. For this to work, you must clearly define your underlying population.

An example of an underlying population might be: women aged 40-70 with diagnosed type II diabetes living in a high-income country in Europe or North America. This is a large underlying population, and it would be impossible to study all these individuals. Therefore, you take a sample to represent the underlying population.

Determening the sampling frame

The sampling frame is those who are practically accessible for your project. For example, it could be women aged 40-70 with diagnosed type II diabetes who are in contact with a healthcare center in a specific municipality. You rarely include everyone in the sampling frame, but rather a sample from it.

It’s important to consider to what extent your sampling frame resembles or differs from the underlying population you want to make statements about. Sometimes the sampling frame is the same as the underlying population, for instance, using the entire Swedish population register to get a sample from the Swedish population to draw conclusions about that the Swedish population.

Having the sampling frame and underlying population be exactly the same is an exception. In most cases, the sampling frame is much smaller than the underlying population. Usually, the sampling frame resembles the underlying population, but the match is rarely 100%. There might be certain types of people in your sampling frame who aren’t in the underlying population (called overcoverage) and vice versa, where the underlying population has certain types that are missing from your sampling frame (called undercoverage).

It’s important to try to get a rough idea of the extent of overcoverage and undercoverage. Sometimes, you can obtain figures by comparing available overall descriptive statistics for both the underlying population and the sampling frame. Often, it’s not possible to get precise figures, but a brief discussion of 1-2 sentences about over- and undercoverage is desirable when presenting your results.

Determine sampling method

There are two main approaches when it comes to sampling strategies: non-random sampling and random sampling. In the latter, each individual’s probability (chance) of being selected is equal and known in advance. In the former, each individual’s probability of being selected is unknown.

Non-random sampling

“One uses what one has,” i.e., you examine the individuals that are easiest to access. The results then apply to the individuals who are studied. Do they also apply to individuals who are not studied? Maybe, but this is often unknown. This is the big problem that makes non-random sampling something to avoid, especially in epidemiological studies. Despite this, it is used for cost reasons or in situations where there are not high demands for precisely accurate answers. Below are descriptions of some different specific types of non-random sampling:

Convenience sampling

One subjectively selects individuals with no other consideration than that they are as easy as possible to access. These could be coworkers, friends, or relatives. The question that remains unanswered is how the selected individuals resemble the underlying population to which the results should apply. There are few situations when this is scientifically acceptable.

Typical sampling

One subjectively selects individuals considered typical for the underlying population of individuals about which one later wants to make statements. The probability of each individual being selected is generally completely unknown, so it is uncertain if they represent the underlying population.

Snowball sampling

The first participants you include are asked to recruit a few others, who in turn are asked to recruit more, and so on. The sample grows like a snowball rolling. Scientifically, this is not a good option, but it can often be the only way to reach people who are usually difficult to access, such as the homeless, drug users, or sex workers.

Telephone Sampling or “On the Street” Sampling

You call phone numbers you have obtained, and those who answer and agree to participate are included in the study. The phone numbers might come from a list or be randomly generated. This procedure risks various systematic errors. If calling from a list provided by any of the directories available online, it’s likely that those with unlisted numbers are excluded, and they may differ from those without unlisted numbers.

Another variant is standing on a street and asking people if they want to participate in a project. There is no control over who frequently visits the area and who is often elsewhere (at home, work, or exercising), leading to serious sources of error that should be avoided.

Web surveys

There are many techniques for collecting responses through web surveys. Almost all of these use non-random sampling without any non-response analysis. In the end, you have no idea about the response rate or which underlying population the responses represent. The value of this is very limited.

If you have a list of individuals you want to invite to participate, and you send out a personalized link with the ability to see who has responded and who hasn’t, this becomes much better. If the list is also randomly generated, the web survey can be a tool for collecting data that represents a random sample of the sampling frame.

Self-selection sampling

Everyone in an underlying population is asked/invited to participate, and those who agree/opt-in are studied. If only a small number opt out, it is akin to a full survey. If a large number opt out, the question remains of whom the participants could represent. Self-selection sampling should be avoided.

Quota sampling

Suppose we know that in the underlying population we want to study, there are 40% men and 60% women. You then decide to examine 40 men and 60 women, handpicked in the easiest way possible, such as the first 40 men and 60 women who arrive at the clinic. Individuals are selected through a convenience sample to achieve predetermined numbers in each group. This is sometimes inaccurately called stratified sampling. The difference is that in stratified sampling, which is a type of random sampling, the probability of each individual theoretically being selected is determined in advance. In quota sampling, the selection is not random, so the probability of an individual being selected cannot be determined in advance.

Consecutive sampling

All individuals who visit a clinic during a predefined time period and meet the inclusion criteria make up our sample. A risk with this approach is the presence of a time effect, some phenomenon active during that period that affects the results.

To avoid time effects, data collection should continue long enough to reasonably avoid these impacts. A specific type of consecutive sampling is examining a certain proportion of patients visiting a clinic over time. This could be every 7th patient, or selecting all patients visiting the clinic every 7th day, choosing a particular weekday. Monday patients might differ from Friday patients, so it might be better to alternate and include patients from different weekdays and times. If a consecutive sample includes a large portion of the clinic’s patient base, it becomes similar to systematic sampling.

If patients return for follow-ups according to a predetermined schedule that is the same for everyone, the risk of systematic error is smaller, although this is rare. Typically, patients visit clinics with varying frequency. Frequent visitors have a higher chance of being included than those who visit less often. Frequent visitors may differ from infrequent ones, making the sample unrepresentative of the underlying population. Waiting room surveys often fall into this trap, leaving uncertainty about which population the respondents represent.

Consecutive sampling is frequently used in experimental studies where the primary interest is not in precisely mapping factors like blood pressure but rather studying the effects of various medications (or other interventions) on blood pressure.

Random sampling

Simple random sampling

All individuals in the underlying population have the same chance of being selected here. If you want to know the proportion of the population that is overweight, a random sample is drawn from our sampling frame.

Systematic sampling

Here, a system is established for selecting individuals. For example, it might be every 10th or every 50th patient in the clinic’s entire medical records archive. This is based on the order they appear in the electronic records (it could be a unique ID number assigned to each patient). In practice, this can be considered a variant of simple random sampling. However, there is a serious potential source of error if there is a temporal phenomenon (or similar issue) introducing systematic bias.

Stratified sampling

All individuals in our sampling frame are divided into conceptual groups (strata), such as age groups and/or gender. From each group, a simple random sample is drawn. If the proportion of individuals chosen from each group matches the group’s proportion in the sampling frame, it’s called proportional stratified sampling. However, if there is also a need to report results separately for each stratum, the proportion of individuals selected from each group may differ from their representation in the population by letting a group that is small in the sampling frame to be over-represented. The latter is named non-proportional stratified sampling. The number of groups (strata) should not exceed six .

Cluster sampling

Individuals are often naturally grouped. Examples of natural groupings include students in school classes, doctors at a clinic, patients treated at different hospitals, or individuals attended by various instructors. It can be impractical to select individuals randomly, such as just a few students in a class. The solution is to randomly select a sample of groups (classes or similar).

There are two variants. If the group size is not too large, all individuals in the selected groups are studied. This is called single-stage cluster sampling. If each group is large, it may be practical to take a simple random sample from each selected group instead. This is called two-stage cluster sampling.

The disadvantage of cluster sampling is that most of the groups (and thus the population) are often not represented in the sample, which can lead to systematic errors if the groups differ in unexpected ways.

Which sampling method should I choose?

• Most non-random samples are acceptable for pilot studies where the aim is to gather more information before planning a larger randomized controlled trial.
• Consecutive sampling is suitable for an early phase 1 or phase 2 experimental study where the goal is to test if an intervention has an effect. Here, the focus is less on providing an exact current description and more on examining the changes that different interventions bring about. Since the baseline description is not the main focus, a simpler procedure can be used to select the sample.
• A type of random sampling is desirable for a larger phase 3 experimental study. However, most phase 3 studies use consecutive sampling. This can be acceptable if you can demonstrate that the chosen sample reasonably represents the underlying population where the intervention is intended to be applied after the study.
• In observational studies, striving for some form of random sampling is essential. Systematic sampling and cluster sampling have the advantage of being practically simple and often reduce the cost of collecting the sample .
• In data analysis, simple random sampling or stratified sampling is generally somewhat better than systematic or cluster sampling due to the lower risk of systematic bias in the former.
• Proportional stratified sampling and simple random sampling work similarly, providing equivalent results for estimating the population’s means and variance (e.g., through standard deviation) .
• Non-proportional stratified sampling also allows smaller groups in the population to be better represented. This variant of stratified sampling is more complex to use (especially in statistical processing) but offers better precision (less random variation) than simple random sampling .

Should the sample be homogeneous or heterogeneous?

Suppose you want to study the effect of a new revolutionary physiotherapy treatment for back pain. Should you only include patients with a narrow definition of the specific type of back pain the treatment is believed to be most effective for (a homogeneous group), or should you include patients with varying types of back pain (a heterogeneous group) similar to what is seen in most physiotherapy clinics? This question isn’t black and white but involves a spectrum of options between a very homogeneous group and a very heterogeneous one.

From a statistical standpoint, conducting a purely experimental study with a very homogeneous group is optimal, and this approach is often used when the goal is to simply demonstrate whether a treatment has any effect (explanatory study). For generalization purposes and to mimic real life, it’s better to aim for a reasonably heterogeneous sample that reflects the diverse reality. Heterogeneous sampling is commonly used in phase 3 studies of new medications when the aim is to obtain results that can be generalized and applied in everyday healthcare settings (pragmatic study).

Eligibility criteria (inklusion and exklusion criteria)

Eligibility criteria consists of inclusion and exclusion criteria and are the criteria used to identify those suitable for inclusion. There are two different ways to use inclusion and exclusion criteria:

• Option A: Individuals must meet all inclusion criteria. Some inclusion criteria might include absence of pregnancy, dementia, cancer, etc. Exclusion criteria are applied later if it becomes necessary to exclude some who were initially included. Here, only inclusion criteria are used to include, and exclusion criteria are applied later if needed. With this method, cross-sectional studies, where individuals are assessed only once, don’t need any exclusion criteria at all.
• Option B: Both inclusion and exclusion criteria are used initially to select participants. Inclusion criteria are often broad, such as age, gender, and geographic location, and all inclusion criteria must be met. Exclusion criteria are often narrower and more specific, focusing on factors that could obscure a treatment effect or indicate participants unlikely to complete the study as planned, such as cancer or psychosis, or factors that might affect the outcome, like a specific disease.

Both options A and B are used, and it’s wise to be aware of the difference. A common misconception is that exclusion criteria are just the opposite of inclusion criteria. For example, stating that being female is an inclusion criterion and being male is an exclusion criterion. If the inclusion criterion is being female, males have never been included, so there is no need to exclude them later. It’s important that exclusion criteria add something not already covered by inclusion criteria.

Operationalization

Now it’s time to determine in detail what data needs to be collected. This is called operationalization. Almost always, various types of data are collected. Collected data is arranged in columns and rows. We use the term “variable” and have a column for each variable. The rows are called observations. An observation is often an individual or patient. Examples of variables include age, gender, weight, blood pressure, etc. We use collected data for two main purposes:

1. To describe the individuals included in the study. This is called descriptive statistics and informs the reader of your report about whether your results might be applicable to their individuals/patients.
2. To draw conclusions about what the study aimed to answer, often using inferential statistics.

Many variables are used for both descriptive and inferential statistics. Variables used for inferential statistics should be submitted to sample size calculations (see below). Sometimes the sample size calculation may show that one variable requires an unreasonably high number of observations / patients. In that scenario this variable might be ditched completely or it might be kept solely for descriptive statistics. There is usually an interplay between the preliminary list of desired variables and the sample size calculation before you end up with the final list of variables intended for descriptive and / or inferential statistics. The type of data to be collected can be:

1. Direct measurements (such as measurements of the body and its chemistry, body reactions)
2. Indirect measurements of knowledge, attitudes or perceptions using surveys or structured interviews
   1. Binary questions (Yes/No)
   2. Surveys measuring attitudes or perceptions (Likert scale, Visual analogue scale or similar)
   3. Surveys with other fixed response alternatives
3. Structured observations
   1. Structured observations of behavior
   2. Structured observations of events or processes

Sample size estimation

In a quantitative approach, estimating the sample size is crucial, especially for variables used in inferential statistics. This involves making some assumptions and then running the planned inferential statistics in reverse. You start with a hypothetical result and determine how many observations are needed to achieve that result. Learn more about this on the page about sample size estimation.

Plan the practicalities of data collection

Once operationalization (se above) is done you need to decide how these variables are going to be collected. If you collect indirect measurements you may have to find, or develop, a suitable survey. Pilot test your data collection and make changes to avoid potential problems. Plan the logistics and resources needed. Once this is done train those involved in data collection. Finally establish a reasonable time line and make a contingency plan in case something goes wrong.

Do data collection

Regularly check the quality and completeness of the data being collected. This is very important in the beginning of data collection. You may find surprising errors that hopefylly can be fixed by small alteration to the data collection procedure.

Data collection in a qualitative approach

Determining sampling method

Strategic sampling is often the best selection strategy in qualitative research. It involves deliberately handpicking individuals to obtain a reasonable representation of variation in terms of gender, age, experience with the phenomenon being studied, etc. The aim is usually to capture variation rather than achieving the same degree of variation present in an underlying population. In qualitative studies, there is generally no discussion of underlying populations. Random sampling is typically the wrong selection strategy for studies with a qualitative approach.

Decide appropriate data collection technique

1. Interviews with one person at a time
   1. Open (unstructured) interviews
   2. Partly open (semi-structured) interviews
2. Interviews or discussions in groups = focus groups
3. Documents
   1. Diaries
   2. Written accounts of experiences
   3. Fiction (invented stories) or poetry
4. Open (unstructured) observations
   1. Non-participant observations
      1. Non-participant hidden observations (using a one-way mirror or a hidden video camera)
      2. Non-participant non-hidden observations (like sitting fully visible and passive or using a visible video camera)
   2. Participating observations
      1. Participating hidden observations (example Günter Wallraff)
      2. Participating non-hidden observations (common in studies using ethnography, grounded theory or social antrophology)

References