We will experience brief service disruptions (see "Updates")
(the number of current visitors is automatically updated every 4 minutes)
Cite this page as:
-
Pilot studies (feasibility studies)
-
First published:
on:
Last updated:
If you want to share information about this web page...
This webpage gives you an understanding of what pilot studies (feasibility studies) are. It will also clarify the difference between a planned pilot study and a study that lacks sufficient power.
This page aim to clarify the difference between a pilot study and other studies. This discussion is most often relevant in relation to studies taking on an empirical-atomistic approach aiming to evaluate the effect of a drug or any other intervention. However, it can be used also for observational studies. The rest of this page will focus the discussion to the situation of pilot studies preparing the way for a larger study to establish effect of an intervention.
It is a common misconception that small underpowered studies trying to prove effect of an intervention compared to placebo or another intervention are pilot studies . This is not true. If they aim to prove effect and are underpowered it means that they either failed to recruit participants up to their pre-calculated target or it was simply a badly designed study, not a pilot study.
Effectiveness and efficacy
Efficacy is a measure of effect of the intervention under ideal circumstances while effectiveness is a measure of the effect if implemented in reality. Pilot studies does not aim to estimate efficacy or effectiveness.
The aims of a pilot study
A pilot study aim to answer questions such as:
- Will a large or small proportion of individuals asked to participate accept participation? Will randomisation make potential participants to hesitate?
- Will the measurement of effect be practical and feasible? Which survey is suitable in this situation? Should a new survey tool be constructed?
- Is the intervention reasonably easy to apply given the resources we could expect to have in a larger study?
- Will a large or small proportion of patients drop out of the study for various reasons?
- What level of random variability and rough changes or effects can be seen after intervention? (to be used as a basis for sample size estimation before the larger adequately powered study)
- How common are unforeseen problems (that may warrant exclusion of a patient or worse termination of the study)?
A pilot study aim to provide answers to some or all of these questions but is not powered to answer the big question is there an effect of the intervention. Hence, the reporting of a true pilot study should completely refrain from trying to reject a null hypothesis concerning possible effect of the studies intervention. Effect size may be calculated only to serve as a guide for a sample size calculation before the final study.
| Pilot study | “The real thing” | |
|---|---|---|
| Sample size calculation before data collection | Rarely (see below) | Should always be done |
| Number of included participants | Usually very small | Usually fairly large |
| Randomization | Often if it is an interventional study | Should be done if the focus is to evaluate effect of something |
| Aim to establish effect | Never ! (but may make a preliminary estimation of effect size to inform a sample size calculation for the final study.) | Establishing / comparing effect is the main aim (may include superiority, equivalence and non-inferiority trials) |
| Aim to establish practical feasibility | Clarifying the practical feasibility is the main aim of a pilot study | Practical feasibility should be sorted out before the real study is done. |
| Inferential statistics used | Is only done to inform a sample size calculation for the coming final study. | Should always be used to answer the research questions. |
What if my pilot study shows an effect?
A pilot study never has the aim to prove an effect of an intervention. Hence, when calculating the effect size (to be used in a sample size calculation for the real study) a finding of no statistical difference between groups says nothing about the true effect size. The pilot study was never designed to detect a difference between groups. However, sometimes the pilot study shows a stattistically significant difference between groups. If that is the case it indicates that there is probably a large effect of the new intervention compared to the comparator if it shows up as statistically significant in a small pilot study.
Sample size in pilot studies
Various recommendations point to a number between 10-50 or higher. However, a proper sample size calculation can only be done for research question 6 above . There is no sensible way to make a proper sample size calculation for research question 1-5.
CONSORT 2010 statement: extension to randomised pilot and feasibility trials
Standard guidelines for randomized controlled trials (CONSORT 2010) focus on reporting results regarding the effectiveness of an intervention. However, pilot and feasibility trials are not designed to test effectiveness, but rather to investigate whether a full-scale trial can be done. The main difference between this extension and the standard CONSORT statement is the shift in focus from hypothesis testing (does the intervention work?) to feasibility assessment.
This extension is important because many researchers previously disguised small pilot studies as full RCTs, leading to “underpowered” negative studies that cluttered the scientific literature. This extension encourages researchers to be honest that their study is a pilot and to report it based on what it actually achieved (feasibility data) rather than what it failed to achieve (statistical significance on clinical outcomes). I recommend reading more on the CONSORT extention website.
Examples
- A pilot study: “A feasibility study for a triple‐blind randomized controlled trial investigating the effects of oral isotretinoin on mood and quality of life in patients with acne vulgaris” .
- A study that failed to recruite participants to the target: “Oral corticosteroids for painful acute otitis externa (swimmer’s ear): A triple-blind randomised controlled trial” . However, the outcome can be used as a pilot study, even if that was not the intention, to finetune the sample size calculation for a new real study with better conditions for success (more money).