This page will explain the difference between “statistical significance” and “effect size”, the latter also labelled clinical significance in health care. The label “effect size” is most often used when discussing effect of an intervention in study designs comparing groups. However, it is also used in observational studies trying to explore effects.

P-value versus effect size

We often talk about the p-value. P stands for “probability”. What kind of probability is it? In the situation of comparing two interventions the p-value states the probability of being wrong when we say that treatment A has a larger effect than treatment B. A low p-value says it is unlikely that we would get the observed observations if the difference in effect between groups is zero. You can read more about this on the page Level of significance. It is important to remember that the p-value only talks about statistical significance but says nothing about importance (clinical significance in health care).

In large studies we may reach statistical significance but the difference between groups may be so small that is is of no importance (no clinical significance). Hence, we need another measure than the p-value showing the importance (clinical significance) of the differences in effect between treatments. This measure is labelled “Effect size”. The p-value is always a probability estimated with a figure between 0-1. The effect size is different. It is not a single figure, rather many different estimates that may all be labelled as effect size when used to evaluate effect of interventions.

Different types of effect size

We use different types of effect sizes depending on the outcome variable and if the scenario is group comparisons with no need to adjust for effect modifiers or confounding variables or, if the scenario focus on associations, the latter commonly used also for group comparisons where there is a need to adjust for effect modifiers and confounding variables. You may find the web page about Choosing statistical analysis clarifying. The most common types of effect sizes are:

*Group allocation means a variable determining if participants had one intervention / exposure compared to another / or placebo.
**Use techniques where you can adjust for confounding variables.

Cohen’s d is sometimes referred to as the effect size and this may cause confusion. It is important to note that Cohen’s d is only one of several different estimates of effect size and Cohen’s d is not always suitable (see table above). Furthermore, it is common to have more than one outcome measure so you may have to mix different types of estimates of effect size.

P-value is given as a single estimate. However, the effect size is given as a point estimate with a confidence interval (usually a 95% confidence interval). The p-value and the effect size are tied to each other so when the p-value increases and approaches 0.05 the 95% confidence interval for effect size will approach the limit for no effect. Hence, the p-value and the effect size are merely the opposite sides of the same coin.

Magnitude of effect size

The table below are rough rules of thumb that should be adjusted for your particular context.

* Irrespective if the effect size is positive or negative.
** Near 1.0 is no effect. The more it moves away from 1.0 (upwards or downwards) the larger the effect.

Cohens d

(Detta avsnitt är under konstruktion. Vi beklagar olägenheten.)

Hedge’s g

(Detta avsnitt är under konstruktion. Vi beklagar olägenheten.)

Useful links

• Download a calculator in Excel calculating Cohens d and Hedges’ g with 95% confidence interval

References