What Is a “Well-Powered” Study?

How do researchers know if their study is well-powered?

Typically, during the design phase of the research process, researchers will conduct what’s called a power analysis to help them calculate the sample size needed to see the desired effect. There are a lot of factors that go into a power analysis, but some of the main ones include:

• Effect size: The effect size is often defined as the standardized difference between the means of two groups. In other words, it describes how large of a change you expect to see in a study. Effect size values most commonly range from 0 to 1, where 0.2 is considered a small effect size, 0.5 medium, and 0.8 large. In general, the relationship between effect size and sample size is inversely proportional, which means you need larger sample sizes to see smaller effects. This is particularly true in heterogeneous populations where the standard deviation in measurements is high.

• Statistical analysis model: There are many statistical models that are used to determine if research results are significant. Which model is used depends on things like how many study arms there are and how many outcomes are being analyzed. Power analyses, therefore, have to consider which statistical model will be used in order to determine how many participants are needed in each study arm.

• Significance level: Statistical analysis results are most often reported using p-values. Prior to performing a study, researchers should set a significance level, which is the threshold for a p-value where the results are considered significant. The majority of studies set their significance level at p<0.05, which essentially means there is a 95% chance that if the p-value is less than 0.05, the results are due to a real effect or relationship. 

To make this concept a little more tangible, let’s use the Life Improvement Trial (LIFT) as an example. Because of the complexity of studying the effect of multiple drugs in one trial, the LIFT will use a statistical model that accounts for data with multiple timepoints and subgroups. The LIFT significance level is set at p<0.05. Given these factors, a power analysis determined that 40 participants per study arm (160 participants total) will allow us to detect both large and medium effect sizes, which we hope will correlate with clinically meaningful changes.

How can I tell if a research study is well-powered?

First and foremost, look at the number of participants included in each group in the study, sometimes reported as an “n” – an n of 20 means there are 20 participants in that group. While there are many factors that go into power analyses, as a general rule of thumb, if there are fewer than 15 participants in each group, the study is likely under-powered. Under-powered studies may still provide useful information, but it’s important that the results are validated with larger, more robust studies before they are used to influence clinical action.