Key Insights

UPDATE (2023–09–15): To detect smaller or trivial findings, more evidence is needed. Conversely, substantial findings require less evidence for discernibility. A p-value that is statistically significant can apply to both scenarios, but only one is genuinely meaningful.

UPDATE (2023–09–13): You can think of the discernibility of a finding as being inversely related to its p-value. Higher values suggest greater discernibility. For example, you might define discernibility as "discernibility = 1 - p" or "discernibility = 1/p". This concept can be particularly beneficial in exploratory studies (i.e., hypothesis generation, discovery, etc.).

Statistical evidence should not be mistaken for practical significance. A "significant" p-value doesn’t equate to a "significant" finding. The significance of the statistical evidence regarding the true variable does not imply that the finding itself is practically meaningful.

UPDATE (2023–11–28): This applies specifically to p-values that decrease with larger sample sizes.

Writing Sample

You are welcome to adapt the following text for your own writings. Ensure to personalize it to reflect your understanding and advocacy on this subject.

Instead of labeling a variable as "significant," refer to it as "discerning," or discuss its "discernibility."

We propose two significant changes in our statistical reporting practices. Firstly, we will avoid the phrase "statistically significant" to prevent confusion with the broader, more commonly understood term "significant," which often implies scientific importance. Statistical significance does not imply scientific significance, yet this misconception has persisted, contributing to the replication crisis in fields such as biomedicine and psychology.

Leading statistical authorities have recommended entirely moving away from the phrase "statistical significance" (Amrhein et al., 2019; McShane et al., 2019; Wasserstein et al., 2019). Thus, we need to adopt alternative phrases that accurately depict the statistical evidence in research findings, such as "statistically discernible," "statistically evident," and "statistically reliable."

Secondly, while we will continue to report p-values, we will refrain from categorizing them as "statistically significant" or "not significant." This common practice oversimplifies the information contained within the p-value, obscuring its utility for informed decision-making. Instead, we will report p-values descriptively based on their size (e.g., small, large) and describe the discernibility of the corresponding findings using terms like "strongly discernible" or "not discernible."

Real-World Applications

The following publications have effectively implemented this type of terminology:

1. Daza EJ, Wac K, Oppezzo M. "Effects of Sleep Deprivation on Blood Glucose, Food Cravings, and Affect in a Non-Diabetic: An N-of-1 Randomized Pilot Study." Healthcare 2020 Mar (Vol. 8, №1, p. 6).
2. Matias I, Daza EJ, Wac K. "What possibly affects nighttime heart rate? Conclusions from N-of-1 observational data." Digital Health. 2022 Aug;8:20552076221120725.

What Needs to Change?

The issue is not technical; it resides in communication and the behaviors we adopt in scientific discourse. The conflation of "statistical significance" with "significance" creates an illusion of certainty based on minimal changes.

Key Concepts

We must maintain rigorous assessments of the statistical quality of our evidence while also recognizing that statistical significance does not equate to scientific merit.

Overview

Statistical evidence remains crucial, but the conflation of "statistical significance" with scientific significance must end. We should only use "significance" to describe scientific findings, not the statistical evidence supporting them.

References

Amrhein, V., Greenland, S., & McShane, B. (2019). "Scientists rise up against statistical significance." Nature, 567, 305–307.

Daza, E. J., Wac, K., & Oppezzo, M. (2020). "Effects of Sleep Deprivation on Blood Glucose, Food Cravings, and Affect in a Non-Diabetic: An N-of-1 Randomized Pilot Study." Healthcare, 8(1), 6.

Kühberger, A., Fritz, A., Lermer, E., & Scherndl, T. (2015). "The significance fallacy in inferential statistics." BMC Research Notes, 8(1), 1–9.

McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). "Abandon statistical significance." The American Statistician, 73(sup1), 235–245.

Wasserstein, R. L., & Lazar, N. A. (2016). "The ASA statement on p-values: context, process, and purpose." The American Statistician, 70(2), 129–133.