Here are six mistakes you definitely don’t want to make, when you see results from clinical research
– and how to avoid them.
Reading and capturing relevance of information in papers referring clinical trials can be both time consuming and challenging for the skilled reader, and virtually impossible for the untrained user.
#1: Taking Statistical Significant Findings for Evidence Based Medicine
Statistical significance does not equal proof of clinical relevance.
Results from clinical trials being statistically significant indicate that the probability of getting the findings in the study just by chance is small. However, even if the findings are not just a case of chance, they still do not need to be relevant for the patient at the point of care. Often they are not.
The validity of scientific conclusions depends on more than the statistical methods themselves. Appropriately chosen techniques, properly conducted analyses and correct interpretation of statistical results play a critical role in ensuring that conclusions are sound and that uncertainty surrounding them is represented correctly.
The word “Evidence” is, unfortunately, misleading. It is custom to use statistical significance as the level of “proof” in medical science. The findings are then dichotomised into either it is “the truth” or “not the truth”. Even though statistical methods used is very far from providing such information, these statements find its way through health care and can even be implemented in guidelines. It is with large studies (many participants) rather easy to get statistical significance, but this kind of significance is not important, what counts is, if the findings are clinically relevant. With increasing access to large data sets (big data) especially by wearables and other ways to self-report, we will, without a doubt, see many studies shortly finding statistical but not clinical significance.
#2: Taking p-value as Proof of Power of Findings
A p-value at 5% (=0.05) or less does not equal proof of clinical relevance. The p-value is not the probability of the research question being true. The most often used cut-off value for “statistical significance” at 5% is chosen arbitrary, and have no root in any scientific discovery or law.
A p-value is a result derived from analyzing data with a mathematical method for calculating the chance of finding the result presented or a more extreme result. This means that a result from research can be named “statistically significant” and have a small p-value without being a meaningful result for the patient.
The test for significance is not at a direct test of the study question(s) or hypothesis, but only an analysis of the data collected. The clinical relevance thereby is highly dependent on how the study is designed and executed, on study time and parameters, study population (and who is excluded) and much, much more.
Every clinical trial has its limitations. One is that the results from a trial only applies to precisely the persons included in the trial, at the time of the study, and by using exactly the treatment and method for analyzing presented in the paper. This is why test results can be bee difficult to reproduce since the settings will inevitably be different.
A study has inclusion criteria (who can be included in the study population), and exclusion criteria (who can not be included in the study). These criteria can include gender, age, diseases, prior conditions and treatments, pregnancy, and much, much more. Data from a trial cannot be generalized beyond this.The chosen inclusion and exclusion criteria affect heavily on the results … and on the broader usefulness of the findings.
Results from a clinical trial cannot necessarily be applied to persons who would be excluded from the trial, and cannot be generalized for a longer time than the study period. If a study excluded pregnant women and persons with cancer in a 12-week study period, the results cannot be used for pregnant women and individuals with cancer, and cannot be generalized after 12 weeks of treatment, since we do not know what would happen at week 26 or 5 years after the start. We can, of course, guess and suppose, but we do not have the “evidence”.
#4: Accepting Surrogate Measurements
Statistical significant clinical trials are expensive and complicated to run. Looking for patient relevant results like an improvement in the quality of life, getting a heart attack or a stroke must often include many persons for a long time, to be able to find relevant results. Researchers therefore often turn to surrogate measurements, which is easier detectable changes associated but not equals what we want to know. This could be a reduction in blood pressure, cholesterol or blood glucose when what we want to know is if a treatment reduces risks associated with having diabetes including the risk of heart disease and stroke.
Surrogate Measurements are very commonly used in clinical trials but do not alone provide the knowledge we want.
#5: Not Evaluating Clinical Relevance and Power of Results
For a result to be clinical relevant it must reflect a detectable and meaningful change for the patient. Reduction in pain may be “statistically significant”, but this does not necessarily tell us if the reduction still is too small for a patient to experience this change in pain.
In other studies, results may show a decline in blood pressure or blood sugar, but what we want to know is if the treatment reduces the risk of death or complications. Some studies have tried to find the smallest change a patient can detect and find relevant in different conditions and symptoms.
#6: Confusing Correlation with Causality
Headlines like “[insert your favorite food here] Linked to Cancer”, or headlines telling us that coffee is bad for our health one week for then to be good for our health next week is part of everyday life. These very basic analyses are made by analyzing if there is same time rise or fall in the frequency of each event.
If people are eating more [insert your favorite food here] have a higher rate of cancer you have your scientific paper and subsequent news headline. However, it may be that these persons also have a higher frequency of smoking or some other risk factors not related to [insert your favorite food here].
Correlation does not equal causality. Sunshine is highly correlated with sales of ice-cream, but the sunshine does not cause the increased sales of ice-cream.
You probably are making one or more of these mistakes, we all are!
That is why we are developing Zignifica. Zignifica is a system and a method available in an app capturing relevant information from papers presenting clinical trials and analyze these findings. These analyses include both method/design of the study and the results. It includes not just all six items above, but many more parameters and also the study’s metadata.
Zignifica present the result of the analyze in score and text, easy to understand for both healthcare professionals and persons without any medical training.
We want to test the concept, idea, and usefulness of Zignifica. That is why we have created a prototype of the app. If you want to test it and provide us with feedbac click here.