Misconceptions and misunderstandings hampering medical research and progress

INTRODUCTION

When scientific projects or articles are evaluated, objections are often raised that may prevent their performance or publication. Sometimes, the flaws noted may not be correct or relevant to the study. In this article, we review the most common types of objections that can hinder the progress of medical research and suggest ways to reduce them.

CLINICAL (OR PROCEDURAL) OBJECTIONS AND STATISTICAL/METHODOLOGICAL OBJECTIONS

The objections an evaluator can make to a research project can be grouped into 2 broad categories: clinical (or procedural) and statistical/methodological.

The former can be addressed and, if necessary, refuted by the author of the project as they relate to the clinical problem per se. In this regard, the author of the project has more expertise and sometimes more up-to-date knowledge than the evaluator on the issue in question. A common example could be the objection, “the project does not specify under which conditions baseline blood pressure should be measured, or the criteria chosen to define hypertension.” The researcher can acknowledge the flaw in his/her protocol and correct it or argue that the objection is incorrect.

The situation is different with statistical/methodological objections. Researchers, whether acting as evaluators or persons who are evaluated, are not usually experts in research methodology and biostatistics. Below are a few examples of this type of objection.

Common erroneous statistical/methodological objections

Sample size

Contrary to what most researchers believe, the objection of an insufficient sample size is only relevant in highly specific situations. In some cases, it is not accurate; for example, if the result has a very small P value that constitutes strong evidence against the null hypothesis. It does not make any sense either in somewhat more complex situations.1

Statistical power

Statistical power depends on 4 parameters, whose value is often not predefined, so by choosing suitable values for these parameters, researchers can obtain almost any value for statistical power. In fact, when researchers are asked about the figure for statistical power, it is often insufficient to give a specific value, because the values of other parameters associated with such power are also necessary. Moreover, it is obvious that by slightly modifying these values within reasonable ranges, very different power values can be obtained.2

Test on the normal distribution of the response variable

In many cases, this objection may be doubly mistaken: either because the response variable is dichotomous and will be treated as such in the analysis, or because the sample size used is greater than, say, 30, and the central limit theorem guarantees a very good approximation to the normal distribution of the statistic used in the test. Naturally, it can never be guaranteed whether the variable has a normal distribution or not. Thus, in cases with a confirmed lack of normal distribution, the robustness of some parametric tests vs nonnormality must be taken into account.3 In cases with a strong association and an extremely small P value in the test, it should be noted that if the true P value of the test were, say, 10 times larger or 10 times smaller than that found in the parametric test, the practical conclusion would be the same.

Control group and study validity

While a control group is a great asset in many situations, demanding its presence should not be a universally or undisputed mantra. In some situations—and when used appropriately—historical controls provide enough information to draw very interesting conclusions. In other cases, each patient serves as his/her own control, thus allowing the use of intraindividual variability, which is often less than interindividual variability and, therefore, provides more powerful tests in many cases.

Pilot trials

Randomized clinical trials (RCTs) add highly useful methodological refinements to effectively determine the safety and efficacy profile of a new drug or procedure. However, pilot trials can add these same methodological refinements and be controlled, randomized, and blinded to a point that the level of scientific evidence they provide can be equivalent to that of RCTs, with significant advantages regarding time and cost savings. In addition, in general, their size is not a determining factor that compromises their validity. Then, what is the main difference between the 2 designs? The difference lies not in the level of evidence they provide, but in the administrative process involved. RCTs require approval from external hospital, regional, or national committees, while pilot trials are endorsed by the expertise of the medical team involved in their design. For external evaluators, it is more challenging to make accurate assessments of each aspect of the project and provide a sound judgment. Moreover, if they have the authority to veto the study, there is a possibility of rejecting it based on insufficiently founded considerations.

Observational trials

Blinded RCTs are widely accepted as the best source of evidence on drug and treatment efficacy. However, observational studies can also provide information on long-term safety and efficacy, which is often lacking in RCTs. Additionally, they are less expensive, allow the study of rare events, and provide information more quickly than RCTs. New and ongoing developments in analytical and data technology offer a promising future for observational studies, which already play a key role in researching treatment outcomes. Data from large observational studies can clarify the tolerability profile of drugs and are particularly suitable for large and heterogeneous populations of patients with complex chronic diseases. RCTs and observational trials should, therefore, be considered to complement each other.

Case-control trials

Rothman4 states that case-control trials have gone from “being the Cinderella of medical research to one of its brightest stars.” In case-control designs, it is much more challenging to avoid the distortion caused by confounding factors. However, these issues are partially mitigated by segmentation, matching, and multivariate analysis techniques. In some cases, they can provide significant statistical evidence much faster and more cheaply than clinical trials. Let’s consider an example of a disease that affects 1% of the population who do not follow a particular diet, and 5% of those who do follow it, knowing that 40% of the population follows that diet. A prospective trial would take 80 people from the diet group and another 80 from the control group, and after the agreed-upon time, we would measure the incidence of the disease in each of the 2 groups. The statistical power of this study for an alpha value of 0.05 would be 8%. A case-control trial would take 80 patients with the disease and 80 without it, and with very detailed health records, we would be able to determine the percentage of people who follow that diet in each of the 2 groups. The statistical power would be 93%.

The list of erroneous objections is much longer, however, and each would require a dedicated article to explain it.

CONCLUSIONS

Some of the methodological objections raised by the evaluators are incorrect. In most cases, the evaluated party assumes that his/her project has a major flaw and ends up abandoning it. Consequently, many projects that could have provided valuable information are unfairly discarded slowing down the progress of medicine.

We believe that this anomaly would largely be avoided if: a) evaluators raised methodological objections only in areas in which they have in-depth knowledge; b) whenever possible, the judgment issued by the evaluators from health agencies and bioethics committees was a suggestion instead of a veto; c) the fundamental role of observational trials, which can be highly effective and generally cheaper than clinical trials, was recognized; d) pilot trials were conducted in many cases where they are indicated, because they can be controlled, randomized, and blinded but without the restrictions associated with RCTs (figure 1).

Figure 1. Measures to expedite medical research by promoting the autonomy of qualified physicians, avoiding unjustified methodological objections, and promoting the use of currently underrated designs.

FUNDING

None declared.

CONFLICTS OF INTEREST

None declared.

REFERENCES