Scientific Method and Scientific Madness

Scientific conclusions need not be accurate, justified, or believed by their authors

Pablo Picasso, Bust of a Woman (Dora Maar), 1938

In a provocative and memorable 2021 paper in Synthese, Scientific conclusions need not be accurate, justified, or believed by their authors, philosophers Haixin Dang and Liam Kofi Bright make the case that “the main results of scientific papers may appropriately be published even if they are false, unjustified, and not believed to be true or justified by their author.”

We ordinarily expect scientific statements to be true (reflect the state of affairs as they are), justified by the totality of available evidence (not just a limited subset of available evidence), or, at least, earnestly believed by the person making the statement. But it appears that, at least in some instances, conclusions presented by scientists in scientific publications (journal articles, conference presentations, etc.) don’t conform to these expectations. Furthermore, this is not a flaw but a feature of how science operates.

“Scientists do sometimes report false, unknown and unjustified things that they do not believe, and this is an acceptable part of the scientific process.”

Dang and Kofi present a vignette that illustrates the basic logic:

“Zahra is a scientist working at the cutting edge of her field. Based on her research, she comes up with a new hypothesis. She diligently pursues inquiry according to the best practices of her field for many months. Her new hypothesis would be considered an important breakthrough discovery. Zahra knows that many more studies will have to be done in the future in order to confirm her hypothesis. Further, she has read the current literature and realizes that the existing research in her field does not, on net, support her hypothesis. She does not believe that she has conclusively proven the new hypothesis. Nonetheless, Zahra sends a paper reporting her hypothesis to the leading journal in her subdiscipline. In the abstract of the paper, the conclusion, and talks she gives on her work, she advocates for her hypothesis. Peer reviewers, while also sceptical of the new hypothesis, believed that her research had been carried out according to best known practices and her paper would be a valuable contribution to the field. Her paper, which purports to have advanced a new hypothesis, is published and widely read by members of her community. In subsequent years, additional research in her field conclusively demonstrates that Zahra’s hypothesis was false.”

In one section of the paper, Dang and Bright focus on a historical case study of the physicist William Henry Bragg — who won the Nobel Prize for physics in 1915 — to illustrate the dynamics around statements that are presented by scientists to other scientists as conclusions of their research.¹ Bragg proposed that gamma and X-rays were material particles, a neutral pair consisting of a positively charged α-particle and a negatively charged β-particle bound together, while knowing the theory was not supported by all available evidence. Despite the eventual disprovement of his hypothesis, it sparked critical research and led to scientific advancements in X-ray crystallography. Furthermore, Bragg’s presentation of his conclusions “did not violate the norms of inter-scientific communication.”

Dang and Bright argue that all this makes sense if we expect the norms governing the presentation of scientific conclusions to scientific peers to align with the reality that science works through division of cognitive labor and collective efforts at error correction.

The unique power of science lies in its collective nature, where individual contributions, however flawed or speculative, play a crucial role in advancing understanding. Scientific communities prioritize the needs of the collective enterprise over the perfection of individual claims, so they do not hold the presentation of scientific conclusions to strict standards of truth, justification, and belief.

Science, they argue, achieves its rigor not through the infallibility of each researcher but through the interaction of diverse, often conflicting contributions of multiple researchers. This process of collaboration, correction, and refinement requires that individuals be free to make bold, sometimes speculative claims. These claims, while not necessarily true or justified by all available evidence, serve as starting points for further inquiry. By allowing this freedom, the scientific community ensures a diversity of ideas and approaches that fuel progress in even the most complex and challenging domains.

The norms governing the scientific presentation of conclusions are deeply contextual. They vary by discipline and are shaped by the historical and methodological standards of each field. These implicit norms are passed down through scientific training and dictate what kind of evidence and reasoning are deemed sufficient to make a claim worthy of consideration. For example, a physicist and a biologist might approach the question of what counts as “justified” or “worth reporting” differently, reflecting the unique demands and practices of their fields.

All this presents us with an apparent paradox: science, which is often seen as the epitome of rigor, operates under rather lax epistemic standards for individual contributions. This leniency, far from being a weakness, is a strength—it allows for the kind of creativity and risk-taking necessary for breakthroughs. Paul Feyerabend, who championed the role of bold conjectures and methodological flexibility in scientific progress, would not be surprised by this at all.

Over time, the scientific community has developed customs that balance the need for creative conjecture with the collective goal of achieving rigor in our knowledge of the world. This ensures that the group can navigate the uncertainties and complexities of scientific inquiry effectively.

“It is not just that we in fact often will say false things in the course of inquiry, but rather that inquiry could not proceed in a way that was even remotely successful if we did not do so.”

Dang and Bright are playfully showing that science is not a collection of justified, true, and believed individual assertions but a dynamic, iterative process of collective inquiry. Science relies on individual scientists to offer creative conclusions that are potentially true but are not necessarily supported by all available evidence, and these conclusions are then taken seriously by other scientists who subject the hypothesis to scrutiny and investigation until a scientific consensus emerges. The success of this process depends on allowing individual scientists the freedom to make speculative and even erroneous claims, trusting that the community will refine these ideas over time. By adopting this perspective, we can better appreciate the practices that make science a uniquely powerful tool for understanding the world.

“It may seem surprising that the norms in science are less, rather than more, strict than everyday life. Intuitively science might seem to be a place where we operate under especially strict epistemic standards. And in a sense this is true, so long as one understands the “we” in a genuinely plural sense. That is to say, it is the group that must achieve high standards of rigour, and how individuals contribute to that may be somewhat indirect. For the group to achieve this goal we may well need the individuals who make up the group to be emboldened to take creative leaps and offer bold conjectures on matters that are complex, abstruse, and generally difficult to gain any epistemic purchase on… for science as a whole to progress individual scientists may operate in a somewhat gung-ho or ad-hoc manner.”

---

Some considerations around “hedging”

Scientists often “hedge” the conclusions of their scientific research, i.e., they qualify and weaken the claims being made. For example, using language such as:

• “We suggest X as a candidate for further investigation.”

• “Y has not yet been ruled out and should be treated as a possible explanation among other competing explanations.”

• “The results of study A [i.e. not the totality of evidence] support the conclusion Z.”

Dang and Bright note that one could “insist that strictly speaking only such weaker avowals would be proper, and fault those scientists who do not live up to this.”

“We do not doubt that it would be possible to reform scientific communication behaviour such that one sticks to scientific public avowals that are proper according to one of the surveyed norms by insisting on appropriate hedging… However results are conveyed, scientists must decide what claims are worthy of further tests. Our point is that it would be inappropriate for scientists to insist that (in the absence of fraud or mistake or misfortune) these pursuit worthy claims must be true, or justified, or believed to be as much by their proponents.”

In a 2022 paper, Cory Dethier writes,

“… even if Dang and Bright are right that neither truth, justification, nor belief is necessary for pursuit-worthiness (and I think they are right on this point), that doesn’t mean that the statements of scientists shouldn’t be assertions that are held to the relevant norm.”

Dethier says that not hedging by scientists is appropriate in a very specialized scientific context, when their conclusions will only be read by other scientists with relevant expertise, but this practice is no longer suitable when the audience is mixed and includes non-experts and experts who don’t share the common ground.

On this point, I am inclined to agree with Dethier. Truth, justification, or belief do not determine the pursuit-worthiness of scientific claims, but I am of the view that it is indeed preferable for scientists to appropriately hedge and justify their claims.

Gungo-ho conclusions by individual scientists work only if we have robust collective norms of inquiry

For individual scientific conclusions to contribute to collective epistemic progress in the manner envisioned by Dang and Bright, we need scientific communities that are appropriately rigorous and skeptical in what they accept to be the case. In other words, we need scientific communities that adhere to standards of methodological objectivity as outlined by Helen Longino (The Fate of Knowledge, 2002):

1. Venues of criticism: “There must be publicly recognized forums for the criticism of evidence, of methods, and of assumptions and reasoning.”

2. Uptake of criticism: “The community must not merely tolerate dissent, but its beliefs and theories must change over time in response to the critical discourse taking place within it.”

3. Public Standards: “There must be publicly recognized standards by reference to which theories, hypotheses, and observational practices are evaluated and by appeal to which criticism is made relevant to the goals of the inquiring community.”

4. Tempered Equality: “Where consensus exists, it must be the result not just of the exercise of political or economic power, or of the exclusion of dissenting perspectives, but a result of critical dialogue in which all relevant perspectives are represented.”

Many times in the medical field, a physician’s social standing or a stakeholder’s lobbying power have led to the creation of a false consensus. As Dang and Bright note, “it is the group that must achieve high standards of rigor, and how individuals contribute to that may be somewhat indirect.” But if the group is falling short of achieving high standards of rigor, the whole process will go awry.

The line between inter-scientific communication and public communication has been blurred, and this creates problems

Dang and Bright are considering a scenario where scientists are presenting conclusions of their research to other scientists. In recent decades, the boundary between inter-scientific and extra-scientific communication has become highly porous. Journalists as well as scientists often communicate scientific conclusions presented in academic journals, preprints, and scientific conferences to the public without appreciating the broader context of scientific inquiry. In addition, scientists have been presenting their pet hypotheses directly to the public in the form of magazine articles or trade books. Consider popular such as Body Keeps The Score (Bessel van der Kolk), How Emotions Are Made (Lisa Feldman Barrett), The Hidden Spring (Mark Solms), Brain Energy (Christopher Palmer), and The Anxious Generation (Jonathan Haidt). All of these are instances where professionals are making a case to general readers for a thesis that is treated by many of their professional expert colleagues with skepticism and/or qualified disagreement.

Dang and Bright write:

“if a layperson were to only follow the specialist journals, the layperson may come to the conclusion that scientists often report false things, after reading many contradictory avowals made at the conclusion of competing cutting-edge research papers. However, it would be a mistake for the layperson to think that when scientists make reports to policy-makers that these reports also have the same epistemic status as papers in scientific journals written for a primarily specialist audience.”

Not only are laypersons reading specialist journals these days without the requisite expertise or familiarity with the relevant scientific norms, but scientists are also speaking directly to laypersons as if they were presenting their conclusions to their scientific peers. It’s one thing for scientists to present a hypothesis to their expert peers and entirely another to present that hypothesis to the general public.

This is a serious problem. Addressing this not only requires the development of customs around what sort of assertions are appropriate for a scientist to make to the public, but it also requires journalists to treat the assertions of an individual scientist or the conclusions of a single paper with more skepticism.

I have great admiration for journalists who have an aptitude for this sort of thing. Here are two examples off the top of my head:

• Shayla Love’s story “Inside the Dispute Over a High-Profile Psychedelic Study” (VICE, May 2022) about a Nature Medicine paper that suggested that psilocybin works for depression by global increases in brain network integration

• Stephanie M. Lee’s story “Jonathan Haidt Started a Social-Media War. Did He Win? How a fight about science and screens got messy, fast.” (Chronicle of Higher Education, Nov 2024) about the scientific controversy around Haidt’s thesis that social media is causally responsible for worsening mental health of teenagers.

A world where scientific communications aimed at expert peers are not necessarily accurate, justified, or believed by their authors is messy enough, but a world where scientific communications aimed at the public are not necessarily accurate, justified, or believed by their authors is disastrous.

---

See also:

• A post inspired by Liam Kofi Bright’s paper White Psychodrama

Psychiatric Psychodrama

• My discussion of Christopher Palmer’s book Brain Energy:

Do all roads lead to mitochondria?

---

1. Dang and Bright use the term “public avowal” in a special sense to refer to claims that scientists feel they are in a position to put forward as a conclusion of their research, and these are aimed at the wider scientific community. This is different from “public scientific testimony,” which are statements from scientists aimed at the general and the policymakers, and public scientific testimony is subject to different standards. A casual reader is going to have a difficult time differentiating between “public avowals” and “public scientific testimony,” so I am not using the term public avowal in my discussion here.