Bad Science | Good Science

The science of vaping is like a big jigsaw puzzle with individual pieces of the puzzle representing specific scientific studies.

Lady Gaga sang about being caught in a “bad romance.” Is the vapor industry caught in “bad science?” Well, that depends on your definition of bad science.

Let’s take a step back and look at the scientific process.

Science can be summarized, in short, as the pursuit of knowledge through observation and/or experimentation. Scientists are an inquisitive bunch and want to know how things work or what happens if they push the big red button (often labeled “don’t push”). Scientists may have a preconceived opinion about what will happen (called a hypothesis), but equally, they would like to test this using robust scientific techniques. A typical hypothesis-based experimental study would thus follow these steps:

1. Issue identification—what is the scientific question being asked?
2. Hypothesis generation—what do you think the answer is?
3. Study design—how do you test the hypothesis?
4. Data collection—conduct the study and collect raw data.
5. Data analysis—format the raw data appropriately and conduct, for example, statistical analyses.
6. Data interpretation—consider the study design, data analysis methods used and other available scientific data related to the topic.
7. Conclusion—is the hypothesis proven? What follow-up studies may be needed?

All pretty logical and linear: why (do the experiment), how (to do it) and what (are the results). However, the dark art is in step six, and getting this wrong is usually the root of most claims of bad science. But is this a fair criticism?

It is important to realize that data is exactly what it says on the tin: data. There is no such thing as bad data. It may come from a poorly designed study, but this can be taken into consideration during the interpretation step. The important question to ask, and keep asking, is what conclusions can actually be drawn based on the study design and what conclusions are a step too far? Many of the recent vaping health-related headlines are classic instances of this.

For example, we have seen results in mice used to confidently predict health outcomes in humans. Now, a study in mice may tell you important things about mouse biology, but to directly equate the findings to humans is fraught with danger (or at least confounding factors). The last time I checked, mice are different from humans, not only in terms of appearance but also in terms of metabolic rate, brain structure, etc. At best, experiments in mice can give some indication of what may be happening in humans (i.e., may give rise to a new hypothesis), but this also needs to be confirmed scientifically.

The real problem occurs when data is misinterpreted, either in error or, worse, to fit a particular preconceived expectation. This can, in many cases, be unintentional—a bit like the placebo effect (a subconscious bias toward a particular interpretation). Unfortunately, and unforgivably, in some cases, however, there can be a deliberate, intentional effort to make the data fit a particular hypothesis. Ideally, the scientific peer-review process should spot such instances of biased interpretation, but it’s not foolproof (and sometimes study results are communicated in the absence of being subject to peer review).

Personally, I like to think about the science of vaping as a big jigsaw puzzle with the individual pieces of the puzzle representing specific scientific studies. On their own, each jigsaw puzzle piece (i.e., study) does not tell you a lot about the bigger picture. Start fitting them together properly (i.e., with appropriate interpretation of the data) and you start to reveal the bigger picture. Forcing pieces together, however, that are not meant to be connected (e.g., by misinterpreting the data) risks distorting the bigger picture.

So where are we at with vaping science? Well, we have many of the jigsaw pieces, and the bigger picture is starting to emerge (as noted by Public Health England). We’re not there yet, though, and there are still some key pieces to gather, such as those related to the long-term health effects of vaping. We just need to ensure that, as they arrive, the new pieces of the jigsaw puzzle are connected properly and not shoehorned into the puzzle to fit someone’s preconceived view of what the final picture should be.

Coming back to my original question—“Is the vapor industry caught in bad science?”—I am not convinced—to be honest, the more science the better at this time. We still have a jigsaw puzzle to complete. I do feel that the industry, at least on occasion, is being subject to poor scientific practice, in particular with respect to the (over) interpretation of scientific study data.

This is distorting the jigsaw puzzle image and leading to overall confusion, especially among consumers. Everyone involved in helping construct the scientific jigsaw puzzle should be pulling in the same direction, fitting the pieces together properly. This is not just a mere scientific exercise. It’s a fundamental responsibility to consumers so that they can make their own informed choices about what products to use or not use.