Playing with Numbers
Credit: Daboost
How research methods distort nicotine effects and risks
By Cheryl K. Olson
“The paper seems like a joke.” That’s what Harvard researcher Miguel Hernan said recently to the journal Science about a report linking e-cigarettes and strokes.
The article was concocted by a dubious research group, founded to help young international medical school graduates get coveted authorship credits. Its analysis of U.S. government survey data claimed that respondents who vaped had a higher risk of stroke, at younger ages, than those who smoked. Its glaring flaws included inflating the number of survey takers by tens of thousands and failing to correct for the relative youth of vapers.
Despite this, the 2022 paper’s findings found their way into media headlines and anti-vaping advertising. The Science article credits Gal Cohen and Floe Foxon with sounding the alarm on this appalling study.
Subtler issues that affect research quality, and how research is perceived by the public, are harder to spot. Research methods may seem a dull or arcane topic. But a peek at how the research sausage is made reveals some simple yet surprising ways that the process can go wrong.
Sometimes old habits or unquestioned assumptions are to blame. Just as typewriters affect how we text on our mobiles, legacy cigarette research methods and mindsets influence how we study noncombustible nicotine products.
Hours of Vaping?
Everyone understands cigarettes. When it comes to totting up use, cigarettes are easy. They come in standard units. You light, puff and extinguish. Not so for products such as vapes. How, then, do researchers compare smoking with these new nicotine-delivery systems?
“There’s a lot of research showing that people who use e-cigarettes graze throughout the day,” says Arielle Selya, who conducts nicotine product research at Pinney Associates. “Unlike cigarettes, there’s no defined stopping and starting. They don’t have to finish a discrete unit; they just puff on and off.” Measuring this kind of variable, intermittent activity is a challenge.
This problem is not unique to vaping. Studying nicotine pouch use, I found unexpectedly wide variations in what people did and what they thought was normal. Some tossed a pouch in the trash after 10 minutes or 15 minutes. Others kept one in their mouth for a couple of hours. A few sometimes reused a pouch they’d started earlier or cheeked pouches of two different flavors at once.
As an example of what can go wrong, Selya pointed to a recent study of vaping and respiratory symptoms. To the authors’ credit, they tried to measure heaviness of e-cigarette use. The problem was the poor fit between their question and the behavior. They asked, “How many hours did you use electronic cigarettes per day?”
“I’m not a vaper, but that seems like such a strange question,” says Selya. “Like asking how many hours do you spend drinking water?”
Better approaches to measuring nicotine product use include writing down what you’re doing whenever a device pings you (ecological momentary assessment) or in a daily diary.
Twisted Terminology
Another holdover from cigarettes is the way tobacco is seen as the default flavor for all nicotine-containing products. “With e-cigarettes, you have to add a tobacco flavor,” notes Selya. “But researchers often say ‘flavored’ when they mean ‘non-tobacco flavored’–in some communications even the NYTS team does this–but tobacco itself is a flavor! This generates misunderstandings.”
Nicotine research terminology can defy common sense. Consider the concept of “abuse liability.” In everyday English, abuse implies harm. When the U.S. Food and Drug Administration assesses new drugs, stricter regulation may be required if there’s abuse potential, defined as “intentional, nontherapeutic use” to “achieve a desired psychological or physical effect.” An effect like euphoria, hallucinations or distorted thoughts or perceptions.
When it comes to reduced-harm nicotine products, abuse potential becomes, weirdly, a plus. A backhanded compliment. If you want to attract someone away from cigarettes, features like rapid nicotine absorption, relaxation and relief of withdrawal encourage that transition.
Abuse liability also illustrates another nicotine methodology vexation: there is no agreed-on way to measure it. One article looked at comments made by the FDA on manufacturers’ submissions for multiple types of nicotine products. Regulators considered a whole range of measures related to abuse liability, from product chemistry and pharmacokinetics to subjective factors. Of the latter, “liking” the product turned out to be the most reliable and sensitive abuse liability measure!
Misleading Measures
Again, cigarettes are simple and familiar. Novel nicotine products, by contrast, come in ever-evolving variations. U.S. government surveys, such as the Population Assessment of Tobacco and Health (PATH) and National Youth Tobacco Survey (NYTS), measure trends in who is using what products. The results are widely used and reported. However, for survey results to make sense, people must understand the questions.
Discrepancies in results suggest that research participants often misunderstand nicotine products and/or the terms being used to describe them. For example, answers about vaping brands and device types often don’t match. In the NYTS, just two-thirds of teens who said they “usually” used a pod/cartridge brand of e-cigarette (such as Juul, Logic or Vuse) also said they “most often” used a pod/cartridge device. Almost one in five adults in the PATH study had these kinds of mismatched answers about their vaping behavior.
Some questions have even larger errors. “The NYTS asks whether your e-cigarette product contains nicotine salts,” says Selya. “And overall, about 50 percent said they don’t know.”
This is also true for so-called “concept” flavors, she notes. “Not strawberry-banana, but something like cosmic fusion. When youth are asked about concept or ice flavors, they don’t know the characteristics of their product, or maybe don’t understand those words.”
NYTS first asked youth about tobacco-free nicotine pouches in 2021. That year, just 1.9 percent of teens reported ever using one. Checking the details, I found a flaw: The questionnaire defined nicotine pouches as “flavored.” However, over a third of teen ever-users said the pouch product they used was unflavored. (Perhaps they confused pouches and snus?)
A further example: the 2023 NYTS found that 1 percent of youth—an estimated 370,000—had ever used a heated-tobacco product. At the time, that product category was not sold in the United States.
As Ray Niaura of New York University told me, “That can’t be right. Literally, it’s impossible. So that means it’s measurement error.”
This suggests young survey takers were befuddled. “Kids aren’t going to know,” says Niaura. “‘Heated tobacco: Yeah. I smoked a cigarette. It’s heated. I light it on fire.’”
Yet the Centers for Disease Control and Prevention reported the result without comment or explanation.
If a product is only used by a small percentage of people, these sorts of errors could create unreal changes in year-to-year trends. The reporting of those potentially misleading trends affect the perceptions of academics, regulators and the public. “With that amount of uncertainty and some of the low numbers, it’s hard to figure out what’s the signal versus the noise,” notes Selya.
Questionable Choices
Another seemingly simple but complicated issue: Who counts as a current product user? Youth surveys typically ask “have you used e-cigarettes at all, even a puff, in the last 30 days?” Surveys aimed at adults commonly ask, “Do you currently use e-cigarettes some days, every day or not at all?”
If you assume capturing any youth e-cigarette use is important, then “even a puff” makes sense. But it also makes it difficult to separate teens who are briefly experimenting from teens at risk for problematic ongoing use.
In studies that look at how using nicotine products affect some aspect of health, researchers choose what outcomes to measure. Their choices can suggest biases or suspicious holes in what’s reported.
A recent study using PATH data tried to compare e-cigarette use and the age at which people developed asthma. “Why age of asthma onset rather than whether they developed asthma at all?” says Selya. “Often, I read a study and think, did you look at these other related outcomes? If so, why weren’t they published?” This issue of results that may exist but aren’t reported are known as the “file drawer problem.” Preregistering study plans would avoid this issue.
Researchers, Meet Users
Before I dove deeply into tobacco harm reduction, my research focused on the effects of violent video games on youth. Finding discrepancies between research reports and what teens told me, I realized that many of the field’s most-cited “experts” had never actually played or even observed the games they studied.
Similarly, many nicotine researchers seem to have never held or used the noncombusted products they study. This leads to findings that don’t reflect real-world situations. One example is an article by Sebastien Soulet and Roberto Sussman on metal contents of e-cigarette aerosols. They found that researchers were overheating tank vaping devices, generating aerosols that would be “likely repellent to human users.”
“I think there’s a big disconnect and abysmally low involvement of actual consumers, the people affected by policies,” says Selya. Partnering with people who actually know and use novel nicotine products would be a giant step toward improved research quality.
References
Foxon F. (2023). Discordant device/brand reporting among adolescents who used e-cigarettes in the National Youth Tobacco Survey. Nicotine and Tobacco Research. https://doi.org/10.1093/ntr/ntad228
Joelving F. (2024). Prescription for controversy. Science. https://www.science.org/content/article/questionable-firms-tempt-young-doctors-with-easy-publications
Selya A, Ruggieri M, Polosa R. (2024). Measures of youth e-cigarette use: strengths, weaknesses and recommendations. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2024.1412406
Soulet S, Sussman RA. (2022). A critical review of recent literature on metal contents in e-cigarette aerosol. Toxics. https://www.mdpi.com/2305-6304/10/9/510
Vansickel A et al. (2022). Human abuse liability assessment of tobacco and nicotine products: approaches for meeting current regulatory recommendations. Nicotine and Tobacco Research. https://doi.org/10.1093/ntr/ntab183
