Should There Be Peer Review After Publication?

N 2020, as a first-year graduate student, Laura Luebbert was asked to present two classic papers on the honeybee waggle dance to her journal club. Individual bees use this dance to communicate with the rest of the hive about the direction and distance of food. Because Luebbert was new to the topic, she decided to read additional studies. In a recent blog post, she wrote that while reading, “I sensed something strange; I had the feeling that I was looking at the same data over and over again.”

Luebbert would eventually team up with her adviser at the California Institute of Technology, a computational biologist named Lior Pachter. Together, they analyzed a series of honeybee papers, which all happened to be co-authored by a renowned scientist, Mandyam Veerambudi Srinivasan. The pair ultimately found what they characterize as “problematic behavior across numerous articles.”

Luebbert and Pachter outlined their findings in a manuscript, which was made public on a preprint server in May. But their concerns didn’t draw much attention until July, when Pachter posted a now widely viewed thread on X, the platform formerly known as Twitter, calling the honeybee studies “junk.”

Srinivasan has since responded on the pair’s blog post, criticizing them for conducting their investigation in an “unprofessional manner.” Rather than contacting him directly, he wrote, “they proceeded to draw conclusions based on limited understanding of the scientific area they were examining.” This kind of behavior, he wrote, “is not conducive to the advancement of science.”

Luebbert recently completed her Ph.D. and will be working as a postdoctoral researcher at the Broad Institute this fall. In a Zoom interview, she spoke about the dust-up and her years’ long effort to correct what she characterizes as flaws in the scientific literature. Our conversation has been edited for length and clarity.

Undark: I’d imagine that it must have been unsettling to be a graduate student reading papers by a world-renowned researcher and finding yourself doubting the data. Do you remember your thoughts and feelings at the time?

Laura Luebbert: I was definitely shocked by seeing, I mean, we’re really talking about identical data. So I definitely spent a lot of time making sure that I was interpreting the experiments correctly. I wanted to discuss my finding with other scientists, specifically other scientists who are experts in insect behavior. So when it was time to [present to] my journal club, I included a very detailed discussion section where I showed all of the instances of duplicated data that I found at the time.

What was really shocking to me — what really stuck with me, and it just became this frustration that I held on to for the rest of my Ph.D. — was the response that I got then. Nobody wanted to talk with me about it. There was just zero interest and also zero surprise. The senior scientists that I reached out to were just kind of like, “Yeah, that’s just how science is.”

UD: What did you do to try to get the word out since it sounds like the scientists that you had spoken with directly were not interested in following this?

LL: I did tweet about it. At the time, I barely had any followers, so it’s not really like that was getting the word out. But one really worthy-of-mentioning comment was by Dr. Elisabeth Bik, who has really made a career of flagging problematic papers. She helped me actually find these papers on PubPeer, and then also encouraged me to write a comment there. So I did write an anonymous comment on PubPeer at the time, which was completely ignored by authors and journals. But at least I was like, “Okay, it’s out there.”

UD: What is PubPeer?

LL: PubPeer is a website where you can add comments to individual papers.

UD: Eventually you joined Lior Pachter’s lab, and it sounds like the two of you decided to take a careful look at this body of work. Why do you think nobody had done this before?

LL: It’s not that easy — it’s very unappreciated work. We really spent a lot of time on these papers, reading the papers, trying to understand the papers, trying to write a very objective report on what we found and what we thought was problematic. This kind of work, it just takes a lot of time, and it’s not worth it for most scientists. There’s just no appreciation for it. Quite the contrary: I was told that it could really be catastrophic to my career if I talked about these issues.

UD: How much time did you spend on this effort?

LL: Quite a lot. During the first preparation for the journal club, this was right at the start of the pandemic, so I was stuck at home anyways. I really worked on them for a month full time.

So, actually, this doesn’t really have anything to do with the question you asked, but the reason we revisited the papers is because this frustration stuck with me so much so, that when Lior and I were talking about what I wanted to do after my Ph.D., I mentioned this as a reason why I thought that maybe academic science wasn’t for me.

Lior’s response was completely different from the response that I had before. He was immediately like, “Hold on. You just found this and nobody ever cared? It never got out?” I showed him my findings, and he was really shocked. Then, as we were reading the papers together, we actually found a lot more issues.

Since October — that’s when Lior and I started working on the manuscript.

UD: Wow, so it’s July right now. You posted the manuscript in —

LL: In May.

UD: That’s a long time. Did this work compete with your Ph.D. research?

LL: Yeah, definitely. I was also trying to finish my Ph.D., which has absolutely nothing to do with insect behavior.

UD: In your blog post, you write about your efforts to contact the publishers of some of the honeybee papers, as well as your efforts to get your manuscript up on a preprint server. The process doesn’t sound very easy. Why do you think journals and preprint servers were reluctant to engage with the findings?

LL: I’ll start with why journals are reluctant. Unfortunately, for a journal, the incentives right now are not aligned. They have their own peer-review process, but once the paper is published, journals don’t want to admit that there might be something wrong with a paper that they have published. This kind of really goes against how science works because the whole point is that we question things, and sometimes we get things wrong.

Obviously, in this case, we’re talking about more egregious errors. But journals are really reluctant to admit that there was something wrong with the paper they published if it wasn’t brought up in their own peer-review process.

For preprint servers: We started with bioRxiv. They have a very specific rule that they only publish new research, which I understand. I think that’s fine, but I think that makes it obvious that that means that we need a different platform for this kind of work. And what Lior and I found after we had written this manuscript is that it’s really hard to get work like this out. There really isn’t a platform.

We’re really grateful to the arXiv that we were able to post our manuscript there. But we also wrote a blog post because we wanted people to be able to comment on our manuscript. All of this just really goes to show that we need a better platform to publish this kind of work.

UD: When researchers see something that strikes them as problematic, should the first step be to contact the study author, the journal, someone else?

LL: I don’t think that is a solved problem. That’s exactly the issue that I had. Even the senior professors at Caltech didn’t know where to send me.

UD: Srinivasan said that you and Pachter conducted your investigation in an “unprofessional manner.” How would you respond to that?

LL: I think it’s a bit difficult to respond to that because I don’t know exactly what he thought was unprofessional. If you read our manuscript, we really took great care to ensure that we don’t mention anyone by name, that it’s a very objective telling of what we found in the papers. The reason we wrote the blog post is to provide a platform for scientists and experts, including Srinivasan, to respond and to talk about the issues that we found. And we also made sure that all of our analyses were very transparent and reproducible.

UD: Do you think peer review should or could be the place where these kinds of discrepancies get caught? Are most peer reviewers looking at researchers’ past work to see if the data or techniques change from study to study?

LL: What Lior and I did was a peer review. It just happened outside of the journal-confined peer review process. I don’t think it’s reasonable to expect a journal’s peer reviewers to look at a whole body of work every time they’re reviewing a paper because they’re doing peer review unpaid.

But I do think that there should be a platform for post-publication peer review. If these kinds of issues are caught prior to publication, journals respond very differently versus post-publication, when somehow they have this incentive to just kind of shove it all under the rug.

UD: What would you like to see happen going forward?

LL: I would love to see more objective responses from the [honeybee research] authors, to see if any of the things we talked about can actually be explained, or if any of the data is still available and the analyses could actually be redone.

UD: Will you and Pachter be submitting your new manuscript to a journal?

LL: We haven’t talked about that yet. Our main goal was to correct the scientific record, and it was very unfortunate that it took a viral tweet to get there, but at least now we have enough of a public response that journals were kind of forced to look into it.

And more importantly, our finding was widely shared within the community of entomology and behavior. However, it would still be nice for the manuscript to be reviewed, so it can be cited as something that is peer reviewed and published in a journal.

UD: In your blog post, you two proposed a journal that would be dedicated to reviewing a body of work by a scientist or a team of scientists. Can you talk about that a little bit?

LL: The Journal of Scientific Integrity was kind of this idea that we need somewhere to go with this kind of work. We wrote a manuscript and realized there’s no venue for this at all. Interestingly, on [X], we actually got a lot of responses from scientists talking about how they actually tried to fund a journal like that, but there’s just very little interest, unfortunately.

UD: Is there anything else that you’d like to add?

LL: Carefully reviewing bodies of work is extremely important to the advancement of science. Very often it saves lives. We’ve seen it in cases like Theranos. Or Dr. Elisabeth Bik has been unveiling a lot of problematic work by Professor Didier Raoult, who is actually working with patients, so there are literally lives on the line when reviewing problematic work.

Obviously, in Srinivasan’s case, he is not working on human health. But I still think that carefully reviewing papers is extremely important to science, and that is true for our paper as well. So anybody can and should carefully review our manuscript, including Srinivasan, and comment on it, especially if they find we’ve done things wrong. I think that the whole point of science is to try to find the truth together.