AI-Generated Images and Videos in Research Publications
In 2024, a research article in Frontiers in Cell and Developmental Biology (Impact Factor: 4.6) was retracted due to fake, AI-generated images (Guo et al., 2024). Retractions occur only when an article has serious flaws indicating that it cannot be trusted. A quick look at the first figure raised doubts: the rat in the image had testicles larger than its head, stem cells in a petri dish were overflowing like a scene from a science-fiction comic, and the text in the image was nonsensical. Unsurprisingly, the figure had been crudely generated by a generative AI model.
The bigger problem was that the editorial team did not reject the article despite the obvious flaws; the figure above was not the only one. After publication, readers raised concerns, whereupon the journal retracted the article. The journal later thanked readers for pointing out the issue, but its failure to detect such an obvious problem triggered further concern within the research community.
This incident was shocking, but not the first warning about generative AI in research. A year earlier, Nature had banned all AI-generated images and videos, citing their unclear and unverifiable sources. However, Nature also relies on authors, peer review, and editorial checks—processes similar to those that failed to detect the flawed article with the infamous rat figure.
The earlier example of the rat was easy to recognize as fake, but this is not always the case. The images above were all generated by AI systems, and it is much more difficult to determine whether they are real or fake. According to Nature, a quiz conducted by the company Proofig showed that 450 researchers were able to correctly identify AI-generated images only about 50% of the time (Kwon, 2024).
In the same report, an expert in image forensics noted that she often found obvious AI-generated text in publications, including standard chatbot phrases that authors or fabricators had forgotten to delete. She saw no reason why similar falsifications could not also occur in images or data. Since not all researchers are honest and reviewers may overlook these problems, trust in research and publications is likely to decline if strict measures are not implemented.
- Optional task:We have learned that generative AI tools can create impressive illustrations in a short time, but the techniques to detect their misuse are unfortunately less developed than the generative AI systems themselves. How, then, can publishers ensure academic integrity in the literature? How can the current review system, which relies heavily on peer review, be improved or even replaced to adapt to the new AI era? As a critical consumer of academic literature, what can you do to verify the accuracy of your sources? The possible answers to these questions are still being debated, so share your suggestions, even if they are still rough ideas. An internet search can also help with brainstorming, for example in articles such as this one from Nature.
