Writing is Thinking

This week, researchers at the University of Florida found that while AI can be a valuable assistant, it falls short of replacing human scientists in many critical areas.

The researchers tested how well popular generative AI models including OpenAI’s ChatGPT, Microsoft’s Copilot and Google’s Gemini could handle various stages of the research process. They put these AI systems through six stages of academic research – ideation, literature review, research design, documenting results, extending the research and final manuscript production – while limiting any human intervention. What they discovered was a mixed bag of capabilities and limitations.

Still, Japanese company Sakana announced this month that a paper written by its "AI Scientist" passed the peer review process at a top machine learning conference workshop, possibly the first time a fully AI-generated paper has passed the peer review process.

The company said: “We believe that the next generation of AI scientists will usher in a new era of science. The fact that AI can generate entire papers that pass peer review at top machine learning international conference workshops is a sure sign of progress to come. But this is just the beginning. AI will continue to improve, perhaps exponentially. At some point in the future, AI will likely be able to generate papers at or above human levels.”

Comparing AI science with human science is not the ultimate goal, says Sakana. “What is most important is that discoveries made by human and AI science will contribute to human prosperity, such as leading to the treatment of diseases and clarifying the laws that govern the universe.”

Commenting on Sakana’s earlier developments last year, Karin Verspoor, Dean of the School of Computing Technologies, RMIT University, in Australia, pointed out that Sakana claims its AI tool can undertake the complete lifecycle of a scientific experiment at a cost of just US$15 per paper – less than the cost of a scientist’s lunch. One of Verspoor’s concerns is that, if AI-generated papers flood the scientific literature, future AI systems may be trained on AI output and become increasingly ineffectual at innovating.

However, the implications for science go well beyond that. “There are already bad actors in science, including ‘paper mills’ churning out fake papers. This problem will only get worse when a scientific paper can be produced with US$15 and a vague initial prompt. The need to check for errors in a mountain of automatically generated research could rapidly overwhelm the capacity of actual scientists.”

A review by Miryam Naddaf in Nature last week highlights the growing use of AI in the peer review process. “AI systems are already transforming peer review — sometimes with publishers’ encouragement and at other times in violation of their rules. Publishers and researchers alike are testing out AI products to flag errors in the text, data, code and references of manuscripts, to guide reviewers toward more-constructive feedback and to polish their prose. Some new websites even offer entire AI-created reviews with one click.”

The article quotes Carl Bergstrom, an evolutionary biologist at the University of Washington in Seattle. He says if reviewers start relying on AI so that they can skip most of the process of writing reviews, they risk providing shallow analysis. “Writing is thinking,” says Bergstrom.