A new study suggests that advanced artificial intelligence models can generate more thorough, accurate, and constructive peer reviews of unpublished academic papers than well-paid human experts operating under ideal conditions. In blind comparisons judged by an AI model and by former journal editors, the AI-written reviews won the large majority of matchups and contained about a third the rate of incorrect claims, according to a working paper posted online by Harvard University researchers.
The findings point to potential changes in how scientific journals evaluate research while raising questions about whether widespread reliance on AI could make scientific thinking more uniform.
The challenges of human and AI peer review
Peer review is the standard quality-control process in scholarly publishing, in which independent researchers in the same field read a draft of a scientific paper to help the authors improve their work and advise journal editors on whether the study merits publication. Despite its central role in science, the traditional review system struggles with frustrating delays, difficulties recruiting qualified volunteers, and low agreement between different experts evaluating the same manuscript. Human referees also frequently miss major errors, offer superficial feedback, or show bias toward familiar methods and prestigious institutions.
Recently, scholars have turned their attention to large language models, which are AI systems trained on vast collections of text to analyze and generate written language. However, applying these tools to scientific work has sparked concerns about fabricated citations, excessive leniency, learned biases, vulnerability to hidden manipulation commands inside manuscripts, and false statements presented with confidence. An analysis of AI-generated research summaries, a different task from reviewing manuscripts, found that chatbots frequently overgeneralized scientific findings compared to human experts, with newer models producing up to 73% more broad generalizations.
Don’t miss the next big psychology finding
Every morning, PsyPost sends a short briefing on the most interesting new peer-reviewed research in psychology and neuroscience.
Because of these risks, most prior analyses concluded that AI should only supplement human reviewers, and two 2026 comparisons of AI and human peer reviews yielded conflicting results. In a 2026 study of 119 manuscripts at BMJ Open, human reviewers raised more detailed and varied points than ChatGPT, whereas in a 2026 blinded study of 40 manuscripts at a cardiology journal, two editors rated reviews from Claude, ChatGPT, and Gemini higher than human reviews on five of seven criteria. Against this backdrop, Gary King and Steven Worthington of Harvard University set out to test how a leading AI model compares not against rushed, unpaid journal referees, but against human peer reviewers performing at their best. Their working paper has not yet been peer reviewed.
Testing AI against well-paid human experts
To establish a strong human benchmark, the researchers examined every eligible submission to rapidPeer, a pre-submission review service developed at Harvard University’s Institute for Quantitative Social Science, which King directs. Through this service, outside experts selected by specialists in the field are paid well to read an anonymized faculty manuscript from start to finish and write at least 1.5 to 2 pages of constructive feedback before the paper is sent to a journal. These human reviews were unusually well regarded: in a survey taken a week after delivery, 88% of the 74 responding authors rated their review as high or very high quality, and former journal editors consulted by the researchers described the reports as among the strongest they had seen.
The complete dataset included 124 scholarly manuscripts and 134 human reviews collected between 2019 and 2026 across statistical, mathematical, and qualitative research fields, with 10 papers receiving two independent human reviews. For each manuscript, the researchers generated an AI review using Anthropic’s Claude Opus 4.8 at its highest reasoning setting under two distinct conditions. In the matched condition, the AI model received the anonymized paper and the exact rapidPeer reviewer instructions with no external tools. In the tuned condition, the model received a custom prompt designed prior to any judging along with internet search access, which it used an average of 12.4 times per review.
To evaluate the reviews without letting a model grade a system from its own developer, the researchers used OpenAI’s GPT-5.5 as a separate AI judge from a different company. After first identifying which of seven rapidPeer evaluation areas (theory and idea, methods, weaknesses, framing, communication, figures and tables, and sources) applied to each manuscript, the AI judge ranked each pair of reviews overall, ranked them head-to-head within each relevant area, and scored each review on a scale from 0 to 10. Every comparison was run in both presentation orders, and a review was credited with a win only when preferred in both orders. The overall ranking was based on the two reviews alone; in a later check that gave the judge the full manuscript, the AI review was still preferred in at least 132 of 134 pairs, and the human review was never preferred in both presentation orders.
In addition, three former editors of leading scholarly journals independently judged 10 randomly selected pairs of reviews through a blinded side-by-side web interface, yielding 30 human judgments.
AI reviews come out ahead by wide margins
Across all evaluations, the AI-generated reviews outperformed the human reviews by wide margins. The headline results were reported descriptively, as win percentages and average ratings, without formal statistical tests or confidence intervals. In the overall head-to-head comparisons, the AI judge preferred the AI review in 100% of decided pairs (132 out of 132), and the blinded former journal editors preferred the AI review in 87% of their judgments (26 out of 30). Notably, the human editors agreed with the AI judge an average of 87% of the time, compared to agreeing with one another 80% of the time.
When the AI judge evaluated the seven specific review areas individually, the tuned AI reviews won 100% of decided pairs in sources, weaknesses, and methods, and 99% in figures and tables. In the remaining three categories (theory and idea, framing, and communication), the AI review won 92% of the time. On the 0-to-10 scale, AI reviews averaged about 2.6 points higher than human reviews, which averaged about 6, a gap that was similar in the editors’ sample.
The researchers ran 10 sets of follow-up checks to verify that the AI model was not winning simply by writing longer reports or using a polished tone. When restricting the comparison to 44 pairs where the AI and human reviews had similar word counts, the AI review still won 100% of the time. Rewriting the human reviews to mimic an AI writing style did not improve their rankings, and the AI reviews outperformed human reviews across all subgroups of manuscript novelty, research methodology, author seniority, and the quality of the journal where the paper was eventually published. Comparing the two AI setups showed that the matched AI condition performed nearly as well as the tuned condition across every category except sources, where adding web search access raised the AI win rate from 62% to 100%.
An audit of factual accuracy against the manuscripts indicated that the web-enabled AI reviews contained zero fabricated sources, and 5.5% of their claims about the paper were judged incorrect by the AI judge, roughly one-third of the 15.9% error rate found in the human reviews. To verify that the AI judge could detect misstatements, the researchers inserted either one accurate sentence or one false sentence contradicting the paper into each review; the judge caught all 258 false statements and flagged none of the 258 accurate ones. Replicating the study across 416 comparisons using both Claude Opus and GPT models as reviewers and judges with a different prompt and scoring rubric yielded AI win rates of 97.1% under the Opus judge and 94.2% under the GPT judge.
“The article demonstrates that AI-generated reviews can now surpass even human (pre-submission) reviews across several dimensions of review quality, suggesting that recent large language models have the potential to substantially improve the quality, consistency, and efficiency of manuscript evaluation,” Kilian M. Pohl, the founder of the CNS Lab and a professor in psychiatry and behavioral sciences and, by courtesy, in electrical engineering at Stanford University who was not involved in the study, told PsyPost. “The findings challenge the longstanding assumption that expert human reviewers provide more useful manuscript feedback than AI.”
George Currie, a senior strategy advisor for open research at UK Research and Innovation (UKRI), the United Kingdom’s national research funding agency, who was not involved in the study, told PsyPost that the results should be weighed against the realities of traditional academic publishing. Currie noted that he was speaking for himself, not for UKRI.
“Before turning our noses up at AI-led peer review, we should be honest about the system we already have: peer review is resource intensive, largely unpaid, variable in quality, functions as journal gatekeeping as much as research improvement, and does not reliably prevent poor research from entering the literature,” Currie said. “The potential attraction of AI is not that it replaces a flawless human process, but that it could reduce burdens and increase transparency across the system while speeding up the communication and reuse of research.”
The absence of fabricated references in these reviews stands in tension with an earlier evaluation of AI-generated mental health literature reviews, which found that nearly two-thirds of citations produced by AI models were invented or contained errors, particularly for specialized topics. However, that earlier study tested literature reviews generated from scratch, whereas the new study evaluated AI critiques of existing manuscripts, with web retrieval enabled in the version reported here.
How journals might use AI and the risks of uniformity
In discussing what these findings mean for academic publishing, the authors distinguish between the two main purposes of peer review: helping authors improve their drafts and helping editors decide what to publish. Their experiment addressed only the first goal, though the authors propose several ways journals might redesign their workflows around AI feedback.
For example, journals could publish an official AI review prompt tailored to their editorial standards and ask authors to submit their manuscript alongside the resulting AI review and a response letter explaining any points where they disagree with the model. Editors could then run a second, undisclosed AI review that includes checks for hidden prompt manipulation, consulting human referees only in difficult cases while keeping final publication decisions in the hands of a human editor.
At the same time, the authors warn that widespread reliance on AI carries collective risks for science. Even if an AI reviewer helps an individual scholar spot flaws and brainstorm ideas, researchers across a field using similar models may be nudged toward conventional, mainstream arguments, making scientific literature more uniform overall.
Eyal Aharoni, an associate professor of psychology, philosophy, and neuroscience at Georgia State University who was not involved in the study, told PsyPost that the paper is likely to be of broad interest to scholars. “As an AI researcher, the basic effect doesn’t surprise me (though the magnitude of the effect does), and I am persuaded by the authors’ argument that such findings put pressure on the field to establish appropriate policies and procedures for balancing its expected benefits and risks,” Aharoni said. “I also appreciate that they sketch out a few concrete ways in which integration of this technological application might be formalized.”
Gustavo A. Patino, an associate dean for undergraduate medical education and associate professor in the Department of Medical Education at Western Michigan University Homer Stryker MD School of Medicine, who was not involved in the research, told PsyPost that he expects AI review tools to become part of journal workflows within about five years as language models improve, though he pointed to a broader trade-off for scholars.
“The authors focus all the discussion around benefits of facilitating the editorial process, but that ignores the benefits that peer review has on the larger research ecosystem,” Patino said. “Serving as a peer reviewer exposes you to a diversity of ideas and scholars in the field, some of which will not end up being published at this time but that still teach you a lot. Non-reviewers will not get the chance to learn from those projects that end up in rejection.”
Caveats, ethical questions, and what to test next
There are several limitations to keep in mind when interpreting the study’s results. First, the researchers evaluated the quality of the review reports themselves, but they did not measure whether revising a paper in response to an AI review actually leads to a stronger final article than revising in response to a human expert. Second, although the full corpus covered 134 review pairs, the blinded human evaluation relied on three former editors judging a subset of 10 manuscripts overall rather than scoring each of the seven review dimensions separately. Third, the human reviews came from a pre-submission service designed to offer constructive coaching rather than gatekeeping for a selective journal.
“Given the overly broad claims made in the abstract, readers might incorrectly conclude that AI is now better than human experts at determining which scientific papers merit publication,” Pohl cautioned. “However, the authors evaluated the quality of constructive feedback, rather than AI’s ability to identify groundbreaking research, detect subtle scientific errors, or assess whether a manuscript meets the standards of a particular journal. As the authors themselves acknowledge, AI-generated reviews may also favor conventional scientific approaches, potentially discouraging innovative or unconventional research. Ultimately, more comprehensive and better-written reviews should not be conflated with superior scientific judgment.”
Pointing to the study’s design features, Pohl said he places moderate confidence in the results and noted that the three former journal editors were not necessarily specialists in the specific subject matter of every paper they rated. “Consequently, their assessments may have favored clarity, comprehensiveness, and presentation quality over the scientific substance of the feedback,” Pohl said. “Much of the evaluation relied on AI judges, potentially biasing findings to AI-generated writing.”
Michael R. Ladomery, a professor of genetics in the School of Applied Sciences at the University of the West of England who was not involved in the study, told PsyPost that he considers the lack of data on whether authors’ revisions actually strengthened their papers a key shortcoming. He also noted that pre-submission reviewers may lack a specific journal’s context, whereas referees for a submitted paper are likely to judge it against that journal’s audience and standards.
“For example, an experiment that a reviewer considers unnecessary for publication in one journal may be regarded as essential for supporting the level of mechanistic insight expected by another,” Ladomery said. He argued that differences between pre-submission coaching and formal journal refereeing may reflect their different settings and goals rather than the reviewers’ ability.
Ladomery also argued that replacing human referees raises ethical and sociological questions, since “performing peer review is an integral part of academic professional life” that helps scholars sharpen their own critical skills. “There is also a particular intellectual satisfaction in critically evaluating the work of one’s peers and in seeing authors respond constructively to that input by improving their manuscript,” he said. “This vital teamwork cannot be replaced by AI.”
Future studies will need to examine whether AI feedback improves revised manuscripts in practice and how journals can guard against authors gaming published AI prompts.
Aharoni also called for follow-up studies, citing a concern about relying on an AI judge. “The methods are appropriate for a first study of its kind,” Aharoni said, while noting that “using a separate AI does not guarantee independence since they may query each other.” He added that “the authors present a balanced tone in their discussion of implications,” and that “it will also be worth developing their brief argument that there still might be good reasons to retain human review even if AI review is better on its face.”
“It would be valuable to conduct a large-scale, prospective randomized study within established scientific journals, comparing traditional human peer review, AI-assisted human review, and fully AI-generated reviews,” Pohl said. “Such a study would move beyond assessing whether AI can produce higher-quality review reports to address the more fundamental question of whether AI-assisted peer review leads to better scientific publications and more reliable editorial decisions, without compromising scientific originality or accountability.”
The study, “AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices,” was authored by Gary King and Steven Worthington.
