Towards Automating Scientific Review with Google's Paper Assistant Tool
The paper presents Google's Paper Assistant, a tool designed to automate aspects of scientific peer review by leveraging large language models. It evaluates the assistant's ability to generate reviews, assess paper quality, and provide constructive feedback, showing potential for reducing reviewer workload while maintaining quality.
Background
- This paper describes Google's "Paper Assistant," an internal tool that uses large language models (LLMs) to automate parts of the scientific peer-review process.
- Peer review — where experts evaluate a paper before publication — is the backbone of science but is under strain: it's slow, inconsistent, and reviewers are overworked.
- The tool was tested on papers from top AI conferences (ICLR, NeurIPS) and asked to make accept/reject decisions and write full reviews.
- Google reports ~80% agreement with human reviewers, but critics argue automated reviews can't detect fraud, miss subtle reasoning, and bake in existing human biases.
- Amazon and others have tried similar projects; Google's version is notable for scale and its proprietary "Scientific Review Dataset."
- Why it matters: if fully automated review becomes viable, it could transform how research is vetted globally — and which companies control that gatekeeping process.