Millions of scientific studies appear every year. But which ones really advance research? Scientists at the Jülich Research Center have developed an AI that can evaluate exactly that. It analyzes the content of a study and assigns a novelty score from 0 to 100. With their approach they even won an international competition and prize money of 300,000 pounds.
Every year countless scientific studies and specialist articles are published worldwide. But not every publication advances research equally. While some works provide new insights and expand existing knowledge, others primarily confirm already known results or only make a limited contribution to scientific progress.
Against this background, scientists at Forschungszentrum Jülich have developed an AI-based process that systematically evaluates the degree of novelty of scientific work and makes innovative research more visible. The development was created as part of the international “Metascience Novelty Indicators Challenge”.
AI recognizes the novelty of scientific studies
The aim of the “Metascience Novelty Indicators Challenge” was to develop a scalable method to assess scientific work in terms of novelty value at the time of publication.
For this purpose, the organizers have provided a data set with 100,000 current scientific publications. These were evaluated by experts in the respective field with regard to their novelty.
The teams participating in the challenge should predict these exact ratings as accurately as possible. The team from Forschungszentrum Jülich achieved the best results and, by winning the challenge, was able to secure prize money of 300,000 pounds to further develop the development.
“So far, the assessment of what is really new and valuable in a scientific work has been reserved primarily for human experts,” explains Dr.-Ing. Jann Michael Weinand, Head of the Integrated Scenarios Department at the Institute of Climate and Energy Systems – Jülich System Analysis (ICE-2). “Our approach shows that modern AI systems can support this task with amazing reliability.”
Jülich AI focuses on content instead of citations
Citation numbers have been considered an important indicator of the importance of scientific publications in science for decades. The more frequently a study is cited by other researchers, the more visible it usually becomes within the scientific community.
However, citations do not necessarily say anything about the degree of novelty of a research work. There is therefore growing interest in new methods that capture the scientific value and originality of research in a more differentiated way.
The Jülich researchers’ AI system therefore relies on analyzing the content. “To evaluate novelty at the time of publication, metadata is not sufficient. Therefore, our system examines the content of a study and relates it to the state of knowledge at the time of its publication,” explains project leader Jan Göpfert from ICE-2.
In this way, the AI collects arguments that speak for or against the novelty of the research work. The novelty value is then rated at the end on a scale from 0 to 100. In addition, the system gives an assessment of how confident it is in its classification and justifies its assessment in writing.
“The real challenge was to define novelty in a meaningful way,” says Kieling. “For us, novelty does not simply mean different. What is crucial is the contribution a work makes to scientific progress.”
With their development, the researchers want to increase the visibility of relevant contributions in the future. The AI system could be embedded in the review or publication process.
Also interesting: