The Missing Connection: Why AI Still Struggles to Find the Research That Could Change Everything

Artificial intelligence can read millions of words, search enormous databases, summarize scientific papers and increasingly help researchers formulate hypotheses. But can it do something that exceptional scientists routinely do—recognize that an idea developed for an apparently different problem may be exactly the idea needed to solve the problem in front of them?

A new research project called ScholarCatalyst suggests that today’s AI systems are still surprisingly poor at this capability. That matters far beyond information retrieval. As artificial intelligence begins accelerating scientific research itself, the ability to discover unexpected connections among hundreds of millions of existing scientific papers could become one of the important determinants of how rapidly science and technology advance.

The research is described in “ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research,” submitted to arXiv on October 1, 2026. The 57-page paper is authored by Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim and Chelsea Finn. The affiliations represented include Stanford University, Seoul National University, the University of Washington, Carnegie Mellon University, the Allen Institute for AI and MIT. The paper is classified under Artificial Intelligence, Computation and Language, and Information Retrieval. (arXiv)

Read the complete ScholarCatalyst paper on arXiv

Finding the Paper You Didn’t Know You Needed

Traditional scientific search is largely built around relevance.

A researcher enters keywords or describes a problem. A retrieval system attempts to find papers containing similar words, concepts or semantic representations. Modern embedding systems are substantially more sophisticated than keyword searches because they can recognize conceptual similarity even when two documents use different terminology.

ScholarCatalyst asks a harder question.

Instead of asking, “Which papers are about this subject?”, it asks something closer to:

“Which previous idea could help me make progress on this problem?”

Those are not equivalent questions.

A paper can be extremely similar to a research question while contributing nothing particularly useful toward solving it. Conversely, an important insight may originate in another specialty, use completely different terminology and have relatively little semantic resemblance to the problem to which it can ultimately be applied.

The researchers call such intellectually useful prior work “catalyst papers”—papers whose ideas did, or plausibly could, materially advance a research project. (arXiv)

Asking the Researchers Who Actually Did the Work

Constructing a benchmark for this problem presents an unusual difficulty: how do you determine which earlier research really mattered?

A bibliography isn’t enough. Papers cite previous work for many reasons. A citation might establish background, acknowledge a competing approach, document a dataset, satisfy scholarly convention or identify a limitation. Meanwhile, an idea that strongly influenced a researcher might never appear in the eventual bibliography.

ScholarCatalyst therefore went directly to researchers who knew how their projects actually developed.

The team collected information from 184 researchers who led 207 recent computer-science projects, principally involving papers published during 2025 and 2026. More than 100 of the source papers had received oral, spotlight or award recognition at major AI conferences. The participating researchers reconstructed their projects as they existed before their principal discoveries were known. (arXiv)

The process produced 894 research questions: 207 broad Core Research Queries and 687 more focused Subfield Queries.

For each query, researchers reviewed candidate earlier papers and classified them as either potentially inspirational or as related but not actually useful in advancing the problem. They also explained their reasoning. (arXiv)

That created something unusual: a retrieval benchmark based not merely on similarity or citations, but on researchers’ firsthand judgments about intellectual usefulness.

Preventing AI From Looking Into the Future

There was another methodological problem.

If an AI system searches today’s literature for the antecedents of an already completed project, information about the eventual solution may have contaminated the search environment—or even the model’s training data.

ScholarCatalyst therefore constructed a temporally restricted corpus. For each research problem, the search system was permitted to retrieve only papers that existed before the source project was completed.

The overall experimental corpus contained 190,896 papers, mostly drawn from computer-science research published from 2020 through 2024, together with resolved references from the source papers. (arXiv)

That is already a substantial search problem, but it is tiny compared with the actual scientific literature. The researchers note that arXiv itself exceeds three million articles, while Semantic Scholar indexes more than 225 million papers. (arXiv)

ScholarCatalyst therefore tests an easier version of the problem that a real scientific research system ultimately must solve.

The Surprising Result: More AI Reasoning Didn’t Solve the Search Problem

The researchers tested conventional lexical search, dense embedding systems, specialized scientific-document retrievers, reasoning-oriented retrieval systems and several forms of AI search agents.

The agentic systems were allowed to do considerably more than issue a single search. Some could reformulate questions, conduct multiple searches, inspect results and reason iteratively about where to search next.

One might expect those agents to dramatically outperform ordinary retrieval.

They didn’t.

Among the pre-cutoff systems, strong embedding retrieval remained extremely competitive. On the subfield queries, Qwen3-Embedding-8B achieved 0.51 Recall@20, meaning that approximately 51 percent of author-identified catalyst papers appeared among its first 20 results. On the broader Core Queries, performance was only 0.37 Recall@20.

The GPT-4.1 tool-calling agent achieved 0.37 on the Core Queries and 0.43 on the Subfield Queries. The o3 deep-research agent achieved 0.33 and 0.38 respectively. (arXiv)

Even a Claude Fable 5.1 agent—treated separately because its later knowledge cutoff meant it might have encountered some completed research—remained around the halfway mark at Recall@20. (arXiv)

The implication is important:

Adding more reasoning after retrieval does not necessarily solve a failure of discovery before reasoning begins.

An AI system cannot recognize the significance of a paper it never encounters.

The Candidate-Coverage Problem

This leads to perhaps ScholarCatalyst’s most important technical finding.

The researchers found evidence that the principal bottleneck is candidate coverage.

An intelligent reranking system may be quite capable of recognizing a valuable paper once the paper has entered its candidate pool. But if the underlying retrieval mechanism never surfaces that paper, the higher-level reasoning system never has an opportunity to evaluate it.

Experiments in which missing correct papers were deliberately injected into candidate pools substantially improved the ability of stronger rerankers to select them.

In simple terms:

The AI may be getting better at recognizing the answer while still being poor at knowing where to look for it. (arXiv)

That distinction has enormous implications for AI-assisted research.

Similarity Is Not Inspiration

ScholarCatalyst also challenges a fundamental assumption underlying much modern information retrieval.

Researchers compared the catalyst papers with deliberately selected “hard negatives”—papers that were clearly related to the research question but that the authors judged not to be inspirational.

The surprising result was that these rejected papers were at least as similar to the research questions as the genuine catalyst papers, using both lexical and embedding-based measures.

In other words:

Semantic similarity and intellectual usefulness are different properties.

The research also found several different kinds of catalytic relationships. Authors frequently identified a method that could be adapted, evidence supporting a direction, an idea that could be generalized into a different setting, or a limitation in earlier work that suggested a new approach. More than one kind of relationship applied in 55.8 percent of the analyzed rationales. (arXiv)

That begins to describe something closer to scientific judgment than ordinary search.

The Most Important Number May Be 49.5 Percent

One result deserves particular attention.

Nearly half—49.5 percent—of the catalyst papers identified by researchers came from outside the subject area of the project.

Furthermore, 43.6 percent of the positive papers identified for the subfield queries weren’t cited in the eventual source paper. (arXiv)

That means conventional bibliographic analysis can miss important intellectual relationships.

More importantly, it illustrates why major advances sometimes arise from connecting ideas across disciplines.

A technique created for language processing might become useful in biology. An optimization technique developed for computer networks might help solve an energy problem. A materials-science discovery could enable an entirely new computing architecture. A control-theory concept might become important in autonomous robotics.

The valuable connection may not share the vocabulary of the problem being solved.

The researchers therefore identify two longer-term milestones. The first would be AI retrieval approaching an expert scientist’s judgment within a field. The second—and potentially much more consequential—would be systems capable of exercising something resembling expert judgment across more fields than any individual human researcher could possibly master. (arXiv)

That second possibility deserves much more attention than merely improving academic search.

From Searching Science to Accelerating Science

ScholarCatalyst itself does not demonstrate that AI is about to autonomously transform every scientific discipline. Its experiment concerns computer-science literature retrieval, and the authors explicitly caution against assuming that their results automatically generalize to other scientific fields. (arXiv)

But it examines one increasingly important component of a much larger development.

Artificial intelligence is beginning to participate in the process by which artificial intelligence itself is researched.

That creates a potential feedback process:

Better AI → better AI-assisted research → faster development of better AI → still better research tools.

This should not automatically be interpreted as an uncontrolled “intelligence explosion.” Research remains constrained by experimental evidence, computing resources, engineering, manufacturing, capital, regulation, human institutions and the physical world.

Nevertheless, even considerably less dramatic acceleration could have historic consequences.

The secondary effect may ultimately be even larger than acceleration of AI research itself.

As AI research tools improve, those capabilities propagate into other scientific and engineering disciplines.

From Medicine to Materials—and From Skateboards to Spaceships

Consider the range of activities involved in research.

Researchers identify unanswered questions. They search previous literature. They formulate hypotheses. They compare competing explanations. They design models. They run simulations. They analyze data. They identify anomalies. They propose experiments. They evaluate results. They discover unexpected relationships and decide what to investigate next.

AI can increasingly assist with portions of this cycle.

In medicine and biology, that can mean analyzing molecular interactions, identifying candidate therapeutic targets, modeling proteins, comparing clinical evidence or helping researchers formulate experiments.

In agriculture, it can mean crop genetics, disease resistance, soil science, precision agriculture, biological pest management and adaptation to changing environmental conditions.

In chemistry and materials science, AI can help search enormous spaces of possible molecules and materials before researchers synthesize the most promising candidates.

Those discoveries propagate outward because materials underpin much of technological civilization.

Better materials affect semiconductors, batteries, motors, buildings, aircraft, automobiles, medical devices, manufacturing equipment and communications systems.

Energy research spans generation, storage, conversion and transmission. It includes nuclear technologies, solar cells, electrochemistry, battery systems, hydrogen, petroleum and its derivatives, catalysts, grid technology and materials capable of operating under extreme temperatures, pressures and radiation.

Transportation ultimately combines many of these disciplines. Advances in materials, energy storage, sensors, computation, communications, control systems and AI can affect virtually everything that moves—from a skateboard or automobile to autonomous trucks, aircraft, drones, ships and spacecraft.

Robotics connects AI to physical action.

Computing research improves the processors, networks and software used to perform still more research.

The relationships are increasingly circular.

Science Has Always Accelerated Science—But AI Changes the Scale

Every generation of scientific instruments has expanded what researchers can discover.

The telescope expanded human vision. The microscope revealed biological structures invisible to the naked eye. Statistical methods allowed scientists to reason systematically about uncertainty. Computers made previously impossible calculations routine. Supercomputers enabled simulation of weather, nuclear processes, molecular systems and entire engineered structures.

Artificial intelligence is different in an important respect.

It is not merely another instrument for observing or calculating.

Increasingly, it can participate in portions of the reasoning, searching, comparison, modeling and hypothesis-development process itself.

ScholarCatalyst shows that today’s systems remain far from mastering one of the most sophisticated parts of that process: recognizing the unexpected previous idea that could unlock a new problem.

But the benchmark also gives researchers a way to measure progress toward precisely that capability.

If systems eventually become good at finding useful connections across hundreds of millions of papers spanning many disciplines, humanity may gain something no individual scientist could possess: an instrument capable of searching a significant portion of accumulated scientific knowledge for connections across disciplinary boundaries.

Important Limitations

ScholarCatalyst should not be oversold.

Its evidence comes from computer-science projects from 2025 and 2026, with uneven representation among specialties. The benchmark relies on retrospective judgments: researchers who already know how their projects succeeded are being asked to reconstruct what earlier work did or could have helped them.

Hindsight can influence those judgments.

There is also no uniquely correct list of papers capable of inspiring a discovery. Different excellent scientists may identify different intellectual paths toward the same result.

The authors tested this issue on a limited basis. In one project with three author annotators, coauthors recovered only 43 to 60 percent of the first author’s selected papers, although the selections they did make had high precision. That suggests genuine human disagreement about what constitutes an inspirational paper. (arXiv)

Consequently, ScholarCatalyst should not be interpreted as measuring whether an AI thinks exactly like an ideal scientist.

It measures something narrower and useful: whether retrieval systems can recover papers that the actual researchers responsible for projects judge capable of materially advancing those projects.

Future work will need larger datasets, additional disciplines, prospective rather than retrospective evaluation, better measures of human expert performance and ultimately experiments testing whether AI-recommended connections actually help researchers produce discoveries.

A New Problem: Managing the Velocity of Discovery

There is a larger societal consequence if AI succeeds at problems like the one ScholarCatalyst measures.

Human institutions have historically had time to absorb technological change. Not always gracefully and not always safely—but technologies generally moved through research, development, commercialization and diffusion over periods long enough for industries, governments and societies to respond.

AI-assisted research could compress portions of that cycle.

And acceleration across multiple interacting disciplines creates a fundamentally different governance problem.

New technology almost always produces both positive and negative consequences.

Automobiles created extraordinary mobility and enormous economic benefits, but also accidents, pollution and infrastructure problems. Pharmaceuticals save lives but can produce adverse effects. Aviation connected the world while introducing new forms of risk. Computing created unprecedented productivity and communications capabilities while also creating cybersecurity, privacy and information-integrity problems.

The appropriate response to technological risk cannot simply be to reject technology.

Nor can responsible societies assume that innovation should proceed without constraints.

The more useful objective is consequence management.

That means identifying foreseeable positive and negative consequences, monitoring for unexpected ones, reducing dangerous consequences, correcting failures rapidly and preserving as much of the beneficial capability as reasonably possible.

Regulation itself therefore faces an optimization problem.

Too little governance can allow preventable harms. Too much or badly designed governance can suppress beneficial innovation, entrench incumbents and prevent society from obtaining technologies that could improve or save lives.

The challenge becomes more difficult as technological development accelerates because governance systems must increasingly adapt while the technologies themselves continue changing.

The Beginning of a Different Scientific Era

It would be premature to claim that AI has already transformed scientific discovery.

ScholarCatalyst actually demonstrates something more interesting: how much capability remains to be developed.

Today’s sophisticated AI systems can retrieve, reason and synthesize enormous amounts of information, yet they still miss roughly half of the earlier papers that researchers themselves identify as potentially important to their discoveries.

Closing that gap could be extraordinarily consequential.

Imagine an AI research system that doesn’t merely answer a scientist’s question, but says:

“Researchers in another discipline solved a structurally similar problem twelve years ago. You haven’t cited their work because their terminology is completely different from yours. Here is the underlying principle, here is why it may apply to your problem, and here are three experiments that could determine whether the connection is real.”

That is not what today’s ScholarCatalyst results show AI can reliably do.

It is what the benchmark helps us measure progress toward.

Human civilization has experienced periods of extraordinary technological change before. But we have never possessed a general-purpose technology capable of participating directly in the intellectual process used to improve that technology—and then carrying increasingly powerful research capabilities into almost every other scientific and engineering discipline.

That does not guarantee an unprecedented acceleration of discovery.

But it creates the possibility.

And if that possibility becomes reality, one of humanity’s central challenges will no longer be simply how rapidly we can invent.

It will be how rapidly we can understand, evaluate and responsibly manage the consequences of what we invent—so that technological progress remains overwhelmingly beneficial to humanity.