The study evaluates language-model navigation on the Wikipedia Game, where a player must reach a target article by following links from one article to another. Background context from the source: This is a constrained traversal problem over an existing document graph, not unrestricted web search or free-form text generation. It tests whether a system can match a destination to the current article, select a useful route, and apply enough linguistic or specialist knowledge to continue through the graph.
The researchers extend the task beyond Wikipedia by evaluating academic-literature graphs built from paper citations. The benchmark compares sentence-embedding models and large language models across five Wikipedia languages and five academic-literature datasets. That combination exposes systems to different graph structures and knowledge distributions: Wikipedia navigation follows hyperlinks between articles, while the academic condition follows citation relationships between papers.
The evaluation uses large local graphs of Wikipedia pages and paper citations, with the citation data derived from OpenAlex. Running the graphs locally removes dependence on third-party APIs during navigation experiments and fixes the underlying graph used for comparison. The researchers also provide code for constructing and navigating the evaluated graphs, supporting reproduction of the benchmark setup rather than limiting the results to an inaccessible hosted service.
The comparison includes human and TF-IDF benchmarks alongside the sentence-embedding and LLM systems. Embedding-based systems provide a semantic-similarity baseline for choosing among linked pages, while LLMs are evaluated as more general navigation systems. The distinction matters because the task requires more than identifying related text: a route must connect the current node to a specific target through the links available at each step.
The strongest named systems—GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B—achieved success rates comparable to human performance. The statistical boundary is important: no evaluated model significantly outperformed humans. The result therefore supports human-level performance under the benchmark’s conditions, not a demonstrated model advantage.
Performance was also uneven. Model performance and rankings varied across languages and topics, so a system that performed well in one condition did not establish a consistent lead across the benchmark. The variation is central to the study’s interpretation: multilingual and cross-domain navigation cannot be represented by a single aggregate ranking without obscuring where systems succeed or fail.
The benchmark’s graph constraints also limit what can be inferred about deployment. A model that reaches targets in the supplied Wikipedia or citation graphs has demonstrated navigation within those fixed structures, using the links and documents available there. That does not establish reliable autonomous retrieval outside the tested languages, domains, graph construction, or route conditions.
In particular, comparable performance to humans does not show broad superiority over people or guarantee that a system will select useful routes in an unfamiliar document collection.
Original source: http://arxiv.org/abs/2610.13948v1