Beyond Context Length:
Characterising the Difficulty of Long-Context Reasoning

Author

Shiwen Qin1, Cameron Barker2, Henry Gouk2, Elliot J. Crowley1, Shay B. Cohen2, Mirella Lapata2, Miao Li2

Affiliations

1 School of Engineering, University of Edinburgh

2 School of Informatics, University of Edinburgh

Posted

October 11, 2026

Venue / status

Under review

arXiv: Coming soon

Overview

Two questions can have equally long contexts yet demand very different reasoning. LURCH is a benchmark of 6,550 questions from 17 sources, with supporting evidence and reasoning graphs that let us measure three properties beyond context length: evidence scope, the proportion of the context needed to answer a question; evidence dispersion, how widely that evidence is scattered; and reasoning depth, the number of sequentially dependent steps needed to reach an answer.

Across four language models, greater scope, dispersion and depth are associated with lower scores even after accounting for context length. Performance also degrades more sharply as contexts grow when the evidence is dispersed. Questions that are moderately demanding on all three properties can be harder than those with an extreme value on just one.

Equal length, different difficulty

Three questions with approximately 92,000-token contexts compare local evidence, dispersed evidence, and broad integrated evidence. Evidence spans and reasoning graphs illustrate differences in scope, dispersion, and depth; bars show the four models' accuracies for each example. Three questions with approximately 92,000-token contexts compare local evidence, dispersed evidence, and broad integrated evidence. Evidence spans and reasoning graphs illustrate differences in scope, dispersion, and depth; bars show the four models' accuracies for each example.
Examples with nearly identical context lengths place different demands on evidence gathering and reasoning. Model accuracies are averaged over three responses per question.