A team led by Zongxia Li released Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0) on arXiv on Oct. 9. The benchmark gives AI agents up to three hours on command-line tasks and scores how much progress they make, not just whether they pass at the end.
The 17 authors are from Tencent’s Hunyuan team and universities including the University of Maryland, Carnegie Mellon and the National University of Singapore, according to the paper.
The first version had 46 tasks in containers, which are self-contained software environments. The authors say repeated runs exposed defects in some tasks and in how grading connected to them. Version 2.0 keeps the 46 tasks, fixes those problems and extends 31 of them, for 77 tasks across 13 domains and eight task types.
How the new tasks work
The new tasks start from a working scientific or engineering program. The team uses a reference version to generate several test scenarios, shows the agent six of them and holds three back for grading. That keeps agents from memorizing public outputs, the authors write.
Scoring uses “progress ladders,” a series of graded steps that give credit for partial work. The time limit grew from 90 minutes to three hours, with a grading checkpoint every 30 minutes. The authors also isolated the grader from the agent and reran results that had been affected by grader access.
How models did
The team scored 1,145 runs from 15 frontier models using the Harbor framework’s Terminus-2 agent, with every model at its highest reasoning setting. GPT-6-astra led with a mean score of 0.69 and 37 tasks at or above 0.95. GPT-5.6-sol followed at 0.67 with 32 tasks.
On the 30 hardest reconstruction tasks, pass-fail checks averaged 0.221 while ladder scores averaged 0.653. The authors say that gap shows outcome-only tests throw away a lot of evidence about how far an agent got. Scientific computing and imaging tasks scored lowest, and 24 tasks went unsolved by every model.
The longer limit mattered for some models. The median successful run finished after 57 minutes, and 42% of solves came after the first hour, the paper says.
GPT-6-astra solved 34 of its 36 tasks within an hour. Hunyuan hy4 and DeepSeek V4 Pro solved many tasks late, and a one-hour test would have ranked them ninth and 13th instead of fourth and eighth.
The authors also tracked when agents claimed to be done. GPT-6-astra and GPT-5.6-sol made their first completion claim after a median of 7 and 8 minutes. Across all models, 92% of completion claims came on runs that never passed verification.
Most of the spending went to failures. Between 83% and 99% of each model’s spending went to tasks it never solved, because every step resends a longer history.
GPT-6-astra spent an average of $664 on an unsolved task and $101 on a solved one. The authors note their estimates apply no prompt-cache discount, so those figures are an upper bound.
Source: arXiv, Zongxia Li et al., “Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress”
Additional sources: arXiv (full paper), Zongxia Li et al., “Long-Horizon-Terminal-Bench 2.0 (HTML full text)”
