Replit released ViBench 2.0 on Oct. 28, a benchmark that asks AI coding agents to build an entire software product from an empty project. None of the open-weight models it tested got a single build fully right.
The benchmark was written up by Replit’s Preeya Kirani, Daniel Furman, Peter Zhong and Michele Catasta. It’s part of the Specialized Intelligence Index from Fireworks AI, Replit’s partner for running open-weight models. Fireworks launched that index on Sept. 22 to compare models on practitioner-built tests of real work.
Most coding benchmarks hand an agent a well-defined task inside an existing codebase, the Replit authors wrote. ViBench 2.0 starts with nothing. The agent writes a first version of a product from a short spec, then adds up to 18 features on top of its own code.
That order is the point. The agent has to pick its database layout and routes before it knows what’s coming, so later features can be harder than earlier ones. Replit’s first version of the benchmark found models score lower when adding to their own code, a setting it called “Vibe-on-Vibe.”
How it’s graded
ViBench 2.0 has 17 products, 241 feature requests and 544 test plans, modeled on apps such as Uber, GitHub and Google Docs. Eight are public on GitHub and nine are held back.
After the last feature, a grading agent runs the test plans in a real browser, clicking and typing like a user.
Replit uses Anthropic’s Opus 5.5 as the grader because it made the fewest mistakes among the models tested. To check it, the team planted 34 bugs in working builds, and the grader never passed a plan while a bug was visible. When OpenAI’s GPT-6.1 Sol re-graded 477 of Opus’s passes, it confirmed 99.2%.
The tests target details a user would hit.
In the ride-hailing product, split fares must add up to the exact cost of the ride, to the cent. In the design tool, undo must take back only your own change, never a teammate’s. The three top models failed that undo plan in eight of their 12 builds.
The results
The official scores come from Fireworks’ runs, with four builds of each product per model. GPT-6.1 Sol leads with a pass@1 of 65%, meaning every test plan passed in 65% of its builds. GPT-6 Astra scored 60% and Opus 5.5 scored 49%.
All three passed more than 90% of their individual test plans, so most failed builds missed by only a few. Cost differed sharply. Sol spent about $12 per build and Astra about $90, by Replit’s figures at list prices.
Open-weight models, whose trained weights are public, did well on individual plans but never cleared a full build. DeepSeek V4.1 Flash passed 77% of its test plans for about $2.50 a build.
Speed was another gap. Opus 5.5 finished a build in about 70 minutes on average, about 2.5 times faster than Sol, which Replit attributes mostly to API provider speed.
Replit also ran older OpenAI models on the eight public products. Pass@1 rose from 0% for GPT-5.4 to 31% for GPT-6 Sol and 53% for GPT-6.1 Sol. OpenAI’s newer GPT-6.1 Terra passed 47% of builds at about half Sol’s cost.
Source: Replit, Preeya Kirani, Daniel Furman, Peter Zhong, Michele Catasta, “ViBench 2.0: A Benchmark for Long-Horizon Software Building”
Additional sources: Fireworks AI, “Specialized Intelligence Index: Real-Work AI Benchmarks”
