Skip to content
METR's Time Horizon Rated GPT-4 at 4 Minutes and Claude Mythos Preview at 17.4 Hours of Human Work, and the Nonprofit Itself Says Readings Above 16 Hours Are Unreliable
AI & Technology

METR's Time Horizon Rated GPT-4 at 4 Minutes and Claude Mythos Preview at 17.4 Hours of Human Work, and the Nonprofit Itself Says Readings Above 16 Hours Are Unreliable

Illustration by tuput

English

METR's 50% time horizon is the length of task, set by how long skilled people take, that an AI agent finishes half the time. METR's data file lists GPT-4 at about 4 minutes and Claude Mythos Preview (early) at about 17.4 hours, and METR says readings above 16 hours are unreliable on its current tasks.

· · 9 min read

METR, a Berkeley nonprofit that tests AI agents, rates a model by the length of task it finishes half the time, with length set by how long skilled people take. METR’s open data file lists GPT-4 (March 2023) at about 4 minutes and Claude Mythos Preview (early), listed with an April 2026 release date, at about 17.4 hours. METR’s own page says “Measurements above 16 hrs are unreliable with our current task suite.”

What a 50% time horizon counts

The measure comes from a paper titled “Measuring AI Ability to Complete Long Software Tasks”, posted to arXiv on 18 March 2025 by 26 authors led by Thomas Kwa. It was a NeurIPS 2025 poster and was last revised on 10 July 2026. METR defines the 50% time horizon as the task duration, measured by how long a human expert takes, at which an agent is predicted to succeed half the time.

The stopwatch runs on the human side. A task that skilled people finish in two hours sits at two hours on the scale, whether a model needs ten minutes or never finishes it. METR’s FAQ says the horizon describes how hard a task is and says nothing of the time an AI spends on it. The same FAQ adds that agents are typically several times faster than humans on the tasks they complete.

A 50% horizon is a fitted curve, and it does not mean the model passes half of every task of that length. The FAQ uses GPT-5 as the example. On tasks that took human experts 90 minutes to 3 hours, a GPT-5 agent succeeded every time on about a third of the tasks, failed every time on about a third, and went either way on the rest.

In its March 2025 post METR reported that the models it tested then succeeded almost every time on tasks that took people under 4 minutes, and less than 10% of the time on tasks that took more than around 4 hours.

How the yardstick is built

The original suite had 170 tasks from three sources. Ninety-seven came from HCAST, a benchmark of 189 machine learning, cybersecurity, software engineering and reasoning tasks. Seven were RE-Bench environments, each eight hours long. RE-Bench’s authors collected 71 eight-hour attempts by 61 human experts and found that the best AI agents scored 4 times higher than the experts on a two-hour budget, while humans narrowly led at eight hours. The last 66 were short tasks METR wrote for the paper, lasting 1 to 30 seconds each.

The paper’s examples show the range. The shortest is a multiple-choice question, “Which file is a shell script?”, at about 3 seconds. A 56-minute task asks for a Python script that converts JSON from one format to another by inferring the rules from sample files. An eight-hour task asks an agent to speed up a trade-backtesting tool with custom CUDA code, aiming for a 30-times gain.

Every task is scored automatically. For lengths, METR hired people: over 800 baselines totalling 2,529 hours, from skilled professionals in software engineering, machine learning or cybersecurity who averaged about 5 years of relevant experience. A task’s length is the geometric mean of the successful human times. For each model, METR fits a curve of success probability against task length and reads off where it crosses 50%.

On 29 January 2026 METR released Time Horizon 1.1. The suite grew from 170 to 228 tasks, with 73 added, all from HCAST, and 15 removed. Tasks estimated at 8 hours or more went from 14 to 31. METR also moved its evaluation code from an in-house tool called Vivaria to Inspect, an open-source framework developed by the UK AI Security Institute (its documentation now names Meridian Labs as a co-developer) that ships with more than 200 ready-made evaluations. The institute’s own agent testing is the subject of its July 2026 incident report.

METR compared the two systems on five models. The estimates differed slightly, and GPT-4o and o3 scored significantly higher under Vivaria, which METR read as sensitivity to the prompts and tools wrapped around the model.

What the data file lists

METR’s time-horizons page links its raw data. The file as read on 12 November 2026 lists these 50% horizons, with release dates as the file gives them. The last column is METR’s own interval from its statistical resampling.

ModelListed release date50% horizonMETR’s interval
GPT-414 March 20234 minutes1.9 to 8 minutes
o15 December 202439 minutes21 to 65 minutes
Claude 3.7 Sonnet24 February 202560 minutes33 to 104 minutes
o316 April 2025120 minutes75 to 191 minutes
GPT-57 August 20253.4 hours1.9 to 6.8 hours
Claude Opus 4.524 November 20254.9 hours2.7 to 10.4 hours
Claude Opus 4.65 February 202612.0 hours5.3 to 60.6 hours
Claude Mythos Preview (early)7 April 202617.4 hours8.5 to 55.1 hours

The intervals widen at the top. Claude Opus 4.6’s 12.0 hours comes with a range of 5.3 to 60.6 hours. The paper’s first estimate for GPT-2 was about 2 seconds, and METR’s file now lists about 3.

The same model’s number has also moved as METR revised its suite. Claude Opus 4.5 was listed at 289 minutes under the original method, 320 minutes in the January 2026 release and 293 minutes in the file now. The page’s update log records that METR corrected a regularization mistake on 3 March 2026. GPT-5 followed a similar path, from 138 minutes to 214 and then 203. The FAQ on the same page still quotes GPT-5 at “around 2 hours and 17 minutes”, which is close to the earliest figure.

How fast the line has climbed

The 2025 paper fitted a straight line, on a logarithmic scale, through horizons from 2019 to early 2025. It found one doubling about every seven months, giving 207 days with a 95% interval of 166 to 240 days, and said the trend may have sped up since 2024. The paper also ran a naive extrapolation to a one-month horizon of 167 working hours, which landed between mid-2028 and mid-2031, and its authors say real forecasts would carry larger error.

With the larger suite in January 2026, METR reported a doubling time of 131 days for models since 2023, against 165 days under the original suite. Across the whole period, joining new estimates to the old ones for earlier models, the figure stayed at 196 days. For models since 2024 it fell from 109 days to 89. METR says the new tasks probably come from a slightly different difficulty mix and that the trend is somewhat sensitive to which tasks are in the suite.

The data file now gives 129 days since 2023 (interval 104 to 158 days) and 188 days overall, and it notes that it leaves out estimates above 16 hours. Kwa wrote in a January 2026 METR note that the slope of the long-run trend is the number to rely on and the exact horizon of any one model is not. The note carries METR’s standard disclaimer that notes are unpolished and do not necessarily reflect the organisation’s views.

In July 2025 Kwa and Vincent Cheng applied the method to nine existing benchmarks. Software and reasoning tests such as GPQA, math contests and LiveCodeBench had horizons of 50 to 200 minutes or more. Visual computer use (OSWorld and WebArena) had horizons 40 to 100 times shorter but rising at a similar rate. Self-driving, measured with Tesla FSD data, improved more slowly, at about 0.6 doublings a year. METR calls any single benchmark’s result fairly noisy.

What METR says the number does not mean

Kwa’s note has a heading aimed at the most common misreading: “Time horizon is not the length of time AIs can work independently”. It describes the amount of serial human labour a model can replace at a 50% success rate.

METR’s FAQ answers “No” to whether an 8-hour horizon means AI can automate all jobs, and gives three reasons. The humans who set the lengths are closer to a new hire with little context than to a veteran of the project. The tasks are mostly software, machine learning and cybersecurity. Most jobs also involve other people and goals that no program can score. The FAQ adds that its human times likely overestimate how long an expert would take, because the baseliners had less context than professionals doing the same work.

Kwa also writes that a 50% horizon of X hours does not mean tasks under X hours can be handed over, because some reliability-critical work needs success rates of 98% or more. He adds that a doubled horizon does not double the degree of automation. Of the 31 tasks estimated at 8 hours or more in the 2026 suite, only 5 have human baseline times, and METR used estimates for the rest. Kwa writes that he cannot say whether Claude Opus 4.5’s true horizon is 3.5 or 6.5 hours.

Cheating changes the reading too. In its 26 June 2026 summary of a pre-release test of GPT-5.6 Sol, METR got about 11.3 hours when cheating attempts counted as failures, over 270 hours when they counted as successes and 71 hours when discarded, and said it did not treat any of the three as a reliable measurement. Other METR readings of recent agents sit alongside independent coding tests, and the wider problem of tests wearing out is covered in our piece on benchmarks.

An outside check from Berkeley statisticians

On 8 October 2026 Drew Nguyen and William Fithian, of the Department of Statistics at UC Berkeley, posted a preprint that recomputes the horizons on METR’s 228 tasks and 26 AI systems. METR fits a straight relationship between the logarithm of human time and AI difficulty. They fitted splines and item-response models, which do not assume it.

Their fitted curve is nearly flat between 2 and 30 minutes of human time and close to linear elsewhere. A jump in horizon from 3 minutes to 30 minutes is therefore easier than a jump from 30 minutes to 5 hours, though both are tenfold. They describe their work as a constructive critique, offer point estimates that score better on cross-validated tests, and suggest that horizons be read with diagnostic plots, especially as longer tasks are added.

As of 12 November 2026, METR’s time-horizons page says it is no longer actively updated, a note added on 8 September 2026. The page says that while it was maintained some models were measured long after release and others were skipped for lack of capacity.

Share
Copied!

Sources & further reading

  1. METR: Measuring AI Ability to Complete Long Software Tasks (19 March 2025)
  2. Kwa and 25 other authors, Measuring AI Ability to Complete Long Software Tasks (arXiv:2503.14499, v4 of 10 July 2026)
  3. Kwa and 25 other authors, Measuring AI Ability to Complete Long Software Tasks (full text, arXiv HTML v4)
  4. NeurIPS 2025 poster page for the time horizon paper
  5. METR: Time Horizon 1.1 (29 January 2026)
  6. METR: Task-Completion Time Horizons of Frontier AI Models (page last updated 8 September 2026)
  7. METR: time horizon data file, benchmark_results_1_1.yaml (read 12 November 2026)
  8. METR note by Thomas Kwa: Clarifying limitations of time horizon (22 January 2026)
  9. METR: How Does Time Horizon Vary Across Domains? (14 July 2025)
  10. METR: Summary of METR's predeployment evaluation of GPT-5.6 Sol (26 June 2026)
  11. Wijk and others, RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (arXiv:2411.15114)
  12. Rein and others, HCAST: Human-Calibrated Autonomy Software Tasks (arXiv:2503.17354)
  13. UK AI Security Institute and Meridian Labs: Inspect, an open-source framework for AI evaluations (documentation)
  14. Nguyen and Fithian, On the estimation and validity of AI time horizons: a statistical look at the METR plot (arXiv:2610.12466, 8 October 2026)
  15. METR: eval-analysis-public, analysis code and data for the time horizon method (GitHub)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#metr#time horizon#ai agents#ai evaluation#ai benchmarks#inspect#hcast#re-bench

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Scale's SWE-Bench Pro Board Puts the Best Coding Agent at 61.5%, Maintainers Reject Many Fixes That Pass the Tests, and METR Says Its Latest Trial Cannot Show How Much Time Agents Save
Benchmarks say agents are strong. Maintainers, a randomised trial and a pile of incident reports say the picture is narrower. Here is what each number actually measures.
DeepSeek's V4.1-Flash, an Open-Weight 552-Billion-Parameter Model Released on 10 September 2026, Cuts Its Per-Token Memory Cache to 890 Bytes, About a Quarter of Its Predecessor's
A model's short-term memory of a long conversation is expensive to keep. DeepSeek's September release compresses it by design, and its own report says where the model still trails. Here is the mechanism, the timeline from 2024, and what outsiders have checked.
An Estimated 6.49% of MMLU Questions Contain Errors, a Fresh Maths Test Cut Some Models' Scores by up to 8 Points, and the Llama 4 Model Meta Entered on LMArena Was Not the One It Released
Leaderboards decide which new model looks best, and their answer keys, test questions and entry rules have all been audited. Here is what the audits found, who ran them, and what the benchmark builders concede.
← all articles