Does AI Actually Make Developers Faster? The Evidence Splits
Four serious studies measured it and they disagree. The disagreement is the finding: what AI does to your speed depends almost entirely on the codebase.
There is no honest single number for how much faster artificial intelligence makes a software developer. There are four credible studies, they disagree, and the disagreement is more useful than any of the individual results.
The study that found a slowdown
In 2025, METR ran a randomised controlled trial with 16 experienced open-source developers across 246 tasks. These were not toy problems: the repositories averaged over 22,000 GitHub stars and more than a million lines of code, and the developers averaged five years of prior experience on the specific repositories they worked in.
Developers took 19% longer when allowed to use AI tools. They had predicted beforehand that AI would make them 24% faster, and even after finishing the tasks they still believed it had made them 20% faster. Economics and machine-learning experts asked to forecast the result were out by an even wider margin.
That result travelled widely. What travelled far less is what METR published in February 2026: a follow-up that did not cleanly replicate. Among the original developers the estimate was a 18% slowdown, and among newly recruited developers 4%, but both confidence intervals crossed zero. METR is redesigning the experiment, partly because developers increasingly refused to take part in a condition that denied them the tools.
Citing the 19% figure without that update is the single most common error in current writing on this subject.
The study that found a large speed-up
The best-known contrary result is a 2023 trial in which 95 programmers recruited on Upwork built an HTTP server in JavaScript. The group with GitHub Copilot finished 55.8% faster.
Three of that paper’s four authors worked for Microsoft or GitHub, the task was a single self-contained exercise of the kind AI handles best, and the confidence interval ran from 21% to 89%. The authors state plainly that the study "does not examine the effects of AI on code quality".
It is a real result about a narrow thing. It is not evidence about maintaining production software, and it is quoted as though it were.
The largest study, and the pattern that holds
The most substantial evidence comes from three field experiments run at Microsoft, Accenture and an anonymous Fortune 100 electronics manufacturer, pooling 4,867 developers. Access to Copilot produced 26% more completed tasks. The authors note that each individual experiment was noisy and only the pooled result reached significance.
Across every one of these studies, one finding repeats: less experienced developers gain most, and the most experienced gain least or lose ground. That is the best-supported result in the entire literature, and it has an uncomfortable implication for anyone arguing these tools reduce the value of learning the fundamentals.
What explains the contradiction
The gap between METR and the others is almost certainly the setting. Greenfield work with clear specifications is where generation is strongest. Mature codebases with established conventions, where most of the work is understanding what already exists, are where it is weakest.
- If you are building something new and self-contained, expect real gains
- If you are working inside a large system you know well, expect less, and possibly a loss
- If you are new to the codebase, the evidence is thinnest and the risk of confidently wrong output is highest
This is the same shape as the finding in the jagged frontier of AI capability. And it is why what a junior engineer needs to know has shifted toward verification rather than production, a change we set out in our piece on what junior now means.
Frequently asked
Does AI make programmers more productive?
It depends heavily on the work. Randomised trials on self-contained greenfield tasks found large gains, while a trial on mature, million-line codebases found experienced developers were 19% slower. The most reliable pattern across all the evidence is that less experienced developers gain most and the most experienced gain least.
Is the METR study that found AI slowed developers down reliable?
It was a properly randomised trial, but it covered 16 developers working in repositories they already knew well, and METR’s own February 2026 follow-up produced statistically inconclusive results and prompted a redesign. It is real evidence about a narrow setting, not a general finding.
Why do AI productivity studies disagree so much?
Because they measure different work. The studies finding large gains used standardised or greenfield tasks; the study finding a slowdown used mature codebases with high quality standards. Almost none of them measured code quality at all, which is where the disagreement probably matters most.