Sixteen open-source developers were split into two groups. Both worked through roughly two hundred and fifty issues. One group had AI tooling; the other did not.
The developers using AI estimated they had been about a fifth more productive. Measurement showed them roughly a fifth slower (4:35).
Anupam Mishra and Raja put this on screen early at re:Invent, alongside separate industry research placing realised velocity gains from AI-assisted development at ten to fifteen per cent (3:34). Neither figure is presented as a case against the tools. They are presented as the problem the rest of the session addresses: the gains are real and much smaller than the experience of using the tools suggests.
Their explanation, drawn from a year of engagements with more than a hundred companies (2:09), is that the two ways almost everyone works fail for opposite reasons.
The first failure: handing over the whole problem
The approach they call AI-managed is the one the marketing implies. Take a complex or ambiguous problem, give it to an agent, expect finished software.
It works for prototypes. It fails wherever hundreds of design decisions have to be made, because the starting point is ambiguous and the model resolves that ambiguity by assumption rather than by asking. The result is not obviously broken — that would be easier.
The specific failure they identify is one of accountability rather than correctness (6:22). A large volume of code arrives that the developer must put their name to, and their confidence in it is low because they did not make the decisions inside it. That code does not move to production at the speed it was generated. It sits in review, or it goes to production carrying risk nobody has priced.
This is the mechanism connecting fast generation to unchanged delivery. The bottleneck was never typing.
The second failure: keeping the hard part
Senior engineers who have tried the first approach usually arrive at its opposite. They do the decomposition and planning themselves, then insert AI into narrow, well-defined slots: implement this function, review this code for a specific class of problem.
This works, in the sense that the output is good. It also leaves the intellectual load exactly where it was before (7:14). If the expensive part of building software is deciding what to build and how to structure it, and a human continues to do all of that, then automating the writing recovers a modest fraction of the total.
Their second observation about this pattern is the more damaging one. The surrounding process is untouched (7:36). The meetings, the documents passed between roles, the ceremonies designed to coordinate people who cannot see each other's work — all of it persists, and it was calibrated to a pace that no longer applies. Time saved in the editor is absorbed by a process that still assumes the editor was the constraint.
Why this reads differently from the usual scepticism
The measured-versus-perceived gap deserves more attention than it usually receives, and the session treats it as an open question rather than as evidence of self-deception (4:53).
There are respectable explanations on both sides. The tools genuinely reduce the unpleasantness of certain work, and reduced friction feels like speed. The study is also from early 2025, and models have moved since — which the presenters note themselves rather than leaving to a critic.
But the finding is uncomfortable regardless of which explanation dominates, because it undermines the primary measurement instrument most organisations are using. Developer surveys are how adoption gets justified. If self-reported productivity can be wrong by forty percentage points in the direction people want it to be wrong, then the case for the investment is resting on the least reliable available evidence.
The shape of an answer
What they propose sits between the two failures, and the reasoning is more useful than the branding.
If handing over everything fails on ambiguity, and retaining everything fails on the human bottleneck, then the target is the specific step where ambiguity is resolved — and that step has to be shared. The lifecycle they describe puts AI inside the process rather than at the coding stage, which is also an argument that the process itself has to change rather than absorb a new tool.
Which is where their diagnosis gets uncomfortable for anyone planning an adoption programme. If the surrounding process consumes the gains, then buying tooling and leaving the organisation intact produces roughly the ten to fifteen per cent the research reports — and does so no matter how good the tools become. The constraint is not in the editor, and it never was.
关键数据
- 10-15%
- industry research on realised velocity gain from AI-assisted development 3:34
- 16 developers, ~250 issues
- design of the controlled experiment cited 3:53
- +20% perceived / −20% measured
- the gap between self-assessment and measurement in that experiment 4:35
- 100+ companies
- engagements underpinning their observations over one year 2:09
演讲章节
关键要点
- 01
Their observations come from a year of engagements spanning more than a hundred companies, from early-stage startups to Fortune 100 firms. 2:09
- 02
They cite industry research putting the realised velocity gain from building software with AI at ten to fifteen per cent, well below the improvement commonly assumed. 3:34
- 03
A controlled experiment split sixteen open-source developers into two groups across roughly two hundred and fifty issues, with only one group permitted AI tooling. 3:53
- 04
Developers in that experiment estimated they had been about a fifth more productive with AI; measurement showed the AI-assisted group roughly a fifth slower. 4:35
- 05
The gap between perceived and measured productivity is treated as the central open question, not as evidence that the tools do not work. 4:53
- 06
The first anti-pattern is handing an ambiguous problem to an agent and awaiting an end-to-end result, which works for prototypes and fails wherever hundreds of design decisions must be made. 5:18
- 07
Its specific failure is one of accountability: a large volume of generated code arrives that the developer must sign for but cannot confidently review, so delivery slows rather than accelerates. 6:22
- 08
The opposite anti-pattern has a senior engineer decompose and plan the work personally, inserting AI into narrow slots — which leaves the intellectual load exactly where it was before. 7:14
- 09
That approach also leaves the surrounding process untouched, so time saved in editing is absorbed by the meetings and document handoffs the pre-AI process still demands. 7:36
提及的实体
相关演讲
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Brooker builds the definition from the bottom up rather than asserting it, using a deliberately absurd arithmetic task to separate three categories: what a model computes reliably as a fixed function of its input, what merely needs to arrive in the system prompt, and what genuinely requires reaching into the world. Only the third category justifies a tool, and the distinction matters because most production disappointment comes from tools built for the first two. His working definition follows — a system given a goal that loops between inference and tool calls until it reaches one — with the observation that modern agents increasingly embed code in their definitions, not for expressiveness but because replacing inference steps with deterministic code improves reliability while lowering both latency and cost. The remainder covers what production actually demands around that loop: somewhere to run, memory that persists preferences, a gateway to internal and external tools, evaluation, and formal methods applied to policy.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The line that explains this session comes from the customer in the last ten minutes: they are preparing for a world where metadata is how agent-based systems find the data they need and access it through the controls being built. That relocates a function — governance has spent two decades as compliance activity describing data that people locate by other means, and if agents navigate by the catalogue then the catalogue stops describing the access path and becomes it. An incomplete catalogue is a documentation problem when humans can ask a colleague; an agent has no such workaround. The most honest moment addresses the perennial failure that rules get written and ignored, with enforcement rather than publication as the argument. Generated descriptions and greyed-out classification suggestions divide the labour correctly, keeping a person accountable while removing the burden of finding candidates.
The most useful sentence here concerns where reliability is achievable: for workflows calling well-structured interfaces and producing checkable code, the path is faster. The implication is that there is a slower path, and it is the one most enterprise pilots are on. The division separates work whose output can be checked mechanically from work where correctness is a judgement, and only the first admits an engineering approach to improvement. The best illustration is browser automation that examines what the customer would observe rather than the underlying markup, walking checkout and booking flows and identifying revenue leakage — a framing that defines success in the business's own terms and is verifiable in the good sense. The demonstration's fifteen to twenty tool calls are the honest measure of what agentic means: not one clever response but a chain of decisions each depending on the last.