The most honest slide in this session is near the end. Most of the security processes they have automated still route their output to security engineers rather than to the business, because the results are not good enough (51:35).
That sits about thirty minutes after a story about two engineers building something in two days that runs a training exercise in seven minutes for ninety-one cents (17:16, 17:45). Both are true, and the gap between them is what the session is actually about.
What separates the two outcomes
The distinguishing factor is not model capability. It is whether the task has a verification loop that a machine can close.
The prototype worked because the exercise had a known answer — a training scenario, run to completion, checked against expectations. The processes still routed to human engineers are the ones where correctness is a judgement call, and no amount of throughput helps if a person has to evaluate every result anyway.
That reframes the automation question. The useful thing to ask about a workflow is not whether a model can do it. It is whether anything other than a human can tell that it did it correctly — and the answer to the second question determines whether the first one matters.
The verification techniques they actually use
Three specific loops appear, and they are the transferable part of the talk.
The first is citation. Rather than trusting an answer, require the model to produce sources, then click through and check them (32:28). The reduction in effort is real and partial: you still verify, but you verify against a pointer rather than searching for one.
The second is having the model check its own output, which costs more in tokens and saves time (33:22) — described plainly as a trade the speaker is happy to make. The framing is right. This is not a claim that self-checking makes output correct; it is an observation that tokens are cheaper than attention.
The third is the strongest, because it is not probabilistic at all. A loop that compiles the generated code guarantees the result is syntactically valid and contains no invented library calls (35:38). The model proposes, the compiler disposes, and no judgement is involved.
Ranking those by how much trust they actually transfer: the compiler loop is verification, self-checking is triage, and citations are a way of making human verification cheap. Only one of them removes a person from the loop.
The rejection of a single number
The most quotable engineering position is about evaluation: a claim that a model scored eighty-three per cent is rejected outright, because one number does not capture result quality — so they work in precision and recall (46:59).
This is worth dwelling on because it is not pedantry. In security work the two error types have wildly different costs. A false negative is a missed intrusion. A false positive is an engineer's afternoon. Any single accuracy figure has averaged those together and thrown away the only information that would let you decide whether the system is usable.
The consequence follows directly from the earlier point. A system with high recall and mediocre precision is exactly the system whose output has to go to a security engineer, which is why the two observations belong together.
Small agents, arranged like an organisation
Their architecture is a set of narrow agents — resource discovery, enumeration, injection testing and others — coordinated by a workflow (45:06). Alongside them sits a set of document-review agents modelled on the company's own meeting culture: a programme manager, a finance reviewer, a product manager, each engaged the way you would engage the corresponding colleague (45:34).
The pattern is worth naming. Each agent is small, which means each can be evaluated against a narrow question, which is the only way the precision-and-recall discipline scales. A single general agent doing all of it would produce results nobody could score.
The advice that actually generalises
The most useful thing said is about adoption rather than technology: the mistake was believing he had to do everything, when the right approach was to pick one tool, apply it to one problem, and iterate (22:24).
Set that against the accessibility argument — there is no barrier to entry, nothing to read, and you can begin in a browser in whatever language you speak (21:02) — and the two form a coherent position. Very low entry cost combined with a narrow first target is a rational strategy under uncertainty about which applications will work.
The scepticism is applied evenly, which is rarer. Negative coverage generates clicks and therefore gets published, which introduces selection bias into the discourse (5:54) — an observation made by someone whose employer has an enormous stake in the technology, about coverage of the technology, and it holds regardless of who is saying it.
关键数据
演讲章节
关键要点
- 01
Most automated security processes still route to human engineers because the results are not good enough to expose to the business. 51:35
- 02
A two-engineer, two-day prototype completed a training exercise in seven minutes for ninety-one cents, eighteen months before the talk. 17:45
- 03
A compile step is the only verification loop discussed that fully removes a person, guaranteeing valid syntax and no invented library calls. 35:38
- 04
He rejects any single accuracy figure outright and insists on precision and recall, because security's two error types have very different costs. 46:59
- 05
His adoption advice is to pick one tool and one problem and iterate, rather than attempting to cover everything. 22:24
提及的实体
相关演讲
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Drawn from a year of engagements with more than a hundred companies, this is the most direct challenge in the season's programme to the assumption that faster code generation produces faster delivery. Mishra and Raja open with external evidence rather than their own: an industry study putting realised velocity gains in the ten to fifteen per cent range, and a controlled experiment in which developers using AI estimated themselves roughly a fifth more productive while measurement showed them a fifth slower. Their diagnosis is that both prevailing working styles fail for opposite reasons. Handing an ambiguous problem to an agent and awaiting a finished result produces a volume of code the developer must nonetheless sign for and cannot confidently review, so it stalls before production. The senior engineer's alternative — decomposing the work personally and inserting AI into narrow slots — keeps the intellectual load exactly where it was, and leaves the surrounding process untouched, so hours saved in editing are consumed by the meetings that process still requires.
Brooker builds the definition from the bottom up rather than asserting it, using a deliberately absurd arithmetic task to separate three categories: what a model computes reliably as a fixed function of its input, what merely needs to arrive in the system prompt, and what genuinely requires reaching into the world. Only the third category justifies a tool, and the distinction matters because most production disappointment comes from tools built for the first two. His working definition follows — a system given a goal that loops between inference and tool calls until it reaches one — with the observation that modern agents increasingly embed code in their definitions, not for expressiveness but because replacing inference steps with deterministic code improves reliability while lowering both latency and cost. The remainder covers what production actually demands around that loop: somewhere to run, memory that persists preferences, a gateway to internal and external tools, evaluation, and formal methods applied to policy.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The hardest problem in this session has nothing to do with machine learning: you cannot reliably tell who owns a machine. Unlike vehicles, which carry an identification number and go through state registration, heavy equipment has no equivalent — someone can simply assert ownership. Everything the connected-product strategy promises depends on solving that, because every step after fault detection requires knowing who to contact. The estate explains why it was not solved earlier: millions of machines with 1.5 million connected, and around 160 dealers who are independent businesses with their own systems, holding the service history that makes telemetry meaningful. The prior state is described directly — multiple accumulated platforms, and dealers confused because the same question returned different answers, which destroys trust in all of them including the correct ones.