Marc Brooker opens with a deliberately ridiculous task: multiply the number of Rs in "strawberry" by the current hour and the outdoor temperature in Seattle, then take the log gamma of the result.
It is a joke that does real work. Because unpacking which parts of it a model can handle turns out to be the cleanest available definition of what an agent is for.
Three categories, only one of which needs a tool
Counting letters in a word is a fixed function of the input. A modern model does it reliably and needs nothing from the world.
The current hour is different — the model cannot know it, but it does not need a tool either, because the value can simply be placed in the system prompt. Basic arithmetic sits in the same category: adequate, as long as the numbers stay small.
The temperature in Seattle is the first thing that genuinely requires reaching outside. So does mathematics with large numbers or floating point, where models are unreliable in ways that do not improve with scale (2:36).
The distinction matters because most disappointing tool integrations are built for the first two categories. A tool that retrieves something the prompt could have carried adds a network round trip, a failure mode and a latency cost in exchange for nothing.
From this Brooker builds his definition, and it is narrower than the usual one: a system given a goal, looping between inference and tool calls until it reaches that goal (3:57).
Why code is moving inside the loop
His refinement of that definition is the observation most worth carrying away.
Increasingly, agent definitions contain code — tools written as code, code the model generates, and code implementing steps of a workflow directly (4:25). The reason he gives is not expressiveness. It is that each workflow step expressed as deterministic code is an inference call not made, which improves reliability and lowers both latency and cost at the same time (4:44).
That is an unusual alignment. Reliability and cost normally trade against each other; here the same substitution improves both. It also implies something about the direction of maturity: as agent systems get better, they should contain proportionally less inference, not more. The model becomes the thing that handles the genuinely ambiguous steps, and the rest hardens into code.
The stack, not the product
The framing of what production requires is deliberately structural, and Brooker is explicit that it is framework-agnostic — the concerns exist whichever agent framework a team picks, or none (5:22).
What an agent needs: somewhere to run. Memory, so preferences persist across sessions rather than being re-supplied. Guardrails. And a gateway connecting it to tools, both inside the company and outside it (13:47).
The gateway is worth pausing on. Connecting an agent to internal systems and connecting it to external services look like different problems — one is an integration question, the other a security question — and his argument is that they are the same problem stated twice, both requiring one governed path with consistent authentication and observability (14:29). Organisations that solve them separately end up with two answers that disagree.
Formal methods, applied to policy
The section he flags as most exciting personally is the one most likely to be skipped: several years of work combining neural and symbolic approaches, applied to policy reasoning (32:20).
The appeal is easy to state. A model asked whether an action is permitted produces a judgement that is usually right and occasionally, unpredictably, not. A symbolic system asked the same question produces an answer that is provably correct or provably fails — but it cannot interpret an ambiguous request. Putting the model at the interpretation layer and the formal system at the decision layer gets a system that can be reasoned about at the point where being wrong is expensive.
For agents with real permissions, that boundary is where the interesting engineering is.
Evaluation as infrastructure
The last section makes a claim by placement rather than argument. Evaluation appears in the same list as runtime, memory and gateway (39:54) — as infrastructure, not as a practice teams adopt.
The logic is straightforward once stated. An agent running a loop autonomously cannot be assessed by inspecting a single output; the output may be correct while the path to it was wrong in ways that will matter next time. What needs examining is the trajectory, and trajectories are not something you inspect by hand at any volume.
Which lands in the same place as the supervision arguments running through this conference season, arriving from an infrastructure direction rather than a cultural one. If the reviewing capacity is the constraint, then evaluation tooling is not a quality initiative. It is the thing that decides how many agents an organisation can actually run.
演讲章节
关键要点
- 01
His worked example separates what a model handles alone as a fixed function of input, what only needs to appear in the system prompt, and what genuinely requires reaching into the world — only the third justifies a tool. 2:36
- 02
The working definition is deliberately narrow: a system given a goal that loops between inference and tool calls until the goal is met. 3:57
- 03
Modern agent definitions increasingly contain code — tools written as code, code the model generates, and code implementing workflow steps. 4:25
- 04
The reason is economic rather than aesthetic: replacing inference steps with deterministic code raises reliability while lowering latency and cost, because each removed step is an inference call not made. 4:44
- 05
The infrastructure is deliberately framework-agnostic, on the argument that the runtime, memory and gateway concerns are independent of whichever agent framework a team picks, or none. 5:22
- 06
Production requirements are enumerated as a stack rather than a product: a place to run, memory that persists preferences across sessions, guardrails, and a gateway connecting internal and external tools. 13:47
- 07
The gateway exists because connecting an agent to company-internal tools and to external ones is the same problem stated twice, and both need one governed path. 14:29
- 08
He points to several years of work combining neural and symbolic approaches, applying formal reasoning to policy as the part that most excites him technically. 32:20
- 09
Evaluation is treated as infrastructure rather than a practice, on the basis that an agent looping autonomously cannot be assessed by inspecting a single output. 39:54
提及的实体
相关演讲
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Drawn from a year of engagements with more than a hundred companies, this is the most direct challenge in the season's programme to the assumption that faster code generation produces faster delivery. Mishra and Raja open with external evidence rather than their own: an industry study putting realised velocity gains in the ten to fifteen per cent range, and a controlled experiment in which developers using AI estimated themselves roughly a fifth more productive while measurement showed them a fifth slower. Their diagnosis is that both prevailing working styles fail for opposite reasons. Handing an ambiguous problem to an agent and awaiting a finished result produces a volume of code the developer must nonetheless sign for and cannot confidently review, so it stalls before production. The senior engineer's alternative — decomposing the work personally and inserting AI into narrow slots — keeps the intellectual load exactly where it was, and leaves the surrounding process untouched, so hours saved in editing are consumed by the meetings that process still requires.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The line that explains this session comes from the customer in the last ten minutes: they are preparing for a world where metadata is how agent-based systems find the data they need and access it through the controls being built. That relocates a function — governance has spent two decades as compliance activity describing data that people locate by other means, and if agents navigate by the catalogue then the catalogue stops describing the access path and becomes it. An incomplete catalogue is a documentation problem when humans can ask a colleague; an agent has no such workaround. The most honest moment addresses the perennial failure that rules get written and ignored, with enforcement rather than publication as the argument. Generated descriptions and greyed-out classification suggestions divide the labour correctly, keeping a person accountable while removing the burden of finding candidates.
The moment in this demonstration that deserves attention is a failure: the agent's first query returned the wrong results, so it consulted a mapping tool to learn what fields exist and rewrote the query with the correct names. That recovery is the capability, because production logging is never fully documented — field names drift and the schema in anyone's head is out of date. An agent that requires accurate documentation requires the thing nobody has. The scenario is well chosen, since with traffic at ten times normal and every service alerting, the difficulty is not detection but correlated noise. Where the session goes past observability is the business impact estimate, which multiplies observed errors by historical averages and produces a number for leadership whose assumptions are not visible — an unverifiable figure delivered alongside verifiable ones.