AI-assisted analysis. See our editorial policy.
Human editorial review not recorded
Brian Beach starts by inventing a programming language, and the invention is the whole methodology.
MathJSON does not exist. He made it up — or rather, had a coding agent make it up — precisely so that no model has ever seen it. Which means the first demonstration is a controlled failure: asked to work in a language absent from every training set, the tool fails completely (2:10).
That baseline is what most sessions about context management skip, and skipping it is why their advice cannot be evaluated. Every subsequent step in this talk has something to be measured against.
The naive fix works and creates a new problem
The obvious response is to give the model the documentation. Beach does exactly that: takes the language's function reference and puts it into a rules file the tool reads at the start of every session (10:21).
It works. The tool now produces valid code in a language it has never encountered.
It also drags roughly eleven hundred lines of documentation into context on every single request, including the many where the language is irrelevant. This is the failure that follows successful context injection everywhere — the fix is invisible because it works, and its cost accrues quietly on requests that had nothing to do with it.
His first correction is to compress: ask the tool to read the full documentation and reduce it to what it actually needs (12:58). The distinction matters — this is not summarising for a human reader, it is the model stating which parts of the reference it uses.
Prescriptive beats descriptive
The more interesting shift is in what the rules file contains.
Documentation describes a language. What Beach converges on is a file that instructs: what this thing is, when to use it, and — the part that changes the results — how to check the work. Once the file tells the agent to run the linter, it runs the linter and validates its own output rather than declaring success (15:19).
His illustration of the principle sits earlier in the talk and is the cleanest statement of it. He does not need to teach the tool Python; models know Python. He needs to teach it the way his organisation writes Python — docstrings always, particular conventions, what good looks like here (5:36).
That is a different category of information from a language reference, and it is the category that is unavailable anywhere else. The model can find documentation. It cannot find your standards.
Context on demand rather than context up front
The final architecture inverts the naive one.
Rather than embedding documentation, the rules file shrinks to a pointer: this language exists, here is when it applies, here is where the documentation lives, go read it when you need it. The agent reads the rules, recognises the situation, and fetches the reference through a tool call at the moment of use (17:56) — rather than from a copy that was current when someone pasted it.
Two properties follow. Context cost is paid only on requests that need it. And the documentation cannot go stale, because it is never duplicated.
This is the same shape as arguments appearing across this conference season from other directions — the tool-search work at Build loading only the tool a task requires, the specification arguments about pinning interfaces rather than describing behaviour exhaustively. Different problems, same resolution: put a small durable thing in context and fetch the large volatile thing when required.
The habit worth stealing
Beach's closing observation is a practice rather than a technique, and it is the one most likely to survive whichever tool a team uses.
Working on a complex file, the agent noticed it was struggling and announced it would try a different approach. His response was to ask it a question: would you like to update your own guidance so this does not happen again (18:59)?
The rules file, in other words, is not written once from what you think the agent needs. It accumulates from the moments where the agent visibly lacked something — and the agent is often the party best positioned to say what that was. Treating those moments as maintenance triggers rather than as friction is the difference between a configuration file that decays and one that improves.
Which reframes what these files are. Not documentation for a machine, but an organisation's accumulated corrections to a very capable colleague who joined recently and does not yet know how things are done here.
Key numbers
- ~1,100 lines
- documentation dragged into every request by the naive approach of embedding a reference 12:58
Talk chapters
Key takeaways
- 01
He establishes a controlled baseline by inventing a domain-specific language absent from every training set, so the first demonstration is a documented failure rather than an assertion. 2:10
- 02
Embedding a full reference into the rules file works, and costs roughly eleven hundred lines of context on every request including those where the language is irrelevant. 12:58
- 03
The shift that changes results is from descriptive to prescriptive: telling the agent when the material applies and instructing it to run the linter, after which it validates its own output. 15:19
- 04
The final architecture keeps a small pointer in context and fetches documentation through a tool call at the moment of use, so the reference cannot go stale. 17:56
- 05
His distinction between what models know and what only you know: you do not teach the tool Python, you teach it the way your organisation writes Python. 5:36
- 06
When an agent announces it is struggling and changing approach, treat that as a maintenance trigger and ask it to update its own guidance. 18:59
Entities mentioned
Organizations
Related talks
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Drawn from a year of engagements with more than a hundred companies, this is the most direct challenge in the season's programme to the assumption that faster code generation produces faster delivery. Mishra and Raja open with external evidence rather than their own: an industry study putting realised velocity gains in the ten to fifteen per cent range, and a controlled experiment in which developers using AI estimated themselves roughly a fifth more productive while measurement showed them a fifth slower. Their diagnosis is that both prevailing working styles fail for opposite reasons. Handing an ambiguous problem to an agent and awaiting a finished result produces a volume of code the developer must nonetheless sign for and cannot confidently review, so it stalls before production. The senior engineer's alternative — decomposing the work personally and inserting AI into narrow slots — keeps the intellectual load exactly where it was, and leaves the surrounding process untouched, so hours saved in editing are consumed by the meetings that process still requires.
Brooker builds the definition from the bottom up rather than asserting it, using a deliberately absurd arithmetic task to separate three categories: what a model computes reliably as a fixed function of its input, what merely needs to arrive in the system prompt, and what genuinely requires reaching into the world. Only the third category justifies a tool, and the distinction matters because most production disappointment comes from tools built for the first two. His working definition follows — a system given a goal that loops between inference and tool calls until it reaches one — with the observation that modern agents increasingly embed code in their definitions, not for expressiveness but because replacing inference steps with deterministic code improves reliability while lowering both latency and cost. The remainder covers what production actually demands around that loop: somewhere to run, memory that persists preferences, a gateway to internal and external tools, evaluation, and formal methods applied to policy.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The line that explains this session comes from the customer in the last ten minutes: they are preparing for a world where metadata is how agent-based systems find the data they need and access it through the controls being built. That relocates a function — governance has spent two decades as compliance activity describing data that people locate by other means, and if agents navigate by the catalogue then the catalogue stops describing the access path and becomes it. An incomplete catalogue is a documentation problem when humans can ask a colleague; an agent has no such workaround. The most honest moment addresses the perennial failure that rules get written and ignored, with enforcement rather than publication as the argument. Generated descriptions and greyed-out classification suggestions divide the labour correctly, keeping a person accountable while removing the burden of finding candidates.