LLM Reliability

Bennett puts an eleven-line agent on screen and runs it repeatedly: pass, pass, pass, fail, fail, fail. It works about half the time, and nothing in the code says so. That breaks the debugging method conventional software allows, because every test run becomes a sample rather than an observation and a fix appears to work when you happen to draw three passes. His demonstration is the argument: changing the model and rerunning raises the success rate to around ninety per cent with no change to the application at all. The number matters less than the method — he knows the change worked because he measured a rate before and after, and without instrumentation the swap would have been indistinguishable from luck. The uncomfortable implication is that if model choice moves reliability that far with no application change, most published comparisons of agent patterns are reporting noise around a variable they did not control.

Microsoft Build

The framing worth separating from the product is that agents are no longer static routers shuffling requests between fixed tools; they acquire skills, generate memory and accomplish things nobody programmed. A router can be tested exhaustively because its behaviour is bounded by configuration. A system that gains capability during operation cannot, because what it does next week depends on what it accumulated this week. The response described is continuous evaluation fed by every action and cost signal — the right shape of answer and a considerable operational commitment, since you end up running an evaluation apparatus permanently at a cost proportional to the thing evaluated. The quieter shift is agents woken by events rather than requests, which removes the natural boundaries a requester provides: unbounded cost, unattributable actions, and no clear answer to who authorised any particular piece of work.

Microsoft Build

The method is stated plainly enough to hold them to: choosing a model is guess and check. What makes the walkthrough useful is that it attaches numbers to steps usually described qualitatively. A customer service agent handling returns starts at 76 per cent quality — their framing is not bad, not great, not shippable, with production needing 90 or better. The progression that follows is ordered by cost rather than sophistication: show the model a pattern to imitate, then supervised fine-tuning of a smaller model with scores moving as you go, then reinforcement where answers are checkable. The observation most likely to survive is about curriculum — you do not teach the hardest problem first, you teach addition before you teach arithmetic — which makes training data ordering a design decision rather than a preprocessing detail. What the session does not supply is the volume at which the effort amortises against serving cost.

Microsoft Build

The chart organising this session is not about productivity. It shows roughly three times more issues arriving and considerably more open pull requests since January, which is the honest shape of a successful agent rollout and almost never the number presented. The team's response was to move from monthly to weekly releases, a change forced by throughput rather than chosen for confidence. Their code-survival metric — the share of a model's output that actually gets committed — is unusually well chosen because it is measured after the human decision and so cannot be inflated by generating more. The most transferable practice is running the editor with and without a change and attaching the comparison to the pull request, which turns the pull request into the specification. They also state plainly that running AI over community-submitted issues means running it on untrusted input, without detailing the mitigation.

Microsoft Build

Three claims here are worth separating from the announcements around them. The speech numbers are meaningful in different ways: blind preference testing is the right evaluation for synthesis where no ground truth exists, and a latency under 150 milliseconds crosses the threshold where spoken exchange stops feeling like a request and starts feeling like conversation. The principles slide contains an actual position rather than decoration — capability should be learned rather than inherited, which takes a side in a live argument about building on existing base models versus training capability directly. And the training description carries the most transferable idea: generate solutions, score against verifiable ground truth, reinforce the better ones, with the clause about verifiability doing all the work. That explains the shape of progress across the field, and it is the same boundary Hassabis drew at Davos, arriving here as engineering description rather than forecast.

Microsoft Build

The most useful moment is an aside rather than a demonstration: why do all that manual work ourselves, asked immediately after an application crashes. What follows is an argument about which parts of operational work were only ever done by people for lack of an alternative. Troubleshooting is one of the few engineering activities that is genuinely unschedulable — it interrupts whatever was planned and costs attention at the worst moment — so reducing the interruption is worth more than the hours saved. The deployment description contains the detail separating a demonstration from something operable: monitoring with automatic rollback, without which an agent that deploys is a mechanism for producing outages faster. What the session does not address is the failure mode this architecture creates, where a fix addresses a symptom, deploys cleanly and passes monitoring while the real defect remains.

Google I/O

The story that should outlive this session is about a question that worked perfectly and cost 1.3 million tokens. The answer was good, it passed the team's question-and-answer checks and acceptance testing, and the tool calls were sensible — yet a single query consumed enough to have destroyed throughput at production scale. That is a category of defect traditional software mostly does not have, because the cost is incurred inside a reasoning process with no fixed shape. The team extended their agent framework for per-turn reasoning visibility and then bounded how much data the agent could pull, a sequence that matters because the guardrail could not have been written before the behaviour was observed. Their architecture separates agent logic from data logic so each side scales independently, and their closing advice is to find the simplest pattern that delivers the benefit and analyse with real data afterwards.

AWS re:Invent

Buried near the end is the most useful sentence in the session: three agentic products are in production, one is about to launch, and one was taken back to the drawing board — and that last one produced some of the most valuable data the team got. The technical argument builds toward verification, starting from a limitation rather than a capability: traditional testing only goes so far because these models are probabilistic, which quietly invalidates most of an enterprise QA apparatus. Their answer is to measure properties rather than check outputs, tracking relevance, completeness and tone while noting other organisations will need different measures. The distinction between hard and soft guardrails clarifies the design question of how much safety requirement can be pushed into a deterministic layer, and their red-teaming runs as a schedule rather than a gate.

AWS re:Invent

The moment in this demonstration that deserves attention is a failure: the agent's first query returned the wrong results, so it consulted a mapping tool to learn what fields exist and rewrote the query with the correct names. That recovery is the capability, because production logging is never fully documented — field names drift and the schema in anyone's head is out of date. An agent that requires accurate documentation requires the thing nobody has. The scenario is well chosen, since with traffic at ten times normal and every service alerting, the difficulty is not detection but correlated noise. Where the session goes past observability is the business impact estimate, which multiplies observed errors by historical averages and produces a number for leadership whose assumptions are not visible — an unverifiable figure delivered alongside verifiable ones.

AWS re:Invent

The most useful sentence here concerns where reliability is achievable: for workflows calling well-structured interfaces and producing checkable code, the path is faster. The implication is that there is a slower path, and it is the one most enterprise pilots are on. The division separates work whose output can be checked mechanically from work where correctness is a judgement, and only the first admits an engineering approach to improvement. The best illustration is browser automation that examines what the customer would observe rather than the underlying markup, walking checkout and booking flows and identifying revenue leakage — a framing that defines success in the business's own terms and is verifiable in the good sense. The demonstration's fifteen to twenty tool calls are the honest measure of what agentic means: not one clever response but a chain of decisions each depending on the last.

AWS re:Invent

The most honest moment in this session comes near the end: most of the security processes the team has automated still route their output to security engineers rather than the business, because the results are not good enough. That sits half an hour after a story about two engineers building a working prototype in two days that runs a training exercise in seven minutes for ninety-one cents. The gap between the two is the subject. What distinguishes them is not model capability but whether the task has a verification loop a machine can close, which reframes the automation question entirely. Three loops appear — citation, self-checking, and a compiler pass that guarantees syntactic validity — and only the last removes a person. The rejection of single-number evaluation scores in favour of precision and recall follows directly, because in security the two error types have wildly different costs.

AWS re:Invent