Inference Optimization

The argument for running models locally used to be privacy. Here it is more specific and better reasoned: with a smaller model you can work at the full context length and largely stop worrying about compaction, because agentic workloads run locally hit hardware limits quickly. Compaction is the failure mode that quietly degrades long agent runs, when an agent forgets a constraint from forty steps ago or keeps a summary that lost the detail that mattered. Framing local inference as a way to avoid it is a real trade with a stated boundary — not that local models are better, but that capacity to hold the whole problem can beat raw capability on any single step. The threshold claim that local models are now viable for real work is the kind that changes behaviour, and thirty seconds from nothing installed to working makes the experiment nearly free.

Microsoft Build

The session builds on the observation that thinking models consume large numbers of tokens while reasoning, which means serving no longer resembles the profile that shaped serving infrastructure. Their account of where difficulty lives transfers past any hardware: the hard problems are memory management, scheduling and utilisation, not arithmetic — the chip does the arithmetic, and the surrounding system decides whether it is busy. The concrete bottleneck is key-value cache management as contexts lengthen and concurrency rises, with reuse rather than recomputation as the optimisation. The batching change is small to state and large in effect, because variable-length output stopped being an edge case and static batches are held hostage by their slowest member. The commercially decisive claim is that none of this requires rewriting the application layer, since gains behind an unchanged interface are adoptable by teams who never think about hardware.

Google I/O

The specification change that matters is the context window moving from 32,000 tokens to 128,000 for smaller models and up to 256,000 for larger ones, because it changes which problems are solvable without infrastructure. A 32,000-token limit means retrieval, chunking and index management; at 128,000 many tasks fit whole and the workarounds become unnecessary. The deployment range runs from a browser with zero ongoing server cost through local runtimes to one-click hosted endpoints, and the strategically significant detail is interface compatibility — a local model speaking the same protocol as hosted APIs is substitutable without changing application code, which makes placement an operational choice rather than an architectural commitment. That is what makes open weights competitive: not being better, but making switching free.

Google I/O

The case for local inference is made in three clauses representing different kinds of constraint: cost, where a local model removes an API call; availability, where the application works on a flight; and data, where requirements prevent information leaving the device. Only the third changes what is buildable rather than what is affordable. What makes this newly practical is unglamorous — instruction set extensions integrated into the runtime rather than a modelling breakthrough — with around thirty per cent improvement reported in image editing functions. The guidance on fine-tuning inverts the usual advice: adaptation matters most for the smallest models, because their capability budget is already spent and getting them to perform on your problem means spending some of it there. The two examples do the real work, since neither is a cheaper version of a cloud application.

Google I/O

The most instructive fix in this session is not a model change: a response was taking too long and containing too much, so the presenter constrained the prompt to a single sentence under fifteen words and got a very fast response. On a phone, output length is the cost, because every token is generated on the device's own processor while the user waits. That reorganises how these features are built, since the constrained version satisfies the product requirement and the performance requirement with the same change. The first section runs in airplane mode, which settles what is actually being demonstrated. The privacy argument is made about receipts containing card numbers, and the demonstration then reaches for the larger cloud model for the harder version of that task — a tension the session does not comment on. The structural improvement, marking a data class as generable, removes an entire category of runtime failure.

Google I/O

The example that justifies this session is a farming application in rural Maharashtra translating agricultural insights and weather warnings into a regional Marathi dialect, offline. That is not a latency argument — the user is unreachable by a hosted model twice over, once through connectivity and once through dialect coverage, and neither problem is solved by a more capable model in a data centre. The most concrete engineering claim concerns speech, where a conventional approach needs a separate 100-to-200-megabyte model per language while the on-device approach needs one small adapter across all supported languages, turning a linear install cost into a nearly fixed one. The guidance to always use the smallest model that reliably solves the use case is the opposite instinct to server-side development, where there is always a later.

Google I/O

Jensen Huang used NVIDIA's 2026 GTC keynote to argue that AI has crossed an inference inflection: models that once only generated text now reason and act, and each step multiplies the compute a single task consumes. He put NVIDIA's forward demand visibility above one trillion dollars through 2027, then spent much of the keynote explaining why that is a factory-economics claim rather than a chip claim — a gigawatt of AI factory costs roughly forty billion dollars before any compute is installed, so throughput per watt is what determines revenue. The technical centrepiece was the Vera Rubin platform; the strategic surprise was NVIDIA absorbing the Groq team to cover the low-latency decode that NVLink alone cannot reach. He closed on two extensions of the agentic thesis: OpenClaw as an emerging operating system for agents, hardened for enterprises as NemoClaw, and physical AI, where four new robotaxi partners add roughly eighteen million vehicles a year.

NVIDIA GTC

The panel is billed as an energy discussion and turns into an argument about who decides. Sridhar's observation is the one that carries: large refineries draw hundreds of megawatts and do not use the grid at all, generating on site because routing industrial demand through infrastructure built for everyone else never made sense — and upgrading that shared infrastructure for a single gigawatt-scale consumer is, in his word, bonkers. Around that sit the constraints that make it tempting. Grids in the United States and Europe are decades old, permitting runs to years even where it has been halved, and Busch's account of Sweden adding ten reactors' worth of unstorable wind capacity while consumption stayed flat is a precise illustration of why installed capacity is the wrong measure. Payne supplies the argument operators will use wherever power is scarce: AI infrastructure produces more economic output per electron than any other industry by an order of magnitude. Persuasive on its own terms, and about to be deployed in places where the competing use is a neighbourhood.

World Economic Forum Annual Meeting

The most useful admission here is that intuition is what fails: the team stepped back because performance work was not going as fast as they wanted and found their instincts were leading them astray. The mechanism is that bottlenecks hide others — you remove the visible one and uncover a worse problem that was masked behind it, which is why a correct fix so often moves no numbers. Their answer is to survey broadly before drilling in, explicitly checking whether the processor back end is worse, finding it is not, and setting it aside. The diagnostic figure is instructions per cycle averaging below one against hardware capable of eight to twelve, which reframes the problem from slow code to an idle processor. Every fix that follows is about memory layout rather than logic, and returns nearly twenty per cent more throughput from configuration alone.

AWS re:Invent

The number that matters here is not a benchmark but a shape: scaling up to 2.2 million virtual cores and back down to zero across four hours. Peak capacity has never been the hard part of scientific computing, because institutions have built large clusters for decades — the difficulty was that the cluster was sized for the peak and idle the rest of the year. Returning to zero removes that calculation, which suits genomics precisely, since analysis is bursty by nature. The acceleration claim is a change in the kind of activity rather than a productivity gain: eight hours means return tomorrow, thirty-five minutes means adjust and run again. The hardware explanation is unusually clear that cores per chip rather than total core count drives the improvement, and two operational findings — provisioning cost and a storage benchmark that showed no difference — are more portable than the hardware itself.

AWS re:Invent

Everything in this session is downstream of one diagram: an accelerator has finite memory bandwidth and finite compute throughput, and where a workload sits between them is decided by how many operations it performs per byte read. Landing memory-bound means expensive compute units idle while data arrives, and every technique demonstrated exists to move a kernel across that line. The natural objection is that a compiler should handle this, and the implicit answer is that it cannot, because the decision depends on tensor shapes whose relative importance a general compiler cannot infer. The kernel structure repeats identically for every operation — allocate on-chip, copy down, compute, copy back — which is an explicit data-movement schedule with arithmetic attached, and which ties the code to one memory hierarchy. Two practices are worth stealing regardless of hardware: measuring on a few layers rather than a whole model, and verifying numerical equivalence rather than assuming it.

AWS re:Invent