The most instructive fix in this session is not a model change. A response was taking too long and containing too much, so the presenter added a constraint to the prompt — one sentence, fewer than fifteen words — and got a very fast response (7:49).
On a phone, output length is the cost. That single observation reorganises how you build these features.
Why length dominates on-device
A hosted model generating a long answer costs money and some latency, both absorbed by infrastructure. A model running on a phone generates every token using the device's own processor and battery, sequentially, while the user waits and the handset warms.
Which means the primary optimisation available to a developer is not a smaller model or a faster runtime. It is asking for less. A summary constrained to a sentence is not a degraded version of a longer summary — it is the version that runs at interactive speed.
The framing that makes this concrete is the original goal: a long itinerary screen was overwhelming, and the feature exists to make information easier to parse (3:15, 7:21). A verbose summary fails the product requirement and the performance requirement simultaneously, and the same fix addresses both.
The demonstration that proves the point
The whole first section runs in airplane mode (2:22).
That is the cleanest possible demonstration and it settles what is actually being shown. Nothing here depends on the network, which means the capability is available on a plane, on a train, in a rural area, or in any of the situations where a travel application is most needed and least connected.
The adoption figure attached — over 140 million devices running versions of the on-device model (4:38) — is what makes it a platform decision rather than a flagship-only feature.
Where the boundary is drawn
The privacy argument is made about a specific case: receipts may contain card numbers and addresses, which makes local processing appropriate (9:13).
That is a good example because the sensitivity is unambiguous and the task is bounded. It is also the case where the demonstration reaches for the larger cloud model, which takes noticeably longer (11:03).
The session does not comment on the tension, and it is worth noting. The reason given for staying local is privacy; the model used for the harder version of the task is the remote one. Both choices are defensible and they point in opposite directions, and the actual architecture of an application like this is going to be a routing decision made per feature — with the sensitive, simple cases local and the complex ones remote, which is precisely the split the privacy argument was meant to avoid.
The change developers should care about
The most consequential improvement described is structural rather than capability-based.
The current approach is to write a prompt describing the fields you want, include an example of the intended output format, then parse the returned text into a usable object and hope it conforms (11:57). Every one of those steps can fail, and the failure appears at parse time, on a device, in front of a user.
The replacement marks a data class as generable and annotates its properties with guidance (12:25). The type system describes the expected output, and the plumbing between text and object stops being application code.
That is the same move that made network interfaces manageable a decade ago: describe the shape once, generate the handling. It is unglamorous and it removes an entire category of runtime failure — which, on a device where you cannot inspect what went wrong, is worth more than any capability improvement in the same session.
关键数据
- 140 million
- devices running versions of the on-device model 4:38
演讲章节
关键要点
- 01
Constraining the prompt to one sentence under fifteen words turned a slow verbose response into a fast usable one. 7:49
- 02
The whole first section runs in airplane mode, which settles that nothing being demonstrated depends on the network. 2:22
- 03
Over 140 million devices run versions of the on-device model, which makes this a platform decision rather than a flagship feature. 4:38
- 04
Receipts containing card numbers are the privacy argument for local processing, yet the harder version of that task uses the cloud model. 9:13
- 05
Marking a data class as generable replaces prompt-plus-example-plus-parse, removing a category of runtime failure you cannot debug on a device. 12:25
提及的实体
相关演讲
Four of Google's model, product and search leads on what changes once agents run for hours rather than seconds, and the most quotable argument comes from Dean: the constraint is moving out of the model and into the tools around it. By Amdahl's law, an agent spending half its time in tools built for human-speed interaction cannot gain more than a doubling however fast the model becomes — which reframes a great deal of current infrastructure work as latency debt. Their internal response is concrete: rewriting Python tooling into Go, framed as a fully specified translation task rather than an open prompt, produced order-of-magnitude speedups overnight. Reid supplies the counterweight from Search, where acceptable latency turns out to scale with how much work is being taken off the user rather than being a fixed budget. Woodward's detail is the quietest and perhaps the most telling: teams that have stopped writing product documents for humans and now write context files for models to act on directly.
The clearest statement at I/O of how an agent API differs from a chat API, and the reasoning behind each departure is stated rather than assumed. Three changes matter. Conversation state moves to the server: a call returns an identifier, and passing it back continues the thread, retiring the client-side history array. The data model abandons alternating user and model turns for discrete steps, on the argument that a trace containing reasoning, tool calls, environment responses and compaction was never really a conversation and modelling it as one distorted it. And agents receive their own persistent remote environment rather than acting on the caller's machine — addressable by identifier, and shareable, so a research agent's output files become an application builder's input without passing through the context window. Schmid is explicit that scaffolded environment files are deliberately not model input, which is what keeps large artefacts out of the context budget. Schaeff's first half covers the real-time voice path, where the notable property is speech-to-speech across ninety languages with transcription of both directions.
The anecdote that opens the panel does the work: a robot asked to pick up the extinct animal selected a dinosaur toy, with nothing in its training data connecting the phrase to the object. That transfer from language models into machines with hands is the premise of the current wave. What the practitioners then describe is where it stops. Physical intelligence is about exerting force and using a body to do it, which is knowledge about consequences — the one thing a corpus of internet images contains almost nothing about. The humanoid question gets an honest treatment: not that human shape is optimal, but that the world is already built for it, plus a development-loop argument about collecting data and deploying on the same hardware. The most useful passage is scepticism about the field's favourite shortcut: generated video looks realistic and does not hold up for dexterous manipulation, because looking right and being physically consistent are different properties.
The observation that organises this session is not about capability but about attention. Engineers increasingly file a ticket rather than opening an editor, and the code comes back — which changes what the surrounding tools are for. If the agent works while you do something else, the conversation between you is no longer a workspace; it is the mechanism by which the agent surfaces a question it cannot resolve alone. Interfaces built for continuous conversation optimise for flow, and interfaces built for interruption should optimise for the opposite. A runtime constraint follows immediately: an agent that starts a long-running job cannot block until it finishes, which turns out to be a workflow-engine problem rather than a model one. The panel's closing formulation — that deciding what to build is the hard skill and always was — reads as reassurance and functions as a warning, since that judgement is downstream of exactly the work now being delegated.
The most honest moment here is an aside about how the presenters have tracked their own projects: plans in documents, plans in spreadsheets, plans in bug comments, and once a plan written on a receipt. That describes the actual category being addressed — not software nobody has built, but the small internal tool every team improvises badly because building it properly was never worth the effort. The demo turns on a single question: the generated app is strictly single player, so what happens when you want to share it with the team? That boundary is where improvised tools historically died, because it is where accounts, shared storage and access rules begin. Here it is crossed in one step, with the access rules generated and deployed automatically — which is convenient, and is also the moment the application acquires obligations nobody reviewed.
Pichai's framing of Google as the buffer between people and the raw internet is offered as continuity — search did it, browsers did it, agents do it more — and it is also the most contested claim in the industry, because a buffer decides what passes through. He reaches immediately for the counterweight, the connection people feel to creators they follow, which is precisely the tension the company is currently managing without resolving. Two answers are sharper than the format usually produces. On competition he describes participants running on different pre-training and release cadences rather than at different speeds in one race, which is a more honest account than the leaderboard framing and comes from someone with an interest in leaderboards. On security he acknowledges models improving at cyber work, which is the one domain where better capability does not obviously net out positive, since an attacker needs one vulnerability and a defender needs all of them.