Even with the screen off, Android assembles a snapshot of you: movement, sound, device state. And when you dictate, it recognizes your own words, learned on the device. Two parts: the captured signals, then the learned vocabulary.
Context detection does not live inside an app: it sits in Android's system layers, upstream of apps, and runs continuously. Three on-device building blocks compose the picture: where you are and how you move, what the microphone can hear around you, and the state of the device. Every captured signal maps to a service you already use.
It is not an app that watches. It is the system itself.
At regular intervals, the device assembles a single object bringing together ten dimensions. Each comes from a different sensor or system service, but they are merged into one timestamped snapshot, compared against the previous one. Scroll: the sensors light up one by one around the core.
inferred from motion sensors: still, on foot, running, cycling or in a vehicle
position computed by crossing GPS, visible WiFi networks and cell towers
headset plugged in or not, plus paired devices in range (watch, car, speaker)
small Bluetooth transmitters installed in some venues (shops, museums), detected as you pass
connected to a recognised WiFi (home, office), on mobile data, or offline
battery level, device charging or plugged into the mains
on or off: is the phone actively in use or set down somewhere
local conditions (temperature, rain) supplied by the weather service at capture time
category of the moment: morning, evening, night, weekday, weekend, public holiday
the generic container from the Play Services context manager: signals already interpreted elsewhere, added as one block
Each snapshot embeds three tracking fields found in the code: the time of the current capture, the time of the previous one, and the number of motion events recorded in between. The previous timestamp is what changes everything: each capture is linked to the one before it, and the series forms a continuous chain rather than a run of isolated photos. The system therefore knows not only your state at this moment, but also how long ago it last observed you and how much you moved in the meantime.
The day cut into segments, each labelled with a mode and a place.
An on-device model takes the position trace (GPS, WiFi, cell, altitude) and cuts it into segments. For each one it assigns two things: a travel mode and the most likely place among candidates. Thirteen transport modes, one stop score, one segment-division score, and ten candidate-place scores: twenty-five outputs in total.
// 25 score outputs of the mobility segmenter // 13 transport modes WALKING RUNNING CYCLING MOTORCYCLING IN_PASSENGER_VEHICLE IN_BUS IN_SUBWAY IN_TRAIN IN_TRAM IN_FERRY FLYING SAILING SKIING // stop at a place + boundary between two segments VISIT_SCORE DIVISION_SCORE // 10 candidate places ranked by a score FLOAT_CANDIDATE_SCORE_N00 … FLOAT_CANDIDATE_SCORE_N09
The two scores in the middle structure everything else. VISIT_SCORE measures the probability that a segment is a stop: the device is no longer moving, it is staying somewhere. This score competes with the thirteen transport modes: at any point of the trace, you are either moving in a given mode or you are on a visit. DIVISION_SCORE measures something different: the probability that a boundary falls at this exact point of the trace. It is what decides where the day gets cut, the instant where the walk ends and the visit begins. The resulting alternation, trips on one side and visits on the other, is exactly the structure Google Timeline displays: a high VISIT_SCORE creates a visited place, a high DIVISION_SCORE closes the previous segment and opens a new one.
To decide between two places at the same address but on different floors, the model compares the device altitude with the known height of the place. Strung together, these segments form the pattern of life: the repetition of trips, stops and schedules that describes an ordinary life.
527 scorable sounds, 45 armed with a trigger threshold.
The model can score 527 types of sound (the full AudioSet ontology), but the app only acts on 45 of them: the ones carrying a coded trigger threshold. On the gauges, the bar is that threshold. Short bar = low threshold = easy trigger, the sound is caught at the faintest hint. Long bar = near-certainty required before reacting.
The contrast is striking. The floors (slam at 0.0045, moan at 0.008, applause at 0.010) are set to fire at the first suspicion. At the other end, music demands 0.9259, the highest threshold of all: the system waits for near-certainty before believing a track is playing, to avoid triggering a "music" context on mere background noise.
Movement, sound, device state: taken separately, these are fragments. Merged into a single snapshot, they draw a moment of your life. The signals converge.
The repetition of trips turns into anticipated automations.
The places the mobility segmenter sees returning day after day do not stay dots on a map. The device turns them into geofences: zones around which it arms triggers. Arriving somewhere, leaving it, and an action gets ready: a routine, a reminder, a suggestion.
The mechanics read in the code as three steps. First, repeated visits accumulate: the mobility segmenter produces scored candidate places, which the code groups into an object named GroupedScoredPlaces. Then, the places that come back often enough get promoted: a component called RoutinesGeofenceHandler registers a virtual perimeter around them with the location service. Finally, crossing that perimeter wakes the app, even closed: the context snapshot qualifies the moment (time, activity, headset, network) and the prepared action fires.
What the system knows of your comings and goings thus becomes the implicit schedule of what it will offer you. The three scenarios below are plausible reconstructions, not captures: the sequences are illustrative, but every small-caps event maps to a real signal found in the code.
After a few weeks of the same sequence, simply leaving the zone is enough: the screen gets ready before you have asked for anything.
The weekly repetition (same place, Tuesday and Thursday) arms the trigger; the plugged-in headset, one of the snapshot's ten dimensions, confirms the moment.
The reminder was set days earlier; it is the crossing of the perimeter, not a clock, that surfaces it.
The "to serve you better": every convenience has a named capture on the other side.
The loop closes here. Signals are not captured for their own sake: they feed products you open every day. Each service maps to a named set of signals.
The article feed is tuned to where you are, at what time, how you move and what state the device is in.
What the search box proposes before you type a single letter, anchored to the current context.
The same question does not produce the same answer. Headset plugged in: the answer is read aloud. In the car: stripped-down interface, everything goes through voice. At home in the evening: shortcuts to connected devices and routines; at the office: the day's calendar and reminders.
A smoke alarm, a doorbell or a crying baby: the phone vibrates, fires the flashlight and alerts on screen and watch (the accessibility feature for hard-of-hearing users). Ordinary sounds become an ambience label that cross-checks the other signals: traffic confirms the GPS trip, music qualifies the moment.
Sensing context is only half the story. The other half plays out on the device: your phone learns your own vocabulary and fixes what you dictate, with on-device models, without a single byte going to the cloud. Here is that path, in six blocks.
Why your phone knows how to write your own words.
How does your phone know? Two memories: what you say, and what you have already typed. They are about to meet up.
Sliced, cleaned, then projected into the same space as words.
Sound is not "recognized" in one go. It is first turned into a series of numbers, vectors (what engineers call embeddings), placed in the same mathematical space as written words. That is what later lets the phone compare your voice to your vocabulary. One important point: these voice vectors are transient. They serve to transcribe the current sentence, then get discarded; your voice itself is not archived in this form.
The file audio_adapter_lora_asr_v3.tflite takes acoustic features of shape [3 × 1536] and projects them into tokens of shape [3 × 2048]. It is a LoRA adapter: a small trained layer that "plugs" audio into the language model space, without retraining the whole model.
Upstream, an audio front-end (ROSIE_ROBOT: Framer, Stacker, Subsample, AutoGain) shapes the signal continuously, frame by frame.
Contacts, words your keyboard has learned, apps and calendar, encoded into an encrypted memory bank.
Your proper nouns are not in the base model. Your phone collects them: your contacts, the keyboard's personal dictionary (the names, acronyms and expressions you type often, which autocorrect eventually learned instead of underlining), the names of your apps, your calendar. It turns them into a compact, encrypted "cheat-sheet", limited in size, that it can read back on every sentence.
An entry is a short sequence specific to you: a contact name, an app name, an expression. It is tokenized then encoded, within a limit of 32 tokens (a few words).
How many words is that? The cap is expressed in entries and tokens, not words, but a range can be given. A real entry holds one to five words ("Linh Nguyen", "Venmo", "quarterly review"): full, the personal dictionary therefore retains on the order of 2,000 to 10,000 words of your own. And once vectorized, this memory is tiny: the full bank weighs about 0.26 MB, less than a single photo.
The encoder nam_retriever.tflite takes phrases of [N × 32 tokens] and outputs a memory bank [N × 128]. It is a frozen Gemma-style mini-transformer: 4 layers, GQA attention, GeGLU activation, RMSNorm, INT8 quantized, about 5 MB.
The list size is not free: sub-graphs are pre-compiled for fixed sizes (128, 256, 512, 1024, 2048 phrases). Beyond 2048, it has to prioritize.
Volume: 2,048 entries of 32 tokens give a theoretical maximum of 65,536 tokens, roughly 40,000 to 50,000 words if every entry were packed full. In practice entries are far shorter, hence the 2,000 to 10,000 word range. The encoded bank is [2048 × 128] in INT8, i.e. 262,144 bytes: about 0.26 MB.
At first it only knows the language; its personal lexicon builds up with use.
Out of the box, a phone knows the language but not your jargon: it will write "CEO". It ships with a generic "factory brain", then its personal lexicon fills up, and the list grows in steps.
A misconception to clear up: your phone does not retrain its "brain" on every word you type. It keeps a personal lexicon up to date instead, a vocabulary list living next to the model. Your usage does the sorting: a newly saved contact, a freshly installed app, a word your keyboard sees you type often earn their place on the list; as the 2,048 slots fill up, the least-used entries give theirs up to the most frequent and most recent ones. On every change, the lexicon is re-encoded into vectors, ready to be consulted. It is memory being read back, not on-device tuning of the model weights.
The "factory brain" is proven by the *_default_adapter.bin files (generic LoRA, about 36 to 37 MB). The step-by-step growth (128, 256, 512, 1024, 2048) is proven too, via the pre-compiled sub-graphs.
How the lexicon fills, concretely: the device collects entries specific to you (contacts via the READ_CONTACTS permission, person names, keyboard dictionary, app names, calendar). They are tokenized (nam_tokenizer_spm), then encoded by nam_retriever into [N × 128] vectors. When the list changes, the bank is re-encoded. This is not retraining: personalization lives in a retrieval bank the decoder consults, not in the model weights, which are frozen and baked into the shipped model.
Usage frequency matters, but upstream: the keyboard and contacts database are what track which words recur, and that frequency is used to pick then prioritize entries once the 2048 cap is reached. Harvesting (contacts, person names) is proven by the files; frequency-based curation and re-encoding on change follow documented design [D].
On each session start, prefill graphs (.nncz files) pre-fill the attention cache (KV-cache) to start faster.
The model matches your voice against your personal lexicon, and leans toward your words.
This is the core moment. Two flows converge: your encoded voice (blue) and your memory bank (orange). When picking words, the Gemini Nano decoder consults your lexicon and leans toward your own names.
« Text Nguyen I'm running 10 minutes late » thanks to your contacts
« Open Venmo and pay Priya back » thanks to your app names
« Book Dr. Nguyen for June 26 » thanks to your calendar
* dashed line: link from Google's documented design, not a tensor found in the open files.
The memory bank [N × 128] is linked to the decoder through a 128-dimension projection. Concretely, during generation, the decoder "consults" the bank and raises the probability of your entries: that is the bias.
Capitals, punctuation, digits: one last layer makes the text readable, still on the device.
What the decoder produces is still raw text: all lowercase, no punctuation, numbers spelled out, like a telegram. A clean-up chain then passes over it: it splits sentences, places capitals and punctuation, and rewrites as digits what was dictated as words. "june twenty sixth" becomes "June 26", "three thirty pm" becomes "3:30 PM". Nine languages are covered, and everything still happens on the device.
raw output: meeting with dr nguyen on june twenty sixth at three thirty pm
The same mechanics apply to amounts, numbers and titles:
"twenty five hundred dollars" → "$2,500"
"five five five one two one two" → "555-1212"
"doctor nguyen" → "Dr. Nguyen"
byte sent to the cloud for this sentence
Finishing relies on the greco_* family (de-normalization, punctuation, capitalization, sentence splitting), multilingual: de, en, es, fr, it, ja, pt, hi, zh. Nine languages attested in the files.
Analysis of the on-device AICore models (PublicComputeServices manifests), June 2026. The CDN blobs are zlib, decoded and verified. The encoding path (voice and lexicon into vectors) is proven by the files; the acoustic bias link (decoder and memory bank cross-attention) follows Google's documented NAM design, not a tensor found in the open files.
One distinction kept for rigor: mT5_cached_bias16.bin is a model-internal relative attention bias, not the personal vocabulary.
The captured context and the learned vocabulary are the base layer. Here is what Google is building on top.
These six scenarios are no longer science fiction: they are being built in Google's code, and they rest on the building blocks described in this page. Excerpts from our study "Tomorrow's AI phone, seen from inside a Google APK".
