An Agent in Your Pocket

Gemma 4 on a phone through LiteRT-LM — the app, the 270M model that only calls tools, and the Kotlin API — with the laptop measured here and the phone numbers labelled Google’s

Author

Behzad Samadi

Published

September 13, 2026

A paper-craft scene: a smartphone tucked into the chest pocket of a navy jacket, its screen glowing amber, the five-dot AI glyph floating above it, and a cut-paper hillside path with a cottage and one cloud behind. Title: An Agent in Your Pocket.

The last guide put an agent on the computer you can unplug. This one asks the next question: what can the same agent honestly do on the device that is actually with you — in a field, on a train, in a clinic corridor with one bar of signal — and what does it cost in seconds? The answer changed in 2026. Google’s on-device runtime, LiteRT-LM, now calls tools; Gemma 4 ships in files built for phones; and the app Google uses to show it off has a benchmark screen, agent skills, and an MCP client. The pieces exist. Whether they add up to an agent depends on one number, and this guide spends most of its time on it.

One honesty, updated. A phone did arrive — a Galaxy S26 Ultra, the exact device Google’s own table names — and this guide now carries real numbers from it, alongside Google’s, said so wherever they appear. What follows is not what the table promised: the flagship phone Google benchmarked turned in prefill throughput close to this laptop’s, not several times it, using Google’s own benchmark app on Google’s own published configuration.

NoteTL;DR
  1. The runtime is LiteRT-LM. One .litertlm file per model, one engine on Android, iOS, the web, the desktop and a Raspberry Pi, with Kotlin, Python and C++ APIs marked stable, Swift and JavaScript in preview. It calls tools. Version 0.17.0 on the day of writing.
  2. The phone models are Gemma 4 E2B and E4B, 2.6 and 3.7 GB on disk, text, image and audio in, 32K context. The 12B is “coming soon” for the phone runtime. Google claims 3,808 prefill and 52 decode tokens a second on a Galaxy S26 Ultra GPU; measured here, on that same phone, with Google’s own app: 1,632 and 37.
  3. Start with the app. The Google AI Edge Gallery (Android 12+, iOS 17+) downloads the model, has a benchmark screen for your hardware, and already does skills, MCP servers and on-device actions.
  4. The phone-shaped trick is a 270M model that only calls tools. FunctionGemma, fine-tuned, goes from 58 to 85 percent on Google’s Mobile Actions set and runs at 142 tokens a second on a Pixel 7 Pro CPU. A small router in front of a small reasoner is an architecture the laptop never needed.
  5. The number that matters is prefill, and the measured phone rate is the one to plan from. A minimal Hermes turn measured here is about 3,000 tokens before the model says a word. At the S26 Ultra’s measured GPU rate that is 1.8 seconds a turn, not the “under a second” Google’s claimed rate implies; on its measured CPU rate, 11 seconds; a full 16K agent prompt, 9.8 seconds on GPU and just under a minute on CPU.
  6. Same runtime, same server. litert-lm serve speaks OpenAI on port 9379, and Hermes ran a tool loop through it here — after it refused the model’s real 32K context and was told a bigger one. Hermes also installs on Android through Termux, and the runtime publishes Android wheels. The two have not been tried together on a phone. Not yet.

Four things, not three

The laptop guide had three parts — model, server, agent — joined at a URL. A phone has the same three and one more, and the extra one is where the phone stops being a small laptop.

Part On the laptop On the phone
The model gemma4:12b, a 7.6 GB GGUF-style file Gemma 4 E2B (2.6 GB) or E4B (3.7 GB), a .litertlm file — “E” for effective: per-layer embeddings keep the working set near the small number
The runtime Ollama, Docker Model Runner, vLLM LiteRT-LM: one engine, GPU through OpenCL or Metal, NPU builds for Tensor, Snapdragon and Intel chips
The agent Hermes, in a terminal or Hermes Desktop The Gallery app’s agent chat, your own Kotlin or Swift app, or Hermes in Termux
The action model — FunctionGemma, 270M parameters, tuned to turn a sentence into a function call and nothing else

The model file deserves a sentence. Google’s card says the phone builds use “a mixture of 2bit, 4bit and 8 bit weights,” so that for text the weights need as little as 0.8 GB in memory, with the 1.12 GB of embeddings memory-mapped and the vision and audio encoders loaded only when an image or a recording arrives. That is why the peak-memory column in the tables below is smaller than the file, and why an E2B fits beside a running phone.

Google’s numbers, and this laptop’s

Google benchmarks its files at 1,024 prefill and 256 decode tokens with a 2,048-token context, caches warm, and publishes the table. These rows are copied from it, read on 13 September 2026, and are Google’s, not mine.

Gemma 4 E2B, 2.58 GB (Google’s figures)

Device Backend Prefill tok/s Decode tok/s Time to first token Peak memory
Galaxy S26 Ultra CPU 557 47 1.8 s 1.7 GB
Galaxy S26 Ultra GPU 3,808 52 0.3 s 0.7 GB
iPhone 17 Pro CPU 532 25 1.9 s 0.6 GB
iPhone 17 Pro GPU 2,878 56 0.3 s 1.5 GB
macOS, M4 GPU 7,835 160 0.1 s 1.6 GB
Raspberry Pi 5, 16 GB CPU 133 8 7.8 s 1.5 GB

Gemma 4 E4B, 3.65 GB (Google’s figures)

Device Backend Prefill tok/s Decode tok/s Time to first token Peak memory
Galaxy S26 Ultra CPU 195 18 5.3 s 3.3 GB
Galaxy S26 Ultra GPU 1,293 22 0.8 s 0.7 GB
iPhone 17 Pro GPU 1,189 25 0.9 s 3.4 GB
Raspberry Pi 5, 16 GB CPU 51 3 20.5 s 3.1 GB

Now the same runtime, the same files, on the M4 MacBook this site is written on — 24 GB, not freshly booted, one run each — with the CLI’s own benchmark command. Google’s M4 row is in the first table; mine is here.

Measured here, 13 September 2026 (litert-lm benchmark, 0.17.0)

Model Backend Prefill / decode tokens Prefill tok/s Decode tok/s Time to first token
E2B CPU 256 / 256 445 34 0.6 s
E2B GPU 256 / 256 1,387 76 0.2 s
E2B GPU 4,096 / 128 1,641 65 2.5 s
E2B CPU 4,096 / 128 318 27 12.9 s
E4B GPU 256 / 256 483 35 0.6 s
E4B GPU 4,096 / 128 581 30 7.1 s
E4B CPU 4,096 / 128 115 9 35.7 s

And now a third set: this phone. A Galaxy S26 Ultra, the device Google’s own table names, running the AI Edge Gallery app (version 1.0.19, the day of writing), its own Benchmark screen, default settings — 256 prefill and 256 decode tokens, three runs, averaged. Not Google’s 1,024-token methodology; the same 256 this guide’s laptop table already uses, so the phone row and the laptop rows sit on equal footing.

Measured on the S26 Ultra, 14 September 2026 (AI Edge Gallery 1.0.19, Gemma 4 E2B, 256/256 tokens, avg of 3 runs)

Backend Prefill tok/s Decode tok/s Time to first token
GPU 1,632 37 0.19 s
CPU 274 26 1.00 s

Four things to take from the three sets together.

  • The phone did not outrun the laptop; it matched it, on GPU, and trailed it on CPU. Google’s table claims 3,808 prefill tokens a second for this exact phone on GPU — more than double this laptop’s 1,387–1,641. The phone actually measured 1,632: inside the laptop’s own range, not ahead of it. On CPU the phone’s claimed 557 becomes a measured 274, while the laptop’s CPU row (256/256) was 445 — the two-year-old MacBook prefills faster than the flagship phone Google benchmarked. Treat every vendor row in this guide, Google’s and the laptop’s alike, as a ceiling a real device may not reach.
  • The E4B costs three times the E2B for the same prompt. On this laptop, and in Google’s phone rows. The E4B is the better model; whether it is better enough is the question the ten-run comparison from the local-agent guide exists to answer, and on a phone it is worth answering before shipping the bigger file.
  • Read the last column as an agent, not a chat, and use the measured rate, not the claimed one. A chat prompt is a sentence. An agent’s prompt is its system instructions plus every tool schema, re-read each turn. Measured below: a minimal Hermes turn is about 3,000 tokens. At the phone’s measured GPU rate that is 1.8 seconds of reading, not Google’s implied “under a second”; at its measured CPU rate, 11 seconds. The 16,000-token prompt the local-agent guide quoted from Rob Braxman works out to 9.8 seconds on the phone’s GPU and just under a minute on its CPU — every turn, unless the cache holds. Plan from the row you can reproduce, not the row in the marketing table.
  • The claimed number was still the useful starting point. It named the right order of magnitude and the right shape of the trade-off (GPU roughly six times CPU prefill here, and about five times in Google’s own table); it was the absolute figure that did not survive contact with a real S26 Ultra, running Google’s own app.

Tier one: the app

The Google AI Edge Gallery is the reference app and the fastest way to see the numbers on your own hardware. Android 12 and up from the Play Store or the GitHub releases as an APK; iOS 17 and up from the App Store; there is a macOS build too. Version 1.0.19 on the day of writing. It downloads the Gemma 4 files itself, and its features are the list of what an on-device agent can do today:

  • Model Management & Benchmark. Google’s words: “run benchmark tests to understand exactly how each model performs on your specific hardware.” This screen is the first thing to open. It gives you the row of the table above that Google could not.
  • AI Chat, Ask Image, Audio Scribe. Text, a photo from the camera, a recording transcribed and translated — all three of Gemma 4’s input modalities, on the device.
  • Agent Skills. A skill is a SKILL.md — the same idea as the skills in the local-agent guide — whose name and description are appended to the system prompt so the model can invoke it. But a phone has no shell for a skill to run in, so the Gallery gives skills two bodies: JavaScript in a hidden webview, or a native intent (send an email, open a map). Skills load from the community list, a URL, or a file.
  • MCP servers. The app is an MCP client: it fetches a server’s tool schemas, injects them into the prompt, and routes calls with per-invocation permission prompts. Two constraints that the README is candid about: servers must speak StreamableHTTP, not stdio (it wraps the fetch example with supergateway), and the phone needs a publicly routable URL — the README suggests a Cloudflare quick tunnel. That is a phone reaching your laptop over the internet to run a tool. Read that twice before doing it with a tool that touches files.
  • Mobile Actions. Offline device control — flashlight, contact, calendar event, Wi-Fi settings, a map — “powered entirely by a finetune of FunctionGemma 270m.” This is the tier-two model, shipped.

The MCP note is marked experimental and the recommended model for it is the E4B, “for better model quality.” Everything in the Gallery runs on the device; the MCP servers are the one place data leaves it, and the app makes you approve each call.

Tier two: the action model

The thing a phone needs that a laptop did not is a model that answers in under a second and never writes prose. FunctionGemma is Google’s version: “a specialized version of our Gemma 3 270M model tuned for function calling,” 32K context, open weights under the Gemma terms, meant “to be fine-tuned for your specific function-calling task.” Out of the box it scores 58 percent on Google’s Mobile Actions set; the published fine-tune scores 85. On a Pixel 7 Pro — a 2022 phone — the fine-tune prefills at 1,916 tokens a second and generates 142, on the CPU.

Its output is not an answer. It is a call:

<start_function_call>call:create_calendar_event{title:<escape>Dentist<escape>,…}<end_function_call>

The Gallery’s Function_Calling_Guide.md shows how thin the app around it is. A tool is a Kotlin method on a class implementing ToolSet:

class MobileActionsTools(val onFunctionCalled: (Action) -> Unit) : ToolSet {
  @Tool(description = "Turns the flashlight on")
  fun turnOnFlashlight(): Map<String, String> {
    onFunctionCalled(FlashlightOnAction())
    return mapOf("result" to "success")
  }
}

The description is what the model reads; the return value is what it hears back; the app performs the action with an Intent or a system service. Adding an action is three edits and ./gradlew installDebug.

The architecture this suggests is the one the laptop never needed: a 270 MB router that turns speech or text into a structured call, in front of a 2.6 GB model that only gets woken for the calls that need reasoning, in front of — if the data may leave — a hosted model for the hard ones. The local-agent guide called the last two steps a pipeline. On a phone the first step is the one that makes the pipeline feel instant.

Tier three: the API, on the laptop first

Everything Google ships for the phone runs on the laptop from the same file, so the developer path starts where the measurements can be taken.

uv tool install litert-lm                     # 0.17.0; Python 3.10+
litert-lm import --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm gemma-4-E2B-it.litertlm
litert-lm run gemma-4-E2B-it.litertlm --backend gpu --prompt "In one sentence, what is a bank statement?"
litert-lm benchmark gemma-4-E2B-it.litertlm --backend gpu -p 4096 -d 128

The import took 53 seconds for 2.4 GiB. The first run compiles the model for the backend and keeps the result on disk beside it — the model’s folder doubles to 5 GB — so the second start is fast. A prompt on GPU answered in 1.8 seconds wall, engine start included.

Tools, from the CLI, are a Python file:

system_instruction = ("You are a careful bookkeeping assistant. Use the tools "
                      "to answer; never invent a figure.")

def list_unclassified(month: str) -> list[dict]:
    """Lists the outflows in a month that no rule has classified yet.

    Args:
        month: The month as YYYY-MM, for example 2026-08.
    """
    ...

def total_for(description: str) -> float:
    """Adds up every outflow whose description contains the given text."""
    ...
litert-lm run gemma-4-E2B-it.litertlm --backend gpu --preset owl_tools.py \
  --prompt "Which outflows in 2026-08 are unclassified, and what did the cafe cost in total?"

Every top-level function becomes a tool, declared from its docstring and type hints. The E2B, on GPU, called list_unclassified for the month, then total_for with the cafe’s exact description, and answered with the four rows and the total — 3.9 seconds wall, two tool calls, engine start included. On the CPU it made the same two calls in 7.6 seconds and then listed three of the four rows in its summary, dropping one that its own tool result contained. The E4B on GPU took 9.6 seconds and got all four. Read the run note for the transcripts. The lesson is the one the local-agent guide ended on, smaller: the 2B model can drive tools, and its summary of what the tools returned is the part to show beside the raw result, not instead of it.

Tier three, continued: the Kotlin app

The example is a one-screen Android app — a question, an Ask button, the answer, the time it took — with the same two tools. The whole runtime is one dependency:

implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0")

The tools are the Python preset in Kotlin:

class StatementTools : ToolSet {
    @Tool(description = "Adds up every outflow whose description contains the given text.")
    fun totalFor(
        @ToolParam(description = "Text to match, case-insensitively, such as CAFE.") description: String,
    ): Double = rows.filter { it.second.contains(description, ignoreCase = true) }.sumOf { it.third }
}

And the engine, the conversation and a turn:

val engine = Engine(EngineConfig(modelPath = "/data/local/tmp/gemma-4-E2B-it.litertlm",
                                 backend = Backend.GPU()))
engine.initialize()                       // seconds; not on the UI thread
val conversation = engine.createConversation(ConversationConfig(
    systemInstruction = Contents.of("You are a careful bookkeeping assistant…"),
    tools = listOf(tool(StatementTools())),
))
val answer = conversation.sendMessage(question)   // tools are called for you

automaticToolCalling = false in the config hands the calls back to the app instead, which is what you want when a call must be approved or must touch a permission. The GPU backend needs two uses-native-library lines in the manifest, which the example has.

It built here — a 55 MB debug APK — after one trap: the 0.17.0 library is compiled with Kotlin 2.4 metadata, and a project on the Kotlin 2.2 plugin fails with “can read versions up to 2.3.0.” Move the plugin to 2.4.0. The Gallery itself still pins 0.11.0 with Kotlin 2.2, which is how the trap stays hidden.

A second trap, found only by running it: adb push to /data/local/tmp/ puts the model where the app can read it, but the GPU backend also writes a weight-cache file next to the model on first load, and /data/local/tmp refused that write — delegate_opencl.cc:340] Failed to create litert::ml_drift::DelegateKernelLiteRt: INTERNAL: Could not open file (…): Permission denied, surfacing in the app as a bare “Failed to create engine.” The model has to live somewhere the app owns: pushing it to /sdcard/Android/data/<package>/files/ instead — the app’s own external-files directory, writable without root — fixed it outright. The CLI’s own --cache disk never hits this, because /data/local/tmp is shell’s, not the CLI’s; an installed app has no such privilege over a world-writable directory.

With the model in a directory the app can write to, on an S26 Ultra: the GPU backend reported “Ready on GPU in 7,091 ms,” then answered the example’s two-tool question — both rows, the $13.50 total, both tools called — in 3,491 ms. Rebuilt for CPU, it read “Ready on CPU in 3,434 ms,” faster to start than GPU, whose first load compiles an OpenCL kernel graph; but the CPU turn took 7,407 ms, twice the GPU turn, and its summary dropped one of the four unclassified rows from the answer while the correct total still printed — the exact CPU-only omission the local-agent guide found on the laptop, reproduced here on the phone with the same model. The lesson carries over unchanged: GPU costs more to start and less per turn; CPU starts faster and its prose summary is the part to double-check against the tool’s own result, not the number in it.

The same shape exists in Swift (EngineConfig(modelPath:, backend: .gpu), try await engine.initialize(), Metal underneath, a Tool protocol) and in JavaScript for the browser (@litert-lm/core, WebGPU, a -web.litertlm file, text only, “early preview”). Neither was run for this guide.

The server, and Hermes

The CLI has a fourth command, and it is the bridge back to the last guide:

litert-lm serve            # OpenAI-compatible: /v1/models, /v1/chat/completions, /v1/embeddings, port 9379

A chat request with a tools array came back with finish_reason: "tool_calls" and a well-formed call in two seconds. So anything that speaks OpenAI can sit in front of this runtime — including Hermes. A profile pointed at the port, with the model card’s 32K context declared:

hermes profile create pocket --no-skills
hermes -p pocket config set model.provider custom
hermes -p pocket config set model.base_url http://localhost:9379/v1
hermes -p pocket config set model.default gemma-4-E2B-it.litertlm
hermes -p pocket config set model.context_length 32000
hermes -p pocket chat -Q -t terminal -q "Run the shell command 'date' and tell me the year it printed."

Hermes refused: “a context window of 32,000 tokens, which is below the minimum 64,000 required by Hermes Agent.” The local-agent guide called this the rule documented on both sides and mentioned by neither quick start; here it lands on a model whose card says 32K, full stop. The error text itself says the declared length may be raised only “if your server reports a window smaller than the model’s true window.” Declared as 64,000 anyway, to see what happens: the loop ran, three requests, 25 seconds wall, “The year printed is 2026.”

A logging proxy between the two measured what Hermes actually sent, with one toolset enabled and no skills:

Request Messages Tools Prompt tokens
Session title 2 0 273
First turn 2 4 2,956
After the tool result 4 4 3,019

Three thousand tokens is the floor of a Hermes turn on this model, and it fitted inside the server’s cache — which the verbose log showed to be 4,096 tokens, on the CPU, with nothing on the serve command line to change either. So the honest statement is narrow: a Hermes tool loop ran through LiteRT-LM on a laptop, for a short session, against a declared context the model does not have. A full toolset would not fit in that cache at all until it is raised, and the day a session grows past 32K the declaration comes due.

And the phone? Hermes lists “Android (Termux)” as a supported platform, with a page that says what works there (the CLI, cron, the terminal tool, MCP, memory) and what does not (voice, Docker isolation, the browser; and Android “may still suspend Termux background jobs”). The runtime’s Python package publishes android_23_arm64_v8a wheels, so litert-lm serve in Termux is plausible, with Hermes on the same phone pointed at 127.0.0.1:9379. Nobody has shown that working here. It is the first experiment for the phone when it arrives, and it will meet the 64K rule on the way in.

TipWhat this means for choosing
  • For a phone app that acts — a form filled from speech, a note classified, a device controlled — start with FunctionGemma or the E2B with tools, and measure the first turn on the benchmark screen yourself. A couple of seconds on a current flagship’s GPU, by what this guide measured, not the “under a second” the claimed rate implies; five to ten on its CPU. Fine-tune the 270M model if the actions are fixed; it is what it is for.
  • For summarising, transcribing, or answering about a document on the device, the E2B is the honest default and the E4B the upgrade — three times the reading cost for a better answer, decided by a ten-run comparison on your own inputs, not by the benchmark.
  • For an agent loop with a full toolset, the phone is a first pass, not the loop. Three thousand tokens a turn is the floor; sixteen thousand is normal; the cache on the phone’s server is four thousand until you raise it. Let the phone read the private thing, classify, and prepare; let a machine with a 64K window run the loop; let a person approve.
  • Do not point the Gallery’s MCP client at your laptop over a public tunnel for a tool that touches files. The README is clear about what it needs; be clear about what it exposes.

What to do this week

  • Install the Gallery on the phone you have and open Benchmark before Chat. Its numbers for your hardware replace every table in this guide.
  • Run the CLI on your laptop with the same file — import, run, benchmark — and compare your row with Google’s for the same chip. The gap tells you how to read the phone rows.
  • Give it two tools from a preset and a question that needs both. Then check its summary against the raw tool result, on GPU and on CPU. That is the whole trust question for a 2B model, in a minute.
  • If the app fails to load a model you pushed to /data/local/tmp, that is the GPU cache-write permission, not a broken model: move it to /sdcard/Android/data/<package>/files/ instead. This guide’s own numbers (below) came from that fix.
  • Leave Hermes on the laptop for now. Let the phone be the first pass and the laptop the loop, and revisit when the phone runtime’s server takes a context flag.

Sources

Research notes — what was read in full, what was measured, and the phone run — are in the accompanying folder, with the CLI preset and the Android project in its example directory. The phone measurements — the Gallery benchmark, the Kotlin example’s engine-start and turn times, and the storage-permission fix — are in notes/phone-run.md.