Running AI on glasses is a different problem from running it in the data center. Peak memory, battery, heat and a sub-500 ms response budget change what “good” means across data, models and inference, and UX is part of the design, not an afterthought.

The work in this space has exploded over the last two years. To make it easier to find your way in, we mapped the key research directions as a set of gears, with a companion table that links every technique to its source and calls out open problems.

The Edge AI for Wearables gearbox: interlocking gears for models, data, compression and inference, each labelled with its knobs
Click the image to see in full screen
Level 1Level 2Level 3Knob / techniqueWhat it isOpen problems
MODELSDense SLMs0.1–1B paramsThe size band that fits on the wearable itself: at 4-bit, a 1B model needs about 0.5 GB of weights, which fits glasses-class memory. 2–4B models (e.g., Qwen3-4B) run only on the companion phone.
  • MoE prefill: multi-token prompts activate most experts, erasing MoE's advantage on time to first token.
  • Low-bit reasoning: small reasoning models degrade sharply below 4 bits; recipes that keep chain-of-thought intact at 2–3 bits are missing.
  • Temporal memory: all-day egocentric streams need bounded state that can still recall an event from minutes ago, within ~1–2 W.
  • Visual evidence: token pruning and low resolution can drop exactly the detail (text, small objects) a task needs.
  • Honest benchmarks: static QA and warm tokens/s omit capture, speech, missed events and long-session power.
Deep & thinBelow ~1B parameters, more layers at narrower width beat wide-and-shallow at equal size (MobileLLM: +2.7 / +4.3 points at 125M / 350M), at some cost in serial latency.
Tied embedsThe input embedding (token → vector) and the output projection (final vector → a score for each vocabulary word) share one matrix. Embeddings are a large share of a small model's weights, so tying frees memory for more layers.
GQAGrouped-query attention: groups of query heads share one key/value head, shrinking the KV cache 4–8× with close to full-attention quality. Standard in current small models.
Weight sharingAdjacent layers reuse one transformer block's weights (MobileLLM-LS runs each block twice): more depth without more storage, and little added latency because the weights are already in cache.
MatFormerA nested model (Gemma 3n): a smaller sub-model lives inside the larger one, so one download serves several sizes; switching between them at run time, by device load, is described as future work.
PLE offloadPer-layer embeddings (PLE): Gemma 3n and Gemma 4 keep these in CPU memory instead of accelerator memory, so a model with ~5B raw parameters runs with a ~2B-class footprint (E2B), which is still above most glasses budgets today.
Early exitStop at an intermediate layer when the prediction is already confident (LayerSkip), saving compute on easy tokens; the early layers can double as a self-speculative drafter.
MoE (mixture of experts)# expertsHow many expert feed-forward blocks a layer holds. Each token runs only its top-k experts, so adding experts adds capacity without adding per-token compute; all of them must still be stored; MobileMoE finds moderate sparsity best on-device.
Top-k routingEach token is sent to its k highest-scoring experts (e.g., top-2, top-4). Larger k improves quality but raises compute and the number of experts that must be loaded.
GranularitySplit experts into many smaller ones (fine-grained experts). MobileMoE uses 60 routed experts with top-4 routing, which gives more diverse routing at equal compute.
Shared expertOne always-on expert processes every token alongside the routed ones; it captures common knowledge and improves quality at fixed compute (MobileMoE, DeepSeek).
Active / totalCompute scales with active parameters, memory with total. MobileMoE: 0.3–0.9B active of 1.3–5.3B total. A 30B-A3B model still needs ~15 GB at 4-bit.
Expert cacheKeep recently or frequently used experts in RAM and stream the rest from flash (EdgeMoE, Mixtral-offloading). The cache hit rate largely sets decode speed.
PrefetchPredict the next layer's experts and load them while the current layer computes (routers placed before attention, next-layer gate speculation), hiding flash I/O behind compute.
Mixed-bitStore less important experts at lower precision, 2–4 bits by importance (EdgeMoE, HOBBIT); shrinks memory and flash traffic with limited quality loss.
Prune / mergeDelete rarely useful experts or merge similar ones (NAEE, MC-SMoE), or slim every expert and distill (SlimMoE: 41.9B → 7.6B total).
Hybrid & recurrentConv + attnAbout two-thirds to three-quarters of layers are gated short convolutions, the rest attention (LFM2-1.2B: 10 of 16; LFM2-2.6B: 22 of 30). LFM2 reports up to 2× faster CPU prefill and decode than similar-size transformers.
SSM (Mamba-2)State-space model (SSM): recurrent layers (e.g., Mamba-2) compress the past into a fixed-size state instead of a growing KV cache, so memory stays flat with sequence length (Granite 4.0-H mixes them 9:1 with attention).
Attn+SSM headsAttention and SSM heads run in parallel inside each layer (Hymba): an 11.67× smaller cache and 3.49× the throughput of Llama-3.2-3B, with higher accuracy.
Gated DeltaNetA gated linear-attention layer with delta-rule state updates; Qwen3.5 small models interleave three of these with each full-attention block to bound long-context cost.
RWKV-7Receptance Weighted Key Value (RWKV): a fully recurrent RNN-style language model with constant memory and time per token and no KV cache; competitive up to ~3B parameters.
Fixed-size stateThe key edge property of recurrent layers: memory does not grow with context. State size and precision still matter, and recall of distant details can suffer.
Few attn layersKeep only a handful of full-attention layers for precise retrieval and make everything else cheap. Those layers still need a KV cache that grows with context.
VLM (vision-language)Encoder costThe vision encoder often dominates time to first token. Efficient encoders such as FastViTHD (FastVLM) and MobileNet-V5 (Gemma 3n) cut encoding time sharply.
Visual tokensEvery image token must be prefilled by the language model, so token count drives latency: from 1 (LLaVA-Mini) to 144 (MobileVLM v2) to thousands at high resolution.
ResolutionHigher resolution helps OCR and small objects but multiplies tokens and encoder cost; tiling one large image can add hundreds of tokens.
ROI cropsEncode a small high-resolution crop of the region that matters (text, hands) plus a coarse view of the scene, instead of the whole frame at full resolution.
ProjectorThe module that maps encoder features into language-model tokens; pooling projectors (LDPv2, pixel shuffle) compress 576 tokens to 144 or fewer.
Token pruningDrop or merge low-importance visual tokens after encoding (FastV, SparseVLM, VisionZip, ToMe), often halving compute; can remove evidence needed for counting or OCR.
Frame rateFor video, process fewer frames or only frames that change. Cuts compute and power, at the risk of missing brief events.
Video KV capStreaming video grows the KV cache without bound; cap it with attention sinks plus a window (StreamingVLM) or redundancy removal (InfiniPot-V: up to −94% memory).
VRM (visual reasoning)Think / no-thinkHybrid models (Qwen3, MiniCPM-V 4.5, Phi-4-reasoning-vision) switch between answering directly and chain-of-thought, skipping reasoning on plain perception tasks.
CoT budgetCap the number of thinking tokens (budget forcing, as in s1) so reasoning fits the latency and energy budget.
Adaptive lengthTrain the model to reason only as long as needed (AdaptThink: −53% length at 1.5B) or stop once confident (DEER: 19–80% shorter chains of thought).
Re-look imageInstead of long textual reasoning, look again at a higher-resolution view only when needed (VisionThink starts at 1/4 resolution).
RL (GRPO)Group Relative Policy Optimization (GRPO): reinforcement learning with rule-based rewards (VLM-R1 with GRPO; LMM-R1 at 3B with PPO) builds visual reasoning in small models more robustly than supervised fine-tuning.
Short-CoT KDDistill a large reasoner's traces into short, correct chains for a small student, keeping the accuracy gain while cutting output tokens.
Cloud offloadDeep visual reasoning rarely fits a wearable: 1,000 tokens at 20 tok/s ≈ 50 s. Send it to the companion or the cloud and keep a short-answer path on the wearable.
Open problems · Models
  • MoE prefill: multi-token prompts activate most experts, erasing MoE's advantage on time to first token.
  • Low-bit reasoning: small reasoning models degrade sharply below 4 bits; recipes that keep chain-of-thought intact at 2–3 bits are missing.
  • Temporal memory: all-day egocentric streams need bounded state that can still recall an event from minutes ago, within ~1–2 W.
  • Visual evidence: token pruning and low resolution can drop exactly the detail (text, small objects) a task needs.
  • Honest benchmarks: static QA and warm tokens/s omit capture, speech, missed events and long-session power.
DATAReal dataCalibration & sync

Each sensor's own parameters, the relative pose between sensors, and their timestamps must line up before video, IMU, audio and gaze can be fused.

  • Commercial gap: the big egocentric datasets are non-commercial, so there is no shared public baseline for shipped glasses.
  • Bystander consent at scale: blurring misses voices, reflections, gaze and heart-rate signals; anonymizing on the device before storage is unsolved.
  • Sensor-faithful generation: generators rarely reproduce rolling shutter, fisheye lenses, low-light noise or IMU drift, or keep video, IMU, audio and gaze consistent.
  • Small models and synthetic data: the right real:synthetic ratio for ≤1B models is unknown, and on-device personalization risks recursive collapse.
  • Compute-aware benchmarks: egocentric benchmarks score accuracy only, never the < 500 ms / ~2 W regime.
Collection protocol

Scripted vs unscripted activity, who wears the device, where, for how long, and with what consent; together these decide how well a model generalizes.

  • Ego4D: 3,670 h from 931 wearers in 74 locations across 9 countries.
  • Ego-Exo4D: 1,286 h, 740 participants, 13 cities, skilled activities of 1–42 min.
  • EgoLife: 300 h from 6 people living together for one week.
  • HD-EPIC: 41 h of unscripted recording in 9 kitchens.
Multimodal capture

Wearables need synchronized video, IMU, spatial audio, gaze, hand and body pose and physiological signals, ideally with precise ground truth.

  • Nymeria: 300 h, 264 participants, 50 locations, full-body motion ground truth plus Aria video, gaze and IMU.
  • Aria Gen 2 pilot: 800 Hz IMUs, 8-channel audio, 128 Hz PPG, on-device VIO, hand tracking and gaze.
  • Ground truth: Aria Digital Twin has 200 sequences with 398 objects; Aria Everyday Activities has 143 sequences with SLAM trajectories, gaze and transcripts.
Annotation

Timestamped narrations and dense action, audio and QA labels are the main cost of real data; LLMs can densify cheaper weak labels.

  • HD-EPIC: 263 annotations per minute of video (59K actions, 51K audio events).
  • EPIC-KITCHENS-100: 100 h, 700 videos, 90K actions.
  • Ego-Exo4D adds expert commentary from coaches and teachers.
  • LLM-assisted: LaViLa, using half of Ego4D's human narrations, beat baselines (+10.1% EGTEA, +5.9% EK-100 retrieval).
Privacy & consent

Glasses record people who never agreed to it; faces and licence plates must be anonymized, and biometric laws apply to wearers and bystanders.

  • EgoBlur: Aria's face and licence-plate blurring, evaluated for fairness across demographics.
  • Ego4D: consenting participants plus de-identification.
  • Law: Illinois BIPA, GDPR; gaze and heart-rate (PPG) data are biometric-adjacent.
  • Gap: blurring does not cover voices, reflections or physiological signals.
Federated

Real user data stays on the device; training, evaluation and analytics run federated, ideally with differential privacy.

Domain shift

Models degrade across devices (sensor, sampling rate, placement), users (gait, height, handedness), environments and time; personalization data closes part of the gap.

  • EPIC-KITCHENS-100 includes an unsupervised domain-adaptation challenge and a 'test of time' split recorded two years later.
  • IMU activity recognition shifts with sensor placement, sampling rate and the wearer's movement style.
  • Factory calibration drifts as glasses deform, heat up and age.
Synthetic dataWorld models

Video diffusion and world models generate controllable scenes, rare events and 'what if' variations (lighting, objects, weather).

  • NVIDIA Cosmos: open platform with video curation, tokenizers, pre-trained world models and post-training recipes.
  • Genie 3: interactive worlds at 720p / 24 fps that stay consistent for a few minutes.
  • Egocentric: EgoControl (video driven by 3D body pose), EgoLCD (long-context egocentric video).
LoRA generators

Fine-tune a diffusion model with LoRA on a few real images so its output matches the device's camera and classes.

Scenario gen.

An LLM or program writes scenario descriptions, including edge cases, which are compiled into simulator code.

  • ChatScene (driving, CARLA): 15% more collisions found than baselines; fine-tuning on them cut collisions by 9%.
  • Transfer to egocentric scenes (falls, hazards, social situations) is still an extrapolation.
3D sim & render

Physically rendered scenes give free, pixel-accurate labels (depth, segmentation, 6-DoF pose); randomizing layouts and assets helps transfer to real data.

  • Aria Synthetic Environments: 100K indoor scenes, ~8K 3D objects, simulated Aria camera and lens.
  • ProcTHOR: procedurally generated houses for embodied AI.
  • EgoGen: virtual humans moving by their own egocentric view; localization recall 66.9% → 76.7%, EgoBody pose error −20.7%.
Sensor sim.

Turn other modalities into IMU, audio or raw-camera signals that match wearable hardware.

LLM distillation

A large teacher model writes pre-training, instruction or multimodal QA data for small models.

  • phi-1 (1.3B): 6B filtered web tokens plus 1B GPT-3.5 synthetic tokens, 50.6% on HumanEval.
  • Cosmopedia: 25B tokens generated by Mixtral-8x7B, used to train the Cosmo-1B small model.
  • LLaVA: 158K GPT-4-generated multimodal instruction samples (conversations, descriptions, reasoning).
Augmentation

Cheap, label-preserving transforms of real signals; still the first thing to try.

  • IMU: rotation, permutation and time-warp raised accuracy from 77.5% to 86.9% on Parkinson's data; jitter and scaling did not help.
  • Images and video: crops, colour, blur and mixup; audio: noise and reverberation.
Private synthesis

Differentially private synthetic data, and replacing bystanders in always-on footage.

Curation & evalFilter & dedup

Remove duplicates, low-quality and unsafe samples, and verify synthetic samples before use; small models cannot absorb noisy data.

  • Verification filters for generated data (e.g., IMUGPT's motion filter) catch physically implausible samples.
  • Deduplicate across real and synthetic sources to avoid leaking eval data into training.
Data selection

Quality scoring and mixture design decide what a small model sees; data quality can substitute for data volume.

  • MobileLLM-R1: ~2T high-quality tokens were enough; 4.2T pre-training tokens via resampling (~11.7% of Qwen3's 36T); the 950M model scores 15.5 on AIME.
Real : synth mix

The ratio of real to synthetic data matters; synthetic data should add to real data, not replace it.

  • Model collapse: training recursively on generated data erodes the tails of the distribution (Shumailov et al., Nature 2024).
  • Accumulating synthetic data alongside real data avoids collapse (Gerstgrasser et al., 2024).
  • The right ratio for models of 1B parameters or less is unknown.
Long-tail coverage

Everyday egocentric data is dominated by a few common actions; rare but important events (falls, hazards, unusual objects) are sparse.

  • Report tail-class and unseen-participant results separately (as EPIC-KITCHENS-100 does).
  • Target synthetic generation and extra collection at the tail.
Egocentric evals

Held-out real benchmarks for long-horizon memory and question answering, which is what an assistant on glasses needs.

  • EgoSchema: 5,000+ questions over 3-minute clips; humans ~76%, models under 33% at release.
  • HourVideo: 12,976 questions over 20–120 min videos; experts 85.0%, Gemini 1.5 Pro 37.3%.
  • HD-EPIC VQA: 26K questions; Gemini Pro 38.5%. EgoLifeQA: memory recall and habit questions.
Compute-aware eval

None of the egocentric benchmarks measure latency or energy; wearables need task success within the budget.

  • Report accuracy together with time to first response (< 500 ms), energy per task and behaviour under thermal throttling.
  • Measure on the actual device, including capture, speech and preprocessing.
Provenance

Track where each sample came from and what its licence allows; most egocentric corpora are research-only.

Open problems · Data
  • Commercial gap: the big egocentric datasets are non-commercial, so there is no shared public baseline for shipped glasses.
  • Bystander consent at scale: blurring misses voices, reflections, gaze and heart-rate signals; anonymizing on the device before storage is unsolved.
  • Sensor-faithful generation: generators rarely reproduce rolling shutter, fisheye lenses, low-light noise or IMU drift, or keep video, IMU, audio and gaze consistent.
  • Small models and synthetic data: the right real:synthetic ratio for ≤1B models is unknown, and on-device personalization risks recursive collapse.
  • Compute-aware benchmarks: egocentric benchmarks score accuracy only, never the < 500 ms / ~2 W regime.
COMPRESSIONQuantizationFP1616-bit reference precision: 2 bytes per weight (a 2B model is 4 GB). Keep it for sensitive parts or as the baseline.
  • Portable low-bit execution: 2-bit and ternary storage often lacks equally efficient arithmetic and kernels on mobile NPUs.
  • Kernel fragmentation: group-wise scales, 2:4 sparsity and rotations speed up GPUs but often not mobile or wearable NPUs.
  • Well-trained small models: the harm from post-training quantization grows with training data, so the smallest, best-trained models are the hardest to compress.
  • Compression order: distill → prune → QAT → adapters is still largely empirical; changing one step invalidates the others.
  • Adapters on quantized bases: keeping many adapters accurate after merging and quantization, and invalidating cached KV when weights change.
INT88-bit weights and/or activations (LLM.int8(), SmoothQuant W8A8): close to lossless, half the memory, and a good match for integer NPUs.
INT4 · default4-bit weights (GPTQ, AWQ, or QAT as in Gemma 3 / Llama 3.2): about 4× smaller than FP16 with small loss. The safe post-training default.
3-bitPost-training quantization starts to hurt (about −4 points on average for 3-bit AWQ on Llama-3-8B); small and reasoning models degrade most (R1-Distill-1.5B: −18.6 points with 3-bit AWQ).
2-bit + QATPost-training 2-bit collapses; with QAT or vector quantization plus fine-tuning (AQLM, PV-Tuning, ParetoQ) it becomes Pareto-optimal. Apple's on-device model uses 2-bit QAT.
1.58-bitTernary weights (−1, 0, +1) trained natively (BitNet b1.58 2B4T): 0.4 GB of non-embedding weights and 0.028 J per token, but it needs custom kernels.
Mixed-bitGive sensitive layers or channels more bits and the rest fewer (SqueezeLLM, SliM-LLM); nested formats (MatQuant) serve int8, int4 and int2 from one model.
W4A8 / W4A16Weight vs activation bits. W4A16 keeps activations at 16-bit (simple, accurate); W4A8 uses integer activations for faster NPU math (QServe), at some accuracy risk.
Group sizeOne scale per group of 32–128 weights preserves accuracy, but some NPUs support only per-channel scales (Hexagon: Llama-3.2-1B scores 2.1% vs 15.9% on MATH500).
RotationsMultiply by rotation matrices (QuaRot, SpinQuant) to spread activation outliers, enabling 4-bit weights, activations and KV cache; the runtime must support or fuse them.
Pruning & sparsityStructuredRemove whole channels, heads or blocks, then retrain; the result is a smaller dense model that standard kernels accelerate (LLM-Pruner, SliceGPT).
UnstructuredZero individual weights (SparseGPT: 50–60% in one shot; Wanda needs no weight updates); saves little on devices unless the kernel actually skips the zeros.
N:M (2:4)At most 2 non-zeros in each group of 4 weights; fast on NVIDIA sparse tensor cores, but support on mobile NPUs is limited.
BlockZero whole tiles (e.g., 16×16) so kernels can skip them; the speedup depends on block size, sparsity level and device support.
Channel / headRemove feed-forward channels or attention heads and physically shrink the tensors; keep dimensions aligned to the accelerator's tile sizes.
Layer dropRemove whole layers ranked by low importance (ShortGPT); simple and stacks with quantization, but deep cuts need recovery training.
Width / depthPrune hidden width and depth together, then distill from the original (Minitron: 40× fewer training tokens than training from scratch).
Activation sp.Activation sparsity: skip neurons whose activations are near zero for the current input (Deja Vu, TEAL, ProSparse at ~89%); pays off with CPU/NPU co-design or flash offload (PowerInfer-2).
Fine-tuning (PEFT)LoRALow-Rank Adaptation: freeze the base model and learn a low-rank update B·A (rank 8–64, under 1% of the weights); merge it into the weights or keep it as a swappable adapter.
QLoRA (NF4)Quantized LoRA: LoRA on a 4-bit NormalFloat (NF4) base with double quantization and paged optimizers. A training-side technique: adapters are built off-device, then shipped to the wearable. NF4 is a training format, not an NPU format.
DoRAWeight-Decomposed Low-Rank Adaptation: splits each weight into magnitude and direction and applies LoRA only to the direction; consistently beats LoRA and merges with no inference cost.
RoRA (α/√r)Reliability Optimization for Rank Adaptation: changes LoRA's scaling from α/r to α/√r so accuracy keeps rising with rank; notable gains on pruned models (+5.7% over LoRA on Sheared-LLaMA-1.3B).
QA-LoRAQuantization-Aware LoRA: a LoRA variant whose update merges cleanly into an INT4 model, avoiding a precision mismatch after merging.
LoftQLoRA-Fine-Tuning-aware Quantization: initializes LoRA jointly with quantization so the adapter starts by correcting quantization error; strongest at 2-bit and mixed 2/4-bit.
Rank rThe adapter's size and capacity; higher rank adds parameters and memory. With standard scaling, gains can plateau as rank grows (the motivation for RoRA).
Adapter swapKeep one quantized base model and load small task adapters on demand (Apple: rank-16 adapters of tens of MB) instead of storing several full models.
DistillationLogit KDTrain the student to match the teacher's output probability distribution, which carries more signal than hard labels.
Feature KDAlso match intermediate hidden states or attention maps; useful after pruning, when student and teacher share structure.
Teacher outputsFine-tune on answers generated by a larger model (sequence-level distillation); simple and effective for narrow tasks.
Short CoTDistill concise reasoning traces so the student reasons in fewer tokens, directly cutting decode time and energy.
Task narrowingTrain a small specialist for the device's actual tasks (intents, captions, OCR) rather than a generalist; often a bigger win than another bit of quantization.
System 2 → 1Distill answers that needed chain-of-thought into direct answers (Meta, 2024), baking repeated reasoning into the weights; works for many tasks, not multi-step math.
Prune + KDPrune the model, then distill from the unpruned original to recover quality (Minitron, SlimMoE); Sheared-LLaMA recovers with continued pre-training instead.
Open problems · Compression
  • Portable low-bit execution: 2-bit and ternary storage often lacks equally efficient arithmetic and kernels on mobile NPUs.
  • Kernel fragmentation: group-wise scales, 2:4 sparsity and rotations speed up GPUs but often not mobile or wearable NPUs.
  • Well-trained small models: the harm from post-training quantization grows with training data, so the smallest, best-trained models are the hardest to compress.
  • Compression order: distill → prune → QAT → adapters is still largely empirical; changing one step invalidates the others.
  • Adapters on quantized bases: keeping many adapters accurate after merging and quantization, and invalidating cached KV when weights change.
INFERENCEAttentionMHAMulti-head attention: every query head has its own key/value head. Best quality, largest KV cache.
  • Router calibration: query-value estimators trained on chat data must handle egocentric, multimodal intents under battery, thermal and network shifts.
  • Speculation energy: speedups can raise energy per token when acceptance drops; wearable-scale evidence is thin.
  • KV eviction vs recall: bounded caches forget, and there are no guarantees for recalling earlier scenes.
  • Wearable–companion–cloud split: radio start-up, queueing and disconnects dominate tail latency; per-token cross-device protocols are fragile.
  • No glasses-class measurements: tokens/s, joules per token and thermal curves for AR1-class chips remain unpublished.
MQAMulti-query attention: all query heads share a single key/value head. Smallest cache, some quality loss.
GQAGrouped-query attention: groups of query heads share key/value heads (e.g., 32 → 8, a 4× smaller cache); existing models can be uptrained with ~5% of pretraining compute.
MLAMulti-head latent attention (DeepSeek): cache one small compressed vector per token (−93% KV); existing models can be converted (MHA2MLA, TransMLA).
Local : globalInterleave sliding-window layers with occasional full-attention layers (Gemma 3: 5:1 with a 1,024-token window), cutting KV memory overhead from ~60% to under 15%.
Window sizeHow many recent tokens a local layer sees (e.g., 512–4,096). Smaller windows save memory and compute; long-range recall then relies on the global layers.
Sparse (NSA)Native sparse attention: each token attends to a compressed summary, a few selected blocks and a local window; the model is trained sparse from the start.
Block (MoBA)Mixture of block attention: a gate picks the top-k context blocks for each query, like MoE routing over the context. Saves compute, not KV storage.
Skip attnRemove attention from some layers entirely (dropping half of Llama-2-70B's: 48% faster for −2.4%, shown at server scale); MobileLLM-Flash searches for where skipping is safe.
KV cache8-bit KVStore keys and values in 8-bit: half the KV memory with negligible loss. The shipped default in Apple's on-device model; the Gemma 4 report sizes memory with an int8 KV cache.
4-bit KVAnother 2× saving; near-lossless even for reasoning models in reported tests, with extra packing and dequantization overhead.
2-bit KVAggressive (KIVI: keys per channel, values per token): 2.6× less peak memory, but more sensitive on retrieval-heavy tasks.
Sinks + windowKeep the first few 'attention sink' tokens plus a recent window (StreamingLLM): bounded memory for always-on streams, but older details are dropped.
EvictionKeep only important tokens (H2O heavy hitters, SnapKV, PyramidKV at 12% of the cache) and drop the rest; selection overhead and changing queries can hurt recall.
Prefix reuseCache the KV of recurring prompts (system prompt, agent context) and reload it; one study restored a context in 577 ms instead of 15.7 s.
KV sharingAdjacent layers reuse one KV cache (CLA, Gemma 3n and Gemma 4, Apple's on-device model: −37.5% KV); must be trained in, not bolted on.
Flash tieringKeep hot KV in RAM and move cold KV to flash or a low-rank form (TierKV: up to 17.6× faster prefill; phone-measured).
Context ≤ 4KShipped assistants on phones and browsers cap context near 4K tokens (Apple on-device model: 4,096; Gemini Nano in Chrome: 4,096 per session); wearable budgets are tighter. A 128K advertised context is a ceiling, not a default.
Speculative decodingDraft modelA small model proposes several tokens and the target verifies them in one pass; needs a second model and its KV cache resident in memory.
Self-specDraft with the target's own early layers and verify with the rest (LayerSkip, Draft & Verify), so no second model is needed in memory.
MTP headsMulti-token prediction (MTP): extra heads predict several future tokens at once; Gemini Nano v3 on Pixel is 50%+ faster than with a separate drafter and saves 130 MB (phone-measured).
EAGLE headsExtrapolation Algorithm for Greater Language-model Efficiency (EAGLE): a light head drafts from the target's hidden features (EAGLE-2 and EAGLE-3: 3–6.5×, GPU-measured; gains on phones are smaller); a strong drafter with a small memory footprint.
Prompt lookupDraft by copying n-grams from the prompt or context; free, and effective for summarizing, editing and grounded answers.
Token treesVerify a tree of candidate continuations in one pass instead of a single chain (EdgeLLM, EAGLE-2), raising the tokens accepted per step.
Draft length kTokens proposed per round; longer drafts pay off only when acceptance is high and verification cost grows slowly with k.
Accept rateThe fraction of drafted tokens the target accepts: the main driver of speedup, and of whether energy per token goes up or down.
Cross-deviceDraft on the wearable and verify on the companion or in the cloud (−35% latency in one study); every round adds radio and queueing delay.
RoutingQuery valueEstimate how much a bigger model would improve this request, weighed against latency, energy and privacy; learned routers (RouteLLM) cut cost by over 85% on MT-Bench while keeping 95% of GPT-4's quality.
Difficulty est.Predict up front whether the small model will do well enough (Hybrid LLM: 40% fewer large-model calls with no quality drop).
CascadesTry the small model first and escalate if a check fails (FrugalGPT, AutoMix): cheap on easy queries, but hard queries pay for both models.
Token-levelSend only the critical tokens to the large model (R2R: a 1.5B + 32B pair is 2.8× faster than the 32B model alone at similar accuracy; the 32B model sits off-device).
Think budgetDecide how much reasoning a query deserves (thinking vs non-thinking modes, token budgets) as part of the same routing decision.
Confidence gateEscalate or stop based on the model's confidence; thresholds need calibrating on the deployed model, since quantization and distillation change its confidence.
Where to runChoose among the wearable, its companion phone and the cloud, weighing network latency, availability and privacy; keep an offline path for time-critical actions.
Battery-awareFactor battery level and temperature into the decision; sustained load can cut throughput ~40% (iPhone 16 Pro: 40.5 → 23.7 tok/s, phone-measured; glasses throttle sooner).
Input & systemNPU placementRun supported operators on the NPU, DSP or GPU and keep large graph segments on one processor to avoid transfers and synchronization.
No CPU fallbackAn unsupported operator forces a fallback to the CPU; fragmented execution can be slower than running everything on the CPU. Build only from supported ops.
Op fusionMerge adjacent operations (e.g., matrix multiply + activation, fused attention) into one kernel, cutting memory traffic and dispatch overhead.
BN foldingFold batch normalization into the preceding convolution's weights and bias; mathematically exact, and removes a separate pass over the tensor.
AOT compileCompile and optimize the graph ahead of time for the target runtime (ExecuTorch, QNN, Core ML), reducing startup and dispatch work.
Buffer reusePreallocate and reuse memory buffers instead of allocating on every inference; lowers peak memory and allocator overhead.
Fewer copiesRemove redundant copies and layout conversions between camera, preprocessing and model; these can cost more than the model itself.
TilingSplit computation into tiles that fit in on-chip memory, reducing slow and energy-hungry external-memory traffic.
Shorter contextFewer prompt tokens mean faster prefill and a smaller KV cache; at 0.6 ms per prompt token, 1K tokens alone cost ~600 ms.
Fewer callsRun the model less often (event triggers, batching, lower duty cycle). On wearables, average power depends more on how often you run than on per-call speed.
Open problems · Inference
  • Router calibration: query-value estimators trained on chat data must handle egocentric, multimodal intents under battery, thermal and network shifts.
  • Speculation energy: speedups can raise energy per token when acceptance drops; wearable-scale evidence is thin.
  • KV eviction vs recall: bounded caches forget, and there are no guarantees for recalling earlier scenes.
  • Wearable–companion–cloud split: radio start-up, queueing and disconnects dominate tail latency; per-token cross-device protocols are fragile.
  • No glasses-class measurements: tokens/s, joules per token and thermal curves for AR1-class chips remain unpublished.