Edge AI for Wearables: The Gearbox
Running AI on glasses is a different problem from running it in the data center. Peak memory, battery, heat and a sub-500 ms response budget change what “good” means across data, models and inference, and UX is part of the design, not an afterthought.
The work in this space has exploded over the last two years. To make it easier to find your way in, we mapped the key research directions as a set of gears, with a companion table that links every technique to its source and calls out open problems.
| Level 1 | Level 2 | Level 3 | Knob / technique | What it is | Open problems |
|---|---|---|---|---|---|
| MODELS | Dense SLMs | 0.1–1B params | The size band that fits on the wearable itself: at 4-bit, a 1B model needs about 0.5 GB of weights, which fits glasses-class memory. 2–4B models (e.g., Qwen3-4B) run only on the companion phone. |
| |
| Deep & thin | Below ~1B parameters, more layers at narrower width beat wide-and-shallow at equal size (MobileLLM: +2.7 / +4.3 points at 125M / 350M), at some cost in serial latency. | ||||
| Tied embeds | The input embedding (token → vector) and the output projection (final vector → a score for each vocabulary word) share one matrix. Embeddings are a large share of a small model's weights, so tying frees memory for more layers. | ||||
| GQA | Grouped-query attention: groups of query heads share one key/value head, shrinking the KV cache 4–8× with close to full-attention quality. Standard in current small models. | ||||
| Weight sharing | Adjacent layers reuse one transformer block's weights (MobileLLM-LS runs each block twice): more depth without more storage, and little added latency because the weights are already in cache. | ||||
| MatFormer | A nested model (Gemma 3n): a smaller sub-model lives inside the larger one, so one download serves several sizes; switching between them at run time, by device load, is described as future work. | ||||
| PLE offload | Per-layer embeddings (PLE): Gemma 3n and Gemma 4 keep these in CPU memory instead of accelerator memory, so a model with ~5B raw parameters runs with a ~2B-class footprint (E2B), which is still above most glasses budgets today. | ||||
| Early exit | Stop at an intermediate layer when the prediction is already confident (LayerSkip), saving compute on easy tokens; the early layers can double as a self-speculative drafter. | ||||
| MoE (mixture of experts) | # experts | How many expert feed-forward blocks a layer holds. Each token runs only its top-k experts, so adding experts adds capacity without adding per-token compute; all of them must still be stored; MobileMoE finds moderate sparsity best on-device. | |||
| Top-k routing | Each token is sent to its k highest-scoring experts (e.g., top-2, top-4). Larger k improves quality but raises compute and the number of experts that must be loaded. | ||||
| Granularity | Split experts into many smaller ones (fine-grained experts). MobileMoE uses 60 routed experts with top-4 routing, which gives more diverse routing at equal compute. | ||||
| Shared expert | One always-on expert processes every token alongside the routed ones; it captures common knowledge and improves quality at fixed compute (MobileMoE, DeepSeek). | ||||
| Active / total | Compute scales with active parameters, memory with total. MobileMoE: 0.3–0.9B active of 1.3–5.3B total. A 30B-A3B model still needs ~15 GB at 4-bit. | ||||
| Expert cache | Keep recently or frequently used experts in RAM and stream the rest from flash (EdgeMoE, Mixtral-offloading). The cache hit rate largely sets decode speed. | ||||
| Prefetch | Predict the next layer's experts and load them while the current layer computes (routers placed before attention, next-layer gate speculation), hiding flash I/O behind compute. | ||||
| Mixed-bit | Store less important experts at lower precision, 2–4 bits by importance (EdgeMoE, HOBBIT); shrinks memory and flash traffic with limited quality loss. | ||||
| Prune / merge | Delete rarely useful experts or merge similar ones (NAEE, MC-SMoE), or slim every expert and distill (SlimMoE: 41.9B → 7.6B total). | ||||
| Hybrid & recurrent | Conv + attn | About two-thirds to three-quarters of layers are gated short convolutions, the rest attention (LFM2-1.2B: 10 of 16; LFM2-2.6B: 22 of 30). LFM2 reports up to 2× faster CPU prefill and decode than similar-size transformers. | |||
| SSM (Mamba-2) | State-space model (SSM): recurrent layers (e.g., Mamba-2) compress the past into a fixed-size state instead of a growing KV cache, so memory stays flat with sequence length (Granite 4.0-H mixes them 9:1 with attention). | ||||
| Attn+SSM heads | Attention and SSM heads run in parallel inside each layer (Hymba): an 11.67× smaller cache and 3.49× the throughput of Llama-3.2-3B, with higher accuracy. | ||||
| Gated DeltaNet | A gated linear-attention layer with delta-rule state updates; Qwen3.5 small models interleave three of these with each full-attention block to bound long-context cost. | ||||
| RWKV-7 | Receptance Weighted Key Value (RWKV): a fully recurrent RNN-style language model with constant memory and time per token and no KV cache; competitive up to ~3B parameters. | ||||
| Fixed-size state | The key edge property of recurrent layers: memory does not grow with context. State size and precision still matter, and recall of distant details can suffer. | ||||
| Few attn layers | Keep only a handful of full-attention layers for precise retrieval and make everything else cheap. Those layers still need a KV cache that grows with context. | ||||
| VLM (vision-language) | Encoder cost | The vision encoder often dominates time to first token. Efficient encoders such as FastViTHD (FastVLM) and MobileNet-V5 (Gemma 3n) cut encoding time sharply. | |||
| Visual tokens | Every image token must be prefilled by the language model, so token count drives latency: from 1 (LLaVA-Mini) to 144 (MobileVLM v2) to thousands at high resolution. | ||||
| Resolution | Higher resolution helps OCR and small objects but multiplies tokens and encoder cost; tiling one large image can add hundreds of tokens. | ||||
| ROI crops | Encode a small high-resolution crop of the region that matters (text, hands) plus a coarse view of the scene, instead of the whole frame at full resolution. | ||||
| Projector | The module that maps encoder features into language-model tokens; pooling projectors (LDPv2, pixel shuffle) compress 576 tokens to 144 or fewer. | ||||
| Token pruning | Drop or merge low-importance visual tokens after encoding (FastV, SparseVLM, VisionZip, ToMe), often halving compute; can remove evidence needed for counting or OCR. | ||||
| Frame rate | For video, process fewer frames or only frames that change. Cuts compute and power, at the risk of missing brief events. | ||||
| Video KV cap | Streaming video grows the KV cache without bound; cap it with attention sinks plus a window (StreamingVLM) or redundancy removal (InfiniPot-V: up to −94% memory). | ||||
| VRM (visual reasoning) | Think / no-think | Hybrid models (Qwen3, MiniCPM-V 4.5, Phi-4-reasoning-vision) switch between answering directly and chain-of-thought, skipping reasoning on plain perception tasks. | |||
| CoT budget | Cap the number of thinking tokens (budget forcing, as in s1) so reasoning fits the latency and energy budget. | ||||
| Adaptive length | Train the model to reason only as long as needed (AdaptThink: −53% length at 1.5B) or stop once confident (DEER: 19–80% shorter chains of thought). | ||||
| Re-look image | Instead of long textual reasoning, look again at a higher-resolution view only when needed (VisionThink starts at 1/4 resolution). | ||||
| RL (GRPO) | Group Relative Policy Optimization (GRPO): reinforcement learning with rule-based rewards (VLM-R1 with GRPO; LMM-R1 at 3B with PPO) builds visual reasoning in small models more robustly than supervised fine-tuning. | ||||
| Short-CoT KD | Distill a large reasoner's traces into short, correct chains for a small student, keeping the accuracy gain while cutting output tokens. | ||||
| Cloud offload | Deep visual reasoning rarely fits a wearable: 1,000 tokens at 20 tok/s ≈ 50 s. Send it to the companion or the cloud and keep a short-answer path on the wearable. | ||||
Open problems · Models
| |||||
| DATA | Real data | Calibration & sync | Each sensor's own parameters, the relative pose between sensors, and their timestamps must line up before video, IMU, audio and gaze can be fused.
|
| |
| Collection protocol | Scripted vs unscripted activity, who wears the device, where, for how long, and with what consent; together these decide how well a model generalizes. | ||||
| Multimodal capture | Wearables need synchronized video, IMU, spatial audio, gaze, hand and body pose and physiological signals, ideally with precise ground truth.
| ||||
| Annotation | Timestamped narrations and dense action, audio and QA labels are the main cost of real data; LLMs can densify cheaper weak labels.
| ||||
| Privacy & consent | Glasses record people who never agreed to it; faces and licence plates must be anonymized, and biometric laws apply to wearers and bystanders.
| ||||
| Federated | Real user data stays on the device; training, evaluation and analytics run federated, ideally with differential privacy.
| ||||
| Domain shift | Models degrade across devices (sensor, sampling rate, placement), users (gait, height, handedness), environments and time; personalization data closes part of the gap.
| ||||
| Synthetic data | World models | Video diffusion and world models generate controllable scenes, rare events and 'what if' variations (lighting, objects, weather).
| |||
| LoRA generators | Fine-tune a diffusion model with LoRA on a few real images so its output matches the device's camera and classes.
| ||||
| Scenario gen. | An LLM or program writes scenario descriptions, including edge cases, which are compiled into simulator code.
| ||||
| 3D sim & render | Physically rendered scenes give free, pixel-accurate labels (depth, segmentation, 6-DoF pose); randomizing layouts and assets helps transfer to real data.
| ||||
| Sensor sim. | Turn other modalities into IMU, audio or raw-camera signals that match wearable hardware.
| ||||
| LLM distillation | A large teacher model writes pre-training, instruction or multimodal QA data for small models.
| ||||
| Augmentation | Cheap, label-preserving transforms of real signals; still the first thing to try.
| ||||
| Private synthesis | Differentially private synthetic data, and replacing bystanders in always-on footage.
| ||||
| Curation & eval | Filter & dedup | Remove duplicates, low-quality and unsafe samples, and verify synthetic samples before use; small models cannot absorb noisy data.
| |||
| Data selection | Quality scoring and mixture design decide what a small model sees; data quality can substitute for data volume.
| ||||
| Real : synth mix | The ratio of real to synthetic data matters; synthetic data should add to real data, not replace it.
| ||||
| Long-tail coverage | Everyday egocentric data is dominated by a few common actions; rare but important events (falls, hazards, unusual objects) are sparse.
| ||||
| Egocentric evals | Held-out real benchmarks for long-horizon memory and question answering, which is what an assistant on glasses needs.
| ||||
| Compute-aware eval | None of the egocentric benchmarks measure latency or energy; wearables need task success within the budget.
| ||||
| Provenance | Track where each sample came from and what its licence allows; most egocentric corpora are research-only.
| ||||
Open problems · Data
| |||||
| COMPRESSION | Quantization | FP16 | 16-bit reference precision: 2 bytes per weight (a 2B model is 4 GB). Keep it for sensitive parts or as the baseline. |
| |
| INT8 | 8-bit weights and/or activations (LLM.int8(), SmoothQuant W8A8): close to lossless, half the memory, and a good match for integer NPUs. | ||||
| INT4 · default | 4-bit weights (GPTQ, AWQ, or QAT as in Gemma 3 / Llama 3.2): about 4× smaller than FP16 with small loss. The safe post-training default. | ||||
| 3-bit | Post-training quantization starts to hurt (about −4 points on average for 3-bit AWQ on Llama-3-8B); small and reasoning models degrade most (R1-Distill-1.5B: −18.6 points with 3-bit AWQ). | ||||
| 2-bit + QAT | Post-training 2-bit collapses; with QAT or vector quantization plus fine-tuning (AQLM, PV-Tuning, ParetoQ) it becomes Pareto-optimal. Apple's on-device model uses 2-bit QAT. | ||||
| 1.58-bit | Ternary weights (−1, 0, +1) trained natively (BitNet b1.58 2B4T): 0.4 GB of non-embedding weights and 0.028 J per token, but it needs custom kernels. | ||||
| Mixed-bit | Give sensitive layers or channels more bits and the rest fewer (SqueezeLLM, SliM-LLM); nested formats (MatQuant) serve int8, int4 and int2 from one model. | ||||
| W4A8 / W4A16 | Weight vs activation bits. W4A16 keeps activations at 16-bit (simple, accurate); W4A8 uses integer activations for faster NPU math (QServe), at some accuracy risk. | ||||
| Group size | One scale per group of 32–128 weights preserves accuracy, but some NPUs support only per-channel scales (Hexagon: Llama-3.2-1B scores 2.1% vs 15.9% on MATH500). | ||||
| Rotations | Multiply by rotation matrices (QuaRot, SpinQuant) to spread activation outliers, enabling 4-bit weights, activations and KV cache; the runtime must support or fuse them. | ||||
| Pruning & sparsity | Structured | Remove whole channels, heads or blocks, then retrain; the result is a smaller dense model that standard kernels accelerate (LLM-Pruner, SliceGPT). | |||
| Unstructured | Zero individual weights (SparseGPT: 50–60% in one shot; Wanda needs no weight updates); saves little on devices unless the kernel actually skips the zeros. | ||||
| N:M (2:4) | At most 2 non-zeros in each group of 4 weights; fast on NVIDIA sparse tensor cores, but support on mobile NPUs is limited. | ||||
| Block | Zero whole tiles (e.g., 16×16) so kernels can skip them; the speedup depends on block size, sparsity level and device support. | ||||
| Channel / head | Remove feed-forward channels or attention heads and physically shrink the tensors; keep dimensions aligned to the accelerator's tile sizes. | ||||
| Layer drop | Remove whole layers ranked by low importance (ShortGPT); simple and stacks with quantization, but deep cuts need recovery training. | ||||
| Width / depth | Prune hidden width and depth together, then distill from the original (Minitron: 40× fewer training tokens than training from scratch). | ||||
| Activation sp. | Activation sparsity: skip neurons whose activations are near zero for the current input (Deja Vu, TEAL, ProSparse at ~89%); pays off with CPU/NPU co-design or flash offload (PowerInfer-2). | ||||
| Fine-tuning (PEFT) | LoRA | Low-Rank Adaptation: freeze the base model and learn a low-rank update B·A (rank 8–64, under 1% of the weights); merge it into the weights or keep it as a swappable adapter. | |||
| QLoRA (NF4) | Quantized LoRA: LoRA on a 4-bit NormalFloat (NF4) base with double quantization and paged optimizers. A training-side technique: adapters are built off-device, then shipped to the wearable. NF4 is a training format, not an NPU format. | ||||
| DoRA | Weight-Decomposed Low-Rank Adaptation: splits each weight into magnitude and direction and applies LoRA only to the direction; consistently beats LoRA and merges with no inference cost. | ||||
| RoRA (α/√r) | Reliability Optimization for Rank Adaptation: changes LoRA's scaling from α/r to α/√r so accuracy keeps rising with rank; notable gains on pruned models (+5.7% over LoRA on Sheared-LLaMA-1.3B). | ||||
| QA-LoRA | Quantization-Aware LoRA: a LoRA variant whose update merges cleanly into an INT4 model, avoiding a precision mismatch after merging. | ||||
| LoftQ | LoRA-Fine-Tuning-aware Quantization: initializes LoRA jointly with quantization so the adapter starts by correcting quantization error; strongest at 2-bit and mixed 2/4-bit. | ||||
| Rank r | The adapter's size and capacity; higher rank adds parameters and memory. With standard scaling, gains can plateau as rank grows (the motivation for RoRA). | ||||
| Adapter swap | Keep one quantized base model and load small task adapters on demand (Apple: rank-16 adapters of tens of MB) instead of storing several full models. | ||||
| Distillation | Logit KD | Train the student to match the teacher's output probability distribution, which carries more signal than hard labels. | |||
| Feature KD | Also match intermediate hidden states or attention maps; useful after pruning, when student and teacher share structure. | ||||
| Teacher outputs | Fine-tune on answers generated by a larger model (sequence-level distillation); simple and effective for narrow tasks. | ||||
| Short CoT | Distill concise reasoning traces so the student reasons in fewer tokens, directly cutting decode time and energy. | ||||
| Task narrowing | Train a small specialist for the device's actual tasks (intents, captions, OCR) rather than a generalist; often a bigger win than another bit of quantization. | ||||
| System 2 → 1 | Distill answers that needed chain-of-thought into direct answers (Meta, 2024), baking repeated reasoning into the weights; works for many tasks, not multi-step math. | ||||
| Prune + KD | Prune the model, then distill from the unpruned original to recover quality (Minitron, SlimMoE); Sheared-LLaMA recovers with continued pre-training instead. | ||||
Open problems · Compression
| |||||
| INFERENCE | Attention | MHA | Multi-head attention: every query head has its own key/value head. Best quality, largest KV cache. |
| |
| MQA | Multi-query attention: all query heads share a single key/value head. Smallest cache, some quality loss. | ||||
| GQA | Grouped-query attention: groups of query heads share key/value heads (e.g., 32 → 8, a 4× smaller cache); existing models can be uptrained with ~5% of pretraining compute. | ||||
| MLA | Multi-head latent attention (DeepSeek): cache one small compressed vector per token (−93% KV); existing models can be converted (MHA2MLA, TransMLA). | ||||
| Local : global | Interleave sliding-window layers with occasional full-attention layers (Gemma 3: 5:1 with a 1,024-token window), cutting KV memory overhead from ~60% to under 15%. | ||||
| Window size | How many recent tokens a local layer sees (e.g., 512–4,096). Smaller windows save memory and compute; long-range recall then relies on the global layers. | ||||
| Sparse (NSA) | Native sparse attention: each token attends to a compressed summary, a few selected blocks and a local window; the model is trained sparse from the start. | ||||
| Block (MoBA) | Mixture of block attention: a gate picks the top-k context blocks for each query, like MoE routing over the context. Saves compute, not KV storage. | ||||
| Skip attn | Remove attention from some layers entirely (dropping half of Llama-2-70B's: 48% faster for −2.4%, shown at server scale); MobileLLM-Flash searches for where skipping is safe. | ||||
| KV cache | 8-bit KV | Store keys and values in 8-bit: half the KV memory with negligible loss. The shipped default in Apple's on-device model; the Gemma 4 report sizes memory with an int8 KV cache. | |||
| 4-bit KV | Another 2× saving; near-lossless even for reasoning models in reported tests, with extra packing and dequantization overhead. | ||||
| 2-bit KV | Aggressive (KIVI: keys per channel, values per token): 2.6× less peak memory, but more sensitive on retrieval-heavy tasks. | ||||
| Sinks + window | Keep the first few 'attention sink' tokens plus a recent window (StreamingLLM): bounded memory for always-on streams, but older details are dropped. | ||||
| Eviction | Keep only important tokens (H2O heavy hitters, SnapKV, PyramidKV at 12% of the cache) and drop the rest; selection overhead and changing queries can hurt recall. | ||||
| Prefix reuse | Cache the KV of recurring prompts (system prompt, agent context) and reload it; one study restored a context in 577 ms instead of 15.7 s. | ||||
| KV sharing | Adjacent layers reuse one KV cache (CLA, Gemma 3n and Gemma 4, Apple's on-device model: −37.5% KV); must be trained in, not bolted on. | ||||
| Flash tiering | Keep hot KV in RAM and move cold KV to flash or a low-rank form (TierKV: up to 17.6× faster prefill; phone-measured). | ||||
| Context ≤ 4K | Shipped assistants on phones and browsers cap context near 4K tokens (Apple on-device model: 4,096; Gemini Nano in Chrome: 4,096 per session); wearable budgets are tighter. A 128K advertised context is a ceiling, not a default. | ||||
| Speculative decoding | Draft model | A small model proposes several tokens and the target verifies them in one pass; needs a second model and its KV cache resident in memory. | |||
| Self-spec | Draft with the target's own early layers and verify with the rest (LayerSkip, Draft & Verify), so no second model is needed in memory. | ||||
| MTP heads | Multi-token prediction (MTP): extra heads predict several future tokens at once; Gemini Nano v3 on Pixel is 50%+ faster than with a separate drafter and saves 130 MB (phone-measured). | ||||
| EAGLE heads | Extrapolation Algorithm for Greater Language-model Efficiency (EAGLE): a light head drafts from the target's hidden features (EAGLE-2 and EAGLE-3: 3–6.5×, GPU-measured; gains on phones are smaller); a strong drafter with a small memory footprint. | ||||
| Prompt lookup | Draft by copying n-grams from the prompt or context; free, and effective for summarizing, editing and grounded answers. | ||||
| Token trees | Verify a tree of candidate continuations in one pass instead of a single chain (EdgeLLM, EAGLE-2), raising the tokens accepted per step. | ||||
| Draft length k | Tokens proposed per round; longer drafts pay off only when acceptance is high and verification cost grows slowly with k. | ||||
| Accept rate | The fraction of drafted tokens the target accepts: the main driver of speedup, and of whether energy per token goes up or down. | ||||
| Cross-device | Draft on the wearable and verify on the companion or in the cloud (−35% latency in one study); every round adds radio and queueing delay. | ||||
| Routing | Query value | Estimate how much a bigger model would improve this request, weighed against latency, energy and privacy; learned routers (RouteLLM) cut cost by over 85% on MT-Bench while keeping 95% of GPT-4's quality. | |||
| Difficulty est. | Predict up front whether the small model will do well enough (Hybrid LLM: 40% fewer large-model calls with no quality drop). | ||||
| Cascades | Try the small model first and escalate if a check fails (FrugalGPT, AutoMix): cheap on easy queries, but hard queries pay for both models. | ||||
| Token-level | Send only the critical tokens to the large model (R2R: a 1.5B + 32B pair is 2.8× faster than the 32B model alone at similar accuracy; the 32B model sits off-device). | ||||
| Think budget | Decide how much reasoning a query deserves (thinking vs non-thinking modes, token budgets) as part of the same routing decision. | ||||
| Confidence gate | Escalate or stop based on the model's confidence; thresholds need calibrating on the deployed model, since quantization and distillation change its confidence. | ||||
| Where to run | Choose among the wearable, its companion phone and the cloud, weighing network latency, availability and privacy; keep an offline path for time-critical actions. | ||||
| Battery-aware | Factor battery level and temperature into the decision; sustained load can cut throughput ~40% (iPhone 16 Pro: 40.5 → 23.7 tok/s, phone-measured; glasses throttle sooner). | ||||
| Input & system | NPU placement | Run supported operators on the NPU, DSP or GPU and keep large graph segments on one processor to avoid transfers and synchronization. | |||
| No CPU fallback | An unsupported operator forces a fallback to the CPU; fragmented execution can be slower than running everything on the CPU. Build only from supported ops. | ||||
| Op fusion | Merge adjacent operations (e.g., matrix multiply + activation, fused attention) into one kernel, cutting memory traffic and dispatch overhead. | ||||
| BN folding | Fold batch normalization into the preceding convolution's weights and bias; mathematically exact, and removes a separate pass over the tensor. | ||||
| AOT compile | Compile and optimize the graph ahead of time for the target runtime (ExecuTorch, QNN, Core ML), reducing startup and dispatch work. | ||||
| Buffer reuse | Preallocate and reuse memory buffers instead of allocating on every inference; lowers peak memory and allocator overhead. | ||||
| Fewer copies | Remove redundant copies and layout conversions between camera, preprocessing and model; these can cost more than the model itself. | ||||
| Tiling | Split computation into tiles that fit in on-chip memory, reducing slow and energy-hungry external-memory traffic. | ||||
| Shorter context | Fewer prompt tokens mean faster prefill and a smaller KV cache; at 0.6 ms per prompt token, 1K tokens alone cost ~600 ms. | ||||
| Fewer calls | Run the model less often (event triggers, batching, lower duty cycle). On wearables, average power depends more on how often you run than on per-call speed. | ||||
Open problems · Inference
| |||||