Running a language model on a phone, laptop, vehicle, or embedded gateway is not simply cloud inference with a smaller model. The device shares memory with the application, changes performance as it heats, may lose power mid-generation, and must support hardware and operating-system versions already in users’ hands. Network absence can be the reason for the feature, while network availability can still be a useful escape path.
The core engineering thesis is: an on-device model is one component in a resource-bounded product contract. Choose the task and device envelope first, then co-design model, quantization, runtime, context, user experience, and fallback. Parameter count alone cannot tell whether a feature fits, responds promptly, preserves battery, or remains accurate after compression.
This chapter works through an offline field-note assistant that converts a technician’s short notes into a structured service record. The feature must work without connectivity, keep customer details local by default, and decline tasks that exceed its bounded extraction and rewriting capabilities.
Define the Task and Device Envelope
“Run an LLM locally” is a technology goal, not a product requirement. Specify what the model must do, how quickly the interaction must begin, how much context it needs, which outputs are permitted, and what happens when it cannot complete the task. Classification, extraction, autocomplete, and short rewriting generally fit smaller models more readily than open-ended research or long-form reasoning.
Define a supported device matrix rather than an imaginary average device. Relevant dimensions include operating-system version, CPU instruction support, GPU or neural accelerator availability, addressable memory, free storage, battery state, thermal design, and whether the application can run while backgrounded. A desktop with unified memory and active cooling behaves differently from a low-cost phone that suspends background work.
Reserve resources for the rest of the product. If a device has 6 GB of memory, the model does not own 6 GB. The operating system, UI, camera pipeline, local database, and other applications compete for it. Set a model resident-memory budget, peak temporary-allocation budget, package-download budget, and cache budget. Define behavior below the minimum: disable the feature, load a smaller variant, or offer a network path with informed consent.
The task contract for the field-note assistant accepts at most a bounded note and an approved local equipment catalog. It returns a strict record containing equipment identifier, reported symptom, action taken, unresolved issue, and a list of source spans. It never invents a part number, schedules work, or submits the record automatically. Those constraints reduce context, simplify validation, and make a small model useful without pretending it is a general assistant.
Budget Memory, Compute, Energy, and Heat
Model weights establish a lower bound on storage and resident memory. For parameters represented with an average of bits, the idealized weight payload is
Real packages are larger because they include metadata, tokenizer files, alignment, lookup tables, and sometimes mixed-precision tensors. Runtime memory also includes activations, scratch buffers, graph state, and the key-value cache used during autoregressive generation. Measure peak resident memory on the target runtime; do not accept the model-file size as a proxy.
Context is a resource decision. The key-value cache grows with sequence length and model architecture, so a nominally supported long context can exhaust memory or make prompt processing unacceptably slow. Bound input length by the actual task, retrieve only relevant local records, and summarize through deterministic domain logic where possible. A smaller prompt also reduces the amount of sensitive text exposed to model memory.
Latency has at least two visible phases. Prefill processes the prompt before the first output token. Decode generates subsequent tokens, often one step at a time. Time to first token governs whether the interface feels responsive; tokens per second governs whether a longer response drags. For structured extraction, total completion time and schema-valid rate may matter more than a high decode rate.
Energy and temperature make performance history-dependent. A short cold benchmark can look excellent before the device throttles. Sustained generation occupies compute units, wakes memory, and competes with rendering or camera work. Measure battery discharge or platform energy counters over a controlled workload, and run repeated sessions long enough to observe thermal stabilization. Avoid claiming a universal energy cost: screen state, radio activity, silicon, ambient temperature, and runtime version all alter it.
Compress Deliberately with Quantization and Distillation
Quantization stores weights, and sometimes activations or cache values, at lower precision. Moving from 16-bit to 8-bit or 4-bit weights can substantially reduce payload and memory bandwidth, but the ideal ratio is not guaranteed in the full application. Scales, zero points, mixed-precision layers, unpacking kernels, and unsupported operations add overhead.
Post-training quantization is convenient because it starts from an existing model. Calibration-based methods use representative samples to choose ranges or reconstruction parameters. Quantization-aware training exposes the model to approximate low-precision effects during training and can recover quality, at greater cost and pipeline complexity. Some architectures or layers are unusually sensitive; mixed precision may keep selected tensors at higher precision.
Quality loss is task-specific. A quantized model can preserve fluent prose while damaging exact extraction, multilingual behavior, rare identifiers, or calibrated confidence. Evaluate schema fields and critical slices, not only visual plausibility. Tokenizer and embedding tables also matter for languages and domain symbols. A model compact in English may tokenize another supported language inefficiently, increasing context and latency.
Distillation trains a smaller student to reproduce selected behavior from a larger teacher or labeled corpus. It can focus capacity on the product task, vocabulary, and output contract. The student also inherits teacher errors and any synthetic-data weaknesses. Keep human-authored or independently labeled validation data, document teacher provenance, and avoid measuring the student solely by agreement with its teacher.
Pruning and architecture-specific compression can help, but sparse weights improve speed only when the target runtime and hardware exploit that sparsity. A theoretically smaller operation count may be slower than a dense kernel optimized by the device vendor. Export the actual candidate and benchmark the exact graph; paper specifications do not execute the shipping path.
Treat the Runtime as Part of the Model
The same weights can behave differently across inference engines. A runtime determines supported operators, graph transformations, memory mapping, threading, accelerator delegation, cache layout, and fallback to CPU. An unsupported operation can move part of a graph across processors and erase the expected acceleration through copies and synchronization.
Choose a deployment format and runtime supported across the device matrix. Platform runtimes can integrate tightly with vendor accelerators and application lifecycle. Cross-platform runtimes can reduce packaging differences but still need device-specific execution providers. Verify operator coverage during build and collect runtime diagnostics on representative devices. Silent CPU fallback should be observable.
Thread count is not a “more is faster” knob. Extra threads can contend with the UI, consume efficient cores needed elsewhere, and increase energy without lowering latency. Accelerator use can conflict with camera, graphics, or other ML features. Benchmark scheduling under realistic concurrent application work, and assign quality-of-service priorities where the platform permits it.
Model delivery is also runtime engineering. Packages may be too large for the base application download. Use signed, resumable downloads with checksums, available-space checks, and version pinning. Never activate a partially written package. Keep the previous compatible model until the new model passes an on-device load check, and provide garbage collection for obsolete versions without deleting an active file.
Cold load, warm reuse, and process eviction need separate measurements. Memory mapping may reduce startup allocations but cause page faults during first use. Keeping the model resident improves responsiveness while increasing memory pressure. Make residency conditional on device class and recent use, and respond to operating-system memory warnings instead of fighting them.
Work Through an Offline Field-Note Assistant
A technician writes: “Unit HX-41 vibrates after ten minutes. Tightened left mount, vibration reduced but remains. Needs bearing inspection.” The local equipment catalog contains identifiers and component names. The desired record is concise and attributable:
{
"equipmentId": "HX-41",
"reportedSymptom": "Vibration begins after approximately ten minutes",
"actionTaken": "Tightened left mount",
"outcome": "Vibration reduced but remains",
"followUp": "Inspect bearing",
"sourceSpans": {
"equipmentId": [5, 10],
"actionTaken": [40, 60]
}
}
The application first validates note length and performs deterministic catalog lookup for literal identifier candidates. It supplies only the matching equipment entry and the note to the model. The model fills a constrained schema; application code verifies that equipmentId belongs to the candidate set and that source spans point inside the original note. The user sees a draft with every field editable and must confirm before local persistence.
Assume three candidate builds for illustration: a higher-precision baseline, a 4-bit quantized variant, and a distilled 4-bit student. The team does not select the smallest package immediately. It measures exact identifier accuracy, field omission, unsupported inference, schema validity, cold start, warm completion time, peak memory, energy over repeated notes, and thermal behavior on each supported device tier.
The quantized baseline may retain extraction quality but exceed memory on the lowest tier because its cache and scratch buffers are large. The student may fit comfortably yet omit follow-up details in terse notes. That result suggests task-specific distillation or routing, not a universal winner. The lowest tier might use a shorter context and extraction-only student, while capable devices also offer rewriting.
When a note exceeds the local contract, contains several equipment records, or repeatedly fails validation, the assistant returns a draft with unresolved fields. If connectivity and policy allow, it can offer a cloud analysis. The original note is not uploaded automatically; the user sees what data would leave the device and can decline without losing local editing.
Benchmark the Shipping Path on Real Devices
Desktop emulators and developer laptops are useful for correctness, not device acceptance. Benchmark release builds, production compiler settings, the shipping tokenizer, actual model package, and the same runtime APIs used by the UI. Include minimum, common, and high-end device tiers plus devices from different accelerator families.
Define workloads from product behavior: short and maximum-length notes, supported languages, catalog-hit and miss cases, valid and malformed inputs, repeated interactions, cold process launch, warm reuse, background interruption, and low-memory recovery. Fix input sets and record model, runtime, operating system, application, and device versions. Label all sample numbers as measurements from those conditions, never as general model properties.
Measure at least:
- package size, download expansion, and load failure rate;
- cold model readiness and warm time to first token;
- prefill duration, decode rate, and total task latency;
- peak resident memory and allocation failures;
- task-specific correctness, schema validity, and abstention;
- energy over a defined repeated workload and thermal throttling;
- UI frame stalls, cancellation latency, and recovery after interruption.
Run enough repetitions to expose variance, randomize candidate order, and allow devices to return to a documented thermal state between independent cold tests. For sustained tests, do the opposite: keep the workload active to observe throttling. Report distributions rather than the fastest run. Check outputs so compiler or cache behavior cannot turn the benchmark into no work.
Automated device farms can catch compatibility failures, while energy and thermal studies may require controlled physical setups. Protect performance budgets with broad, device-tier-specific thresholds and trend review; shared devices are noisy. Correctness gates must run for every compressed artifact because conversion itself can change outputs.
Design Privacy, Offline UX, and Cloud Fallback Together
On-device execution reduces network exposure, but it is not synonymous with privacy. Prompts can enter application logs, crash reports, keyboard telemetry, backups, or unencrypted caches. Model outputs may reveal local records to another signed-in user. Apply data minimization, platform-protected storage, explicit retention, lock-screen behavior, and redacted diagnostics.
State clearly what runs locally. A feature labeled “offline” should not fetch a tokenizer, safety classifier, or telemetry endpoint on first use without disclosure. Package required components or explain the one-time download. Queue no hidden upload for later connectivity. Aggregate operational telemetry only with an appropriate consent and privacy design; raw notes are rarely necessary to learn that model loading failed.
Offline UX must handle interruption. Save the user’s source note before inference, expose cancellation, and make retries idempotent. If the operating system suspends or kills the process, reopen to the original note rather than a spinner or partial model state. Streamed output should remain a draft until schema and source checks complete.
Cloud fallback is a capability escalation. Define triggers such as unsupported task, insufficient memory, user-selected higher quality, or repeated local validation failure. Never route merely because the local answer “looks uncertain” unless uncertainty is measured and the user understands the transfer. Minimize uploaded context, apply the same output contract, and prevent fallback loops between local and remote paths.
Operate Models Across a Fragmented Fleet
Shipping creates a matrix of model, quantization, runtime, application, operating system, and hardware. Roll out by device capability and preserve a kill switch that disables a faulty model without breaking manual workflows. A signed manifest should declare package hash, minimum runtime, memory class, tokenizer, context limit, and compatible schema version.
Monitor privacy-safe aggregates: package download and verification failures, load errors, CPU fallback, out-of-memory events, cancellation, schema rejection, local completion, cloud escalation, and latency bands by coarse device class. Avoid collecting raw prompts or stable hardware fingerprints by default. A rising fallback rate after an operating-system update may reveal runtime incompatibility rather than declining model quality.
Failure modes include choosing by parameter count, benchmarking only a warm flagship device, assuming lower precision always runs faster, ignoring tokenizer expansion, retaining too much context, blocking the UI thread, and silently uploading failed requests. Another subtle failure is capability creep: a model approved for extraction gradually becomes a general assistant without new resource, privacy, or quality analysis.
Treat model updates like application code. Validate conversion, run task and device matrices, canary eligible devices, compare resource and quality signals, and retain rollback packages. Ensure old applications cannot load an incompatible schema and new applications can still read records created by older models.
An edge model succeeds when the product remains useful under the actual device’s limits. The winning design may use a distilled local extractor, deterministic validation, and an optional remote path rather than the largest model that barely loads. Resource accounting, honest real-device benchmarks, and explicit degraded behavior turn local inference from a demo into a dependable offline capability.