---
title: Running On-Device SLMs on Mobile: ExecuTorch, 4-Bit Quantization, and Real-Time Token Streaming in React Native
publishedAt: 2026-09-01
summary: A pragmatic engineering guide to running Small Language Models on iOS and Android using Software Mansion’s react-native-executorch, 4-bit quantization, and the useLLM streaming hook.
---

# Running On-Device SLMs on Mobile: ExecuTorch, 4-Bit Quantization, and Real-Time Token Streaming in React Native

### Executive Summary (TL;DR)
Running Small Language Models (SLMs) directly on mobile devices eliminates cloud inference latency, guarantees zero-cost offline availability, and keeps user data private. By leveraging Meta's [PyTorch ExecuTorch](https://github.com/pytorch/executorch) runtime together with Software Mansion's official [react-native-executorch](https://github.com/software-mansion/react-native-executorch) package, you can run 4-bit quantized models like Llama 3.2 1B natively without writing low-level C++ or Swift/Kotlin bridge boilerplate. This guide provides a complete, production-ready walkthrough: preparing 4-bit `.pte` binaries with [torchao](https://github.com/pytorch/ao), installing and configuring the library, and building a responsive token-streaming chat UI with the `useLLM` hook.

---

## Architecture: ExecuTorch and React Native Runtime Flow

Running Small Language Models on mobile devices requires isolating compute-heavy tensor generation on native background threads while streaming tokens directly into React Native UI state. With [Software Mansion's react-native-executorch](https://github.com/software-mansion/react-native-executorch), PyTorch's ExecuTorch runtime is integrated natively via modern C++ TurboModules, bypassing traditional bridge serializations. This architecture allows 4-bit quantized models to download on demand, execute on mobile hardware delegates, and push real-time token updates without blocking JavaScript gesture interactions.

```mermaid
flowchart TD
    subgraph ModelPreparation["1. Model Preparation & Distribution"]
        A[Hugging Face Weights<br/>Llama 3.2 1B] --> B[torchao INT4 Quantization]
        B --> C[ExecuTorch Export & XNNPACK Lowering]
        C --> D[model.pte Binary on CDN / HF]
    end

    subgraph NativeLayer["2. Mobile Native Layer (react-native-executorch)"]
        D -. On-Demand Download .-> E[Resource Fetcher / Local Cache]
        E --> F[ExecuTorch C++ Engine]
        F --> G[Background Worker Thread<br/>Autoregressive Decode Loop]
    end

    subgraph UIThread["3. React Native Application"]
        G -->|Zero-Copy Token Stream| H["useLLM Hook (TurboModule)"]
        H -->|Reactive State: response, messageHistory| I[React Native Chat UI]
        I -->|User Prompt / Interrupt| H
    end
```

By delegating the autoregressive generation loop to dedicated native background threads, the JavaScript event loop and main UI thread remain completely free to process animations, taps, and keyboard events at 60/120 FPS.

---

## Step 1: Exporting and Quantizing Models (INT4 with torchao)

Deploying an SLM to mobile hardware begins with converting a PyTorch model into an optimized `.pte` binary using 4-bit weight-only quantization via [torchao](https://github.com/pytorch/ao). Quantizing weights to INT4 slashes the resident memory footprint of a 1B parameter model from over 2.5 GB to under 800 MB, fitting well inside mobile operating system thresholds. This offline compilation step partitions model operators to mobile backends such as XNNPACK, preparing the weights for zero-copy memory mapping on iOS and Android.

> [!TIP]
> If you prefer not to compile binaries yourself, Software Mansion hosts verified, pre-exported 4-bit `.pte` checkpoints and tokenizers directly on [Hugging Face](https://huggingface.co/software-mansion/react-native-executorch-llama-3.2). You can load those using built-in presets (`LLAMA3_2_1B`) out-of-the-box.

### 1. Environment Setup

To export and quantize custom weights locally, set up a Python 3.10+ environment with PyTorch, TorchAO, and ExecuTorch:

```bash
pip install torch torchvision
pip install torchao
pip install executorch
pip install transformers
```

### 2. Export and Quantization Script

The following script loads `meta-llama/Llama-3.2-1B-Instruct`, applies 4-bit groupwise weight quantization (`group_size=128`), lowers operators to the mobile-optimized XNNPACK CPU delegate, and outputs the `.pte` executable:

```python
# export_llama_int4.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from torchao.quantization.quant_api import quantize_, int4_weight_only
from executorch.exir import to_edge, EdgeCompileConfig
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
OUTPUT_FILE = "./llama3_2_1b_int4.pte"

print("[1/4] Loading Hugging Face model and tokenizer...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float32,
    device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

print("[2/4] Applying INT4 groupwise weight quantization...")
# Reduces weight precision to 4-bit, dropping 1B params under 800MB
quantize_(model, int4_weight_only(group_size=128))

print("[3/4] Tracing and lowering graph to ExecuTorch Edge IR...")
sample_input = (torch.zeros((1, 128), dtype=torch.long),)
edge_program = to_edge(
    torch.export.export(model, sample_input),
    compile_config=EdgeCompileConfig(_check_ir=False)
)

print("[4/4] Partitioning for XNNPACK and serializing .pte binary...")
delegated_program = edge_program.to_backend(XnnpackPartitioner())
exec_program = delegated_program.to_executorch()

with open(OUTPUT_FILE, "wb") as f:
    f.write(exec_program.buffer)

print(f"Export completed successfully: {OUTPUT_FILE}")
```

Once exported, host your `.pte` file and tokenizer assets on a secure CDN or cloud bucket with HTTP Range request support for resumable downloads.

---

## Step 2: Installing and Configuring react-native-executorch

Integrating ExecuTorch into React Native requires the New Architecture (TurboModules) and the official `react-native-executorch` package alongside a resource fetcher adapter. The setup entails installing the library, configuring your bundler to recognize `.pte` and `.bin` model extensions, and initializing the native runtime at your app's root. Once initialized, the native engine handles asynchronous binary fetching, caching, and thread management automatically.

> [!IMPORTANT]
> `react-native-executorch` strictly requires the **React Native New Architecture** (React Native 0.76+). If you are using Expo, you must use a **development build** (`npx expo run:ios` or `npx expo run:android`), as custom native C++ runtimes cannot run inside the standard Expo Go sandbox.

### 1. Package Installation

Install `react-native-executorch` and its Expo resource fetcher adapter (works with both bare React Native apps with Expo modules and managed Expo workflows):

```bash
# Core package
npm install react-native-executorch

# Resource fetcher for on-demand model downloading and caching
npm install react-native-executorch-expo-resource-fetcher expo-file-system expo-asset
```

For iOS bare workflows, install the CocoaPods dependencies:

```bash
cd ios && pod install && cd ..
```

### 2. Configure Metro Bundler (`metro.config.js`)

If you plan to bundle lightweight models or local test binaries directly as app assets, configure Metro to resolve `.pte` and `.bin` file types:

```javascript
// metro.config.js
const { getDefaultConfig } = require('expo/metro-config'); // or '@react-native/metro-config'

const config = getDefaultConfig(__dirname);

config.resolver.assetExts.push('pte', 'bin');

module.exports = config;
```

### 3. Initialize Runtime at Application Entry Point

Initialize `react-native-executorch` before mounting any components that execute inference. Call `initExecutorch` in your root entry file (`App.tsx` or `index.ts`):

```tsx
// App.tsx
import React from 'react';
import { initExecutorch } from 'react-native-executorch';
import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';
import { ChatScreen } from './src/ChatScreen';

// Initialize ExecuTorch native engine with persistent resource caching
initExecutorch({
  resourceFetcher: ExpoResourceFetcher,
});

export default function App() {
  return <ChatScreen />;
}
```

---

## Step 3: Implementing Real-Time Token Streaming with useLLM

Streaming inference tokens smoothly into a chat interface is handled declaratively using the `useLLM` hook provided by `react-native-executorch`. The hook exposes reactive properties for asset download progress, model initialization status, live token generation, and generation interruption. By binding the reactive `response` stream and `messageHistory` to React Native state, you achieve fluid 60 FPS chat updates without writing custom native bridge boilerplate.

### Complete Streaming Chat Screen (`ChatScreen.tsx`)

The following production-ready component downloads the quantized Llama 3.2 1B model on demand, tracks caching progress, streams tokens into the UI in real time, and allows users to stop generation mid-stream:

```tsx
// src/ChatScreen.tsx
import React, { useState, useRef, useEffect } from 'react';
import {
  View,
  Text,
  TextInput,
  TouchableOpacity,
  FlatList,
  ActivityIndicator,
  StyleSheet,
  SafeAreaView,
  KeyboardAvoidingView,
  Platform,
} from 'react-native';
import { useLLM, LLAMA3_2_1B } from 'react-native-executorch';

export function ChatScreen() {
  const [input, setInput] = useState('');
  const flatListRef = useRef<FlatList>(null);

  // Initialize LLM using Software Mansion's pre-configured Llama 3.2 1B preset
  const llm = useLLM({
    model: LLAMA3_2_1B,
  });

  // Auto-scroll as tokens stream in
  useEffect(() => {
    if (llm.response.length > 0 || llm.messageHistory.length > 0) {
      flatListRef.current?.scrollToEnd({ animated: true });
    }
  }, [llm.response, llm.messageHistory]);

  const handleSend = async () => {
    const prompt = input.trim();
    if (!prompt || !llm.isReady || llm.isGenerating) return;

    setInput('');
    try {
      await llm.sendMessage(prompt);
    } catch (err) {
      console.error('Inference error:', err);
    }
  };

  const handleInterrupt = () => {
    llm.interrupt();
  };

  // State 1: Model downloading or initializing
  if (!llm.isReady) {
    const progressPercent = Math.round((llm.downloadProgress ?? 0) * 100);
    return (
      <SafeAreaView style={styles.loadingContainer}>
        <ActivityIndicator size="large" color="#0284c7" />
        <Text style={styles.loadingTitle}>Loading On-Device SLM</Text>
        <Text style={styles.loadingSubtitle}>
          {progressPercent > 0 && progressPercent < 100
            ? `Downloading model: ${progressPercent}%`
            : 'Initializing ExecuTorch runtime...'}
        </Text>
        <View style={styles.progressBarBackground}>
          <View style={[styles.progressBarFill, { width: `${progressPercent}%` }]} />
        </View>
      </SafeAreaView>
    );
  }

  // State 2: Active Chat Interface
  return (
    <SafeAreaView style={styles.container}>
      <KeyboardAvoidingView
        style={styles.container}
        behavior={Platform.OS === 'ios' ? 'padding' : undefined}
      >
        <View style={styles.header}>
          <Text style={styles.headerTitle}>Llama 3.2 1B (On-Device)</Text>
          <View style={styles.badge}>
            <Text style={styles.badgeText}>Offline INT4</Text>
          </View>
        </View>

        <FlatList
          ref={flatListRef}
          data={llm.messageHistory}
          keyExtractor={(_, index) => index.toString()}
          contentContainerStyle={styles.messageList}
          renderItem={({ item }) => (
            <View
              style={[
                styles.messageBubble,
                item.role === 'user' ? styles.userBubble : styles.assistantBubble,
              ]}
            >
              <Text style={styles.roleLabel}>{item.role === 'user' ? 'You' : 'Llama'}</Text>
              <Text
                style={[
                  styles.messageText,
                  item.role === 'user' ? styles.userText : styles.assistantText,
                ]}
              >
                {item.content}
              </Text>
            </View>
          )}
          ListFooterComponent={
            llm.isGenerating ? (
              <View style={[styles.messageBubble, styles.assistantBubble]}>
                <Text style={styles.roleLabel}>Llama (generating...)</Text>
                <Text style={[styles.messageText, styles.assistantText]}>
                  {llm.response || '...'}
                </Text>
              </View>
            ) : null
          }
        />

        <View style={styles.inputContainer}>
          <TextInput
            style={styles.textInput}
            value={input}
            onChangeText={setInput}
            placeholder="Type a message..."
            placeholderTextColor="#94a3b8"
            editable={!llm.isGenerating}
            onSubmitEditing={handleSend}
            returnKeyType="send"
          />

          {llm.isGenerating ? (
            <TouchableOpacity style={styles.stopButton} onPress={handleInterrupt}>
              <Text style={styles.buttonText}>Stop</Text>
            </TouchableOpacity>
          ) : (
            <TouchableOpacity
              style={[styles.sendButton, !input.trim() && styles.disabledButton]}
              onPress={handleSend}
              disabled={!input.trim()}
            >
              <Text style={styles.buttonText}>Send</Text>
            </TouchableOpacity>
          )}
        </View>
      </KeyboardAvoidingView>
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1, backgroundColor: '#f8fafc' },
  loadingContainer: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
    backgroundColor: '#f8fafc',
    padding: 24,
  },
  loadingTitle: { fontSize: 18, fontWeight: '700', color: '#0f172a', marginTop: 16 },
  loadingSubtitle: { fontSize: 14, color: '#64748b', marginTop: 8 },
  progressBarBackground: {
    width: '80%',
    height: 6,
    backgroundColor: '#e2e8f0',
    borderRadius: 3,
    marginTop: 16,
    overflow: 'hidden',
  },
  progressBarFill: { height: '100%', backgroundColor: '#0284c7' },
  header: {
    paddingHorizontal: 16,
    paddingVertical: 12,
    backgroundColor: '#ffffff',
    borderBottomWidth: 1,
    borderColor: '#e2e8f0',
    flexDirection: 'row',
    alignItems: 'center',
    justifyContent: 'space-between',
  },
  headerTitle: { fontSize: 16, fontWeight: '700', color: '#0f172a' },
  badge: { backgroundColor: '#e0f2fe', paddingHorizontal: 8, paddingVertical: 4, borderRadius: 6 },
  badgeText: { fontSize: 12, fontWeight: '600', color: '#0284c7' },
  messageList: { padding: 16, gap: 12 },
  messageBubble: { padding: 12, borderRadius: 12, maxWidth: '85%' },
  userBubble: { alignSelf: 'flex-end', backgroundColor: '#0284c7' },
  assistantBubble: { alignSelf: 'flex-start', backgroundColor: '#ffffff', borderWidth: 1, borderColor: '#e2e8f0' },
  roleLabel: { fontSize: 11, fontWeight: '600', color: '#94a3b8', marginBottom: 4 },
  messageText: { fontSize: 15, lineHeight: 22 },
  userText: { color: '#ffffff' },
  assistantText: { color: '#0f172a' },
  inputContainer: {
    flexDirection: 'row',
    padding: 12,
    backgroundColor: '#ffffff',
    borderTopWidth: 1,
    borderColor: '#e2e8f0',
    alignItems: 'center',
  },
  textInput: {
    flex: 1,
    backgroundColor: '#f1f5f9',
    borderRadius: 8,
    paddingHorizontal: 14,
    paddingVertical: 10,
    fontSize: 15,
    color: '#0f172a',
    marginRight: 8,
  },
  sendButton: {
    backgroundColor: '#0284c7',
    paddingHorizontal: 18,
    paddingVertical: 10,
    borderRadius: 8,
  },
  stopButton: {
    backgroundColor: '#ef4444',
    paddingHorizontal: 18,
    paddingVertical: 10,
    borderRadius: 8,
  },
  disabledButton: { backgroundColor: '#cbd5e1' },
  buttonText: { color: '#ffffff', fontWeight: '600', fontSize: 14 },
});
```

---

## Step 4: Mobile Memory Budgets, Context Windows, and Production Pitfalls

Deploying on-device generative AI models introduces real-world mobile limitations around RAM headroom, thermal throttling, and app binary size constraints. Exceeding platform memory limits triggers immediate OS watchdog terminations (`SIGKILL`), while continuous generation on mobile SoCs can induce severe thermal throttling. Addressing these constraints requires dynamic asset downloading over Wi-Fi, strict context window clamping, and graceful error handling for device lifecycle transitions.

### 1. Never Bundle `.pte` Weights Inside the App Store Binary

Embedding a 700 MB `.pte` model directly into `ipa` or `apk` assets will exceed Apple’s and Google’s cellular download ceilings (typically 200 MB OTA limit) and inflate app update payloads.
- Store your `.pte` binaries on a CDN or Hugging Face repository.
- Use `react-native-executorch`'s resource fetcher, which downloads the model to persistent app storage upon initial launch.
- Subsequent app launches instantly load the cached binary from local storage via memory mapping (`mmap`), consuming near-zero download bandwidth.

### 2. Context Window and KV Cache Overflow

On-device inference reserves an in-memory Key-Value (KV) cache for multi-turn history. If conversational turns accumulate unchecked, ExecuTorch will fail or the OS will terminate the app due to memory pressure.

To prevent context overflow:
- Cap maximum generation turns by truncating older messages from history.
- If using `generate()` in manual mode instead of `sendMessage()`, implement a sliding window strategy retaining only the initial system prompt and the last 3–5 conversation turns.

```typescript
// Truncate message history to prevent KV cache memory blowout
const prunedHistory = messageHistory.slice(-6);
```

### 3. Thermal Throttling & Battery Conservation

Autoregressive token generation places heavy sustained load across CPU cores or NPUs. Continuous generation generates substantial heat, causing iOS and Android to aggressively downclock processor frequencies and drain battery.
- Enforce conservative generation limits (`maxTokens: 150` to `256`) per request.
- Always provide an interruption trigger via `llm.interrupt()` so users can abort long responses immediately.
- Prevent background inference: pause or terminate generation when the app transitions into `background` via React Native's `AppState` API.

---

## Frequently Asked Questions (FAQ)

Adopting on-device inference with ExecuTorch raises important technical questions regarding device compatibility, hardware acceleration, and runtime stability. Understanding how mobile operating systems handle memory allocation and how backends leverage NPUs is essential for delivering robust native experiences. Below are direct answers to the most common engineering questions encountered when shipping SLMs in production mobile apps.

### What are the minimum device hardware requirements for running a 1B model?
A 4-bit quantized 1B parameter model requires approximately 750 MB to 900 MB of resident RAM for model weights and active KV cache buffers. For stable multi-app multitasking without OS memory pressure terminations, target devices with at least 4 GB of RAM on Android and 3 GB of RAM on iOS (iPhone 11 or newer).

### How does token streaming work without dropping UI frames?
`react-native-executorch` runs the ExecuTorch C++ runtime on a dedicated background POSIX thread. As each token is generated, it writes directly to native memory and notifies the React Native TurboModule layer. The UI updates through the hook's reactive `response` property, keeping the JavaScript thread and main UI frame rate decoupled from tensor matrix calculations.

### Can custom fine-tuned models be loaded into react-native-executorch?
Yes. Any model exported to the `.pte` format using ExecuTorch's export toolchain can be loaded by passing custom file URIs or URLs into `modelSource`, `tokenizerSource`, and `tokenizerConfigSource`. You are not limited to pre-configured library presets.

---

### About the Author
**Furkan Çetinkaya** is a Mobile-focused Software Developer specializing in React Native, native bridge integrations (Kotlin & Swift), and supporting backend services. Experienced in maintaining high-impact mobile applications and developer SDKs.
- [GitHub](https://github.com/cetfu)
- [LinkedIn](https://www.linkedin.com/in/cetfu)