Running On-Device SLMs on Mobile: ExecuTorch, 4-Bit Quantization, and Real-Time Token Streaming in React Native

Executive Summary (TL;DR)

Running Small Language Models (SLMs) directly on mobile devices eliminates cloud inference latency, guarantees zero-cost offline availability, and keeps user data private. By leveraging Meta's PyTorch ExecuTorch runtime together with Software Mansion's official react-native-executorch package, you can run 4-bit quantized models like Llama 3.2 1B natively without writing low-level C++ or Swift/Kotlin bridge boilerplate. This guide provides a complete, production-ready walkthrough: preparing 4-bit .pte binaries with torchao, installing and configuring the library, and building a responsive token-streaming chat UI with the useLLM hook.

Architecture: ExecuTorch and React Native Runtime Flow

Running Small Language Models on mobile devices requires isolating compute-heavy tensor generation on native background threads while streaming tokens directly into React Native UI state. With Software Mansion's react-native-executorch, PyTorch's ExecuTorch runtime is integrated natively via modern C++ TurboModules, bypassing traditional bridge serializations. This architecture allows 4-bit quantized models to download on demand, execute on mobile hardware delegates, and push real-time token updates without blocking JavaScript gesture interactions.

By delegating the autoregressive generation loop to dedicated native background threads, the JavaScript event loop and main UI thread remain completely free to process animations, taps, and keyboard events at 60/120 FPS.

Step 1: Exporting and Quantizing Models (INT4 with torchao)

Deploying an SLM to mobile hardware begins with converting a PyTorch model into an optimized .pte binary using 4-bit weight-only quantization via torchao. Quantizing weights to INT4 slashes the resident memory footprint of a 1B parameter model from over 2.5 GB to under 800 MB, fitting well inside mobile operating system thresholds. This offline compilation step partitions model operators to mobile backends such as XNNPACK, preparing the weights for zero-copy memory mapping on iOS and Android.

[!TIP] If you prefer not to compile binaries yourself, Software Mansion hosts verified, pre-exported 4-bit .pte checkpoints and tokenizers directly on Hugging Face. You can load those using built-in presets (LLAMA3_2_1B) out-of-the-box.

1. Environment Setup

To export and quantize custom weights locally, set up a Python 3.10+ environment with PyTorch, TorchAO, and ExecuTorch:

pip install torch torchvision
pip install torchao
pip install executorch
pip install transformers

2. Export and Quantization Script

The following script loads meta-llama/Llama-3.2-1B-Instruct, applies 4-bit groupwise weight quantization (group_size=128), lowers operators to the mobile-optimized XNNPACK CPU delegate, and outputs the .pte executable:

# export_llama_int4.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from torchao.quantization.quant_api import quantize_, int4_weight_only
from executorch.exir import to_edge, EdgeCompileConfig
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
OUTPUT_FILE = "./llama3_2_1b_int4.pte"

print("[1/4] Loading Hugging Face model and tokenizer...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float32,
    device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

print("[2/4] Applying INT4 groupwise weight quantization...")
# Reduces weight precision to 4-bit, dropping 1B params under 800MB
quantize_(model, int4_weight_only(group_size=128))

print("[3/4] Tracing and lowering graph to ExecuTorch Edge IR...")
sample_input = (torch.zeros((1, 128), dtype=torch.long),)
edge_program = to_edge(
    torch.export.export(model, sample_input),
    compile_config=EdgeCompileConfig(_check_ir=False)
)

print("[4/4] Partitioning for XNNPACK and serializing .pte binary...")
delegated_program = edge_program.to_backend(XnnpackPartitioner())
exec_program = delegated_program.to_executorch()

with open(OUTPUT_FILE, "wb") as f:
    f.write(exec_program.buffer)

print(f"Export completed successfully: {OUTPUT_FILE}")

Once exported, host your .pte file and tokenizer assets on a secure CDN or cloud bucket with HTTP Range request support for resumable downloads.

Step 2: Installing and Configuring react-native-executorch

Integrating ExecuTorch into React Native requires the New Architecture (TurboModules) and the official react-native-executorch package alongside a resource fetcher adapter. The setup entails installing the library, configuring your bundler to recognize .pte and .bin model extensions, and initializing the native runtime at your app's root. Once initialized, the native engine handles asynchronous binary fetching, caching, and thread management automatically.

[!IMPORTANT] react-native-executorch strictly requires the React Native New Architecture (React Native 0.76+). If you are using Expo, you must use a development build (npx expo run:ios or npx expo run:android), as custom native C++ runtimes cannot run inside the standard Expo Go sandbox.

1. Package Installation

Install react-native-executorch and its Expo resource fetcher adapter (works with both bare React Native apps with Expo modules and managed Expo workflows):

# Core package
npm install react-native-executorch

# Resource fetcher for on-demand model downloading and caching
npm install react-native-executorch-expo-resource-fetcher expo-file-system expo-asset

For iOS bare workflows, install the CocoaPods dependencies:

cd ios && pod install && cd ..

2. Configure Metro Bundler (metro.config.js)

If you plan to bundle lightweight models or local test binaries directly as app assets, configure Metro to resolve .pte and .bin file types:

// metro.config.js
const { getDefaultConfig } = require('expo/metro-config'); // or '@react-native/metro-config'

const config = getDefaultConfig(__dirname);

config.resolver.assetExts.push('pte', 'bin');

module.exports = config;

3. Initialize Runtime at Application Entry Point

Initialize react-native-executorch before mounting any components that execute inference. Call initExecutorch in your root entry file (App.tsx or index.ts):

// App.tsx
import React from 'react';
import { initExecutorch } from 'react-native-executorch';
import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';
import { ChatScreen } from './src/ChatScreen';

// Initialize ExecuTorch native engine with persistent resource caching
initExecutorch({
  resourceFetcher: ExpoResourceFetcher,
});

export default function App() {
  return <ChatScreen />;
}

Step 3: Implementing Real-Time Token Streaming with useLLM

Streaming inference tokens smoothly into a chat interface is handled declaratively using the useLLM hook provided by react-native-executorch. The hook exposes reactive properties for asset download progress, model initialization status, live token generation, and generation interruption. By binding the reactive response stream and messageHistory to React Native state, you achieve fluid 60 FPS chat updates without writing custom native bridge boilerplate.

Complete Streaming Chat Screen (ChatScreen.tsx)

The following production-ready component downloads the quantized Llama 3.2 1B model on demand, tracks caching progress, streams tokens into the UI in real time, and allows users to stop generation mid-stream:

// src/ChatScreen.tsx
import React, { useState, useRef, useEffect } from 'react';
import {
  View,
  Text,
  TextInput,
  TouchableOpacity,
  FlatList,
  ActivityIndicator,
  StyleSheet,
  SafeAreaView,
  KeyboardAvoidingView,
  Platform,
} from 'react-native';
import { useLLM, LLAMA3_2_1B } from 'react-native-executorch';

export function ChatScreen() {
  const [input, setInput] = useState('');
  const flatListRef = useRef<FlatList>(null);

  // Initialize LLM using Software Mansion's pre-configured Llama 3.2 1B preset
  const llm = useLLM({
    model: LLAMA3_2_1B,
  });

  // Auto-scroll as tokens stream in
  useEffect(() => {
    if (llm.response.length > 0 || llm.messageHistory.length > 0) {
      flatListRef.current?.scrollToEnd({ animated: true });
    }
  }, [llm.response, llm.messageHistory]);

  const handleSend = async () => {
    const prompt = input.trim();
    if (!prompt || !llm.isReady || llm.isGenerating) return;

    setInput('');
    try {
      await llm.sendMessage(prompt);
    } catch (err) {
      console.error('Inference error:', err);
    }
  };

  const handleInterrupt = () => {
    llm.interrupt();
  };

  // State 1: Model downloading or initializing
  if (!llm.isReady) {
    const progressPercent = Math.round((llm.downloadProgress ?? 0) * 100);
    return (
      <SafeAreaView style={styles.loadingContainer}>
        <ActivityIndicator size="large" color="#0284c7" />
        <Text style={styles.loadingTitle}>Loading On-Device SLM</Text>
        <Text style={styles.loadingSubtitle}>
          {progressPercent > 0 && progressPercent < 100
            ? `Downloading model: ${progressPercent}%`
            : 'Initializing ExecuTorch runtime...'}
        </Text>
        <View style={styles.progressBarBackground}>
          <View style={[styles.progressBarFill, { width: `${progressPercent}%` }]} />
        </View>
      </SafeAreaView>
    );
  }

  // State 2: Active Chat Interface
  return (
    <SafeAreaView style={styles.container}>
      <KeyboardAvoidingView
        style={styles.container}
        behavior={Platform.OS === 'ios' ? 'padding' : undefined}
      >
        <View style={styles.header}>
          <Text style={styles.headerTitle}>Llama 3.2 1B (On-Device)</Text>
          <View style={styles.badge}>
            <Text style={styles.badgeText}>Offline INT4</Text>
          </View>
        </View>

        <FlatList
          ref={flatListRef}
          data={llm.messageHistory}
          keyExtractor={(_, index) => index.toString()}
          contentContainerStyle={styles.messageList}
          renderItem={({ item }) => (
            <View
              style={[
                styles.messageBubble,
                item.role === 'user' ? styles.userBubble : styles.assistantBubble,
              ]}
            >
              <Text style={styles.roleLabel}>{item.role === 'user' ? 'You' : 'Llama'}</Text>
              <Text
                style={[
                  styles.messageText,
                  item.role === 'user' ? styles.userText : styles.assistantText,
                ]}
              >
                {item.content}
              </Text>
            </View>
          )}
          ListFooterComponent={
            llm.isGenerating ? (
              <View style={[styles.messageBubble, styles.assistantBubble]}>
                <Text style={styles.roleLabel}>Llama (generating...)</Text>
                <Text style={[styles.messageText, styles.assistantText]}>
                  {llm.response || '...'}
                </Text>
              </View>
            ) : null
          }
        />

        <View style={styles.inputContainer}>
          <TextInput
            style={styles.textInput}
            value={input}
            onChangeText={setInput}
            placeholder="Type a message..."
            placeholderTextColor="#94a3b8"
            editable={!llm.isGenerating}
            onSubmitEditing={handleSend}
            returnKeyType="send"
          />

          {llm.isGenerating ? (
            <TouchableOpacity style={styles.stopButton} onPress={handleInterrupt}>
              <Text style={styles.buttonText}>Stop</Text>
            </TouchableOpacity>
          ) : (
            <TouchableOpacity
              style={[styles.sendButton, !input.trim() && styles.disabledButton]}
              onPress={handleSend}
              disabled={!input.trim()}
            >
              <Text style={styles.buttonText}>Send</Text>
            </TouchableOpacity>
          )}
        </View>
      </KeyboardAvoidingView>
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1, backgroundColor: '#f8fafc' },
  loadingContainer: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
    backgroundColor: '#f8fafc',
    padding: 24,
  },
  loadingTitle: { fontSize: 18, fontWeight: '700', color: '#0f172a', marginTop: 16 },
  loadingSubtitle: { fontSize: 14, color: '#64748b', marginTop: 8 },
  progressBarBackground: {
    width: '80%',
    height: 6,
    backgroundColor: '#e2e8f0',
    borderRadius: 3,
    marginTop: 16,
    overflow: 'hidden',
  },
  progressBarFill: { height: '100%', backgroundColor: '#0284c7' },
  header: {
    paddingHorizontal: 16,
    paddingVertical: 12,
    backgroundColor: '#ffffff',
    borderBottomWidth: 1,
    borderColor: '#e2e8f0',
    flexDirection: 'row',
    alignItems: 'center',
    justifyContent: 'space-between',
  },
  headerTitle: { fontSize: 16, fontWeight: '700', color: '#0f172a' },
  badge: { backgroundColor: '#e0f2fe', paddingHorizontal: 8, paddingVertical: 4, borderRadius: 6 },
  badgeText: { fontSize: 12, fontWeight: '600', color: '#0284c7' },
  messageList: { padding: 16, gap: 12 },
  messageBubble: { padding: 12, borderRadius: 12, maxWidth: '85%' },
  userBubble: { alignSelf: 'flex-end', backgroundColor: '#0284c7' },
  assistantBubble: { alignSelf: 'flex-start', backgroundColor: '#ffffff', borderWidth: 1, borderColor: '#e2e8f0' },
  roleLabel: { fontSize: 11, fontWeight: '600', color: '#94a3b8', marginBottom: 4 },
  messageText: { fontSize: 15, lineHeight: 22 },
  userText: { color: '#ffffff' },
  assistantText: { color: '#0f172a' },
  inputContainer: {
    flexDirection: 'row',
    padding: 12,
    backgroundColor: '#ffffff',
    borderTopWidth: 1,
    borderColor: '#e2e8f0',
    alignItems: 'center',
  },
  textInput: {
    flex: 1,
    backgroundColor: '#f1f5f9',
    borderRadius: 8,
    paddingHorizontal: 14,
    paddingVertical: 10,
    fontSize: 15,
    color: '#0f172a',
    marginRight: 8,
  },
  sendButton: {
    backgroundColor: '#0284c7',
    paddingHorizontal: 18,
    paddingVertical: 10,
    borderRadius: 8,
  },
  stopButton: {
    backgroundColor: '#ef4444',
    paddingHorizontal: 18,
    paddingVertical: 10,
    borderRadius: 8,
  },
  disabledButton: { backgroundColor: '#cbd5e1' },
  buttonText: { color: '#ffffff', fontWeight: '600', fontSize: 14 },
});

Step 4: Mobile Memory Budgets, Context Windows, and Production Pitfalls

Deploying on-device generative AI models introduces real-world mobile limitations around RAM headroom, thermal throttling, and app binary size constraints. Exceeding platform memory limits triggers immediate OS watchdog terminations (SIGKILL), while continuous generation on mobile SoCs can induce severe thermal throttling. Addressing these constraints requires dynamic asset downloading over Wi-Fi, strict context window clamping, and graceful error handling for device lifecycle transitions.

1. Never Bundle .pte Weights Inside the App Store Binary

Embedding a 700 MB .pte model directly into ipa or apk assets will exceed Apple’s and Google’s cellular download ceilings (typically 200 MB OTA limit) and inflate app update payloads.

• Store your .pte binaries on a CDN or Hugging Face repository.
• Use react-native-executorch's resource fetcher, which downloads the model to persistent app storage upon initial launch.
• Subsequent app launches instantly load the cached binary from local storage via memory mapping (mmap), consuming near-zero download bandwidth.

2. Context Window and KV Cache Overflow

On-device inference reserves an in-memory Key-Value (KV) cache for multi-turn history. If conversational turns accumulate unchecked, ExecuTorch will fail or the OS will terminate the app due to memory pressure.

To prevent context overflow:

• Cap maximum generation turns by truncating older messages from history.
• If using generate() in manual mode instead of sendMessage(), implement a sliding window strategy retaining only the initial system prompt and the last 3–5 conversation turns.

// Truncate message history to prevent KV cache memory blowout
const prunedHistory = messageHistory.slice(-6);

3. Thermal Throttling & Battery Conservation

Autoregressive token generation places heavy sustained load across CPU cores or NPUs. Continuous generation generates substantial heat, causing iOS and Android to aggressively downclock processor frequencies and drain battery.

• Enforce conservative generation limits (maxTokens: 150 to 256) per request.
• Always provide an interruption trigger via llm.interrupt() so users can abort long responses immediately.
• Prevent background inference: pause or terminate generation when the app transitions into background via React Native's AppState API.

Frequently Asked Questions (FAQ)

Adopting on-device inference with ExecuTorch raises important technical questions regarding device compatibility, hardware acceleration, and runtime stability. Understanding how mobile operating systems handle memory allocation and how backends leverage NPUs is essential for delivering robust native experiences. Below are direct answers to the most common engineering questions encountered when shipping SLMs in production mobile apps.

What are the minimum device hardware requirements for running a 1B model?

A 4-bit quantized 1B parameter model requires approximately 750 MB to 900 MB of resident RAM for model weights and active KV cache buffers. For stable multi-app multitasking without OS memory pressure terminations, target devices with at least 4 GB of RAM on Android and 3 GB of RAM on iOS (iPhone 11 or newer).

How does token streaming work without dropping UI frames?

react-native-executorch runs the ExecuTorch C++ runtime on a dedicated background POSIX thread. As each token is generated, it writes directly to native memory and notifies the React Native TurboModule layer. The UI updates through the hook's reactive response property, keeping the JavaScript thread and main UI frame rate decoupled from tensor matrix calculations.

Can custom fine-tuned models be loaded into react-native-executorch?

Yes. Any model exported to the .pte format using ExecuTorch's export toolchain can be loaded by passing custom file URIs or URLs into modelSource, tokenizerSource, and tokenizerConfigSource. You are not limited to pre-configured library presets.

About the Author

Furkan Çetinkaya is a Mobile-focused Software Developer specializing in React Native, native bridge integrations (Kotlin & Swift), and supporting backend services. Experienced in maintaining high-impact mobile applications and developer SDKs.

• GitHub
• LinkedIn