Building Production-Ready Computer Vision in React Native: VisionCamera, Custom C++ Frame Processors, and Real-Time On-Device Inference

Real-time computer vision in mobile apps demands sub-frame latency and efficient memory management between the camera hardware and inference runtime. By coupling react-native-vision-camera with custom C++ frame processors and lightweight inference engines like ONNX Runtime Mobile or TensorFlow Lite, React Native applications can process live camera streams directly on native worker threads without blocking the JavaScript runtime. This practical guide walks through native toolchain setup, writing cross-platform C++ frame processing logic, and piping structured predictions back to your React Native UI.

Why Native Frame Processors Are Essential for Mobile Computer Vision

Standard React Native bridges and asynchronous event loops cannot handle raw video frames, as streaming 30 to 60 megabyte-sized buffers across threads introduces severe latency, garbage collection pressure, and dropped frames. Real-world computer vision requires executing inference directly against native frame memory pointers before returning only lightweight, structured results (such as bounding boxes or classification labels) back to the JavaScript thread. Running this pipeline inside a native frame processor decouples compute-heavy tensor operations from UI rendering, keeping your application responsive.

In legacy React Native camera architectures, developers frequently resorted to capturing image snapshots, base64-encoding the byte arrays, and sending them over the asynchronous bridge to JavaScript. At 30 FPS with high-definition frames, this approach quickly causes out-of-memory (OOM) crashes and thermal throttling.

Modern computer vision workflows avoid serialization entirely. Native camera libraries yield direct memory addresses to the frame buffer (CVPixelBufferRef on iOS and ImageProxy / AHardwareBuffer on Android). By leveraging C++ frame processor plugins, developers inspect and transform raw pixel buffers directly inside an isolated native execution context, running inference models without touching the main JavaScript thread.

Architectural Overview: High-Throughput Camera Pipelines in React Native

Modern mobile vision pipelines rely on a unidirectional data flow that separates continuous sensor capture from lightweight UI updates. The camera sensor outputs hardware buffers directly into a dedicated native worker thread, where a custom C++ processor preprocesses the pixels and executes model inference. Once predictions are ready, small JSON-compatible primitives are dispatched back to the React Native JavaScript thread using worklets.

This architecture ensures that:

1. Memory remains localized: Raw camera frames never leave native memory and are automatically recycled by the operating system once the processing cycle completes.
2. The JavaScript thread remains free: Complex mathematical operations, pixel normalization, and matrix multiplications run entirely in C++.
3. UI rendering is decoupled: If a complex model requires 45 milliseconds to compute, the camera preview remains buttery smooth at 60 FPS while detection overlays update at ~22 FPS.

Step 1: Project Configuration and Native Dependencies

Configuring real-time vision processing requires installing the core camera and worklet modules, followed by setting up platform build systems for C++ compilation. Both iOS (Podfile and clang toolchains) and Android (CMakeLists.txt with NDK) must link to your native model runtime, such as TensorFlow Lite or ONNX Runtime Mobile. Verifying camera permissions and target compilation standards during initial setup avoids elusive runtime symbol errors.

First, install react-native-vision-camera and react-native-worklets-core into your project:

npm install react-native-vision-camera react-native-worklets-core

Ensure your babel.config.js includes the worklets plugin:

module.exports = {
  presets: ['module:@react-native/babel-preset'],
  plugins: [
    ['react-native-worklets-core/plugin'],
  ],
};

iOS Setup and Permissions

Add camera usage descriptions to ios/YourApp/Info.plist:

<key>NSCameraUsageDescription</key>
<string>This application requires camera access for real-time visual inspection.</string>

Ensure your ios/Podfile specifies C++20 or C++17 support:

post_install do |installer|
  installer.pods_project.targets.each do |target|
    target.build_configurations.each do |config|
      config.build_settings['CLANG_CXX_LANGUAGE_STANDARD'] = 'c++20'
    end
  end
end

Android Setup and Permissions

Add camera permissions to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />

In android/app/build.gradle, configure the NDK and CMake build paths:

android {
    compileSdkVersion 34

    defaultConfig {
        externalNativeBuild {
            cmake {
                cppFlags "-std=c++20", "-O3", "-frtti", "-fexceptions"
                arguments "-DANDROID_STL=c++_shared"
            }
        }
    }

    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
            version "3.22.1"
        }
    }
}

Step 2: Building the Cross-Platform C++ and Native Frame Processor Plugin

A production-grade frame processor separates platform-specific buffer extraction from the portable C++ inference engine. On iOS, Objective-C++ unpacks CMSampleBufferRef to access raw BGRA or YUV byte pointers, while Android extracts pixel planes from an ImageProxy via JNI or NDK handles. Once retrieved, these raw pointers are passed to a portable C++ class that handles tensor pre-processing, inference execution, and returns bounding box coordinates back to the platform wrapper.

The Portable C++ Vision Core

Create cpp/VisionCore.hpp to handle inference logic independently of the mobile platform:

#pragma once

#include <vector>
#include <cstdint>
#include <string>

struct DetectionResult {
    float x;
    float y;
    float width;
    float height;
    float confidence;
    std::string label;
};

class VisionCore {
public:
    VisionCore();
    ~VisionCore();

    bool initialize(const std::string& modelPath);
    std::vector<DetectionResult> processFrame(
        const uint8_t* pixelBuffer,
        int width,
        int height,
        int bytesPerRow,
        bool isBGRA
    );

private:
    bool isModelLoaded = false;
    // Model session handles (e.g., Ort::Session or TfLiteModel) live here
};

Implement frame evaluation in cpp/VisionCore.cpp:

#include "VisionCore.hpp"
#include <algorithm>

VisionCore::VisionCore() {}
VisionCore::~VisionCore() {}

bool VisionCore::initialize(const std::string& modelPath) {
    // Initialize ONNX Runtime / TFLite session here
    isModelLoaded = !modelPath.empty();
    return isModelLoaded;
}

std::vector<DetectionResult> VisionCore::processFrame(
    const uint8_t* pixelBuffer,
    int width,
    int height,
    int bytesPerRow,
    bool isBGRA
) {
    std::vector<DetectionResult> results;
    if (!pixelBuffer || width <= 0 || height <= 0) {
        return results;
    }

    // Example: Read pixel intensity or feed input tensor directly
    // Preprocess: Resize frame buffer into model input tensor (e.g., 224x224x3)
    // Run inference: session.Run(...)

    // Simulated detection output for illustration:
    results.push_back({
        .x = 0.15f,
        .y = 0.20f,
        .width = 0.45f,
        .height = 0.35f,
        .confidence = 0.92f,
        .label = "target_object"
    });

    return results;
}

iOS Native Plugin Registration

VisionCamera allows custom plugins using FrameProcessorPlugin. According to the VisionCamera Frame Processor Guide, you register native plugins using the plugin registry.

Create ios/ObjectDetectorPlugin.mm:

#import <VisionCamera/FrameProcessorPlugin.h>
#import <VisionCamera/FrameProcessorPluginRegistry.h>
#import <VisionCamera/Frame.h>
#import <CoreMedia/CoreMedia.h>
#include "VisionCore.hpp"

static VisionCore gVisionCore;

@interface ObjectDetectorPlugin : FrameProcessorPlugin
@end

@implementation ObjectDetectorPlugin

- (instancetype)initWithOptions:(NSDictionary*)options {
    if (self = [super initWithOptions:options]) {
        static dispatch_once_t onceToken;
        dispatch_once(&onceToken, ^{
            // Load your model asset from app bundle
            NSString* modelPath = [[NSBundle mainBundle] pathForResource:@"detector" ofType:@"tflite"];
            if (modelPath) {
                gVisionCore.initialize([modelPath UTF8String]);
            }
        });
    }
    return self;
}

- (id)callback:(Frame*)frame withArguments:(NSDictionary*)arguments {
    CMSampleBufferRef sampleBuffer = frame.buffer;
    CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    if (!pixelBuffer) {
        return @[];
    }

    CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);

    uint8_t* baseAddress = (uint8_t*)CVPixelBufferGetBaseAddress(pixelBuffer);
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
    OSType pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);

    bool isBGRA = (pixelFormat == kCVPixelFormatType_32BGRA);

    auto detections = gVisionCore.processFrame(baseAddress, (int)width, (int)height, (int)bytesPerRow, isBGRA);

    CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);

    NSMutableArray* output = [NSMutableArray arrayWithCapacity:detections.size()];
    for (const auto& item : detections) {
        [output addObject:@{
            @"x": @(item.x),
            @"y": @(item.y),
            @"width": @(item.width),
            @"height": @(item.height),
            @"confidence": @(item.confidence),
            @"label": [NSString stringWithUTF8String:item.label.c_str()]
        }];
    }

    return output;
}

VISION_EXPORT_FRAME_PROCESSOR(ObjectDetectorPlugin, detectObjects)

@end

Android Native Plugin Registration

Create the corresponding Kotlin frame processor plugin in android/app/src/main/java/com/yourapp/ObjectDetectorPlugin.kt:

package com.yourapp

import androidx.camera.core.ImageProxy
import com.mrousavy.camera.frameprocessors.Frame
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin
import com.mrousavy.camera.frameprocessors.VisionCameraProxy

class ObjectDetectorPlugin(proxy: VisionCameraProxy, options: Map<String, Any>?) : FrameProcessorPlugin() {

    init {
        // Load native library containing VisionCore C++ logic
        System.loadLibrary("vision_camera_native")
        initModelNative("/data/local/tmp/detector.tflite")
    }

    override fun callback(frame: Frame, params: Map<String, Any>?): Any {
        val imageProxy: ImageProxy = frame.imageProxy ?: return emptyList<Map<String, Any>>()
        
        // Call C++ via JNI passing direct byte buffer or planes
        val planes = imageProxy.planes
        if (planes.isEmpty()) return emptyList<Map<String, Any>>()

        val buffer = planes[0].buffer
        val detections = processBufferNative(
            buffer,
            imageProxy.width,
            imageProxy.height,
            planes[0].rowStride
        )

        return detections
    }

    private external fun initModelNative(modelPath: String): Boolean
    private external fun processBufferNative(
        buffer: java.nio.ByteBuffer,
        width: Int,
        height: Int,
        rowStride: Int
    ): List<Map<String, Any>>
}

Register the plugin in your PackageList or initialization class:

import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry

class MainApplication : Application(), ReactApplication {
    override fun onCreate() {
        super.onCreate()
        FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectObjects") { proxy, options ->
            ObjectDetectorPlugin(proxy, options)
        }
    }
}

Step 3: Integrating the Frame Processor in React Native UI

React Native interfaces with custom native plugins inside the useFrameProcessor worklet hook, which runs on an isolated secondary thread. Because vision frame processors can fire dozens of times per second, developers must throttle updates before using runOnJS to send state back to React. This guarantees high-frequency camera detection without exhausting the React rendering cycle.

Here is a practical, production-ready screen implementation:

import React, { useEffect, useState } from 'react';
import { StyleSheet, Text, View } from 'react-native';
import {
  Camera,
  useCameraDevice,
  useCameraFormat,
  useFrameProcessor,
  VisionCameraProxy,
} from 'react-native-vision-camera';
import { Worklets } from 'react-native-worklets-core';

interface Detection {
  x: number;
  y: number;
  width: number;
  height: number;
  confidence: number;
  label: string;
}

// Initialize native plugin handle
const plugin = VisionCameraProxy.initFrameProcessorPlugin('detectObjects', {});

export const VisionCameraScreen: React.FC = () => {
  const [hasPermission, setHasPermission] = useState(false);
  const [detections, setDetections] = useState<Detection[]>([]);

  const device = useCameraDevice('back');
  
  // Choose an optimal format (e.g. 1080p or 720p) to balance resolution and model latency
  const format = useCameraFormat(device, [
    { videoResolution: { width: 1280, height: 720 } },
    { fps: 30 },
  ]);

  useEffect(() => {
    (async () => {
      const status = await Camera.requestCameraPermission();
      setHasPermission(status === 'granted');
    })();
  }, []);

  // Worklet callback that sends results to the React JavaScript thread safely
  const updateDetectionsJS = Worklets.createRunOnJS((results: Detection[]) => {
    setDetections(results);
  });

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet';
    if (plugin == null) {
      return;
    }

    // Call native C++ plugin
    const rawResults = plugin.call(frame) as unknown as Detection[];

    if (rawResults && rawResults.length > 0) {
      updateDetectionsJS(rawResults);
    }
  }, [plugin, updateDetectionsJS]);

  if (!hasPermission) {
    return (
      <View style={styles.centered}>
        <Text style={styles.statusText}>Awaiting camera permissions...</Text>
      </View>
    );
  }

  if (device == null) {
    return (
      <View style={styles.centered}>
        <Text style={styles.statusText}>No suitable camera hardware found.</Text>
      </View>
    );
  }

  return (
    <View style={styles.container}>
      <Camera
        style={StyleSheet.absoluteFill}
        device={device}
        format={format}
        isActive={true}
        frameProcessor={frameProcessor}
        pixelFormat="native"
      />

      {/* Detection HUD Bounding Boxes */}
      <View style={StyleSheet.absoluteFill} pointerEvents="none">
        {detections.map((item, index) => (
          <View
            key={`${item.label}-${index}`}
            style={[
              styles.box,
              {
                left: `${item.x * 100}%`,
                top: `${item.y * 100}%`,
                width: `${item.width * 100}%`,
                height: `${item.height * 100}%`,
              },
            ]}
          >
            <Text style={styles.boxLabel}>
              {item.label} ({(item.confidence * 100).toFixed(0)}%)
            </Text>
          </View>
        ))}
      </View>
    </View>
  );
};

const styles = StyleSheet.create({
  container: { flex: 1, backgroundColor: '#000' },
  centered: { flex: 1, justifyContent: 'center', alignItems: 'center' },
  statusText: { color: '#fff', fontSize: 16 },
  box: {
    position: 'absolute',
    borderWidth: 2,
    borderColor: '#00FF66',
    borderRadius: 4,
    backgroundColor: 'rgba(0, 255, 102, 0.1)',
  },
  boxLabel: {
    backgroundColor: '#00FF66',
    color: '#000',
    fontSize: 12,
    fontWeight: 'bold',
    paddingHorizontal: 4,
    alignSelf: 'flex-start',
  },
});

Step 4: Production Pitfalls and Performance Best Practices

Deploying computer vision to production requires vigilant management of hardware constraints, memory lifecycle, and thermal limits across heterogeneous mobile devices. Failing to drop backpressured frames or running inference on oversized 4K buffers will swiftly trigger thermal throttling and OS watchdog terminations. Adhering to strict native buffer recycling patterns and downscaling sensor dimensions at the driver level ensures consistent performance in consumer environments.

1. Match Hardware Capture Format to Model Input Dimensions

Mobile camera sensors frequently capture in 4K (3840×2160) or 1080p by default. Running color space conversions or bilateral filters on millions of pixels when your model only accepts 224×224 or 640×640 inputs wastes immense CPU and memory bandwidth.

• Always restrict camera resolution using useCameraFormat to target 720p (1280×720) or 480p (640×480).
• Select pixelFormat="yuv" or "native" to prevent the driver from performing an unneeded BGRA conversion if your inference engine natively ingests YUV/greyscale data.

2. Guard Against Memory Leaks and Buffer Retention

Mobile operating systems provide a fixed pool of hardware image buffers (typically 3 to 5 buffers in the pipeline). If your C++ processor fails to release base addresses or holds a reference to a frame across asynchronous dispatch boundaries, the camera feed stalls completely.

• iOS: Always pair CVPixelBufferLockBaseAddress with CVPixelBufferUnlockBaseAddress inside a try/finally block or RAII guard.
• Android: Close the ImageProxy immediately after copying required bytes into model memory. Failing to close the proxy results in the camera sensor dropping subsequent frames.

3. Handle Frame Dropping and Backpressure

If model inference takes 60ms, the processor can only sustain 16 FPS. Attempting to process every single 30 FPS camera frame creates backpressure queues that lead to input lag and memory inflation.

• Execute frame processing synchronously within the frame processor thread, allowing the camera hardware driver to automatically discard intermediate frames when the processor is busy.
• Alternatively, maintain an atomic std::atomic<bool> isProcessing{false} flag in C++ to instantly skip frames whenever an active inference cycle is running.

4. Optimize On-Device Models Before Deployment

Running full float32 neural networks will rapidly drain the device battery and generate excessive heat. Refer to the ONNX Runtime Mobile Documentation to apply quantization (INT8/FP16) and prune unnecessary operators prior to bundling model artifacts into your mobile binary.

Frequently Asked Questions

Understanding common edge cases in mobile computer vision helps teams troubleshoot pipeline stalls and performance degradation early in the development lifecycle. The answers below address real-world deployment challenges encountered when shipping VisionCamera and C++ plugins to end users.

How do I handle pixel format conversions between camera frames and model inputs?

Camera hardware predominantly outputs frames in YUV formats (such as NV12 on iOS or YUV_420_888 on Android), while computer vision models typically require planar RGB or BGR float tensors. Rather than converting the entire image to RGB on the JavaScript or platform layer, perform planar slicing and normalization directly inside your C++ core. For models that only evaluate luminance (such as barcode scanning or edge detection), you can feed the Y-plane directly to the model as a greyscale matrix, bypassing color space conversion entirely.

Can I run model inference asynchronously without dropping camera frames?

Yes, but you must avoid holding references to native frame buffers across asynchronous threads. Because mobile hardware frame pools are small, keeping a frame alive while an asynchronous thread executes inference will exhaust the camera pool and freeze the preview. If asynchronous processing is required, copy the raw pixel bytes into an internal, pre-allocated C++ memory buffer during the synchronous frame callback, release the original camera frame immediately, and pass the cloned buffer to your background inference thread pool.

How do I prevent thermal throttling and battery drain during continuous scanning?

Thermal buildup occurs when the GPU or NPU continuously executes heavy matrix multiplications at maximum clock speeds. To maintain low device temperatures during extended camera sessions, throttle your inference frequency to 10–15 FPS using a timestamp delta check inside your worklet. Additionally, select the lowest acceptable sensor resolution from useCameraFormat that meets your model's input dimension requirements, and employ INT8 quantized models to take advantage of low-power mobile DSP and NPU accelerators.

About the Author

Furkan Çetinkaya is a Mobile-focused Software Developer specializing in React Native, native bridge integrations (Kotlin & Swift), and supporting backend services. Experienced in maintaining high-impact mobile applications and developer SDKs.

• GitHub
• LinkedIn