--- title: Building Production-Ready Computer Vision in React Native: VisionCamera, Custom C++ Frame Processors, and Real-Time On-Device Inference publishedAt: 2026-09-09 summary: A pragmatic, production-focused guide to implementing real-time on-device computer vision in React Native using VisionCamera, native C++ frame processors, and mobile inference engines. --- # Building Production-Ready Computer Vision in React Native: VisionCamera, Custom C++ Frame Processors, and Real-Time On-Device Inference Real-time computer vision in mobile apps demands sub-frame latency and efficient memory management between the camera hardware and inference runtime. By coupling `react-native-vision-camera` with custom C++ frame processors and lightweight inference engines like ONNX Runtime Mobile or TensorFlow Lite, React Native applications can process live camera streams directly on native worker threads without blocking the JavaScript runtime. This practical guide walks through native toolchain setup, writing cross-platform C++ frame processing logic, and piping structured predictions back to your React Native UI. --- ## Why Native Frame Processors Are Essential for Mobile Computer Vision Standard React Native bridges and asynchronous event loops cannot handle raw video frames, as streaming 30 to 60 megabyte-sized buffers across threads introduces severe latency, garbage collection pressure, and dropped frames. Real-world computer vision requires executing inference directly against native frame memory pointers before returning only lightweight, structured results (such as bounding boxes or classification labels) back to the JavaScript thread. Running this pipeline inside a native frame processor decouples compute-heavy tensor operations from UI rendering, keeping your application responsive. In legacy React Native camera architectures, developers frequently resorted to capturing image snapshots, base64-encoding the byte arrays, and sending them over the asynchronous bridge to JavaScript. At 30 FPS with high-definition frames, this approach quickly causes out-of-memory (OOM) crashes and thermal throttling. Modern computer vision workflows avoid serialization entirely. Native camera libraries yield direct memory addresses to the frame buffer (`CVPixelBufferRef` on iOS and `ImageProxy` / `AHardwareBuffer` on Android). By leveraging C++ frame processor plugins, developers inspect and transform raw pixel buffers directly inside an isolated native execution context, running inference models without touching the main JavaScript thread. --- ## Architectural Overview: High-Throughput Camera Pipelines in React Native Modern mobile vision pipelines rely on a unidirectional data flow that separates continuous sensor capture from lightweight UI updates. The camera sensor outputs hardware buffers directly into a dedicated native worker thread, where a custom C++ processor preprocesses the pixels and executes model inference. Once predictions are ready, small JSON-compatible primitives are dispatched back to the React Native JavaScript thread using worklets. ```mermaid flowchart TD A[Camera Hardware Sensor] -->|Native Video Stream| B[VisionCamera Frame Processor Thread] B -->|CVPixelBuffer / ImageProxy| C[Custom Native Plugin iOS / Android] C -->|Raw Byte Pointer| D[C++ Vision Core / Inference Engine] D -->|TFLite / ONNX Runtime| E[Raw Tensor Output] E -->|Bounding Boxes & Confidences| C C -->|JSI Primitives Object| F[JS Worklet Context] F -->|runOnJS Throttled Update| G[React Native UI Component] G -->|Render| H[SVG / Canvas HUD Overlay] ``` This architecture ensures that: 1. **Memory remains localized:** Raw camera frames never leave native memory and are automatically recycled by the operating system once the processing cycle completes. 2. **The JavaScript thread remains free:** Complex mathematical operations, pixel normalization, and matrix multiplications run entirely in C++. 3. **UI rendering is decoupled:** If a complex model requires 45 milliseconds to compute, the camera preview remains buttery smooth at 60 FPS while detection overlays update at ~22 FPS. --- ## Step 1: Project Configuration and Native Dependencies Configuring real-time vision processing requires installing the core camera and worklet modules, followed by setting up platform build systems for C++ compilation. Both iOS (`Podfile` and clang toolchains) and Android (`CMakeLists.txt` with NDK) must link to your native model runtime, such as TensorFlow Lite or ONNX Runtime Mobile. Verifying camera permissions and target compilation standards during initial setup avoids elusive runtime symbol errors. First, install [react-native-vision-camera](https://github.com/mrousavy/react-native-vision-camera) and `react-native-worklets-core` into your project: ```bash npm install react-native-vision-camera react-native-worklets-core ``` Ensure your `babel.config.js` includes the worklets plugin: ```javascript module.exports = { presets: ['module:@react-native/babel-preset'], plugins: [ ['react-native-worklets-core/plugin'], ], }; ``` ### iOS Setup and Permissions Add camera usage descriptions to `ios/YourApp/Info.plist`: ```xml NSCameraUsageDescription This application requires camera access for real-time visual inspection. ``` Ensure your `ios/Podfile` specifies C++20 or C++17 support: ```ruby post_install do |installer| installer.pods_project.targets.each do |target| target.build_configurations.each do |config| config.build_settings['CLANG_CXX_LANGUAGE_STANDARD'] = 'c++20' end end end ``` ### Android Setup and Permissions Add camera permissions to `android/app/src/main/AndroidManifest.xml`: ```xml ``` In `android/app/build.gradle`, configure the NDK and CMake build paths: ```groovy android { compileSdkVersion 34 defaultConfig { externalNativeBuild { cmake { cppFlags "-std=c++20", "-O3", "-frtti", "-fexceptions" arguments "-DANDROID_STL=c++_shared" } } } externalNativeBuild { cmake { path "src/main/cpp/CMakeLists.txt" version "3.22.1" } } } ``` --- ## Step 2: Building the Cross-Platform C++ and Native Frame Processor Plugin A production-grade frame processor separates platform-specific buffer extraction from the portable C++ inference engine. On iOS, Objective-C++ unpacks `CMSampleBufferRef` to access raw BGRA or YUV byte pointers, while Android extracts pixel planes from an `ImageProxy` via JNI or NDK handles. Once retrieved, these raw pointers are passed to a portable C++ class that handles tensor pre-processing, inference execution, and returns bounding box coordinates back to the platform wrapper. ### The Portable C++ Vision Core Create `cpp/VisionCore.hpp` to handle inference logic independently of the mobile platform: ```cpp #pragma once #include #include #include struct DetectionResult { float x; float y; float width; float height; float confidence; std::string label; }; class VisionCore { public: VisionCore(); ~VisionCore(); bool initialize(const std::string& modelPath); std::vector processFrame( const uint8_t* pixelBuffer, int width, int height, int bytesPerRow, bool isBGRA ); private: bool isModelLoaded = false; // Model session handles (e.g., Ort::Session or TfLiteModel) live here }; ``` Implement frame evaluation in `cpp/VisionCore.cpp`: ```cpp #include "VisionCore.hpp" #include VisionCore::VisionCore() {} VisionCore::~VisionCore() {} bool VisionCore::initialize(const std::string& modelPath) { // Initialize ONNX Runtime / TFLite session here isModelLoaded = !modelPath.empty(); return isModelLoaded; } std::vector VisionCore::processFrame( const uint8_t* pixelBuffer, int width, int height, int bytesPerRow, bool isBGRA ) { std::vector results; if (!pixelBuffer || width <= 0 || height <= 0) { return results; } // Example: Read pixel intensity or feed input tensor directly // Preprocess: Resize frame buffer into model input tensor (e.g., 224x224x3) // Run inference: session.Run(...) // Simulated detection output for illustration: results.push_back({ .x = 0.15f, .y = 0.20f, .width = 0.45f, .height = 0.35f, .confidence = 0.92f, .label = "target_object" }); return results; } ``` ### iOS Native Plugin Registration VisionCamera allows custom plugins using `FrameProcessorPlugin`. According to the [VisionCamera Frame Processor Guide](https://react-native-vision-camera.com/docs/guides/frame-processors), you register native plugins using the plugin registry. Create `ios/ObjectDetectorPlugin.mm`: ```objc #import #import #import #import #include "VisionCore.hpp" static VisionCore gVisionCore; @interface ObjectDetectorPlugin : FrameProcessorPlugin @end @implementation ObjectDetectorPlugin - (instancetype)initWithOptions:(NSDictionary*)options { if (self = [super initWithOptions:options]) { static dispatch_once_t onceToken; dispatch_once(&onceToken, ^{ // Load your model asset from app bundle NSString* modelPath = [[NSBundle mainBundle] pathForResource:@"detector" ofType:@"tflite"]; if (modelPath) { gVisionCore.initialize([modelPath UTF8String]); } }); } return self; } - (id)callback:(Frame*)frame withArguments:(NSDictionary*)arguments { CMSampleBufferRef sampleBuffer = frame.buffer; CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); if (!pixelBuffer) { return @[]; } CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly); uint8_t* baseAddress = (uint8_t*)CVPixelBufferGetBaseAddress(pixelBuffer); size_t width = CVPixelBufferGetWidth(pixelBuffer); size_t height = CVPixelBufferGetHeight(pixelBuffer); size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer); OSType pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer); bool isBGRA = (pixelFormat == kCVPixelFormatType_32BGRA); auto detections = gVisionCore.processFrame(baseAddress, (int)width, (int)height, (int)bytesPerRow, isBGRA); CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly); NSMutableArray* output = [NSMutableArray arrayWithCapacity:detections.size()]; for (const auto& item : detections) { [output addObject:@{ @"x": @(item.x), @"y": @(item.y), @"width": @(item.width), @"height": @(item.height), @"confidence": @(item.confidence), @"label": [NSString stringWithUTF8String:item.label.c_str()] }]; } return output; } VISION_EXPORT_FRAME_PROCESSOR(ObjectDetectorPlugin, detectObjects) @end ``` ### Android Native Plugin Registration Create the corresponding Kotlin frame processor plugin in `android/app/src/main/java/com/yourapp/ObjectDetectorPlugin.kt`: ```kotlin package com.yourapp import androidx.camera.core.ImageProxy import com.mrousavy.camera.frameprocessors.Frame import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin import com.mrousavy.camera.frameprocessors.VisionCameraProxy class ObjectDetectorPlugin(proxy: VisionCameraProxy, options: Map?) : FrameProcessorPlugin() { init { // Load native library containing VisionCore C++ logic System.loadLibrary("vision_camera_native") initModelNative("/data/local/tmp/detector.tflite") } override fun callback(frame: Frame, params: Map?): Any { val imageProxy: ImageProxy = frame.imageProxy ?: return emptyList>() // Call C++ via JNI passing direct byte buffer or planes val planes = imageProxy.planes if (planes.isEmpty()) return emptyList>() val buffer = planes[0].buffer val detections = processBufferNative( buffer, imageProxy.width, imageProxy.height, planes[0].rowStride ) return detections } private external fun initModelNative(modelPath: String): Boolean private external fun processBufferNative( buffer: java.nio.ByteBuffer, width: Int, height: Int, rowStride: Int ): List> } ``` Register the plugin in your `PackageList` or initialization class: ```kotlin import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry class MainApplication : Application(), ReactApplication { override fun onCreate() { super.onCreate() FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectObjects") { proxy, options -> ObjectDetectorPlugin(proxy, options) } } } ``` --- ## Step 3: Integrating the Frame Processor in React Native UI React Native interfaces with custom native plugins inside the `useFrameProcessor` worklet hook, which runs on an isolated secondary thread. Because vision frame processors can fire dozens of times per second, developers must throttle updates before using `runOnJS` to send state back to React. This guarantees high-frequency camera detection without exhausting the React rendering cycle. Here is a practical, production-ready screen implementation: ```tsx import React, { useEffect, useState } from 'react'; import { StyleSheet, Text, View } from 'react-native'; import { Camera, useCameraDevice, useCameraFormat, useFrameProcessor, VisionCameraProxy, } from 'react-native-vision-camera'; import { Worklets } from 'react-native-worklets-core'; interface Detection { x: number; y: number; width: number; height: number; confidence: number; label: string; } // Initialize native plugin handle const plugin = VisionCameraProxy.initFrameProcessorPlugin('detectObjects', {}); export const VisionCameraScreen: React.FC = () => { const [hasPermission, setHasPermission] = useState(false); const [detections, setDetections] = useState([]); const device = useCameraDevice('back'); // Choose an optimal format (e.g. 1080p or 720p) to balance resolution and model latency const format = useCameraFormat(device, [ { videoResolution: { width: 1280, height: 720 } }, { fps: 30 }, ]); useEffect(() => { (async () => { const status = await Camera.requestCameraPermission(); setHasPermission(status === 'granted'); })(); }, []); // Worklet callback that sends results to the React JavaScript thread safely const updateDetectionsJS = Worklets.createRunOnJS((results: Detection[]) => { setDetections(results); }); const frameProcessor = useFrameProcessor((frame) => { 'worklet'; if (plugin == null) { return; } // Call native C++ plugin const rawResults = plugin.call(frame) as unknown as Detection[]; if (rawResults && rawResults.length > 0) { updateDetectionsJS(rawResults); } }, [plugin, updateDetectionsJS]); if (!hasPermission) { return ( Awaiting camera permissions... ); } if (device == null) { return ( No suitable camera hardware found. ); } return ( {/* Detection HUD Bounding Boxes */} {detections.map((item, index) => ( {item.label} ({(item.confidence * 100).toFixed(0)}%) ))} ); }; const styles = StyleSheet.create({ container: { flex: 1, backgroundColor: '#000' }, centered: { flex: 1, justifyContent: 'center', alignItems: 'center' }, statusText: { color: '#fff', fontSize: 16 }, box: { position: 'absolute', borderWidth: 2, borderColor: '#00FF66', borderRadius: 4, backgroundColor: 'rgba(0, 255, 102, 0.1)', }, boxLabel: { backgroundColor: '#00FF66', color: '#000', fontSize: 12, fontWeight: 'bold', paddingHorizontal: 4, alignSelf: 'flex-start', }, }); ``` --- ## Step 4: Production Pitfalls and Performance Best Practices Deploying computer vision to production requires vigilant management of hardware constraints, memory lifecycle, and thermal limits across heterogeneous mobile devices. Failing to drop backpressured frames or running inference on oversized 4K buffers will swiftly trigger thermal throttling and OS watchdog terminations. Adhering to strict native buffer recycling patterns and downscaling sensor dimensions at the driver level ensures consistent performance in consumer environments. ### 1. Match Hardware Capture Format to Model Input Dimensions Mobile camera sensors frequently capture in 4K (3840×2160) or 1080p by default. Running color space conversions or bilateral filters on millions of pixels when your model only accepts 224×224 or 640×640 inputs wastes immense CPU and memory bandwidth. * Always restrict camera resolution using `useCameraFormat` to target 720p (1280×720) or 480p (640×480). * Select `pixelFormat="yuv"` or `"native"` to prevent the driver from performing an unneeded BGRA conversion if your inference engine natively ingests YUV/greyscale data. ### 2. Guard Against Memory Leaks and Buffer Retention Mobile operating systems provide a fixed pool of hardware image buffers (typically 3 to 5 buffers in the pipeline). If your C++ processor fails to release base addresses or holds a reference to a frame across asynchronous dispatch boundaries, the camera feed stalls completely. * **iOS:** Always pair `CVPixelBufferLockBaseAddress` with `CVPixelBufferUnlockBaseAddress` inside a `try/finally` block or RAII guard. * **Android:** Close the `ImageProxy` immediately after copying required bytes into model memory. Failing to close the proxy results in the camera sensor dropping subsequent frames. ### 3. Handle Frame Dropping and Backpressure If model inference takes 60ms, the processor can only sustain 16 FPS. Attempting to process every single 30 FPS camera frame creates backpressure queues that lead to input lag and memory inflation. * Execute frame processing synchronously within the frame processor thread, allowing the camera hardware driver to automatically discard intermediate frames when the processor is busy. * Alternatively, maintain an atomic `std::atomic isProcessing{false}` flag in C++ to instantly skip frames whenever an active inference cycle is running. ### 4. Optimize On-Device Models Before Deployment Running full float32 neural networks will rapidly drain the device battery and generate excessive heat. Refer to the [ONNX Runtime Mobile Documentation](https://onnxruntime.ai/docs/tutorials/mobile/) to apply quantization (INT8/FP16) and prune unnecessary operators prior to bundling model artifacts into your mobile binary. --- ## Frequently Asked Questions Understanding common edge cases in mobile computer vision helps teams troubleshoot pipeline stalls and performance degradation early in the development lifecycle. The answers below address real-world deployment challenges encountered when shipping VisionCamera and C++ plugins to end users. ### How do I handle pixel format conversions between camera frames and model inputs? Camera hardware predominantly outputs frames in YUV formats (such as NV12 on iOS or YUV_420_888 on Android), while computer vision models typically require planar RGB or BGR float tensors. Rather than converting the entire image to RGB on the JavaScript or platform layer, perform planar slicing and normalization directly inside your C++ core. For models that only evaluate luminance (such as barcode scanning or edge detection), you can feed the Y-plane directly to the model as a greyscale matrix, bypassing color space conversion entirely. ### Can I run model inference asynchronously without dropping camera frames? Yes, but you must avoid holding references to native frame buffers across asynchronous threads. Because mobile hardware frame pools are small, keeping a frame alive while an asynchronous thread executes inference will exhaust the camera pool and freeze the preview. If asynchronous processing is required, copy the raw pixel bytes into an internal, pre-allocated C++ memory buffer during the synchronous frame callback, release the original camera frame immediately, and pass the cloned buffer to your background inference thread pool. ### How do I prevent thermal throttling and battery drain during continuous scanning? Thermal buildup occurs when the GPU or NPU continuously executes heavy matrix multiplications at maximum clock speeds. To maintain low device temperatures during extended camera sessions, throttle your inference frequency to 10–15 FPS using a timestamp delta check inside your worklet. Additionally, select the lowest acceptable sensor resolution from `useCameraFormat` that meets your model's input dimension requirements, and employ INT8 quantized models to take advantage of low-power mobile DSP and NPU accelerators. --- ### About the Author **Furkan Çetinkaya** is a Mobile-focused Software Developer specializing in React Native, native bridge integrations (Kotlin & Swift), and supporting backend services. Experienced in maintaining high-impact mobile applications and developer SDKs. - [GitHub](https://github.com/cetfu) - [LinkedIn](https://www.linkedin.com/in/cetfu)