MCUNeRF: Packing NeRF into an MCU with 1MB Memory

摘要

Neural Radiance Fields (NeRFs) have revolutionized 3D scene synthesis. Voxel grids are commonly employed to enhance training or rendering speed, but they entail additional storage requirements. The large model size and high computational and memory demands impede their progress on resource-constrained devices, e.g., Microcontroller Units (MCUs). Besides, there is currently no NeRF rendering framework available on MCU devices. In this paper, we propose a NeRF method named MCUNeRF for 3D scene synthesis on MCU devices. The proposed MCUNeRF compresses voxel grids via a hybrid quantization algorithm merging learned step-size quantization (LSQ) and optimized product quantization (OPQ). To further reduce the model storage, we also propose a codebook-sharing method that renders multiple objects with a single quantization codebook. Then we implement a NeRF-based rendering framework for MCU devices, which leverages a low-bit neural network computation framework, i.e. CMSIS-NN, to accelerate the rendering progress. Extensive experiments on four datasets such as Synthetic-NeRF demonstrate that our proposed method could compress model data by 20-40 times with comparable rendering quality, which enables NeRF-based scene rendering on MCU devices with only 1M SRAM.