MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

• Published in: 2022 IEEE International Conference on Computational Photography (ICCP) pp. 1 - 12
• Main Authors: So, Haley M., Martel, Julien N.P., Wetzstein, Gordon, Dudek, Piotr
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.08.2022
• Subjects: computational photography; Computer vision; Encoding; end-to-end optimization; Image sensors; in-pixel intelligence; Photography; programmable sensors; Prototypes; Sensor systems; Wrapping
• ISSN: 2472-7636
• DOI: 10.1109/ICCP54855.2022.9887659

The ability to image high-dynamic-range (HDR) scenes is crucial in many computer vision applications. The dynamic range of conventional sensors, however, is fundamentally limited by their well capacity, resulting in saturation of bright scene parts. To overcome this limitation, emerging sensors offer in-pixel processing capabilities to encode the incident irradiance. Among the most promising encoding schemes is modulo wrapping, which results in a computational photography problem where the HDR scene is computed by an irradiance unwrapping algorithm from the wrapped low-dynamic-range (LDR) sensor image. Here, we design a neural network-based algorithm that outperforms previous irradiance unwrapping methods and we design a perceptually inspired "mantissa," or log-modulo, encoding scheme that more efficiently wraps an HDR scene into an LDR sensor. Combined with our reconstruction framework, MantissaCam achieves state-of-the-art results among modulo-type snapshot HDR imaging approaches. We demonstrate the efficacy of our method in simulation and show benefits of our algorithm on modulo images captured with a prototype implemented with a programmable sensor.