Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

• Published in: Nature photonics Vol. 17; no. 5; pp. 442 - 450
• Main Authors: Zhou, Kevin C., Harfouche, Mark, Cooke, Colin L., Park, Jaehee, Konda, Pavan C., Kreiss, Lucas, Kim, Kanghyun, Jönsson, Joakim, Doman, Thomas, Reamey, Paul, Saliu, Veton, Cook, Clare B., Zheng, Maxwell, Bechtel, John P., Bègue, Aurélien, McCarroll, Matthew, Bagwell, Jennifer, Horstmeyer, Gregor, Bagnat, Michel, Horstmeyer, Roarke
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 01.05.2023; Nature Publishing Group
• Subjects: 631/1647/245/2226; 631/1647/328; 639/624/1075/1083; 639/624/1107/328; 639/624/1107/510; Algorithms; Applied and Technical Physics; Arrays; Cameras; Computer applications; Design optimization; Frames per second; Image acquisition; Image reconstruction; Larvae; Microscopy; Neural networks; Optical instruments; Optical properties; Organisms; Parallel processing; Physics; Physics and Astronomy; Quantum Physics; Redundancy; Spatial discrimination; Spatial resolution; Topography; Video; Zebrafish; parallelized microscopy; self-supervised learning; behavioral imaging; 3D imaging; computational microscopy; camera array
• ISSN: 1749-4885; 1749-4893
• DOI: 10.1038/s41566-023-01171-7

Wide-field-of-view microscopy that can resolve three-dimensional (3D) information at high speed and spatial resolution is particularly desirable for studying the behaviour of freely moving organisms. However, it is challenging to design an optical instrument that optimizes all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising speed and throughput. Here we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135 cm 2 area, achieving up to 230 frames per second at a spatiotemporal throughput exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with several collections of freely behaving organisms: ants, fruit flies and zebrafish larvae. 3D-RAPID, a scalable computational microscope using 54 cameras, records 3D topographic videos of freely moving organisms over an area of 135 cm 2 at a spatial resolution of tens of micrometres and at a throughput exceeding 5 gigapixels per second.