A vision chip with complementary pathways for open-world sensing

• Published in: Nature (London) Vol. 629; no. 8014; pp. 1027 - 1033
• Main Authors: Yang, Zheyu, Wang, Taoyi, Lin, Yihan, Chen, Yuguo, Zeng, Hui, Pei, Jing, Wang, Jiazheng, Liu, Xue, Zhou, Yichun, Zhang, Jianqiang, Wang, Xin, Lv, Xinhao, Zhao, Rong, Shi, Luping
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 30.05.2024; Nature Publishing Group
• Subjects: 639/166/987; 639/624/1107/510; 639/925/930/2735; Bandwidths; Cognition; Cognition & reasoning; Design; Driving ability; Dynamic range; Figure of merit; High speed; Humanities and Social Sciences; multidisciplinary; Science; Science (multidisciplinary); Semantics; Sensors; Sparsity; Spatial discrimination; Spatial resolution; Vision systems; Visual system
• ISSN: 0028-0836; 1476-4687
• DOI: 10.1038/s41586-024-07358-4

Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications. Inspired by the human visual system, a vision chip with primitive-based complementary pathways is developed to overcome the power and bandwidth wall of vision systems, achieving fast, precise, robust and high-dynamic-range sensing efficiently in the open world.