A soft thumb-sized vision-based sensor with accurate all-round force perception

• Published in: Nature machine intelligence Vol. 4; no. 2; pp. 135 - 145
• Main Authors: Sun, Huanbo, Kuchenbecker, Katherine J., Martius, Georg
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 01.02.2022; Nature Publishing Group
• Subjects: 639/166/987; 639/166/988; 639/705/1042; Accuracy; Artificial neural networks; Calibration; Cameras; Collimators; Computer vision; Contact force; Data processing; Design; Elastomers; Engineering; Force distribution; Haptics; Localization; Low cost; Machine learning; Neural networks; Robots; Robustness; Sensors; Spatial distribution; Spatial resolution
• ISSN: 2522-5839; 2522-5839
• DOI: 10.1038/s42256-021-00439-3

Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer vision techniques; however, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized three-dimensional haptic sensor named Insight, which continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-moulded on a stiff frame to guarantee sensitivity, robustness and soft contact. Furthermore, Insight uniquely combines photometric stereo and structured light using a collimator to detect the three-dimensional deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of three-dimensional contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, a force magnitude accuracy of around 0.03 N and a force direction accuracy of around five degrees over a range of 0.03–2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts. High-fidelity haptic sensors with three-dimensional sensing surfaces are needed to advance dexterous robotic manipulation. The authors develop a sensor design that offers accurate force sensation across a three-dimensional surface while being robust, low-cost and easy to fabricate.