Robot grabbing method and system based on multi-modal feature extraction and dense prediction

• Main Authors: LIU XUEBING, MAO JIANXU, FENG MINGTAO, ZHU QING, YUAN XIAOFANG, ZHOU XIAN'EN, WU CHENGZHONG, HUANG JIANAN, WANG YAONAN, ZHOU JIAMING
• Format: Patent
• Language: Chinese; English
• Published: 06.01.2023
• Subjects: CALCULATING; CHAMBERS PROVIDED WITH MANIPULATION DEVICES; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; HAND TOOLS; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; MANIPULATORS; PERFORMING OPERATIONS; PHYSICS; PORTABLE POWER-DRIVEN TOOLS; TRANSPORTING

The invention discloses a robot grabbing method and system based on multi-modal feature extraction and dense prediction, and the method comprises the steps: obtaining a scene color image and a depth image, calculating a scene three-dimensional point cloud and adaptive convolution receptive fields of different scales from the depth image, and obtaining a surface normal vector image according to the scene three-dimensional point cloud; a multi-modal feature extraction and dense prediction network is constructed, a scene color image and a surface normal vector image are processed, dense three-dimensional attitude information and three-dimensional position information predicted by each type of objects are obtained, three-dimensional attitudes and three-dimensional positions of corresponding objects are obtained through calculation, and the three-dimensional attitudes and the three-dimensional positions jointly form three-dimensional poses. And the three-dimensional pose is sent to a robot grabbing system, and the