Learning Physics-Based Manipulation in Clutter: Combining Image-Based Generalization and Look-Ahead Planning

• Published in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) pp. 6562 - 6569
• Main Authors: Bejjani, Wissam, Dogar, Mehmet R., Leonetti, Matteo
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.11.2019
• ISSN: 2153-0866
• DOI: 10.1109/IROS40897.2019.8967717

Physics-based manipulation in clutter involves complex interaction between multiple objects. In this paper, we consider the problem of learning, from interaction in a physics simulator, manipulation skills to solve this multi-step sequential decision making problem in the real world. Our approach has two key properties: (i) the ability to generalize and transfer manipulation skills (over the type, shape, and number of objects in the scene) using an abstract image-based representation that enables a neural network to learn useful features; and (ii) the ability to perform look-ahead planning in the image space using a physics simulator, which is essential for such multi-step problems. We show, in sets of simulated and real-world experiments (video available on https://youtu.be/EmkUQfyvwkY), that by learning to evaluate actions in an abstract image-based representation of the real world, the robot can generalize and adapt to the object shapes in challenging real-world environments.