How Google Wants to Solve Robotic Grasping by Letting Robots Learn for Themselves

You are likely pretty good at picking things up. That’s nice. Part of the reason that you’re pretty good at picking things up is that when you were little, you spent a lot of time trying and failing to pick things up, and learning from your experiences. For roboticists who don’t want to wait through the equivalent of an entire robotic childhood, there are ways to streamline the process: at Google Research, they’ve set up more than a dozen robotic arms and let them work for months on picking up objects that are heavy, light, flat, large, small, rigid, soft, and translucent (although not all at once). We talk to the researchers about how their approach is unique, and why 800,000 grasps (!) is just the beginning.

Part of what makes animals so good at grasping things are our eyes, as opposed to just our hands. You can grab stuff with your eyes closed, but you’re much better at it if you watch your hand interacting with the object that you’re trying to pick up. In robotics, this is referred to as visual servoing, and in addition to improving grasping accuracy, it makes grasping possible when objects are moving around or changing orientation during the grasping process, a very common thing to have happen in those pesky “real-world situations.”

Image: Google Research

One of the robotic manipulators used in the data collection experiments. Each unit consisted of a 7-degree-of-freedom arm with a 2-finger gripper, and a camera mounted over the shoulder of the robot. The researchers say the camera recorded monocular RGB and depth images, but only the monocular RGB images were used for grasp success prediction.

Teaching robots this skill can be tricky, because there aren’t necessarily obvious connections between sensor data and actions, especially if you have gobs of sensor data coming in all the time (like you do with vision systems). A cleverer way to do it is to just let the robots learn for themselves, instead of trying to teach them at all. At Google Research, a team of researchers, with help from colleagues at X, tasked a 7-DoF robot arm with picking up objects in clutter using monocular visual servoing, and used a deep convolutional neural network (CNN) to predict the outcome of the grasp. The CNN was continuously retraining itself (starting with a lot of fail but gradually getting better), and to speed the process along, Google threw 14 robots at the problem in parallel. This is completely autonomous: all the humans had to do was fill the bins with stuff and then turn the power on.