Robotic Grasping of Novel Objects

Home | Publications | STAIR manipulation | Cluttered Grasping | Stanford Object Grasping Data

Learning to Grasp Novel Objects

Image	Label	*Learning*	Prediction	Grasping experiments	Dishwasher unloading
Training Data

We consider the problem of grasping novel objects, specifically ones that are being seen for the first time through vision. We present a learning algorithm that predicts, directly as a function of the image, a point at which to grasp the object.

We demonstrate on two robotic manipulation platforms that this approach successfully grasps a wide variety of objects, such as wine glasses, duct tape, markers, a translucent box, jugs, knife-cutters, cellphones, keys, screwdrivers, staplers, toothbrushes, a thick coil of wire, a strangely shaped power horn, and others, none of which were seen in the training set. We also apply our method to the task of unloading items from dishwashers.

More Data Code Cluttered grasping

Videos

These videos show the robotic arm picking up previously unknown objects using only vision completely autonomously:

1 minute video (3-x speed; Jan 31, 2006): mp4, avi, wmv.
Unload a dishwasher (2-x speed; Nov 30, 2006): wmv.
Detailed videos (realtime; Jan 31, 2006): Translucent Box, jugs/mug, cellphone/stapler, wine Glass (with and without water), duct tape rolls and a "powerhorn", Jugs, solder tool holder, grasping person's Hands, untrained Algorithm: for comparison
Fetching a stapler in response to a verbal command (integrated with navigation, object recognition and speech, with Morgan Quigley; May 2, 2007): mov
Autonomous Grasping using 7-dof 3-finger Barrett (also with Brad Gulko, Lawson Wong; June 2, 2007): avi (200mb),wmv

Publications

Learning grasp strategies with partial shape information,
Ashutosh Saxena, Lawson Wong, Andrew Y. Ng. To appear in AAAI, 2008. [pdf]

Learning to Open New Doors,
Ellen Klingbeil, Ashutosh Saxena, Andrew Y. Ng. To be presented in AAAI 17th Annual Robot Workshop and Exhibition, 2008. [pdf]

Robotic Grasping of Novel Objects using Vision,
Ashutosh Saxena, Justin Driemeyer, Andrew Y. Ng. In International Journal of Robotics Research (IJRR), 2008. [pdf]

A Vision-based System for Grasping Novel Objects in Cluttered Environments,
Ashutosh Saxena, Lawson Wong, Morgan Quigley, Andrew Y. Ng. In International Symposium of Robotics Research (ISRR), 2007. [pdf]

Robotic Grasping of Novel Objects,
Ashutosh Saxena, Justin Driemeyer, Justin Kearns, Andrew Y. Ng. In NIPS 19, 2006. (spotlight paper) [pdf]

Learning to Grasp Novel Objects using Vision,
Ashutosh Saxena, Justin Driemeyer, Justin Kearns, Chioma Osondu, Andrew Y. Ng, ISER, 2006. [pdf]

Shorter version appeared as:
Learning to Grasp Novel Objects using Vision, Ashutosh Saxena, Justin Driemeyer, Justin Kearns, Chioma Osondu, Andrew Y. Ng,
RSS Workshop on Manipulation for Human Environments, 2006. [pdf]

Media Coverage

Getting a Grip: Building the Ultimate Robotic Hand, Wired Magazine, Issue 15.12, Dec 2007.

Newspapers: Brainy robots start stepping into daily life, Front page of NY Times (July 18, 2006). Also International Herald Tribune; the Sunday Times (UK); Mercury News; Apple News (Hong Kong); Science Central; and PC Magazine

News Channels: Kron 4 News, 6pm, April 12, 2007. Robots will 'do housework'. BBC News. 6pm, Feb 16, 2007. Also ABC channel 7 news, 11pm, April 6, 2007.

Magazines: Challenges for Robot Manipulation in Human Environments, Charles C. Kemp, Aaron Edsinger, Eduardo Torres-Jara, IEEE Robotics and Automation Magazine, vol. 14, issue 1, 2007.

Stanford Press Release: Robot learns to grasp everyday chores, Brian D. Lee, Stanford Report, Nov 8, 2006. Also: Artificial Intelligence might keep you from doing the dishes, Jack Hubbard, Nov 8, 2005. Also: A multitasking machine, Stanford Magazine, March 2007.

More information about Perception/Manipulation group. (available to STAIR members only.)

STAIR 1.0