Ajay U. Mandlekar – Publications

For a complete list please see Google Scholar.

PhD Thesis

Building Robot Intelligence by Scaling Human Supervision

[pdf] [video]

Preprints

Thumbnail 

Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations

Julen Urain, Ajay Mandlekar, Yilun Du, Mahi Shafiullah, Danfei Xu, Katerina Fragkiadaki, Georgia Chalvatzaki, Jan Petersn
2024

[pdf]

Thumbnail 

NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors

Shuo Cheng, Caelan Garrett*, Ajay Mandlekar*, Danfei Xu

2024

[pdf] [website]

Publications

Thumbnail 

RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots

Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, Yuke Zhu

RSS 2024

[pdf] [website] [code]

Thumbnail 

Signatures Meet Dynamic Programming: Generalizing Bellman Equations for Trajectory Following

Motoya Ohnishi, Iretiayo Akinola, Jie Xu, Ajay Mandlekar, Fabio Ramos

L4DC 2024

[pdf] [website]

Thumbnail 

IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning

Ryan Hoque, Ajay Mandlekar*, Caelan Garrett*, Ken Goldberg, Dieter Fox

IROS 2024

[pdf] [website]

Thumbnail 

Voyager: An Open-Ended Embodied Agent with Large Language Models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang*, Ajay Mandlekar*, Chaowei Xiao, Yuke Zhu, Linxi Fan†, Anima Anandkumar†

TMLR 2024

[pdf] [website] [code]

Thumbnail 

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Ajay Mandlekar, Soroush Nasiriany*, Bowen Wen*, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox

CoRL 2023

Oral at CoRL 2023 Towards Generalist Robots Workshop

[pdf] [website] [code]

Thumbnail 

Human-in-the-Loop Task and Motion Planning for Imitation Learning

Ajay Mandlekar*, Caelan Garrett*, Danfei Xu, Dieter Fox

CoRL 2023

Best Paper Runner-Up and Oral at CoRL 2023 Deployable Workshop

[pdf] [website]

Thumbnail 

Imitating Task and Motion Planning with Visuomotor Transformers

Murtaza Dalal, Ajay Mandlekar*, Caelan Garrett*, Ankur Handa, Ruslan Salakhutdinov, Dieter Fox

CoRL 2023

[pdf] [website] [code]

Thumbnail 

Active Task Randomization: Learning Visuomotor Skills for Sequential Manipulation by Proposing Feasible and Novel Tasks

Kuan Fang*, Toki Migimatsu*, Ajay Mandlekar, Li Fei-Fei, Jeannette Bohg

IROS 2023

[pdf] [website]

Thumbnail 

ORBIT: A Unified Simulation Framework for Interactive Robot Learning Environments

Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Pooria Poorsarvi Tehrani, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, Animesh Garg

IEEE Robotics and Automation Letters (RA-L) and ICRA 2023

[pdf] [website] [code]

Thumbnail 

Learning and Retrieval from Prior Data for Skill-based Imitation Learning

Soroush Nasiriany, Tian Gao, Ajay Mandlekar, Yuke Zhu

CoRL 2022

[pdf] [website]

Thumbnail 

MoCoDA: Model-based Counterfactual Data Augmentation

Silviu Pitis, Elliot Creager, Ajay Mandlekar, Animesh Garg

NeurIPS 2022

[pdf] [website] [code]

Thumbnail 

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Linxi Fan, Guanzhi Wang*, Yunfan Jiang*, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, Anima Anandkumar

NeurIPS 2022 Datasets and Benchmarks Track

Outstanding Paper Award

[pdf] [website] [code] [blog]

Thumbnail 

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

CoRL 2021

Oral (6.5% acceptance)

[pdf] [website] [video] [code] [blog] [talk]

Thumbnail 

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Roberto Martín-Martín

CoRL 2021

[pdf] [website] [video]

Thumbnail 

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

Samarth Sinha, Ajay Mandlekar, Animesh Garg

CoRL 2021

[pdf]

Thumbnail 

Generalization through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

Chen Wang*, Rui Wang*, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Danfei Xu

IROS 2021

[pdf] [website]

Thumbnail 

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Albert Tung*, Josiah Wong*, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

ICRA 2021

Best Multi-Robotic Systems Paper Finalist

[pdf] [website] [video]

Thumbnail 

Deep Affordance Foresight: Planning Through What Can Be Done in the Future

Danfei Xu, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Silvio Savarese, Li Fei-Fei

ICRA 2021

[pdf] [website] [video] [talk]

Thumbnail 

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Ajay Mandlekar, Danfei Xu*, Roberto Martin-Martin*, Yuke Zhu, Li Fei-Fei, Silvio Savarese

Technical Report 2020

[pdf] [website] [video]

Thumbnail 

Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

Ajay Mandlekar*, Danfei Xu*, Roberto Martín-Martín, Silvio Savarese, Li Fei-Fei

RSS 2020

[pdf] [website] [video]

Thumbnail 

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox

ICRA 2020

[pdf] [website] [video] [talk]

Thumbnail 

Controlling Assistive Robots with Learned Latent Actions

Dylan P. Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Dorsa Sadigh

ICRA 2020

[pdf] [blog] [video]

RoboTurk Real Dataset 

Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

IROS 2019

Best Cognitive Robotics Paper Finalist

[pdf] [website] [blog] [talk]

AC-Teach Figure 

AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

Andrey Kurenkov*, Ajay Mandlekar*, Roberto Martín-Martín, Silvio Savarese, Animesh Garg

CoRL 2019

[pdf] [website] [blog] [talk]

RoboTurk Figure 

RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, Li Fei-Fei

CoRL 2018

[pdf] [website] [talk]

ARPL Figure 

Adversarially Robust Policy Learning: Active Construction of Physically-Plausible Perturbations

Ajay Mandlekar*, Yuke Zhu*, Animesh Garg*, Li Fei-Fei, Silvio Savarese

IROS 2017

[pdf] [website] [video]

RSIRL Figure 

Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, Marco Pavone

RSS 2017

[pdf]

Technical Reports

Thumbnail 

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin

Technical Report

[pdf] [website]