Yuke Zhu

I am currently a senior research scientist at NVIDIA Research and a visiting scholar at Stanford University. I will be joining UT-Austin as an Assistant Professor in Computer Science starting Fall 2020 (see Opportunies below).

My goal is to build intelligence for general-purpose robots that understand and interact with the real world. My research lies at the intersection of robotics, computer vision, and machine learning. I focus on developing methods and mechanisms of perception and control for general-purpose autonomy.

I received my Ph.D. from Stanford University in September 2019, where I work in Stanford Vision and Learning Lab with Prof. Fei-Fei Li and Prof. Silvio Savarese. I am a founder of Stanford People, AI & Robots Group.

Email: yukez@cs.stanford.edu


•  At UT-Austin, I look forward to working with strong and motivated students who are passionate about AI + Robotics. If you are interested in joining my lab, please submit your PhD applications to UTCS and mention my name in your statement. Please email me at yukez@cs.utexas.edu if you have any question.

•  At NVIDIA Research, we have year-round internship opportunities in Machine Learning and Robotics. Please contact me at yukez@nvidia.com for more information.


[new] I will be attending CoRL 2019 and IROS 2019 and giving two invited talks at AnSWeR19 and LRPC workshops.

[new] We released our papers on probabilistic planning under uncertainty and causal reasoning for goal-directed tasks.

[new] We released our technical report on the SURREAL System for distributed reinfrocement learning.

One paper accepted in CoRL 2019 and one paper accepted in NeurIPS 2019.

Two papers accepted in IROS 2019 and one paper accepted in ICCV 2019.

We won the Best Conference Paper Award at ICRA 2019 in Montreal.

Talks and Tutorials

  • Closing the Perception-Action Loop: Towards General-Purpose Robot Autonomy
    Stanford University Ph.D. Dissertation (August 2019)
  • Learning Keypoint Representations for Robot Manipulation
    IROS'19 Workshop on Learning Representations for Planning and Control (November 2019)

Selected Publications

  • 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
    Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, Yuke Zhu
  • KETO: Learning Keypoint Representations for Tool Manipulation
    Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, Silvio Savarese
  • Causal Induction from Visual Observations for Goal Directed Tasks
    Suraj Nair, Yuke Zhu, Silvio Savarese, Li Fei-Fei
  • Dual Sequential Monte Carlo: Tunneling Filtering and Planning in Continuous POMDPs
    Yunbo Wang*, Bo Liu*, Jiajun Wu, Yuke Zhu, Simon S Du, Li Fei-Fei, Joshua B Tenenbaum
  • Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
    Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei
    CoRL 2019 (Oral Presentation)
  • Regression Planning Networks
    Danfei Xu, Roberto Martín-Martín, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei
    NeurIPS 2019
  • Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset
    through Human Reasoning and Dexterity
    Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg,
    Silvio Savarese, Li Fei-Fei
    IROS 2019 (Best Cognitive Robotics Paper Finalist)
  • DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
    Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, Silvio Savarese
    CVPR 2019
  • Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
    De-An Huang*, Suraj Nair*, Danfei Xu*, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese,
    Juan Carlos Niebles
    CVPR 2019 (Oral Presentation)
  • Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for
    Contact-Rich Tasks
    Michelle A. Lee*, Yuke Zhu*, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg,
    Jeannette Bohg
    ICRA 2019 (Best Conference Paper)
  • SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark
    Linxi Fan*, Yuke Zhu*, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese,
    Li Fei-Fei
    CoRL 2018
  • RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
    Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao,
    John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, Li Fei-Fei
    CoRL 2018
  • Reinforcement and Imitation Learning for Diverse Visuomotor Skills
    Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool,
    János Kramár, Raia Hadsell, Nando de Freitas, Nicolas Heess
    RSS 2018
  • Learning Task-Oriented Grasping for Tool Manipulation with Simulated Self-Supervision
    Kuan Fang, Yuke Zhu, Animesh Garg, Virja Mehta, Andrey Kuryenkov, Li Fei-Fei, Silvio Savarese
    RSS 2018
  • Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
    Danfei Xu*, Suraj Nair*, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese
    ICRA 2018
  • Visual Semantic Planning using Deep Successor Representations
    Yuke Zhu*, Daniel Gordon*, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, Ali Farhadi
    ICCV 2017
  • Scene Graph Generation by Iterative Message Passing
    Danfei Xu, Yuke Zhu, Christopher B. Choy, Li Fei-Fei
    CVPR 2017
  • Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi
    ICRA 2017
  • Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen,
    Yannis Kalanditis, Li-Jia Li, David A. Shamma, Michael Bernstein, Li Fei-Fei
    IJCV 2017
  • Reasoning About Object Affordances in a Knowledge Base Representation
    Yuke Zhu, Alireza Fathi, Li Fei-Fei
    ECCV 2014