Yunfan Jiang

I am a first-year CS PhD student at Stanford University, advised by Prof. Fei-Fei Li. I am affiliated with the Stanford Vision and Learning Lab. I work on the intersection of machine learning and robotics.
Previously, I received my master's degree from Stanford University and bachelor's degree from The University of Edinburgh. I was fortunate to collaborate with Dr. Jim Fan and Prof. Yuke Zhu. I have also worked as research intern at NVIDIA and Boston Dynamics AI Institute.

News

[May 2024] Introducing TRANSIC, a holistic human-in-the-loop method to tackle sim-to-real transfer for contact-rich robot manipulation.
[March 2024] Voyager is accepted at TMLR.
[September 2023] One paper called Cross-Episodic Curriculum is accepted at NeurIPS 2023. See you again in NOLA.
[June 2023] I am glad to receive the prestigious Stanford School of Engineering Exceptional Master’s Student Award [press].
[June 2023] I will be interning at Boston Dynamics AI Institute this summer.
[May 2023] We introduce Voyager, the first LLM-powered embodied lifelong learning agent in an open-ended Minecraft world.
[April 2023] VIMA is accepted at ICML 2023. I'm also glad to receive the conference travel grant. See y'all in Hawaii!
[April 2023] 📢 Life Update 📢 Decide to stay at Stanford for my PhD study!
[November 2022] I talk about VIMA in Foundation Models for Decision Making workshop at NeurIPS 2022.
[November 2022] MineDojo is accepted at NeurIPS 2022 D&B track and wins the Outstanding Paper Award.
[October 2022] We design a Transformer agent, VIMA, that ingests multimodal prompts and solves a wide variety of robot manipulation tasks.
[June 2022] We release a new framework called MineDojo for building generalist agents that can solve open-ended tasks with internet-scale knowledge in Minecraft.

Research [selected | all]

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

Yunfan Jiang, Chen Wang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei
arXiv | paper | website | code | tldr

Voyager: An Open-Ended Embodied Agent with Large Language Models

Guanzhi Wang, Yuqi Xie, Yunfan Jiang^*, Ajay Mandlekar^*, Chaowei Xiao, Yuke Zhu, Linxi "Jim" Fan^†, Anima Anandkumar^†
* Equal contribution † Equal advising
arXiv | paper | website | code | tldr
TMLR
Also Oral Presentation at NeurIPS 2023 Agent Learning in Open-Endedness Workshop and Intrinsically Motivated Open-Ended Learning Workshop
Covered by Forbes, WIRED, and Two Minute Papers

Cross-Episodic Curriculum for Transformer Agents

Lucy Xiaoyang Shi^*, Yunfan Jiang^*, Jake Grigsby, Linxi "Jim" Fan^†, Yuke Zhu^†
* Equal contribution † Equal advising
NeurIPS 2023
arXiv | paper | website | code | tldr

VIMA: General Robot Manipulation with Multimodal Prompts

Yunfan Jiang, Agrim Gupta^*, Zichen "Charles" Zhang^*, Guanzhi Wang^*, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu^†, Linxi "Jim" Fan^†
* Equal contribution † Equal advising
ICML 2023
Also Oral Presentation at NeurIPS 2022 Foundation Models for Decision Making Workshop
arXiv | paper | website | code | tldr
Covered by NVIDIA GTC Jensen Huang Keynote

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Linxi "Jim" Fan, Guanzhi Wang^*, Yunfan Jiang^*, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu^†, Anima Anandkumar^†
* Equal contribution † Equal advising
NeurIPS 2022 Datasets and Benchmarks Track
🎉 Outstanding Paper Award 🎉
Featured Paper (Oral-Equivalent)
arXiv | paper | website | code | tldr
Covered by NVIDIA Blog, Two Minute Papers, and Ars Technica