Andy Zeng

Andy is a co-founder at Generalist. Before that, he was a research scientist and tech lead at Google DeepMind – and before that, a student at Princeton and UC Berkeley. Andy has worked on training large-scale foundation models, developing robots that write their own code, and inventing handheld data collection for robots. These days, he is interested in training large neural nets on massive amounts of robot data.


X (Twitter)     G. Scholar     LinkedIn     Github

andyzeng at generalistai dot com

2026

Generalist article "GEN-1: Scaling Embodied Foundation Models to Mastery"

Generalist blog post "The Dark Matter of Robotics: Physical Commonsense"

2025

Generalist article "GEN-0: Embodied Foundation Models That Scale with Physical Interaction"

Out of stealth! Co-founded Generalist to build general intelligence for the physical world

2024

IEEE Robotics and Automation (RAS) Early Career Award

2023

Conference on Robot Learning (CoRL) Best Student Paper Award

IEEE International Conference on Robotics and Automation (ICRA) Outstanding Learning Paper Award

2022

Google AI blog post "Robots That Write Their Own Code"

AXIOS article "Unleash All This Creativity: Google AI's Breathtaking Potential"

CNET video "Google’s Most Advanced Robot Brain Just Got a Body"

2020

IEEE Transactions on Robotics (T-RO) Best Paper Award

2019

New York Times article "A New Lab Full of Fast Learners"
For a more up-to-date list, visit my Google Scholar

PaLM-E: An Embodied Multimodal Language Model

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence
International Conference on Machine Learning (ICML) 2023
Webpage  •   PDF  •   Demo  •   Google AI Blog

Read summary

PaLM-E combines a pretrained language model with continuous visual and state inputs so one embodied model can reason across robotics, visual question answering, and captioning. Joint training across these domains improves transfer, while the largest model retains general language abilities and performs strongly as a vision-language model.

Implicit Behavioral Cloning

Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson
Conference on Robot Learning (CoRL) 2021
Webpage  •   PDF  •   Code  •   Google AI Blog

Read summary

This work treats behavioral cloning as learning an implicit energy-based policy instead of directly predicting an action. Across simulated and real robot tasks, the implicit policies better represent complex, discontinuous, or multi-valued behaviors and often outperform common explicit cloning approaches.

Video Language Planning

Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan
Tompson
International Conference on Learning Representations (ICLR) 2024
Webpage  •   PDF  •   Code

Read summary

Video Language Planning uses tree search with vision-language policy and value models plus text-to-video dynamics models to generate multimodal plans for long-horizon tasks. Those video plans can guide goal-conditioned robot policies and improve success across simulated and real robot platforms.

Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng
IEEE International Conference on Robotics and Automation (ICRA) 2023
★ Outstanding Learning Paper Award, ICRA ★
Webpage  •   PDF  •   Code  •   Colab  •   Google AI Blog  •   TechCrunch  •   AXIOS

Multiscale Sensor Fusion and Continuous Control with Neural CDEs

Sumeet Singh, Francis McCann Ramirez, Jacob Varley, Andy Zeng, Vikas Sindhwani
IEEE International Conference on Intelligent Robots and Systems (IROS) 2022
PDF

Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners

Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar
Conference on Robot Learning (CoRL) 2023
★ Oral Presentation, Best Student Paper Award, CoRL ★
Webpage  •   PDF  •   Video

Hybrid Random Features

Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S Ryoo, Jake Varley, Andy Zeng, Valerii Likhosherstov, Dmitry Kalashnikov, Vikas Sindhwani, Adrian Weller
The International Conference on Learning Representations (ICLR) 2022
PDF

XIRL: Cross-Embodiment Inverse Reinforcement Learning

Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, Debidatta Dwibedi
Conference on Robot Learning (CoRL) 2021
★ Best Paper Award Finalist, CoRL ★
Webpage  •   PDF  •   Code  •   Benchmark  •   Google AI Blog

Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations

Shuran Song, Andy Zeng, Johnny Lee, Thomas Funkhouser
IEEE International Conference on Intelligent Robots and Systems (IROS) 2020
IEEE Robotics and Automation Letters (RA-L) 2020
Webpage  •   PDF

TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser
Robotics: Science and Systems (RSS) 2019
IEEE Transactions on Robotics (T-RO) 2020
★ Best Paper Award, T-RO & Best Systems Paper Award, RSS ★
Webpage  •   PDF  •   Google AI Blog  •   New York Times  •   IEEE Spectrum

Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R. Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, Ian Taylor, Weber Liu, Thomas Funkhouser, Alberto Rodriguez
IEEE International Conference on Robotics and Automation (ICRA) 2018
The International Journal of Robotics Research (IJRR) 2019
★ 1st Place (Stow Task), Amazon Robotics Challenge 2017 ★
Webpage  •   PDF  •   Code  •   MIT News  •   Engadget

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017
★ Oral Presentation, CVPR ★
Webpage  •   PDF  •   Code  •   Talk  •   2 Minute Papers

Show more publications

2026

CES K-Innovation Night

2024

UIUC CSL Student Conference

2023

EMNLP Workshop on RoboNLP

Cornell Robotics Seminar

MILA Robot Learning Seminar

UPenn GRASP SFI Seminar

RSS Workshop on Symmetries in Robot Learning (video)

RSS Workshop on Generalizable Manipulation Policy Learning: Paradigms and Debates

CVPR Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics

ICRA Workshop on Embracing Contact: Making Robots Physically Interact with our World

ICRA Round-Table Discussion with Science Robotics

ICRA Workshop on Life-Long Learning with Human Help

Panel on Northwestern Grand Challenges in Robotics (video)

Guest Lecture at Columbia COMS 4733: Computational Aspects of Robotics

Stanford Vision and Learning Lab

2022

CoRL Workshop on Language and Robot Learning (video)

NeurIPS Workshop on Robot Learning (video)

NeurIPS Workshop on Habitat Rearrangement Challenge

Guest Lecture at Columbia IEOR E6617: Machine Learning and High-Dimensional Data

Princeton Robotics Seminar

RSS Workshop on Implicit Representations (video)

CVPR Tutorial on Vision-Based Robot Learning

MIT CSL Seminar (video)

Guest Lecture at Columbia COMS 4733: Computational Aspects of Robotics

2021

CVPR Workshop on 3D Vision and Robotics

Guest Lecture at Stony Brook University CSE525: Introduction to Robotics

Guest Lecture at NYU CSCI-UA 74: Big Ideas in Artificial Intelligence

2020

RSS Workshop on Self-Supervised Robot Learning

RSS Workshop on Robotics Retrospectives (video)

2019

Amazon Research Robotics Symposium

2018

RE·WORK Deep Learning for Robotics Summit

Guest Lecture at NCTU Robotics Seminar: Robotic Manipulation

2025

Advanced Robotics (AR) Best Survey Paper Award

2024

IEEE Robotics and Automation (RAS) Early Career Award

Human-Robot Interaction (HRI) Best Paper Award (Technical Track)

IEEE International Conference on Intelligent Robots and Systems (IROS) RoboCup Best Paper Award

2023

Conference on Robot Learning (CoRL) Best Student Paper Award

Associate Editor, IEEE Robotics and Automation Letters (RA-L)

Robotics: Science and Systems (RSS) Best Demo Paper Award Finalist

Google AI blog post "Modular Visual Question Answering via Code Generation"

Demo, Robotics: Science and Systems (RSS) - Large Language Models on Robots (memories)

Co-Organizer, RSS Workshop on Articulate Robots

IEEE International Conference on Robotics and Automation (ICRA) Outstanding Learning Paper Award

Associate Editor, IEEE International Conference on Robotics and Automation (ICRA)

Demo, Google I/O - Code as Policies with PaLI and PaLM 2 (memories)

Google AI blog post "Visual Language Maps for Robot Navigation"

Co-Organizer, AAAI Tutorial on Everything You Need to Know about Transformers

Google Research PaLM-E debut - "An Embodied Multimodal Language Model" (20+ derived YouTube videos)

Google AI blog post "PaLM-E: An Embodied Multimodal Language Model"

2022

Conference on Robot Learning (CoRL) Special Innovation Award

Area Chair, Conference on Robot Learning (CoRL)

Co-Organizer, CoRL Workshop on Pre-training Robot Learning

Co-Organizer, CoRL Workshop on Inductive Bias in Robot Learning

Google AI blog post "Robots That Write Their Own Code"

Google AI@ Event Code as Policies debut (memories) - AXIOS, TechCrunch, ZDNet (60+ articles)

Session Chair, Robotics: Science and Systems (RSS)

Co-Organizer, 2nd RSS Workshop on Scaling Robot Learning

Sponsor, Google Research Scholar Program

Co-Organizer, CVPR Tutorial on Vision-Based Robot Learning

Session Chair, IEEE International Conference on Robotics and Automation (ICRA)

Co-Organizer, ICRA Workshop on Scaling Robot Learning

AXIOS article "Unleash All This Creativity: Google AI's Breathtaking Potential"

CNET video "Google’s Most Advanced Robot Brain Just Got a Body"

Google Research PaLM-SayCan debut - Wired, Washington Post, The Verge (260+ articles)

Google AI blog post "Robot See, Robot Do"

Princeton News article "In Picking Up Trash, Robots Pick Up New Approaches to Work"

2021

Conference on Robot Learning (CoRL) Best Paper Award Finalist

Google AI blog post "Decisiveness in Imitation Learning for Robots"

Session Chair, IEEE International Conference on Robotics and Automation (ICRA)

Co-Organizer, RSS Workshop on Advancing AI and Manipulation for Robotics

Area Chair, Conference on Robot Learning (CoRL)

Google AI blog post "Learning to Manipulate Deformable Objects"

Sponsor, Google Research Scholar Program

Japan Factory Automation (FA) Foundation Paper Award

Google AI blog post "Rearranging the Visual World"

2020

IEEE Transactions on Robotics (T-RO) Best Paper Award

Conference on Robot Learning (CoRL) Best Paper Presentation Award Finalist

Google AI blog post "Visual Transfer Learning for Robotic Manipulation"

IEEE International Conference on Robotics and Automation (ICRA) Best Paper in Automation Award Finalist

Google AI blog post "Learning to See Transparent Objects"

Mentor, Google CS Research Mentorship Program

Sponsor, Google Faculty Research Awards

2019

New York Times article "A New Lab Full of Fast Learners" (100+ articles)

Google AI blog post "Learning to Assemble and to Generalize"

Robotics: Science and Systems (RSS) Best System Paper Award

Google AI blog post "Unifying Physics and Deep Learning with TossingBot"

2018

Honored to be a recipient of the Princeton SEAS Award for Excellence

Demo, Google TGIF - TossingBot (memories)

IEEE International Conference on Intelligent Robots and Systems (IROS) Best Cognitive Paper Award Finalist

Amazon Robotics Best System Paper Award

Honored to be a recipient of the NVIDIA Fellowship

MIT News article "Robo-Picker Grasps and Packs"

2017

1st place winners (stow) at the worldwide Amazon Picking Challenge 2017 with Team MIT-Princeton

2016

3rd place winners at the worldwide Amazon Picking Challenge 2016 with Team MIT-Princeton

Co-Organizer, CVPR Workshop on 3D Deep Learning

2015

Honored to be a recipient of the Gordon Y.S. Wu Fellowship in Engineering and Wu Prize

2015+

Reviewer, T-RO, RSS, CoRL, IJRR, RA-L, ICRA, NeurIPS, CVPR, IROS, ECCV, ICCV

Reviewer, SIGGRAPH, PR Journal, Eurographics, TMM, TIP, CASE, TCSVT

Member, IEEE