Lecture Course Overview and Computer Vision Basics

Lecture Robotics Basics and Machine Learning Basics

2D Perception

• DINOv2: Learning Robust Visual Features without Supervision. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski (2023) [Presenter: Maksym Bondarenko]
• Segment Anything 2. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer (2024) [Presenter: Seyoung Ree]

• Emerging Properties in Self-Supervised Vision Transformers. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin (2021)
• Segment Anything. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick (2023)

3D Perception

• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield (2024) [Presenter: Lingyi Zhang]
• DUSt3R: Geometric 3D Vision Made Easy. Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, Jerome Revaud (2024) [Presenter: Dan Harvey]

• PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox (2018)
• Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation. He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, Leonidas J. Guibas (2020)

Representation Learning for Robotics

• R3M: A Universal Visual Representation for Robotic Manipulation. Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, Abhinav Gupta (2022) [Presenter: TA Yixuan Wang & Code Reporter: Sammy Agrawal]
• D³Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement. Yixuan Wang*, Mingtong Zhang*, Zhuoran Li*, Tarik Kelestemur, Katherine Driggs-Campbell, Jiajun Wu, Li Fei-Fei, Yunzhu Li (2024) [Presenter: Prof. Yunzhu Li]

• Real-World Robot Learning with Masked Visual Pre-training. Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, Trevor Darrell (2022)
• Momentum Contrast for Unsupervised Visual Representation Learning. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick (2019)

Model Learning for Robotics

• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, Antonio Torralba (2019) [Presenter: Nikolaus Holzer]
• RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools. Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, Jiajun Wu (2023) [Presenter: Shuo Sha]

• Learning Mesh-Based Simulation with Graph Networks. Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, Peter W. Battaglia (2020)
• Mesh-Based Dynamics with Occlusion Reasoning for Cloth Manipulation. Zixuan Huang, Xingyu Lin, David Held (2022)

Gaussian Splatting

• GraspSplats: Efficient Manipulation with 3D Feature Splatting. Mazeyu Ji*, Ri-Zhao Qiu*, Xueyan Zou, Xiaolong Wang (2024) [Presenter: Yutao Mao]
• Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics. Jad Abou-Chakra, Krishan Rana, Feras Dayoub, Niko Suenderhauf (2024) [Presenter: Philippe Wu]

• DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation. Bardienus P. Duisterhof, Zhao Mandi, Yunchao Yao, Jia-Wei Liu, Jenny Seidenschwarz, Mike Zheng Shou, Deva Ramanan, Shuran Song, Stan Birchfield, Bowen Wen, Jeffrey Ichnowski (2024)
• Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling. Mingtong Zhang*, Kaifeng Zhang*, Yunzhu Li (2024)

Multimodal Perception

• 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing. Binghao Huang, Yixuan Wang, Xinyi Yang, Yiyue Luo, Yunzhu Li (2024) [Presenter: Bingyao Du]
• NeuralFeels with neural fields Visuo-tactile perception for in-hand manipulation. Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, Joseph Ortiz, Mustafa Mukadam (2024) [Presenter: Jifeng Li]

• Cable Manipulation with a Tactile-Reactive Gripper Yu She, Shaoxiong Wang, Siyuan Dong, Neha Sunil, Alberto Rodriguez, Edward Adelson (2019)
• See, Feel, Act: Hierarchical Learning for Complex Manipulation Skills with Multisensory Fusion. Nima Fazeli, Miquel Oller, Jiajun Wu, Zheng Wu, Joshua B Tenenbaum, Alberto Rodriguez (2019)

Visual Tracking

• CoTracker: It is Better to Track Together. Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht (2024) [Presenter: Aakash Aanegola]
• CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht (2024) [Presenter: Frank Fan]

• Tracking Everything Everywhere All at Once. Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely (2023)
• DART: Dense Articulated Real-Time Tracking. Tanner Schmidt, Richard Newcombe, Dieter Fox (2010)

Grasping

• Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a Multi-Armed Bandit model with correlated rewards. Jeffrey Mahler, Florian T Pokorny, Brian Hou, Melrose Roderick, Michael Laskey, Mathieu Aubry, Kai Kohlhoff, Torsten Kröger, James Kuffner, Ken Goldberg (2016) [Presenter: Charles Xu]
• AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains. Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, Cewu Lu (2023) [Presenter: Rodolfo Raimundo]

• 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation. Arsalan Mousavian, Clemens Eppner, Dieter Fox (2019)
• Learning Ambidextrous Robot Grasping Policies. Jeffrey Mahler, Matthew Matl, Vishal Satish, Michael Danielczuk, Bill DeRose, Stephen McKinley, Ken Goldberg (2019)

Visual Affordance

• TossingBot: Learning to Throw Arbitrary Objects with Residual Physics. Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser (2019) [Presenter: Jason Zou]
• Transporter Networks: Rearranging the Visual World for Robotic Manipulation. Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Ayzaan Wahid, Vikas Sindhwani, Johnny Lee (2020) [Presenter: Priyanka Varghese]

• Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robotic Manipulation. Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Mingrun Jiang, Huazhe Xu (2024)
• AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, Hao Dong (2023)

Imitation Learning

• Implicit Behavioral Cloning. Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson (2021) [Presenter: Tianjun Zhong]
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song (2023) [Presenter: Yolanda Zhu & Code Reporter: Naian Tao]

• MimicPlay: Long-Horizon Imitation Learning by Watching Human Play. Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, Anima Anandkumar (2023)
• ALOHA Unleashed: A Simple Recipe for Robot Dexterity. Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, Ayzaan Wahid (2024)

Model-Based Planning

• Deep Dynamics Models for Learning Dexterous Manipulation. Anusha Nagabandi, Kurt Konolige, Sergey Levine, Vikash Kumar (2019) [Presenter: Hongyu Li]
• BaB-ND: Long-Horizon Motion Planning with Branch-and-Bound and Neural Dynamics. Keyi Shen, Jiangwei Yu, Huan Zhang, Yunzhu Li (2024) [Presenter: Kheri Hughes & Code Reporter: Alexander Du]

• Model-Based Diffusion for Trajectory Optimization. Chaoyi Pan, Zeji Yi, Guanya Shi, Guannan Qu (2024)
• Model-Based Control with Sparse Neural Dynamics. Ziang Liu, Genggeng Zhou*, Jeff He*, Tobia Marcucci, Jiajun Wu, Li Fei-Fei, Yunzhu Li (2023)

Task and Motion Planning

• Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances. Aidan Curtis, Xiaolin Fang, Leslie Pack Kaelbling, Tomás Lozano-Pérez, Caelan Reed Garrett (2021) [Presenter: Feiyang Chen]
• Task and Motion Planning with Large Language Models for Object Rearrangement. Yan Ding, Xiaohan Zhang, Chris Paxton, Shiqi Zhang (2023) [Presenter: Carl Gross]

• Learning Resuable Manipulation Strategies. Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling (2023)
• Object-Centric Task and Motion Planning in Dynamic Environments. Toki Migimatsu, Jeannette Bohg (2019)

Embodied AI Benchmark

• BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. Chengshu Li*, Ruohan Zhang*, Josiah Wong*, Cem Gokmen*, Sanjana Srivastava*, Roberto Martín-Martín*, Chen Wang*, Gabrael Levine*, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, Li Fei-Fei (2022) [Presenter: Prof. Yunzhu Li]
• Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, Roozbeh Mottaghi (2024) [Presenter: Adithi Narayan & Code Reporter: Yuan Fang]

• AI2-THOR: An Interactive 3D Environment for Visual AI. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, Ali Farhadi (2017)
• ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, Roozbeh Mottaghi (2022)

Dexterous Manipulation

• Learning Dexterous In-Hand Manipulation. OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, Wojciech Zaremba (2018) [Presenter: William Wang]
• Visual Dexterity: In-hand Dexterous Manipulation from Depth. Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, Pulkit Agrawal (2022) [Presenter: Danelle Tuchman]

• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality. Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche, Dieter Fox, Gavriel State (2022)
• A System for General In-Hand Object Re-Orientation. Tao Chen, Jie Xu, Pulkit Agrawal (2021)

Scalable Robot Data

• GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robotic Manipulators. Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, Pieter Abbeel (2023) [Presenter: Charan Santhirasegaran]
• Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, Shuran Song (2024) [Presenter: Ishaan Mahajan]

• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Tony Z. Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn (2023)
• Open-TeleVision: Teleoperation with Immersive Active Visual Feedback. Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, Xiaolong Wang (2024)

Program Synthesis for Embodied Agents

• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei (2023) [Presenter: Nico Bykhovsky]
• ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, Li Fei-Fei (2024) [Presenter: Karla Zúñiga]

• Code as Policies: Language Model Programs for Embodied Control. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng (2022)
• Voyager: An Open-Ended Embodied Agent with Large Language Models. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (2023)

Vision-Language-Action Model

• OpenVLA: An Open-Source Vision-Language-Action Model. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn (2024) [Presenter: Tom Zollo & Code Reporter: John Wendlandt]
• $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, Ury Zhilinsky (2024) [Presenter: Emre Adabag]

• Open X-Embodiment: Robotic Learning Datasets and RT-X Models. Open-X Collaboration (2023)
• FAST: Efficient Action Tokenization for Vision-Language-Action Models. Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, Sergey Levine (2025)

World Model

• Video Language Planning. Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan Tompson (2023) [Presenter: Hao Zou]
• Cosmos World Foundation Model Platform for Physical AI. NVIDIA: Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, Artur Zolkowski (2025) [Presenter: Kuo Gong]

• World Models. David Ha, Jürgen Schmidhuber (2018)
• DayDreamer: World Models for Physical Robot Learning. Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel (2022)

Locomotion

• Learning Quadrupedal Locomotion over Challenging Terrain. Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter (2020) [Presenter: Boshra Khalili]
• HumanPlus: Humanoid Shadowing and Imitation from Humans. Zipeng Fu*, Qingqing Zhao*, Qi Wu*, Gordon Wetzstein, Chelsea Finn (2024) [Presenter: Linlin Zhang & Code Reporter: Seojin Yoon]

• OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning. Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, Guanya Shi (2024)
• RMA: Rapid Motor Adaptation for Legged Robots. Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik (2021)

Mobile Robots

• TidyBot: Personalized Robot Assistance with Large Language Models. Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas Funkhouser (2023) [Presenter: Zirun Wang]
• OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics. Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto (2024) [Presenter: Lei Huang]

• SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, Niko Suenderhauf (2023)
• TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning. Jimmy Wu, William Chong, Robert Holmberg, Aaditya Prasad, Yihuai Gao, Oussama Khatib, Shuran Song, Szymon Rusinkiewicz, Jeannette Bohg (2024)