CSCI 645: Robot Manipulation

Course Staff

Professor: Daniel Seita (Office: GCS 305A)

Contact him by email with questions. Unfortunately, there is no TA, so please be mindful of this in our correspondence.

Lectures

Thursdays: 4:00 pm to 7:20 pm. Location: SLH 102.

We will have an extended 20-25 minute break approximately halfway through each session to give everyone time to rest, use the restroom, grab a snack, etc.

We will record lectures and share them on this website (see the "Recording" column below).

Office Hours

Daniel: Thursdays 11:20 am to 12:30 pm (GCS 305A), except for Thursdays when there is no class or when I am traveling. In those cases, I will offer alternative times. However, feel free to simply send me an email if you want to meet at an alternative time. If you do this, please suggest multiple options to reduce back-and-forth communication.

Course Description

This course covers state-of-the-art advances in robot manipulation, focusing on how robots physically interact with and affect their environments through actions such as grasping, pushing, pick-and-place, tool use, and dynamic contact behaviors. Due to recent advances in robot learning, robot perception, and robot hardware, robot manipulation has experienced tremendous interest and growth in recent years. Despite this progress, real-world manipulation remains fundamentally hard, and progress can be tricky to measure due to lack of standardized benchmarks and cherry-picked demonstrations.

Students will study core principles of manipulation (including perception, control, planning, and dynamics) with modern data-driven approaches such as imitation learning, reinforcement learning, diffusion-based policy generation, and multimodal (e.g., vision-language-tactile) reasoning. While assignments will use simulators for faster and more reproducible work, this class also discusses how to get such systems to work more reliably in unstructured real-world settings, so that students develop an appreciation of the challenges of building such systems.

The class mostly consists of instructor-led lectures but will have some student-led discussions of recent research papers. There will also be a substantial final project. This class is aimed at PhD students doing research in this topic; advanced undergraduates and master's students are welcome to enroll with permission of the instructor.

Learning Objectives

This course is intended to:

• Equip students with knowledge of both core principles and recent data-driven methods for robot manipulation.
• Develop skills to understand, implement, and experiment with state-of-the-art manipulation-related algorithms for assignments and the final project.
• Provide experience reading, understanding, analyzing, and debating papers from robotics venues.
  • Conferences: CoRL, RSS, ICRA, IROS, ISRR, ISER, WAFR.
  • Journals: T-RL, RA-L, T-RO, IJRR, AURO, Science Robotics.
• Encourage students to develop new ideas for research projects in robot manipulation.
• Improve academic writing and presentation skills.

At the end of this course, students will be able to:

• Implement and experiment with state-of-the-art manipulation-related algorithms.
• Analyze and intelligently discuss current papers from robotics venues.
• Propose new ideas for research projects in robot manipulation.
• Communicate technical ideas through written reports and oral presentations.

Prerequisites

Recommended preparation: familiarity with robotics at the level of CSCI 445L or CSCI 545, and familiarity with machine learning at the level of CSCI 467. Concurrent enrollment with these courses can be helpful. Students should be comfortable programming in Python and working in a Linux/Ubuntu shell environment, and they should understand how to utilize AI coding agents for help when needed (without being 100% dependent on them). However, at the end of the day, what ultimately matters is how fast students can pick up new material, and how much time/effort they are willing to invest in learning. This is why I don't think it makes sense to really enforce formal prerequisites.

Reading Material

Primary textbooks (both freely available online):

• Robot Manipulation: Perception, Planning, and Control, by Russ Tedrake. Link. (RT)
• Modern Robotics, by Kevin Lynch and Frank Park. Link. (MR)

Secondary textbooks (also open source):

• Underactuated Robotics, by Russ Tedrake. Link. (UR)
• Reinforcement Learning: An Introduction, by Richard Sutton and Andrew Barto. Link. (SB)

Additional readings (research papers, code, etc.) will be provided as needed and will all be open-source and free to access.

Grading

We will use Brightspace for communication and grading. We do not use a fixed grading scale (e.g., 90–100 = A); grades are computed at the end of the semester. Students must submit reasonable versions of all assignments to pass the course. To reduce administrative burden on course staff and to encourage long-term planning, we do not have "late days" in this course.

In lieu of a final exam, students will work on a substantial final project and may work in groups of 1–2 (with expectations scaling for teams of two). The final project grade is subdivided into: project presentation (1/5), final written report (1/5), and quality of results (3/5). The final project report should ideally be of sufficient quality to form the basis of a future submission to a top-tier robotics conference.

Acknowledgments

Daniel acknolwedges, in alphabetical order by last name: Pieter Abbeel, John Canny, Ken Goldberg, David Held, Dan Klein, Sergey Levine, Kevin Lynch, Stefanos Nikolaidis, Frank Park, and Russ Tedrake.