CSCI 645: Robot Manipulation

Course Staff

Professor: Daniel Seita (Office: GCS 305A)

Use Brightspace to contact the course staff.

Lectures

Thursdays: 4:00 pm to 7:20 pm. Location: TBD.

We will have an extended 25-minute break approximately halfway through each session to give everyone time to rest, use the restroom, grab a snack, etc.

We will record lectures and share them on this website (see the "Recording" column below).

Office Hours

Daniel: Thursdays 11:00 am to 12:00 pm (GCS 305A), except for Thursdays when there is no class or when I am traveling. In those cases, I will offer alternative times, but send me an email if you want to meet at other times.

Course Description

This course covers state-of-the-art advances in robot manipulation, focusing on how robots physically interact with and affect their environments through actions such as grasping, pushing, pick-and-place, tool use, and dynamic contact behaviors. Due to recent advances in robot learning, robot perception, and robot hardware, robot manipulation has experienced tremendous interest and growth in recent years. Despite this progress, real-world manipulation remains fundamentally hard, and progress can be tricky to measure due to lack of standardized benchmarks and cherry-picked demonstrations.

Students will study core principles of manipulation (including perception, control, planning, and dynamics) with modern data-driven approaches such as imitation learning, reinforcement learning, diffusion-based policy generation, and multimodal (e.g., vision-language-tactile) reasoning. While assignments will use simulators for faster and more reproducible work, this class also discusses how to get such systems to work more reliably in unstructured real-world settings, so that students develop an appreciation of the challenges of building such systems.

The class mostly consists of instructor-led lectures but will have some student-led discussions of recent research papers. There will also be a substantial final project. This class is aimed at PhD students doing research in this topic; advanced undergraduates and master's students are welcome to enroll with permission of the instructor.

Learning Objectives

This course is intended to:

• Equip students with knowledge of both core principles and recent data-driven methods for robot manipulation.
• Develop skills to understand, implement, and experiment with state-of-the-art manipulation-related algorithms for assignments and the final project.
• Provide experience reading, understanding, analyzing, and debating papers from robotics venues.
  • Conferences: CoRL, RSS, ICRA, IROS, ISRR, ISER, WAFR.
  • Journals: T-RL, RA-L, T-RO, IJRR, AURO, Science Robotics.
• Encourage students to develop new ideas for research projects in robot manipulation.
• Improve academic writing and presentation skills.

At the end of this course, students will be able to:

• Implement and experiment with state-of-the-art manipulation-related algorithms.
• Analyze and intelligently discuss current papers from robotics venues.
• Propose new ideas for research projects in robot manipulation.
• Communicate technical ideas through written reports and oral presentations.

Prerequisites

No formal prerequisites. Recommended preparation: familiarity with robotics at the level of CSCI 445L or CSCI 545, and familiarity with machine learning at the level of CSCI 467. Concurrent enrollment with these courses can be helpful. Students should be comfortable programming in Python and working in a Linux/Ubuntu shell environment.

Reading Material

Primary textbooks (both freely available online):

• Robot Manipulation: Perception, Planning, and Control, by Russ Tedrake. Link. (RT)
• Modern Robotics, by Kevin Lynch and Frank Park. Link. (MR)

Secondary textbooks (also open source):

• Underactuated Robotics, by Russ Tedrake. Link. (UR)
• Reinforcement Learning: An Introduction, by Richard Sutton and Andrew Barto. Link. (SB)

Additional readings (research papers, code, etc.) will be provided as needed and will all be open-source and free to access.

Grading

We will use Brightspace for communication and grading. We do not use a fixed grading scale (e.g., 90–100 = A); grades are computed at the end of the semester. Missing a homework assignment will reduce the maximum achievable grade by one letter. Students will be allowed a small number of late days for the two homework assignments; late days cannot be used for the final project.

In lieu of a final exam, students will work on a substantial final project and may work in groups of 1–2 (with expectations scaling for teams of two). The final project grade is subdivided into: project presentation (1/5), final written report (1/5), and quality of results (3/5). The final project report should ideally be of sufficient quality to form the basis of a future submission to a top-tier robotics conference.

Acknowledgments

TBD.