CS699: Robotic Manipulation

Daniel Seita

None.

Sign up for Piazza here: https://piazza.com/usc/fall2024/csci699robotmanipulation

Thursdays: 3:30 pm to 6:50 pm. Location: DMC 101.

Daniel: Thursdays, 1:45pm to 3:00pm (at PHE 212).

This special topics class and seminar will cover state of the art advances in robot manipulation. This is about robots interacting with and affecting their environment, and could refer to things such as grasping, pushing, picking-and-placing, tossing, and many other actions. Aided in part by the rise of deep learning and subsequent advances in robot perception, the research area of robot manipulation has experienced tremendous growth in recent years. Nonetheless, despite this progress, real-world manipulation remains fundamentally hard. In this course, we will review and understand robot manipulation with a focus on using learning-based techniques. We will explore how to get such systems to work more reliably in unstructured real world settings.

For Fall 2024, we will continue the "multimodality" theme from last fall, where we study robots that can handle multiple types of inputs or perform multiple types of actions. For example, this would include robot manipulation based on language and images as input. For 2024, we will also focus on higher-DOF control, including bimanual robots, quadrupeds with arms, and humanoid manipulation. We will also emphasize learning from human feedback.

In this class, students will read, review, and present research papers. Most research papers will be recent and have cutting-edge results. Students will also work on a substantial final project. This class is aimed at PhD students who are doing research in this topic. However, undergraduates and master's students can enroll with permission of the instructor. If interested, please talk to the instructor at the beginning of the semester.

Here is the weekly schedule and paper list:

https://docs.google.com/spreadsheets/d/1XKmv1bkNQxRsvQ4lCfzR2_4hJIzxWg8Vvi1jG7sW8B0/edit?usp=sharing

See this document for more details about the presentations, paper reviews, scribe notes, and class final project:

https://docs.google.com/document/d/1lZ-JtT-dc22wAPNRUGpeYKemF68MXlcMV49iEFMy9xE/edit?usp=sharing

See the spreadsheet above for the schedule. The first two lectures will provide background material. After the first two lectures, each 3+ hour class will involve presentations and discussions of research papers. The class will be structured (roughly) as follows:

• At the beginning, Daniel may talk briefly about recent robotics updates.
• Presentation of 3 related papers.
• A discussion / debate of those 3 papers.
• A break, because the class is 3 hours and 20 minutes long (let me know if you don't want one).
• Presentation of 3 more papers.
• A discussion / debate of those 3 papers.

Instead of paper discussions, the final week will involve a summary of what we have learned, and final project presentations. Thus, besides the first and last parts of the class, most weeks will involve students learning from academic robot manipulation research papers. We will not record the lectures but the slides will be made publicly available.

Courses such as CSCI 467: Introduction to Machine Learning, and CSCI 545: Introduction to Robotics, are recommended but not required. Students are expected to be familiar with the fundamentals of robotics (especially manipulation) and machine learning, or be willing to learn the material as needed. This class will consist largely of reading and discussing papers, which requires some degree of technical maturity.

Your grade will be based on:

• 25%: Class presentations
• 25%: Paper reviews
• 10%: Scribe notes
• 05%: Class participation
• 35%: A substantial final project

Depending on the number of students in class, we may adjust the relative grading portion of the class presentations versus paper reviews. In addition, since this is a seminar-style class, students will be expected to attend class, unless they have notified the instructor in advance. You must inform the instructor regarding every absence. Frequent absences will be handled on a case-by-case basis and will result in some grading penalty.

Tentative grading scale, conditioned on adequate class attendance:

• [90, 100]: A
• [85, 90): A-
• [80, 85): B+
• Grades lower than this will be handled on a case-by-case basis.

The grading scale may change over the semester.

• A better understanding of the current research frontiers in robot manipulation, and robot learning more broadly.
• Gain experience reading, understanding, analyzing, and debating papers from robotics venues.
• Main conferences: CoRL, RSS, ICRA, IROS, ISRR, ISER, WAFR.
• Main journals: RA-L, T-RO, IJRR, AURO, Science Robotics.
• Understand, implement, and experiment with state-of-the-art algorithms for the final project.
• Improve academic writing and presentation skills.

Discussing with other students at a high level is allowed, but you must write up your own paper reviews and write, prepare, and deliver your own class presentation (if these are done in groups, then you can discuss with your group members). For final projects, you must also acknowledge whether this is ongoing and concurrent research (and to what extent you will be implementing or working on class-related material), and whether you are re-using materials for another class.

You are welcome to use whatever resources you need to best learn the material. In general, we agree with Jeff Erickson's citation policy. For example:

• If you talked with a robotic manipulation researcher to help understand a technical concept, you must acknowledge that person.

We will provide templates for assignments where you must acknowledge the ingredients you used for your assignment and/or project.

If you have a disability and need academic accommodations, please discuss any needs with me as soon as possible. I will work with you to ensure that your needs are met and that we are able to effectively achieve the educational goals of this course. I hope that students with diverse backgrounds can all benefit from the course. If there is something I can do to make this course better achieve that objective, feel free to contact me with suggestions.