Logistics
Description and Objectives
Recent progress of Artificial Intelligence has been largely driven by advances in large language models (LLMs) and other generative methods. These models are often very large (e.g. 175 billion parameters for GPT3) and requires increasingly larger data to train (e.g. 300 billion tokens for ChatGPT). Training, serving, fine-tuning, and evaluating LLMs require sophisticated engineering with modern hardware and software stacks. Developing scalable systems for large language models is critical to advance AI.
In this course, students will learn the essential skills to design and implement LLM systems. This includes algorithms and system techniques to efficiently train LLMs with huge data, efficient embedding storage and retrieval, data efficient fine-tuning, communication efficient algorithms, efficient implementation of reinforcement learning with human feedback, acceleration on GPU and other hardware, model compression for deployment, and online maintenance. We will cover the latest advances about LLM systems in machine learning, natural language processing, and system research.
Time and Location
Monday and Wednesday, 12:30-1:50pm, TEP 1403 (in-person expected)
Office Hours
| Day | Time | Location | Instructor/TA |
|---|---|---|---|
| Monday | 5pm - 6pm | GHC 6403 | Lei Li |
| Tuesday | 5pm - 6pm | GHC 5417 | Rishi Shah |
| Wednesday | 3pm - 4pm | GHC 5417 | Kath Choi |
| Thursday | 4pm - 5pm | GHC 5417 | Amogh Gyaneshwar |
| Friday | 3pm - 4pm | TCS 244 | Hector He |
Starting Feb 3, we will change to the following office hours where instructor and/or multiple TAs will be there to answer any questions.
| Day | Time | Location |
|---|---|---|
| Monday | 5pm - 6pm | GHC 6417 |
| Friday | 3pm - 4pm | GHC 6417 |
Prerequisites
You are highly recommended to take either Deep Learning (11785) or Advanced NLP (11-611 or 11-711) course previously. Please see FAQ for alternative courses.
Class Format
Each class may contain some combination of the following
- Lectures
- Code-walk throuth (if available)
- Small quiz problems
- Homework review
Occasionally, we will invite industrial speakers to present latest advancement and engineering practice in building real LLM systems.
Discussion Forum
We will use the Ed platform for discussions, but coming to office hours is also encouraged. Please do not send email to individual TA/Instructor.
Textbook and Course Material
No text book is required. A select set of recent papers on LLM systems and algorithms will be provided. Students are expected to read the assigned material and papers before each lecture.
Homework, Exam and Grading
The course will have four graded components. Please submit your homework on canvas.
| Percentage | |
|---|---|
| Homework | 10% each, 40% in total |
| Quiz | 10% |
| Forum Answer | 5% (+5%) |
| Project | 45% |
Required Reading
Students are highly encouraged to read the material or paper before each session.
Computing Resources
Each student will receive $150 credits for computation on AWS cloud. We will also provide access to Pittsburgh Supercomputing clusters (PSC). Students may also use Google co-lab if they are new users.
Policies
Late Day Policy
Each student could use at most 100 late hours for each individual assignments (counted in whole hours!). Each late hour incurs a penalty of 1%. However, NO late hour is allowed for final project report.
We still encourage everybody to complete their work by the designated deadlines. This prevents cascading tardiness from overwhelming both students and teaching staff.
Extensions However, sometimes there are situations that call for extensions. Some examples from the last few years include the following:
- The death of friend or family member
- A wedding in the family
- A serious accident
- A surgery
- A significant illness
- A mental health crisis or episode
- An important religious or national holiday
We care about you and your well being more than we care about deadlines and if something difficult is happening in your life which is making it hard for you to complete an assignment on time please contact us so we can talk. We have found that, often, the students who most need some leeway are those least likely to ask for it. It never hurts to ask. We will work out a plan so you can complete the requirements of the course with your physical and psychological health intact. Do not feel ashamed to reach out to us. We are eager to see you succeed.
Academic integrity
Any cheating or plagiarism will be dealt with according to the University policies on academic integrity. In general, high-level discussion of tools, concepts, and formalisms is acceptable collaboration and is encouraged. Sharing specific aspects of solutions or results with other students, or consulting work from previous semesters or other universities, is considered cheating. Using Github copilot or any AI is prohibited. You are responsible for any content you submitted.
Disability
Many people have disabilities, including members of our own families. We see disabilities as deficits not in disabled people but in the institutions and societies that are structured such that they are disadvantaged. We wish to do our part to overcome this disparate treatment. If you have a disability (visible or invisible), please let us know as soon as possible (you don’t need to tell us the nature of the disability) and work with Disability Service to develop a set of accommodations which we can then approve. These may include extra time on exams, a quiet place in which to take an exam, alt text on all images, documents that work for people with differences in vision, sign language interpretation, captioning, etc.
Diversity, Equity, and Inclusion
Throughout human history, some people have been denied the rights and opportunities available to others on the basis of their race, gender, economic class, caste, ancestry, language community, age, religion, beliefs, political affiliation, and abilities (visible and invisible). A single course cannot undo the injustices of history, but we—as a teaching staff—are committed to fighting inequity and promoting inclusion. We encourage you to join us. If you feel that you, or those around you, have been treated unfairly based upon their identity (or perceived identity) by us, by other members of the teaching staff, or by other students in the course, we ask that you share your experience with Ethics Reporting Hotline. Students, faculty, and staff can anonymously file a report by calling 844-587-0793 or visiting cmu.ethicspoint.com.