Logistics
Description and Objectives
Recent progress of Artificial Intelligence has been largely driven by advances in large language models (LLMs) and other generative methods. These models are often very large (e.g. 175 billion parameters for GPT3) and requires increasingly larger data to train (e.g. 300 billion tokens for ChatGPT). Training, serving, fine-tuning, and evaluating LLMs require sophisticated engineering with modern hardware and software stacks. Developing scalable systems for large language models is critical to advance AI.
In this course, students will learn the essential skills to design and implement LLM systems. This includes algorithms and system techniques to efficiently train LLMs with huge data, efficient embedding storage and retrieval, data efficient fine-tuning, communication efficient algorithms, efficient implementation of reinforcement learning with human feedback, acceleration on GPU and other hardware, model compression for deployment, and online maintenance. We will cover the latest advances about LLM systems in machine learning, natural language processing, and system research.
Time and Location
Monday and Wednesday, 12:30-1:50pm, TEP 1403 (in-person expected)
Office Hours
| Day | Time | Location | Instructor/TA |
|---|---|---|---|
| Monday | am - am | ||
| Tuesday | 5pm - 6pm | ||
| Wednesday | 3pm - 4pm | ||
| Thursday | 3pm - 4pm | ||
| Thursday | 4pm - 5pm | ||
| Friday | 2pm - 3pm | ||
| Friday | 3pm - 4pm | ||
| Friday | 4pm - 5pm |
Prerequisites
You are highly recommended to take either Deep Learning (11785) or Advanced NLP (11-611 or 11-711) course previously.
Class Format
Each class will contain
- Lectures
- Code-walk throuth (if available)
- Discussion and review of the content.
- Homework review
Occasionally, we will invite industrial speakers to present latest advancement and engineering practice in building real LLM systems.
Discussion Forum
We will use the Ed platform for discussions, but coming to office hours is also encouraged. You may send private message on edstem platform as well.
Textbook and Course Material
No text book is required. A select set of recent papers on LLM systems and algorithms will be provided. Students are expected to read the assigned material and papers before each lecture.
Homework, Exam and Grading
The course will have four graded components. Please submit your homework on canvas.
| Percentage | |
|---|---|
| Homework | 10% each, 40% in total |
| Participation, Quiz, Discussion | 10% |
| In-class Presentation | 10% |
| Project | 40% |
Required Reading
Each student is required to read the material or paper before each session. Each student will write a review for the paper. Students assigned to the paper will be the reviewing committee. The review is similar to a normal conference paper review and should contain description of major innovation/highlight from the paper, major strengths and weakness of the paper. Students are expected to discuss and exchange reviews.
Rationale: Being able to review papers and provide insightful feedback is critical to being a productive member of the research community. This is an important skill to be developed in the graduate program.
In-class Presentation
Student will form a team of 3 to present a recent paper assigned by the instructor about the topic of the session. The presentation will be 30 minutes and 10 minutes for discussion. Students will turn in the presentation slides.
Rationale: Learning how to read, synthesize, and present ideas in a collection of related papers is an important part of the graduate program. This is a safe environment to develop the critical skills. In addition, we have found that the best way to learn the material is to try to teach it.
Computing Resources
Each student will receive $150 credits for computation on AWS cloud. We will also provide access to Pittsburgh Supercomputing clusters (PSC). Students may also use Google co-lab if they are new users. We are applying for hardware grant from Nvidia as well for this course.
Policies
Late Day Policy
Each student will have 3 late days in total for all the individual assignments. Each assignment is allowed to use at most 1 late day. No penalty is applied for late days, but there will be penalties if you submit the homework late.
We still encourage everybody to complete their work by the designated deadlines. This prevents cascading tardiness from overwhelming both students and teaching staff.
Extensions However, sometimes there are situations that call for extensions. Some examples from the last few years include the following:
- The death of friend or family member
- A wedding in the family
- A serious accident
- A surgery
- A significant illness
- A mental health crisis or episode
- An important religious or national holiday
We care about you and your well being more than we care about deadlines and if something difficult is happening in your life which is making it hard for you to complete an assignment on time please contact us so we can talk. We have found that, often, the students who most need some leeway are those least likely to ask for it. It never hurts to ask. We will work out a plan so you can complete the requirements of the course with your physical and psychological health intact. Do not feel ashamed to reach out to us. We are eager to see you succeed.
Academic integrity
Any cheating or plagiarism will be dealt with according to the University policies on academic integrity. In general, high-level discussion of tools, concepts, and formalisms is acceptable collaboration and is encouraged. Sharing specific aspects of solutions or results with other students, or consulting work from previous semesters or other universities, is considered cheating. Using Github copilot is discouraged. You are responsible for any content you submitted.
Disability
Many people have disabilities, including members of our own families. We see disabilities as deficits not in disabled people but in the institutions and societies that are structured such that they are disadvantaged. We wish to do our part to overcome this disparate treatment. If you have a disability (visible or invisible), please let us know as soon as possible (you don’t need to tell us the nature of the disability) and work with Disability Service to develop a set of accommodations which we can then approve. These may include extra time on exams, a quiet place in which to take an exam, alt text on all images, documents that work for people with differences in vision, sign language interpretation, captioning, etc.
Diversity, Equity, and Inclusion
Throughout human history, some people have been denied the rights and opportunities available to others on the basis of their race, gender, economic class, caste, ancestry, language community, age, religion, beliefs, political affiliation, and abilities (visible and invisible). A single course cannot undo the injustices of history, but we—as a teaching staff—are committed to fighting inequity and promoting inclusion. We encourage you to join us. If you feel that you, or those around you, have been treated unfairly based upon their identity (or perceived identity) by us, by other members of the teaching staff, or by other students in the course, we ask that you share your experience with Ethics Reporting Hotline. Students, faculty, and staff can anonymously file a report by calling 844-587-0793 or visiting cmu.ethicspoint.com.