Syllabus

Week	Dates	Topic	Reading/Content	Homework
Week 1	8/24	Introduction to LLM[slides]
	8/26	GPU Programming Basics 1[slides]	Chap 2,3,4 of Programming Massively Parallel Processors, 4th Ed
	8/28	Recitation 1: PSC Guidelines, Simple CUDA Demo, HW1 slides
Week 2	8/31	GPU Acceleration[slides]	Chap 5,6 of Programming Massively Parallel Processors, 4th Ed
	9/2	Deep Learning Frameworks and Auto Differentiation [slides]	Tensorflow Auto Diff survey Differentiable Programming	HW1 Due
	9/4	Recitation 2: HW2, MiniTorch, More GPU
Week 3	9/7	no class
	9/9	LLM Architecture and Pre-training [slides]	Attention is all you need, Deepseek-V2, GPT3	HW2 Due
	9/11	Recitation 3: Annotated Transformers	Annotated Transformer
Week 4	9/14	Tokenization and Embedding [slides]	BPE, Sentence-Piece, VOLT
	9/16	Generation and Speculative Decoding [slides]	Speculative Decoding	Project Team Due
	9/18	Recitation 4: Tokens and Decoding
Week 5	9/21	Accelerating Transformer on GPU Part 1[slides]	LightSeq
	9/23	Accelerating Transformer on GPU Part 2[slides]	LightSeq2	HW3 Due
	9/25	Recitation 5: Lightseq demo
Week 6	9/28	Guest Lecture by Srinath Mandalapu (Google): TPU and JAX [slides]
	9/30	Guest Lecture by Srinath Mandalapu (Google): Pallas and Splash Attention [slides]		Project Proposal Due
	10/2	Recitation 6: JAX and TPU
Week 7	10/5	Distributed Data Parallel Training [slides]	DDP
	10/7	Model Parallel Training [slides]	GPipe, Megatron-LM	HW4 Due
	10/9	Recitation 7 Distributed training
Week 8	10/12	spring break
Week 9	10/19	Systems for Mixture-of-Expert Models [slides]	GShard, Switch Transformer, DeepSpeed-MOE, Deepseek-MoE
	10/21	Memory Optimization in Distributed Training [slides]	ZeRO (DeepSpeed)
	10/23	Recitation 8 MoE
Week 10	10/26	Model Quantization [slides]	NN Quantization, AdaQuant, LLM.int8()	HW5 Due
	10/28	Model Quantization II [slides]	GPTQ
	10/30	Recitation 9 Quantization
Week 11	11/2	Efficient fine-tuning for Large Models [slides]	CIAT, LORA, QLoRA
	11/4	Optimizing Attention for Modern Hardware (Tri Dao) [slides]	FlashAttention FlashAttention2 FlashAttention3 FlashAttention4	Mid-term Report Due
	11/6	Recitation 10 LoRA and FlashAttention
Week 12	11/9	LLM serving with SGL [slides]	ORCA SGLang
	11/11	Efficient LLM Inference with Paged Attention and vLLM (Woosuk Kwon) [slides]	vLLM
	11/13	Recitation 11 SGLang and vLLM		HW6 Due
Week 13	11/16	Acceleration on TPU1
	11/18	Acceleration on TPU2
	11/20	Recitation 12: TPU acceleration	Distributed Training
Week 14	11/23	Efficient Reinforcement Learning System for LLMs	ReaLHF
	11/25	no class
Week 15	11/30	Serving with Disaggregated Prefill-Decoding [slides]	DistServe	HW7 Due
	12/2	Better KV Cache for LLM Serving (Junchen Jiang) [slides]	CacheGen CacheBlend
	12/4	Final project presentation
	12/6			Final report due
		DistServe: Disaggregated Prefill-Decoding (Hao Zhang) [slides]	DistServe
		App Stack and Model Serving[slides]	Triton, LightLLM
		Triton for Kernel Optimization	JAX
		Retrieval-augmented Language Models	RAG
		Nearest Vector Search for Embeddings	HNSW
		Multimodal LLMs	Flamingo
		Efficient Streaming Language Models with Attention Sinks	Attention Sink
		LLM Serving on Heterogeneous Hardware [slides]	Mooncake, kTransformer