CS 59200-MLS | Schedule

Date	Lecture	Readings	Presenters
Week#1 08/25	Lecture #1 Course Introduction [ slides ]		Xupeng Miao
Week#1 08/27	Lecture #2 Introduction to Machine Learning Systems [ slides ]	(Optional) TensorFlow: A System for Large-Scale Machine Learning (Optional) PyTorch: An Imperative Style, High-Performance Deep Learning Library (Optional) PyTorch Distributed: Experiences on Accelerating Data Parallel Training (Optional) Scaling Distributed Machine Learning with the Parameter Server (Optional) XGBoost: A Scalable Tree Boosting System	Xupeng Miao
Week#1 08/29	Lecture #3 Introduction to LLMs [ slides ]	(Optional) Transformer: Attention is All you Need (Optional) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Optional) Training language models to follow instructions with human feedback	Xupeng Miao
Week#2 09/03	Lecture #4 Introduction to GPU & CUDA Programming [ slides ]		Xupeng Miao
Week#2 09/05	Lecture #5 ML Compilation (1) [ slides ]	TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Halide: Decoupling Algorithms from Schedules for Easy Optimization of Image Processing Pipelines (Optional) MLIR: A Compiler Infrastructure for the End of Moore's Law (Optional) Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
Week#3 09/08	Lecture #6 ML Compilation (2) [ slides ]	Learning to Optimize Tensor Programs Ansor: Generating High-Performance Tensor Programs for Deep Learning
Week#3 09/10	Lecture #7 Advanced ML Compilation (1) [ slides ]	TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions EINNET: Optimizing Tensor Programs with Derivation-Based Transformations
Week#3 09/12	Lecture #8 Advanced ML Compilation (2) [ slides ]	Mirage: A Multi-Level Superoptimizer for Tensor Programs ThunderMLA: Flashmla, faster and fuseder!	Xupeng Miao
Week#4 09/15	Lecture #9 ML Parallelization: Data & Model Parallelism [ slides ]	ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Optional) PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Week#4 09/17	Lecture #10 ML Parallelization: Pipeline Parallelism [ slides ]	GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism PipeDream: Generalized Pipeline Parallelism for DNN Training (Optional) Memory-Efficient Pipeline-Parallel DNN Training
Week#4 09/19	Lecture #11 Automatic Parallelization (1) [ slides ]	Beyond Data and Model Parallelism for Deep Neural Networks Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning (Optional) GSPMD: General and Scalable Parallelization for ML Computation Graphs
Week#5 09/22	Lecture #12 Memory Rematerialization [ slides ]	Training Deep Nets with Sublinear Memory Cost Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (Optional) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training
Week#5 09/24	Lecture #13 Automatic Parallelization (2) [ slides ]	GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism Improving Automatic Parallel Training via Balanced Memory Workload Optimization (Optional) Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization
Week#5 09/26	Lecture #14 ML Parallelization: Embeddings [ slides ]	HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training
Week#6 09/29	Lecture #15 ML Parallelization: Graph Neural Networks [ slides ]	Reducing Communication in Graph Neural Network Training GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUs
Week#6 10/01	Lecture #16 ML Parallelization: Expert Parallelism [ slides ]	FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement (Optional) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Week#6 10/03	Lecture #17 ML Parallelization: Sequence Parallelism [ slides ]	USP: A Unified Sequence Parallelism Approach for Long Context Generative AI WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
Week#7 10/06	Lecture #18 ML Parallelization: Hardware Heterogeneity [ slides ]	SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient SDPipe: A Semi-Decentralized Framework for Heterogeneity-Aware Pipeline-parallel Training (Optional) Pathways: Asynchronous Distributed Dataflow for ML
Week#7 10/08	Lecture #19 ML Parallelization: Fault Tolerance [ slides ]	Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs (Optional) GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints (Optional) Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances
Week#7 10/10	Lecture #20 ML Parallelization: Overlapping [ slides ] Final Team Signup Due	Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
Week#8 10/15	Lecture #21 LLM Inference Systems: Attention (1) [ slides ]	Fast and Memory-Efficient Exact Attention with IO-Awareness Efficient Memory Management for Large Language Model Serving with PagedAttention (Optional) Orca: A Distributed Serving System for Transformer-Based Generative Models (Optional) vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Week#8 10/17	Lecture #22 LLM Inference Systems: Attention (2) [ slides ]	Efficient Streaming Language Models with Attention Sinks LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention (Optional) Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems, Table 1 (Optional) Gemma 2: Improving Open Language Models at a Practical Size, Section 2
Week#9 10/20	Lecture #23 LLM Inference Systems: Decoding [ slides ]	SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding (Optional) Fast Inference from Transformers via Speculative Decoding (Optional) Sequoia: Scalable and Robust Speculative Decoding (Optional) Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Week#9 10/22	Lecture #24 LLM Inference Systems: Attention (3) [ slides ]	FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference
Week#9 10/24	Lecture #25 LLM Serving Workloads [ slides ] One-page Proposal Due	BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
Week#10 10/27	Lecture #26 LLM Inference Systems: Scheduling [ slides ]	Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
Week#10 10/29	Lecture #27 LLM Inference Systems: KV Cache (1) [ slides ]	Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention Fast State Restoration in LLM Serving with HCache
Week#10 10/31	Lecture #28 LLM Inference Systems: KV Cache (2) [ slides ] Intermediate Check-in Starting	CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (Optional) Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
Week#11 11/03	Lecture #29 LLM Inference Systems: RAG [ slides ]	CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models (Optional) Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
Week#11 11/05	Lecture #30 Affordable LLM Serving [ slides ]	SpotServe: Serving Generative Large Language Models on Preemptible Instances Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Week#11 11/07	Lecture #31 LLM Inference Systems: Quantization [ slides ]	AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
Week#12 11/10	Lecture #32 LLM Inference Systems: MoE [ slides ]	MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference (Optional) MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Week#12 11/12	Lecture #33 LLM Inference Systems: Multiplexing [ slides ]	MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
Week#12 11/14	Lecture #34 LLM Finetuning & Serving [ slides ]	S-LoRA: Serving Thousands of Concurrent LoRA Adapters FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning (Optional) LoRA: Low-Rank Adaptation of Large Language Models (Optional) QLoRA: Efficient Finetuning of Quantized LLMs
Week#13 11/17	Lecture #35 LLM Inference Systems: Applications [ slides ]	Teola: Towards End-to-End Optimization of LLM-based Applications Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
Week#13 11/19	Lecture #36 Reinforcement Learning [ slides ]	HybridFlow: A Flexible and Efficient RLHF Framework Optimizing RLHF Training for Large Language Models with Stage Fusion (Optional) ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
Week#13 11/21	Lecture #37 Diffusion Models (1) [ slides ]	Approximate Caching for Efficiently Serving Diffusion Models SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers
Week#14 11/24	Lecture #38 Diffusion Models (2) [ slides ]	Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding ScaleFusion: Scalable Inference of Spatial-Temporal Diffusion Transformers for High-Resolution Long Video Generation
Week#15 12/01	Lecture #39 LLM & Database [ slides ]	Semantic Operators: A Declarative Model for Rich, AI-based Data Processing Optimizing LLM Queries in Relational Workloads
Week#15 12/03	Lecture #40 Hardware for LLM [ slides ]	WaferLLM: Large Language Model Inference at Wafer Scale LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
Week#15 12/05	Lecture #41 Video Generation Systems [ slides ]	Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity Fast Video Generation with Sliding Tile Attention
Week#16 12/08	Lecture #42 Final Presentations [ slides ]		Group 1 Group 2 Group 3
Week#16 12/10	Lecture #43 Final Presentations [ slides ]		Group 4 Group 5 Group 6
Week#16 12/12	Lecture #44 Final Presentations [ slides ]		Group 7 Group 8 Group 9
12/17	Final Course Project Report Due