CS67800 - (Graduate) Deep Learning
CS67800: Fall 2018, LWSN 1106 — Time: MW 4:30-5:45pm
Schedule
Instructor
Bruno Ribeiro
LWSN 2142C (please use Piazza to communicate with me)
Description
This is the website for CS67800 (graduate) Deep Learning.
Space in this course is quite limited, see Prerequisites to better understand enrollment priority.
Students that fulfill the prerequisits should enroll on the waiting list.
Completion of CS 57300 or CS 57800 with a B+ or better required.
There is NO waiving of pre-requisites. We will be selecting students from the waiting list according to depatmental needs and the student grades at CS57300 and CS57800.
Submit a request by August 13, 2018
here.
TA, Communication and Help
All communication will be handled via Piazza. Please do not email me or the TA.
Higher priority is given to messages that are public to all students.
Learning objectives
This is an in-depth
research-oriented course on deep learning.
Please consider other offers on campus if you are only interested in deep learning as a tool.
A lot of our coding will be in numpy (or in pytorch as a numpy for GPU) rather than existing deep learning packages.
Upon completing the course, students should be able to:
- Understand the mathematical foundations of deep learning, including:
- Backpropagation and Feedforward Networks
- Robbins-Monro-type stochastic optimization
- Auto-encoders, Representation Learning, Vector Spaces and Manifolds
- Neural network models for structured data and the role of canonical representations in deep learning
- Conditional Random Fields, Neural CRFs, Neural Turing Machines
- Markov chains and Sequence learning, Backpropagation Through Time
- Generative models, Partition Functions, Noise-Contrastive Estimation, Variational Auto-encoders, RBMs, DBMs, GANs
- Posterior estimation in Bayesian neural networks, Hamiltonian Monte Carlo and variants
- Understand how to evaluate the performance of neural networks, as well as formulate and test hypotheses
- Understand how algorithmic elements interact to impact performance
Prerequisites
CS57800 (Statistical Machine Learning) or CS57300 (Data Mining) are prerequisites, absolutely no exceptions.
Enrollent priority is given to CS and STAT graduate students the highest grades in CS57800 or CS57300.
Research objectives also counts but in a limited fashion.
This course has heavy programming assigments and a heavy load of mathematical tools and concepts. All assigments are in Python 3. We assume all students are procifient in python or can learn it quickly (say, become experts in less than one week).
There will be no waiving of prerequisites.
Text
- (Required) Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, FREE online copy
The texts below are recommended but not required. Reading materials will be distributed as necessary. Reading assignments will be posted on the schedule, please check regularly.
Assignments and exams
There will be 5 homework assignments. The lowest homework grade will be discarded. Homework assignments should be submitted on Blackboard. Details will be provided in the assignments. Programming assignments should written in Python 3 and some assigments require pyTorch, unless otherwise noted.
All homework will be due on the designated deadlines. There are no homework extensions. Late homework assignments will not be accepted.
Questions about the details of homework assignments should be posted on Piazza, and will be answered by the TAs or instructor.
There will be one individual course project.
There will be one comprehensive (in-class) final exam. The final will be closed book and closed notes.
Grading
- Homework: 40%
- Midterm: 30%
- Final Project: 30%
Grades posting
here.
Late policy
All homework will be due on the designated deadlines. There are no homework extensions. Late homework assignments will not be accepted.
Academic honesty
Please read the departmental
academic integrity policy. This will be followed unless we provide written documentation of exceptions.
Unless stated otherwise, each student should write up their own solutions independently. You need to
indicate the names of the people you discussed a problem with; ideally you should discuss with no more than
two other people.
NO PART OF THE STUDENT'S ASSIGMENT SHOULD BE COPIED FROM ANOTHER STUDENT (Plagiarism). We encourage you to interact amongst yourselves: you may discuss and obtain help with basic concepts covered in lectures or the textbook, homework specification (but not solution), and program implementation (but not design). However, unless otherwise noted, work turned in should reflect your own efforts and knowledge. Sharing or copying solutions is unacceptable and could result in failure. We use copy detection software, so do not copy code and make changes (either from the Web or from other students). You are expected to take reasonable precautions to prevent others from using your work.
Any student not following these guidelines are subject to an automatic F (final grade).
Additional course policies
Please read the general course policies
here.
Schedule (Subject to Change)
| Date |
Topic |
Notes |
Reading |
Slides |
| 08/20 |
Course Overview and Machine Learning Review |
ML Review
|
Chapters 2 and 3 of Deep Learning book
|
Policy and Course Overview |
| 08/22 |
Types of Data, Probabilistic Models |
Data and Models
|
Chapters 3 of Deep Learning book
|
|
| 08/27 |
Learning Probabilistic Models (Maximum Likelihood Estimation, Gradient-based Optimization, Stochastic Gradient Methods) |
Learning Probabilistic Models
|
Chapters 3, 4, and 5 of Deep Learning book
|
|
| 08/29 |
Learning and Generalization Error (Capacity, Overfitting, Underfitting, Learning Curves) |
Learning and Generalization
|
Chapters 2, 3, 4, and 5 of Deep Learning book
|
|
| 08/27 |
Shallow vs Deep Networks (Logistic Regression and Deeper Models) |
Learning and Generalization
|
Chapters 2, 3, 4, and 5 of Deep Learning book
|
|
| 08/29 |
(Logistic Regression and Deeper Models) |
Learning and Generalization
|
Chapters 2, 3, 4, and 5 of Deep Learning book
|
|