Prof. Nicholas Dronen
Department of Computer Science
Engineering Center South Tower 121
dronen@colorado.edu
Neural networks have enjoyed several waves of popularity over the past half century. Each time they become popular, they promise to provide a general purpose artificial intelligence–a computer that can learn to do any task that you could program it to do. The first wave of popularity, in the late 1950s, was crushed by theoreticians who proved serious limitations to the techniques of the time. These limitations were overcome by advances that allowed neural networks to discover internal representations, leading to another wave of enthusiasm in the late 1980s. The second wave died out as more elegant, mathematically principled algorithms were developed (e.g., support-vector machines, Bayesian models). Around 2010, neural nets had a third resurgence. What happened over the past 20 years? Basically, computers got much faster and data sets got much larger, and the algorithms from the 1980s—with a few critical tweaks and improvements—appear to once again be state of the art, consistently winning competitions in computer vision, speech recognition, and natural language processing. The many accomplishments of the field have helped move research from academic journals into systems that improve our daily lives: apps that identify our friends in photos, automated vision systems that match or outperform humans in large-scale object recognition, phones and home appliances that recognize continuous, natural speech, self-driving cars, and software that translates from any language to any other language.
In this course, we’ll examine the history of neural networks and state-of-the-art approaches to deep learning. Students will learn to design neural network architectures and training procedures via hands-on assignments. Students will read current research articles to appreciate state-of-the-art approaches as well as to question some of the hype that comes with the resurgence of popularity.
The course is open to any students who have some background in cognitive science or artificial intelligence and who have taken introductory probability/statistics and linear algebra. Students must be competent in Python and NumPy – or be able to learn NumPy quickly.
Even though the lecture notes cover most of the material I care for you to know, the text will provide a more detailed and formal treatment of some of the topics. I’m very happy if you read the text in advance of class so that you can ask informed questions, or ask me to clarify material.
The primary text will be Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. The text is available online by chapter in html, but it should serve as a good reference and is worth purchasing. If you wish additional background reading, consult:
Wikipedia is often a useful resource.
In the second half of the course, we’ll discuss current articles from the literature, all of which will be available on arXiv or other online sources.
The grandfather of the modern neural net field is Geoffrey Hinton from the University of Toronto (now at Google). He taught a Coursera class in 2012; it is a bit dated, but he gives such beautiful explanations and intuitions that his lectures are well worth viewing. Coursera has since deprecated the material, but it is available via YouTube. Many of his tutorials and invited talks are available on the web and I recommend viewing these talks over pretty much any other way you might spend your time.
We can all delude ourselves into believing we understand some math or algorithm by reading, but implementing and experimenting with the algorithm is both fun and valuable for obtaining a true understanding. Students will implement small-scale versions of as many of the models we discuss as possible. I will give nine homework assignments that involve implementation over the semester, details to be determined. You must implement your solutions in Python. One or more of the assignments may involve writing a commentary on a research article or presenting the article to the class.
We will use Piazza for class discussion. Rather than emailing me, I encourage you to post your questions on Piazza. Assignments will be distributed and submitted via the CS department’s Moodle. Here’s the course Moodle page.
Instructions for viewing the Zoom livestream are in the Announcements section of the course’s Moodle page. Recordings are available after the lecture on Canvas; it can take up to four hours for the recording to be posted.
Semester grades will be weighted with the following proportions :
| Date | Topic | Subtopics | Readings | Lecture Materials | Assignments |
|---|---|---|---|---|---|
| Jan. 14 (M) | Introduction (I) |
|
#0 released (Self Assessment) | ||
| Jan. 16 (W) | Introduction (II) |
|
|||
| Jan. 21 (M) | MLK Day, no lecture | ||||
| Jan. 23 (W) | Learning (I) |
|
|||
| Jan. 28 (M) | Learning (II) |
|
|
||
| Jan. 30 (W) | Automatic differentiation (AD) |
|
|||
| Feb. 4 (M) | Optimization |
|
|
||
| Feb. 6 (W) | Regularization |
|
|
||
| Feb. 11 (M) | Convolutional Networks (I) |
|
|
||
| Feb. 13 (W) | Convolutional Networks (II) |
|
|||
| Feb. 18 (M) | Recurrent Networks (RNN) I |
|
|
||
| Feb. 20 (W) | Recurrent Networks (RNN) II |
|
|
||
| Feb. 25 (M) | Unsupervised and Representation Learning |
|
|
||
| Feb. 27 (W) | Generative Models (I) |
|
|||
| Mar. 4 (M) | Generative Models (II) |
|
|
||
| Mar. 6 (W) | Generative Models (III) |
|
|||
| Mar. 11 (M) | Hardware for Deep Learning | Guest lecture by Phil James-Roxby, Distinguished Engineer at Xilinx |
|
||
| Mar. 13 (W) | Campus closed | Bomb cyclone! | |||
| Mar. 18 (M) | No lecture |
|
|||
| Mar. 20 (W) | Midterm | 4:30-6:30, Room G125, Duane Physics and Astrophysics |
|||
| Mar. 25 (M) | Spring break, no lecture | ||||
| Mar. 27 (W) | Spring break, no lecture | ||||
| Apr. 1 (M) | Autoregressive Networks, Density Estimation | ||||
| Apr. 3 (W) | Recent Advances in Deep Learning for Natural Language Processing | Language Models |
|
||
| Apr. 8 (M) | Software for Deep Learning (I) |
|
|||
| Apr. 10 (W) | No lecture | Bomb cyclone! | |||
| Apr. 15 (M) | Software for Deep Learning (II) | ||||
| Apr. 17 (W) | Graph Neural Networks (I) |
|
|||
| Apr. 22 (M) | Deep Learning and Society | Guest lecture by Paul Diduch of CU's Herbst Program of Humanities in Engineering |
|
||
| Apr. 24 (W) | Graph Neural Networks (II) | ||||
| Apr. 29 (M) | No lecture | Work on your projects, presentations | h
|
||
| May. 1 (W) | Project presentations | Work on your projects, presentations |
Many thanks to Prof. Mike Mozer for allowing me to re-use many of his materials from a previous administration of this course.
All students enrolled in a University of Colorado Boulder course are responsible for knowing and adhering to the academic integrity policy of the institution. Violations of the policy may include: plagiarism, cheating, fabrication, lying, bribery, threat, unauthorized access, clicker fraud, resubmission, and aiding academic dishonesty. All incidents of academic misconduct will be reported to the Honor Code Council (honor@colorado.edu; 303-735-2273). Students who are found responsible for violating the academic integrity policy will be subject to nonacademic sanctions from the Honor Code Council as well as academic sanctions from the faculty member. Additional information regarding the academic integrity policy can be found at www.colorado.edu/honorcode/.
If you qualify for accommodations because of a disability, please submit your accommodation letter from Disability Services to your faculty member in a timely manner (for exam accommodations provide your letter at least one week prior to the exam) so that your needs can be addressed. Disability Services determines accommodations based on documented disabilities in the academic environment. Information on requesting accommodations is located on the Disability Services website. Contact Disability Services at 303-492-8671 or dsinfo@colorado.edu for further assistance. If you have a temporary medical condition or injury, see Temporary Medical Conditions under the Students tab on the Disability Services website and discuss your needs with your professor.
Campus policy regarding religious observances requires that faculty make every effort to deal reasonably and fairly with all students who, because of religious obligations, have conflicts with scheduled exams, assignments, or required attendance. If you have an exam or assignment conflict due to a religious observance please notify your instructor in a timely manner. See the campus policy regarding religious observances for full details.
Students and faculty each have responsibility for maintaining an appropriate learning environment. Those who fail to adhere to such behavioral standards may be subject to discipline. Professional courtesy and sensitivity are especially important with respect to individuals and topics dealing with race, color, national origin, sex, pregnancy, age, disability, creed, religion, sexual orientation, gender identity, gender expression, veteran status, political affiliation or political philosophy. Class rosters are provided to the instructor with the student’s legal name. We will gladly honor your request to address you by an alternate name or gender pronoun. Please advise me of this preference early in the semester so that I may make appropriate changes to my records. For more information, see the policies on classroom behavior and the Student Code of Conduct.
The University of Colorado Boulder (CU Boulder) is committed to maintaining a positive learning, working, and living environment. CU Boulder will not tolerate acts of sexual misconduct, discrimination, harassment or related retaliation against or by any employee or student. CU’s Sexual Misconduct Policy prohibits sexual assault, sexual exploitation, sexual harassment, intimate partner abuse (dating or domestic violence), stalking or related retaliation. CU Boulder’s Discrimination and Harassment Policy prohibits discrimination, harassment or related retaliation based on race, color, national origin, sex, pregnancy, age, disability, creed, religion, sexual orientation, gender identity, gender expression, veteran status, political affiliation or political philosophy. Individuals who believe they have been subject to misconduct under either policy should contact the Office of Institutional Equity and Compliance (OIEC) at 303-492-2127. Information about the OIEC, the above referenced policies, and the campus resources available to assist individuals regarding sexual misconduct, discrimination, harassment or related retaliation can be found at the OIEC website.