Objectives

This class will provide an introduction to deep learning. We will focus on the underlying scientific principles for deep learning, and will apply these principles to build networks using TensorFlow/Keras. The course will describe techniques to specify how to build deep networks that are effective at solving a range of tasks, such as classification and reinforcement learning/control.

Administrivia

Class schedule: Wednesday 9-10 (WGB G-08)/Friday 11-12 (WGB G-03)
Midterm exam: week 7        Final exam: TBA

Instructor: Gregory Provan
Office hours: Tuesdays 11-1 (WGB 1-71), but please feel free to email me (g.provan@cs.ucc.ie) for another time.

The curriculum and course lectures can be found here.

Free Online Books

I am drawing material from a range of different texts. Texts that may be helpful are:

1. Deep Learning by Yoshua Bengio, Ian Goodfellow and Aaron Courville
   
2. Neural Networks and Deep Learning by Michael Nielsen
   
3. Deep Learning by Microsoft Research
   
4. Deep Learning Tutorial by LISA lab, University of Montreal
   
5. neuraltalk by Andrej Karpathy : numpy-based RNN/LSTM implementation
6. Deep Learning in Neural Networks: An Overview
7. Artificial intelligence and machine learning: Topic wise explanation

Mathematical Background in Deep Learning

• Neural Networks and Deep Learning, ch.2 - Overview of  the backpropagation algorithm.
• Neural Networks and Deep Learning, ch.4 - A visual proof that neural nets can compute any function.
• Yes you should understand backprop - Exposing backprop's caveats and the importance of knowing that while training models.
• Artificial Neural Networks: Mathematics of Backpropagation - Picturing backprop, mathematically.
• Deep Learning Lecture 12: Recurrent Neural Nets and LSTMs - Unfolding of RNN graphs is explained, and potential problems about gradient descent algorithms discussed.
• Gradient descent algorithms in a saddle point - Visualize how different optimizers interacts with a saddle points.
• Gradient descent algorithms in an almost flat landscape - Visualize how different optimizers interacts with an almost flat landscape.
• Gradient Descent - From Andrew Ng's Coursera class above, showing an introduction of the gradient descent algorithm.
• Gradient Descent: Intuition - Andrew Ng's Coursera class:  intuition of the gradient descent algorithm.
• Gradient Descent in Practice 2: Learning Rate - How to adjust the learning rate of a neural network.

Grading: Your final grade will be assigned based on the following weighting:

• 35% midterm     week 7
  
• 35% final
• 30% continuous assessment
  

Notes: