Optimization methods are the engines underlying neural networks that enable them to learn from data. In this lecture, DeepMind Research Scientist James Martens covers the fundamentals of gradient-based optimization methods, and their application to training neural networks. Major topics include gradient descent, momentum methods, 2nd-order methods, and stochastic methods. James analyzes these methods through the interpretive framework of local 2nd-order approximations.

Download the slides here:
https://storage.googleapis.com/deepmi...

Find out more about how DeepMind increases access to science here:
https://deepmind.com/about#access_to_...

Speaker Bio:

James Martens is a Research Scientist at DeepMind working on the fundamentals of deep learning including optimization, initialization, and regularization. Before that he received his BMath from the University of Waterloo, and did his Masters and PhD at University of Toronto, coadvised by Geoff Hinton and Rich Zemel. During his PhD he helped revive interest in deep neural network training by showing how deep networks could be effectively trained using pure optimization methods (which has now become the standard approach).

About the lecture series:

The Deep Learning Lecture Series is a collaboration between DeepMind and the UCL Centre for Artificial Intelligence. Over the past decade, Deep Learning has evolved as the leading artificial intelligence paradigm providing us with the ability to learn complex functions from raw data at unprecedented accuracy and scale. Deep Learning has been applied to problems in object recognition, speech recognition, speech synthesis, forecasting, scientific computing, control and many more. The resulting applications are touching all of our lives in areas such as healthcare and medical research, human-computer interaction, communication, transport, conservation, manufacturing and many other fields of human endeavour. In recognition of this huge impact, the 2019 Turing Award, the highest honour in computing, was awarded to pioneers of Deep Learning.

In this lecture series, research scientists from leading AI research lab, DeepMind, deliver 12 lectures on an exciting selection of topics in Deep Learning, ranging from the fundamentals of training neural networks via advanced ideas around memory, attention, and generative modelling to the important topic of responsible innovation.