# CS 294: Deep Reinforcement Learning, Fall 2017

Instructor: Sergey Levine |

GSIs:
Abhishek Gupta, Joshua Achiam |

Lectures:Monday and Wednesday 10:30am-12pm in 306 Soda Hall |

Office Hours: MW 1-2pm, F 9-10am, by appointment (see signup sheet on Piazza) |

Communication: Piazza will be used for announcements, general questions and discussions, clarifications about assignments, student questions to each other, and so on. To sign up, go to Piazza and sign up with “UC Berkeley” and “CS294-112”. We request that enrollment be restricted to students enrolled in the class or on the waitlist. |

If you are not enrolled, but are interested in following and discussing the course, there is a subreddit forum here: reddit.com/r/berkeleydeeprlcourse/ |

### Table of Contents

- Lecture Videos
- Lectures, Readings, and Assignments
- Prerequisites
- Related Materials
- Previous Offerings

# Fall 2017 Materials

## Lecture Videos

The course lectures are available below. The course is **not** being offered as an online course, and the videos are provided only for your personal informational and entertainment purposes. They are not part of any course requirement or degree-bearing university program.

For all videos, **click here**.

For live
stream, **click here**.

## Lectures, Readings, and Assignments

Below you can find an outline of the course. Slides and references will be posted as the course proceeds.

- Aug 23: Introduction and course overview (Levine)
- Aug 28: Supervised learning and imitation (Levine)
- Slides
**Homework 1 is out: Imitation Learning**

- Aug 30: Reinforcement learning introduction (Levine)
- Sep 4: Holiday - no class.
- Sep 6: Policy gradients introduction (Levine)
- Slides
**Homework 1 milestone due**

- Sep 8: Neural networks review (Achiam)
- Sep 11: Actor-critic introduction (Levine)
- Slides
**Homework 1 is due, Homework 2 is out: Policy Gradients**

- Sep 13: Value functions introduction (Levine)
- Sep 18: Advanced Q-learning algorithms (Levine)
- Sep 20: Optimal control and planning (Levine)
- Sep 25: Learning dynamical systems from data (Levine)
- Sep 27: Learning policies by imitating optimal controllers (Levine)
- Oct 2: Advanced model learning and images (Guest lecture: Chelsea Finn)
- Oct 4: Connection between inference and control (Levine)
- Slides
**Homework 3 is due, Homework 4 is out: Model Based RL**

- Oct 9: Inverse reinforcement learning (Levine)
- Slides
**Project proposal is due**

- Oct 11: Advanced policy gradients (natural gradient, importance sampling) (Achiam)
- Oct 16: Exploration (Levine)
- Oct 18: Exploration (part 2) and transfer learning (Levine)
- Slides
**Homework 4 is due**

- Oct 23: Meta-learning and other advanced topics (Levine)
- Oct 25: Parallelism and asynchrony, open problems and challenges (Levine)
- Oct 30: Advanced imitation learning (Gupta)
- Nov 1: Guest lecture: Igor Mordatch, OpenAI
- Nov 6: Guest lecture: Mohammad Norouzi, Google
**Project milestone**

- Nov 8: Guest lecture: Pieter Abbeel, UC Berkeley and OpenAI
- Nov 13: Guest lecture: Emma Brunskill, Stanford University
- Nov 15: Guest lecture: Aviv Tamar, UC Berkeley
- Nov 20: Guest lecture: John Schulman, OpenAI
- Nov 22: Holiday - no class
- Nov 27: Final project presentations 1
- Nov 29: Final project presentations 2

## Previous Offerings

A full version of this course was offered in Spring 2017. Lecture videos from Spring 2017, are available**here**

An abbreviated version of this course was offered in Fall 2015.

## Prerequisites

CS189 or equivalent is a prerequisite for the course. This course will assume some familiarity with reinforcement learning, numerical optimization and machine learning. Students who are not familiar with the concepts below are encouraged to brush up using the references provided right below this list. We’ll review this material in class, but it will be rather cursory.

- Reinforcement learning and MDPs
- Definition of MDPs
- Exact algorithms: policy and value iteration
- Search algorithms

- Numerical Optimization
- gradient descent, stochastic gradient descent
- backpropagation algorithm

- Machine Learning
- Classification and regression problems: what loss functions are used, how to fit linear and nonlinear models
- Training/test error, overfitting.

For introductory material on RL and MDPs, see

- CS188 EdX course, starting with
*Markov Decision Processes I* - Sutton & Barto, Ch 3 and 4.
- For a concise intro to MDPs, see Ch 1-2 of Andrew Ng’s thesis
- David Silver’s course, links below

For introductory material on machine learning and neural networks, see

## Related Materials

### John's lecture series at MLSS

- Lecture 1: intro, derivative free optimization
- Lecture 2: score function gradient estimation and policy gradients
- Lecture 3: actor critic methods
- Lecture 4: trust region and natural gradient methods, open problems

### Courses

- Dave Silver’s course on reinforcement learning / Lecture Videos
- Nando de Freitas’ course on machine learning
- Andrej Karpathy’s course on neural networks

### Relevant Textbooks

- Deep Learning
- Sutton & Barto, Reinforcement Learning: An Introduction
- Szepesvari, Algorithms for Reinforcement Learning
- Bertsekas, Dynamic Programming and Optimal Control, Vols I and II
- Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming
- Powell, Approximate Dynamic Programming