Guided Policy Search¶

Dependencies¶

The following are required

• python 2.7, numpy (v1.7.0+), matplotlib (v1.5.0+), scipy (v0.11.0+)
• boost, including boost-python
• protobuf (apt-get packages libprotobuf-dev and protobuf-compiler)

One or more of the following agent interfaces is required. Set up instructions for each are below.

• Box2D
• ROS
• MuJoCo

One of the following neural network libraries is required for the full guided policy search algorithm

• Caffe (master branch as of 11/2015, with pycaffe compiled, python layer enabled, PYTHONPATH configured)
• TensorFlow (Tested on versions 0.5-0.8. Works with and without GPU support).

Setup¶

Follow the following steps to get set up:

1. Install necessary dependencies above. To install protobuf and boost:

   sudo apt-get install libprotobuf-dev protobuf-compiler libboost-all-dev
   sudo pip install protobuf
   

2. Clone the repo:

   git clone https://github.com/cbfinn/gps.git
   

3. Compile protobuffer:

   cd gps
   ./compile_proto.sh
   

4. Set up one or more agents below.

Box2D Setup (optional)

Here are the instructions for setting up Pybox2D.

1. Install Swig and Pygame:

   sudo apt-get install build-essential python-dev swig python-pygame git
   

2. Check out the Pybox2d code via GitHub

   git clone https://github.com/pybox2d/pybox2d
   

3. Build and install the library:

   python setup.py build
   sudo python setup.py install
   

MuJoCo Setup (optional)

In addition to the dependencies listed above, OpenSceneGraph(v3.0.1+) is also needed. It can be installed by running sudo apt-get install openscenegraph libopenscenegraph-dev.

1. Install MuJoCo (v1.22+) and place the downloaded mjpro directory into gps/src/3rdparty. MuJoCo is a high-quality physics engine and requires requires a license. Obtain a key, which should be named mjkey.txt, and place the key into the mjpro directory.

2. Build gps/src/3rdparty by running:

   cd gps/build
   cmake ../src/3rdparty
   make -j
   

3. Set up paths by adding the following to your ~/.bashrc file:

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/gps/build/lib
   export PYTHONPATH=$PYTHONPATH:/path/to/gps/build/lib
   

   Don’t forget to run source ~/.bashrc afterward.

ROS Setup (optional)

1. Install ROS, including the standard PR2 packages

2. Set up paths by adding the following to your ~/.bashrc file:

   export ROS_PACKAGE_PATH=$ROS_PACKAGE_PATH:/path/to/gps:/path/to/gps/src/gps_agent_pkg
   

   Don’t forget to run source ~/.bashrc afterward.

3. Compilation:

   cd src/gps_agent_pkg/
   cmake .
   make -j
   

ROS Setup with Caffe (optional)

This is required if you intend to run neural network policies with the ROS agent.

1. Run step 1 and 2 of the above section.

2. Checkout and build caffe, including running make -j && make distribute within caffe.

3. Compilation:

   cd src/gps_agent_pkg/
   cmake . -DUSE_CAFFE=1 -DCAFFE_INCLUDE_PATH=/path/to/caffe/distribute/include -DCAFFE_LIBRARY_PATH=/path/to/caffe/build/lib
   make -j
   

   To compile with GPU, also include the option -DUSE_CAFFE_GPU=1.

Box2D example¶

There are two examples of running trajectory optimizaiton using a simple 2D agent in Box2D. Before proceeding, be sure to set up Box2D.

Each example starts from a random controller and learns through experience to minimize cost.

The first is a point mass learning to move to goal position.

To try it out, run the following from the gps directory:

python python/gps/gps_main.py box2d_pointmass_example

The progress of the algorithm is displayed on the GUI. The point mass should start reaching the visualized goal by around the 4th iteration.

The second example is a 2-link arm learning to move to goal state.

To try it out, run this:

python python/gps/gps_main.py box2d_arm_example

The arm should start reaching the visualized goal after around 6 iterations.

All settings for these examples are located in experiments/box2d_[name]_example/hyperparams.py, which can be modified to input different target positions and change various hyperparameters of the algorihtm.

MuJoCo example¶

To run the mujoco example, be sure to first set up MuJoCo.

The first example is using trajectory optimizing for peg insertion. To try it, run the following from the gps directory:

python python/gps/gps_main.py mjc_example

Here the robot starts with a random initial controller and learns to insert the peg into the hole. The progress of the algorithm is displayed on the GUI.

Now let’s learn to generalize to different positions of the hole. For this, run the guided policy search algorithm:

python python/gps/gps_main.py mjc_badmm_example

The robot learns a neural network policy for inserting the peg under varying initial conditions.

To tinker with the hyperparameters and input, take a look at experiments/mjc_badmm_example/hyperparams.py. Additionally, the neural network library can be changed through the ALGORITHM_NN_LIBRARY variable which can be set to caffe or tf.

PR2 example¶

To run the code on a real or simulated PR2, be sure to first follow the instructions above for ROS setup.

roslaunch gps_agent_pkg pr2_real.launch

roslaunch gps_agent_pkg pr2_gazebo.launch

python python/gps/gps_main.py pr2_example

python python/gps/gps_main.py pr2_badmm_example

Running a new experiment¶

1. Set up a new experiment directory by running:

   python python/gps/gps_main.py my_experiment -n
   

   This will create a new directory called my_experiment/ in the experiments directory, with a blank hyperparams.py file.

2. Fill in a hyperparams.py file in your experiment. See pr2_example and mjc_example for examples.

3. If you wish to set the initial and/or target positions for the pr2 robot agent, run target setup:

   python python/gps/gps_main.py my_experiment -t
   

   See the GUI documentation for details on using the GUI.

4. Finally, run your experiment

   python python/gps/gps_main.py my_experiment
   

All of the output logs and data will be routed to your experiment directory. For more details, see intended usage.