Abracadabra

Reproduce DQN result

Posted on | | Visitors

论文链接：https://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

源代码地址：https://sites.google.com/a/deepmind.com/dqn/

由于源代码中只有训练阶段，没有测试阶段，因此我才用了这个项目的测试脚本，并且生成游戏动图。

实验复现的步骤如下（这里引用作者原文）：

DQN 3.0

This project contains the source code of DQN 3.0, a Lua-based deep reinforcement learning architecture, necessary to reproduce the experiments described in the paper “Human-level control through deep reinforcement learning”, Nature 518, 529–533 (26 February 2015) doi:10.1038/nature14236.

To replicate the experiment results, a number of dependencies need to be installed, namely:

  • LuaJIT and Torch 7.0
  • nngraph
  • Xitari (fork of the Arcade Learning Environment (Bellemare et al., 2013))
  • AleWrap (a lua interface to Xitari) An install script for these dependencies is provided.

Two run scripts are provided: run_cpu and run_gpu. As the names imply, the former trains the DQN network using regular CPUs, while the latter uses GPUs (CUDA), which typically results in a significant speed-up.

Installation instructions

The installation requires Linux with apt-get.

Note: In order to run the GPU version of DQN, you should additionally have the NVIDIA® CUDA® (version 5.5 or later) toolkit installed prior to the Torch installation below. This can be downloaded from https://developer.nvidia.com/cuda-toolkit and installation instructions can be found in http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux

To train DQN on Atari games, the following components must be installed:

  • LuaJIT and Torch 7.0
  • nngraph
  • Xitari
  • AleWrap

To install all of the above in a subdirectory called ‘torch’, it should be enough to run

1
./install_dependencies.sh

from the base directory of the package.

Note: The above install script will install the following packages via apt-get: build-essential, gcc, g++, cmake, curl, libreadline-dev, git-core, libjpeg-dev, libpng-dev, ncurses-dev, imagemagick, unzip

Training DQN on Atari games

Prior to running DQN on a game, you should copy its ROM in the ‘roms’ subdirectory. It should then be sufficient to run the script

1
./run_cpu <game name>

Or, if GPU support is enabled,

1
./run_gpu <game name>

Note: On a system with more than one GPU, DQN training can be launched on a specified GPU by setting the environment variable GPU_ID, e.g. by

1
GPU_ID=2 ./run_gpu <game name>

If GPU_ID is not specified, the first available GPU (ID 0) will be used by default.

这之后是我采用另一个项目的测试步骤：

Storing a .gif for a trained network

Once you have a snapshot of a network you can run

1
./test_gpu <game name> <snapshopt filename>

to make it play one game and store the .gif under gifs. For example

1
./test_gpu breakout DQN3_0_1_breakout_FULL_Y.t7

Options

Options to DQN are set within run_cpu (respectively, run_gpu). You may, for example, want to change the frequency at which information is output to stdout by setting ‘prog_freq’ to a different value.

我在实验过程中碰到了一系列的问题，实验环境为Windows 10 Vmware Workstation中运行的Ubuntu 16.04 LTS虚拟机。需要声明的是，我才用原始代码并没有运行成功，经分析应该是虚拟机的问题，但以下碰到的问题应当是具有一般性地，下一步准备采用测试项目代码运行。

Some Problem

./run_cpu之后出现Segmentation fault错误

  1. 可能是因为其后参数名中出现大写字母
  2. 可能是因为内存不足，可以尝试换一个游戏运行

./test_cpu之后提示找不到gd

这个时候需要手动安装gd，具体安装方法如下：

下载地址：https://ittner.github.io/lua-gd/manual.html#download

我下载的是这个版本http://files.luaforge.net/releases/lua-gd/lua-gd/lua-gd-2.0.33r2forLua5.1/lua-gd-2.0.33r2.tar.gz

下载解压后，进到对应的目录，

执行命令：

make

make成功后，执行：

sudo make install

如果中间出现错误的话，请把下面的几个包都安装上：

sudo apt-get install lua5.1
sudo apt-get install lua5.1-0-dev
sudo apt-get install liblua5.1-0-dev
sudo apt-get install libgd2-dev

安装成功之后会有如下提示：

1
2
3
4
5
6
7
8
9
10
11
12
13
gcc -o gd.so `gdlib-config --features |sed -e "s/GD_/-DGD_/g"`
`gdlib-config --cflags` `pkg-config lua5.1 --cflags` -O3 -Wall -shared
`gdlib-config --ldflags` `gdlib-config --libs` `pkg-config lua5.1 --libs`
-lgd luagd.c
lua test_features.lua
Lua-GD version: lua-gd 2.0.33r2
Lua-GD features:
    PNG support ..................... Enabled
    GIF support ..................... Enabled
    JPEG support .................... Enabled
    XPM/XBM support ................. Enabled
    FreeType support ................ Enabled
    Fontconfig support .............. Enabled

安装gd时进行make的时候出现gd.h: No such file or directory

Try to install this package if you are in debian : libgd2-noxpm-dev

安装gd时进行make的时候出现srlua makefile error lua.h No such file or directory

sudo apt-get install liblua5.1-0-dev

在解决以上问题后依然通不过编译

这里引用了https://groups.google.com/forum/#!topic/bamboo-cn/myYzVk5XLgc