Curiosity-Driven Learning made easy Part I

This article is part of Deep Reinforcement Learning Course with Tensorflow 🕹️. Check the syllabus here.

OpenAI Five contest

In the recent years, we’ve seen a lot of innovations in Deep Reinforcement Learning. From DeepMind and the Deep Q learning architecture in 2014 to OpenAI playing Dota2 with OpenAI five in 2018, we live in an exciting and promising moment.

And today we’ll learn about Curiosity-Driven Learning, one of the most exciting and promising strategy in Deep Reinforcement Learning.

Reinforcement Learning is based on the reward hypothesis, which is the idea that each goal can be described as the maximization of the rewards. However, the current problem of extrinsic rewards (aka rewards given by the environment) is that this function is hard coded by a human, which is not scalable.

The idea of Curiosity-Driven learning, is to build a reward function that is intrinsic to the agent (generated by the agent itself). It means that the agent will be a self-learner since he will be the student but also the feedback master.

Data Splitting Strategies

• Random
• Timewise
• By id (maybe hidden)
• Combined

Notices

• Make sure the strategy used by train/val splitting is same as train/test splitting.
• Different models trained from different data splitting strategies have much performance gap.
• Logic of feature generation depends on the data splitting strategy.

Validation problems

• Validation stage
  • Causes of different scores and optimal parameters
    • Too little data
    • Too diverse and inconsistent data
  • Solutions
    • Average scores from different K-Fold splits
    • Tune model on one split and evaluate score on the other
• Submission stage
  • We can observe that
    • LB score is consistently higher/lower than validation score
    • LB score is not correlated with validation score at all
  • Causes
    • We may already have quite different scores in K-Fold
      • make sure split train/validation correct
    • too litter data in public LB
      • Just trust your validation scores
    • train and test data are from different distributions
      • classes show in the test set not show in the train set
        • make a shift to your prediction (mean of train minus mean of test) – LB probing
      • classes ratio is not same
        • make the validation classes ratio is same as test classes ratio

Expect LB shuffle because of

• Randomness
• Litter amount of data
• Different public/private distributions

I can't download the video ~

state: closed opened by: jenkey2011 on: 2017-05-04

I can't download the video ~

Your environment

• win10
• Python version : 3.6
• coursera-dl version:0.8
• PS：I'm from China……

  Steps to reproduce

  > coursera-dl -u xxx -p xxxx -b html-css-javascript

Then it works , but only the subtitles were downloaded ; And the cmd shows "The following URLs (64) could not be downloaded:" , they're all video links;

Comments

from: wanghoppe on: 2017-05-17

Me too have the issue…. Could someone help??

from: lvhuiyang on: 2017-05-18

I met the same problem yesterday.

I use the command `coursera-dl -u xx@xxx.com -p xxx course_name –wget` to solve it.

(env: Mac OS, wget, python3.6)

from: balta2ar on: 2017-05-18

Are you guys all from China? Did you try downloading over a VPN or Tor? If it's your government's firewall, we can't do anything about it, use proxies, VPNs and Tor.

from: jenkey2011 on: 2017-05-19

I guess so…… What a pity.

from: FBryce on: 2017-06-02

Hi, If you are from China, adding "52.84.246.72 d3c33hcgiwev3.cloudfront.net" in the host file and fresh dns with " ipconfig/flushdns" may work

from: wanghoppe on: 2017-06-02

This is a notebook, used in the screencast video. Note, that the data files are not present here in Jupyter hub and you will not be able to run it. But you can always download the notebook to your local machine as well as the competition data and make it interactive.

1
2
3
4
5
6
7
8
9
10
11

import os
import numpy as np
import pandas as pd 
from tqdm import tqdm_notebook
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
import seaborn

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

def autolabel(arrayA):
    ''' label each colored square with the corresponding data value. 
    If value > 20, the text is in black, else in white.
    '''
    arrayA = np.array(arrayA)
    for i in range(arrayA.shape[0]):
        for j in range(arrayA.shape[1]):
                plt.text(j,i, "%.2f"%arrayA[i,j], ha='center', va='bottom',color='w')
def hist_it(feat):
    plt.figure(figsize=(16,4))
    feat[Y==0].hist(bins=range(int(feat.min()),int(feat.max()+2)),normed=True,alpha=0.8)
    feat[Y==1].hist(bins=range(int(feat.min()),int(feat.max()+2)),normed=True,alpha=0.5)
    plt.ylim((0,1))
    
def gt_matrix(feats,sz=16):
    a = []
    for i,c1 in enumerate(feats):
        b = [] 
        for j,c2 in enumerate(feats):
            mask = (~train[c1].isnull()) & (~train[c2].isnull())
            if i>=j:
                b.append((train.loc[mask,c1].values>=train.loc[mask,c2].values).mean())
            else:
                b.append((train.loc[mask,c1].values>train.loc[mask,c2].values).mean())
        a.append(b)
    plt.figure(figsize = (sz,sz))
    plt.imshow(a, interpolation = 'None')
    _ = plt.xticks(range(len(feats)),feats,rotation = 90)
    _ = plt.yticks(range(len(feats)),feats,rotation = 0)
    autolabel(a)

1
2
3
4
5

def hist_it1(feat):
    plt.figure(figsize=(16,4))
    feat[Y==0].hist(bins=100,range=(feat.min(),feat.max()),normed=True,alpha=0.5)
    feat[Y==1].hist(bins=100,range=(feat.min(),feat.max()),normed=True,alpha=0.5)
    plt.ylim((0,1))

Read the data

1
2

train = pd.read_csv('train.csv.zip')
Y = train.target

1
2

test = pd.read_csv('test.csv.zip')
test_ID = test.ID