使用深度学习来进行击剑动作识别的裁判工作

在击剑比赛中，当双方几乎同时击中对方时，记分板两边都会亮起。这时裁判需要决定哪一方得分。一般而言，谁更主动或控制了局势就会得分。我尝试训练了一个模型来辅助裁判做这样的判断！目前该模型在花剑测试集上的准确率大约为60%，相比随机选择（左、右或无得分）的33%有了提升。接下来我将对佩剑数据进行测试。（在重剑比赛中，双方同时击中则双方都得分。）

这里有几个主要挑战，首要的是缺乏标注过的击剑视频数据集！通常来说，视频分类领域还没有图像分类那么成熟，即使是像SPORTS 1-M这样的流行视频数据集，仅通过单帧图像就能达到非常不错的准确度。大多数用于视频分类的模型架构结合了卷积神经网络和循环神经网络：卷积层提取每帧的特征，而循环网络则基于这些特征做出最终判断。一个特别有趣的想法是构建一个全循环卷积网络，其中每一层都是循环的。目前我采用了一种需要较少数据的方法，即使用预训练的InceptionV3网络对每个视频帧提取卷积特征（即将每帧转换为1x2048维的向量）。Inception网络中这个特征向量接着被一个全连接层用来分类ImageNet数据库中的图像，但同样的特征向量也可以很好地表示我的视频帧中的情况。然后一个多层LSTM网络在这些特征向量序列上进行训练。我在AWS的p2 spot实例上进行了模型训练。

为了使网络更快地捕捉到相对运动的概念，在将每帧转换为特征向量之前，计算出该帧与前一帧之间的密集光流，并将其映射到彩色轮盘上叠加在该帧上。原始帧被转为黑白，以便网络不必学习区分运动和原始颜色。使用光流的灵感来源于Karen Simonyan和Andrew Zisserman的论文《Two-Stream Convolutional Networks for Action Recognition in Videos》。

我希望对探索机器学习新应用感兴趣的人能够从这里提供的代码中获益。它将引导你从创建自己的数据集一直到如何加载和使用预训练模型，以及训练和部署自己的模型。

创建击剑视频片段数据库简而言之，我下载了所有世界杯击剑比赛的视频，并使用OpenCV将视频分割成短片段，这些片段涵盖了记分板每次亮起前的时间。然后，我训练了一个小型逻辑回归分类器来识别记分板数字的变化。在某些情况下（仅当双方同时击中且至少有一方命中有效区域时），裁判必须做出判断。根据记分板的变化方式，我们可以自动标记这些片段来表明是谁得分。所有只有一个灯亮的片段都被丢弃，因为它们无法自动标注。

随后，我对视频进行了下采样处理，保留了更多结尾部分的帧。这些片段被水平翻转以增加数据集的大小。之后，我使用OpenCV在片段上叠加了光流图。最后，这些片段被转换为numpy数组并使用hickle包进行保存和压缩。由于文件很大，数据集是以每100个片段为一个块来保存的，每个块的尺寸为100 x 帧数 x 高 x 宽 x 深度。最终我从能找到的比赛视频中获得了大约5,500个片段，加上水平翻转后变成了约11,000个片段。

模型架构因为我们没有大量的样本，使用迁移学习来提取特征而不是训练自己的卷积层是有意义的。这样只有顶部的循环网络需要从头开始学习。我尝试了几种架构，发现4层带有0.2的dropout率效果不错。如果不使用dropout作为正则化手段，模型会开始过拟合。我计划不久后研究批量归一化作为正则化方法。

使用预训练的Inception网络这里所做的就是取用InceptionV3网络倒数第二层的张量。~ 正在继续编写中。

下一步我已经在玩具问题上实现了全循环网络的例子（例如在MNIST数据集上只显示图像的部分切片，模拟时间维度）。很快我会启动服务器并重新下载/处理所有数据，因为这里的互联网速度太慢以至于无法上传完整数据（约40GB），但在笔记本电脑上处理数据然后上传特征向量是没有问题的。首先，我很好奇模型在佩剑数据集上的表现是否会更好，所以将在接下来的一周内运行测试。


# coding: utf-8

# In[1]:

############ To use tensorboard, include all the summary op code below, and type: 
# tensorboard --logdir=/tmp/logs_path
# to launch type http://0.0.0.0:6006/ in a search bar 
# to kill ps -e | grep tensorboard in terminal
# the first number is the PID, type kill PID

import tensorflow as tf
import numpy as np
import argparse
import time
import subprocess as sp
import os
import hickle as hkl
print tf.__version__

num_layers = 4
drop_out_prob = 0.8
batch_size = 30
epochs = 30
learning_rate = 0.00001
test_size = 800
validation_size = 600
# In[40]:

videos_loaded = 0
for i in os.listdir(os.getcwd()):
    if i.endswith(".hkl"):
        if 'features' in i:
            print i
            if videos_loaded == 0:
                loaded = hkl.load(i)
            else:
                loaded = np.concatenate((loaded,hkl.load(i)), axis = 0)
            videos_loaded = videos_loaded + 1
            print loaded.shape


# In[41]:

videos_loaded = 0
for i in os.listdir(os.getcwd()):
    if i.endswith(".hkl"):
        if "labels" in i:
            print i
            if videos_loaded == 0:
                labels = hkl.load(i)
            else:
                labels = np.concatenate((labels,hkl.load(i)), axis = 0)
            videos_loaded = videos_loaded + 1
            print labels.shape
## Orders match up! Thats good.



def unison_shuffled_copies(a, b):
    assert len(a) == len(b)
    p = np.random.permutation(len(a))
    return a[p,:,:], b[p,:]

#reorder identically.
loaded, labels = unison_shuffled_copies(loaded,labels)
print loaded.shape, labels.shape

## Take the first 'test_size' examples as a test set. 
test_set = loaded[:test_size]
test_labels = labels[:test_size]
## Take indices from end of test_set to end of validation set size as our validation set
validation_set = loaded[test_size:(validation_size+test_size)]
validation_labels = labels[test_size:(validation_size+test_size)]
test_set_size = len(test_set)
## Now cut off the first 'test_size' + valid set numbers examples, because those are the test and validation sets
loaded = loaded[(test_size+validation_size):]
labels = labels[(test_size+validation_size):]
print "Test Set Shape: ", test_set.shape
print "Validation Set Shape: ", validation_set.shape
print "Training Set Shape: ", loaded.shape

## Save our test to test when we load the model we've trained. 
hkl.dump(test_set, 'test_data.hkl', mode='w', compression='gzip', compression_opts=9)
hkl.dump(test_labels, 'test_lbls.hkl', mode='w', compression='gzip', compression_opts=9)

device_name = "/gpu:0"

with tf.device(device_name):

    tf.reset_default_graph()


    logs_path = '/tmp/4_d-0.8'
    
    ## How often to print results and save checkpoint.
    display_step = 40

    # Network Parameters
    n_input = 2048 # 2048 long covolutional feature vector
    n_hidden = 1024 # hidden layer numfeatures
    n_classes = 3 # There are 3 possible outcomes, left, together, right
    

    # tf Graph input
    x = tf.placeholder("float32", [None, None, n_input]) ## this is [batch_size, n_steps, len_input_feature_vector]
    y = tf.placeholder("float32", [None, n_classes])
    input_batch_size = tf.placeholder("int32", None)

    # Define weights
    weights = {
        'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
    }
    biases = {
        'out': tf.Variable(tf.random_normal([n_classes]))
    }


    def RNN(x, weights, biases):

        
        cell = tf.contrib.rnn.LSTMCell(n_hidden, state_is_tuple=True)
        ## Variational recurrent = True means the same dropout mask is applied at each step. 
        cell = tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=drop_out_prob)
        cell = tf.contrib.rnn.MultiRNNCell([cell] * num_layers, state_is_tuple=True)
        init_state = cell.zero_state(input_batch_size, tf.float32)
        outputs, states = tf.nn.dynamic_rnn(cell,x, initial_state = init_state, swap_memory = True) # swap_memory = True is incredibly important, otherwise gpu will probably run out of RAM
        print states
        # Linear activation, outputs -1 is the last 'frame'.
        return tf.matmul(outputs[:,-1,:], weights['out']) + biases['out']







    with tf.name_scope('Model'):
        pred = RNN(x, weights, biases)
    print "prediction", pred
    # Define loss and optimizer
    with tf.name_scope('Loss'):
        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
    with tf.name_scope('SGD'):
        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    # Evaluate model
    with tf.name_scope('Accuracy'):
        correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
        accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

    # Initializing the variables
    init = tf.global_variables_initializer()

    # Create a summary to monitor cost tensor
    tf.summary.scalar("loss", cost)
    # Create a summary to monitor accuracy tensor
    tf.summary.scalar("training_accuracy", accuracy)
    # Merge all summaries into a single op
    merged_summary_op = tf.summary.merge_all()


# In[ ]:

current_epochs = 0
# Launch the graph
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
    saver = tf.train.Saver()
    sess.run(init)
    summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())
    step = 0
    
    
        
    # Keep training until we don't have any more data
    while step < (len(labels)/batch_size):
        batch_x = loaded[step*batch_size:(step+1)*batch_size]
        batch_y = labels[step*batch_size:(step+1)*batch_size]
       
        # x is in form batch * frames * conv_feature_size
        # y is in form batch * labels
        # Run optimization op (backprop)
        _,acc,loss,summary = sess.run([optimizer,accuracy,cost,merged_summary_op], feed_dict={x: batch_x, y: batch_y, input_batch_size:batch_size})
        print 'ran'
        ## The divide/multiply makes sense because it is reducing it to an int, i.e getting the whole number, could also cast as int, but this shows intent better.
        summary_writer.add_summary(summary, step*batch_size+current_epochs * (len(labels)/batch_size)*batch_size)

        if step % display_step == 0:
            # Calculate batch accuracy

            # calculate validation-set accuracy, because the GPU RAM is too small for such a large batch (the entire validation set, feed it through bit by bit)
            accuracies = []
            train_drop_out_prob = drop_out_prob
            drop_out_prob = 1.0
            for i in range(0,validation_size/batch_size):
                validation_batch_data = validation_set[i*batch_size:(i+1)*batch_size]
                validation_batch_labels = validation_labels[i*batch_size:(i+1)*batch_size]
                validation_batch_acc,_ = sess.run([accuracy,cost], feed_dict={x: validation_batch_data, y: validation_batch_labels, input_batch_size: batch_size})    
                accuracies.append(validation_batch_acc)
                

            # Create a new Summary object with your measure and add the summary
            summary = tf.Summary()
            summary.value.add(tag="Validation_Accuracy", simple_value=sum(accuracies)/len(accuracies))
            summary_writer.add_summary(summary, step*batch_size +current_epochs * (len(labels)/batch_size)*batch_size)
            # Save a checkpoint
            saver.save(sess, 'fencing_AI_checkpoint')
            
            print "Validation Accuracy - All Batches:", sum(accuracies)/len(accuracies)
            
            ## Set dropout back to regularizaing
            drop_out_prob = train_drop_out_prob
                
            print "Iter " + str(step*batch_size+current_epochs * (len(labels)/batch_size)*batch_size) + ", Minibatch Loss= " + \
                  "{:.6f}".format(loss) + ", Train Accuracy= " + \
                  "{:.5f}".format(acc)
                    
            


        step += 1
        if current_epochs < epochs:
            if step >= (len(labels)/batch_size):
                print "###################### New epoch ##########"
                current_epochs = current_epochs + 1
                learning_rate = learning_rate- (learning_rate*0.15)
                step = 0

                # Randomly reshuffle every batch.
                loaded, labels = unison_shuffled_copies(loaded,labels)

        
    print "Learning finished!"