Abstract: Long-Short Term Memory Recurrent Neural Networks (LSTM RNN) is one of the most important and widely used deep learning networks. It is capable of learning and deploying problems that require sequential data processing, e.g. language translation engine, speech recognition, text processing, etc. In this talk, I will introduce a set of optimizations for implementing LSTM RNN on NVIDIA's GPU from the perspective of increasing scheduling efficiency. Specifically, the major optimizations include 1) manipulating the core operations -- matrix-matrix multiplication (GEMM) -- to increase computational efficiency; 2) scheduling the dependency graph generated from the LSTM RNN to increase GPU occupancy and 3) re-arranging GPU allocation to increase memory bandwidth utilization.

Short Bio: Jin Wang is a senior GPU architect at NVIDIA. She works on designing and implementing new scheduling software and hardware inspired by deep learning applications for future GPU products. She received her PhD degree at Georgia Institute of Technology where she performed research on optimizing irregular applications on GPU using new execution models for dynamic parallelism with efficient compiler and architectural support.