Performance of Parallel Computing in Training Deep Learning Models

Abstract

The demand for computing resources required to train new deep learning models is currently constantly growing. This growth has led to the introduction of specialized processors called accelerators. Accelerators such as graphics processing units (GPUs) and tensor processing units (TPUs) achieve better performance than a central processing unit (CPU) due to their parallel processing resources and high memory bandwidth. Google Colaboratory (Colab) is a cloud service that allows you to write Python code in Jupyter Notebook, provides free access to TPU and GPU hardware platforms, allowing you to develop modern deep learning models in the browser. The main goal of this article is to compare the training performance of models running on Colaboratory cloud platform accelerators designed for processing text and graphic data. A comparison of CPUs and GPU and TPU accelerators is presented when training image recognition using convolutional neural network (CNN) and deep belief network (DBN) models. The performance of these platforms is compared when training bidirectional long short-term memory (BILSTM) and guided recurrent unit (BIGRU) models with text input. Dependences of time characteristics and quality metrics of model training on their parameters were obtained, which make it possible to quantitatively assess the performance of the platforms. To implement the parallelism of all selected models, the Tensorflow Google library was selected, which supports parallel computing on GPU and TPU accelerators. Image recognition training was performed using the MNIST (Modified National Institute of Standards and Technology) dataset; text classification training was performed using the IMDB (Internet Movie Database) dataset. It is shown that in this case, the training time on graphic and text data is reduced significantly. However, in the case of texts, the degree of reduction was less.