Abstract: GAN is one of the most popular and commonly used neural network models. When the model is large and there is a lot of data, the learning process can be delayed. The standard way out is to use multiple devices. Therefore, the methods of distributed and federated training for GANs are an important question. But from an optimization point of view, GANs are saddle-point problems: $\min_x \max_y f(x,y)$. Therefore, this paper focuses on the distributed optimization of smooth stochastic saddle-point problems. The first part of the paper is devoted to lower bounds for the distributed methods for saddle-point problems as well as the optimal algorithms by which these bounds are achieved. Next, we present a new algorithm for distributed saddle-point problems - Extra Step Local SGD. In the experimental part of the paper, we use the Local SGD technique in practice. In particular, we train GANs in a distributed manner.
Publication Year: 2020
Publication Date: 2020-10-25
Language: en
Type: article
Access and Citation
Cited By Count: 8
AI Researcher Chatbot
Get quick answers to your questions about the article from our AI researcher chatbot