Sultana, Khadija (2022) Elastic Optimization for Stragglers in Edge Federated Learning. Research Master thesis, Victoria University.

Abstract

With the increasing proliferation of intelligent edge devices and their applications, machine learning has contributed significantly to regression analysis. Over the past few decades, the sub-fields of machine learning have evolved, such as evolutionary computing, computer vision, natural language processing, neural networks and speech recognition. Traditionally, machine learning is performed by collecting a huge amount of data at a centralized location, often raising privacy concerns. More recently, edge computing has been introduced where computation moves closer to the edge devices which decreases the latency associated with the centralized cloud. Moreover, edge servers help in computation offloading which decreases the data transit time to the central cloud. However, data privacy issues remain unresolved. Hence, federated learning has emerged as a thought-provoking technology for keeping data at the source and obtaining a collaborative predictive model. Further, federated learning follows rigid client-server architecture where one server communicates with many clients. A single global server is a single point of failure. In addition, clients have to communicate with global servers and communication latency is high in this context. Therefore, to deal with these issues, federated learning is seen as a critical application in edge computing. Hence, to fully exploit the enormous data generated by devices in edge computing, edge federated learning is a promising solution. The distributed collaborative training in EFL deals with delay and privacy issues compared to traditional model training methods. However, the existence of straggling devices degrades model performance. Stragglers are caused by data and system heterogeneity. The straggler effect can be alleviated by reinforced device selection by edge servers which can solve device heterogeneity to some extent. But, the challenge of statistical heterogeneity remains unsolved. We investigate heterogeneity in data from two aspects: high-dimensional data generated at edge devices where the number of features is greater than the number of observations and the heterogeneity caused by partial device participation. With a large number of features, the computation overhead on the devices increases, causing edge devices to become stragglers. Also, the submission of partial results causes gradients to be diverged as more local training is performed to reach local optima. In this thesis, we introduce elastic optimization for stragglers due to data heterogeneity in edge federated learning. Specifically, we define the problem of stragglers in edge federated learning. Then, we formu-late an optimization problem to be solved at the edge devices. We customize the benchmark algorithm, FedAvg, to obtain a new elastic optimization algorithm (FedEN) which is applied in the local training of edge devices. FedEN eradicates stragglers by achieving a balance between lasso and ridge penalization, thereby generating sparse model updates and forcing the parameters to be as close as possible to local optima. We experiment on the MNIST and CIFAR- 10 datasets for the proposed model. Simulated experiments demonstrate that the proposed approach improves the run time training performance by achieving target accuracy in less communication rounds. The results confirm the improved performance of the FedEN approach over benchmark algorithms.

Search Google Scholar

Repository staff login