Improving Efficacy and Efficiency of Hypothesis Adaptation and Federated Learning

Abstract

Small computing devices, such as smartphones, constantly collect and store data that can potentially help machines learn complex tasks. However, the data contained on a single device is relatively small and biased, making Machine Learning difficult. Although it is possible to transfer data from other devices, doing so poses storage, communication, and privacy concerns. Hypothesis Transfer Learning (HTL) provides a solution to this dilemma by transferring and importing knowledge learned from data on other devices in form of pretrained Machine Learning models (hypotheses). In its simplest form, a hypothesis can be transferred from one task or domain to another. Training a model on the target task can benefit from the knowledge embedded in the transferred hypothesis with no direct access to the source data. This often leads to significantly lower storage and communication costs as well as fewer privacy concerns. HTL can be extended to a chain of transfers and adaptations in the context of Continual Learning. Federated Learning (FL) further extends this idea by coordinating concurrent transfers from multiple sources at each transfer iteration. It enables a large number of remote devices to train a model collaboratively without sharing their data. In this thesis, we study some challenges associated with transferring and adapting pretrained models in the context of task adaptation and FL. We discuss an often overlooked source of inefficiency in feature-tuning, the most popular method of task adaptation. Accordingly, we propose two novel methods for improving the efficiency of feature-tuning which work by gradually increasing the magnitude of updates to pretrained feature-extractors. In addition, we present a new algorithm that addresses the issue of client drift which is known to make existing FL algorithms sub-optimal in heterogeneous settings. The thesis includes several experiments with image classification benchmarks for each of the learning settings under the study. Our findings show that the methods we propose can significantly improve their baselines in terms of accuracy and efficiency. Although our methods provide practical improvements over existing baselines, there is still room for further improvement as well as better understanding of the underlying mechanisms, which we leave to future research.