Posts

Improving Federated Learning Personalization via MAML Abstract FL algorithms share many similarities with MAML and can be interpreted through meta-learning algorithms Fine-tuning gives the global model stronger accuracy while making personalization easier Models trained on standard centralized datasets are harder to personalize than those trained with FedAvg Introduction Points out the connection between FL and MAML, and interprets the FL algorithm via MAML Improves FedAvg with a two-stage approach of training and fine-tuning Finds that FedAvg is essentially a meta-learning algorithm that optimizes personalization performance rather than the global model. Interpreting FedAvg as a Meta Learning Algorithm The figure below shows MAML applied in the FL setting (left), the Reptile algorithm (middle), and FedAvg, the training algorithm of FL (right). Let L be the loss function. In each round of iteration, MAML trains by randomly sampling a batch of tasks T. For each task T there is an inner loop, and the outer loop aggregates the gradient updates obtained from each task. The FL algorithm randomly samples several clients T. For each T and its weight, several rounds of local optimization are performed, and the updated gradients are aggregated into a new global model. If we simplify the setting and assume all clients have the same amount of data, all weights become equal, and Reptile and FedAvg are then effectively the same algorithm. ...

Federated Adversarial Domain Adaptation Federated learning currently has the problem of knowledge domain shift, which prevents it from having good generalization ability. Domain shift occurs when the labeled data of the source nodes and the unlabeled data of the target node are different. This paper mainly proposes a way to address the use of knowledge domains in federated learning, so that the knowledge learned by different nodes can be aligned with the data distribution of the target node. ...

Memory GAN Reading Notes Introduction This paper mainly solves two problems in training unsupervised GANs. First, since generative adversarial networks use only a continuous latent distribution to represent multiple classes or clusters of data, they usually cannot correctly handle the structurally discontinuous nature between different classes in the latent space. (Caused by the mode collapse problem.) For example, if a GAN embeds buildings and cats in the same continuous latent distribution, the GAN may generate realistic images in the transition region between the two classes. ...

Getting Started with Kettle Since I was still one credit short in my final semester, I had to take one more course, and looking at the schedule the only option was OLAP — yet another completely unfamiliar subject. Tutorials about Kettle online are scarce, so I gathered some materials and jotted down these notes on the tool this course requires Introduction to Kettle Kettle is an ETL (Extract, Transform and Load) tool for data extraction, transformation, and loading. ETL tools are used very frequently in data warehouse projects, and Kettle can also be applied in the following scenarios: ...

One-shot Learning with Memory-Augmented Neural Networks Paper Notes Introduction Traditional gradient-based deep learning methods require large amounts of data to learn. When confronted with new data, the model has to relearn new parameters and cannot quickly adapt to it. This paper proposes a memory-augmented neural network that can rapidly assimilate new data and, after learning from only a few samples, use that data to make accurate predictions. For the design of the MANN model, the authors had two requirements: ...