Posts on eree's blog

[Paper] TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH Reading Notes

Wed, 24 Jun 2020 18:55:03 +0800

TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH

Abstract

This paper mainly proposes a feature fusion approach to accelerate and improve the performance of federated learning.

Introduction

Nowadays many smart devices rely on pretrained models, which makes the machine’s inference capability lack personalization and flexibility. At the same time, smart terminals also generate a large amount of valid privacy-sensitive data, which can improve the personalization capability of these models. Federated learning, a distributed training algorithm that can train models directly on terminals, solves this problem. Among federated learning algorithms, those represented by FedAvg effectively alleviate the privacy issues in information exchange, but later research has also shown that federated learning still has problems such as: computational consumption, model accuracy.

This paper proposes a feature-fusion federated learning algorithm, FedFusion, which fuses the features of the global model and the local model. The three main contributions of this paper: 1. introducing a feature fusion mechanism; 2. fusing the features of the local model and the global model in an effective and personalized way; 3. experiments show that the model outperforms the baselines in both accuracy and generalization ability while reducing communication by more than 60%.

Mainly the FedAvg algorithm of Federated Learning; not much to say here.

Methods

It is mainly divided into the feature fusion module and the FedFusion algorithm.

Feature Fusion Modules

The blue features in the figure are the two-channel features extracted by the local model, and the gray ones are the two-channel features extracted by the global model. The figure shows three feature fusion methods: Conv, Multi, Single.

Conv:

F_{c o n v}\left(E_{l}(x), E_{g}(x)\right)=W_{c o n v}\left(E_{g}(x) \| E_{l}(x)\right)

where $W_{c o n v}$ denotes a learnable weight matrix of shape 2C*C. The concrete operation is to concatenate the global and local features and then apply a convolution.

Multi:

F_{m u l t i}\left(E_{l}(x), E_{g}(x)\right)=\lambda E_{g}(x)+(1-\lambda) E_{l}(x)

The multiplication operation uses a lambda weight matrix to perform a weighted sum of local and global.

Single:

F_{\text {single}}\left(E_{l}(x), E_{g}(x)\right)=\lambda E_{g}(x)+(1-\lambda) E_{l}(x)

The addition operation uses a scalar lambda weight to perform a weighted sum of local and global.

FedFusion

The training procedure uses the features of the previous round’s global model to participate in this round’s model feature aggregation training.

Experiment

Experiment setup

Datasets: Mnist, Cifar10

Data partition methods:

An artificial non-IID partition, where each node contains only two classes
A user-specific non-IID partition, where each node contains similar classes but with different distributions, similar to multi-task learning
IID distribution

Artificial Non-IID Partition

Experimental results of two random runs of the artificial non-IID sampling scheme. The experimental results show that the multi fusion method works best; the conv fusion method converges slightly faster, but its final result is not as good as multi; none and single are both mediocre.

The paper’s explanation is that the multi operation allows the model to select the feature maps that are effective for local data for fusion, while the single operation is a scalar and cannot select specific channels of the feature maps.

User-Specific Non-IID Partition

In terms of accuracy, FedFusion is much higher than FedAvg, with conv converging faster and also achieving higher accuracy.

The figure above shows the degree of communication reduction of FedFusion compared with FedAvg. From the results, in the user-specific non-IID scenario, the conv fusion method works better. This is because, under the user-specific non-IID partition, the classes of the data are similar, only with different distributions. And the conv fusion method is stronger at integrating the feature maps from the local and global models — that is, the knowledge of the data distributions of different nodes.

For the impact on generalization ability: when a new node joins, FedFusion only needs about 60 local epochs to fit, having a better initialization than the other methods.

IID Partition

The authors believe that the IID distribution also needs to be evaluated, because if a strategy cannot handle the IID distribution, its effectiveness is questionable.

The multi and conv fusion methods can achieve better accuracy with the least communication cost. In terms of the final converged accuracy, there is a great improvement compared with the other methods.

To summarize the three fusion methods:

The multi operation mainly makes a more flexible and more interpretable selection between the local feature maps and the global feature maps. Each channel of the weight vector represents the weight of the corresponding channel of the global feature map. When a gap appears between the classes of the data, the multi operation can select the most effective feature maps for fusion. The conv operation is more effective at integrating the knowledge of the global and local models. If the data on the nodes has similar classes but different distributions, the conv fusion method is better. The single fusion method brings a slight improvement.

Conclusion

Using feature map fusion reduces the communication volume, improves model performance, and also improves the generalization ability for new nodes.

Watermelon Book Reading Notes 1. Introduction

Wed, 24 Jun 2020 18:15:38 +0800

Introduction

Preamble

Machine learning: suppose we use P to evaluate the performance of a computer program on a certain class of tasks T; if a program improves its performance on the tasks in T through experience E, then we say that, with respect to T and P, the program has learned from E.

Basic Terminology

Data-related:

A dataset contains a series of records, and each record is an instance or a sample. That which reflects certain properties is an attribute or a feature. The value an attribute takes is called the attribute value, and the space spanned by the attributes is called the attribute space, the sample space, or the input space. Every point in this space is called a feature vector.

Training-related:

The process of learning a model from data is called learning or training. The data used is called training data, each sample is a training sample, and the whole collection is called the training set. The learned model corresponds to some underlying regularity in the data, which is called a hypothesis. The regularity itself is called the ground truth. The model can also be called a learner.

After training, predictions need to be made. Information about the outcome of an instance is called a label, and an instance with a label is an example. The space spanned by the labels is the label space or the output space.

Classification: the predicted values are discrete. Regression: the predicted values are continuous. Binary classification: the outputs are the positive class and the negative class.

The stage of making predictions with the learned model is called testing; the samples being predicted are called testing samples, and likewise there is a testing set

Supervised learning: training with labeled data, such as classification and regression.

Unsupervised learning: training with unlabeled data, such as clustering, which divides training samples into clusters.

Generalization: the ability of a model to work well on new samples

Independent and identically distributed (iid): the samples are all drawn independently from the same distribution.

Hypothesis Space

Induction: from the specific to the general. (generalization)

Deduction: from the general to the specific. (specialization)

Inductive learning: in the broad sense, learning from samples; in the narrow sense, acquiring concepts from data, which is called concept learning.

Version space: the set of hypotheses consistent with the training set

Inductive Bias

Inductive bias: the preference of a machine learning algorithm for a certain type of hypothesis during the learning process; it corresponds to the assumption the learning algorithm itself makes about “what kind of model is better”

Watermelon Book Reading Notes 2. Model Evaluation and Selection

Wed, 24 Jun 2020 16:06:16 +0800

Model Evaluation and Selection

Empirical Error and Overfitting

Error rate: the overall proportion of misclassified samples

Accuracy: the proportion of correctly classified samples

Error: the difference between the actual output and the true output; on training samples it is the empirical error or training error, and on new samples it is the generalization error

Overfitting: generalization performance degrades

Underfitting: performs poorly even on the training samples

Evaluation Methods

A testing set is typically used to measure a model’s performance, with the testing error serving as an approximation of the generalization error

Hold-Out Method

Split the dataset D into two mutually exclusive sets, one part as the training set S and the other as the testing set T. When S is large and T is small, the evaluation may not be accurate enough. If T is made larger, then the model trained on S may differ too much from the model that would be trained on D, and the evaluation result will not be accurate enough either.

Cross-Validation

Split the dataset D into k mutually exclusive subsets, use k-1 of them as the training set and the remaining one as the testing set; this yields k groups of data, and the average of the testing results of these k groups is returned as the final result. The extreme case is leave-one-out, which gives more accurate results but at a greater computational cost.

Bootstrap

Use bootstrap sampling to draw a new dataset $D'$ with replacement. Train on this new dataset and test on the rest. Advantage: this method is useful when the dataset is small and it is difficult to split training/testing sets effectively. Disadvantage: it changes the distribution of the initial dataset and introduces estimation bias.

Parameter Tuning and the Final Model

Algorithm parameters have a great impact on the algorithm’s performance. Besides choosing an algorithm, its parameters also need to be adjusted, i.e., parameter tuning. After the model has been evaluated and selected using a validation set and the model and parameter configuration are determined, the model must be retrained once on the entire dataset D before testing — this is the finally submitted model. To distinguish it, the data used during model evaluation is called the validation set. The validation set is used to evaluate algorithm selection and parameter tuning, while the testing set is used to evaluate the generalization performance of the algorithm.

Performance Measures

Performance measure: the evaluation criterion used to measure a model’s generalization ability

The most commonly used performance measure for regression tasks is MSE, the mean squared error:

E(f ; D)=\frac{1}{m} \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2}

More generally, it can be written as:

E(f ; D)=\int_{x \sim D}(f(x)-y)^{2} p(x) d x

Error Rate and Accuracy

The two most commonly used performance measures: error rate and accuracy

Error rate: $E(f ; D)=\frac{1}{m} \sum_{i=1}^{m} I\left(f\left(x_{i}\right) \neq y_{i}\right)$

Accuracy: $\begin{aligned} \operatorname{acc}(f ; D) &=\frac{1}{m} \sum_{i=1}^{m} \mathbb{I}\left(f\left(x_{i}\right)=y_{i}\right) \\ &=1-E(f ; D) \end{aligned}$

Precision, Recall, and F1

In information retrieval, one usually cares about “how much of the retrieved information the user is interested in” and “how much of what the user is interested in has been retrieved”. This motivates the concepts of precision and recall.

For binary classification problems,

Precision = true positives / (true positives + false positives)

Recall = true positives / (true positives + false negatives)

These two metrics usually conflict with each other: when recall is high, precision is relatively low. For example, to select as many good melons as possible, simply selecting every melon guarantees that all the good ones are selected.

P-R curve: the precision-recall curve. If one learner’s PR curve can completely envelop another learner’s curve, the former learner is better. A more reasonable criterion is the size of the area under the curve, but it is not easy to estimate, so there are three evaluation approaches.

Break-Even Point (BEP): the value at which precision equals recall.
F1 measure: F1=2PR/(P+R)= 2*TP/(total number of examples+TP-TN), based on the harmonic mean of P and R $\frac{1}{F_{1}}=\frac{1}{2}\left(\frac{1}{P}+\frac{1}{R}\right)$
The general form of the F1 measure (when P and R are weighted differently): $F_{\beta}=\frac{\left(1+\beta^{2}\right) \times P \times R}{\left(\beta^{2} \times P\right)+R}$ , based on the harmonic mean $\frac{1}{F_{\beta}}=\frac{1}{1+\beta^{2}}\left(\frac{1}{P}+\frac{\beta^{2}}{R}\right)$ $\beta$ represents the relative importance of the two: it degenerates to F1 when equal to 1, recall matters more when it is greater than 1, and precision matters more when it is less than 1.

Sometimes P and R need to be computed over n binary confusion matrices and then averaged, giving macro-precision, macro-recall, and macro-F1. Alternatively, the corresponding elements of the confusion matrices can be averaged first to obtain averaged positive and negative counts, from which micro-precision, micro-recall, and micro-F1 are then computed.

ROC and AUC

ROC: Receiver Operating Characteristic curve.

A learner typically outputs a probability prediction for an input sample, which is then compared against a classification threshold. The model ranks the samples most likely to be positive at the front and the least likely at the back, then splits the samples into two parts at a certain cut point: the first part is judged positive and the second part negative.

Different classification tasks adopt different cut points: if precision matters more, the cut point is moved forward; if recall is valued more, it is moved backward.

Sort the samples according to the learner’s prediction results, and compute two values in that order.

Vertical axis: True Positive Rate (TPR): $T P R=\frac{T P}{T P+F N}$

Horizontal axis: False Positive Rate (FPR): $FPR=\frac {FP} {TN+FP}$

When comparing learners, if one curve can envelop the other, the former is better; if they cross, neither clearly dominates. A more reasonable judgment is to compare the area under the ROC curve, i.e., the AUC.

A U C=\frac{1}{2} \sum_{i=1}^{m-1}\left(x_{i+1}-x_{i}\right)\left(y_{i}+y_{i+1}\right)

Loss: $\operatorname{l_{rank}}=\frac{1}{m^{+} m^{-}} \sum_{x^{+} \in D^{+}} \sum_{x \in D^{-}}\left(\mathbb{I}\left(f\left(x^{+}\right)$

AUC= 1-lrank

https://datawhalechina.github.io/pumpkin-book/#/chapter2/chapter2 For more details, see the Pumpkin Book

Cost-Sensitive Error Rate and Cost Curve

To weigh the different losses caused by different types of errors, errors can be assigned “unequal costs”

Taking binary classification as an example, set up a cost matrix

Binary classification cost matrix

True class	Predicted class 0	Predicted class 1
0	0	cost01
1	cost10	0

If misclassifying 0 as 1 is considered the greater loss, then cost01 is greater than cost10; the greater the difference in loss, the greater the difference between the values.

The performance measures introduced above all implicitly assume equal costs; under unequal costs, the goal becomes minimizing the total cost overall

The cost-sensitive error rate is:

E(f ; D ; \cos t)=\frac{1}{m}\left(\sum_{x_{i} \in D^{+}} \mathbb{I}\left(f\left(x_{i}\right) \neq y_{i}\right) \times cost_{01}, \quad+\sum_{X_{i} \in D^{-}} \mathbb{I}\left(f\left(x_{i}\right) \neq y_{i}\right) \times cost_{1 0}\right)

Under unequal costs, the ROC curve cannot reflect the learner’s expected total cost, but the cost curve can serve this purpose.

The horizontal axis is the positive-class probability cost over [0,1]:

P(+) cost=\frac{p \times cost_{01}}{p \times cost_{01}+(1-p) \times cost_{10}}

where p is the probability that an example is positive

The vertical axis is the normalized cost over [0,1]:

cost_{\text {norm }}=\frac{ F N R \times p \times cost_{01}+FP R \times(1-p) \times \cos t_{10}}{p \times cost_{01} +(1-p) \times cost_{10}}

FNR is the false negative rate, and FPR is the false positive rate. FNR=1-TPR

Comparative Tests

Performance comparison in machine learning usually relies on statistical hypothesis tests, for the following reasons:

We want to compare generalization performance, but what experiments evaluate is performance on the testing set, and the two comparison results do not necessarily agree
Performance on the testing set depends heavily on the choice of testing set
The algorithm itself involves a certain degree of randomness

Hypothesis Testing

The error rate ${\epsilon}$ serves as the performance measure. For a learner with error rate ${\epsilon}$ , suppose we obtain a testing error of $\hat{\epsilon}$ on the testing set; this means that $\hat{\epsilon} \times m$ of the $m$ samples are misclassified. It is easy to derive that the probability of obtaining testing error $\hat{\epsilon}$ on a test set of $m$ samples, for a learner with generalization error rate ${\epsilon}$ , is:

P(\hat{\epsilon} ; \epsilon)=\left(\begin{array}{c}m \\ \hat{\epsilon} \times m\end{array}\right) \epsilon^{\hat{\epsilon} \times m}(1-\epsilon)^{m-\hat{\epsilon} \times m}

A binomial distribution is used for the hypothesis test. The hypothesis takes the form $\epsilon \leq \epsilon_{0}$ , and the maximum error rate we can observe with probability $1-\alpha$ is:

\bar{\epsilon}=\max \epsilon \quad \text { s.t. } \sum_{i=\epsilon_{0} \times m+1}^{m}\left(\begin{array}{c}m \\ i\end{array}\right) \epsilon^{i}(1-\epsilon)^{m-i}<\alpha

If the testing error $\hat{\epsilon}$ is smaller than the critical value $\bar{\epsilon}$ , we accept the hypothesis at the $1-\alpha$ confidence level; otherwise, at the significance level $\alpha$ , we conclude that the generalization error rate is greater than $\epsilon$ .

If the hold-out method or cross-validation is applied multiple times, k testing error rates are obtained. The mean and variance can then be computed, and a t-test can be used for the hypothesis test. The hypothesis in this case is that the mean equals the error rate. The variable:

\tau_{t}=\frac{\sqrt{k}\left(\mu-\epsilon_{0}\right)}{\epsilon}

follows a t-distribution with k-1 degrees of freedom.

More test methods are described on p.41 of the Watermelon Book

Bias and Variance

Besides estimating generalization performance, we also want to understand why the model performs the way it does, which calls for bias-variance decomposition. Taking a regression task as an example, y_D is the label of x on the dataset, y is the true label of x, and f(x,D) is the predicted output.

The expected prediction is then: $\bar{f}(x)=\mathbb{E}_{D}[f(x ; D)]$

Training sets with different numbers of samples give rise to variance. Then perform the decomposition:

\begin{aligned} E(f ; D) &=\mathbb{E}_{-} D\left[\left(f(x ; D)-y_{D}\right)^{2}\right] \\ &=\mathbb{E}_{-} D\left[(f(x ; D)-\bar{f}(x))^{2}\right]+(\bar{f}(x)-y)^{2}+\mathbb{E}_{-} D\left[\left(y_{D}-y\right)^{2}\right] \\ &=\operatorname{bias}^{2}(x)+\operatorname{var}(x)+\varepsilon^{2} \end{aligned}

The generalization error can be decomposed into the sum of bias, variance, and noise

Bias measures the deviation of the algorithm’s expected prediction from the true result, characterizing the fitting capability of the learning algorithm itself
Variance measures the change in learning performance caused by changes in training sets of the same size, characterizing the impact of data perturbation
Noise expresses the lower bound of the expected generalization error that any learning algorithm can achieve on the current task, characterizing the difficulty of the learning problem itself.

A Survey on Personalization Capabilities in Federated Learning

Wed, 29 Apr 2020 12:37:55 +0000

Abstract

The goal of federated learning is to obtain a shared global model that all nodes can use. However, due to Non-IID data distributions, local models trained solely on local data often outperform the global model, which makes these nodes reluctant to participate in the federation process. This post introduces techniques currently used to personalize the global model so as to improve its performance on individual nodes.

Introduction

Federated learning is a distributed machine learning algorithm designed for Non-IID data. It enables multiple nodes to collaboratively train a shared global model without sharing their own data. At the beginning of each training round, the central server sends the current global model to every node, and each node trains the model on its local data. The central server then collects the model updates from all nodes and applies them to the global model.

The main reason for a node to participate in federated learning is to obtain a better model. When a node does not have enough data to train a good local model on its own, it can obtain a decent model through federated learning. But for nodes with abundant data, the benefit of federated learning remains in question. Yu, in Salvaging Federated Learning by Local Adaptation, noted that for many tasks some nodes gain no benefit because the globally shared model underperforms a model trained on their local data. Hanzely, in Federated Learning of a Mixture of Global and Local Models, questioned whether the global model deviates too far from the classical use cases of users.

The purpose of this post is to survey recent techniques for building personalized models on local nodes, where the personalized models are expected to outperform either the shared global model or the model trained locally by the node.

Need for Personalization

Wu, in Personalized federated learning for intelligent iot applications, described the three major challenges federated learning systems face with respect to personalization:

Heterogeneity of devices in terms of storage, computation, and communication capabilities.
Data heterogeneity caused by Non-IID distributions.
Model heterogeneity arising from the need for different models due to different user environments.

To address the challenges brought by data heterogeneity and Non-IID data, personalizing the global model has become increasingly important. Most personalization techniques consist of two steps: 1. learn a global model collaboratively across the nodes; 2. personalize the global model using each node’s private data. Jiang argued that optimizing solely for global-model accuracy hardly produces models that personalize well, and proposed that, for personalization in federated learning to be useful in practice, the following three objectives must be solved simultaneously rather than independently:

Build improved personalized models that benefit the majority of nodes
Build a more accurate global model, which enables better personalization for nodes with little data
Achieve fast model convergence within few training rounds

In addition, only a fraction of the samples in a node’s local data may be relevant to the specific task, while the remaining samples are irrelevant ones that hurt model training. Tuor proposed a method that uses a relevance model built on a small benchmark to distinguish relevant from irrelevant samples, and then uses only the relevant samples during federated learning.

Techniques

Adding User Context

Before introducing methods for personalizing the global model for each node, it must be pointed out that a shared global model can also make highly personalized predictions, provided that the node’s context and personal information are properly featurized and integrated into the dataset. However, most public datasets do not contain contextual features, and developing techniques that can effectively incorporate context remains an important open problem—one with great potential to improve federated learning models. At the same time, whether such contextual featurization can be done without adversely affecting privacy remains to be studied. As an intermediate approach between a single global model and purely local models, Masour, in Three approaches for personalization with applications to federated learning, suggested clustering users—grouping similar clients together—and training a separate model for each group.

Transfer Learning

Transfer learning allows deep learning models to leverage the capabilities acquired while solving problem A to solve another related problem. Schneider and Vlachos, in Mass personalization of deep learning, discussed using transfer learning to achieve model personalization in non-federated settings. Transfer learning has likewise been applied in federated settings. Wang, in Federated evalua- tion of on-device personalization, proposed re-learning some or all parameters of the trained global model on local data; the previously cited work provides the theoretical justification. By initializing local training with the parameters of the trained global model, transfer learning exploits the knowledge extracted by the global model instead of learning from scratch. Meanwhile, to avoid catastrophic forgetting, care must be taken not to retrain/finetune on local data for too long. One variant of this technique freezes the base layers of the global model and trains only the top layers on local data (similar to freezing the feature layers and finetuning the fully connected layers).

Multi-task Learning

In multi-task learning, a model exploits the commonalities and differences across tasks through joint learning, so that multiple related tasks can be solved simultaneously. Smith, in Federated multi-task learning, showed that multi-task learning is a natural choice for constructing personalized federated models and developed the MOCHA algorithm. A drawback of using multi-task learning in federated settings, however, is that, because it produces one model per task, all nodes must participate in every round of training.

Meta Learning

Meta-learning involves training across multiple learning tasks to produce models that adapt quickly—models that can fit and solve a new task with only a few training samples. Finn proposed the Model-Agnostic Meta-Learning algorithm (MAML), which is compatible with any model trained by gradient descent. MAML builds internal representations that work across multiple tasks, so for a new task, fine-tuning the top layers can produce good results.

Jiang pointed out that the federated learning process can be viewed as meta-training, while the personalization process can be viewed as meta-testing; the FedAvg algorithm is then very similar to Reptile. The author also observed that careful fine-tuning can yield accurate global models that are easy to personalize, whereas optimizing purely for global-model accuracy sacrifices the model’s subsequent personalization capability. Other personalization approaches in federated learning treat the construction of the global model and personalization as two separate processes; Jiang proposed an improved FedAvg algorithm that achieves both a better global model and better personalized models.

Fallah, in Personalized federated learning: A meta-learning approach, formulated a new version of the standard federated learning problem that incorporates MAML, seeking a global model that performs well on every node after being updated with that node’s own loss function; they further proposed Per-FedAvg to solve this problem. Khodak, in Adaptive gradient-based meta-learning methods, proposed ARUBA and demonstrated improved performance by applying it to FedAvg. Chen, in Federated meta-learning for recommendation, proposed a federated meta-learning framework for building personalized recommendation models, in which both the algorithm and the model are parameterized and need to be optimized.

Knowledge Distillation

Caruana showed that the knowledge of an ensemble of models can be compressed into a single model that is easy to deploy. Knowledge distillation develops this idea further, extracting the knowledge of a large teacher network into a smaller student network by having the student imitate the teacher. Overfitting poses a major obstacle during personalization, especially for nodes with small local datasets. Yu proposed that treating the global model as the teacher and the node’s personalized model as the student can alleviate overfitting. Li proposed FedMD, a federated learning framework based on knowledge distillation and transfer learning that allows nodes to independently design their own networks while leveraging both their private data and a global public dataset.

Base+Personalization Layers

In the conventional federated learning setting, data is distributed across the multiple devices participating in training. To mitigate the adverse effects of such data heterogeneity, Arivazhagan et al., in Federated learning with personalization layers, proposed the FedPer architecture, in which the base layers are trained with FedAvg while the top layers (the personalization layers) are trained on the local node with a variant of gradient descent. The transfer-learning approach first trains all layers on global data and then retrains some or all layers on the local node’s data, whereas FedPer trains the base layers on global data and the personalization layers on local data, separately.

Mixtrue of Global and Local Models

Hanzely, in Federated learning of a mixture of global and local models, proposed a new problem formulation that seeks a balance between the global model and local models. Instead of learning a single global model, each node learns a mixture of the global model and its own local model. To solve this problem, the authors proposed LLGD, Loopless Local Gradient Descent (LLGD), a variant of gradient descent. Unlike full averaging, this method only takes averaged steps, because full averaging can be too aggressive.

Discussion

In federated learning, when a node’s local dataset is small and IID (isn’t this similar to the meta-learning scenario?), the global model usually surpasses local models, and most nodes benefit from the federation process. However, when nodes possess sufficiently large private datasets with Non-IID distributions, local models usually outperform the global model, and nodes are generally reluctant to participate in federated learning. An open theoretical question is: how to determine when the global model will outperform the model on a single node

This post has mainly summarized several techniques for optimizing the personalization of the global model. With a few exceptions, most prior work has focused on measuring the performance of global models on aggregated data rather than on individual nodes. But if the global model is to be personalized before use, aggregate global performance becomes meaningless.

Personalized models usually outperform both the global model and purely local models on individual nodes. In some cases, however, a personalized model cannot reach the same capability as a purely local model, especially under differential privacy and similar conditions.

A Survey on Personalization Capabilities in Federated Learning

Wed, 29 Apr 2020 12:37:55 +0800

A Survey on Personalization in Federated Learning

Abstract

Introduction

Need for Personalization

Wu, in Personalized federated learning for intelligent iot applications, described the three major challenges federated learning systems face with respect to personalization:

Heterogeneity of devices in terms of storage, computation, and communication capabilities.
Data heterogeneity caused by Non-IID distributions.
Model heterogeneity arising from the need for different models due to different user environments.

Build improved personalized models that benefit the majority of nodes
Build a more accurate global model, which enables better personalization for nodes with little data
Achieve fast model convergence within few training rounds

Techniques

Adding User Context

Transfer Learning

Multi-task Learning

Meta Learning

Knowledge Distillation

Base+Personalization Layers

Mixtrue of Global and Local Models

Discussion

Decoupled Neural Interfaces using Synthetic Gradients

Tue, 18 Feb 2020 11:04:59 +0800

Decoupled Neural Interfaces using Synthetic Gradients

Abstract

Training neural networks typically requires pushing the computation graph forward and then backpropagating errors to update the weights. In this sense, all layers of the network are locked, because they must wait for other parts of the network to complete their forward inference and backward pass before they can be updated. In this work, we introduce models of future computation in the network graph, in order to decouple modules and thereby break this constraint. These models use only local information to predict the outcome of a subgraph. In particular, we focus on modeling the error gradients: by replacing the true backpropagated error gradients with modeled synthetic gradients, subgraphs are decoupled and can be updated independently and asynchronously—namely, we realize decoupled neural interfaces. We present results for feed-forward models in which every layer is trained asynchronously; for RNNs, where predicting the future gradients of one layer extends the effective modeling horizon of the RNN; and for hierarchical RNN systems with different timescales. Finally, we show that, beyond predicting gradients, the same framework can also be used to predict inputs, resulting in models that are decoupled in both the forward and backward passes—equivalent to two independent networks that can learn together so as to be composed into a single functional network.

Introduction

Each layer (or module) in a directed neural network can be viewed as a computation step that transforms its input. These modules are connected by directed edges, forming a feed-forward graph that defines the flow of data from the network’s input, through each module, to the network’s output. A loss is defined on the output to produce errors, which are backpropagated through the network graph to update each module’s weights.

This mechanism leads to the following forms of locking:

Forward Locking: no module can process its input before the earlier nodes of the forward computation graph have been executed.
Update Locking: no module can be updated until all relevant modules have finished executing in the forward graph.
Backwards Locking: no module can be updated until all relevant modules have executed both the feed-forward and the backward model—for example, the BP algorithm.

Forward, update, and backwards locking force neural networks to run and update sequentially and synchronously. For simple networks this seems benign, but it becomes a serious problem for large, complex, irregular networks, or network systems running in multiple environments with asynchronous timescales.

For example, consider distributed models in which part of the model is shared and used by many downstream clients. This means that all clients must finish executing and pass their error gradients back to the shared model before it can be updated—which means the training speed of the system is dictated by the slowest client. If training of the network could be parallelized, computation time could be greatly reduced.

The goal of this work is to remove update locking from neural networks. This can be achieved by removing backpropagation. To update the weights $\theta_{i}$ of module i, we approximate the backpropagation function as closely as possible:

\begin{aligned} \frac{\partial L}{\partial \theta_{i}} &=f_{\text {Bprop }}\left(\left(h_{i}, x_{i}, y_{i}, \theta_{i}\right), \ldots\right) \frac{\partial h_{i}}{\partial \theta_{i}} \\ & \simeq \hat{f}_{\text {Bprop }}\left(h_{i}\right) \frac{\partial h_{i}}{\partial \theta_{i}} \end{aligned}

where h denotes the layer’s activations, x the input, y the supervision (labels), and L the total loss. The update now depends entirely on h—that is, on information local to module i.

The premise of this method is to allow neural network modules to interact and to be trained without update locking. Here the authors replace the conventional neural interface (the connection between two modules in a network) with a Decoupled Neural Interface (DNI). In short, when one network layer passes activations to another layer, there is an associated model that produces a predicted error gradient for those activations. The predicted gradient is a function of the activation alone and does not depend on other events, states, or losses. The sending layer can then use this synthetic gradient to update immediately. By removing update and backwards locking, we can train networks without synchronization. We also present preliminary results extending this idea to remove forward locking as well, so that the modules of the network can likewise be trained without a synchronized forward pass. When applied to RNNs, we show that using synthetic gradients allows an RNN to model a time horizon far beyond the limit of BPTT. We further show that using synthetic gradients to decouple two RNNs operating at different timescales can greatly improve training.

Decoupled Neural Interfaces

We first describe the high-level communication protocol used to allow asynchronous learning agents to communicate.

As shown in Figure 1, Sender A sends a message (activation) $h_A$ to Receiver B. Intuitively, A can be thought of as the earlier layer and B as the later layer. B has a utility $M_B$ that processes the signal $h_A$ to predict the feedback. The error signal: $\hat{\delta}_{A}=M_{B}\left(h_{A}, s_{B}, c\right)$ , where $h_A$ is the message (activation), $s_B$ is B’s state, and c is some other potentially relevant information, such as labels or context. A can update immediately using this error signal. B can also, in time, fully evaluate the true $\delta_A$ ; thus B’s utility model can be updated to fit the true utility, reducing the discrepancy between the true and synthetic errors.

This protocol allows A to send messages to B in a way that decouples the updates of A and B—A does not have to wait for B to evaluate the true utility before updating—and A can still learn to send messages to B.

We can apply this protocol to the communication process within networks, giving rise to so-called Decoupled Neural Interfaces (DNI). For neural networks, the feedback error signal $\hat{\delta}_A$ can take different forms: for example, gradients can serve as error signals used together with backpropagation; target information can serve as error signals used together with target propagation; or they can even be used as values incorporated into a reinforcement learning framework. This paper focuses on differentiable networks trained via backpropagation with gradient-based updates. We therefore concentrate on the produced error gradient as the feedback $\hat{\delta}_A$ , called the synthetic gradient.

Notation. We define the function at step i as fi, and the composition of the functions from step i to step j as Fij. The loss at layer i is defined as Li.

Synthetic Gradient for Feed-Forward Networks

Consider the DNI formulation for feed-forward networks: N layers fi, each taking input hi-1 and producing output hi = fi(hi-1), where h0 = x is the input data. The feed-forward graph of the whole network can be written as F1N. See Figure 3(a)

Define the loss function on the network output as L = LN. Each layer fi has parameters thetai, updated by a gradient update rule to minimize L(hN)

\theta_{i} \leftarrow \theta_{i}-\alpha \delta_{i} \frac{\partial h_{i}}{\partial \theta_{i}} ; \quad \delta_{i}=\frac{\partial L}{\partial h_{i}}

α is the learning rate, and the derivative with respect to hi is obtained via backpropagation. The dependence on $\delta_i$ means that layer i’s update must wait until the rest of the network has been updated—that is, all of F(i+1)N has executed its feed-forward and feedback phases. Layer i is thus update-locked to F(i+1)N.

To remove update locking, we adopt the communication protocol described earlier. Layer i sends the message hi to the following layer, which has a communication model Mi+1 that produces the synthetic error gradient $\hat{\delta}_{i} = M_{i+1}(h_{i})$ . As shown in the figure:

Layer i, as well as the other layers within F1i, can be updated immediately

\theta_{n} \leftarrow \theta_{n}-\alpha \hat{\delta}_{i} \frac{\partial h_{i}}{\partial \theta_{n}}, n \in\{1, \ldots, i\}

To train the parameters of the synthetic gradient model Mi+1, we wait until the true error gradient $\delta_i$ has been computed, and then compute the MSE between the two.

Moreover, for feed-forward networks, we can use synthetic gradients as the communication feedback to decouple every layer in the network. As shown in the figure:

The complete execution of this mechanism:

In this case, the target error gradient $\delta_i$ is produced by backpropagating $\hat{\delta}_{i+1}$ through layer i+1. Thus $\delta_i$ is not the true error gradient but an estimate obtained from the synthetic gradient models further downstream. Surprisingly, this does not make the errors worse—learning remains stable even across many layers, as the experiments below show.

In addition, supervision or context c can be incorporated when computing the synthetic gradient.

\hat{\delta}_{i} = {M_{i+1}\left(h_{i}, c\right) }

This procedure allows a layer to be updated immediately after its forward pass has been executed. This paves the way for training sub-parts—or individual layers—of a network in an asynchronous manner.

Experiment

Feed-Forward Networks

We apply DNI to feed-forward networks to allow asynchronous or sporadic training of individual layers, as might arise in distributed training.

As explained earlier, introducing synthetic gradients decouples the network layers, allowing them to communicate with one another without being affected by update locking.

Asynchronous Updates To demonstrate the improvements brought by the DNI-decoupled layers, we ran experiments on a four-layer fully connected network for MNIST, in which each layer’s backward pass and update occur in random order with probability pupdate. (That is, a layer is updated only for a fraction pupdate of the forward passes.) This completely breaks backpropagation—for example, the first layer would only receive error-gradient updates with probability pupdate³—and even then, the system would still be constrained to be synchronous. With DNI, however, the communication gap between layers is bridged, and the randomness of layer updates does not affect downstream layers, because synthetic gradients are used. We sampled different values of pupdate uniformly between 0 and 1. DNI with and without labels is shown in Figure 7.

With pupdate = 0.2, the network can still be trained to 2% error. Incredibly, when DNI is conditioned on the data labels (a reasonable assumption if training is performed in a distributed manner), the network trains perfectly with an update probability of just 5%, albeit more slowly.

Complete Unlock

By also eliminating forward locking, feed-forward networks are made fully asynchronous. In this case, every network layer has a synthetic gradient model as well as a synthetic input model used to predict the input data, as shown in the figure below:

Each network layer can then be trained independently, relying on the synthetic gradient and the synthetic input model. The figure below shows the experimental results.

The experiments show that in this setting the model can be trained in a fully asynchronous and independent manner, although it takes somewhat longer to reach the 2% error rate.

Discussion & Conclusion

This paper introduced DNI, a method that uses synthetic gradients to decouple the communication between network layers, allowing them to update independently. It also demonstrated that the method can fully separate all layers of the network, so that they can train in a completely asynchronous, unordered, and sporadic manner.

It is worth noting that, although this paper presented empirical evidence for the effectiveness of DNI and synthetic gradients, the work of Czarnecki et al. investigated the theoretical understanding in greater depth and established convergence.

Stagewise Knowledge Distillation

Mon, 17 Feb 2020 13:15:34 +0800

Stagewise Knowledge Distillation

Abstract

Most modern deep learning models require high computational power, but embedded devices lack such computing capability. Therefore, for this kind of device, models that can reduce computation while maintaining performance are very important. Knowledge distillation is one of the methods that solves this kind of problem. Traditional knowledge distillation methods directly transfer knowledge from the teacher to the student in a single stage. We propose a stagewise training approach to improve knowledge transfer. This approach can even use only a portion of the data used to train the teacher model without affecting performance. This approach can complement other model compression techniques, and can even be regarded as a universal model compression technique.

Introduction

This paper is mainly about model compression techniques based on knowledge distillation, so let me first introduce the categories of model compression.

Model compression can mainly be divided into the following five categories:

Parameter Pruning and Sharing: mainly aims to reduce redundancy in network parameters and eliminate unnecessary parameters.
Low Rank Factorization techniques: mainly use tensor/matrix factorization to determine the effective parameters of the network.
Transferred/Compact Convolutional Filters: aim to use specially designed convolutional filters to reduce computation and storage space.
Knowledge Distillation: aims to use a larger pretrained model, the teacher, to train a small model, the student.
Quantization: aims to reduce the number of bits of each weight while preserving network performance.

This work focuses mainly on the knowledge distillation approach. Ideally, the teacher should be able to pass on all the knowledge it has learned to the student, but this is not the case. Moreover, not all of the teacher’s knowledge is necessarily relevant to the student; the ideal situation is that the student learns the important parts and ignores the unimportant ones. This paper mainly uses ResNet34 as the teacher model, and the student model also uses a ResNet-like model, but is much smaller in storage structure and computational complexity. This paper presents a method of training the student using multiple feature maps of a pretrained teacher model.

This paper mainly adopts a new way of training: the student model is trained using the teacher model’s feature maps fixed at a particular layer. For each feature map, the student model is trained in a stagewise manner, and the final classification layer is trained directly on the dataset without the teacher. It is finally demonstrated that this approach can learn directly on a subset of the teacher’s training data.

Methodology

This paper mainly uses ResNet networks; the specific structure of ResNet will not be repeated here. It mainly consists of:

Basic Block
Downsample Block
ResNet18 or 34 type models

Teacher Network

This paper uses ResNet34 as the teacher model.

Student Network

A reduced version of ResNet34 is used as the student model, mainly by reducing the number of basic blocks.

Dataset

Three datasets are used: Imagenette, Imagewoof and CIFAR10. The first two datasets are subsets of ImageNet. The first is relatively easy, and the latter is relatively harder. The purpose of this work is not to improve accuracy as much as possible, but to make the student’s accuracy as close as possible to the teacher’s.

Proposed Training Method

In the early stage of the experiments, we trained the student model’s multiple feature maps to simultaneously mimic the corresponding feature maps of the teacher model and the labels. So the mean squared error of each pair of feature maps would be accumulated. In addition, the cross-entropy loss is also accumulated. Therefore the total loss function can be expressed as:

\begin{aligned} L\left(y, \hat{y}, y_{c l s}, \mathrm{class}\right) &=\frac{1}{M N} \sum_{i=1}^{N} \sum_{j=1}^{M}(y(i, j)-\hat{y}(i, j))^{2} \\ &+\frac{1}{M C} \sum_{j=1}^{M} \sum_{k=1}^{C}\left\{-\log \left(\frac{\exp \left(y_{c l s}(j, \text { class })\right)}{\sum_{k=1}^{C} \exp \left(y_{c l s}(j, k)\right)}\right)\right\} \end{aligned}

N denotes the number of blocks, and y(i,j) is the intermediate output of the teacher model’s i-th block for the j-th input. Likewise, y-hat is that of the student model. M is the batch size. y_cls(j,k) is the model’s output for the j-th input with respect to the k-th class, C is the number of classes, and class is the correct class that each particular input represents.

These early experiments showed a very small improvement of student models trained with the teacher model over those trained without. This can be attributed to the fact that multiple feature maps and labels must be mimicked at the same time, i.e., the conditions imposed on the optimization algorithm are very strict. Assigning weights to each MSE loss and the cross-entropy loss did not help either, because the training process was still very strict. Another possible reason is gradient vanishing and accumulation. To reduce this strictness of training, a stagewise training approach was proposed.

We train the student model in a stagewise manner, i.e., one block at a time. The image is the input to both the teacher and the student model, and the outputs of the first block are taken from both models. The MSE error is applied between the outputs, and then backpropagation is performed on the student model. After training the first block for 100 epochs, training stops. In the next step, the input is again passed to the teacher and the student, but the features of the second block are taken, and the same procedure as in the first stage is followed, i.e., the MSE loss between the outputs of the second block. Then backpropagation for 100 epochs. This operation is repeated for all the blocks. At the end of the student model, the classifier part is trained directly to predict classes from the dataset, i.e., the images are passed through the student model and trained with the cross-entropy loss for class prediction. At this stage, no teacher model is used, and the rest of the student model (the parts of the model other than the classifier part) is frozen. Figure 2 can help understand this point. The training loss function of stage i can be expressed as:

L_{i}(y, \hat{y})=\frac{1}{M} \sum_{j=1}^{M}(y(i, j)-\hat{y}(i, j))^{2}

The classifier uses the standard cross-entropy loss:

L_{c l s}\left(y_{c l s}, \mathrm{class}\right)=\frac{1}{M C} \sum_{j=1}^{M} \sum_{k=1}^{C}\left\{-\log \left(\frac{\exp \left(y_{c l s}(j, \mathrm{class}\right)}{\sum_{k=1}^{C} \exp \left(y_{c l s}(j, k)\right)}\right)\right\}

We have shown that stagewise training has its own advantages; the main advantage is that the number of parameters to be optimized at a time is limited. Compared with training a larger number of parameters at once, this limited number of parameters can relax the strictness during training. The results show that stagewise training works better than training everything at once.

Less Data Approach

Datasets like ImageNet are so large that performing stagewise training of the student model with the teacher model on limited hardware would take a great deal of time. Therefore, it becomes useful if we can perform stagewise training using only a subset of the data while maintaining accuracy. Thus, the stagewise training experiments were repeated using 1/4 of the original training data. Note that the original training data refers to the data on which the teacher model was trained, and the remaining 3/4 of the data is kept as a test set for evaluation.

Results

Figures 3 and 4 give the results of simultaneous training, stagewise training, and stagewise training with partial data.

These figures show that the student model with the entire dataset achieves almost the same accuracy as the teacher. But in the case of less data, there is a huge gap in accuracy between training with and without a teacher. The following paragraphs discuss the possible reasons behind these results. It should be pointed out that the purpose of model compression is to reduce the gap between teacher and student, not to obtain better accuracy. Obviously, if a better teacher is used, the student’s accuracy will improve; sometimes the student’s accuracy even exceeds that of the teacher model.

The experimental results can be explained as follows. Since the teacher has already learned the complete dataset, it has already learned the features necessary for classifying the entire dataset. When this teacher is used to train the student, even if training uses a small dataset, its “knowledge” is passed on to the student. Using less data can also be justified by the number of parameters that must be trained in a single stage. Since the proposed method trains only a small part of the network at a time, the number of parameters to be optimized is much smaller than that of the full network mentioned in the preceding paragraphs. The results show that this approach greatly improves accuracy: without a teacher, the accuracy of the student network trained on a small dataset is much lower than that of the student trained on the same dataset with the proposed method. Of course, the main advantage is the reduction in training time, which is very important, because without the teacher, stagewise training would take N+1 times as long. (Because each stage trains the same number of epochs separately.) Here N denotes the number of stages.

Figure 4 shows the experimental results using a smaller amount of data. It can be seen that when using less data and training independently, the student performs very poorly. On the other hand, if a teacher trained on less data is used to train the student stagewise, the prediction accuracy improves greatly. This capability would be very useful when applied to very large datasets.

The results of simultaneous training are close to those of training without a teacher. In particular, for the two ImageNet subsets, simultaneous training is slightly better, but for CIFAR10 it is slightly worse. This again demonstrates that the conditions of simultaneous training are too strict, and there is no obvious advantage over training without a teacher. However, the results of stagewise training are much better than those of both simultaneous training and training without a teacher. Since simultaneous training on the complete dataset did not give optimistic results, it was not performed on smaller datasets.

Conclusion

This work presents a novel method for transferring knowledge from one network to another. Since the number of parameters optimized in one stage is reduced, the method performs better than transferring knowledge directly with the full network all at once. This also allows the student network to be trained with less data than the teacher. It will be very useful when training on larger datasets such as ImageNet.

In addition, the method is very flexible and can be used together with other model compression techniques and with other models. At the same time, it is not limited to image classification and can also be used in applications such as object detection and image segmentation. It can be regarded as a universal compression technique.

Federated Meta-Learning with Fast Convergence and Efficient Communication

Wed, 05 Feb 2020 18:36:17 +0800

Federated Meta-Learning with Fast Convergence and Efficient Communication

Abstract

This paper proposes a federated meta-learning framework, FedMeta, which shares a parameterized algorithm (meta learner) instead of the previous global model. It is evaluated on the LEAF datasets and a real-world dataset, and demonstrates that the communication cost required by FedMeta is reduced by 2.82-4.33 times, that it converges faster, and that accuracy is even improved by 3.23~14.84 percentage points compared with FedAvg. Moreover, FedMeta preserves user privacy, since only the parameterized algorithm is shared and no data.

Introduction

The well-known FedAvg algorithm in federated learning can flexibly use SGD for local training, achieving high accuracy while balancing computation and communication costs.

In meta-learning, initialization-based meta-learning algorithms such as MAML excel at fast convergence on new tasks and possess good generalization. This makes them well suited to decentralized data that is Non-IID and highly personalized.

Inspired by this, the paper develops a federated meta-learning framework that differs greatly from previous federated learning work. It first connects meta-learning methods with federated learning. In meta-learning, a parameterized algorithm (meta learner) is slowly learned from a large number of tasks through the meta-training process, during which the algorithm quickly trains a specific model within each task.

A task usually consists of a support set and a query set, which are disjoint. The task-specific model is trained on the support set and tested on the query set. The test results are then used to update the algorithm. In federated meta-learning, the algorithm (meta learner) is maintained on the server and distributed to the clients for model training. During each episode of meta-training, a sampled batch of clients receives the parameters of the algorithm (meta learner) and trains their models. The test results on the query sets are then uploaded to the server to update the meta learner. The overall flow is shown in the figure:

Comparing federated meta-learning with federated learning A comparison of FML and FL. Federated meta-learning is similar to federated learning; the main difference is that what is shared is no longer a global model but the parameters of an algorithm (the meta-model parameters). Moreover, meta-learning is conceptually different from distributed model training, and sharing a meta-learning algorithm can be applied more flexibly than sharing a model. For example, in image classification, images of n classes may be unevenly distributed across clients, with each client having at most k classes. Federated learning would need to train a large n-class classifier in order to exploit data from all clients, whereas in fact a k-class classifier suffices, since it only ever makes predictions for a single client. Such a large model in federated learning entails substantial communication cost. One could send only part of the model to a client to update the relevant parameters, but this requires prior knowledge of the client’s private data. Meta-learning, on the other hand, allows an algorithm to train on tasks containing different classes. For example, MAML can provide an initialization for a k-class classifier by meta-training on k-class tasks, regardless of what the specific classes are. Therefore, within the FML framework, MAML can be used to meta-train initializations of k-class classifiers over all n classes—that is, letting a k-class classifier be meta-learned for initialization in the context of tasks drawn from n classes. This reduces both the communication and computation costs of FML.

Contributions The contributions of this paper focus on algorithm design for the federated learning setting; to this end, the paper proposes a new framework and carries out extensive experiments. The contributions are threefold: 1. Proposing the FedMeta framework, which combines meta-learning and federation, integrating MAML and Meta-SGD into federated learning. 2. Conducting experiments on the LEAF datasets with comparisons against FedAvg; the results show that FedMeta reduces overhead while achieving higher accuracy. 3. Applying FedMeta to a recommendation task in which each client has highly personalized records; the experiments demonstrate that meta-learning algorithms achieve higher accuracy than standalone or federated learning methods.

Federated Meta-Learning

The Meta-Learning Approach

The goal of meta-learning is to meta-train an algorithm (meta learner) A that can train a model quickly—that is, to obtain an initial model that converges fast. The algorithm $A_{\varphi}$ is a parameterized model whose parameters are updated across a series of tasks during meta-training. During meta-training, a task T consists of a support set and a query set, each with corresponding labels and data. The algorithm (meta learner) A trains the model f on the support set and outputs the parameters $\theta_T$ —this is the inner update. The model $f_{\theta_T}$ is then evaluated on the query set, and the test loss $\mathcal{L}_{D_{Q}^{T}}\left(\theta_{T}\right)$ is computed to reflect how well algorithm A trains. Finally, A is updated by minimizing the test loss—this step is called the outer update. Note that the query set and the support set are disjoint, so as to maximize the generalization ability of A. During meta-training, each episode samples a batch of tasks from a meta-training set. The optimization objective of algorithm A can thus be expressed as:

\min _{\varphi} \mathbb{E}_{T \sim \mathcal{T}}\left[\mathcal{L}_{D_{Q}^{T}}\left(\theta_{T}\right)\right]=\min _{\varphi} \mathbb{E}_{T \sim \mathcal{T}}\left[\mathcal{L}_{D_{Q}^{T}}\left(\mathcal{A}_{\varphi}\left(D_{S}^{T}\right)\right)\right]

MAML is a representative meta-learning algorithm. For MAML, algorithm A is what produces the model’s initial state. Concretely, for each task T, the algorithm sets $\alpha = \theta$ , making the algorithm’s parameters equal to the parameters of model f. The parameters of model f are then trained on the support set and updated according to the loss: $\mathcal{L}_{D_{S}^{T}}(\theta):=\frac{1}{\left|D_{S}^{T}\right|} \sum_{(x, y) \in D_{S}^{T}} \ell\left(f_{\theta}(x), y\right)$ . Finally, the model parameters are tested on the query set, and the test loss is computed: $\mathcal{L}_{D_{Q}^{T}}\left(\theta_{T}\right):=\frac{1}{\left|D_{Q}^{T}\right|} \sum_{\left(x^{\prime}, y^{\prime}\right) \in D_{Q}^{T}} \ell\left(f_{\theta_{T}}\left(x^{\prime}\right), y^{\prime}\right)$

Equation 1 can be simplified as:

\min _{\theta} \mathbb{E}_{T \sim \mathcal{T}}\left[\mathcal{L}_{D_{Q}^{T}}\left(\theta-\alpha \nabla \mathcal{L}_{D_{S}^{T}}(\theta)\right)\right]

Building on MAML, Meta-SGD further learns the initial parameters and the inner-loop learning rate at the same time. The test loss can be viewed as a function of both the model parameters and the learning rate, and both can be updated by taking gradients with SGD in the outer loop. Moreover, the learning rate is a vector of the same dimension as the model parameters, so that the learning rate corresponds to the model parameter vector coordinate-wise.

Therefore the optimization condition can be rewritten as

\min _{\theta, \alpha} \mathbb{E}_{T \sim \mathcal{T}}\left[\mathcal{L}_{D_{Q}^{T}}\left(\theta-\alpha \circ \nabla \mathcal{L}_{D_{S}^{T}}(\theta)\right)\right]

The Federated Meta-Learning Framework

The goal of FML is to meta-train an algorithm collaboratively from data distributed across clients. Taking MAML as an example, the aim is to use the data of all clients to complete the initialization of a model. MAML involves two levels of optimization: the inner loop trains a task-specific model from the initial parameters, and the outer loop updates the initial parameters according to the test loss. In the federated learning setting, each client u receives the initial parameters $\theta$ from the server, then trains the model on the support-set data on its device and transmits the test loss to the server. The server mainly maintains the initial parameters and updates them according to the clients’ test losses.

The information transmitted in this process includes: the model’s initial parameters (from server to clients) and the test loss (from clients to server). For the Meta-SGD algorithm, the vector $\alpha$ is also part of the transmission, serving as the learning-rate parameters for inner-loop model training.

Algorithm 1 describes the FedMeta procedure using MAML and Meta-SGD, where each communication round corresponds to an episode. The algorithm is maintained in the AlgorithmUpdate step. In each round of updates, the server collects the test losses obtained by training MAML or Meta-SGD on the sampled clients. The initial parameters $\theta$ are updated after training on the clients’ training sets, and the updated parameters are then tested on the test set. After the meta-training process, the model is deployed on the clients.

Experiments

Experiments on the LEAF datasets demonstrate fast convergence and high accuracy.
Experiments on a recommendation task in a real-world setting show that strong adaptability is maintained even at smaller scales.

Evaluation Scheme

In all experiments, 80% of the clients serve as training clients, 10% as validation clients, and the remainder as test clients. Each client’s local data is split into a support set and a query set. The paper also varies the fraction p of each client’s data used as the support set, in order to study how effectively FedMeta adapts to new clients with limited data; this is abbreviated psupport hereafter.

The authors ran experiments with FedAvg, a meta version of FedAvg, and two variants of FedMeta. The meta version of FedAvg uses the support sets of the test clients to finetune the initial model received from the server before testing, whereas during training both use all of the data on the training clients.

As for FML, three methods were tested—MAML, FOMAML, and Meta-SGD—all of which are model-agnostic and easy to implement.

LEAF Datasets

Accuracy and Convergence Comparison Considering the limited computational capabilities of edge devices, each client’s local epoch is set to 1.

As shown above, all methods within the FedMeta framework converge faster and more stably, achieving clear improvements. MAML and Meta-SGD reach similar convergence speeds and accuracy on the first two datasets; on Sent140, Meta-SGD performs somewhat better.

The table shows the accuracy of the four methods after several communication rounds. First, it can be seen that FedAvg performs far worse than FedMeta, especially on the image classification task. MAML and Meta-SGD achieve the highest accuracy, with gains of 3.23-14.84 percentage points. It is also found that FedAvg (meta) attains higher accuracy than FedAvg in most cases; there are two exceptions, however, when the support fraction is 20%. Unexpectedly, FedAvg (Meta) even shows a slight drop in accuracy. This may be because, after being finetuned on a small amount of data, the model deviates excessively from the global optimum. Second, as we increase the probability p of the support set, the accuracy of both FedAvg (Meta) and FedMeta improves in almost all cases, but the growth rate of FedAvg (meta) is greater than that of FedMeta. This indicates that FedMeta generalizes better and adapts more effectively to clients with limited data.

Fairness Comparison The paper also compares FedAvg and FedMeta across multiple runs, using the distribution of final accuracies. The last row of the previous figure shows kernel density estimates for the different methods. For FEMNIST, MAML and Meta-SGD not only yield a higher mean but also lower variance. For the Shakespeare dataset, FedMeta has larger variance but also a larger mean. For Sent140, the accuracy distributions are roughly the same; still, it can be seen that MAML and Meta-SGD have more clients close to 100%. Overall, for image classification tasks, FedMeta’s accuracy distribution is more stable and concentrated.

Conclusion

The FedMeta framework outperforms the original FedAvg in accuracy, convergence speed, and communication cost.

CS330 Lecture 1&2 Study Notes (Incomplete)

Fri, 10 Jan 2020 11:31:05 +0800

CS330 lecture 1&2 notes

Informal Problem Definitions

The multi-task learning problem: Learn all of the tasks more quickly or more proficiently than learning them independently.
The meta-learning problem: Given data/experience on previous tasks, learn a new task more quickly and/or more proficiently.

Multi-Task Learning Basics

Traditional single-task learning:

\begin{array}{l}{\mathscr{D}=\left\{(\mathbf{x}, \mathbf{y})_{k}\right\}} \\ {\min _{\theta} \mathscr{L}(\theta, \mathscr{D})}\end{array}

Typical loss: negative log likelihood

\mathscr{L}(\theta, \mathscr{D})=-\mathbb{E}_{(x, y) \sim \mathscr{D}}\left[\log f_{\theta}(\mathbf{y} | \mathbf{x})\right]

What’s a task?

A task: $\mathscr{T}_{i} \triangleq\left\{p_{i}(\mathbf{x}), p_{i}(\mathbf{y} | \mathbf{x}), \mathscr{L}_{i}\right\}$

data generating distributions

Here a task is defined as the distribution over data samples, the distribution over data labels, and a loss function.

Corresponding datasets: $\mathscr{D}_{i}^{tr}$ training set, $\mathscr{D}_{i}^{t s t}$ test set.

Usually $\mathscr{D}_{i}$ denotes the training set.

Multi-task classification: $\mathscr{L}_{i}$ same across all tasks. E.g., in handwritten character recognition across different languages, the form of the loss function may be the same.

Multi-label learning: $\mathscr{L}_{i}, {p}_{i}(x)$ same across all tasks. E.g., in the CelebA multi-label recognition task, the samples and the loss function are identical.

The loss function may vary across tasks in the following cases:

mixed discrete, continuous labels across tasks
caring more about one task than another (i.e., different weights for different tasks?)

Conditioning on the task

The multi-task learning problem requires introducing a task descriptor as a variable that describes the task; the question is how to design this variable.

Assume ${z}_{i}$ is the task index. The most straightforward approach is multiplicative gating, which effectively trains each task in the multi-task setting with its own separate network, without sharing parameters.

The other extreme is to directly concat $z_i$ , in which case all parameters are shared except those that come after the input $z_i$ .

Yet another idea is to split $\theta$ into shared parameters $\theta^{sh}$ and task-specific parameters $\theta^i$ — i.e., shared and non-shared parameters.

The optimization objective then becomes

\min _{\theta^{s h}, \theta^{1}, \ldots, \theta^{T}} \sum_{i=1}^{T} \mathscr{L}_{i}\left(\left\{\theta^{s h}, \theta^{i}\right\}, \mathscr{D}_{i}\right)

The problem then becomes which parameters to share and when.

Common Choices

The common choices are mainly concatenation and addition — the figures make them clear at a glance.

Concatenation-based conditioning

Additive conditioning

In fact, the two are equivalent.

Multi-head architecture

Multiplicative conditioning

The multiplicative approach offers:

stronger expressive power
multiplication gating for regression tasks
better generalization across independent networks and heads

Complex Choices

There are also many other more complex choices.

But where the design inspiration comes from is just like choosing the hyperparameters of a neural network:

different problems are independent of one another
for any specific problem, it mostly relies on the designer’s intuition and background knowledge
current approaches are more art than science

Optimizing the objective

Objective: $\min _{\theta} \sum_{i=1}^{T} \mathscr{L}_{i}\left(\theta, \mathscr{D}_{i}\right)$

The typical procedure:

Sample a minibatch of tasks $\mathscr{B} \sim\left\{\mathscr{T}_{i}\right\}$
Sample a minibatch of data from each task $\mathscr{D}_{i}^{b} \sim \mathscr{D}_{i}$
Compute the loss on each minibatch-task: $\hat{\mathscr{L}}(\theta, \mathscr{B})=\sum_{\mathcal{T}_{k} \in \mathscr{B}} \mathscr{L}_{k}\left(\theta, \mathscr{D}_{k}^{b}\right)$
Backpropagate to compute gradients $\nabla_{\theta} \hat{\mathscr{L}}$
Update the gradients with your favorite optimizer

Note: this ensures that tasks are sampled uniformly regardless of their data size.

Tip: for regression tasks, make sure task labels are on the same scale.

Challenge

Negative transfer

Multi-task training on CIFAR-100 performs worse than training tasks independently.

Possible causes:

optimization challenges
- interference between different tasks
- different learning rates across tasks
limited expressive capacity
- multi-task networks are large

Solution:

share less across tasks (soft parameter sharing)

allows for more fluid degrees of parameter sharing (advantage)
yet another set of design decisions/hyperparameters (drawback)

\min _{\theta^{sh}, \theta^{1}, \ldots, \theta^{T}} \sum_{i=1}^{T} \mathscr{L}_{i}\left(\left\{\theta^{s h}, \theta^{i}\right\}, \mathscr{D}_{i}\right)+\sum_{t=1}^{T}\left\|\theta^{t}-\theta^{t'}\right\|

The latter term is soft parameter sharing: the difference between one task’s parameters and the previous one is used as a regularization term, which effectively makes each task’s parameters as similar as possible — i.e., the parameters are shared.

Overfitting

Overfitting is usually caused by not sharing enough parameters; the solution is to share more. Intuitively, insufficient sharing makes each task overfit, which resembles independent training.

Meta-Learning Basics

Two views of meta-learning:

Mechanistic view
- a deep neural network that can take in an entire dataset and make predictions on new data
- the network is trained on a meta-dataset that contains different datasets for different tasks
- this view makes it easy to implement a meta-learning algorithm
Probabilistic view
- extract prior knowledge from a series of meta-learning tasks
- use a small amount of data plus prior information to infer a relatively effective posterior
- this view leads to a better understanding of meta-learning algorithms

Problem definitions

First, recall supervised learning:

Existing issues:

requires a large amount of labeled data
labels are very limited for some tasks nowadays

To be continued

Paper Notes: A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms [Unfinished]

Thu, 09 Jan 2020 16:02:59 +0800

A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

Abstract

This paper proposes exploiting a meta-learning objective that maximizes the speed of transfer to a changed distribution in order to learn how to acquire knowledge in a modular fashion. In particular, we care about how to factor the joint distribution into appropriate conditionals consistent with the causal relationships. If this is achievable, changes in the distribution are assumed to be localized (distributions are localized) (e.g., one of the marginals changes due to an intervention on one of the variables). We show that, under such localized changes of the assumed causal mechanisms, the correct causal graph will tend to have only a few parameters with non-zero gradients, i.e., the parameters that need to be adjusted (the parameters of the modified variables). This is observed experimentally to lead to faster adaptation, and we exploit this property to define a meta-learning surrogate score which, in addition to a continuous parametrization of the graph, will also favor the correct causal graph. Finally, considering AI agents (e.g., a robot autonomously discovering its environment), we consider how the same objective can discover the causal variables themselves, since the observed low-level variables carry no causal meaning. Experiments on the bivariate case validate the proposed ideas and theoretical results.

Introduction

Data is assumed to be independent and identically distributed (IID). Likewise, a model’s performance is typically evaluated with test samples drawn from the same distribution, on the assumption that they represent how the learned system will be used. Although these assumptions are well analyzed from a statistical point of view, they are hard to apply in many practical situations. For example: a medical diagnosis system trained on historical data from one hospital may perform poorly on patients from another hospital because the distribution has shifted. Ideally, we would like our models to generalize well and to adapt quickly to out-of-distribution data.

However, successfully transferring to a new distribution calls for more data. In this paper, rather than focusing on assumptions about the data distribution itself, we focus on how the data distribution changes (e.g., the change from the training distribution to the transfer distribution may be caused by the actions of some agents). The assumption we care about is that when knowledge is represented in an appropriately modular way, only one or a few modules change — such changes are sparse. This is especially important when the distribution change is caused by the action of one or more agents, because an agent intervenes at a particular place and time, which takes the form of the interventions discussed in the causal literature, where one of the causal variables is clamped to a particular value or to a random variable. In general, it is hard for an agent to affect more than one of the underlying causal variables at once. Although this paper is not about agent learning per se, this is a property we explore to help discover these variables and the causal relationships between them. In this respect, the causal graph is a powerful tool, because it tells us how a perturbation of the distribution of an intervened variable will propagate to all the other variables and affect their distributions.

Usually, the causal structure is not known in advance. The problem of causal discovery is generally needed to obtain the causal graph, but this is usually achievable only under strong assumptions. One such assumption is that a learner that has captured the correct structure of the true underlying data-generating process should generalize to perturbations of that structure in a specific way. This can be illustrated by considering temperature and altitude: roughly speaking, a learner trained on data from Switzerland should still be valid when tested on data drawn from the distribution of a less mountainous country such as the Netherlands. It is thus proposed to use the out-of-distribution robustness of the predictive model to guide inference about the true causal structure.

So how can we exploit the assumption of local changes? As we explain theoretically and validate experimentally here, if we have the correct representation of knowledge, then starting from a model pre-trained on the training distribution we should be able to adapt very quickly to the transferred data distribution. This happens because we assume the data-generating process is acquired as a set of independent mechanisms, and moving from the training distribution to the transfer distribution requires changing very few of the mechanisms and parameters. A model corresponding to the correct decomposition of knowledge therefore needs only a few updates and examples to adapt to the transfer distribution. As shown below, the expected gradient on the invariant parameters will be close to 0 (if the model has already been well trained on the training distribution), so the effective search space is greatly reduced during adaptation to the transfer distribution, which speeds up adaptation, as the experiments show. Therefore, based on the assumption of small changes in the space of correct knowledge representations, we can define a meta-learning objective that measures the speed of adaptation — a form of regret — to optimize how knowledge is represented, decomposed, and structured

Back to the earlier temperature and altitude example: if we receive out-of-distribution data from the Netherlands, we expect the model to adapt faster because a small number of transfer samples from the Netherlands have been collected. Similarly to robustness, adaptation speed can be used to guide inference about the true causal structure of the problem at hand, possibly together with other sources of signal about the causal structure.

Main contributions: We first validate on synthetic data that, after certain interventions are performed on the true bivariate causal graph (unknown to the learner), a model that correctly captures the underlying causal structure adapts faster when samples are provided. This shows that adaptation speed can serve as a score that adequately assesses how well a learner fits the underlying causal graph. We then use a smooth parametrization of the causal graph to directly optimize this score in an end-to-end manner. Finally, we show that, when the mixing of variables is unknown, this score can be used to distinguish the correct causal variables.

Which is Cause and Which is Effect?

To be continued

Improving Federated Learning Personalization via MAML Paper Notes (Incomplete)

Mon, 02 Dec 2019 15:07:30 +0800

Improving Federated Learning Personalization via MAML

Abstract

FL algorithms share many similarities with MAML and can be interpreted through meta-learning algorithms
Fine-tuning gives the global model stronger accuracy while making personalization easier
Models trained on standard centralized datasets are harder to personalize than those trained with FedAvg

Introduction

Points out the connection between FL and MAML, and interprets the FL algorithm via MAML
Improves FedAvg with a two-stage approach of training and fine-tuning
Finds that FedAvg is essentially a meta-learning algorithm that optimizes personalization performance rather than the global model.

Interpreting FedAvg as a Meta Learning Algorithm

The figure below shows MAML applied in the FL setting (left), the Reptile algorithm (middle), and FedAvg, the training algorithm of FL (right). Let L be the loss function. In each round of iteration, MAML trains by randomly sampling a batch of tasks T. For each task T there is an inner loop, and the outer loop aggregates the gradient updates obtained from each task. The FL algorithm randomly samples several clients T. For each T and its weight, several rounds of local optimization are performed, and the updated gradients are aggregated into a new global model. If we simplify the setting and assume all clients have the same amount of data, all weights become equal, and Reptile and FedAvg are then effectively the same algorithm.

Assume the weights in FedAvg are equal, denoted wi. Consider T clients, and let the model parameters be $\theta$ . For each client i, the loss function is $L_{i}(\theta)$ ; let $g_{j}^{i}$ denote the gradient computed during the $j^{t h}$ local training step.

The gradient update of FedSGD is:

g_{F e d S G D}=\frac{-\beta}{T} \sum_{i=1}^{T} \frac{\partial L_{i}(\theta)}{\partial \theta}=\frac{1}{T} \sum_{i=1}^{T} g_{1}^{i}

Now let us express FOMAML in the same terms. Assume the client learning rate is $\beta$ ; after K steps, the personalized model of client i obtains the updated parameters $\theta_{K}^{i}=U_{K}^{i}(\theta)=\theta-\beta \sum_{j=1}^{K} g_{j}^{i}=\theta-\beta \sum_{j=1}^{K} \frac{\partial L_{i}\left(\theta_{j}\right)}{\partial \theta}$

Differentiating gives:

\frac{\partial U_{K}^{i}(\theta)}{\partial \theta}=I-\beta \frac{\partial \sum_{j=1}^{K} g_{j}^{i}}{\partial \theta}=I-\beta \sum_{j=1}^{K} \frac{\partial^{2} L_{i}\left(\theta_{j}\right)}{\partial \theta^{2}}

After K gradient updates, the whole model is updated:

g_{M A M L}=\frac{\partial L_{M A M L}}{\partial \theta}=\frac{1}{T} \sum_{i=1}^{T} \frac{\partial L_{i}\left(U_{K}^{i}(\theta)\right)}{\partial \theta}=\frac{1}{T} \sum_{i=1}^{T} L_{i}^{\prime}\left(U_{K}^{i}(\theta)\right)\left(I-\beta \sum_{j=1}^{K} \frac{\partial^{2} L_{i}\left(\theta_{j}\right)}{\partial \theta^{2}}\right)

To avoid the computational cost of second-order derivatives, FOMAML was introduced: after K gradient updates, it directly uses the (K+1)-th gradient as the local update.

g_{F O M A M L}(K)=\frac{1}{T} \sum_{i=1}^{T} L_{i}^{\prime}\left(U_{K}^{i}(\theta)\right) I=\frac{1}{T} \sum_{i=1}^{T} L_{i}^{\prime}\left(\theta_{K}^{i}\right)=\frac{1}{T} \sum_{i=1}^{T} g_{K+1}^{i}

From the formulas above, it is easy to see that the FedAvg update — the average of all clients’ updates — is essentially a linear combination of the two ideas above.

g_{F e d A v g}=\frac{1}{T} \sum_{i=1}^{T} \sum_{j=1}^{K} g_{j}^{i}=\frac{1}{T} \sum_{i=1}^{T} g_{1}^{i}+\sum_{j=1}^{K-1} \frac{1}{T} \sum_{i=1}^{T} g_{j+1}^{i}=g_{F e d S G D}+\sum_{j=1}^{K-1} g_{F O M A M L}(j)

Personalized FedAvg

As shown in the figure above, FedAvg E in Algorithm 1 trains for E local epochs, weighting the gradient updates by the local data volume. Then, in the FL setting, Reptile (K) trains for K local steps, ignoring the local data volume.

Generally speaking, in terms of the number of communication rounds, FedAvg with several local epochs converges quickly within just a few rounds of communication. Due to the complexity of production environments, this metric is used to measure the convergence speed of FL algorithms. This paper finds that using momentum SGD as the server optimizer already benefits the personalized model, whereas the initial model remains relatively unstable. Previous approaches reduced the number of local training epochs or the learning rate.

This paper proposes using Reptile (K) for fine-tuning and then Adam as the server optimizer, which improves the initial model while also stabilizing the personalized model.

To be continued

Federated Adversarial Domain Adaptation Paper Notes (Incomplete)

Sun, 24 Nov 2019 14:04:02 +0800

Federated Adversarial Domain Adaptation

Federated learning currently has the problem of knowledge domain shift, which prevents it from having good generalization ability. Domain shift occurs when the labeled data of the source nodes and the unlabeled data of the target node are different.

This paper mainly proposes a way to address the use of knowledge domains in federated learning, so that the knowledge learned by different nodes can be aligned with the data distribution of the target node.

The paper’s method mainly applies adversarial adaptation techniques to federated learning. In addition, a dynamic attention mechanism is designed, and feature disentanglement is used to enhance knowledge transfer.

Introduction

Traditional federated learning has a problem: since the data on each node is collected in a non-IID manner, a domain shift problem arises. For example, the images taken by one device are mostly indoor scenes, while those taken by another are mostly outdoor scenes. The method proposed in this paper is: transferring the knowledge in the data of decentralized nodes to a node of a new and different data domain, without requiring additional user supervision. This method is also known as Unsupervised Federated Domain Adaptation.

There are currently many unsupervised domain adaptation methods, but the federated learning setting gives rise to the following problems:

Data is stored locally and cannot be shared.
Model parameters are trained independently on different nodes and converge at different rates, and the contribution to the global model depends on the correlation between the two domains.
Since the knowledge learned from the source nodes is highly integrated and difficult to disentangle, it may cause negative transfer.

The method proposed in this paper is named Federated Adversarial Domain Adaptation (FADA), which uses adversarial techniques to solve the domain shift problem in federated learning systems.

The main steps of the method are:

First, the federated domain adaptation problem is analyzed from a theoretical perspective, and a general, universal bound is provided.
Inspired by the theoretical results, an effective adaptation algorithm is proposed, based on adversarial adaptation and representation disentanglement techniques applied to the federated learning setting.
A dynamic attention model is designed to cope with the constantly changing convergence rates in federated learning systems.

Unsupervised Domain Adaptation: The goal of UDA is to transfer knowledge from a labeled data domain to an unlabeled data domain.

Federated Learning: I will not go into detail here.

Feature Disentanglement: The features extracted by neural networks through complex hidden layers are usually highly integrated. Therefore, learning disentangled features can help remove irrelevant or domain-specific knowledge, and then model the needed domain knowledge.

Generalization Bound for FDA

First, recall the definition of the theoretical error bound for the adaptation problem in the single-data-source setting.
Then describe the definition of the error bound in the unsupervised federated domain adaptation setting.

Federated Adversarial Domain Adaptation

From the theory in the previous section, it is not difficult to see the importance of the weight $\alpha$ and the distance. The method of this paper learns the weights by proposing a dynamic attention mechanism, and reduces the distance between the source domains and the target domain through a federated adversarial alignment mechanism. In addition, a feature disentanglement method is adopted to extract domain-invariant features to strengthen knowledge transfer.

Dynamic Attention Mechanism:

The principle of this mechanism is mainly to increase the weights of nodes with high contributions and decrease the weights of nodes with low contributions. So how do we judge the degree of a node’s contribution? This paper uses the gap statistics method to evaluate how well the target features can be clustered.

I=\sum_{r=1}^{k} \frac{1}{2 n_{r}} \sum_{i, j \in C_{r}}\left\|f_{i}^{t}-f_{j}^{t}\right\|_{2}

Assume there are $C_{1}, C_{2}, \ldots, C_{k}$ clusters, where $C_r$ denotes the indices of the objects in the r-th cluster, and $n_{r}=\left|C_{r}\right|$ .

Intuitively, a smaller gap statistics value indicates that the feature distribution has smaller intra-class variance. The degree of contribution of each source node is measured by computing the difference between the gap statistics of two iterations.

I_{i}^{g a i n}=I_{i}^{p-1}-I_{i}^{p}

It represents how much the clusters can be improved after the target domain is updated with a source domain’s gradient. For the gradient updates, the following is used to decide how much each gradient contributes:

\text { Softmax }\left(I_{1}^{\text {gain }}, I_{2}^{\text {gain }}, \ldots, I_{N}^{\text {gain}}\right)

Federated Adversarial Alignment:

Machine learning models can suffer greatly degraded performance due to domain distance. To solve this problem, this paper improves the traditional adversarial training method in the federated learning setting. In the federated learning context, this paper divides the adversarial alignment optimization process into two steps: 1. For each domain, train a local feature extractor, with $G_i$ and $G_t$ corresponding to $D_i$ and $D_t$ respectively. 2. For each ( $D_i, D_t$ ) source-target domain pair, train an adversarial domain discriminator DI to align the two distributions in an adversarial learning way. First, train DI to recognize which feature domain a feature comes from; then train the generator, i.e., the feature extractors ( $G_i, G_t$ ), to fool DI. Note: D can only access the outputs of $G_i$ and $G_t$ .

Given the i-th source data domain $\mathbf{X}^{S_{i}}$ and the target domain $\mathbf{X}^T$ , the optimization objective of $D I_{i}$ is:

\underset{\Theta^{D I_{i}}{L}}{L_{a d v_{D I}}\left(\mathbf{X}^{S_{i}}, \mathbf{X}^{T}, G_{i}, G_{t}\right)=-\mathbb{E}_{\mathbf{x}^{s_{i} \sim \mathbf{X}^{s_{i}}}}\left[\log D I_{i}\left(G_{i}\left(\mathbf{x}^{s_{i}}\right)\right)\right]-\mathbb{E}_{\mathbf{x}^{t} \sim \mathbf{x}^{t}}\left[\log \left(1-D I_{i}\left(G_{t}\left(\mathbf{x}^{t}\right)\right)\right)\right]}

Then, keeping D fixed, the objective function for updating G is:

\underset{\Theta^{G}{G}_{i}, \Theta^{G_{t}}}{L}\left(\mathbf{X}^{S_{i}}, \mathbf{X}^{T}, D I_{i}\right)=-\mathbb{E}_{\mathbf{x}^{s_{i} \sim \mathbf{X}^{s_{i}}}\left[\log D I_{i}\left(G_{i}\left(\mathbf{x}^{s_{i}}\right)\right)\right]-\mathbb{E}_{\mathbf{x}^{t} \sim \mathbf{X}^{t}}\left[\log D I_{i}\left(G_{t}\left(\mathbf{x}^{t}\right)\right)\right]}

Feature Disentanglement:

Under the adversarial learning framework, this paper also adopts adversarial disentanglement to extract domain-invariant features. The paper argues that the extracted features can be divided into domain-specific features and domain-invariant features. As shown in the first figure, the feature disentangler D divides the features into two categories. First, a K-class classifier and a K-class category recognizer are trained to predict labels from the features.

\begin{array}{c}{L_{cross entropy}} \\ {\Theta^{G_i}, \Theta^{D_i}, \Theta^{C_i}, \Theta^{CI_i}}\end{array}=-\mathbb{E}_{\left(\mathbf{x}^{s_i}, \mathbf{y}^{s_i}\right) \sim \widehat{\mathcal{D}}_{s_i}} \sum_{k=1}^{K} \mathbb{1}\left[k=\mathbf{y}^{s_{i}}\right] \log \left(C_{i}\left(f_{d i}\right)\right)-\mathbb{E}_{\left(\mathbf{x}^{s} i, \mathbf{y}^{s} i\right) \sim \widehat{\mathcal{D}}_{s_{i}}} \sum_{k=1}^{K} \mathbb{1}\left[k=\mathbf{y}^{s_{i}}\right] \log \left(C I_{i}\left(f_{d s}\right)\right)

Then, with the category classifier fixed, only the feature disentangler is trained, producing domain-specific features to fool the category classifier.

\underset{\Theta^{D} i, \Theta^{G_{i}}}{L_{e n t}}=-\frac{1}{N_{s_{i}}} \sum_{j=1}^{N_{s_{i}}} \log C I_{i}\left(f_{d s}^{j}\right)=-\frac{1}{N_{s_{i}}} \sum_{j=1}^{N_{s_{i}}} \log C I_{i}\left(D_{i}\left(G_{i}\left(\mathbf{x}^{s_{i}}\right)\right)\right)

The feature disentanglement method transfers knowledge by keeping the class-invariant features and removing the class-specific features. To enhance the disentanglement effect, this paper minimizes the mutual information between the two.

Mutual information is defined as $I\left(f_{d i} ; f_{d s}\right)=\int_{\mathcal{P} \times \mathcal{Q}} \log \frac{d \mathbb{P}_{\mathcal{P} Q}}{d \mathbb{P}_{\mathcal{P}} \otimes \mathbb{P}_{\mathcal{Q}}} d \mathbb{P}_{\mathcal{P} \mathcal{Q}}$ , where $\mathbb{P}_{\mathcal{P} \mathcal{Q}}$ is the joint probability distribution of the two kinds of features, and $\mathbb{P}_{\mathcal{P}}=\int_{\mathcal{P}} d \mathbb{P}_{\mathcal{P} \mathcal{Q}}, \mathbb{P}_{\mathcal{Q}}=\int_{\mathcal{Q}} d \mathbb{P}_{\mathcal{P} \mathcal{Q}}$ are the corresponding marginal probability distributions.

The Mutual Information Neural Estimator (MINE) is used to estimate the mutual information:

T_{\theta}: \widehat{I(\mathcal{P} ; \mathcal{Q})}_{n}=\sup _{\theta \in \Theta} \mathbb{E}_{\mathbb{P}_{\mathcal{P} Q}^{(n)}}\left[T_{\theta}\right]-\log \left(\mathbb{E}_{\mathbb{P}_{P}^{(n)} \otimes \mathbb{P}_{Q}^{(n)}}\left[e^{T_{\theta}}\right]\right)

In practice, it can be computed by:

I(\mathcal{P} ; \mathcal{Q})=\iint \mathbb{P}_{\mathcal{P} \mathcal{Q}}^{n}(p, q) T(p, q, \theta)-\log \left(\iint \mathbb{P}_{\mathcal{P}}^{n}(p) \mathbb{P}_{\mathcal{Q}}^{n}(q) e^{T(p, q, \theta)}\right)

To avoid computing integrals, the Monte Carlo method is adopted:

I(\mathcal{P}, \mathcal{Q})=\frac{1}{n} \sum_{i=1}^{n} T(p, q, \theta)-\log \left(\frac{1}{n} \sum_{i=1}^{n} e^{T\left(p, q^{\prime}, \theta\right)}\right)

where (p, q) are sampled from the joint distribution, $q^{\prime}$ is sampled from the marginal distribution, and $T(p, q, \theta)$ is a neural network determined by $\theta$ that estimates the mutual information between the P and Q distributions.

The domain-invariant features and the domain-specific features are fed into a reconstructor to obtain recombined features; the loss function uses an L2 loss to reconstruct the original features, in order to preserve the composability of the representation. The L2 constraint and the mutual information loss constraint can be balanced by tuning the hyperparameters.

To be continued

MemoryGAN Paper Notes (Incomplete)

Mon, 22 Apr 2019 14:05:02 +0800

Memory GAN Reading Notes

Introduction

This paper mainly solves two problems in training unsupervised GANs.

First, since generative adversarial networks use only a continuous latent distribution to represent multiple classes or clusters of data, they usually cannot correctly handle the structurally discontinuous nature between different classes in the latent space. (Caused by the mode collapse problem.) For example, if a GAN embeds buildings and cats in the same continuous latent distribution, the GAN may generate realistic images in the transition region between the two classes.

Second, the discriminator of a generative adversarial network very easily forgets the samples generated in the past, which causes instability during adversarial training.

The authors believe that these two main problems can be solved by a learnable memory network accessible to both the generator and the discriminator. The generator can effectively learn representations of the training samples to understand the underlying clustering distribution of the data, thereby alleviating the structural discontinuity problem. Meanwhile, the discriminator can better memorize the clusters of previously generated samples, which can alleviate the forgetting problem.

This paper proposes an end-to-end generative adversarial network model, Memory GAN, which involves an unsupervised memory network integrated with existing generative adversarial network models.

Von Mises-Fisher (vMF) mixture model. The memory module can effectively alleviate the instability problem. First, to alleviate the structural discontinuity problem, the memory can learn representations of the training samples, helping the generator better understand the class and cluster distributions. Therefore, we can separate the modeling of discrete clusters from the embedding of data attributes on the continuous latent space, which can alleviate the discontinuity problem.

Second, the memory network can alleviate the forgetting problem by learning to memorize the clusters of previously generated sample data, including very rare samples.

Architecture

The overall architecture is divided into the memory discriminative network (DMN) and the memory conditional generative network (MCGN).

Memory Discriminative Network

The memory discriminative network consists of a feedforward network $\mu$ and a memory network.

The feedforward network $\mu$ is a convolutional neural network, whose input is $x \in \mathbb{R}^{D}$ and whose output is a normalized vector $q=\mu(x) \in \mathbb{R}^{M}$ , where $\|q\|=1$ . This is then used as the input of the memory module, which outputs the discriminative result.

The memory network is formulated as $\mathcal{M}=(K, v, a, h)$ , where $K \in \mathbb{R}^{N \times M}$ is the memory key matrix, $N$ is the memory size and $M$ is the dimension. $v \in\{0,1\}^{N}$ is the memory value matrix. Conceptually, each key vector stores a representation of a clustering center learned through the vMF mixture model, and its corresponding key value is the real/fake judgment. $a \in \mathbb{R}^{N}$ denotes the age of the object stored in each memory block. And $h \in \mathbb{R}^{N}$ denotes the slot histogram, where each $h_{i}$ represents the amount of valid data belonging to the i-th memory slot.

life-long memory network: the memory space can grow freely, and there is no need to reset it during training.

k-nearest neighbor indexing for efficient memory lookup: k-nearest neighbor indexing is used for lookup.

least recently used (LRU) scheme for memory update. The LRU scheme is used for memory update.

Innovations:

Probabilistic derivation: the data likelihood, as well as the prior and posterior distributions of the categorical distribution over memory indices, can be computed.
The memory learns an approximate distribution of queries by maximizing the likelihood with an incremental EM algorithm.
The memory is optimized through the GAN loss rather than through a memory loss.
The contribution of each sample is decided by tracking the slot histogram.

Discriminator Output

For each input $x$ , we first need to decide which memory slot to use to compute the discriminative probability.

$c \in\{1,2, \ldots, N\}$ denotes the memory slot index.

A Von Mises-Fisher (vMF) mixture model is used to represent the posterior distribution of the memory index.

p(c=i | x)=\frac{p(x | c=i) p(c=i)}{\sum_{j=1}^{N} p(x | c=j) p(c=j)}=\frac{\exp \left(\kappa K_{i}^{T} \mu(x)\right) p(c=i)}{\sum_{j=1}^{N} \exp \left(\kappa K_{j}^{T} \mu(x)\right) p(c=j)}

In $p(x | c=i)=C(\kappa) \exp \left(\kappa K_{i}^{T} \mu(x)\right)$ , $\kappa=1$ is a constant attention parameter.

The vMF is effectively equivalent to a properly normalized Gaussian distribution defined on the unit sphere.

The categorical distribution over the memory index, $p(c)$ , is obtained by normalizing the slot histogram, $p(c=i)=\frac{h_{i}+\beta}{\sum_{j=1}^{N}\left(h_{j}+\beta\right)}$ , where $\beta\left(=10^{-8}\right)$ is a smoothing constant for numerical stability. By using $p(y=1 | c=i, x)=v_{i}$ , we marginalize the joint probability $p(y=1, c | x)$ over c to obtain $p(y=1 | x)$ :

p(y=1 | x)=\sum_{i=1}^{N} p(y=1 | c=i, x) p(c=i | x)=\sum_{i=1}^{N} v_{i} p(c=i | x)=\mathbb{E}_{i \sim p(c | x)}\left[v_{i}\right]

However, for each sample x, this is not scalable for the whole memory of size N.

The maximum a posteriori probability is adopted to consider the top-k slots $S=\left\{s_{1}, \dots, s_{k}\right\}$ :

S=\underset{c_{1}, \ldots, c_{k}}{\operatorname{argmax}} p(c | x)=\underset{c_{1}, \ldots, c_{k}}{\operatorname{argmax}} p(x | c) p(c)=\underset{c_{1}, \ldots, c_{k}}{\operatorname{argmax}} \exp \left(\kappa K_{c}^{T} \mu(x)\right)\left(h_{c}+\beta\right)

where $p(x | c)$ is the vMF likelihood and $p(c)$ is the prior distribution of the memory index.

Here we omit the normalization of the vMF likelihood and the denominator of the prior, because they are both constants. Once we have obtained S, we can compute

p(y | x) \approx \frac{\sum_{i \in S} v_{i} p(x | c=i) p(c=i)}{\sum_{j \in S} p(x | c=j) p(c=j)}

Memory Update Mechanism

The memory keys and values are updated during training. The update mechanism includes a traditional memory update mechanism and an incremental EM algorithm.

Let the sample be $x$ with label $y$ , where real is 1 and fake is 0. For each x, we first find the k-nearest slots $S_{y}$ , but using the conditional posterior $p\left(c | x, v_{c}=y\right)$ . This is so that in the subsequent EM algorithm, only the slots belonging to the same class as y are considered.

After that, we update the memory in different ways depending on whether $S_y$ contains the correct label.

Without the correct label:

The oldest memory slot is found via $n_{a}=\operatorname{argmax}_{i \in\{1, \ldots, N\}} a_{i}$ , and the information of x is copied to the corresponding position: $K_{n_{a}} \leftarrow q=\mu(x)$ , $v_{n_{a}} \leftarrow y$ , $a_{n_{a}} \leftarrow 0$ , $h_{n_{a}} \leftarrow \frac{1}{N} \sum_{i=1}^{N} h_{i}$ .

With the correct label:

The memory keys are updated through the following custom incremental EM algorithm run for T iterations, which partly incorporates the information of the new sample.

In the inference step, by applying the keys of the previous moment $\hat{K}_{i}^{t-1}$ and $\hat{h}_{i}^{t-1}$ to Equation 1, the posterior distribution $\gamma_{i}^{t}=p\left(c_{i} | x\right)$ is computed, where $i \in S_{y}$ . In the maximization step, the following updates are performed:

\hat{h}_{i}^{t} \leftarrow \hat{h}_{i}^{t-1}+\gamma^{t}-\gamma^{t-1}, \quad \hat{K}_{i}^{t} \leftarrow \hat{K}_{i}^{t-1}+\frac{\gamma^{t}-\gamma^{t-1}}{\hat{h}_{i}^{t}}\left(q_{i}-\hat{K}_{i}^{t}\right)

where $t \in 1, \ldots, T, \gamma^{0}=0, \hat{K}_{i}^{1}=K_{i}, \hat{h}_{i}^{1}=\alpha h_{i}$ , and $\alpha=0.5$ .

After T iterations, the slots of $S_y$ are updated through $K_{i} \leftarrow \hat{K}_{i}^{t}$ and $h_{i} \leftarrow \hat{h}_{i}^{t}$ .

The decay rate $\alpha$ controls the degree to which the contribution of old queries to the slot position in the average direction of the mixture component is exponentially reduced.

$\alpha$ is crucial for performance, because the old queries used to update the keys no longer fit the current mixture distribution, as the feedforward network itself is also being updated.

Finally, it is worth noting that this memory update mechanism is orthogonal to the adversarial training algorithm, because when the discriminator is updated, the memory update is carried out independently. In addition, adding the memory module does not affect the speed of the model at test time, because the memory is only updated during training.

Memory Conditional Generative Network

The memory conditional generative network is based on the generator of InfoGAN. The difference is that it is conditioned not only on random noise but also on memory information.

In other words, the generator not only samples randomly from the noise distribution, but also samples the memory index $i$ from $P\left(c=i | v_{c}=1\right)=\frac{h_{i} v_{i}}{\sum_{j}^{N} h_{j} v_{j}}$ . The formula above represents the frequency of occurrence of the unit i that stores real data. The final input is $[K_i, z]$ , where $K_i$ is the key vector of memory index i.

The difference from other CGANs is that MCGN does not require extra annotations or an extra encoder. Instead, MCGN can make full use of the memory information learned by the DMN in an unsupervised way. The DMN learns the vMF mixture memory using only each sample in the sequence and its label.

The training procedure of the whole MemoryGAN is:

for number of training iterations do Sample a minibatch of samples from the training set Sample a minibatch from the noise distribution and the memory indices Update the discriminator loss Find S_y for each sample in the minibatch Initialize the key, h and gamma for each slot in S_y for number of EM iterations do Estimate $\gamma_s$ for each s Update $h_s$ Update $K_s$ Update the vMF mixture model, $h_{s_{y}} \leftarrow \hat{h}_{s_{y}}^{T}, K_{s_{y}} \leftarrow \hat{K}_{s_{y}}^{T}$ for $s_{y} \in S_{y}$ Sample a minibatch from the noise distribution and the memory indices Update the generator loss

objective function

The goal of MemoryGAN is based on the objective of InfoGAN, which is to maximize the mutual information between the latent variables and the observed content. (For details, see InfoGAN.)

A mutual information loss is added between $K_i$ and $G(z, K_i)$ to ensure the continuity between the sampled memory information and the structured information of the generated samples:

I\left(K_{i} ; G\left(z, K_{i}\right)\right) \geq H\left(K_{i}\right)-\hat{I}-\log C(\kappa)

where $\hat{I}$ denotes the expectation of the negative cosine similarity, $\hat{I}=-E_{x \sim G\left(z, K_{i}\right)}\left[\kappa K_{i}^{T} \mu(x)\right]$ .

To be continued

Getting Started with Kettle

Sat, 23 Mar 2019 12:44:49 +0800

Getting Started with Kettle

Since I was still one credit short in my final semester, I had to take one more course, and looking at the schedule the only option was OLAP — yet another completely unfamiliar subject. Tutorials about Kettle online are scarce, so I gathered some materials and jotted down these notes on the tool this course requires

Introduction to Kettle

Kettle is an ETL (Extract, Transform and Load) tool for data extraction, transformation, and loading. ETL tools are used very frequently in data warehouse projects, and Kettle can also be applied in the following scenarios:

Integrating data across different applications or databases
Exporting data from a database to text files
Loading bulk data into a database
Data cleansing

It is also used in projects involving application integration

Kettle is very easy to use: you design what business logic to implement through a graphical interface, with no need to write code; therefore, Kettle is designed in a metadata-oriented way

Kettle supports many input and output formats, including text files, data tables, and both commercial and free database engines. In addition, Kettle’s powerful transformation features make it very convenient to manipulate data.

Installing Kettle

Download link

Since my platform is macOS and there is a bug that prevents it from launching on double-click, I could only extract the archive and then run in the terminal

sh spoon.sh

to start Spoon

Data Transformation - Exporting to Excel

Connect to the database
Export to Excel

First, create a new transformation as shown in the figure below, and set up the database connection.

Once the database connection is set up, you can configure the input source; here we use a table in the database as the input source

With the input source configured, you can click preview to take a look at the data first

After setting up the input source, likewise drag the Excel output onto the workspace, but don’t rush to configure the output source yet: in the View pane, add a hop to connect the input and output sources

This way, when you configure the output source, you can directly fetch and select the fields to export in the Fields tab

Finally, click Start to run the transformation

Let’s take a look at the exported file

Hello World

Alright, now that you have learned how transformations work, let’s do a Hello World example. I came across this tutorial online and found it pretty good — it lets you get in touch with more of Kettle’s features, unlike my very basic walkthrough above.

This Hello World outputs a corresponding greeting for each name in the database and exports the result to a file.

First, set up the input source the same way, but the SQL statement needs a small change: here I selected only their names, ordered by id, limited to the first ten.

Then write a small JavaScript script to automatically add a “hello”

Finally, link the three steps with hops; you can run preview first to check the result

Then take a look at the result

The above only demonstrates the most basic features of data transformation; other, more detailed features can all be found in the Design tab In short, Kettle’s operating logic is steps + hops: a step is an operation you perform, and hops are the links between them.

Jobs

The above covered transformations, one of Kettle’s core features; now let’s introduce another Kettle feature, the job.

A job is essentially an automated workflow: when you need to run multiple transformations or add some logical control conditions, you need a job.

Here, following the earlier Hello World example, we build a simple job that detects when the file does not exist in the folder and then automatically exports data from the database and adds “hello”.

Find the widget shown in the figure above in the toolbar on the right, drag it into the workspace, connect things with hops, and configure it — that’s it. The operations are similar to those for creating a transformation; it just combines several of them together

The figure below shows the transformation configuration interface, where you enter the path of the transformation script

Kitchen and Pan

The kitchen and pan commands are used to execute job and transformation scripts.

./kitch.sh -file ./scripts/demo.kjb
./pan.sh -file ./scripts/demo.ktr

From now on, just save the jobs and transformations designed in Spoon and run them directly from the command line — much more convenient.

I’ll update these notes whenever I get to use new features.

References

The ETL Powerhouse Kettle in Practice, Part 1 [Introduction to Kettle]

KETTLE Tutorial

Using Kettle Jobs

Kettle Tutorials

Kettle Chinese Community

One-shot Learning with Memory-Augmented Neural Networks Paper Notes (Incomplete)

Fri, 22 Mar 2019 08:47:11 +0800

One-shot Learning with Memory-Augmented Neural Networks Paper Notes

Introduction

Traditional gradient-based deep learning methods require large amounts of data to learn. When confronted with new data, the model has to relearn new parameters and cannot quickly adapt to it.

This paper proposes a memory-augmented neural network that can rapidly assimilate new data and, after learning from only a few samples, use that data to make accurate predictions.

For the design of the MANN model, the authors had two requirements:

The stored information must be stable and addressed element-wise.
The number of parameters must not be tied to the size of the memory.

In the end, the model combines the following two advantages:

It can learn a general way of learning by extracting useful information from raw data via gradient descent.
It can rapidly learn never-before-seen information by relying on an additional memory module.

Meta-Learning Method Design

Usually, parameters $\theta$ are chosen on some dataset $D$ to minimize a learning cost $\mathcal{L}$ .

In meta-learning, however, what is minimized is the expected cost with respect to a distribution over datasets $p(D)$ :

\theta^{*}=\operatorname{argmin}_{\theta} E_{D \sim p(D)}[\mathcal{L}(D ; \theta)]

To achieve this, the paper makes the following design choices:

First, the inputs are presented as a sequence, where each input carries the label of the previous input:

\left(\mathbf{x}_{1}, \text { null }\right),\left(\mathbf{x}_{2}, y_{1}\right), \ldots,\left(\mathbf{x}_{T}, y_{T-1}\right)

Across different datasets, the labels are shuffled (to prevent the network from gradually learning a fixed mapping between samples and labels). Instead, this forces the network to learn the trick of holding a data sample in memory until the correct label arrives, so that sample-label information is bound together and stored for later use.

Without relying on the actual content of the data and labels, the meta-learning model learns to bind the data distribution to the corresponding labels, and generalizes into a universal model that captures the data-label relationship for prediction.

Memory-Augmented Model

Neural Turing Machine

The composition of the Neural Turing Machine is largely similar to the MANN in this paper. An NTM consists of a controller, such as an LSTM or a feedforward network, which interacts with an external memory module through a set of read/write heads.

In this model, the controller likewise uses an LSTM or a feedforward network.

To be continued

Notes on Generative Adversarial Networks (GAN)

Fri, 01 Mar 2019 17:00:22 +0800

Generative Adversarial Network

Overview of GAN

The idea behind GAN is a two-player zero-sum game in which the sum of the two players’ interests is a constant. Consider arm wrestling: assuming the total space is fixed, if your strength is greater, you gain more space and I get less; conversely, if I am stronger, I gain more. But one thing is certain — our total space is fixed. This is a two-player game in which the total interest is constant.

A vivid analogy: GAN is like a large network containing two smaller networks. One is the generative network, which can be regarded as a counterfeiter producing fake banknotes, and the other is the discriminative network, i.e., the person authenticating the banknotes. The goal of the generative network is to fool the discriminator, while the discriminator strives not to be fooled by the generator. Through alternating optimization during training, both networks improve. It is theoretically proven that, at best, the generative model makes the discriminator unable to tell real from fake, i.e., the probability of real versus fake is fifty-fifty.

The figure above is a structural diagram of a generative adversarial network: the discriminator takes in both real samples and fake samples produced by the generator, and outputs a real/fake verdict. The generator takes noise as input and produces fake samples.

The Principle of GAN

The following is the objective function of GAN:

\min _ { G } \max _ { D } V ( D , G ) = \mathbb { E } _ { \boldsymbol { x } \sim p _ { \mathrm { data } } ( \boldsymbol { x } ) } [ \log D ( \boldsymbol { x } ) ] + \mathbb { E } _ { \boldsymbol { z } \sim p _ { \boldsymbol { z } } ( z ) } [ \log ( 1 - D ( G ( \boldsymbol { z } ) ) ) ]

From the objective function we can see that the overall cost function minimizes the generator and maximizes the discriminator. When tackling this optimization problem, we can first fix G, maximize D, and then minimize G to obtain the optimal solution. Here, for a given G, maximizing V(D,G) measures the divergence or distance between P_G and P_data.

First, after fixing G, the optimal D can be expressed as:

D _ { G } ^ { * } = \operatorname { argmax } _ { D } V ( G , D )

The problem of optimizing G can then be expressed as:

G ^ { * } = \operatorname { argmin } _ { G } V \left( G , D _ { G } ^ { * } \right)

Theoretical Derivation

The original paper’s derivation uses the JS divergence to describe the similarity between two distributions, and the JS divergence is in turn built from the KL divergence. Therefore, before carrying out the full derivation, we first introduce some theoretical foundations, derive the conditions required for the optimal discriminator and the optimal generator, and finally use the derived results to restate the training procedure.

KL Divergence

For a single random variable $x$ with two separate probability distributions $P(x)$ and $Q(x)$ , the KL divergence can measure the difference between the two distributions (the appendix proves why the KL divergence reflects the difference between two distributions):

D _ { \mathrm { KL } } ( P \| Q ) = \mathbb { E } _ { \mathrm { x } \sim P } \left[ \log \frac { P ( x ) } { Q ( x ) } \right] = \mathbb { E } _ { \mathrm { x } \sim P } [ \log P ( x ) - \log Q ( x ) ]

Properties of the KL divergence:

Non-negativity (used later in the derivation); moreover, the KL divergence equals 0 if and only if P and Q are the same distribution. Because of this non-negativity, it is often used to measure the difference between two distributions. (The appendix proves its non-negativity.)
Asymmetry. Although it can measure the difference between distributions, this difference is not a symmetric distance: the KL divergence of P with respect to Q differs from that of Q with respect to P.

Issues in the Paper’s Derivation

In the original paper, one idea differs from many other approaches: the generator G is not required to satisfy an invertibility condition, and in practice G is indeed non-invertible. However, in the proof, the change-of-variables formula for integrals was incorrectly used, while this change of variables is valid only when G is invertible. So the proof should be based on the validity of the following equality:

E _ { z \sim p _ { x } ( z ) } \log ( 1 - D ( G ( z ) ) ) = E _ { x \sim p _ { c } ( x ) } \log ( 1 - D ( x ) )

This equality comes from the Radon-Nikodym theorem in measure theory; it is presented as Proposition 1 in the original paper and expressed as the following equality:

\begin{array} { c } { \int _ { x } p _ { d a t a } ( x ) \log D ( x ) \mathrm { d } x + \int _ { z } p ( z ) \log ( 1 - D ( G ( z ) ) ) \mathrm { d } z } \\ { = \int _ { x } p _ { d a t a } ( x ) \log D ( x ) + p _ { G } ( x ) \log ( 1 - D ( x ) ) \mathrm { d } x } \end{array}

This formula uses the change-of-variables formula for integrals, but doing so requires computing the inverse of G, whose existence is not assumed. Moreover, in the practice of neural networks it does not exist. However, this practice is so common in ML that it is simply overlooked.

Optimal Discriminator

In the minimax game, we first fix the generator G and maximize the value function, which yields the optimal discriminator D. The maximized value function measures the divergence between the distribution produced by the generator and the distribution of the dataset (proven later).

The expectations in the original paper’s value function can be expanded into integral form:

V ( G , D ) = \int _ { x } p _ { \mathrm { data } } ( \boldsymbol { x } ) \log ( D ( \boldsymbol { x } ) ) + p _ { g } ( \boldsymbol { x } ) \log ( 1 - D ( \boldsymbol { x } ) ) d x

Maximizing the integral can be converted into maximizing the integrand. Maximizing the integrand yields the optimal discriminator D, so all terms not involving the discriminator can be treated as constants. Letting the discriminator D(x) be y, the integrand can be written as:

f ( y ) = a \log y + b \log ( 1 - y )

To find the optimal extreme point, if $a + b \neq 0$ , we can solve using the first-order derivative:

f ^ { \prime } ( y ) = 0 \Rightarrow \frac { a } { y } - \frac { b } { 1 - y } = 0 \Rightarrow y = \frac { a } { a + b }

Taking the second derivative at the stationary point gives:

f ^ { \prime \prime } \left( \frac { a } { a + b } \right) = - \frac { a } { \left( \frac { a } { a + b } \right) ^ { 2 } } - \frac { b } { 1 - \left( \frac { a } { a + b } \right) ^ { 2 } } < 0

where $a , b \in ( 0,1 )$ . Since the first derivative equals 0 and the second derivative is less than 0, $\frac { a } { a + b }$ is a maximum.

Finally, the value function can be written as:

\begin{aligned} V ( G , D ) = & \int _ { x } p _ { d a t a } ( x ) \log D ( x ) + p _ { G } ( x ) \log ( 1 - D ( x ) ) \mathrm { d } x \\ & \leq \int \max _ { y } \max _ { y } p _ { d a t a } ( x ) \log y + p _ { G } ( x ) \log ( 1 - y ) \mathrm { d } x \end{aligned}

Letting D(x)=P_data/(P_data+p_G) attains the maximum, because f(y) has a unique maximum on its domain; that is, the optimal D is unique, and no other D can attain the maximum.

In fact, the optimal D cannot be computed in practice, but it is mathematically important. Moreover, we do not know the prior Pdata, so we cannot use it directly in training. On the other hand, the existence of the optimal D implies that of the optimal G, and it suffices that we approach the optimal D.

Optimal Generator

The training process of GAN aims to make P_G=P_data, in which case the optimal D can be written as:

D _ { G } ^ { * } = \frac { p _ { \text {data} } } { p _ { \text {data} } + p _ { G } } = \frac { 1 } { 2 }

That is, the optimal generator makes the discriminator unable to distinguish P_data from P_G. Based on this observation, the authors proved that G is the solution of the minimax game.

Theorem: The global minimum of C(G)=maxV(G,D) is attained if and only if P_G=P_data.

The theorem states an if-and-only-if result, so we prove it from both directions. First we work backwards to derive the value of C(G), then we prove it forward.

Assume P_G=P_data (deriving backwards from the known result). We can derive:

V \left( G , D _ { G } ^ { * } \right) = \int _ { x } p _ { d a t a } ( x ) \log \frac { 1 } { 2 } + p _ { G } ( x ) \log \left( 1 - \frac { 1 } { 2 } \right) \mathrm { d } x

V \left( G , D _ { G } ^ { * } \right) = - \log 2 \int _ { x } p _ { G } ( x ) \mathrm { d } x - \log 2 \int _ { x } p _ { d a t a } ( x ) \mathrm { d } x = - 2 \log 2 = - \log 4

So -log4 is a candidate for the minimum, since it arises only when P_G=P_data. Now we must prove forward that this value is always the minimum, i.e., that the if-and-only-if condition is fully satisfied.

Now drop the condition P_G=P_data, pick any G, and rewrite the formula as:

C ( G ) = \int _ { x } p _ { d a t a } ( x ) \log \left( \frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) + p _ { G } ( x ) \log \left( \frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) \mathrm { d } x

Next we apply a trick: add a zero to the equation. It does not change the value of the equation, but it lets us construct a log2, since we know -log4 is the candidate global minimum.

\begin{aligned} C ( G ) & = \int _ { x } ( \log 2 - \log 2 ) p _ { d a t a } ( x ) + p _ { d a t a } ( x ) \log \left( \frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) \\ & + ( \log 2 - \log 2 ) p _ { G } ( x ) + p _ { G } ( x ) \log \left( \frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) \mathrm { d } x \end{aligned}

\begin{array} { c } { C ( G ) = - \log 2 \int _ { x } p _ { G } ( x ) + p _ { d a t a } ( x ) d x } \\ { + \int _ { x } p _ { d a t a } ( x ) \left( \log 2 + \log \left( \frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) \right) } \\ { + p _ { G } ( x ) \left( \log 2 + \log \left( \frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \right) \right) \mathrm { d } x } \end{array}

Finally, simplification yields:

\begin{aligned} C ( G ) = & - \log 4 + \int _ { x } p _ { d a t a } ( x ) \log \left( \frac { p _ { d a t a } ( x ) } { \left( p _ { G } ( x ) + p _ { \text {data} } ( x ) \right) / 2 } \right) \mathrm { d } x \\ & + \int _ { x } p _ { G } ( x ) \log \left( \frac { p _ { G } ( x ) } { \left( p _ { G } ( x ) + p _ { d a t a } ( x ) \right) / 2 } \right) \mathrm { d } x \end{aligned}

If you have read the earlier section on KL divergence, you will notice that this can be simplified into the form of KL divergences:

C ( G ) = - \log 4 + K L \left( p _ { d a t a } | \frac { p _ { d a t a } + p _ { G } } { 2 } \right) + K L \left( p _ { G } | \frac { p _ { d a t a } + p _ { G } } { 2 } \right)

Since the KL divergence is non-negative, -log4 is the global minimum.

It remains to show that only one G can attain this value, so that P_G=P_data is the unique solution, completing the proof.

From the earlier discussion we know the KL divergence is asymmetric and can only measure the similarity of distribution a with respect to distribution b. But after adding the second term, their sum becomes symmetric, and this sum of the two terms can be expressed as the JS divergence:

\operatorname { JSD } ( P \| Q ) = \frac { 1 } { 2 } D ( P \| M ) + \frac { 1 } { 2 } D ( Q \| M )

M = \frac { 1 } { 2 } ( P + Q )

Suppose there are two distributions P and Q whose average distribution is M=(P+Q)/2; then the JS divergence between the two distributions is the KL divergence between P and M plus the KL divergence between Q and M, divided by 2.

Therefore, the JS divergence ranges from 0 to log2. It equals log2 when the two distributions have no overlap at all, and reaches the minimum 0 when they are identical.

Hence C(G) can be rewritten as:

C ( G ) = - \log 4 + 2 \cdot J S D \left( p _ { \text { data } } | p _ { G } \right)

This proves that the JSD is 0 when P_G=P_data. In summary, we obtain the optimal generator if and only if the generated distribution equals the true data distribution.

Convergence

Regarding whether the training process converges to the optimal generator, the original paper gives an additional proof that, with sufficient training data and sufficient capacity of d and g, training converges to the optimal G. Since this part is not particularly important, I will put the proof in the appendix (mainly because my own understanding of it is only partial).

Training Procedure

Parameter optimization process

If we want to find the optimal generator, then after fixing a discriminator D, we can treat the original value function as the loss function L(G) for training the generator. With a loss function, we can update the generator using optimization algorithms such as SGD or Adam. The gradient descent update is as follows:

\theta _ { G } \leftarrow \theta _ { G } - \eta \partial L ( G ) / \partial \theta _ { G }

Now, given an initial G_0, we need to find the D_0* that maximizes V(G_0,D); thus the discriminator’s update process amounts to minimizing the loss function -V(G,D). Moreover, from the earlier derivation, V(G,D) differs from the JS divergence between the distributions P_data(x) and P_G(x) only by a constant term. Therefore this alternating adversarial process can be described as:

Given G_0, maximize V(G_0,D) to obtain D_0*, i.e., max[JSD(P_data(x)||P_G0(x))];
Fix D_0*, compute $\mathrm { \theta } _ { - } \mathrm { G } 1 \leftarrow \theta _ { - } \mathrm { G0 } - \mathrm { \eta } \left( \partial \mathrm { V } \left( \mathrm { G } , \mathrm { D } _ { - } \mathrm { 0 } ^ { * } \right) / \partial \theta _ { - } \mathrm { G } \right)$ to obtain the updated G_1;
Fix G_1, maximize V(G_1,D_0*) to obtain D_1*, i.e., max[JSD(P_data(x)||P_G1(x)];
Fix D_1*, compute $\theta _ { - } \mathrm { G } 2 \leftarrow \theta _ { - } \mathrm { G } 1 - \eta \left( \partial \mathrm { V } \left( G , D _ { - } \mathrm { 0 } ^ { * } \right) / \partial \theta _ { - } \mathrm { G } \right)$ to obtain the updated G_2;

And so the loop continues.

Practical training process

According to the definition of the value function V(G,D) above, we need to compute two expectations, E[log(D(x))] and E[log(1-D(G(z)))], where x follows the real data distribution and z follows the initialization distribution. In practice, there is no way to compute these two expectations by integration, so we generally approximate them by sampling from the infinite real data and an infinite generator.

Suppose the generator G is given and we wish to compute maxV(G,D) to find the discriminator D. First we need to sample m samples from P_data(x) and m samples from the generator P_G(x). Maximizing the value function can then be replaced by the following expression:

\text { Maximize } \tilde { V } = \frac { 1 } { m } \sum _ { i = 1 } ^ { m } \log D \left( x ^ { i } \right) + \frac { 1 } { m } \sum _ { i = 1 } ^ { m } \log \left( 1 - D \left( \tilde { x } ^ { i } \right) \right)

Now we treat the samples drawn from P_data(x) as positive samples and those drawn from P_G(x) as negative samples, while using an approximation of the negative V(G,D) as the loss function. This can therefore be formulated as the standard training procedure of a binary classifier:

\text { Minimize } L = - \frac { 1 } { m } \sum _ { i = 1 } ^ { m } \log D \left( x ^ { i } \right) - \frac { 1 } { m } \sum _ { i = 1 } ^ { m } \log \left( 1 - D \left( \tilde { x } ^ { i } \right) \right)

In practice, we must implement the minimax game with iterative and numerical methods. Fully optimizing D in the inner loop of training is computationally prohibitive, and a finite dataset would lead to overfitting. Therefore, we can alternate between k steps of optimizing D and one step of optimizing G. As long as G is updated slowly, D stays near its optimal solution.

In summary, over the whole training procedure, for each iteration:

Draw m samples from the real data distribution P_data
Draw m noise samples from the prior distribution P_prior(z)
Feed the noise samples into the generator G to produce data $\left\{ \tilde { x } ^ { 1 } , \tilde { x } ^ { 2 } , \ldots , \tilde { x } ^ { m } \right\} , \tilde { x } ^ { i } = G \left( z ^ { i } \right)$ , and update the discriminator parameters $\theta _ { - } d$ by maximizing the approximation of V; the discriminator’s update rule is $\theta _ { d } \leftarrow \theta _ { d } + \eta \nabla \tilde { V } \left( \theta _ { d } \right)$

The above is the process of learning the discriminator D. The process computes the JS divergence and is repeated k times, because we want to maximize the value function.

Draw another m noise samples from the prior distribution P_prior(z)
Update the generator by minimizing V, i.e., $\tilde { V } = \frac { 1 } { m } \sum _ { i = 1 } ^ { m } \log \left( 1 - D \left( G \left( z ^ { i } \right) \right) \right)$ , with the generator’s update rule $\theta _ { g } \leftarrow \theta _ { g } - \eta \nabla \tilde { V } \left( \theta _ { g } \right)$

The above is the learning process of the generator’s parameters. This process occurs only once per iteration, which avoids excessive updates that would make the JS divergence rise.

This concludes the complete derivation and argumentation of GAN.

Several Issues in GAN Training

Training Instability

Training the original GAN is very difficult. This mainly manifests as the training process failing to converge, or the trained generator being unable to produce meaningful content, among other issues. On the other hand, although our optimization objective is the JS divergence, which should reflect the distance between the two distributions, ideally this distance should be relatively large at the start and gradually shrink as the training of G progresses.

In practice, the discriminator’s loss function very easily drops to 0 and then stays at 0 thereafter. The JS divergence measures the distance between two distributions, but in practice two situations can cause the JS divergence to deem the distance between the two distributions infinite, making the loss function forever 0.

Case 1: the discriminator D is too strong, causing overfitting.

Solution: try regularization, or reduce the number of model parameters.

Case 2: properties of the data itself. The low-dimensional manifold produced by the generator indeed does not overlap easily with the data distribution.

Solution: one option is to add noise to the data so that the generator’s distribution and the real data distribution overlap more easily. Another is the GAN to be discussed next time.

Mode Collapse

All outputs are the same! This phenomenon is known as Mode Collapse. A likely cause is that the real data has large probability mass in many places in the space, but our generative model has not directly learned the characteristics of the real distribution. To guarantee minimal loss, it would rather always produce the same but certainly correct output than try other, different but possibly wrong outputs. In other words, our generator sometimes cannot cover all internal modes of the data distribution and conservatively picks just one mode that is certainly correct.

Summary

GAN combines a generative model with a discriminative model, eliminating the difficulty of defining a loss function for generative models
It operates on probability distributions and is not constrained by the dimensionality of the generation
It can be used for semi-supervised learning

Posts on eree's blog

[Paper] TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH Reading Notes

TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH

Abstract

Introduction

Related Work

Methods

Feature Fusion Modules

FedFusion

Experiment

Experiment setup

Artificial Non-IID Partition

User-Specific Non-IID Partition

IID Partition

Conclusion

Watermelon Book Reading Notes 1. Introduction

Introduction

Preamble

Basic Terminology

Hypothesis Space

Inductive Bias

Watermelon Book Reading Notes 2. Model Evaluation and Selection

Model Evaluation and Selection

Empirical Error and Overfitting

Evaluation Methods

Hold-Out Method

Cross-Validation

Bootstrap

Parameter Tuning and the Final Model

Performance Measures

Error Rate and Accuracy

Precision, Recall, and F1

ROC and AUC

Cost-Sensitive Error Rate and Cost Curve

Comparative Tests

Hypothesis Testing

Bias and Variance

A Survey on Personalization Capabilities in Federated Learning

Abstract

Introduction

Need for Personalization

Techniques

Adding User Context

Transfer Learning

Multi-task Learning

Meta Learning

Knowledge Distillation

Base+Personalization Layers

Mixtrue of Global and Local Models

Discussion

A Survey on Personalization Capabilities in Federated Learning

A Survey on Personalization in Federated Learning

Abstract

Introduction

Need for Personalization

Techniques

Adding User Context

Transfer Learning

Multi-task Learning

Meta Learning

Knowledge Distillation

Base+Personalization Layers

Mixtrue of Global and Local Models

Discussion

Decoupled Neural Interfaces using Synthetic Gradients

Decoupled Neural Interfaces using Synthetic Gradients

Abstract

Introduction

Decoupled Neural Interfaces

Synthetic Gradient for Feed-Forward Networks

Experiment

Feed-Forward Networks

Complete Unlock

Discussion & Conclusion

Stagewise Knowledge Distillation

Stagewise Knowledge Distillation

Abstract

Introduction

Related Work

Methodology