[{"content":"TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH Abstract This paper mainly proposes a feature fusion approach to accelerate and improve the performance of federated learning.\nIntroduction Nowadays many smart devices rely on pretrained models, which makes the machine\u0026rsquo;s inference capability lack personalization and flexibility. At the same time, smart terminals also generate a large amount of valid privacy-sensitive data, which can improve the personalization capability of these models. Federated learning, a distributed training algorithm that can train models directly on terminals, solves this problem. Among federated learning algorithms, those represented by FedAvg effectively alleviate the privacy issues in information exchange, but later research has also shown that federated learning still has problems such as: computational consumption, model accuracy.\nThis paper proposes a feature-fusion federated learning algorithm, FedFusion, which fuses the features of the global model and the local model. The three main contributions of this paper: 1. introducing a feature fusion mechanism; 2. fusing the features of the local model and the global model in an effective and personalized way; 3. experiments show that the model outperforms the baselines in both accuracy and generalization ability while reducing communication by more than 60%.\nRelated Work Mainly the FedAvg algorithm of Federated Learning; not much to say here.\nMethods It is mainly divided into the feature fusion module and the FedFusion algorithm.\nFeature Fusion Modules The blue features in the figure are the two-channel features extracted by the local model, and the gray ones are the two-channel features extracted by the global model. The figure shows three feature fusion methods: Conv, Multi, Single.\nConv:\nFconv(El(x),Eg(x))=Wconv(Eg(x)∥El(x))F_{c o n v}\\left(E_{l}(x), E_{g}(x)\\right)=W_{c o n v}\\left(E_{g}(x) \\| E_{l}(x)\\right)where WconvW_{c o n v} denotes a learnable weight matrix of shape 2C*C. The concrete operation is to concatenate the global and local features and then apply a convolution.\nMulti:\nFmulti(El(x),Eg(x))=λEg(x)+(1−λ)El(x)F_{m u l t i}\\left(E_{l}(x), E_{g}(x)\\right)=\\lambda E_{g}(x)+(1-\\lambda) E_{l}(x)The multiplication operation uses a lambda weight matrix to perform a weighted sum of local and global.\nSingle:\nFsingle(El(x),Eg(x))=λEg(x)+(1−λ)El(x)F_{\\text {single}}\\left(E_{l}(x), E_{g}(x)\\right)=\\lambda E_{g}(x)+(1-\\lambda) E_{l}(x)The addition operation uses a scalar lambda weight to perform a weighted sum of local and global.\nFedFusion The training procedure uses the features of the previous round\u0026rsquo;s global model to participate in this round\u0026rsquo;s model feature aggregation training.\nExperiment Experiment setup Datasets: Mnist, Cifar10\nData partition methods:\nAn artificial non-IID partition, where each node contains only two classes A user-specific non-IID partition, where each node contains similar classes but with different distributions, similar to multi-task learning IID distribution Artificial Non-IID Partition Experimental results of two random runs of the artificial non-IID sampling scheme. The experimental results show that the multi fusion method works best; the conv fusion method converges slightly faster, but its final result is not as good as multi; none and single are both mediocre.\nThe paper\u0026rsquo;s explanation is that the multi operation allows the model to select the feature maps that are effective for local data for fusion, while the single operation is a scalar and cannot select specific channels of the feature maps.\nUser-Specific Non-IID Partition In terms of accuracy, FedFusion is much higher than FedAvg, with conv converging faster and also achieving higher accuracy.\nThe figure above shows the degree of communication reduction of FedFusion compared with FedAvg. From the results, in the user-specific non-IID scenario, the conv fusion method works better. This is because, under the user-specific non-IID partition, the classes of the data are similar, only with different distributions. And the conv fusion method is stronger at integrating the feature maps from the local and global models — that is, the knowledge of the data distributions of different nodes.\nFor the impact on generalization ability: when a new node joins, FedFusion only needs about 60 local epochs to fit, having a better initialization than the other methods.\nIID Partition The authors believe that the IID distribution also needs to be evaluated, because if a strategy cannot handle the IID distribution, its effectiveness is questionable.\nThe multi and conv fusion methods can achieve better accuracy with the least communication cost. In terms of the final converged accuracy, there is a great improvement compared with the other methods.\nTo summarize the three fusion methods:\nThe multi operation mainly makes a more flexible and more interpretable selection between the local feature maps and the global feature maps. Each channel of the weight vector represents the weight of the corresponding channel of the global feature map. When a gap appears between the classes of the data, the multi operation can select the most effective feature maps for fusion. The conv operation is more effective at integrating the knowledge of the global and local models. If the data on the nodes has similar classes but different distributions, the conv fusion method is better. The single fusion method brings a slight improvement.\nConclusion Using feature map fusion reduces the communication volume, improves model performance, and also improves the generalization ability for new nodes.\n","permalink":"https://blog.ereebay.me/en/posts/fedfusion/","summary":"\u003ch1 id=\"towards-faster-and-better-federated-learning-a-feature-fusion-approach\"\u003eTOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThis paper mainly proposes a feature fusion approach to accelerate and improve the performance of federated learning.\u003c/p\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eNowadays many smart devices rely on pretrained models, which makes the machine\u0026rsquo;s inference capability lack personalization and flexibility. At the same time, smart terminals also generate a large amount of valid privacy-sensitive data, which can improve the personalization capability of these models. Federated learning, a distributed training algorithm that can train models directly on terminals, solves this problem. Among federated learning algorithms, those represented by FedAvg effectively alleviate the privacy issues in information exchange, but later research has also shown that federated learning still has problems such as: \u003cstrong\u003ecomputational consumption\u003c/strong\u003e, \u003cstrong\u003emodel accuracy\u003c/strong\u003e.\u003c/p\u003e","title":"[Paper] TOWARDS FASTER AND BETTER FEDERATED LEARNING: A FEATURE FUSION APPROACH Reading Notes"},{"content":"Introduction Preamble Machine learning: suppose we use P to evaluate the performance of a computer program on a certain class of tasks T; if a program improves its performance on the tasks in T through experience E, then we say that, with respect to T and P, the program has learned from E.\nBasic Terminology Data-related:\nA dataset contains a series of records, and each record is an instance or a sample. That which reflects certain properties is an attribute or a feature. The value an attribute takes is called the attribute value, and the space spanned by the attributes is called the attribute space, the sample space, or the input space. Every point in this space is called a feature vector.\nTraining-related:\nThe process of learning a model from data is called learning or training. The data used is called training data, each sample is a training sample, and the whole collection is called the training set. The learned model corresponds to some underlying regularity in the data, which is called a hypothesis. The regularity itself is called the ground truth. The model can also be called a learner.\nAfter training, predictions need to be made. Information about the outcome of an instance is called a label, and an instance with a label is an example. The space spanned by the labels is the label space or the output space.\nClassification: the predicted values are discrete. Regression: the predicted values are continuous. Binary classification: the outputs are the positive class and the negative class.\nThe stage of making predictions with the learned model is called testing; the samples being predicted are called testing samples, and likewise there is a testing set\nSupervised learning: training with labeled data, such as classification and regression.\nUnsupervised learning: training with unlabeled data, such as clustering, which divides training samples into clusters.\nGeneralization: the ability of a model to work well on new samples\nIndependent and identically distributed (iid): the samples are all drawn independently from the same distribution.\nHypothesis Space Induction: from the specific to the general. (generalization)\nDeduction: from the general to the specific. (specialization)\nInductive learning: in the broad sense, learning from samples; in the narrow sense, acquiring concepts from data, which is called concept learning.\nVersion space: the set of hypotheses consistent with the training set\nInductive Bias Inductive bias: the preference of a machine learning algorithm for a certain type of hypothesis during the learning process; it corresponds to the assumption the learning algorithm itself makes about \u0026ldquo;what kind of model is better\u0026rdquo;\n","permalink":"https://blog.ereebay.me/en/posts/melonbook-1/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003ch2 id=\"preamble\"\u003ePreamble\u003c/h2\u003e\n\u003cp\u003eMachine learning: suppose we use P to evaluate the performance of a computer program on a certain class of tasks T; if a program improves its performance on the tasks in T through experience E, then we say that, with respect to T and P, the program has learned from E.\u003c/p\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"basic-terminology\"\u003eBasic Terminology\u003c/h2\u003e\n\u003cp\u003eData-related:\u003c/p\u003e\n\u003cp\u003eA dataset contains a series of records, and each record is an instance or a sample. That which reflects certain properties is an attribute or a feature. The value an attribute takes is called the attribute value, and the space spanned by the attributes is called the attribute space, the sample space, or the input space. Every point in this space is called a feature vector.\u003c/p\u003e","title":"Watermelon Book Reading Notes 1. Introduction"},{"content":"Model Evaluation and Selection Empirical Error and Overfitting Error rate: the overall proportion of misclassified samples\nAccuracy: the proportion of correctly classified samples\nError: the difference between the actual output and the true output; on training samples it is the empirical error or training error, and on new samples it is the generalization error\nOverfitting: generalization performance degrades\nUnderfitting: performs poorly even on the training samples\nEvaluation Methods A testing set is typically used to measure a model\u0026rsquo;s performance, with the testing error serving as an approximation of the generalization error\nHold-Out Method Split the dataset D into two mutually exclusive sets, one part as the training set S and the other as the testing set T. When S is large and T is small, the evaluation may not be accurate enough. If T is made larger, then the model trained on S may differ too much from the model that would be trained on D, and the evaluation result will not be accurate enough either.\nCross-Validation Split the dataset D into k mutually exclusive subsets, use k-1 of them as the training set and the remaining one as the testing set; this yields k groups of data, and the average of the testing results of these k groups is returned as the final result. The extreme case is leave-one-out, which gives more accurate results but at a greater computational cost.\nBootstrap Use bootstrap sampling to draw a new dataset D′D\u0026#x27; with replacement. Train on this new dataset and test on the rest. Advantage: this method is useful when the dataset is small and it is difficult to split training/testing sets effectively. Disadvantage: it changes the distribution of the initial dataset and introduces estimation bias.\nParameter Tuning and the Final Model Algorithm parameters have a great impact on the algorithm\u0026rsquo;s performance. Besides choosing an algorithm, its parameters also need to be adjusted, i.e., parameter tuning. After the model has been evaluated and selected using a validation set and the model and parameter configuration are determined, the model must be retrained once on the entire dataset D before testing — this is the finally submitted model. To distinguish it, the data used during model evaluation is called the validation set. The validation set is used to evaluate algorithm selection and parameter tuning, while the testing set is used to evaluate the generalization performance of the algorithm.\nPerformance Measures Performance measure: the evaluation criterion used to measure a model\u0026rsquo;s generalization ability\nThe most commonly used performance measure for regression tasks is MSE, the mean squared error:\nE(f;D)=1m∑i=1m(f(xi)−yi)2E(f ; D)=\\frac{1}{m} \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2}More generally, it can be written as:\nE(f;D)=∫x∼D(f(x)−y)2p(x)dxE(f ; D)=\\int_{x \\sim D}(f(x)-y)^{2} p(x) d xError Rate and Accuracy The two most commonly used performance measures: error rate and accuracy\nError rate: E(f;D)=1m∑i=1mI(f(xi)≠yi)E(f ; D)=\\frac{1}{m} \\sum_{i=1}^{m} I\\left(f\\left(x_{i}\\right) \\neq y_{i}\\right)\nAccuracy: acc⁡(f;D)=1m∑i=1mI(f(xi)=yi)=1−E(f;D)\\begin{aligned} \\operatorname{acc}(f ; D) \u0026amp;=\\frac{1}{m} \\sum_{i=1}^{m} \\mathbb{I}\\left(f\\left(x_{i}\\right)=y_{i}\\right) \\\\ \u0026amp;=1-E(f ; D) \\end{aligned}\nPrecision, Recall, and F1 In information retrieval, one usually cares about \u0026ldquo;how much of the retrieved information the user is interested in\u0026rdquo; and \u0026ldquo;how much of what the user is interested in has been retrieved\u0026rdquo;. This motivates the concepts of precision and recall.\nFor binary classification problems,\nPrecision = true positives / (true positives + false positives)\nRecall = true positives / (true positives + false negatives)\nThese two metrics usually conflict with each other: when recall is high, precision is relatively low. For example, to select as many good melons as possible, simply selecting every melon guarantees that all the good ones are selected.\nP-R curve: the precision-recall curve. If one learner\u0026rsquo;s PR curve can completely envelop another learner\u0026rsquo;s curve, the former learner is better. A more reasonable criterion is the size of the area under the curve, but it is not easy to estimate, so there are three evaluation approaches.\nBreak-Even Point (BEP): the value at which precision equals recall. F1 measure: F1=2PR/(P+R)= 2*TP/(total number of examples+TP-TN), based on the harmonic mean of P and R 1F1=12(1P+1R)\\frac{1}{F_{1}}=\\frac{1}{2}\\left(\\frac{1}{P}+\\frac{1}{R}\\right) The general form of the F1 measure (when P and R are weighted differently): Fβ=(1+β2)×P×R(β2×P)+RF_{\\beta}=\\frac{\\left(1+\\beta^{2}\\right) \\times P \\times R}{\\left(\\beta^{2} \\times P\\right)+R}, based on the harmonic mean 1Fβ=11+β2(1P+β2R)\\frac{1}{F_{\\beta}}=\\frac{1}{1+\\beta^{2}}\\left(\\frac{1}{P}+\\frac{\\beta^{2}}{R}\\right) β\\beta represents the relative importance of the two: it degenerates to F1 when equal to 1, recall matters more when it is greater than 1, and precision matters more when it is less than 1. Sometimes P and R need to be computed over n binary confusion matrices and then averaged, giving macro-precision, macro-recall, and macro-F1. Alternatively, the corresponding elements of the confusion matrices can be averaged first to obtain averaged positive and negative counts, from which micro-precision, micro-recall, and micro-F1 are then computed.\nROC and AUC ROC: Receiver Operating Characteristic curve.\nA learner typically outputs a probability prediction for an input sample, which is then compared against a classification threshold. The model ranks the samples most likely to be positive at the front and the least likely at the back, then splits the samples into two parts at a certain cut point: the first part is judged positive and the second part negative.\nDifferent classification tasks adopt different cut points: if precision matters more, the cut point is moved forward; if recall is valued more, it is moved backward.\nSort the samples according to the learner\u0026rsquo;s prediction results, and compute two values in that order.\nVertical axis: True Positive Rate (TPR): TPR=TPTP+FNT P R=\\frac{T P}{T P+F N}\nHorizontal axis: False Positive Rate (FPR): FPR=FPTN+FPFPR=\\frac {FP} {TN+FP} When comparing learners, if one curve can envelop the other, the former is better; if they cross, neither clearly dominates. A more reasonable judgment is to compare the area under the ROC curve, i.e., the AUC.\nAUC=12∑i=1m−1(xi+1−xi)(yi+yi+1)A U C=\\frac{1}{2} \\sum_{i=1}^{m-1}\\left(x_{i+1}-x_{i}\\right)\\left(y_{i}+y_{i+1}\\right)Loss: lrank⁡=1m+m−∑x+∈D+∑x∈D−(I(f(x+)\u0026lt;f(x−))+12I(f(x+)=f(x−)))\\operatorname{l_{rank}}=\\frac{1}{m^{+} m^{-}} \\sum_{x^{+} \\in D^{+}} \\sum_{x \\in D^{-}}\\left(\\mathbb{I}\\left(f\\left(x^{+}\\right)\u0026lt;f\\left(x^{-}\\right)\\right)+\\frac{1}{2} \\mathbb{I}\\left(f\\left(x^{+}\\right)=f\\left(x^{-}\\right)\\right)\\right)\nAUC= 1-lrank\nhttps://datawhalechina.github.io/pumpkin-book/#/chapter2/chapter2 For more details, see the Pumpkin Book\nCost-Sensitive Error Rate and Cost Curve To weigh the different losses caused by different types of errors, errors can be assigned \u0026ldquo;unequal costs\u0026rdquo;\nTaking binary classification as an example, set up a cost matrix\nBinary classification cost matrix\nTrue class Predicted class 0 Predicted class 1 0 0 cost01 1 cost10 0 If misclassifying 0 as 1 is considered the greater loss, then cost01 is greater than cost10; the greater the difference in loss, the greater the difference between the values.\nThe performance measures introduced above all implicitly assume equal costs; under unequal costs, the goal becomes minimizing the total cost overall\nThe cost-sensitive error rate is:\nE(f;D;cos⁡t)=1m(∑xi∈D+I(f(xi)≠yi)×cost01,+∑Xi∈D−I(f(xi)≠yi)×cost10) E(f ; D ; \\cos t)=\\frac{1}{m}\\left(\\sum_{x_{i} \\in D^{+}} \\mathbb{I}\\left(f\\left(x_{i}\\right) \\neq y_{i}\\right) \\times cost_{01}, \\quad+\\sum_{X_{i} \\in D^{-}} \\mathbb{I}\\left(f\\left(x_{i}\\right) \\neq y_{i}\\right) \\times cost_{1 0}\\right) Under unequal costs, the ROC curve cannot reflect the learner\u0026rsquo;s expected total cost, but the cost curve can serve this purpose.\nThe horizontal axis is the positive-class probability cost over [0,1]:\nP(+)cost=p×cost01p×cost01+(1−p)×cost10 P(+) cost=\\frac{p \\times cost_{01}}{p \\times cost_{01}+(1-p) \\times cost_{10}} where p is the probability that an example is positive\nThe vertical axis is the normalized cost over [0,1]:\ncostnorm =FNR×p×cost01+FPR×(1−p)×cos⁡t10p×cost01+(1−p)×cost10 cost_{\\text {norm }}=\\frac{ F N R \\times p \\times cost_{01}+FP R \\times(1-p) \\times \\cos t_{10}}{p \\times cost_{01} +(1-p) \\times cost_{10}} FNR is the false negative rate, and FPR is the false positive rate. FNR=1-TPR\nComparative Tests Performance comparison in machine learning usually relies on statistical hypothesis tests, for the following reasons:\nWe want to compare generalization performance, but what experiments evaluate is performance on the testing set, and the two comparison results do not necessarily agree Performance on the testing set depends heavily on the choice of testing set The algorithm itself involves a certain degree of randomness Hypothesis Testing The error rate ϵ{\\epsilon} serves as the performance measure. For a learner with error rate ϵ{\\epsilon}, suppose we obtain a testing error of ϵ^\\hat{\\epsilon} on the testing set; this means that ϵ^×m\\hat{\\epsilon} \\times m of the mm samples are misclassified. It is easy to derive that the probability of obtaining testing error ϵ^\\hat{\\epsilon} on a test set of mm samples, for a learner with generalization error rate ϵ{\\epsilon}, is:\nP(ϵ^;ϵ)=(mϵ^×m)ϵϵ^×m(1−ϵ)m−ϵ^×m P(\\hat{\\epsilon} ; \\epsilon)=\\left(\\begin{array}{c}m \\\\ \\hat{\\epsilon} \\times m\\end{array}\\right) \\epsilon^{\\hat{\\epsilon} \\times m}(1-\\epsilon)^{m-\\hat{\\epsilon} \\times m} A binomial distribution is used for the hypothesis test. The hypothesis takes the form ϵ≤ϵ0\\epsilon \\leq \\epsilon_{0}, and the maximum error rate we can observe with probability 1−α1-\\alpha is:\nϵˉ=max⁡ϵ s.t. ∑i=ϵ0×m+1m(mi)ϵi(1−ϵ)m−i\u0026lt;α \\bar{\\epsilon}=\\max \\epsilon \\quad \\text { s.t. } \\sum_{i=\\epsilon_{0} \\times m+1}^{m}\\left(\\begin{array}{c}m \\\\ i\\end{array}\\right) \\epsilon^{i}(1-\\epsilon)^{m-i}\u0026lt;\\alpha If the testing error ϵ^\\hat{\\epsilon} is smaller than the critical value ϵˉ\\bar{\\epsilon}, we accept the hypothesis at the 1−α1-\\alpha confidence level; otherwise, at the significance level α\\alpha, we conclude that the generalization error rate is greater than ϵ\\epsilon.\nIf the hold-out method or cross-validation is applied multiple times, k testing error rates are obtained. The mean and variance can then be computed, and a t-test can be used for the hypothesis test. The hypothesis in this case is that the mean equals the error rate. The variable:\nτt=k(μ−ϵ0)ϵ \\tau_{t}=\\frac{\\sqrt{k}\\left(\\mu-\\epsilon_{0}\\right)}{\\epsilon} follows a t-distribution with k-1 degrees of freedom.\nMore test methods are described on p.41 of the Watermelon Book\nBias and Variance Besides estimating generalization performance, we also want to understand why the model performs the way it does, which calls for bias-variance decomposition. Taking a regression task as an example, y_D is the label of x on the dataset, y is the true label of x, and f(x,D) is the predicted output.\nThe expected prediction is then: fˉ(x)=ED[f(x;D)]\\bar{f}(x)=\\mathbb{E}_{D}[f(x ; D)]\nTraining sets with different numbers of samples give rise to variance. Then perform the decomposition:\nE(f;D)=E−D[(f(x;D)−yD)2]=E−D[(f(x;D)−fˉ(x))2]+(fˉ(x)−y)2+E−D[(yD−y)2]=bias⁡2(x)+var⁡(x)+ε2 \\begin{aligned} E(f ; D) \u0026amp;=\\mathbb{E}_{-} D\\left[\\left(f(x ; D)-y_{D}\\right)^{2}\\right] \\\\ \u0026amp;=\\mathbb{E}_{-} D\\left[(f(x ; D)-\\bar{f}(x))^{2}\\right]+(\\bar{f}(x)-y)^{2}+\\mathbb{E}_{-} D\\left[\\left(y_{D}-y\\right)^{2}\\right] \\\\ \u0026amp;=\\operatorname{bias}^{2}(x)+\\operatorname{var}(x)+\\varepsilon^{2} \\end{aligned} The generalization error can be decomposed into the sum of bias, variance, and noise\nBias measures the deviation of the algorithm\u0026rsquo;s expected prediction from the true result, characterizing the fitting capability of the learning algorithm itself Variance measures the change in learning performance caused by changes in training sets of the same size, characterizing the impact of data perturbation Noise expresses the lower bound of the expected generalization error that any learning algorithm can achieve on the current task, characterizing the difficulty of the learning problem itself. ","permalink":"https://blog.ereebay.me/en/posts/melonbook-2/","summary":"\u003ch1 id=\"model-evaluation-and-selection\"\u003eModel Evaluation and Selection\u003c/h1\u003e\n\u003ch2 id=\"empirical-error-and-overfitting\"\u003eEmpirical Error and Overfitting\u003c/h2\u003e\n\u003cp\u003eError rate: the overall proportion of misclassified samples\u003c/p\u003e\n\u003cp\u003eAccuracy: the proportion of correctly classified samples\u003c/p\u003e\n\u003cp\u003eError: the difference between the actual output and the true output; on training samples it is the empirical error or training error, and on new samples it is the generalization error\u003c/p\u003e\n\u003cp\u003eOverfitting: generalization performance degrades\u003c/p\u003e\n\u003cp\u003eUnderfitting: performs poorly even on the training samples\u003c/p\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"evaluation-methods\"\u003eEvaluation Methods\u003c/h2\u003e\n\u003cp\u003eA testing set is typically used to measure a model\u0026rsquo;s performance, with the testing error serving as an approximation of the generalization error\u003c/p\u003e","title":"Watermelon Book Reading Notes 2. Model Evaluation and Selection"},{"content":"Abstract The goal of federated learning is to obtain a shared global model that all nodes can use. However, due to Non-IID data distributions, local models trained solely on local data often outperform the global model, which makes these nodes reluctant to participate in the federation process. This post introduces techniques currently used to personalize the global model so as to improve its performance on individual nodes.\nIntroduction Federated learning is a distributed machine learning algorithm designed for Non-IID data. It enables multiple nodes to collaboratively train a shared global model without sharing their own data. At the beginning of each training round, the central server sends the current global model to every node, and each node trains the model on its local data. The central server then collects the model updates from all nodes and applies them to the global model.\nThe main reason for a node to participate in federated learning is to obtain a better model. When a node does not have enough data to train a good local model on its own, it can obtain a decent model through federated learning. But for nodes with abundant data, the benefit of federated learning remains in question. Yu, in Salvaging Federated Learning by Local Adaptation, noted that for many tasks some nodes gain no benefit because the globally shared model underperforms a model trained on their local data. Hanzely, in Federated Learning of a Mixture of Global and Local Models, questioned whether the global model deviates too far from the classical use cases of users.\nThe purpose of this post is to survey recent techniques for building personalized models on local nodes, where the personalized models are expected to outperform either the shared global model or the model trained locally by the node.\nNeed for Personalization Wu, in Personalized federated learning for intelligent iot applications, described the three major challenges federated learning systems face with respect to personalization:\nHeterogeneity of devices in terms of storage, computation, and communication capabilities. Data heterogeneity caused by Non-IID distributions. Model heterogeneity arising from the need for different models due to different user environments. To address the challenges brought by data heterogeneity and Non-IID data, personalizing the global model has become increasingly important. Most personalization techniques consist of two steps: 1. learn a global model collaboratively across the nodes; 2. personalize the global model using each node\u0026rsquo;s private data. Jiang argued that optimizing solely for global-model accuracy hardly produces models that personalize well, and proposed that, for personalization in federated learning to be useful in practice, the following three objectives must be solved simultaneously rather than independently:\nBuild improved personalized models that benefit the majority of nodes Build a more accurate global model, which enables better personalization for nodes with little data Achieve fast model convergence within few training rounds In addition, only a fraction of the samples in a node\u0026rsquo;s local data may be relevant to the specific task, while the remaining samples are irrelevant ones that hurt model training. Tuor proposed a method that uses a relevance model built on a small benchmark to distinguish relevant from irrelevant samples, and then uses only the relevant samples during federated learning.\nTechniques Adding User Context Before introducing methods for personalizing the global model for each node, it must be pointed out that a shared global model can also make highly personalized predictions, provided that the node\u0026rsquo;s context and personal information are properly featurized and integrated into the dataset. However, most public datasets do not contain contextual features, and developing techniques that can effectively incorporate context remains an important open problem—one with great potential to improve federated learning models. At the same time, whether such contextual featurization can be done without adversely affecting privacy remains to be studied. As an intermediate approach between a single global model and purely local models, Masour, in Three approaches for personalization with applications to federated learning, suggested clustering users—grouping similar clients together—and training a separate model for each group.\nTransfer Learning Transfer learning allows deep learning models to leverage the capabilities acquired while solving problem A to solve another related problem. Schneider and Vlachos, in Mass personalization of deep learning, discussed using transfer learning to achieve model personalization in non-federated settings. Transfer learning has likewise been applied in federated settings. Wang, in Federated evalua- tion of on-device personalization, proposed re-learning some or all parameters of the trained global model on local data; the previously cited work provides the theoretical justification. By initializing local training with the parameters of the trained global model, transfer learning exploits the knowledge extracted by the global model instead of learning from scratch. Meanwhile, to avoid catastrophic forgetting, care must be taken not to retrain/finetune on local data for too long. One variant of this technique freezes the base layers of the global model and trains only the top layers on local data (similar to freezing the feature layers and finetuning the fully connected layers).\nMulti-task Learning In multi-task learning, a model exploits the commonalities and differences across tasks through joint learning, so that multiple related tasks can be solved simultaneously. Smith, in Federated multi-task learning, showed that multi-task learning is a natural choice for constructing personalized federated models and developed the MOCHA algorithm. A drawback of using multi-task learning in federated settings, however, is that, because it produces one model per task, all nodes must participate in every round of training.\nMeta Learning Meta-learning involves training across multiple learning tasks to produce models that adapt quickly—models that can fit and solve a new task with only a few training samples. Finn proposed the Model-Agnostic Meta-Learning algorithm (MAML), which is compatible with any model trained by gradient descent. MAML builds internal representations that work across multiple tasks, so for a new task, fine-tuning the top layers can produce good results.\nJiang pointed out that the federated learning process can be viewed as meta-training, while the personalization process can be viewed as meta-testing; the FedAvg algorithm is then very similar to Reptile. The author also observed that careful fine-tuning can yield accurate global models that are easy to personalize, whereas optimizing purely for global-model accuracy sacrifices the model\u0026rsquo;s subsequent personalization capability. Other personalization approaches in federated learning treat the construction of the global model and personalization as two separate processes; Jiang proposed an improved FedAvg algorithm that achieves both a better global model and better personalized models.\nFallah, in Personalized federated learning: A meta-learning approach, formulated a new version of the standard federated learning problem that incorporates MAML, seeking a global model that performs well on every node after being updated with that node\u0026rsquo;s own loss function; they further proposed Per-FedAvg to solve this problem. Khodak, in Adaptive gradient-based meta-learning methods, proposed ARUBA and demonstrated improved performance by applying it to FedAvg. Chen, in Federated meta-learning for recommendation, proposed a federated meta-learning framework for building personalized recommendation models, in which both the algorithm and the model are parameterized and need to be optimized.\nKnowledge Distillation Caruana showed that the knowledge of an ensemble of models can be compressed into a single model that is easy to deploy. Knowledge distillation develops this idea further, extracting the knowledge of a large teacher network into a smaller student network by having the student imitate the teacher. Overfitting poses a major obstacle during personalization, especially for nodes with small local datasets. Yu proposed that treating the global model as the teacher and the node\u0026rsquo;s personalized model as the student can alleviate overfitting. Li proposed FedMD, a federated learning framework based on knowledge distillation and transfer learning that allows nodes to independently design their own networks while leveraging both their private data and a global public dataset.\nBase+Personalization Layers In the conventional federated learning setting, data is distributed across the multiple devices participating in training. To mitigate the adverse effects of such data heterogeneity, Arivazhagan et al., in Federated learning with personalization layers, proposed the FedPer architecture, in which the base layers are trained with FedAvg while the top layers (the personalization layers) are trained on the local node with a variant of gradient descent. The transfer-learning approach first trains all layers on global data and then retrains some or all layers on the local node\u0026rsquo;s data, whereas FedPer trains the base layers on global data and the personalization layers on local data, separately.\nMixtrue of Global and Local Models Hanzely, in Federated learning of a mixture of global and local models, proposed a new problem formulation that seeks a balance between the global model and local models. Instead of learning a single global model, each node learns a mixture of the global model and its own local model. To solve this problem, the authors proposed LLGD, Loopless Local Gradient Descent (LLGD), a variant of gradient descent. Unlike full averaging, this method only takes averaged steps, because full averaging can be too aggressive.\nDiscussion In federated learning, when a node\u0026rsquo;s local dataset is small and IID (isn\u0026rsquo;t this similar to the meta-learning scenario?), the global model usually surpasses local models, and most nodes benefit from the federation process. However, when nodes possess sufficiently large private datasets with Non-IID distributions, local models usually outperform the global model, and nodes are generally reluctant to participate in federated learning. An open theoretical question is: how to determine when the global model will outperform the model on a single node\nThis post has mainly summarized several techniques for optimizing the personalization of the global model. With a few exceptions, most prior work has focused on measuring the performance of global models on aggregated data rather than on individual nodes. But if the global model is to be personalized before use, aggregate global performance becomes meaningless.\nPersonalized models usually outperform both the global model and purely local models on individual nodes. In some cases, however, a personalized model cannot reach the same capability as a purely local model, especially under differential privacy and similar conditions.\n","permalink":"https://blog.ereebay.me/en/posts/notes-fl-overview/","summary":"\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThe goal of federated learning is to obtain a shared global model that all nodes can use. However, due to Non-IID data distributions, local models trained solely on local data often outperform the global model, which makes these nodes reluctant to participate in the federation process. This post introduces techniques currently used to personalize the global model so as to improve its performance on individual nodes.\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eFederated learning is a distributed machine learning algorithm designed for Non-IID data. It enables multiple nodes to collaboratively train a shared global model without sharing their own data. At the beginning of each training round, the central server sends the current global model to every node, and each node trains the model on its local data. The central server then collects the model updates from all nodes and applies them to the global model.\u003c/p\u003e","title":"A Survey on Personalization Capabilities in Federated Learning"},{"content":"A Survey on Personalization in Federated Learning Abstract The goal of federated learning is to obtain a shared global model that all nodes can use. However, due to Non-IID data distributions, local models trained solely on local data often outperform the global model, which makes these nodes reluctant to participate in the federation process. This post introduces techniques currently used to personalize the global model so as to improve its performance on individual nodes.\nIntroduction Federated learning is a distributed machine learning algorithm designed for Non-IID data. It enables multiple nodes to collaboratively train a shared global model without sharing their own data. At the beginning of each training round, the central server sends the current global model to every node, and each node trains the model on its local data. The central server then collects the model updates from all nodes and applies them to the global model.\nThe main reason for a node to participate in federated learning is to obtain a better model. When a node does not have enough data to train a good local model on its own, it can obtain a decent model through federated learning. But for nodes with abundant data, the benefit of federated learning remains in question. Yu, in Salvaging Federated Learning by Local Adaptation, noted that for many tasks some nodes gain no benefit because the globally shared model underperforms a model trained on their local data. Hanzely, in Federated Learning of a Mixture of Global and Local Models, questioned whether the global model deviates too far from the classical use cases of users.\nThe purpose of this post is to survey recent techniques for building personalized models on local nodes, where the personalized models are expected to outperform either the shared global model or the model trained locally by the node.\nNeed for Personalization Wu, in Personalized federated learning for intelligent iot applications, described the three major challenges federated learning systems face with respect to personalization:\nHeterogeneity of devices in terms of storage, computation, and communication capabilities. Data heterogeneity caused by Non-IID distributions. Model heterogeneity arising from the need for different models due to different user environments. To address the challenges brought by data heterogeneity and Non-IID data, personalizing the global model has become increasingly important. Most personalization techniques consist of two steps: 1. learn a global model collaboratively across the nodes; 2. personalize the global model using each node\u0026rsquo;s private data. Jiang argued that optimizing solely for global-model accuracy hardly produces models that personalize well, and proposed that, for personalization in federated learning to be useful in practice, the following three objectives must be solved simultaneously rather than independently:\nBuild improved personalized models that benefit the majority of nodes Build a more accurate global model, which enables better personalization for nodes with little data Achieve fast model convergence within few training rounds In addition, only a fraction of the samples in a node\u0026rsquo;s local data may be relevant to the specific task, while the remaining samples are irrelevant ones that hurt model training. Tuor proposed a method that uses a relevance model built on a small benchmark to distinguish relevant from irrelevant samples, and then uses only the relevant samples during federated learning.\nTechniques Adding User Context Before introducing methods for personalizing the global model for each node, it must be pointed out that a shared global model can also make highly personalized predictions, provided that the node\u0026rsquo;s context and personal information are properly featurized and integrated into the dataset. However, most public datasets do not contain contextual features, and developing techniques that can effectively incorporate context remains an important open problem—one with great potential to improve federated learning models. At the same time, whether such contextual featurization can be done without adversely affecting privacy remains to be studied. As an intermediate approach between a single global model and purely local models, Masour, in Three approaches for personalization with applications to federated learning, suggested clustering users—grouping similar clients together—and training a separate model for each group.\nTransfer Learning Transfer learning allows deep learning models to leverage the capabilities acquired while solving problem A to solve another related problem. Schneider and Vlachos, in Mass personalization of deep learning, discussed using transfer learning to achieve model personalization in non-federated settings. Transfer learning has likewise been applied in federated settings. Wang, in Federated evalua- tion of on-device personalization, proposed re-learning some or all parameters of the trained global model on local data; the previously cited work provides the theoretical justification. By initializing local training with the parameters of the trained global model, transfer learning exploits the knowledge extracted by the global model instead of learning from scratch. Meanwhile, to avoid catastrophic forgetting, care must be taken not to retrain/finetune on local data for too long. One variant of this technique freezes the base layers of the global model and trains only the top layers on local data (similar to freezing the feature layers and finetuning the fully connected layers).\nMulti-task Learning In multi-task learning, a model exploits the commonalities and differences across tasks through joint learning, so that multiple related tasks can be solved simultaneously. Smith, in Federated multi-task learning, showed that multi-task learning is a natural choice for constructing personalized federated models and developed the MOCHA algorithm. A drawback of using multi-task learning in federated settings, however, is that, because it produces one model per task, all nodes must participate in every round of training.\nMeta Learning Meta-learning involves training across multiple learning tasks to produce models that adapt quickly—models that can fit and solve a new task with only a few training samples. Finn proposed the Model-Agnostic Meta-Learning algorithm (MAML), which is compatible with any model trained by gradient descent. MAML builds internal representations that work across multiple tasks, so for a new task, fine-tuning the top layers can produce good results.\nJiang pointed out that the federated learning process can be viewed as meta-training, while the personalization process can be viewed as meta-testing; the FedAvg algorithm is then very similar to Reptile. The author also observed that careful fine-tuning can yield accurate global models that are easy to personalize, whereas optimizing purely for global-model accuracy sacrifices the model\u0026rsquo;s subsequent personalization capability. Other personalization approaches in federated learning treat the construction of the global model and personalization as two separate processes; Jiang proposed an improved FedAvg algorithm that achieves both a better global model and better personalized models.\nFallah, in Personalized federated learning: A meta-learning approach, formulated a new version of the standard federated learning problem that incorporates MAML, seeking a global model that performs well on every node after being updated with that node\u0026rsquo;s own loss function; they further proposed Per-FedAvg to solve this problem. Khodak, in Adaptive gradient-based meta-learning methods, proposed ARUBA and demonstrated improved performance by applying it to FedAvg. Chen, in Federated meta-learning for recommendation, proposed a federated meta-learning framework for building personalized recommendation models, in which both the algorithm and the model are parameterized and need to be optimized.\nKnowledge Distillation Caruana showed that the knowledge of an ensemble of models can be compressed into a single model that is easy to deploy. Knowledge distillation develops this idea further, extracting the knowledge of a large teacher network into a smaller student network by having the student imitate the teacher. Overfitting poses a major obstacle during personalization, especially for nodes with small local datasets. Yu proposed that treating the global model as the teacher and the node\u0026rsquo;s personalized model as the student can alleviate overfitting. Li proposed FedMD, a federated learning framework based on knowledge distillation and transfer learning that allows nodes to independently design their own networks while leveraging both their private data and a global public dataset.\nBase+Personalization Layers In the conventional federated learning setting, data is distributed across the multiple devices participating in training. To mitigate the adverse effects of such data heterogeneity, Arivazhagan et al., in Federated learning with personalization layers, proposed the FedPer architecture, in which the base layers are trained with FedAvg while the top layers (the personalization layers) are trained on the local node with a variant of gradient descent. The transfer-learning approach first trains all layers on global data and then retrains some or all layers on the local node\u0026rsquo;s data, whereas FedPer trains the base layers on global data and the personalization layers on local data, separately.\nMixtrue of Global and Local Models Hanzely, in Federated learning of a mixture of global and local models, proposed a new problem formulation that seeks a balance between the global model and local models. Instead of learning a single global model, each node learns a mixture of the global model and its own local model. To solve this problem, the authors proposed LLGD, Loopless Local Gradient Descent (LLGD), a variant of gradient descent. Unlike full averaging, this method only takes averaged steps, because full averaging can be too aggressive.\nDiscussion In federated learning, when a node\u0026rsquo;s local dataset is small and IID (isn\u0026rsquo;t this similar to the meta-learning scenario?), the global model usually surpasses local models, and most nodes benefit from the federation process. However, when nodes possess sufficiently large private datasets with Non-IID distributions, local models usually outperform the global model, and nodes are generally reluctant to participate in federated learning. An open theoretical question is: how to determine when the global model will outperform the model on a single node\nThis post has mainly summarized several techniques for optimizing the personalization of the global model. With a few exceptions, most prior work has focused on measuring the performance of global models on aggregated data rather than on individual nodes. But if the global model is to be personalized before use, aggregate global performance becomes meaningless.\nPersonalized models usually outperform both the global model and purely local models on individual nodes. In some cases, however, a personalized model cannot reach the same capability as a purely local model, especially under differential privacy and similar conditions.\n","permalink":"https://blog.ereebay.me/en/posts/survey-personal/","summary":"\u003ch1 id=\"a-survey-on-personalization-in-federated-learning\"\u003eA Survey on Personalization in Federated Learning\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThe goal of federated learning is to obtain a shared global model that all nodes can use. However, due to Non-IID data distributions, local models trained solely on local data often outperform the global model, which makes these nodes reluctant to participate in the federation process. This post introduces techniques currently used to personalize the global model so as to improve its performance on individual nodes.\u003c/p\u003e","title":"A Survey on Personalization Capabilities in Federated Learning"},{"content":"Decoupled Neural Interfaces using Synthetic Gradients Abstract Training neural networks typically requires pushing the computation graph forward and then backpropagating errors to update the weights. In this sense, all layers of the network are locked, because they must wait for other parts of the network to complete their forward inference and backward pass before they can be updated. In this work, we introduce models of future computation in the network graph, in order to decouple modules and thereby break this constraint. These models use only local information to predict the outcome of a subgraph. In particular, we focus on modeling the error gradients: by replacing the true backpropagated error gradients with modeled synthetic gradients, subgraphs are decoupled and can be updated independently and asynchronously—namely, we realize decoupled neural interfaces. We present results for feed-forward models in which every layer is trained asynchronously; for RNNs, where predicting the future gradients of one layer extends the effective modeling horizon of the RNN; and for hierarchical RNN systems with different timescales. Finally, we show that, beyond predicting gradients, the same framework can also be used to predict inputs, resulting in models that are decoupled in both the forward and backward passes—equivalent to two independent networks that can learn together so as to be composed into a single functional network.\nIntroduction Each layer (or module) in a directed neural network can be viewed as a computation step that transforms its input. These modules are connected by directed edges, forming a feed-forward graph that defines the flow of data from the network\u0026rsquo;s input, through each module, to the network\u0026rsquo;s output. A loss is defined on the output to produce errors, which are backpropagated through the network graph to update each module\u0026rsquo;s weights.\nThis mechanism leads to the following forms of locking:\nForward Locking: no module can process its input before the earlier nodes of the forward computation graph have been executed. Update Locking: no module can be updated until all relevant modules have finished executing in the forward graph. Backwards Locking: no module can be updated until all relevant modules have executed both the feed-forward and the backward model—for example, the BP algorithm. Forward, update, and backwards locking force neural networks to run and update sequentially and synchronously. For simple networks this seems benign, but it becomes a serious problem for large, complex, irregular networks, or network systems running in multiple environments with asynchronous timescales.\nFor example, consider distributed models in which part of the model is shared and used by many downstream clients. This means that all clients must finish executing and pass their error gradients back to the shared model before it can be updated—which means the training speed of the system is dictated by the slowest client. If training of the network could be parallelized, computation time could be greatly reduced.\nThe goal of this work is to remove update locking from neural networks. This can be achieved by removing backpropagation. To update the weights θi\\theta_{i} of module i, we approximate the backpropagation function as closely as possible:\n∂L∂θi=fBprop ((hi,xi,yi,θi),…)∂hi∂θi≃f^Bprop (hi)∂hi∂θi \\begin{aligned} \\frac{\\partial L}{\\partial \\theta_{i}} \u0026amp;=f_{\\text {Bprop }}\\left(\\left(h_{i}, x_{i}, y_{i}, \\theta_{i}\\right), \\ldots\\right) \\frac{\\partial h_{i}}{\\partial \\theta_{i}} \\\\ \u0026amp; \\simeq \\hat{f}_{\\text {Bprop }}\\left(h_{i}\\right) \\frac{\\partial h_{i}}{\\partial \\theta_{i}} \\end{aligned} where h denotes the layer\u0026rsquo;s activations, x the input, y the supervision (labels), and L the total loss. The update now depends entirely on h—that is, on information local to module i.\nThe premise of this method is to allow neural network modules to interact and to be trained without update locking. Here the authors replace the conventional neural interface (the connection between two modules in a network) with a Decoupled Neural Interface (DNI). In short, when one network layer passes activations to another layer, there is an associated model that produces a predicted error gradient for those activations. The predicted gradient is a function of the activation alone and does not depend on other events, states, or losses. The sending layer can then use this synthetic gradient to update immediately. By removing update and backwards locking, we can train networks without synchronization. We also present preliminary results extending this idea to remove forward locking as well, so that the modules of the network can likewise be trained without a synchronized forward pass. When applied to RNNs, we show that using synthetic gradients allows an RNN to model a time horizon far beyond the limit of BPTT. We further show that using synthetic gradients to decouple two RNNs operating at different timescales can greatly improve training.\nDecoupled Neural Interfaces We first describe the high-level communication protocol used to allow asynchronous learning agents to communicate.\nAs shown in Figure 1, Sender A sends a message (activation) hAh_A to Receiver B. Intuitively, A can be thought of as the earlier layer and B as the later layer. B has a utility MBM_B that processes the signal hAh_A to predict the feedback. The error signal: δ^A=MB(hA,sB,c)\\hat{\\delta}_{A}=M_{B}\\left(h_{A}, s_{B}, c\\right), where hAh_A is the message (activation), sBs_B is B\u0026rsquo;s state, and c is some other potentially relevant information, such as labels or context. A can update immediately using this error signal. B can also, in time, fully evaluate the true δA\\delta_A; thus B\u0026rsquo;s utility model can be updated to fit the true utility, reducing the discrepancy between the true and synthetic errors.\nThis protocol allows A to send messages to B in a way that decouples the updates of A and B—A does not have to wait for B to evaluate the true utility before updating—and A can still learn to send messages to B.\nWe can apply this protocol to the communication process within networks, giving rise to so-called Decoupled Neural Interfaces (DNI). For neural networks, the feedback error signal δ^A\\hat{\\delta}_A can take different forms: for example, gradients can serve as error signals used together with backpropagation; target information can serve as error signals used together with target propagation; or they can even be used as values incorporated into a reinforcement learning framework. This paper focuses on differentiable networks trained via backpropagation with gradient-based updates. We therefore concentrate on the produced error gradient as the feedback δ^A\\hat{\\delta}_A, called the synthetic gradient.\nNotation. We define the function at step i as fi, and the composition of the functions from step i to step j as Fij. The loss at layer i is defined as Li.\nSynthetic Gradient for Feed-Forward Networks Consider the DNI formulation for feed-forward networks: N layers fi, each taking input hi-1 and producing output hi = fi(hi-1), where h0 = x is the input data. The feed-forward graph of the whole network can be written as F1N. See Figure 3(a)\nDefine the loss function on the network output as L = LN. Each layer fi has parameters thetai, updated by a gradient update rule to minimize L(hN)\nθi←θi−αδi∂hi∂θi;δi=∂L∂hi \\theta_{i} \\leftarrow \\theta_{i}-\\alpha \\delta_{i} \\frac{\\partial h_{i}}{\\partial \\theta_{i}} ; \\quad \\delta_{i}=\\frac{\\partial L}{\\partial h_{i}} α is the learning rate, and the derivative with respect to hi is obtained via backpropagation. The dependence on δi\\delta_i means that layer i\u0026rsquo;s update must wait until the rest of the network has been updated—that is, all of F(i+1)N has executed its feed-forward and feedback phases. Layer i is thus update-locked to F(i+1)N.\nTo remove update locking, we adopt the communication protocol described earlier. Layer i sends the message hi to the following layer, which has a communication model Mi+1 that produces the synthetic error gradient δ^i=Mi+1(hi)\\hat{\\delta}_{i} = M_{i+1}(h_{i}). As shown in the figure:\nLayer i, as well as the other layers within F1i, can be updated immediately\nθn←θn−αδ^i∂hi∂θn,n∈{1,…,i} \\theta_{n} \\leftarrow \\theta_{n}-\\alpha \\hat{\\delta}_{i} \\frac{\\partial h_{i}}{\\partial \\theta_{n}}, n \\in\\{1, \\ldots, i\\} To train the parameters of the synthetic gradient model Mi+1, we wait until the true error gradient δi\\delta_i has been computed, and then compute the MSE between the two.\nMoreover, for feed-forward networks, we can use synthetic gradients as the communication feedback to decouple every layer in the network. As shown in the figure:\nThe complete execution of this mechanism:\nIn this case, the target error gradient δi\\delta_i is produced by backpropagating δ^i+1\\hat{\\delta}_{i+1} through layer i+1. Thus δi\\delta_i is not the true error gradient but an estimate obtained from the synthetic gradient models further downstream. Surprisingly, this does not make the errors worse—learning remains stable even across many layers, as the experiments below show.\nIn addition, supervision or context c can be incorporated when computing the synthetic gradient.\nδ^i=Mi+1(hi,c) \\hat{\\delta}_{i} = {M_{i+1}\\left(h_{i}, c\\right) } This procedure allows a layer to be updated immediately after its forward pass has been executed. This paves the way for training sub-parts—or individual layers—of a network in an asynchronous manner.\nExperiment Feed-Forward Networks We apply DNI to feed-forward networks to allow asynchronous or sporadic training of individual layers, as might arise in distributed training.\nAs explained earlier, introducing synthetic gradients decouples the network layers, allowing them to communicate with one another without being affected by update locking.\nAsynchronous Updates To demonstrate the improvements brought by the DNI-decoupled layers, we ran experiments on a four-layer fully connected network for MNIST, in which each layer\u0026rsquo;s backward pass and update occur in random order with probability pupdate. (That is, a layer is updated only for a fraction pupdate of the forward passes.) This completely breaks backpropagation—for example, the first layer would only receive error-gradient updates with probability pupdate³—and even then, the system would still be constrained to be synchronous. With DNI, however, the communication gap between layers is bridged, and the randomness of layer updates does not affect downstream layers, because synthetic gradients are used. We sampled different values of pupdate uniformly between 0 and 1. DNI with and without labels is shown in Figure 7.\nWith pupdate = 0.2, the network can still be trained to 2% error. Incredibly, when DNI is conditioned on the data labels (a reasonable assumption if training is performed in a distributed manner), the network trains perfectly with an update probability of just 5%, albeit more slowly.\nComplete Unlock By also eliminating forward locking, feed-forward networks are made fully asynchronous. In this case, every network layer has a synthetic gradient model as well as a synthetic input model used to predict the input data, as shown in the figure below:\nEach network layer can then be trained independently, relying on the synthetic gradient and the synthetic input model. The figure below shows the experimental results.\nThe experiments show that in this setting the model can be trained in a fully asynchronous and independent manner, although it takes somewhat longer to reach the 2% error rate.\nDiscussion \u0026amp; Conclusion This paper introduced DNI, a method that uses synthetic gradients to decouple the communication between network layers, allowing them to update independently. It also demonstrated that the method can fully separate all layers of the network, so that they can train in a completely asynchronous, unordered, and sporadic manner.\nIt is worth noting that, although this paper presented empirical evidence for the effectiveness of DNI and synthetic gradients, the work of Czarnecki et al. investigated the theoretical understanding in greater depth and established convergence.\n","permalink":"https://blog.ereebay.me/en/posts/dnisg/","summary":"\u003ch1 id=\"decoupled-neural-interfaces-using-synthetic-gradients\"\u003eDecoupled Neural Interfaces using Synthetic Gradients\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eTraining neural networks typically requires pushing the computation graph forward and then backpropagating errors to update the weights. In this sense, all layers of the network are locked, because they must wait for other parts of the network to complete their forward inference and backward pass before they can be updated. In this work, we introduce models of future computation in the network graph, in order to decouple modules and thereby break this constraint. These models use only local information to predict the outcome of a subgraph. In particular, we focus on modeling the error gradients: by replacing the true backpropagated error gradients with modeled synthetic gradients, subgraphs are decoupled and can be updated independently and asynchronously—namely, we realize decoupled neural interfaces. We present results for feed-forward models in which every layer is trained asynchronously; for RNNs, where predicting the future gradients of one layer extends the effective modeling horizon of the RNN; and for hierarchical RNN systems with different timescales. Finally, we show that, beyond predicting gradients, the same framework can also be used to predict inputs, resulting in models that are decoupled in both the forward and backward passes—equivalent to two independent networks that can learn together so as to be composed into a single functional network.\u003c/p\u003e","title":"Decoupled Neural Interfaces using Synthetic Gradients"},{"content":"Stagewise Knowledge Distillation Abstract Most modern deep learning models require high computational power, but embedded devices lack such computing capability. Therefore, for this kind of device, models that can reduce computation while maintaining performance are very important. Knowledge distillation is one of the methods that solves this kind of problem. Traditional knowledge distillation methods directly transfer knowledge from the teacher to the student in a single stage. We propose a stagewise training approach to improve knowledge transfer. This approach can even use only a portion of the data used to train the teacher model without affecting performance. This approach can complement other model compression techniques, and can even be regarded as a universal model compression technique.\nIntroduction This paper is mainly about model compression techniques based on knowledge distillation, so let me first introduce the categories of model compression.\nModel compression can mainly be divided into the following five categories:\nParameter Pruning and Sharing: mainly aims to reduce redundancy in network parameters and eliminate unnecessary parameters. Low Rank Factorization techniques: mainly use tensor/matrix factorization to determine the effective parameters of the network. Transferred/Compact Convolutional Filters: aim to use specially designed convolutional filters to reduce computation and storage space. Knowledge Distillation: aims to use a larger pretrained model, the teacher, to train a small model, the student. Quantization: aims to reduce the number of bits of each weight while preserving network performance. This work focuses mainly on the knowledge distillation approach. Ideally, the teacher should be able to pass on all the knowledge it has learned to the student, but this is not the case. Moreover, not all of the teacher\u0026rsquo;s knowledge is necessarily relevant to the student; the ideal situation is that the student learns the important parts and ignores the unimportant ones. This paper mainly uses ResNet34 as the teacher model, and the student model also uses a ResNet-like model, but is much smaller in storage structure and computational complexity. This paper presents a method of training the student using multiple feature maps of a pretrained teacher model.\nThis paper mainly adopts a new way of training: the student model is trained using the teacher model\u0026rsquo;s feature maps fixed at a particular layer. For each feature map, the student model is trained in a stagewise manner, and the final classification layer is trained directly on the dataset without the teacher. It is finally demonstrated that this approach can learn directly on a subset of the teacher\u0026rsquo;s training data.\nRelated Work Methodology This paper mainly uses ResNet networks; the specific structure of ResNet will not be repeated here. It mainly consists of:\nBasic Block Downsample Block ResNet18 or 34 type models Teacher Network This paper uses ResNet34 as the teacher model.\nStudent Network A reduced version of ResNet34 is used as the student model, mainly by reducing the number of basic blocks.\nDataset Three datasets are used: Imagenette, Imagewoof and CIFAR10. The first two datasets are subsets of ImageNet. The first is relatively easy, and the latter is relatively harder. The purpose of this work is not to improve accuracy as much as possible, but to make the student\u0026rsquo;s accuracy as close as possible to the teacher\u0026rsquo;s.\nProposed Training Method In the early stage of the experiments, we trained the student model\u0026rsquo;s multiple feature maps to simultaneously mimic the corresponding feature maps of the teacher model and the labels. So the mean squared error of each pair of feature maps would be accumulated. In addition, the cross-entropy loss is also accumulated. Therefore the total loss function can be expressed as:\nL(y,y^,ycls,class)=1MN∑i=1N∑j=1M(y(i,j)−y^(i,j))2+1MC∑j=1M∑k=1C{−log⁡(exp⁡(ycls(j, class ))∑k=1Cexp⁡(ycls(j,k)))} \\begin{aligned} L\\left(y, \\hat{y}, y_{c l s}, \\mathrm{class}\\right) \u0026amp;=\\frac{1}{M N} \\sum_{i=1}^{N} \\sum_{j=1}^{M}(y(i, j)-\\hat{y}(i, j))^{2} \\\\ \u0026amp;+\\frac{1}{M C} \\sum_{j=1}^{M} \\sum_{k=1}^{C}\\left\\{-\\log \\left(\\frac{\\exp \\left(y_{c l s}(j, \\text { class })\\right)}{\\sum_{k=1}^{C} \\exp \\left(y_{c l s}(j, k)\\right)}\\right)\\right\\} \\end{aligned} N denotes the number of blocks, and y(i,j) is the intermediate output of the teacher model\u0026rsquo;s i-th block for the j-th input. Likewise, y-hat is that of the student model. M is the batch size. y_cls(j,k) is the model\u0026rsquo;s output for the j-th input with respect to the k-th class, C is the number of classes, and class is the correct class that each particular input represents.\nThese early experiments showed a very small improvement of student models trained with the teacher model over those trained without. This can be attributed to the fact that multiple feature maps and labels must be mimicked at the same time, i.e., the conditions imposed on the optimization algorithm are very strict. Assigning weights to each MSE loss and the cross-entropy loss did not help either, because the training process was still very strict. Another possible reason is gradient vanishing and accumulation. To reduce this strictness of training, a stagewise training approach was proposed.\nWe train the student model in a stagewise manner, i.e., one block at a time. The image is the input to both the teacher and the student model, and the outputs of the first block are taken from both models. The MSE error is applied between the outputs, and then backpropagation is performed on the student model. After training the first block for 100 epochs, training stops. In the next step, the input is again passed to the teacher and the student, but the features of the second block are taken, and the same procedure as in the first stage is followed, i.e., the MSE loss between the outputs of the second block. Then backpropagation for 100 epochs. This operation is repeated for all the blocks. At the end of the student model, the classifier part is trained directly to predict classes from the dataset, i.e., the images are passed through the student model and trained with the cross-entropy loss for class prediction. At this stage, no teacher model is used, and the rest of the student model (the parts of the model other than the classifier part) is frozen. Figure 2 can help understand this point. The training loss function of stage i can be expressed as:\nLi(y,y^)=1M∑j=1M(y(i,j)−y^(i,j))2 L_{i}(y, \\hat{y})=\\frac{1}{M} \\sum_{j=1}^{M}(y(i, j)-\\hat{y}(i, j))^{2} The classifier uses the standard cross-entropy loss:\nLcls(ycls,class)=1MC∑j=1M∑k=1C{−log⁡(exp⁡(ycls(j,class)∑k=1Cexp⁡(ycls(j,k)))} L_{c l s}\\left(y_{c l s}, \\mathrm{class}\\right)=\\frac{1}{M C} \\sum_{j=1}^{M} \\sum_{k=1}^{C}\\left\\{-\\log \\left(\\frac{\\exp \\left(y_{c l s}(j, \\mathrm{class}\\right)}{\\sum_{k=1}^{C} \\exp \\left(y_{c l s}(j, k)\\right)}\\right)\\right\\} We have shown that stagewise training has its own advantages; the main advantage is that the number of parameters to be optimized at a time is limited. Compared with training a larger number of parameters at once, this limited number of parameters can relax the strictness during training. The results show that stagewise training works better than training everything at once.\nLess Data Approach Datasets like ImageNet are so large that performing stagewise training of the student model with the teacher model on limited hardware would take a great deal of time. Therefore, it becomes useful if we can perform stagewise training using only a subset of the data while maintaining accuracy. Thus, the stagewise training experiments were repeated using 1/4 of the original training data. Note that the original training data refers to the data on which the teacher model was trained, and the remaining 3/4 of the data is kept as a test set for evaluation.\nResults Figures 3 and 4 give the results of simultaneous training, stagewise training, and stagewise training with partial data.\nThese figures show that the student model with the entire dataset achieves almost the same accuracy as the teacher. But in the case of less data, there is a huge gap in accuracy between training with and without a teacher. The following paragraphs discuss the possible reasons behind these results. It should be pointed out that the purpose of model compression is to reduce the gap between teacher and student, not to obtain better accuracy. Obviously, if a better teacher is used, the student\u0026rsquo;s accuracy will improve; sometimes the student\u0026rsquo;s accuracy even exceeds that of the teacher model.\nThe experimental results can be explained as follows. Since the teacher has already learned the complete dataset, it has already learned the features necessary for classifying the entire dataset. When this teacher is used to train the student, even if training uses a small dataset, its \u0026ldquo;knowledge\u0026rdquo; is passed on to the student. Using less data can also be justified by the number of parameters that must be trained in a single stage. Since the proposed method trains only a small part of the network at a time, the number of parameters to be optimized is much smaller than that of the full network mentioned in the preceding paragraphs. The results show that this approach greatly improves accuracy: without a teacher, the accuracy of the student network trained on a small dataset is much lower than that of the student trained on the same dataset with the proposed method. Of course, the main advantage is the reduction in training time, which is very important, because without the teacher, stagewise training would take N+1 times as long. (Because each stage trains the same number of epochs separately.) Here N denotes the number of stages.\nFigure 4 shows the experimental results using a smaller amount of data. It can be seen that when using less data and training independently, the student performs very poorly. On the other hand, if a teacher trained on less data is used to train the student stagewise, the prediction accuracy improves greatly. This capability would be very useful when applied to very large datasets.\nThe results of simultaneous training are close to those of training without a teacher. In particular, for the two ImageNet subsets, simultaneous training is slightly better, but for CIFAR10 it is slightly worse. This again demonstrates that the conditions of simultaneous training are too strict, and there is no obvious advantage over training without a teacher. However, the results of stagewise training are much better than those of both simultaneous training and training without a teacher. Since simultaneous training on the complete dataset did not give optimistic results, it was not performed on smaller datasets.\nConclusion This work presents a novel method for transferring knowledge from one network to another. Since the number of parameters optimized in one stage is reduced, the method performs better than transferring knowledge directly with the full network all at once. This also allows the student network to be trained with less data than the teacher. It will be very useful when training on larger datasets such as ImageNet.\nIn addition, the method is very flexible and can be used together with other model compression techniques and with other models. At the same time, it is not limited to image classification and can also be used in applications such as object detection and image segmentation. It can be regarded as a universal compression technique.\n","permalink":"https://blog.ereebay.me/en/posts/skd/","summary":"\u003ch1 id=\"stagewise-knowledge-distillation\"\u003eStagewise Knowledge Distillation\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eMost modern deep learning models require high computational power, but embedded devices lack such computing capability. Therefore, for this kind of device, models that can reduce computation while maintaining performance are very important. Knowledge distillation is one of the methods that solves this kind of problem. Traditional knowledge distillation methods directly transfer knowledge from the teacher to the student in a single stage. We propose a stagewise training approach to improve knowledge transfer. This approach can even use only a portion of the data used to train the teacher model without affecting performance. This approach can complement other model compression techniques, and can even be regarded as a universal model compression technique.\u003c/p\u003e","title":"Stagewise Knowledge Distillation"},{"content":"Federated Meta-Learning with Fast Convergence and Efficient Communication Abstract This paper proposes a federated meta-learning framework, FedMeta, which shares a parameterized algorithm (meta learner) instead of the previous global model. It is evaluated on the LEAF datasets and a real-world dataset, and demonstrates that the communication cost required by FedMeta is reduced by 2.82-4.33 times, that it converges faster, and that accuracy is even improved by 3.23~14.84 percentage points compared with FedAvg. Moreover, FedMeta preserves user privacy, since only the parameterized algorithm is shared and no data.\nIntroduction The well-known FedAvg algorithm in federated learning can flexibly use SGD for local training, achieving high accuracy while balancing computation and communication costs.\nIn meta-learning, initialization-based meta-learning algorithms such as MAML excel at fast convergence on new tasks and possess good generalization. This makes them well suited to decentralized data that is Non-IID and highly personalized.\nInspired by this, the paper develops a federated meta-learning framework that differs greatly from previous federated learning work. It first connects meta-learning methods with federated learning. In meta-learning, a parameterized algorithm (meta learner) is slowly learned from a large number of tasks through the meta-training process, during which the algorithm quickly trains a specific model within each task.\nA task usually consists of a support set and a query set, which are disjoint. The task-specific model is trained on the support set and tested on the query set. The test results are then used to update the algorithm. In federated meta-learning, the algorithm (meta learner) is maintained on the server and distributed to the clients for model training. During each episode of meta-training, a sampled batch of clients receives the parameters of the algorithm (meta learner) and trains their models. The test results on the query sets are then uploaded to the server to update the meta learner. The overall flow is shown in the figure:\nComparing federated meta-learning with federated learning A comparison of FML and FL. Federated meta-learning is similar to federated learning; the main difference is that what is shared is no longer a global model but the parameters of an algorithm (the meta-model parameters). Moreover, meta-learning is conceptually different from distributed model training, and sharing a meta-learning algorithm can be applied more flexibly than sharing a model. For example, in image classification, images of n classes may be unevenly distributed across clients, with each client having at most k classes. Federated learning would need to train a large n-class classifier in order to exploit data from all clients, whereas in fact a k-class classifier suffices, since it only ever makes predictions for a single client. Such a large model in federated learning entails substantial communication cost. One could send only part of the model to a client to update the relevant parameters, but this requires prior knowledge of the client\u0026rsquo;s private data. Meta-learning, on the other hand, allows an algorithm to train on tasks containing different classes. For example, MAML can provide an initialization for a k-class classifier by meta-training on k-class tasks, regardless of what the specific classes are. Therefore, within the FML framework, MAML can be used to meta-train initializations of k-class classifiers over all n classes—that is, letting a k-class classifier be meta-learned for initialization in the context of tasks drawn from n classes. This reduces both the communication and computation costs of FML.\nContributions The contributions of this paper focus on algorithm design for the federated learning setting; to this end, the paper proposes a new framework and carries out extensive experiments. The contributions are threefold: 1. Proposing the FedMeta framework, which combines meta-learning and federation, integrating MAML and Meta-SGD into federated learning. 2. Conducting experiments on the LEAF datasets with comparisons against FedAvg; the results show that FedMeta reduces overhead while achieving higher accuracy. 3. Applying FedMeta to a recommendation task in which each client has highly personalized records; the experiments demonstrate that meta-learning algorithms achieve higher accuracy than standalone or federated learning methods.\nFederated Meta-Learning The Meta-Learning Approach The goal of meta-learning is to meta-train an algorithm (meta learner) A that can train a model quickly—that is, to obtain an initial model that converges fast. The algorithm AφA_{\\varphi} is a parameterized model whose parameters are updated across a series of tasks during meta-training. During meta-training, a task T consists of a support set and a query set, each with corresponding labels and data. The algorithm (meta learner) A trains the model f on the support set and outputs the parameters θT\\theta_T—this is the inner update. The model fθTf_{\\theta_T} is then evaluated on the query set, and the test loss LDQT(θT)\\mathcal{L}_{D_{Q}^{T}}\\left(\\theta_{T}\\right) is computed to reflect how well algorithm A trains. Finally, A is updated by minimizing the test loss—this step is called the outer update. Note that the query set and the support set are disjoint, so as to maximize the generalization ability of A. During meta-training, each episode samples a batch of tasks from a meta-training set. The optimization objective of algorithm A can thus be expressed as:\nmin⁡φET∼T[LDQT(θT)]=min⁡φET∼T[LDQT(Aφ(DST))] \\min _{\\varphi} \\mathbb{E}_{T \\sim \\mathcal{T}}\\left[\\mathcal{L}_{D_{Q}^{T}}\\left(\\theta_{T}\\right)\\right]=\\min _{\\varphi} \\mathbb{E}_{T \\sim \\mathcal{T}}\\left[\\mathcal{L}_{D_{Q}^{T}}\\left(\\mathcal{A}_{\\varphi}\\left(D_{S}^{T}\\right)\\right)\\right] MAML is a representative meta-learning algorithm. For MAML, algorithm A is what produces the model\u0026rsquo;s initial state. Concretely, for each task T, the algorithm sets α=θ\\alpha = \\theta, making the algorithm\u0026rsquo;s parameters equal to the parameters of model f. The parameters of model f are then trained on the support set and updated according to the loss: LDST(θ):=1∣DST∣∑(x,y)∈DSTℓ(fθ(x),y)\\mathcal{L}_{D_{S}^{T}}(\\theta):=\\frac{1}{\\left|D_{S}^{T}\\right|} \\sum_{(x, y) \\in D_{S}^{T}} \\ell\\left(f_{\\theta}(x), y\\right). Finally, the model parameters are tested on the query set, and the test loss is computed: LDQT(θT):=1∣DQT∣∑(x′,y′)∈DQTℓ(fθT(x′),y′)\\mathcal{L}_{D_{Q}^{T}}\\left(\\theta_{T}\\right):=\\frac{1}{\\left|D_{Q}^{T}\\right|} \\sum_{\\left(x^{\\prime}, y^{\\prime}\\right) \\in D_{Q}^{T}} \\ell\\left(f_{\\theta_{T}}\\left(x^{\\prime}\\right), y^{\\prime}\\right)\nEquation 1 can be simplified as:\nmin⁡θET∼T[LDQT(θ−α∇LDST(θ))] \\min _{\\theta} \\mathbb{E}_{T \\sim \\mathcal{T}}\\left[\\mathcal{L}_{D_{Q}^{T}}\\left(\\theta-\\alpha \\nabla \\mathcal{L}_{D_{S}^{T}}(\\theta)\\right)\\right] Building on MAML, Meta-SGD further learns the initial parameters and the inner-loop learning rate at the same time. The test loss can be viewed as a function of both the model parameters and the learning rate, and both can be updated by taking gradients with SGD in the outer loop. Moreover, the learning rate is a vector of the same dimension as the model parameters, so that the learning rate corresponds to the model parameter vector coordinate-wise.\nTherefore the optimization condition can be rewritten as\nmin⁡θ,αET∼T[LDQT(θ−α∘∇LDST(θ))] \\min _{\\theta, \\alpha} \\mathbb{E}_{T \\sim \\mathcal{T}}\\left[\\mathcal{L}_{D_{Q}^{T}}\\left(\\theta-\\alpha \\circ \\nabla \\mathcal{L}_{D_{S}^{T}}(\\theta)\\right)\\right] The Federated Meta-Learning Framework The goal of FML is to meta-train an algorithm collaboratively from data distributed across clients. Taking MAML as an example, the aim is to use the data of all clients to complete the initialization of a model. MAML involves two levels of optimization: the inner loop trains a task-specific model from the initial parameters, and the outer loop updates the initial parameters according to the test loss. In the federated learning setting, each client u receives the initial parameters θ\\theta from the server, then trains the model on the support-set data on its device and transmits the test loss to the server. The server mainly maintains the initial parameters and updates them according to the clients\u0026rsquo; test losses.\nThe information transmitted in this process includes: the model\u0026rsquo;s initial parameters (from server to clients) and the test loss (from clients to server). For the Meta-SGD algorithm, the vector α\\alpha is also part of the transmission, serving as the learning-rate parameters for inner-loop model training.\nAlgorithm 1 describes the FedMeta procedure using MAML and Meta-SGD, where each communication round corresponds to an episode. The algorithm is maintained in the AlgorithmUpdate step. In each round of updates, the server collects the test losses obtained by training MAML or Meta-SGD on the sampled clients. The initial parameters θ\\theta are updated after training on the clients\u0026rsquo; training sets, and the updated parameters are then tested on the test set. After the meta-training process, the model is deployed on the clients.\nExperiments Experiments on the LEAF datasets demonstrate fast convergence and high accuracy. Experiments on a recommendation task in a real-world setting show that strong adaptability is maintained even at smaller scales. Evaluation Scheme In all experiments, 80% of the clients serve as training clients, 10% as validation clients, and the remainder as test clients. Each client\u0026rsquo;s local data is split into a support set and a query set. The paper also varies the fraction p of each client\u0026rsquo;s data used as the support set, in order to study how effectively FedMeta adapts to new clients with limited data; this is abbreviated psupport hereafter.\nThe authors ran experiments with FedAvg, a meta version of FedAvg, and two variants of FedMeta. The meta version of FedAvg uses the support sets of the test clients to finetune the initial model received from the server before testing, whereas during training both use all of the data on the training clients.\nAs for FML, three methods were tested—MAML, FOMAML, and Meta-SGD—all of which are model-agnostic and easy to implement.\nLEAF Datasets Accuracy and Convergence Comparison Considering the limited computational capabilities of edge devices, each client\u0026rsquo;s local epoch is set to 1.\nAs shown above, all methods within the FedMeta framework converge faster and more stably, achieving clear improvements. MAML and Meta-SGD reach similar convergence speeds and accuracy on the first two datasets; on Sent140, Meta-SGD performs somewhat better.\nThe table shows the accuracy of the four methods after several communication rounds. First, it can be seen that FedAvg performs far worse than FedMeta, especially on the image classification task. MAML and Meta-SGD achieve the highest accuracy, with gains of 3.23-14.84 percentage points. It is also found that FedAvg (meta) attains higher accuracy than FedAvg in most cases; there are two exceptions, however, when the support fraction is 20%. Unexpectedly, FedAvg (Meta) even shows a slight drop in accuracy. This may be because, after being finetuned on a small amount of data, the model deviates excessively from the global optimum. Second, as we increase the probability p of the support set, the accuracy of both FedAvg (Meta) and FedMeta improves in almost all cases, but the growth rate of FedAvg (meta) is greater than that of FedMeta. This indicates that FedMeta generalizes better and adapts more effectively to clients with limited data.\nFairness Comparison The paper also compares FedAvg and FedMeta across multiple runs, using the distribution of final accuracies. The last row of the previous figure shows kernel density estimates for the different methods. For FEMNIST, MAML and Meta-SGD not only yield a higher mean but also lower variance. For the Shakespeare dataset, FedMeta has larger variance but also a larger mean. For Sent140, the accuracy distributions are roughly the same; still, it can be seen that MAML and Meta-SGD have more clients close to 100%. Overall, for image classification tasks, FedMeta\u0026rsquo;s accuracy distribution is more stable and concentrated.\nConclusion The FedMeta framework outperforms the original FedAvg in accuracy, convergence speed, and communication cost.\n","permalink":"https://blog.ereebay.me/en/posts/fmfe/","summary":"\u003ch1 id=\"federated-meta-learning-with-fast-convergence-and-efficient-communication\"\u003eFederated Meta-Learning with Fast Convergence and Efficient Communication\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThis paper proposes a federated meta-learning framework, FedMeta, which shares a parameterized algorithm (meta learner) instead of the previous global model. It is evaluated on the LEAF datasets and a real-world dataset, and demonstrates that the communication cost required by FedMeta is reduced by 2.82-4.33 times, that it converges faster, and that accuracy is even improved by 3.23~14.84 percentage points compared with FedAvg. Moreover, FedMeta preserves user privacy, since only the parameterized algorithm is shared and no data.\u003c/p\u003e","title":"Federated Meta-Learning with Fast Convergence and Efficient Communication"},{"content":"CS330 lecture 1\u0026amp;2 notes Informal Problem Definitions The multi-task learning problem: Learn all of the tasks more quickly or more proficiently than learning them independently. The meta-learning problem: Given data/experience on previous tasks, learn a new task more quickly and/or more proficiently. Multi-Task Learning Basics Traditional single-task learning:\nD={(x,y)k}min⁡θL(θ,D) \\begin{array}{l}{\\mathscr{D}=\\left\\{(\\mathbf{x}, \\mathbf{y})_{k}\\right\\}} \\\\ {\\min _{\\theta} \\mathscr{L}(\\theta, \\mathscr{D})}\\end{array} Typical loss: negative log likelihood\nL(θ,D)=−E(x,y)∼D[log⁡fθ(y∣x)] \\mathscr{L}(\\theta, \\mathscr{D})=-\\mathbb{E}_{(x, y) \\sim \\mathscr{D}}\\left[\\log f_{\\theta}(\\mathbf{y} | \\mathbf{x})\\right] What\u0026rsquo;s a task? A task: Ti≜{pi(x),pi(y∣x),Li}\\mathscr{T}_{i} \\triangleq\\left\\{p_{i}(\\mathbf{x}), p_{i}(\\mathbf{y} | \\mathbf{x}), \\mathscr{L}_{i}\\right\\}\ndata generating distributions\nHere a task is defined as the distribution over data samples, the distribution over data labels, and a loss function.\nCorresponding datasets: Ditr\\mathscr{D}_{i}^{tr} training set, Ditst\\mathscr{D}_{i}^{t s t} test set.\nUsually Di\\mathscr{D}_{i} denotes the training set.\nMulti-task classification: Li\\mathscr{L}_{i} same across all tasks. E.g., in handwritten character recognition across different languages, the form of the loss function may be the same.\nMulti-label learning: Li,pi(x)\\mathscr{L}_{i}, {p}_{i}(x) same across all tasks. E.g., in the CelebA multi-label recognition task, the samples and the loss function are identical.\nThe loss function may vary across tasks in the following cases:\nmixed discrete, continuous labels across tasks caring more about one task than another (i.e., different weights for different tasks?) Conditioning on the task The multi-task learning problem requires introducing a task descriptor as a variable that describes the task; the question is how to design this variable.\nAssume zi{z}_{i} is the task index. The most straightforward approach is multiplicative gating, which effectively trains each task in the multi-task setting with its own separate network, without sharing parameters.\nThe other extreme is to directly concat ziz_i, in which case all parameters are shared except those that come after the input ziz_i.\nYet another idea is to split θ\\theta into shared parameters θsh\\theta^{sh} and task-specific parameters θi\\theta^i — i.e., shared and non-shared parameters.\nThe optimization objective then becomes\nmin⁡θsh,θ1,…,θT∑i=1TLi({θsh,θi},Di) \\min _{\\theta^{s h}, \\theta^{1}, \\ldots, \\theta^{T}} \\sum_{i=1}^{T} \\mathscr{L}_{i}\\left(\\left\\{\\theta^{s h}, \\theta^{i}\\right\\}, \\mathscr{D}_{i}\\right) The problem then becomes which parameters to share and when.\nCommon Choices The common choices are mainly concatenation and addition — the figures make them clear at a glance.\nConcatenation-based conditioning Additive conditioning In fact, the two are equivalent.\nMulti-head architecture Multiplicative conditioning The multiplicative approach offers:\nstronger expressive power multiplication gating for regression tasks better generalization across independent networks and heads Complex Choices There are also many other more complex choices.\nBut where the design inspiration comes from is just like choosing the hyperparameters of a neural network:\ndifferent problems are independent of one another for any specific problem, it mostly relies on the designer\u0026rsquo;s intuition and background knowledge current approaches are more art than science Optimizing the objective Objective: min⁡θ∑i=1TLi(θ,Di)\\min _{\\theta} \\sum_{i=1}^{T} \\mathscr{L}_{i}\\left(\\theta, \\mathscr{D}_{i}\\right)\nThe typical procedure:\nSample a minibatch of tasks B∼{Ti}\\mathscr{B} \\sim\\left\\{\\mathscr{T}_{i}\\right\\} Sample a minibatch of data from each task Dib∼Di\\mathscr{D}_{i}^{b} \\sim \\mathscr{D}_{i} Compute the loss on each minibatch-task: L^(θ,B)=∑Tk∈BLk(θ,Dkb)\\hat{\\mathscr{L}}(\\theta, \\mathscr{B})=\\sum_{\\mathcal{T}_{k} \\in \\mathscr{B}} \\mathscr{L}_{k}\\left(\\theta, \\mathscr{D}_{k}^{b}\\right) Backpropagate to compute gradients ∇θL^\\nabla_{\\theta} \\hat{\\mathscr{L}} Update the gradients with your favorite optimizer Note: this ensures that tasks are sampled uniformly regardless of their data size.\nTip: for regression tasks, make sure task labels are on the same scale.\nChallenge Negative transfer Multi-task training on CIFAR-100 performs worse than training tasks independently.\nPossible causes:\noptimization challenges interference between different tasks different learning rates across tasks limited expressive capacity multi-task networks are large Solution:\nshare less across tasks (soft parameter sharing)\nallows for more fluid degrees of parameter sharing (advantage) yet another set of design decisions/hyperparameters (drawback) min⁡θsh,θ1,…,θT∑i=1TLi({θsh,θi},Di)+∑t=1T∥θt−θt′∥ \\min _{\\theta^{sh}, \\theta^{1}, \\ldots, \\theta^{T}} \\sum_{i=1}^{T} \\mathscr{L}_{i}\\left(\\left\\{\\theta^{s h}, \\theta^{i}\\right\\}, \\mathscr{D}_{i}\\right)+\\sum_{t=1}^{T}\\left\\|\\theta^{t}-\\theta^{t\u0026#x27;}\\right\\| The latter term is soft parameter sharing: the difference between one task\u0026rsquo;s parameters and the previous one is used as a regularization term, which effectively makes each task\u0026rsquo;s parameters as similar as possible — i.e., the parameters are shared.\nOverfitting Overfitting is usually caused by not sharing enough parameters; the solution is to share more. Intuitively, insufficient sharing makes each task overfit, which resembles independent training.\nMeta-Learning Basics Two views of meta-learning:\nMechanistic view a deep neural network that can take in an entire dataset and make predictions on new data the network is trained on a meta-dataset that contains different datasets for different tasks this view makes it easy to implement a meta-learning algorithm Probabilistic view extract prior knowledge from a series of meta-learning tasks use a small amount of data plus prior information to infer a relatively effective posterior this view leads to a better understanding of meta-learning algorithms Problem definitions First, recall supervised learning:\nExisting issues:\nrequires a large amount of labeled data labels are very limited for some tasks nowadays To be continued\n","permalink":"https://blog.ereebay.me/en/posts/cs330-1/","summary":"\u003ch1 id=\"cs330-lecture-12-notes\"\u003eCS330 lecture 1\u0026amp;2 notes\u003c/h1\u003e\n\u003ch2 id=\"informal-problem-definitions\"\u003eInformal Problem Definitions\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eThe multi-task learning problem: Learn all of the tasks more quickly or more proficiently than learning them independently.\u003c/li\u003e\n\u003cli\u003eThe meta-learning problem: Given data/experience on previous tasks, learn a new task more quickly and/or more proficiently.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"multi-task-learning-basics\"\u003eMulti-Task Learning Basics\u003c/h2\u003e\n\u003cp\u003eTraditional single-task learning:\u003c/p\u003e\n\u003cspan class=\"katex\"\u003e\u003cmath xmlns=\"http://www.w3.org/1998/Math/MathML\"\u003e\u003csemantics\u003e\u003cmtable rowspacing=\"0.16em\" columnalign=\"left\" columnspacing=\"1em\"\u003e\u003cmtr\u003e\u003cmtd\u003e\u003cmstyle scriptlevel=\"0\" displaystyle=\"false\"\u003e\u003cmrow\u003e\u003cmi mathvariant=\"script\"\u003eD\u003c/mi\u003e\u003cmo\u003e=\u003c/mo\u003e\u003cmrow\u003e\u003cmo fence=\"true\"\u003e{\u003c/mo\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi mathvariant=\"bold\"\u003ex\u003c/mi\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmi mathvariant=\"bold\"\u003ey\u003c/mi\u003e\u003cmsub\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmi\u003ek\u003c/mi\u003e\u003c/msub\u003e\u003cmo fence=\"true\"\u003e}\u003c/mo\u003e\u003c/mrow\u003e\u003c/mrow\u003e\u003c/mstyle\u003e\u003c/mtd\u003e\u003c/mtr\u003e\u003cmtr\u003e\u003cmtd\u003e\u003cmstyle scriptlevel=\"0\" displaystyle=\"false\"\u003e\u003cmrow\u003e\u003cmsub\u003e\u003cmrow\u003e\u003cmi\u003emin\u003c/mi\u003e\u003cmo\u003e⁡\u003c/mo\u003e\u003c/mrow\u003e\u003cmi\u003eθ\u003c/mi\u003e\u003c/msub\u003e\u003cmi mathvariant=\"script\"\u003eL\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi\u003eθ\u003c/mi\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmi mathvariant=\"script\"\u003eD\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003c/mrow\u003e\u003c/mstyle\u003e\u003c/mtd\u003e\u003c/mtr\u003e\u003c/mtable\u003e\u003cannotation encoding=\"application/x-tex\"\u003e\n\\begin{array}{l}{\\mathscr{D}=\\left\\{(\\mathbf{x}, \\mathbf{y})_{k}\\right\\}} \\\\ {\\min _{\\theta} \\mathscr{L}(\\theta, \\mathscr{D})}\\end{array}\n\u003c/annotation\u003e\u003c/semantics\u003e\u003c/math\u003e\u003c/span\u003e\u003cp\u003eTypical loss: negative log likelihood\u003c/p\u003e\n\u003cspan class=\"katex\"\u003e\u003cmath xmlns=\"http://www.w3.org/1998/Math/MathML\"\u003e\u003csemantics\u003e\u003cmrow\u003e\u003cmi mathvariant=\"script\"\u003eL\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi\u003eθ\u003c/mi\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmi mathvariant=\"script\"\u003eD\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmo\u003e=\u003c/mo\u003e\u003cmo\u003e−\u003c/mo\u003e\u003cmsub\u003e\u003cmi mathvariant=\"double-struck\"\u003eE\u003c/mi\u003e\u003cmrow\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi\u003ex\u003c/mi\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmi\u003ey\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmo\u003e∼\u003c/mo\u003e\u003cmi mathvariant=\"script\"\u003eD\u003c/mi\u003e\u003c/mrow\u003e\u003c/msub\u003e\u003cmrow\u003e\u003cmo fence=\"true\"\u003e[\u003c/mo\u003e\u003cmi\u003elog\u003c/mi\u003e\u003cmo\u003e⁡\u003c/mo\u003e\u003cmsub\u003e\u003cmi\u003ef\u003c/mi\u003e\u003cmi\u003eθ\u003c/mi\u003e\u003c/msub\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi mathvariant=\"bold\"\u003ey\u003c/mi\u003e\u003cmi mathvariant=\"normal\"\u003e∣\u003c/mi\u003e\u003cmi mathvariant=\"bold\"\u003ex\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmo fence=\"true\"\u003e]\u003c/mo\u003e\u003c/mrow\u003e\u003c/mrow\u003e\u003cannotation encoding=\"application/x-tex\"\u003e\n\\mathscr{L}(\\theta, \\mathscr{D})=-\\mathbb{E}_{(x, y) \\sim \\mathscr{D}}\\left[\\log f_{\\theta}(\\mathbf{y} | \\mathbf{x})\\right]\n\u003c/annotation\u003e\u003c/semantics\u003e\u003c/math\u003e\u003c/span\u003e\u003ch3 id=\"whats-a-task\"\u003eWhat\u0026rsquo;s a task?\u003c/h3\u003e\n\u003cp\u003eA task: \u003cspan class=\"katex\"\u003e\u003cmath xmlns=\"http://www.w3.org/1998/Math/MathML\"\u003e\u003csemantics\u003e\u003cmrow\u003e\u003cmsub\u003e\u003cmi mathvariant=\"script\"\u003eT\u003c/mi\u003e\u003cmi\u003ei\u003c/mi\u003e\u003c/msub\u003e\u003cmo\u003e≜\u003c/mo\u003e\u003cmrow\u003e\u003cmo fence=\"true\"\u003e{\u003c/mo\u003e\u003cmsub\u003e\u003cmi\u003ep\u003c/mi\u003e\u003cmi\u003ei\u003c/mi\u003e\u003c/msub\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi mathvariant=\"bold\"\u003ex\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmsub\u003e\u003cmi\u003ep\u003c/mi\u003e\u003cmi\u003ei\u003c/mi\u003e\u003c/msub\u003e\u003cmo stretchy=\"false\"\u003e(\u003c/mo\u003e\u003cmi mathvariant=\"bold\"\u003ey\u003c/mi\u003e\u003cmi mathvariant=\"normal\"\u003e∣\u003c/mi\u003e\u003cmi mathvariant=\"bold\"\u003ex\u003c/mi\u003e\u003cmo stretchy=\"false\"\u003e)\u003c/mo\u003e\u003cmo separator=\"true\"\u003e,\u003c/mo\u003e\u003cmsub\u003e\u003cmi mathvariant=\"script\"\u003eL\u003c/mi\u003e\u003cmi\u003ei\u003c/mi\u003e\u003c/msub\u003e\u003cmo fence=\"true\"\u003e}\u003c/mo\u003e\u003c/mrow\u003e\u003c/mrow\u003e\u003cannotation encoding=\"application/x-tex\"\u003e\\mathscr{T}_{i} \\triangleq\\left\\{p_{i}(\\mathbf{x}), p_{i}(\\mathbf{y} | \\mathbf{x}), \\mathscr{L}_{i}\\right\\}\u003c/annotation\u003e\u003c/semantics\u003e\u003c/math\u003e\u003c/span\u003e\u003c/p\u003e","title":"CS330 Lecture 1\u00262 Study Notes (Incomplete)"},{"content":"A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms Abstract This paper proposes exploiting a meta-learning objective that maximizes the speed of transfer to a changed distribution in order to learn how to acquire knowledge in a modular fashion. In particular, we care about how to factor the joint distribution into appropriate conditionals consistent with the causal relationships. If this is achievable, changes in the distribution are assumed to be localized (distributions are localized) (e.g., one of the marginals changes due to an intervention on one of the variables). We show that, under such localized changes of the assumed causal mechanisms, the correct causal graph will tend to have only a few parameters with non-zero gradients, i.e., the parameters that need to be adjusted (the parameters of the modified variables). This is observed experimentally to lead to faster adaptation, and we exploit this property to define a meta-learning surrogate score which, in addition to a continuous parametrization of the graph, will also favor the correct causal graph. Finally, considering AI agents (e.g., a robot autonomously discovering its environment), we consider how the same objective can discover the causal variables themselves, since the observed low-level variables carry no causal meaning. Experiments on the bivariate case validate the proposed ideas and theoretical results.\nIntroduction Data is assumed to be independent and identically distributed (IID). Likewise, a model\u0026rsquo;s performance is typically evaluated with test samples drawn from the same distribution, on the assumption that they represent how the learned system will be used. Although these assumptions are well analyzed from a statistical point of view, they are hard to apply in many practical situations. For example: a medical diagnosis system trained on historical data from one hospital may perform poorly on patients from another hospital because the distribution has shifted. Ideally, we would like our models to generalize well and to adapt quickly to out-of-distribution data.\nHowever, successfully transferring to a new distribution calls for more data. In this paper, rather than focusing on assumptions about the data distribution itself, we focus on how the data distribution changes (e.g., the change from the training distribution to the transfer distribution may be caused by the actions of some agents). The assumption we care about is that when knowledge is represented in an appropriately modular way, only one or a few modules change — such changes are sparse. This is especially important when the distribution change is caused by the action of one or more agents, because an agent intervenes at a particular place and time, which takes the form of the interventions discussed in the causal literature, where one of the causal variables is clamped to a particular value or to a random variable. In general, it is hard for an agent to affect more than one of the underlying causal variables at once. Although this paper is not about agent learning per se, this is a property we explore to help discover these variables and the causal relationships between them. In this respect, the causal graph is a powerful tool, because it tells us how a perturbation of the distribution of an intervened variable will propagate to all the other variables and affect their distributions.\nUsually, the causal structure is not known in advance. The problem of causal discovery is generally needed to obtain the causal graph, but this is usually achievable only under strong assumptions. One such assumption is that a learner that has captured the correct structure of the true underlying data-generating process should generalize to perturbations of that structure in a specific way. This can be illustrated by considering temperature and altitude: roughly speaking, a learner trained on data from Switzerland should still be valid when tested on data drawn from the distribution of a less mountainous country such as the Netherlands. It is thus proposed to use the out-of-distribution robustness of the predictive model to guide inference about the true causal structure.\nSo how can we exploit the assumption of local changes? As we explain theoretically and validate experimentally here, if we have the correct representation of knowledge, then starting from a model pre-trained on the training distribution we should be able to adapt very quickly to the transferred data distribution. This happens because we assume the data-generating process is acquired as a set of independent mechanisms, and moving from the training distribution to the transfer distribution requires changing very few of the mechanisms and parameters. A model corresponding to the correct decomposition of knowledge therefore needs only a few updates and examples to adapt to the transfer distribution. As shown below, the expected gradient on the invariant parameters will be close to 0 (if the model has already been well trained on the training distribution), so the effective search space is greatly reduced during adaptation to the transfer distribution, which speeds up adaptation, as the experiments show. Therefore, based on the assumption of small changes in the space of correct knowledge representations, we can define a meta-learning objective that measures the speed of adaptation — a form of regret — to optimize how knowledge is represented, decomposed, and structured\nBack to the earlier temperature and altitude example: if we receive out-of-distribution data from the Netherlands, we expect the model to adapt faster because a small number of transfer samples from the Netherlands have been collected. Similarly to robustness, adaptation speed can be used to guide inference about the true causal structure of the problem at hand, possibly together with other sources of signal about the causal structure.\nMain contributions: We first validate on synthetic data that, after certain interventions are performed on the true bivariate causal graph (unknown to the learner), a model that correctly captures the underlying causal structure adapts faster when samples are provided. This shows that adaptation speed can serve as a score that adequately assesses how well a learner fits the underlying causal graph. We then use a smooth parametrization of the causal graph to directly optimize this score in an end-to-end manner. Finally, we show that, when the mixing of variables is unknown, this score can be used to distinguish the correct causal variables.\nWhich is Cause and Which is Effect? To be continued\n","permalink":"https://blog.ereebay.me/en/posts/meta/","summary":"\u003ch1 id=\"a-meta-transfer-objective-for-learning-to-disentangle-causal-mechanisms\"\u003eA Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eThis paper proposes exploiting a meta-learning objective that maximizes the speed of transfer to a changed distribution in order to learn how to acquire knowledge in a modular fashion. In particular, we care about how to factor the joint distribution into appropriate conditionals consistent with the causal relationships. If this is achievable, changes in the distribution are assumed to be localized (distributions are localized) (e.g., one of the marginals changes due to an intervention on one of the variables). We show that, under such localized changes of the assumed causal mechanisms, the correct causal graph will tend to have only a few parameters with non-zero gradients, i.e., the parameters that need to be adjusted (the parameters of the modified variables). This is observed experimentally to lead to faster adaptation, and we exploit this property to define a meta-learning surrogate score which, in addition to a continuous parametrization of the graph, will also favor the correct causal graph. Finally, considering AI agents (e.g., a robot autonomously discovering its environment), we consider how the same objective can discover the causal variables themselves, since the observed low-level variables carry no causal meaning. Experiments on the bivariate case validate the proposed ideas and theoretical results.\u003c/p\u003e","title":"Paper Notes: A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms [Unfinished]"},{"content":"Improving Federated Learning Personalization via MAML Abstract FL algorithms share many similarities with MAML and can be interpreted through meta-learning algorithms Fine-tuning gives the global model stronger accuracy while making personalization easier Models trained on standard centralized datasets are harder to personalize than those trained with FedAvg Introduction Points out the connection between FL and MAML, and interprets the FL algorithm via MAML Improves FedAvg with a two-stage approach of training and fine-tuning Finds that FedAvg is essentially a meta-learning algorithm that optimizes personalization performance rather than the global model. Interpreting FedAvg as a Meta Learning Algorithm The figure below shows MAML applied in the FL setting (left), the Reptile algorithm (middle), and FedAvg, the training algorithm of FL (right). Let L be the loss function. In each round of iteration, MAML trains by randomly sampling a batch of tasks T. For each task T there is an inner loop, and the outer loop aggregates the gradient updates obtained from each task. The FL algorithm randomly samples several clients T. For each T and its weight, several rounds of local optimization are performed, and the updated gradients are aggregated into a new global model. If we simplify the setting and assume all clients have the same amount of data, all weights become equal, and Reptile and FedAvg are then effectively the same algorithm.\nAssume the weights in FedAvg are equal, denoted wi. Consider T clients, and let the model parameters be θ\\theta. For each client i, the loss function is Li(θ)L_{i}(\\theta); let gjig_{j}^{i} denote the gradient computed during the jthj^{t h} local training step.\nThe gradient update of FedSGD is:\ngFedSGD=−βT∑i=1T∂Li(θ)∂θ=1T∑i=1Tg1i g_{F e d S G D}=\\frac{-\\beta}{T} \\sum_{i=1}^{T} \\frac{\\partial L_{i}(\\theta)}{\\partial \\theta}=\\frac{1}{T} \\sum_{i=1}^{T} g_{1}^{i} Now let us express FOMAML in the same terms. Assume the client learning rate is β\\beta; after K steps, the personalized model of client i obtains the updated parameters θKi=UKi(θ)=θ−β∑j=1Kgji=θ−β∑j=1K∂Li(θj)∂θ\\theta_{K}^{i}=U_{K}^{i}(\\theta)=\\theta-\\beta \\sum_{j=1}^{K} g_{j}^{i}=\\theta-\\beta \\sum_{j=1}^{K} \\frac{\\partial L_{i}\\left(\\theta_{j}\\right)}{\\partial \\theta}\nDifferentiating gives:\n∂UKi(θ)∂θ=I−β∂∑j=1Kgji∂θ=I−β∑j=1K∂2Li(θj)∂θ2 \\frac{\\partial U_{K}^{i}(\\theta)}{\\partial \\theta}=I-\\beta \\frac{\\partial \\sum_{j=1}^{K} g_{j}^{i}}{\\partial \\theta}=I-\\beta \\sum_{j=1}^{K} \\frac{\\partial^{2} L_{i}\\left(\\theta_{j}\\right)}{\\partial \\theta^{2}} After K gradient updates, the whole model is updated:\ngMAML=∂LMAML∂θ=1T∑i=1T∂Li(UKi(θ))∂θ=1T∑i=1TLi′(UKi(θ))(I−β∑j=1K∂2Li(θj)∂θ2) g_{M A M L}=\\frac{\\partial L_{M A M L}}{\\partial \\theta}=\\frac{1}{T} \\sum_{i=1}^{T} \\frac{\\partial L_{i}\\left(U_{K}^{i}(\\theta)\\right)}{\\partial \\theta}=\\frac{1}{T} \\sum_{i=1}^{T} L_{i}^{\\prime}\\left(U_{K}^{i}(\\theta)\\right)\\left(I-\\beta \\sum_{j=1}^{K} \\frac{\\partial^{2} L_{i}\\left(\\theta_{j}\\right)}{\\partial \\theta^{2}}\\right) To avoid the computational cost of second-order derivatives, FOMAML was introduced: after K gradient updates, it directly uses the (K+1)-th gradient as the local update.\ngFOMAML(K)=1T∑i=1TLi′(UKi(θ))I=1T∑i=1TLi′(θKi)=1T∑i=1TgK+1i g_{F O M A M L}(K)=\\frac{1}{T} \\sum_{i=1}^{T} L_{i}^{\\prime}\\left(U_{K}^{i}(\\theta)\\right) I=\\frac{1}{T} \\sum_{i=1}^{T} L_{i}^{\\prime}\\left(\\theta_{K}^{i}\\right)=\\frac{1}{T} \\sum_{i=1}^{T} g_{K+1}^{i} From the formulas above, it is easy to see that the FedAvg update — the average of all clients\u0026rsquo; updates — is essentially a linear combination of the two ideas above.\ngFedAvg=1T∑i=1T∑j=1Kgji=1T∑i=1Tg1i+∑j=1K−11T∑i=1Tgj+1i=gFedSGD+∑j=1K−1gFOMAML(j) g_{F e d A v g}=\\frac{1}{T} \\sum_{i=1}^{T} \\sum_{j=1}^{K} g_{j}^{i}=\\frac{1}{T} \\sum_{i=1}^{T} g_{1}^{i}+\\sum_{j=1}^{K-1} \\frac{1}{T} \\sum_{i=1}^{T} g_{j+1}^{i}=g_{F e d S G D}+\\sum_{j=1}^{K-1} g_{F O M A M L}(j) Personalized FedAvg As shown in the figure above, FedAvg E in Algorithm 1 trains for E local epochs, weighting the gradient updates by the local data volume. Then, in the FL setting, Reptile (K) trains for K local steps, ignoring the local data volume.\nGenerally speaking, in terms of the number of communication rounds, FedAvg with several local epochs converges quickly within just a few rounds of communication. Due to the complexity of production environments, this metric is used to measure the convergence speed of FL algorithms. This paper finds that using momentum SGD as the server optimizer already benefits the personalized model, whereas the initial model remains relatively unstable. Previous approaches reduced the number of local training epochs or the learning rate.\nThis paper proposes using Reptile (K) for fine-tuning and then Adam as the server optimizer, which improves the initial model while also stabilizing the personalized model.\nTo be continued\n","permalink":"https://blog.ereebay.me/en/posts/fl-maml/","summary":"\u003ch1 id=\"improving-federated-learning-personalization-via-maml\"\u003eImproving Federated Learning Personalization via MAML\u003c/h1\u003e\n\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003eFL algorithms share many similarities with MAML and can be interpreted through meta-learning algorithms\u003c/li\u003e\n\u003cli\u003eFine-tuning gives the global model stronger accuracy while making personalization easier\u003c/li\u003e\n\u003cli\u003eModels trained on standard centralized datasets are harder to personalize than those trained with FedAvg\u003c/li\u003e\n\u003c/ol\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003ePoints out the connection between FL and MAML, and interprets the FL algorithm via MAML\u003c/li\u003e\n\u003cli\u003eImproves FedAvg with a two-stage approach of training and fine-tuning\u003c/li\u003e\n\u003cli\u003eFinds that FedAvg is essentially a meta-learning algorithm that optimizes personalization performance rather than the global model.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"interpreting-fedavg-as-a-meta-learning-algorithm\"\u003eInterpreting FedAvg as a Meta Learning Algorithm\u003c/h2\u003e\n\u003cp\u003eThe figure below shows MAML applied in the FL setting (left), the Reptile algorithm (middle), and FedAvg, the training algorithm of FL (right). Let L be the loss function. In each round of iteration, MAML trains by randomly sampling a batch of tasks T. For each task T there is an inner loop, and the outer loop aggregates the gradient updates obtained from each task. The FL algorithm randomly samples several clients T. For each T and its weight, several rounds of local optimization are performed, and the updated gradients are aggregated into a new global model. If we simplify the setting and assume all clients have the same amount of data, all weights become equal, and Reptile and FedAvg are then effectively the same algorithm.\u003c/p\u003e","title":"Improving Federated Learning Personalization via MAML Paper Notes (Incomplete)"},{"content":"Federated Adversarial Domain Adaptation Federated learning currently has the problem of knowledge domain shift, which prevents it from having good generalization ability. Domain shift occurs when the labeled data of the source nodes and the unlabeled data of the target node are different.\nThis paper mainly proposes a way to address the use of knowledge domains in federated learning, so that the knowledge learned by different nodes can be aligned with the data distribution of the target node.\nThe paper\u0026rsquo;s method mainly applies adversarial adaptation techniques to federated learning. In addition, a dynamic attention mechanism is designed, and feature disentanglement is used to enhance knowledge transfer.\nIntroduction Traditional federated learning has a problem: since the data on each node is collected in a non-IID manner, a domain shift problem arises. For example, the images taken by one device are mostly indoor scenes, while those taken by another are mostly outdoor scenes. The method proposed in this paper is: transferring the knowledge in the data of decentralized nodes to a node of a new and different data domain, without requiring additional user supervision. This method is also known as Unsupervised Federated Domain Adaptation.\nThere are currently many unsupervised domain adaptation methods, but the federated learning setting gives rise to the following problems:\nData is stored locally and cannot be shared. Model parameters are trained independently on different nodes and converge at different rates, and the contribution to the global model depends on the correlation between the two domains. Since the knowledge learned from the source nodes is highly integrated and difficult to disentangle, it may cause negative transfer. The method proposed in this paper is named Federated Adversarial Domain Adaptation (FADA), which uses adversarial techniques to solve the domain shift problem in federated learning systems.\nThe main steps of the method are:\nFirst, the federated domain adaptation problem is analyzed from a theoretical perspective, and a general, universal bound is provided. Inspired by the theoretical results, an effective adaptation algorithm is proposed, based on adversarial adaptation and representation disentanglement techniques applied to the federated learning setting. A dynamic attention model is designed to cope with the constantly changing convergence rates in federated learning systems. Related Work Unsupervised Domain Adaptation: The goal of UDA is to transfer knowledge from a labeled data domain to an unlabeled data domain.\nFederated Learning: I will not go into detail here.\nFeature Disentanglement: The features extracted by neural networks through complex hidden layers are usually highly integrated. Therefore, learning disentangled features can help remove irrelevant or domain-specific knowledge, and then model the needed domain knowledge.\nGeneralization Bound for FDA First, recall the definition of the theoretical error bound for the adaptation problem in the single-data-source setting. Then describe the definition of the error bound in the unsupervised federated domain adaptation setting. Federated Adversarial Domain Adaptation From the theory in the previous section, it is not difficult to see the importance of the weight α\\alpha and the distance. The method of this paper learns the weights by proposing a dynamic attention mechanism, and reduces the distance between the source domains and the target domain through a federated adversarial alignment mechanism. In addition, a feature disentanglement method is adopted to extract domain-invariant features to strengthen knowledge transfer.\nDynamic Attention Mechanism:\nThe principle of this mechanism is mainly to increase the weights of nodes with high contributions and decrease the weights of nodes with low contributions. So how do we judge the degree of a node\u0026rsquo;s contribution? This paper uses the gap statistics method to evaluate how well the target features can be clustered.\nI=∑r=1k12nr∑i,j∈Cr∥fit−fjt∥2 I=\\sum_{r=1}^{k} \\frac{1}{2 n_{r}} \\sum_{i, j \\in C_{r}}\\left\\|f_{i}^{t}-f_{j}^{t}\\right\\|_{2} Assume there are C1,C2,…,CkC_{1}, C_{2}, \\ldots, C_{k} clusters, where CrC_r denotes the indices of the objects in the r-th cluster, and nr=∣Cr∣n_{r}=\\left|C_{r}\\right|.\nIntuitively, a smaller gap statistics value indicates that the feature distribution has smaller intra-class variance. The degree of contribution of each source node is measured by computing the difference between the gap statistics of two iterations.\nIigain=Iip−1−Iip I_{i}^{g a i n}=I_{i}^{p-1}-I_{i}^{p} It represents how much the clusters can be improved after the target domain is updated with a source domain\u0026rsquo;s gradient. For the gradient updates, the following is used to decide how much each gradient contributes:\nSoftmax (I1gain ,I2gain ,…,INgain) \\text { Softmax }\\left(I_{1}^{\\text {gain }}, I_{2}^{\\text {gain }}, \\ldots, I_{N}^{\\text {gain}}\\right) Federated Adversarial Alignment:\nMachine learning models can suffer greatly degraded performance due to domain distance. To solve this problem, this paper improves the traditional adversarial training method in the federated learning setting. In the federated learning context, this paper divides the adversarial alignment optimization process into two steps: 1. For each domain, train a local feature extractor, with GiG_i and GtG_t corresponding to DiD_i and DtD_t respectively. 2. For each (Di,DtD_i, D_t) source-target domain pair, train an adversarial domain discriminator DI to align the two distributions in an adversarial learning way. First, train DI to recognize which feature domain a feature comes from; then train the generator, i.e., the feature extractors (Gi,GtG_i, G_t), to fool DI. Note: D can only access the outputs of GiG_i and GtG_t.\nGiven the i-th source data domain XSi\\mathbf{X}^{S_{i}} and the target domain XT\\mathbf{X}^T, the optimization objective of DIiD I_{i} is:\nLadvDI(XSi,XT,Gi,Gt)=−Exsi∼Xsi[log⁡DIi(Gi(xsi))]−Ext∼xt[log⁡(1−DIi(Gt(xt)))]ΘDIiL \\underset{\\Theta^{D I_{i}}{L}}{L_{a d v_{D I}}\\left(\\mathbf{X}^{S_{i}}, \\mathbf{X}^{T}, G_{i}, G_{t}\\right)=-\\mathbb{E}_{\\mathbf{x}^{s_{i} \\sim \\mathbf{X}^{s_{i}}}}\\left[\\log D I_{i}\\left(G_{i}\\left(\\mathbf{x}^{s_{i}}\\right)\\right)\\right]-\\mathbb{E}_{\\mathbf{x}^{t} \\sim \\mathbf{x}^{t}}\\left[\\log \\left(1-D I_{i}\\left(G_{t}\\left(\\mathbf{x}^{t}\\right)\\right)\\right)\\right]} Then, keeping D fixed, the objective function for updating G is:\nLΘGGi,ΘGt(XSi,XT,DIi)=−Exsi∼Xsi[log⁡DIi(Gi(xsi))]−Ext∼Xt[log⁡DIi(Gt(xt))] \\underset{\\Theta^{G}{G}_{i}, \\Theta^{G_{t}}}{L}\\left(\\mathbf{X}^{S_{i}}, \\mathbf{X}^{T}, D I_{i}\\right)=-\\mathbb{E}_{\\mathbf{x}^{s_{i} \\sim \\mathbf{X}^{s_{i}}}\\left[\\log D I_{i}\\left(G_{i}\\left(\\mathbf{x}^{s_{i}}\\right)\\right)\\right]-\\mathbb{E}_{\\mathbf{x}^{t} \\sim \\mathbf{X}^{t}}\\left[\\log D I_{i}\\left(G_{t}\\left(\\mathbf{x}^{t}\\right)\\right)\\right]} Feature Disentanglement:\nUnder the adversarial learning framework, this paper also adopts adversarial disentanglement to extract domain-invariant features. The paper argues that the extracted features can be divided into domain-specific features and domain-invariant features. As shown in the first figure, the feature disentangler D divides the features into two categories. First, a K-class classifier and a K-class category recognizer are trained to predict labels from the features.\nLcrossentropyΘGi,ΘDi,ΘCi,ΘCIi=−E(xsi,ysi)∼D^si∑k=1K1[k=ysi]log⁡(Ci(fdi))−E(xsi,ysi)∼D^si∑k=1K1[k=ysi]log⁡(CIi(fds)) \\begin{array}{c}{L_{cross entropy}} \\\\ {\\Theta^{G_i}, \\Theta^{D_i}, \\Theta^{C_i}, \\Theta^{CI_i}}\\end{array}=-\\mathbb{E}_{\\left(\\mathbf{x}^{s_i}, \\mathbf{y}^{s_i}\\right) \\sim \\widehat{\\mathcal{D}}_{s_i}} \\sum_{k=1}^{K} \\mathbb{1}\\left[k=\\mathbf{y}^{s_{i}}\\right] \\log \\left(C_{i}\\left(f_{d i}\\right)\\right)-\\mathbb{E}_{\\left(\\mathbf{x}^{s} i, \\mathbf{y}^{s} i\\right) \\sim \\widehat{\\mathcal{D}}_{s_{i}}} \\sum_{k=1}^{K} \\mathbb{1}\\left[k=\\mathbf{y}^{s_{i}}\\right] \\log \\left(C I_{i}\\left(f_{d s}\\right)\\right) Then, with the category classifier fixed, only the feature disentangler is trained, producing domain-specific features to fool the category classifier.\nLentΘDi,ΘGi=−1Nsi∑j=1Nsilog⁡CIi(fdsj)=−1Nsi∑j=1Nsilog⁡CIi(Di(Gi(xsi))) \\underset{\\Theta^{D} i, \\Theta^{G_{i}}}{L_{e n t}}=-\\frac{1}{N_{s_{i}}} \\sum_{j=1}^{N_{s_{i}}} \\log C I_{i}\\left(f_{d s}^{j}\\right)=-\\frac{1}{N_{s_{i}}} \\sum_{j=1}^{N_{s_{i}}} \\log C I_{i}\\left(D_{i}\\left(G_{i}\\left(\\mathbf{x}^{s_{i}}\\right)\\right)\\right) The feature disentanglement method transfers knowledge by keeping the class-invariant features and removing the class-specific features. To enhance the disentanglement effect, this paper minimizes the mutual information between the two.\nMutual information is defined as I(fdi;fds)=∫P×Qlog⁡dPPQdPP⊗PQdPPQI\\left(f_{d i} ; f_{d s}\\right)=\\int_{\\mathcal{P} \\times \\mathcal{Q}} \\log \\frac{d \\mathbb{P}_{\\mathcal{P} Q}}{d \\mathbb{P}_{\\mathcal{P}} \\otimes \\mathbb{P}_{\\mathcal{Q}}} d \\mathbb{P}_{\\mathcal{P} \\mathcal{Q}}, where PPQ\\mathbb{P}_{\\mathcal{P} \\mathcal{Q}} is the joint probability distribution of the two kinds of features, and PP=∫PdPPQ,PQ=∫QdPPQ\\mathbb{P}_{\\mathcal{P}}=\\int_{\\mathcal{P}} d \\mathbb{P}_{\\mathcal{P} \\mathcal{Q}}, \\mathbb{P}_{\\mathcal{Q}}=\\int_{\\mathcal{Q}} d \\mathbb{P}_{\\mathcal{P} \\mathcal{Q}} are the corresponding marginal probability distributions.\nThe Mutual Information Neural Estimator (MINE) is used to estimate the mutual information:\nTθ:I(P;Q)^n=sup⁡θ∈ΘEPPQ(n)[Tθ]−log⁡(EPP(n)⊗PQ(n)[eTθ]) T_{\\theta}: \\widehat{I(\\mathcal{P} ; \\mathcal{Q})}_{n}=\\sup _{\\theta \\in \\Theta} \\mathbb{E}_{\\mathbb{P}_{\\mathcal{P} Q}^{(n)}}\\left[T_{\\theta}\\right]-\\log \\left(\\mathbb{E}_{\\mathbb{P}_{P}^{(n)} \\otimes \\mathbb{P}_{Q}^{(n)}}\\left[e^{T_{\\theta}}\\right]\\right) In practice, it can be computed by:\nI(P;Q)=∬PPQn(p,q)T(p,q,θ)−log⁡(∬PPn(p)PQn(q)eT(p,q,θ)) I(\\mathcal{P} ; \\mathcal{Q})=\\iint \\mathbb{P}_{\\mathcal{P} \\mathcal{Q}}^{n}(p, q) T(p, q, \\theta)-\\log \\left(\\iint \\mathbb{P}_{\\mathcal{P}}^{n}(p) \\mathbb{P}_{\\mathcal{Q}}^{n}(q) e^{T(p, q, \\theta)}\\right) To avoid computing integrals, the Monte Carlo method is adopted:\nI(P,Q)=1n∑i=1nT(p,q,θ)−log⁡(1n∑i=1neT(p,q′,θ)) I(\\mathcal{P}, \\mathcal{Q})=\\frac{1}{n} \\sum_{i=1}^{n} T(p, q, \\theta)-\\log \\left(\\frac{1}{n} \\sum_{i=1}^{n} e^{T\\left(p, q^{\\prime}, \\theta\\right)}\\right) where (p, q) are sampled from the joint distribution, q′q^{\\prime} is sampled from the marginal distribution, and T(p,q,θ)T(p, q, \\theta) is a neural network determined by θ\\theta that estimates the mutual information between the P and Q distributions.\nThe domain-invariant features and the domain-specific features are fed into a reconstructor to obtain recombined features; the loss function uses an L2 loss to reconstruct the original features, in order to preserve the composability of the representation. The L2 constraint and the mutual information loss constraint can be balanced by tuning the hyperparameters.\nTo be continued\n","permalink":"https://blog.ereebay.me/en/posts/flpaper/","summary":"\u003ch1 id=\"federated-adversarial-domain-adaptation\"\u003eFederated Adversarial Domain Adaptation\u003c/h1\u003e\n\u003cp\u003eFederated learning currently has the problem of knowledge domain shift, which prevents it from having good generalization ability. Domain shift occurs when the labeled data of the source nodes and the unlabeled data of the target node are different.\u003c/p\u003e\n\u003cp\u003eThis paper mainly proposes a way to address the use of knowledge domains in federated learning, so that the knowledge learned by different nodes can be aligned with the data distribution of the target node.\u003c/p\u003e","title":"Federated Adversarial Domain Adaptation Paper Notes (Incomplete)"},{"content":"Memory GAN Reading Notes Introduction This paper mainly solves two problems in training unsupervised GANs.\nFirst, since generative adversarial networks use only a continuous latent distribution to represent multiple classes or clusters of data, they usually cannot correctly handle the structurally discontinuous nature between different classes in the latent space. (Caused by the mode collapse problem.) For example, if a GAN embeds buildings and cats in the same continuous latent distribution, the GAN may generate realistic images in the transition region between the two classes.\nSecond, the discriminator of a generative adversarial network very easily forgets the samples generated in the past, which causes instability during adversarial training.\nThe authors believe that these two main problems can be solved by a learnable memory network accessible to both the generator and the discriminator. The generator can effectively learn representations of the training samples to understand the underlying clustering distribution of the data, thereby alleviating the structural discontinuity problem. Meanwhile, the discriminator can better memorize the clusters of previously generated samples, which can alleviate the forgetting problem.\nThis paper proposes an end-to-end generative adversarial network model, Memory GAN, which involves an unsupervised memory network integrated with existing generative adversarial network models.\nVon Mises-Fisher (vMF) mixture model. The memory module can effectively alleviate the instability problem. First, to alleviate the structural discontinuity problem, the memory can learn representations of the training samples, helping the generator better understand the class and cluster distributions. Therefore, we can separate the modeling of discrete clusters from the embedding of data attributes on the continuous latent space, which can alleviate the discontinuity problem.\nSecond, the memory network can alleviate the forgetting problem by learning to memorize the clusters of previously generated sample data, including very rare samples.\nArchitecture The overall architecture is divided into the memory discriminative network (DMN) and the memory conditional generative network (MCGN).\nMemory Discriminative Network The memory discriminative network consists of a feedforward network μ\\mu and a memory network.\nThe feedforward network μ\\mu is a convolutional neural network, whose input is x∈RDx \\in \\mathbb{R}^{D} and whose output is a normalized vector q=μ(x)∈RMq=\\mu(x) \\in \\mathbb{R}^{M}, where ∥q∥=1\\|q\\|=1. This is then used as the input of the memory module, which outputs the discriminative result.\nThe memory network is formulated as M=(K,v,a,h)\\mathcal{M}=(K, v, a, h), where K∈RN×MK \\in \\mathbb{R}^{N \\times M} is the memory key matrix, NN is the memory size and MM is the dimension. v∈{0,1}Nv \\in\\{0,1\\}^{N} is the memory value matrix. Conceptually, each key vector stores a representation of a clustering center learned through the vMF mixture model, and its corresponding key value is the real/fake judgment. a∈RNa \\in \\mathbb{R}^{N} denotes the age of the object stored in each memory block. And h∈RNh \\in \\mathbb{R}^{N} denotes the slot histogram, where each hih_{i} represents the amount of valid data belonging to the i-th memory slot.\nRelated Mechanisms: life-long memory network: the memory space can grow freely, and there is no need to reset it during training.\nk-nearest neighbor indexing for efficient memory lookup: k-nearest neighbor indexing is used for lookup.\nleast recently used (LRU) scheme for memory update. The LRU scheme is used for memory update.\nInnovations: Probabilistic derivation: the data likelihood, as well as the prior and posterior distributions of the categorical distribution over memory indices, can be computed. The memory learns an approximate distribution of queries by maximizing the likelihood with an incremental EM algorithm. The memory is optimized through the GAN loss rather than through a memory loss. The contribution of each sample is decided by tracking the slot histogram. Discriminator Output For each input xx, we first need to decide which memory slot to use to compute the discriminative probability.\nc∈{1,2,…,N}c \\in\\{1,2, \\ldots, N\\} denotes the memory slot index.\nA Von Mises-Fisher (vMF) mixture model is used to represent the posterior distribution of the memory index.\np(c=i∣x)=p(x∣c=i)p(c=i)∑j=1Np(x∣c=j)p(c=j)=exp⁡(κKiTμ(x))p(c=i)∑j=1Nexp⁡(κKjTμ(x))p(c=j)p(c=i | x)=\\frac{p(x | c=i) p(c=i)}{\\sum_{j=1}^{N} p(x | c=j) p(c=j)}=\\frac{\\exp \\left(\\kappa K_{i}^{T} \\mu(x)\\right) p(c=i)}{\\sum_{j=1}^{N} \\exp \\left(\\kappa K_{j}^{T} \\mu(x)\\right) p(c=j)}In p(x∣c=i)=C(κ)exp⁡(κKiTμ(x))p(x | c=i)=C(\\kappa) \\exp \\left(\\kappa K_{i}^{T} \\mu(x)\\right), κ=1\\kappa=1 is a constant attention parameter.\nThe vMF is effectively equivalent to a properly normalized Gaussian distribution defined on the unit sphere.\nThe categorical distribution over the memory index, p(c)p(c), is obtained by normalizing the slot histogram, p(c=i)=hi+β∑j=1N(hj+β)p(c=i)=\\frac{h_{i}+\\beta}{\\sum_{j=1}^{N}\\left(h_{j}+\\beta\\right)}, where β(=10−8)\\beta\\left(=10^{-8}\\right) is a smoothing constant for numerical stability. By using p(y=1∣c=i,x)=vip(y=1 | c=i, x)=v_{i}, we marginalize the joint probability p(y=1,c∣x)p(y=1, c | x) over c to obtain p(y=1∣x)p(y=1 | x):\np(y=1∣x)=∑i=1Np(y=1∣c=i,x)p(c=i∣x)=∑i=1Nvip(c=i∣x)=Ei∼p(c∣x)[vi] p(y=1 | x)=\\sum_{i=1}^{N} p(y=1 | c=i, x) p(c=i | x)=\\sum_{i=1}^{N} v_{i} p(c=i | x)=\\mathbb{E}_{i \\sim p(c | x)}\\left[v_{i}\\right] However, for each sample x, this is not scalable for the whole memory of size N.\nThe maximum a posteriori probability is adopted to consider the top-k slots S={s1,…,sk}S=\\left\\{s_{1}, \\dots, s_{k}\\right\\}:\nS=argmax⁡c1,…,ckp(c∣x)=argmax⁡c1,…,ckp(x∣c)p(c)=argmax⁡c1,…,ckexp⁡(κKcTμ(x))(hc+β) S=\\underset{c_{1}, \\ldots, c_{k}}{\\operatorname{argmax}} p(c | x)=\\underset{c_{1}, \\ldots, c_{k}}{\\operatorname{argmax}} p(x | c) p(c)=\\underset{c_{1}, \\ldots, c_{k}}{\\operatorname{argmax}} \\exp \\left(\\kappa K_{c}^{T} \\mu(x)\\right)\\left(h_{c}+\\beta\\right) where p(x∣c)p(x | c) is the vMF likelihood and p(c)p(c) is the prior distribution of the memory index.\nHere we omit the normalization of the vMF likelihood and the denominator of the prior, because they are both constants. Once we have obtained S, we can compute\np(y∣x)≈∑i∈Svip(x∣c=i)p(c=i)∑j∈Sp(x∣c=j)p(c=j) p(y | x) \\approx \\frac{\\sum_{i \\in S} v_{i} p(x | c=i) p(c=i)}{\\sum_{j \\in S} p(x | c=j) p(c=j)} Memory Update Mechanism The memory keys and values are updated during training. The update mechanism includes a traditional memory update mechanism and an incremental EM algorithm.\nLet the sample be xx with label yy, where real is 1 and fake is 0. For each x, we first find the k-nearest slots SyS_{y}, but using the conditional posterior p(c∣x,vc=y)p\\left(c | x, v_{c}=y\\right). This is so that in the subsequent EM algorithm, only the slots belonging to the same class as y are considered.\nAfter that, we update the memory in different ways depending on whether SyS_y contains the correct label.\nWithout the correct label: The oldest memory slot is found via na=argmax⁡i∈{1,…,N}ain_{a}=\\operatorname{argmax}_{i \\in\\{1, \\ldots, N\\}} a_{i}, and the information of x is copied to the corresponding position: Kna←q=μ(x)K_{n_{a}} \\leftarrow q=\\mu(x), vna←yv_{n_{a}} \\leftarrow y, ana←0a_{n_{a}} \\leftarrow 0, hna←1N∑i=1Nhih_{n_{a}} \\leftarrow \\frac{1}{N} \\sum_{i=1}^{N} h_{i}.\nWith the correct label: The memory keys are updated through the following custom incremental EM algorithm run for T iterations, which partly incorporates the information of the new sample.\nIn the inference step, by applying the keys of the previous moment K^it−1\\hat{K}_{i}^{t-1} and h^it−1\\hat{h}_{i}^{t-1} to Equation 1, the posterior distribution γit=p(ci∣x)\\gamma_{i}^{t}=p\\left(c_{i} | x\\right) is computed, where i∈Syi \\in S_{y}. In the maximization step, the following updates are performed:\nh^it←h^it−1+γt−γt−1,K^it←K^it−1+γt−γt−1h^it(qi−K^it) \\hat{h}_{i}^{t} \\leftarrow \\hat{h}_{i}^{t-1}+\\gamma^{t}-\\gamma^{t-1}, \\quad \\hat{K}_{i}^{t} \\leftarrow \\hat{K}_{i}^{t-1}+\\frac{\\gamma^{t}-\\gamma^{t-1}}{\\hat{h}_{i}^{t}}\\left(q_{i}-\\hat{K}_{i}^{t}\\right) where t∈1,…,T,γ0=0,K^i1=Ki,h^i1=αhit \\in 1, \\ldots, T, \\gamma^{0}=0, \\hat{K}_{i}^{1}=K_{i}, \\hat{h}_{i}^{1}=\\alpha h_{i}, and α=0.5\\alpha=0.5.\nAfter T iterations, the slots of SyS_y are updated through Ki←K^itK_{i} \\leftarrow \\hat{K}_{i}^{t} and hi←h^ith_{i} \\leftarrow \\hat{h}_{i}^{t}.\nThe decay rate α\\alpha controls the degree to which the contribution of old queries to the slot position in the average direction of the mixture component is exponentially reduced.\nα\\alpha is crucial for performance, because the old queries used to update the keys no longer fit the current mixture distribution, as the feedforward network itself is also being updated.\nFinally, it is worth noting that this memory update mechanism is orthogonal to the adversarial training algorithm, because when the discriminator is updated, the memory update is carried out independently. In addition, adding the memory module does not affect the speed of the model at test time, because the memory is only updated during training.\nMemory Conditional Generative Network The memory conditional generative network is based on the generator of InfoGAN. The difference is that it is conditioned not only on random noise but also on memory information.\nIn other words, the generator not only samples randomly from the noise distribution, but also samples the memory index ii from P(c=i∣vc=1)=hivi∑jNhjvjP\\left(c=i | v_{c}=1\\right)=\\frac{h_{i} v_{i}}{\\sum_{j}^{N} h_{j} v_{j}}. The formula above represents the frequency of occurrence of the unit i that stores real data. The final input is [Ki,z][K_i, z], where KiK_i is the key vector of memory index i.\nThe difference from other CGANs is that MCGN does not require extra annotations or an extra encoder. Instead, MCGN can make full use of the memory information learned by the DMN in an unsupervised way. The DMN learns the vMF mixture memory using only each sample in the sequence and its label.\nThe training procedure of the whole MemoryGAN is:\nfor number of training iterations do Sample a minibatch of samples from the training set Sample a minibatch from the noise distribution and the memory indices Update the discriminator loss Find S_y for each sample in the minibatch Initialize the key, h and gamma for each slot in S_y for number of EM iterations do Estimate γs\\gamma_s for each s Update hsh_s Update KsK_s Update the vMF mixture model, hsy←h^syT,Ksy←K^syTh_{s_{y}} \\leftarrow \\hat{h}_{s_{y}}^{T}, K_{s_{y}} \\leftarrow \\hat{K}_{s_{y}}^{T} for sy∈Sys_{y} \\in S_{y} Sample a minibatch from the noise distribution and the memory indices Update the generator loss\nobjective function The goal of MemoryGAN is based on the objective of InfoGAN, which is to maximize the mutual information between the latent variables and the observed content. (For details, see InfoGAN.)\nA mutual information loss is added between KiK_i and G(z,Ki)G(z, K_i) to ensure the continuity between the sampled memory information and the structured information of the generated samples:\nI(Ki;G(z,Ki))≥H(Ki)−I^−log⁡C(κ) I\\left(K_{i} ; G\\left(z, K_{i}\\right)\\right) \\geq H\\left(K_{i}\\right)-\\hat{I}-\\log C(\\kappa) where I^\\hat{I} denotes the expectation of the negative cosine similarity, I^=−Ex∼G(z,Ki)[κKiTμ(x)]\\hat{I}=-E_{x \\sim G\\left(z, K_{i}\\right)}\\left[\\kappa K_{i}^{T} \\mu(x)\\right].\nTo be continued\n","permalink":"https://blog.ereebay.me/en/posts/memorygan/","summary":"\u003ch1 id=\"memory-gan-reading-notes\"\u003eMemory GAN Reading Notes\u003c/h1\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eThis paper mainly solves two problems in training unsupervised GANs.\u003c/p\u003e\n\u003cp\u003eFirst, since generative adversarial networks use only a continuous latent distribution to represent multiple classes or clusters of data, they usually cannot correctly handle the structurally discontinuous nature between different classes in the latent space. (Caused by the mode collapse problem.) For example, if a GAN embeds buildings and cats in the same continuous latent distribution, the GAN may generate realistic images in the transition region between the two classes.\u003c/p\u003e","title":"MemoryGAN Paper Notes (Incomplete)"},{"content":"Getting Started with Kettle Since I was still one credit short in my final semester, I had to take one more course, and looking at the schedule the only option was OLAP — yet another completely unfamiliar subject. Tutorials about Kettle online are scarce, so I gathered some materials and jotted down these notes on the tool this course requires\nIntroduction to Kettle Kettle is an ETL (Extract, Transform and Load) tool for data extraction, transformation, and loading. ETL tools are used very frequently in data warehouse projects, and Kettle can also be applied in the following scenarios:\nIntegrating data across different applications or databases\nExporting data from a database to text files\nLoading bulk data into a database\nData cleansing\nIt is also used in projects involving application integration\nKettle is very easy to use: you design what business logic to implement through a graphical interface, with no need to write code; therefore, Kettle is designed in a metadata-oriented way\nKettle supports many input and output formats, including text files, data tables, and both commercial and free database engines. In addition, Kettle\u0026rsquo;s powerful transformation features make it very convenient to manipulate data.\nInstalling Kettle Download link\nSince my platform is macOS and there is a bug that prevents it from launching on double-click, I could only extract the archive and then run in the terminal\nsh spoon.sh to start Spoon\nData Transformation - Exporting to Excel Connect to the database Export to Excel First, create a new transformation as shown in the figure below, and set up the database connection.\nOnce the database connection is set up, you can configure the input source; here we use a table in the database as the input source\nWith the input source configured, you can click preview to take a look at the data first\nAfter setting up the input source, likewise drag the Excel output onto the workspace, but don\u0026rsquo;t rush to configure the output source yet: in the View pane, add a hop to connect the input and output sources\nThis way, when you configure the output source, you can directly fetch and select the fields to export in the Fields tab\nFinally, click Start to run the transformation\nLet\u0026rsquo;s take a look at the exported file\nHello World Alright, now that you have learned how transformations work, let\u0026rsquo;s do a Hello World example. I came across this tutorial online and found it pretty good — it lets you get in touch with more of Kettle\u0026rsquo;s features, unlike my very basic walkthrough above.\nThis Hello World outputs a corresponding greeting for each name in the database and exports the result to a file.\nFirst, set up the input source the same way, but the SQL statement needs a small change: here I selected only their names, ordered by id, limited to the first ten.\nThen write a small JavaScript script to automatically add a \u0026ldquo;hello\u0026rdquo;\nFinally, link the three steps with hops; you can run preview first to check the result\nThen take a look at the result\nThe above only demonstrates the most basic features of data transformation; other, more detailed features can all be found in the Design tab In short, Kettle\u0026rsquo;s operating logic is steps + hops: a step is an operation you perform, and hops are the links between them.\nJobs The above covered transformations, one of Kettle\u0026rsquo;s core features; now let\u0026rsquo;s introduce another Kettle feature, the job.\nA job is essentially an automated workflow: when you need to run multiple transformations or add some logical control conditions, you need a job.\nHere, following the earlier Hello World example, we build a simple job that detects when the file does not exist in the folder and then automatically exports data from the database and adds \u0026ldquo;hello\u0026rdquo;.\nFind the widget shown in the figure above in the toolbar on the right, drag it into the workspace, connect things with hops, and configure it — that\u0026rsquo;s it. The operations are similar to those for creating a transformation; it just combines several of them together\nThe figure below shows the transformation configuration interface, where you enter the path of the transformation script\nKitchen and Pan The kitchen and pan commands are used to execute job and transformation scripts.\n./kitch.sh -file ./scripts/demo.kjb ./pan.sh -file ./scripts/demo.ktr From now on, just save the jobs and transformations designed in Spoon and run them directly from the command line — much more convenient.\nI\u0026rsquo;ll update these notes whenever I get to use new features.\nReferences The ETL Powerhouse Kettle in Practice, Part 1 [Introduction to Kettle]\nKETTLE Tutorial\nUsing Kettle Jobs\nKettle Tutorials\nKettle Chinese Community\n","permalink":"https://blog.ereebay.me/en/posts/kettle%E5%85%A5%E9%97%A8%E6%95%99%E7%A8%8B/","summary":"\u003ch1 id=\"getting-started-with-kettle\"\u003eGetting Started with Kettle\u003c/h1\u003e\n\u003cp\u003eSince I was still one credit short in my final semester, I had to take one more course, and looking at the schedule the only option was OLAP — yet another completely unfamiliar subject. Tutorials about Kettle online are scarce, so I gathered some materials and jotted down these notes on the tool this course requires\u003c/p\u003e\n\u003c!-- more --\u003e\n\u003ch2 id=\"introduction-to-kettle\"\u003eIntroduction to Kettle\u003c/h2\u003e\n\u003cp\u003eKettle is an ETL (Extract, Transform and Load) tool for data extraction, transformation, and loading. ETL tools are used very frequently in data warehouse projects, and Kettle can also be applied in the following scenarios:\u003c/p\u003e","title":"Getting Started with Kettle"},{"content":"One-shot Learning with Memory-Augmented Neural Networks Paper Notes Introduction Traditional gradient-based deep learning methods require large amounts of data to learn. When confronted with new data, the model has to relearn new parameters and cannot quickly adapt to it.\nThis paper proposes a memory-augmented neural network that can rapidly assimilate new data and, after learning from only a few samples, use that data to make accurate predictions.\nFor the design of the MANN model, the authors had two requirements:\nThe stored information must be stable and addressed element-wise. The number of parameters must not be tied to the size of the memory. In the end, the model combines the following two advantages:\nIt can learn a general way of learning by extracting useful information from raw data via gradient descent. It can rapidly learn never-before-seen information by relying on an additional memory module. Meta-Learning Method Design Usually, parameters θ\\theta are chosen on some dataset DD to minimize a learning cost L\\mathcal{L}.\nIn meta-learning, however, what is minimized is the expected cost with respect to a distribution over datasets p(D)p(D):\nθ∗=argmin⁡θED∼p(D)[L(D;θ)] \\theta^{*}=\\operatorname{argmin}_{\\theta} E_{D \\sim p(D)}[\\mathcal{L}(D ; \\theta)] To achieve this, the paper makes the following design choices:\nFirst, the inputs are presented as a sequence, where each input carries the label of the previous input: (x1, null ),(x2,y1),…,(xT,yT−1) \\left(\\mathbf{x}_{1}, \\text { null }\\right),\\left(\\mathbf{x}_{2}, y_{1}\\right), \\ldots,\\left(\\mathbf{x}_{T}, y_{T-1}\\right) Across different datasets, the labels are shuffled (to prevent the network from gradually learning a fixed mapping between samples and labels). Instead, this forces the network to learn the trick of holding a data sample in memory until the correct label arrives, so that sample-label information is bound together and stored for later use. Without relying on the actual content of the data and labels, the meta-learning model learns to bind the data distribution to the corresponding labels, and generalizes into a universal model that captures the data-label relationship for prediction.\nMemory-Augmented Model Neural Turing Machine The composition of the Neural Turing Machine is largely similar to the MANN in this paper. An NTM consists of a controller, such as an LSTM or a feedforward network, which interacts with an external memory module through a set of read/write heads.\nIn this model, the controller likewise uses an LSTM or a feedforward network.\nTo be continued\n","permalink":"https://blog.ereebay.me/en/posts/mann/","summary":"\u003ch1 id=\"one-shot-learning-with-memory-augmented-neural-networks-paper-notes\"\u003eOne-shot Learning with Memory-Augmented Neural Networks Paper Notes\u003c/h1\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eTraditional gradient-based deep learning methods require large amounts of data to learn. When confronted with new data, the model has to relearn new parameters and cannot quickly adapt to it.\u003c/p\u003e\n\u003cp\u003eThis paper proposes a memory-augmented neural network that can rapidly assimilate new data and, after learning from only a few samples, use that data to make accurate predictions.\u003c/p\u003e\n\u003c!-- more --\u003e\n\u003cp\u003eFor the design of the MANN model, the authors had two requirements:\u003c/p\u003e","title":"One-shot Learning with Memory-Augmented Neural Networks Paper Notes (Incomplete)"},{"content":"Generative Adversarial Network Overview of GAN The idea behind GAN is a two-player zero-sum game in which the sum of the two players\u0026rsquo; interests is a constant. Consider arm wrestling: assuming the total space is fixed, if your strength is greater, you gain more space and I get less; conversely, if I am stronger, I gain more. But one thing is certain — our total space is fixed. This is a two-player game in which the total interest is constant.\nA vivid analogy: GAN is like a large network containing two smaller networks. One is the generative network, which can be regarded as a counterfeiter producing fake banknotes, and the other is the discriminative network, i.e., the person authenticating the banknotes. The goal of the generative network is to fool the discriminator, while the discriminator strives not to be fooled by the generator. Through alternating optimization during training, both networks improve. It is theoretically proven that, at best, the generative model makes the discriminator unable to tell real from fake, i.e., the probability of real versus fake is fifty-fifty.\nThe figure above is a structural diagram of a generative adversarial network: the discriminator takes in both real samples and fake samples produced by the generator, and outputs a real/fake verdict. The generator takes noise as input and produces fake samples.\nThe Principle of GAN The following is the objective function of GAN:\nmin⁡Gmax⁡DV(D,G)=Ex∼pdata(x)[log⁡D(x)]+Ez∼pz(z)[log⁡(1−D(G(z)))] \\min _ { G } \\max _ { D } V ( D , G ) = \\mathbb { E } _ { \\boldsymbol { x } \\sim p _ { \\mathrm { data } } ( \\boldsymbol { x } ) } [ \\log D ( \\boldsymbol { x } ) ] + \\mathbb { E } _ { \\boldsymbol { z } \\sim p _ { \\boldsymbol { z } } ( z ) } [ \\log ( 1 - D ( G ( \\boldsymbol { z } ) ) ) ] From the objective function we can see that the overall cost function minimizes the generator and maximizes the discriminator. When tackling this optimization problem, we can first fix G, maximize D, and then minimize G to obtain the optimal solution. Here, for a given G, maximizing V(D,G) measures the divergence or distance between P_G and P_data.\nFirst, after fixing G, the optimal D can be expressed as:\nDG∗=argmax⁡DV(G,D) D _ { G } ^ { * } = \\operatorname { argmax } _ { D } V ( G , D ) The problem of optimizing G can then be expressed as:\nG∗=argmin⁡GV(G,DG∗) G ^ { * } = \\operatorname { argmin } _ { G } V \\left( G , D _ { G } ^ { * } \\right) Theoretical Derivation The original paper\u0026rsquo;s derivation uses the JS divergence to describe the similarity between two distributions, and the JS divergence is in turn built from the KL divergence. Therefore, before carrying out the full derivation, we first introduce some theoretical foundations, derive the conditions required for the optimal discriminator and the optimal generator, and finally use the derived results to restate the training procedure.\nKL Divergence For a single random variable xx with two separate probability distributions P(x)P(x) and Q(x)Q(x), the KL divergence can measure the difference between the two distributions (the appendix proves why the KL divergence reflects the difference between two distributions):\nDKL(P∥Q)=Ex∼P[log⁡P(x)Q(x)]=Ex∼P[log⁡P(x)−log⁡Q(x)] D _ { \\mathrm { KL } } ( P \\| Q ) = \\mathbb { E } _ { \\mathrm { x } \\sim P } \\left[ \\log \\frac { P ( x ) } { Q ( x ) } \\right] = \\mathbb { E } _ { \\mathrm { x } \\sim P } [ \\log P ( x ) - \\log Q ( x ) ] Properties of the KL divergence:\nNon-negativity (used later in the derivation); moreover, the KL divergence equals 0 if and only if P and Q are the same distribution. Because of this non-negativity, it is often used to measure the difference between two distributions. (The appendix proves its non-negativity.) Asymmetry. Although it can measure the difference between distributions, this difference is not a symmetric distance: the KL divergence of P with respect to Q differs from that of Q with respect to P. Issues in the Paper\u0026rsquo;s Derivation In the original paper, one idea differs from many other approaches: the generator G is not required to satisfy an invertibility condition, and in practice G is indeed non-invertible. However, in the proof, the change-of-variables formula for integrals was incorrectly used, while this change of variables is valid only when G is invertible. So the proof should be based on the validity of the following equality:\nEz∼px(z)log⁡(1−D(G(z)))=Ex∼pc(x)log⁡(1−D(x)) E _ { z \\sim p _ { x } ( z ) } \\log ( 1 - D ( G ( z ) ) ) = E _ { x \\sim p _ { c } ( x ) } \\log ( 1 - D ( x ) ) This equality comes from the Radon-Nikodym theorem in measure theory; it is presented as Proposition 1 in the original paper and expressed as the following equality:\n∫xpdata(x)log⁡D(x)dx+∫zp(z)log⁡(1−D(G(z)))dz=∫xpdata(x)log⁡D(x)+pG(x)log⁡(1−D(x))dx \\begin{array} { c } { \\int _ { x } p _ { d a t a } ( x ) \\log D ( x ) \\mathrm { d } x + \\int _ { z } p ( z ) \\log ( 1 - D ( G ( z ) ) ) \\mathrm { d } z } \\\\ { = \\int _ { x } p _ { d a t a } ( x ) \\log D ( x ) + p _ { G } ( x ) \\log ( 1 - D ( x ) ) \\mathrm { d } x } \\end{array} This formula uses the change-of-variables formula for integrals, but doing so requires computing the inverse of G, whose existence is not assumed. Moreover, in the practice of neural networks it does not exist. However, this practice is so common in ML that it is simply overlooked.\nOptimal Discriminator In the minimax game, we first fix the generator G and maximize the value function, which yields the optimal discriminator D. The maximized value function measures the divergence between the distribution produced by the generator and the distribution of the dataset (proven later).\nThe expectations in the original paper\u0026rsquo;s value function can be expanded into integral form:\nV(G,D)=∫xpdata(x)log⁡(D(x))+pg(x)log⁡(1−D(x))dx V ( G , D ) = \\int _ { x } p _ { \\mathrm { data } } ( \\boldsymbol { x } ) \\log ( D ( \\boldsymbol { x } ) ) + p _ { g } ( \\boldsymbol { x } ) \\log ( 1 - D ( \\boldsymbol { x } ) ) d x Maximizing the integral can be converted into maximizing the integrand. Maximizing the integrand yields the optimal discriminator D, so all terms not involving the discriminator can be treated as constants. Letting the discriminator D(x) be y, the integrand can be written as:\nf(y)=alog⁡y+blog⁡(1−y) f ( y ) = a \\log y + b \\log ( 1 - y ) To find the optimal extreme point, if a+b≠0a + b \\neq 0, we can solve using the first-order derivative:\nf′(y)=0⇒ay−b1−y=0⇒y=aa+b f ^ { \\prime } ( y ) = 0 \\Rightarrow \\frac { a } { y } - \\frac { b } { 1 - y } = 0 \\Rightarrow y = \\frac { a } { a + b } Taking the second derivative at the stationary point gives:\nf′′(aa+b)=−a(aa+b)2−b1−(aa+b)2\u0026lt;0 f ^ { \\prime \\prime } \\left( \\frac { a } { a + b } \\right) = - \\frac { a } { \\left( \\frac { a } { a + b } \\right) ^ { 2 } } - \\frac { b } { 1 - \\left( \\frac { a } { a + b } \\right) ^ { 2 } } \u0026lt; 0 where a,b∈(0,1)a , b \\in ( 0,1 ). Since the first derivative equals 0 and the second derivative is less than 0, aa+b\\frac { a } { a + b } is a maximum.\nFinally, the value function can be written as:\nV(G,D)=∫xpdata(x)log⁡D(x)+pG(x)log⁡(1−D(x))dx≤∫max⁡ymax⁡ypdata(x)log⁡y+pG(x)log⁡(1−y)dx \\begin{aligned} V ( G , D ) = \u0026amp; \\int _ { x } p _ { d a t a } ( x ) \\log D ( x ) + p _ { G } ( x ) \\log ( 1 - D ( x ) ) \\mathrm { d } x \\\\ \u0026amp; \\leq \\int \\max _ { y } \\max _ { y } p _ { d a t a } ( x ) \\log y + p _ { G } ( x ) \\log ( 1 - y ) \\mathrm { d } x \\end{aligned} Letting D(x)=P_data/(P_data+p_G) attains the maximum, because f(y) has a unique maximum on its domain; that is, the optimal D is unique, and no other D can attain the maximum.\nIn fact, the optimal D cannot be computed in practice, but it is mathematically important. Moreover, we do not know the prior Pdata, so we cannot use it directly in training. On the other hand, the existence of the optimal D implies that of the optimal G, and it suffices that we approach the optimal D.\nOptimal Generator The training process of GAN aims to make P_G=P_data, in which case the optimal D can be written as:\nDG∗=pdatapdata+pG=12 D _ { G } ^ { * } = \\frac { p _ { \\text {data} } } { p _ { \\text {data} } + p _ { G } } = \\frac { 1 } { 2 } That is, the optimal generator makes the discriminator unable to distinguish P_data from P_G. Based on this observation, the authors proved that G is the solution of the minimax game.\nTheorem: The global minimum of C(G)=maxV(G,D) is attained if and only if P_G=P_data.\nThe theorem states an if-and-only-if result, so we prove it from both directions. First we work backwards to derive the value of C(G), then we prove it forward.\nAssume P_G=P_data (deriving backwards from the known result). We can derive:\nV(G,DG∗)=∫xpdata(x)log⁡12+pG(x)log⁡(1−12)dx V \\left( G , D _ { G } ^ { * } \\right) = \\int _ { x } p _ { d a t a } ( x ) \\log \\frac { 1 } { 2 } + p _ { G } ( x ) \\log \\left( 1 - \\frac { 1 } { 2 } \\right) \\mathrm { d } x V(G,DG∗)=−log⁡2∫xpG(x)dx−log⁡2∫xpdata(x)dx=−2log⁡2=−log⁡4 V \\left( G , D _ { G } ^ { * } \\right) = - \\log 2 \\int _ { x } p _ { G } ( x ) \\mathrm { d } x - \\log 2 \\int _ { x } p _ { d a t a } ( x ) \\mathrm { d } x = - 2 \\log 2 = - \\log 4 So -log4 is a candidate for the minimum, since it arises only when P_G=P_data. Now we must prove forward that this value is always the minimum, i.e., that the if-and-only-if condition is fully satisfied.\nNow drop the condition P_G=P_data, pick any G, and rewrite the formula as:\nC(G)=∫xpdata(x)log⁡(pdata(x)pG(x)+pdata(x))+pG(x)log⁡(pG(x)pG(x)+pdata(x))dx C ( G ) = \\int _ { x } p _ { d a t a } ( x ) \\log \\left( \\frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) + p _ { G } ( x ) \\log \\left( \\frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) \\mathrm { d } x Next we apply a trick: add a zero to the equation. It does not change the value of the equation, but it lets us construct a log2, since we know -log4 is the candidate global minimum.\nC(G)=∫x(log⁡2−log⁡2)pdata(x)+pdata(x)log⁡(pdata(x)pG(x)+pdata(x))+(log⁡2−log⁡2)pG(x)+pG(x)log⁡(pG(x)pG(x)+pdata(x))dx \\begin{aligned} C ( G ) \u0026amp; = \\int _ { x } ( \\log 2 - \\log 2 ) p _ { d a t a } ( x ) + p _ { d a t a } ( x ) \\log \\left( \\frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) \\\\ \u0026amp; + ( \\log 2 - \\log 2 ) p _ { G } ( x ) + p _ { G } ( x ) \\log \\left( \\frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) \\mathrm { d } x \\end{aligned} C(G)=−log⁡2∫xpG(x)+pdata(x)dx+∫xpdata(x)(log⁡2+log⁡(pdata(x)pG(x)+pdata(x)))+pG(x)(log⁡2+log⁡(pG(x)pG(x)+pdata(x)))dx \\begin{array} { c } { C ( G ) = - \\log 2 \\int _ { x } p _ { G } ( x ) + p _ { d a t a } ( x ) d x } \\\\ { + \\int _ { x } p _ { d a t a } ( x ) \\left( \\log 2 + \\log \\left( \\frac { p _ { d a t a } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) \\right) } \\\\ { + p _ { G } ( x ) \\left( \\log 2 + \\log \\left( \\frac { p _ { G } ( x ) } { p _ { G } ( x ) + p _ { d a t a } ( x ) } \\right) \\right) \\mathrm { d } x } \\end{array} Finally, simplification yields:\nC(G)=−log⁡4+∫xpdata(x)log⁡(pdata(x)(pG(x)+pdata(x))/2)dx+∫xpG(x)log⁡(pG(x)(pG(x)+pdata(x))/2)dx \\begin{aligned} C ( G ) = \u0026amp; - \\log 4 + \\int _ { x } p _ { d a t a } ( x ) \\log \\left( \\frac { p _ { d a t a } ( x ) } { \\left( p _ { G } ( x ) + p _ { \\text {data} } ( x ) \\right) / 2 } \\right) \\mathrm { d } x \\\\ \u0026amp; + \\int _ { x } p _ { G } ( x ) \\log \\left( \\frac { p _ { G } ( x ) } { \\left( p _ { G } ( x ) + p _ { d a t a } ( x ) \\right) / 2 } \\right) \\mathrm { d } x \\end{aligned} If you have read the earlier section on KL divergence, you will notice that this can be simplified into the form of KL divergences:\nC(G)=−log⁡4+KL(pdata∣pdata+pG2)+KL(pG∣pdata+pG2) C ( G ) = - \\log 4 + K L \\left( p _ { d a t a } | \\frac { p _ { d a t a } + p _ { G } } { 2 } \\right) + K L \\left( p _ { G } | \\frac { p _ { d a t a } + p _ { G } } { 2 } \\right) Since the KL divergence is non-negative, -log4 is the global minimum.\nIt remains to show that only one G can attain this value, so that P_G=P_data is the unique solution, completing the proof.\nFrom the earlier discussion we know the KL divergence is asymmetric and can only measure the similarity of distribution a with respect to distribution b. But after adding the second term, their sum becomes symmetric, and this sum of the two terms can be expressed as the JS divergence:\nJSD⁡(P∥Q)=12D(P∥M)+12D(Q∥M) \\operatorname { JSD } ( P \\| Q ) = \\frac { 1 } { 2 } D ( P \\| M ) + \\frac { 1 } { 2 } D ( Q \\| M ) M=12(P+Q) M = \\frac { 1 } { 2 } ( P + Q ) Suppose there are two distributions P and Q whose average distribution is M=(P+Q)/2; then the JS divergence between the two distributions is the KL divergence between P and M plus the KL divergence between Q and M, divided by 2.\nTherefore, the JS divergence ranges from 0 to log2. It equals log2 when the two distributions have no overlap at all, and reaches the minimum 0 when they are identical.\nHence C(G) can be rewritten as:\nC(G)=−log⁡4+2⋅JSD(p data ∣pG) C ( G ) = - \\log 4 + 2 \\cdot J S D \\left( p _ { \\text { data } } | p _ { G } \\right) This proves that the JSD is 0 when P_G=P_data. In summary, we obtain the optimal generator if and only if the generated distribution equals the true data distribution.\nConvergence Regarding whether the training process converges to the optimal generator, the original paper gives an additional proof that, with sufficient training data and sufficient capacity of d and g, training converges to the optimal G. Since this part is not particularly important, I will put the proof in the appendix (mainly because my own understanding of it is only partial).\nTraining Procedure Parameter optimization process If we want to find the optimal generator, then after fixing a discriminator D, we can treat the original value function as the loss function L(G) for training the generator. With a loss function, we can update the generator using optimization algorithms such as SGD or Adam. The gradient descent update is as follows:\nθG←θG−η∂L(G)/∂θG \\theta _ { G } \\leftarrow \\theta _ { G } - \\eta \\partial L ( G ) / \\partial \\theta _ { G } Now, given an initial G_0, we need to find the D_0* that maximizes V(G_0,D); thus the discriminator\u0026rsquo;s update process amounts to minimizing the loss function -V(G,D). Moreover, from the earlier derivation, V(G,D) differs from the JS divergence between the distributions P_data(x) and P_G(x) only by a constant term. Therefore this alternating adversarial process can be described as:\nGiven G_0, maximize V(G_0,D) to obtain D_0*, i.e., max[JSD(P_data(x)||P_G0(x))]; Fix D_0*, compute θ−G1←θ−G0−η(∂V(G,D−0∗)/∂θ−G)\\mathrm { \\theta } _ { - } \\mathrm { G } 1 \\leftarrow \\theta _ { - } \\mathrm { G0 } - \\mathrm { \\eta } \\left( \\partial \\mathrm { V } \\left( \\mathrm { G } , \\mathrm { D } _ { - } \\mathrm { 0 } ^ { * } \\right) / \\partial \\theta _ { - } \\mathrm { G } \\right) to obtain the updated G_1; Fix G_1, maximize V(G_1,D_0*) to obtain D_1*, i.e., max[JSD(P_data(x)||P_G1(x)]; Fix D_1*, compute θ−G2←θ−G1−η(∂V(G,D−0∗)/∂θ−G)\\theta _ { - } \\mathrm { G } 2 \\leftarrow \\theta _ { - } \\mathrm { G } 1 - \\eta \\left( \\partial \\mathrm { V } \\left( G , D _ { - } \\mathrm { 0 } ^ { * } \\right) / \\partial \\theta _ { - } \\mathrm { G } \\right) to obtain the updated G_2; And so the loop continues.\nPractical training process According to the definition of the value function V(G,D) above, we need to compute two expectations, E[log(D(x))] and E[log(1-D(G(z)))], where x follows the real data distribution and z follows the initialization distribution. In practice, there is no way to compute these two expectations by integration, so we generally approximate them by sampling from the infinite real data and an infinite generator.\nSuppose the generator G is given and we wish to compute maxV(G,D) to find the discriminator D. First we need to sample m samples from P_data(x) and m samples from the generator P_G(x). Maximizing the value function can then be replaced by the following expression:\nMaximize V~=1m∑i=1mlog⁡D(xi)+1m∑i=1mlog⁡(1−D(x~i)) \\text { Maximize } \\tilde { V } = \\frac { 1 } { m } \\sum _ { i = 1 } ^ { m } \\log D \\left( x ^ { i } \\right) + \\frac { 1 } { m } \\sum _ { i = 1 } ^ { m } \\log \\left( 1 - D \\left( \\tilde { x } ^ { i } \\right) \\right) Now we treat the samples drawn from P_data(x) as positive samples and those drawn from P_G(x) as negative samples, while using an approximation of the negative V(G,D) as the loss function. This can therefore be formulated as the standard training procedure of a binary classifier:\nMinimize L=−1m∑i=1mlog⁡D(xi)−1m∑i=1mlog⁡(1−D(x~i)) \\text { Minimize } L = - \\frac { 1 } { m } \\sum _ { i = 1 } ^ { m } \\log D \\left( x ^ { i } \\right) - \\frac { 1 } { m } \\sum _ { i = 1 } ^ { m } \\log \\left( 1 - D \\left( \\tilde { x } ^ { i } \\right) \\right) In practice, we must implement the minimax game with iterative and numerical methods. Fully optimizing D in the inner loop of training is computationally prohibitive, and a finite dataset would lead to overfitting. Therefore, we can alternate between k steps of optimizing D and one step of optimizing G. As long as G is updated slowly, D stays near its optimal solution.\nIn summary, over the whole training procedure, for each iteration:\nDraw m samples from the real data distribution P_data Draw m noise samples from the prior distribution P_prior(z) Feed the noise samples into the generator G to produce data {x~1,x~2,…,x~m},x~i=G(zi)\\left\\{ \\tilde { x } ^ { 1 } , \\tilde { x } ^ { 2 } , \\ldots , \\tilde { x } ^ { m } \\right\\} , \\tilde { x } ^ { i } = G \\left( z ^ { i } \\right), and update the discriminator parameters θ−d\\theta _ { - } d by maximizing the approximation of V; the discriminator\u0026rsquo;s update rule is θd←θd+η∇V~(θd)\\theta _ { d } \\leftarrow \\theta _ { d } + \\eta \\nabla \\tilde { V } \\left( \\theta _ { d } \\right) The above is the process of learning the discriminator D. The process computes the JS divergence and is repeated k times, because we want to maximize the value function.\nDraw another m noise samples from the prior distribution P_prior(z) Update the generator by minimizing V, i.e., V~=1m∑i=1mlog⁡(1−D(G(zi)))\\tilde { V } = \\frac { 1 } { m } \\sum _ { i = 1 } ^ { m } \\log \\left( 1 - D \\left( G \\left( z ^ { i } \\right) \\right) \\right), with the generator\u0026rsquo;s update rule θg←θg−η∇V~(θg)\\theta _ { g } \\leftarrow \\theta _ { g } - \\eta \\nabla \\tilde { V } \\left( \\theta _ { g } \\right) The above is the learning process of the generator\u0026rsquo;s parameters. This process occurs only once per iteration, which avoids excessive updates that would make the JS divergence rise.\nThis concludes the complete derivation and argumentation of GAN.\nSeveral Issues in GAN Training Training Instability Training the original GAN is very difficult. This mainly manifests as the training process failing to converge, or the trained generator being unable to produce meaningful content, among other issues. On the other hand, although our optimization objective is the JS divergence, which should reflect the distance between the two distributions, ideally this distance should be relatively large at the start and gradually shrink as the training of G progresses.\nIn practice, the discriminator\u0026rsquo;s loss function very easily drops to 0 and then stays at 0 thereafter. The JS divergence measures the distance between two distributions, but in practice two situations can cause the JS divergence to deem the distance between the two distributions infinite, making the loss function forever 0.\nCase 1: the discriminator D is too strong, causing overfitting.\nSolution: try regularization, or reduce the number of model parameters.\nCase 2: properties of the data itself. The low-dimensional manifold produced by the generator indeed does not overlap easily with the data distribution.\nSolution: one option is to add noise to the data so that the generator\u0026rsquo;s distribution and the real data distribution overlap more easily. Another is the GAN to be discussed next time.\nMode Collapse All outputs are the same! This phenomenon is known as Mode Collapse. A likely cause is that the real data has large probability mass in many places in the space, but our generative model has not directly learned the characteristics of the real distribution. To guarantee minimal loss, it would rather always produce the same but certainly correct output than try other, different but possibly wrong outputs. In other words, our generator sometimes cannot cover all internal modes of the data distribution and conservatively picks just one mode that is certainly correct.\nSummary GAN combines a generative model with a discriminative model, eliminating the difficulty of defining a loss function for generative models It operates on probability distributions and is not constrained by the dimensionality of the generation It can be used for semi-supervised learning ","permalink":"https://blog.ereebay.me/en/posts/gan-notes/","summary":"\u003ch1 id=\"generative-adversarial-network\"\u003eGenerative Adversarial Network\u003c/h1\u003e\n\u003ch2 id=\"overview-of-gan\"\u003eOverview of GAN\u003c/h2\u003e\n\u003cp\u003eThe idea behind GAN is a two-player zero-sum game in which the sum of the two players\u0026rsquo; interests is a constant. Consider arm wrestling: assuming the total space is fixed, if your strength is greater, you gain more space and I get less; conversely, if I am stronger, I gain more. But one thing is certain — our total space is fixed. This is a two-player game in which the total interest is constant.\u003c/p\u003e","title":"Notes on Generative Adversarial Networks (GAN)"}]