Posts

Decoupled Neural Interfaces using Synthetic Gradients Abstract Training neural networks typically requires pushing the computation graph forward and then backpropagating errors to update the weights. In this sense, all layers of the network are locked, because they must wait for other parts of the network to complete their forward inference and backward pass before they can be updated. In this work, we introduce models of future computation in the network graph, in order to decouple modules and thereby break this constraint. These models use only local information to predict the outcome of a subgraph. In particular, we focus on modeling the error gradients: by replacing the true backpropagated error gradients with modeled synthetic gradients, subgraphs are decoupled and can be updated independently and asynchronously—namely, we realize decoupled neural interfaces. We present results for feed-forward models in which every layer is trained asynchronously; for RNNs, where predicting the future gradients of one layer extends the effective modeling horizon of the RNN; and for hierarchical RNN systems with different timescales. Finally, we show that, beyond predicting gradients, the same framework can also be used to predict inputs, resulting in models that are decoupled in both the forward and backward passes—equivalent to two independent networks that can learn together so as to be composed into a single functional network. ...

Stagewise Knowledge Distillation Abstract Most modern deep learning models require high computational power, but embedded devices lack such computing capability. Therefore, for this kind of device, models that can reduce computation while maintaining performance are very important. Knowledge distillation is one of the methods that solves this kind of problem. Traditional knowledge distillation methods directly transfer knowledge from the teacher to the student in a single stage. We propose a stagewise training approach to improve knowledge transfer. This approach can even use only a portion of the data used to train the teacher model without affecting performance. This approach can complement other model compression techniques, and can even be regarded as a universal model compression technique. ...

Federated Meta-Learning with Fast Convergence and Efficient Communication Abstract This paper proposes a federated meta-learning framework, FedMeta, which shares a parameterized algorithm (meta learner) instead of the previous global model. It is evaluated on the LEAF datasets and a real-world dataset, and demonstrates that the communication cost required by FedMeta is reduced by 2.82-4.33 times, that it converges faster, and that accuracy is even improved by 3.23~14.84 percentage points compared with FedAvg. Moreover, FedMeta preserves user privacy, since only the parameterized algorithm is shared and no data. ...

CS330 lecture 1&2 notes Informal Problem Definitions The multi-task learning problem: Learn all of the tasks more quickly or more proficiently than learning them independently. The meta-learning problem: Given data/experience on previous tasks, learn a new task more quickly and/or more proficiently. Multi-Task Learning Basics Traditional single-task learning: D={(x,y)k}min⁡θL(θ,D) \begin{array}{l}{\mathscr{D}=\left\{(\mathbf{x}, \mathbf{y})_{k}\right\}} \\ {\min _{\theta} \mathscr{L}(\theta, \mathscr{D})}\end{array} Typical loss: negative log likelihood L(θ,D)=−E(x,y)∼D[log⁡fθ(y∣x)] \mathscr{L}(\theta, \mathscr{D})=-\mathbb{E}_{(x, y) \sim \mathscr{D}}\left[\log f_{\theta}(\mathbf{y} | \mathbf{x})\right] What’s a task? A task: Ti≜{pi(x),pi(y∣x),Li}\mathscr{T}_{i} \triangleq\left\{p_{i}(\mathbf{x}), p_{i}(\mathbf{y} | \mathbf{x}), \mathscr{L}_{i}\right\} ...

A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms Abstract This paper proposes exploiting a meta-learning objective that maximizes the speed of transfer to a changed distribution in order to learn how to acquire knowledge in a modular fashion. In particular, we care about how to factor the joint distribution into appropriate conditionals consistent with the causal relationships. If this is achievable, changes in the distribution are assumed to be localized (distributions are localized) (e.g., one of the marginals changes due to an intervention on one of the variables). We show that, under such localized changes of the assumed causal mechanisms, the correct causal graph will tend to have only a few parameters with non-zero gradients, i.e., the parameters that need to be adjusted (the parameters of the modified variables). This is observed experimentally to lead to faster adaptation, and we exploit this property to define a meta-learning surrogate score which, in addition to a continuous parametrization of the graph, will also favor the correct causal graph. Finally, considering AI agents (e.g., a robot autonomously discovering its environment), we consider how the same objective can discover the causal variables themselves, since the observed low-level variables carry no causal meaning. Experiments on the bivariate case validate the proposed ideas and theoretical results. ...