/explore
|
You are here 
|
fa.bianp.net | ||
| | | | |
justindomke.wordpress.com
|
|
| | | | | In 2012, I wrote a paper that I probably should have called "truncated bi-level optimization". I vaguely remembered telling the reviewers I would release some code, so I'm finally getting around to it. The idea of bilevel optimization is quite simple. Imagine that you would like to minimize some function $latex L(w)$. However, $latex L$... | |
| | | | |
teddykoker.com
|
|
| | | | | Gradient-descent-based optimizers have long been used as the optimization algorithm of choice for deep learning models. Over the years, various modifications to the basic mini-batch gradient descent have been proposed, such as adding momentum or Nesterov's Accelerated Gradient (Sutskever et al., 2013), as well as the popular Adam optimizer (Kingma & Ba, 2014). The paper Learning to Learn by Gradient Descent by Gradient Descent (Andrychowicz et al., 2016) demonstrates how the optimizer itself can be replaced with a simple neural network, which can be trained end-to-end. In this post, we will see how JAX, a relatively new Python library for numerical computing, can be used to implement a version of the optimizer introduced in the paper. | |
| | | | |
dustintran.com
|
|
| | | | | I went to a talk by Roger Grosse today, who was presenting his very recent work on a new SGD procedure called K-FAC [2]. Using the correct parameterization d... | |
| | | | |
www.hamza.se
|
|
| | | A walkthrough of implementing a neural network from scratch in Python, exploring what makes these seemingly complex systems actually quite straightforward. | ||
