/explore
|
You are here 
|
xcorr.net | ||
| | | | |
teddykoker.com
|
|
| | | | | Gradient-descent-based optimizers have long been used as the optimization algorithm of choice for deep learning models. Over the years, various modifications to the basic mini-batch gradient descent have been proposed, such as adding momentum or Nesterov's Accelerated Gradient (Sutskever et al., 2013), as well as the popular Adam optimizer (Kingma & Ba, 2014). The paper Learning to Learn by Gradient Descent by Gradient Descent (Andrychowicz et al., 2016) demonstrates how the optimizer itself can be replaced with a simple neural network, which can be trained end-to-end. In this post, we will see how JAX, a relatively new Python library for numerical computing, can be used to implement a version of the optimizer introduced in the paper. | |
| | | | |
fa.bianp.net
|
|
| | | | | This blog post discusses the convergence rate of the Stochastic Gradient Descent with Stochastic Polyak Step-size (SGD-SPS) algorithm for minimizing a finite sum objective. Building upon the proof of the previous post, we show that the convergence rate can be improved to O(1/t) under the additional assumption that ... | |
| | | | |
qchu.wordpress.com
|
|
| | | | | Note: this is a repost of a Facebook status I wrote off the cuff about a year ago, lightly edited. As such it has a different style from my other posts, but I still wanted to put it somewhere where it'd be easier to find and share than Facebook. Gradient descent, in its simplest where... | |
| | | | |
futurism.com
|
|
| | | This post was originally written by Manan Shah as a response to a question on Quora. | ||
