ADAM Optimizer

A Stochastic Gradient Descent method under the Machine Learning family, originally introduced in https://arxiv.org/abs/1412.6980, with:

Parameters

• $\alpha$ is the stepsize
• ${\beta }_i\in [0,1]$ is the exponential decay rates for the momentum estimates
• $\theta$ is the parameter vector to be optimized.
• $f(\theta )$ is the stochastic objective function

Algorithm

$$\begin{array}{rlccc}& \text{Variable}& \text{Initial value}& \text{Update rule}& \\ \text{Gradient}& g_t& \text{N/A}& {\nabla }_{\theta }f({\theta }_{t-1})& \\ \text{1st moment vector}& m_t& 0& {\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t& \\ \text{Bias corrected 1st moment vector}& \widehat{m_t}& 0& \frac{m_t}{1-{\beta }_t^1}& \\ \text{2nd moment vector}& v_t& 0& {\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2& \\ \text{Bias corrected 2nd moment vector}& \widehat{v_t}& 0& \frac{v_t}{1-{\beta }_t^1}& \\ \text{Estimated parameters}& {\theta }_t& \text{User given}& {\theta }_t:={\theta }_{t-1}-\alpha \frac{\widehat{m_t}}{\sqrt{\widehat{v_t}}+ϵ}& \end{array}$$

Termination condition

${\theta }_t$ has converged