ADAM Optimizer

A Stochastic Gradient Descent method under the Machine Learning family, originally introduced in https://arxiv.org/abs/1412.6980, with:

Initialization

$$\{\begin{array}{rlc}\alpha & \in {\mathbb{R}}^{+}& \text{Stepsize}\\ {\beta }_i& \in [0,1)& \text{Exponential decay for ith moment}\\ f(\theta )& :{\mathbb{R}}^n\to \mathbb{R}& \text{Stochastic objective function}\\ m_0& =0& \text{1st moment vector}\\ v_0& =0& \text{2nd moment vector}\\ t& =0& \text{time}\end{array}$$

Iteration

$$\{\begin{array}{rlc}t& \in {\mathbb{Z}}_0^{+}& \text{Discrete time}\\ g_t& ={\nabla }_{\theta }{f}_t({\theta }_{t-1})& \text{Gradient}\\ m_t& ={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t& \\ v_t& ={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2& \\ \widehat{m_t}& =\frac{m_t}{1-{\beta }_1^t}& \\ \widehat{v_t}& =\frac{v_t}{1-{\beta }_2^t}& \\ {\theta }_t& ={\theta }_{t-1}-\alpha \frac{\widehat{m_t}}{\sqrt{\widehat{v_t}}+ϵ}& \end{array}$$

Termination condition

${\theta }_t$ has converged