Adam 与自适应学习率

Two averages of the gradient

The initialization bias

When the second moment misleads

Adam divides the momentum estimate by the square root of the second-moment estimate. Coordinate $i$ has seen consistently small gradients and coordinate $j$ consistently large ones. Which statement describes the relative step sizes, holding everything else fixed?

With $\beta_1 = 0.9$ and a first gradient $\mathbf{g}_1$, the momentum state is $\mathbf{v}_1 = (1 - \beta_1)\mathbf{g}_1$. The bias correction divides by $1 - \beta_1^{1}$, and the corrected estimate $\hat{\mathbf{v}}_1$ equals ____ in terms of $\mathbf{g}_1$.