A new modification to Adam called ADOPT enables optimal convergence rates regardless of the β₂ parameter choice. The key insight is adding a simple term to Adam's update rule that compensates for potential convergence issues when β₂ is set suboptimally. Technical details: - ADOPT modifies Adam's update rule by introducing an additional term proportional to (1-β₂) - Theoretical analysis proves O(1/√T) convergence rate for any β₂ ∈ (0,1) - Works for both convex and non-convex optimization - Maintains Adam's practical benefits while improving theoretical guarantees - Requires no additional hyperparameter tuning Key results: - Matches optimal convergence rates of SGD for smooth non-convex optimization - Empirically performs similarly or better than Adam across tested scenarios - Provides more ...
