GitLocker: The Coding Marketplace

Description:

pytorch optimizer 3.1.1

pytorch-optimizer

Build

Quality

Package

Status

License

pytorch-optimizer is optimizer & lr scheduler collections in PyTorch.
I just re-implemented (speed & memory tweaks, plug-ins) the algorithm while based on the original paper. Also, It includes useful and practical optimization ideas.
Currently, 75 optimizers (+ bitsandbytes, qgalore), 16 lr schedulers, and 13 loss functions are supported!
Highly inspired by pytorch-optimizer.
Getting Started
For more, see the documentation.
Most optimizers are under MIT or Apache 2.0 license, but a few optimizers like Fromage, Nero have CC BY-NC-SA 4.0 license, which is non-commercial.
So, please double-check the license before using it at your work.
Installation
$ pip3 install pytorch-optimizer

From v2.12.0, v3.1.0, you can use bitsandbytes, q-galore-torch optimizers respectively!
please check the bnb requirements, q-galore-torch installation
before installing it.
From v3.0.0, drop Python 3.7 support. However, you can still use this package with Python 3.7 by installing with --ignore-requires-python option.
$ pip install "pytorch-optimizer[bitsandbytes]"

Simple Usage
from pytorch_optimizer import AdamP

model = YourModel()
optimizer = AdamP(model.parameters())

# or you can use optimizer loader, simply passing a name of the optimizer.

from pytorch_optimizer import load_optimizer

optimizer = load_optimizer(optimizer='adamp')(model.parameters())

# if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.

from pytorch_optimizer import load_optimizer

opt = load_optimizer(optimizer='bnb_adamw8bit')
optimizer = opt(model.parameters())

Also, you can load the optimizer via torch.hub.
import torch

model = YourModel()
opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
optimizer = opt(model.parameters())

If you want to build the optimizer with parameters & configs, there's create_optimizer() API.
from pytorch_optimizer import create_optimizer

optimizer = create_optimizer(
model,
'adamp',
lr=1e-3,
weight_decay=1e-3,
use_gc=True,
use_lookahead=True,
)

Supported Optimizers
You can check the supported optimizers with below code.
from pytorch_optimizer import get_supported_optimizers

supported_optimizers = get_supported_optimizers()

Optimizer
Description
Official Code
Paper
Citation

AdaBelief
Adapting Step-sizes by the Belief in Observed Gradients
github
https://arxiv.org/abs/2010.07468
cite

AdaBound
Adaptive Gradient Methods with Dynamic Bound of Learning Rate
github
https://openreview.net/forum?id=Bkg3g2R9FX
cite

AdaHessian
An Adaptive Second Order Optimizer for Machine Learning
github
https://arxiv.org/abs/2006.00719
cite

AdamD
Improved bias-correction in Adam

https://arxiv.org/abs/2110.10828
cite

AdamP
Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
github
https://arxiv.org/abs/2006.08217
cite

diffGrad
An Optimization Method for Convolutional Neural Networks
github
https://arxiv.org/abs/1909.11015v3
cite

MADGRAD
A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic
github
https://arxiv.org/abs/2101.11075
cite

RAdam
On the Variance of the Adaptive Learning Rate and Beyond
github
https://arxiv.org/abs/1908.03265
cite

Ranger
a synergistic optimizer combining RAdam and LookAhead, and now GC in one optimizer
github
https://bit.ly/3zyspC3
cite

Ranger21
a synergistic deep learning optimizer
github
https://arxiv.org/abs/2106.13731
cite

Lamb
Large Batch Optimization for Deep Learning
github
https://arxiv.org/abs/1904.00962
cite

Shampoo
Preconditioned Stochastic Tensor Optimization
github
https://arxiv.org/abs/1802.09568
cite

Nero
Learning by Turning: Neural Architecture Aware Optimisation
github
https://arxiv.org/abs/2102.07227
cite

Adan
Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
github
https://arxiv.org/abs/2208.06677
cite

Adai
Disentangling the Effects of Adaptive Learning Rate and Momentum
github
https://arxiv.org/abs/2006.15815
cite

SAM
Sharpness-Aware Minimization
github
https://arxiv.org/abs/2010.01412
cite

ASAM
Adaptive Sharpness-Aware Minimization
github
https://arxiv.org/abs/2102.11600
cite

GSAM
Surrogate Gap Guided Sharpness-Aware Minimization
github
https://openreview.net/pdf?id=edONMAnhLu-
cite

D-Adaptation
Learning-Rate-Free Learning by D-Adaptation
github
https://arxiv.org/abs/2301.07733
cite

AdaFactor
Adaptive Learning Rates with Sublinear Memory Cost
github
https://arxiv.org/abs/1804.04235
cite

Apollo
An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization
github
https://arxiv.org/abs/2009.13586
cite

NovoGrad
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
github
https://arxiv.org/abs/1905.11286
cite

Lion
Symbolic Discovery of Optimization Algorithms
github
https://arxiv.org/abs/2302.06675
cite

Ali-G
Adaptive Learning Rates for Interpolation with Gradients
github
https://arxiv.org/abs/1906.05661
cite

SM3
Memory-Efficient Adaptive Optimization
github
https://arxiv.org/abs/1901.11150
cite

AdaNorm
Adaptive Gradient Norm Correction based Optimizer for CNNs
github
https://arxiv.org/abs/2210.06364
cite

RotoGrad
Gradient Homogenization in Multitask Learning
github
https://openreview.net/pdf?id=T8wHz4rnuGL
cite

A2Grad
Optimal Adaptive and Accelerated Stochastic Gradient Descent
github
https://arxiv.org/abs/1810.00553
cite

AccSGD
Accelerating Stochastic Gradient Descent For Least Squares Regression
github
https://arxiv.org/abs/1704.08227
cite

SGDW
Decoupled Weight Decay Regularization
github
https://arxiv.org/abs/1711.05101
cite

ASGD
Adaptive Gradient Descent without Descent
github
https://arxiv.org/abs/1910.09529
cite

Yogi
Adaptive Methods for Nonconvex Optimization

NIPS 2018
cite

SWATS
Improving Generalization Performance by Switching from Adam to SGD

https://arxiv.org/abs/1712.07628
cite

Fromage
On the distance between two neural networks and the stability of learning
github
https://arxiv.org/abs/2002.03432
cite

MSVAG
Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
github
https://arxiv.org/abs/1705.07774
cite

AdaMod
An Adaptive and Momental Bound Method for Stochastic Learning
github
https://arxiv.org/abs/1910.12249
cite

AggMo
Aggregated Momentum: Stability Through Passive Damping
github
https://arxiv.org/abs/1804.00325
cite

QHAdam
Quasi-hyperbolic momentum and Adam for deep learning
github
https://arxiv.org/abs/1810.06801
cite

PID
A PID Controller Approach for Stochastic Optimization of Deep Networks
github
CVPR 18
cite

Gravity
a Kinematic Approach on Optimization in Deep Learning
github
https://arxiv.org/abs/2101.09192
cite

AdaSmooth
An Adaptive Learning Rate Method based on Effective Ratio

https://arxiv.org/abs/2204.00825v1
cite

SRMM
Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates
github
https://arxiv.org/abs/2201.01652
cite

AvaGrad
Domain-independent Dominance of Adaptive Methods
github
https://arxiv.org/abs/1912.01823
cite

PCGrad
Gradient Surgery for Multi-Task Learning
github
https://arxiv.org/abs/2001.06782
cite

AMSGrad
On the Convergence of Adam and Beyond

https://openreview.net/pdf?id=ryQu7f-RZ
cite

Lookahead
k steps forward, 1 step back
github
https://arxiv.org/abs/1907.08610
cite

PNM
Manipulating Stochastic Gradient Noise to Improve Generalization
github
https://arxiv.org/abs/2103.17182
cite

GC
Gradient Centralization
github
https://arxiv.org/abs/2004.01461
cite

AGC
Adaptive Gradient Clipping
github
https://arxiv.org/abs/2102.06171
cite

Stable WD
Understanding and Scheduling Weight Decay
github
https://arxiv.org/abs/2011.11152
cite

Softplus T
Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM

https://arxiv.org/abs/1908.00700
cite

Un-tuned w/u
On the adequacy of untuned warmup for adaptive optimization

https://arxiv.org/abs/1910.04209
cite

Norm Loss
An efficient yet effective regularization method for deep neural networks

https://arxiv.org/abs/2103.06583
cite

AdaShift
Decorrelation and Convergence of Adaptive Learning Rate Methods
github
https://arxiv.org/abs/1810.00143v4
cite

AdaDelta
An Adaptive Learning Rate Method

https://arxiv.org/abs/1212.5701v1
cite

Amos
An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
github
https://arxiv.org/abs/2210.11693
cite

SignSGD
Compressed Optimisation for Non-Convex Problems
github
https://arxiv.org/abs/1802.04434
cite

Sophia
A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
github
https://arxiv.org/abs/2305.14342
cite

Prodigy
An Expeditiously Adaptive Parameter-Free Learner
github
https://arxiv.org/abs/2306.06101
cite

PAdam
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
github
https://arxiv.org/abs/1806.06763
cite

LOMO
Full Parameter Fine-tuning for Large Language Models with Limited Resources
github
https://arxiv.org/abs/2306.09782
cite

AdaLOMO
Low-memory Optimization with Adaptive Learning Rate
github
https://arxiv.org/abs/2310.10195
cite

Tiger
A Tight-fisted Optimizer, an optimizer that is extremely budget-conscious
github

cite

CAME
Confidence-guided Adaptive Memory Efficient Optimization
github
https://aclanthology.org/2023.acl-long.243/
cite

WSAM
Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term
github
https://arxiv.org/abs/2305.15817
cite

Aida
A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
github
https://arxiv.org/abs/2203.13273
cite

GaLore
Memory-Efficient LLM Training by Gradient Low-Rank Projection
github
https://arxiv.org/abs/2403.03507
cite

Adalite
Adalite optimizer
github
https://github.com/VatsaDev/adalite
cite

bSAM
SAM as an Optimal Relaxation of Bayes
github
https://arxiv.org/abs/2210.01620
cite

Schedule-Free
Schedule-Free Optimizers
github
https://github.com/facebookresearch/schedule_free
cite

FAdam
Adam is a natural gradient optimizer using diagonal empirical Fisher information
github
https://arxiv.org/abs/2405.12807
cite

Grokfast
Accelerated Grokking by Amplifying Slow Gradients
github
https://arxiv.org/abs/2405.20233
cite

Kate
Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
github
https://arxiv.org/abs/2403.02648
cite

StableAdamW
Stable and low-precision training for large-scale vision-language models

https://arxiv.org/abs/2304.13013
cite

AdamMini
Use Fewer Learning Rates To Gain More
github
https://arxiv.org/abs/2406.16793
cite

TRAC
Adaptive Parameter-free Optimization
github
https://arxiv.org/abs/2405.16642
cite

AdamG
Towards Stability of Parameter-free Optimization

https://arxiv.org/abs/2405.04376
cite

Supported LR Scheduler
You can check the supported learning rate schedulers with below code.
from pytorch_optimizer import get_supported_lr_schedulers

supported_lr_schedulers = get_supported_lr_schedulers()

LR Scheduler
Description
Official Code
Paper
Citation

Explore-Exploit
Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

https://arxiv.org/abs/2003.03977
cite

Chebyshev
Acceleration via Fractal Learning Rate Schedules

https://arxiv.org/abs/2103.01338
cite

REX
Revisiting Budgeted Training with an Improved Schedule
github
https://arxiv.org/abs/2107.04197
cite

WSD
Warmup-Stable-Decay learning rate scheduler
github
https://arxiv.org/abs/2404.06395
cite

Supported Loss Function
You can check the supported loss functions with below code.
from pytorch_optimizer import get_supported_loss_functions

supported_loss_functions = get_supported_loss_functions()

Loss Functions
Description
Official Code
Paper
Citation

Label Smoothing
Rethinking the Inception Architecture for Computer Vision

https://arxiv.org/abs/1512.00567
cite

Focal
Focal Loss for Dense Object Detection

https://arxiv.org/abs/1708.02002
cite

Focal Cosine
Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble

https://arxiv.org/abs/2007.07805
cite

LDAM
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
github
https://arxiv.org/abs/1906.07413
cite

Jaccard (IOU)
IoU Loss for 2D/3D Object Detection

https://arxiv.org/abs/1908.03851
cite

Bi-Tempered
The Principle of Unchanged Optimality in Reinforcement Learning Generalization

https://arxiv.org/abs/1906.03361
cite

Tversky
Tversky loss function for image segmentation using 3D fully convolutional deep networks

https://arxiv.org/abs/1706.05721
cite

Lovasz Hinge
A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
github
https://arxiv.org/abs/1705.08790
cite

Useful Resources
Several optimization ideas to regularize & stabilize the training. Most of the ideas are applied in Ranger21 optimizer.
Also, most of the captures are taken from Ranger21 paper.

Adaptive Gradient Clipping
Gradient Centralization
Softplus Transformation

Gradient Normalization
Norm Loss
Positive-Negative Momentum

Linear learning rate warmup
Stable weight decay
Explore-exploit learning rate schedule

Lookahead
Chebyshev learning rate schedule
(Adaptive) Sharpness-Aware Minimization

On the Convergence of Adam and Beyond
Improved bias-correction in Adam
Adaptive Gradient Norm Correction

Adaptive Gradient Clipping
This idea originally proposed in NFNet (Normalized-Free Network) paper. AGC (Adaptive Gradient Clipping) clips gradients based on the unit-wise ratio of gradient norms to parameter norms.

code : github
paper : arXiv

Gradient Centralization

Gradient Centralization (GC) operates directly on gradients by centralizing the gradient to have zero mean.

code : github
paper : arXiv

Softplus Transformation
By running the final variance denom through the softplus function, it lifts extremely tiny values to keep them viable.

paper : arXiv

Gradient Normalization
Norm Loss

paper : arXiv

Positive-Negative Momentum

code : github
paper : arXiv

Linear learning rate warmup

paper : arXiv

Stable weight decay

code : github
paper : arXiv

Explore-exploit learning rate schedule

code : github
paper : arXiv

Lookahead
k steps forward, 1 step back. Lookahead consisting of keeping an exponential moving average of the weights that is updated and substituted to the current weights every k lookahead steps (5 by default).
Chebyshev learning rate schedule
Acceleration via Fractal Learning Rate Schedules.
(Adaptive) Sharpness-Aware Minimization
Sharpness-Aware Minimization (SAM) simultaneously minimizes loss value and loss sharpness.
In particular, it seeks parameters that lie in neighborhoods having uniformly low loss.
On the Convergence of Adam and Beyond
Convergence issues can be fixed by endowing such algorithms with 'long-term memory' of past gradients.
Improved bias-correction in Adam
With the default bias-correction, Adam may actually make larger than requested gradient updates early in training.
Adaptive Gradient Norm Correction
Correcting the norm of a gradient in each iteration based on the adaptive training history of gradient norm.
Frequently asked questions
here
Visualization
here
Citation
Please cite the original authors of optimization algorithms. You can easily find it in the above table!
If you use this software, please cite it below. Or you can get it from "cite this repository" button.
@software{Kim_pytorch_optimizer_optimizer_2021,
author = {Kim, Hyeongchan},
month = jan,
title = {{pytorch_optimizer: optimizer & lr scheduler & loss function collections in PyTorch}},
url = {https://github.com/kozistr/pytorch_optimizer},
version = {3.1.0},
year = {2021}
}

Maintainer
Hyeongchan Kim / @kozistr