precondition-opt 0.3.0

Description:

preconditionopt 0.3.0

precondition: Preconditioning Optimizers


Installation (note package name is precondition but pypi distribution name is precondition-opt):
pip3 install -U precondition-opt

Currently, this contains several preconditioning optimizer implementations. Please refer to the citations below.
Shampoo (distributed_shampoo.py)
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}

Sketchy (distributed_shampoo.py), logical reference implementation as a branch in Shampoo.
@article{feinberg2023sketchy,
title={Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions},
author={Feinberg, Vladimir and Chen, Xinyi and Sun, Y Jennifer and Anil, Rohan and Hazan, Elad},
journal={arXiv preprint arXiv:2302.03764},
year={2023}
}

SM3 (sm3.py).
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}

This external repository was seeded from existing open-source work available at this google-research repository.
This is not an officially supported Google product.