Description:
gptq 0.0.3
🔮 GPTQ - Accurate Post-Training Compression for Generative Pretrained Transformers
This repo is a extended and polished version of the original code for the paper GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers.
🔥 SOTA on LLM PTQ
An efficient implementation of the GPTQ algorithm
2/3/4/8-bit quantized matrix full-precision vector product CUDA kernel
Bug fix for old consumer-grade GPU
📥 Installation
pip install gptq
🛟 Install PyTorch
gptq requires PyTorch and GPU, and installing PyTorch with CUDA is tricky. To install PyTorch correctly, the following steps are recommended:
run nvcc --version to get the version. For example, the following result means we have cuda compiler version 116
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
run pip install light-the-torch to install ltt
run ltt install --pytorch-computation-backend=cu116 torch torchvision torchaudio to install the torch suite. Please replace the 116 according to your environment!
TODO
GPTQ with CNN
Algorithm credits go to IST Austria Distributed Algorithms and Systems Lab
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
