LSQ and LSQ+

LSQ+ net or LSQplus net and LSQ net

commit log

[20260816] if you want to quantize ops like conv + linear + add + div + multiply + sub + concat + softmax + sigmoid + relu, and so on. You can write below classes to replace ops "+ - * / concat softmax relu ......", and replace ops in your network with these classes, add these classes to file quantization/lsqplus_quantize_V1.py

class QuantAdd(nn.Module):
    def __init__(self)
        self.activation_quantizerA = LSQPlusActivationQuantizer(a_bits=a_bits, all_positive=all_positive,batch_init = batch_init)
        self.activation_quantizerC = LSQPlusActivationQuantizer(a_bits=a_bits, all_positive=all_positive,batch_init = batch_init)
    def forward(self, inputA, inputC):
        A = self.activation_quantizerA(inputA)
        C = self.activation_quantizerC(inputC)
        return A + C
class QuantDiv(nn.Module):
class QuantMultiply(nn.Module):
class QuantConcat(nn.Module):
    def __init__(self)
        self.activation_quantizerA = LSQPlusActivationQuantizer(a_bits=a_bits, all_positive=all_positive,batch_init = batch_init)
        self.activation_quantizerC = LSQPlusActivationQuantizer(a_bits=a_bits, all_positive=all_positive,batch_init = batch_init)
    def forward(self, inputA, inputC, d):
        A = self.activation_quantizerA(inputA)
        C = self.activation_quantizerC(inputC)
        return torch.concat([A, C], dim = d)
class QuantRelu(nn.Module):
......

[20260730] suggestion: https://github.com/modeltc/mqbench for QAT, PTQ

[20260730] this repo just quantize the conv + linear, but actually in company or in NPU, you should quantize conv + linear + add + div + multiply + sub + concat + softmax + sigmoid + relu, and so on, so mqbench is a good choice.

2023-01-08 Dorefa and Pact, https://github.com/ZouJiu1/Dorefa_Pact
--------------------------------------------------------------------------------------------------------------
add torch.nn.Parameter .data, retrain models 18-01-2022

I'm not the author, I just complish an unofficial implementation of LSQ+ or LSQplus and LSQ，the origin paper you can find LSQ+ here arxiv.org/abs/2004.09576 and LSQ here arxiv.org/abs/1902.08153.

pytorch==1.8.1

You should train 32-bit float model firstly, then you can finetune a low bit-width quantization QAT model by loading the trained 32-bit float model

Dataset used for training is CIFAR10 and model used is Resnet18 revised

Version introduction

lsqplus_quantize_V1.py: initialize s、beta of activation quantization according to LSQ+ LSQ+: Improving low-bit quantization through learnable offsets and better initialization

lsqplus_quantize_V2.py: initialize s、beta of activation quantization according to min max values

lsqquantize_V1.py：initialize s of activation quantization according to LSQ Learned Step Size Quantization

lsqquantize_V2.py: initialize s of activation quantization = 1

lsqplus_quantize_V2.py has the best result when use cifar10 dataset

The Train Results

For the below table all set a_bit=8, w_bit=8

version	weight per_channel	learning rate	A s initial	A beta initial	best epoch	Accuracy	models
Float 32bit	-	<=66 0.1 <=86 0.01 <=99 0.001 <=112 0.0001	-	-	112	92.6	https://www.aliyundrive.com/s/6B2AZ45fFjx
lsqplus_quantize_V1	×	<=31 0.1 <=61 0.01 <=81 0.001 <112 0.0001	1	-1e-9	90	90.3	https://www.aliyundrive.com/s/FNZRhoTe8uW
lsqplus_quantize_V2	×	as before	-	-	87	92.8	https://www.aliyundrive.com/s/WDH3ZnEa7vy
lsqplus_quantize_V1	✔	as before	-	-	96	91.19	https://www.aliyundrive.com/s/JATsi4vdurp
lsqplus_quantize_V2	✔	as before	-	-	69	92.8	https://www.aliyundrive.com/s/LRWHaBLQGWc
lsqquantize_V1	×	as before	-	-	102	91.89	https://www.aliyundrive.com/s/nR1KZZRuB23
lsqquantize_V2	×	as before	-	-	69	91.82	https://www.aliyundrive.com/s/7fjmViqUvh4
lsqquantize_V1	✔	as before	-	-	108	91.29	https://www.aliyundrive.com/s/
lsqquantize_V2	✔	as before	-	-	72	91.72	https://www.aliyundrive.com/s/7nGvMVZcKp7

all

https://www.aliyundrive.com/s/hng9XsvhYru

A represent activation, I use moving average method to initialize s and beta.

LEARNED STEP SIZE QUANTIZATION
LSQ+: Improving low-bit quantization through learnable offsets and better initialization

References

https://github.com/666DZY666/micronet
https://github.com/hustzxd/LSQuantization
https://github.com/zhutmost/lsq-net
https://github.com/Zhen-Dong/HAWQ
https://github.com/KwangHoonAn/PACT
https://github.com/Jermmy/pytorch-quantization-demo