plinear 0.1.3.3

Description:

plinear 0.1.3.3

plinear
Github for parrallel - linear layer
You can install PLinear using pip:
pip install plinear

Idea inspired from
https://arxiv.org/pdf/2402.17764?trk=public_post_comment-text
Code inspired from
https://github.com/kyegomez/BitNet/blob/main/bitnet/bitlinear.py
Ideas and Road Map
Parrallel neural network (PLinear)
Layer composition
Binarizing ternary layers by making posNet and negNet and compute them.
Both are created with posNet, which returns 1 if the weight if over 0 and 0 else.
Result comes out with posNet - negNet to mimic ternary.
Found out that tanh(weight) makes the model to fit in and learn without normalizing entire layer.
No additional activation function used in test.
tanh makes learning much more stable (quite stunning if you see the curve.)


W_neg is calculated just as W_pos

Learning w' - w as bitNet does

Example usage
import torch
import torch.nn as nn
from plinear import PLinear

class SimpleNN(nn.Module):
def __init__(self):
super(SimpleNN, self).__init__()
self.fc1 = PLinear(28*28, 128)
self.fc2 = PLinear(128, 10)

def forward(self, x):
x = torch.flatten(x, 1)
x = self.fc1(x)
x = self.fc2(x)
return x

Test code for Mnist example (Should clone whole git)
pytest -k mnist -s

Results can be found in tests/result_mnist.
You are offered with precision, accuracy, recall per epochs.
Also confusion matrix and full visualization of weights per epochs will be offered in animation.
Complex layers
parrellized 4 binarized nn.Linears (using same algorithms with PLinear).
2 for real and 2 for complex.
Real and Complex Results
Real Result

Complex Result

Notations

P.S.
I used torch.zeros if there is no complex input to feed.
pretty lovely result comes out, I suggest you to try this.
Example usage
import torch
import torch.nn as nn
from plinear import PLinear_Complex as PL

class SimpleNN(nn.Module):
def __init__(self):
super(SimpleNN, self).__init__()
self.complex = torch.zeros(28*28, 1)
self.fc1 = PL(28*28, 128)
self.fc2 = PL(128, 10)

def forward(self, x):
real = torch.flatten(x, 1)
real, complex = self.fc1(real, complex)
real, complex = self.fc2(real, complex)
return real

Test code for Mnist example (Should clone whole git)
pytest -k mnist_c -s

visualization (Not finised for documentation)
Exhaustive search of 3 x 3 CNN (Only Idea)
Since I parrallelized layers, each 3 x 3 CNN layer can be brute forced in 2^9 * 2 weights for ternary, which is very cheap against previous models.
Even the model is same, the layer is still at least 9 times smaller even if the model seeked through every cases.
And we can reduce the model with simple searching tasks.
I believe this can be used to vectorize images in proper size of vector which can be reused for image generation or more.
I guess vectorizing concepts and dynamically allocating them with layers would be the final goal of this project.
Developer Note
15, July, 2024
Checked plinear works on colab
16, July, 2024
version 0.1.2.2.
version 0.1.2.3.
19, July, 2024
version 0.1.3.0 - PLinear_Complex
fixed some documentaion
14, Auguest, 2024
version 0.1.3.1 - Minor Tester change
changelog
0.1.2.2.
Documented readme.md
Preflight testing done for mnist both layer and visualization.
0.1.2.3
Integrated posNet, negNet functions to posNet.
Layer now does posNet - negNet instead of posNet + negNet since negNet is not negative in real now.
Weight is now processed with tanh and shows much stable learning curve.
Removed test result from the git.
0.1.3.0
Complex layer created and tested on MNIST
mnist run case fixed to show result through cmd
0.1.3.1
Teseted SAM optimizer and somehow it did not work well.
Found out that square models way better than non-square models.
Hypothesis
While backpropagation if the tensor got smaller, the change should spread thinner.
Need compensation since unlike traditional linear, quantized models does not show the change until it passes threshold.
0.1.3.2
Minor instalation version check.
pytorch 2.2.2 for darwin (intel Macos)
0.1.3.3
plinear device setting added