Model quantization based on the principle and math used ISTA-DASLab for low quants of LLMPure proof of conceptTL:DRThe custom int8 x bf16 is very close to base model for only 2.4GBThe custom GGUF is the best I've manage to get before quality take a hit and lose to closeness to base model.Based on ISTA-DAS labs Non-uniform GGUF quantizations via GSQ + RCO: per-tensor mixed precision in standard GGUF form.There is also a int8 convrot created using the same pattern, where a handfull of tensor have been "buffed" to BF16.Both models went throught some overnight gpu compute to be done and manual curating by me.I post here the 2 models I think are the closest to base model at a fraction of the weight.It's mostly experimental, can be usefull for 2 class of people<ul><li>User with "old" 4gb or 6gb gpu, This give some headroom for lora stacks and higher res and batchs</li><li>User with massive workflow who want everything to in VRAM.</li><li>People like me who just want to try ^</li></ul>

Anima Base 1.0 GSQ-RCO quantized (safetensor+gguf) proof of concept

Initial version , looking for feedback

Files with SHA256: e62d2dd90c12f65163e04173aee46007f94abca813c41a5b8ecf95017c7aa639