ggufy: 100 stars, convert text models! Plus TensorPencil update
I woke up this morning and did my normal page refreshes, and realized ggufy has reached 100 stars on GitHub! Thanks for all the support and interest. I'm glad it's been a useful tool.
I posted a new ggufy release yesterday, 0.8.0, which supports the conversion of a limited set of LLM (text generation) families. This includes multiple models that are used as text encoders for diffusion models. See the release notes for a more detailed description.
This release also includes a fix for MageFlow model conversion - ggufy was quantizing some parts that shouldn't be, which was resulting in a loss of precision.
TensorPencil
I've been spending a lot of time with TensorPencil as well. If you're not aware, TensorPencil is a desktop program that can be used to chat with a local LLM model and generate images.
Last update I said I hoped to get a beta build available... Well, that didn't happen. I got sidetracked. I did put in a lot of things of varying usefulness, though.
The biggest one is multi-host! This took a lot of refactoring... You can run tp-serve on a remote headless (no display or GUI) server, connect to it with tp-gui running on your own machine, and it will show a meter bar at the bottom for the remote host. You can then run the gui and it will dispatch work to that remote host.
You can also connect multiple hosts at once! In that case, you select a single host to act as the LLM or chat host, and image generation jobs will be automatically balanced between all available hosts. Your local machine will also run a copy of tp-serve and automatically connect to it, even if you aren't using any other hosts; this keeps the code more consistent and modular.
Lydia doesn't have a lot of VRAM...
For a host to be able to generate images, it has to have all the right models available on that host. Each host will display on its meter if it is missing any needed files for the models you have selected, and there is an option in the settings page to transfer any needed models to the remote host. You can also see a grid of all models available, which hosts they are available on, and buttons to copy them between hosts.
Aside from multi-host, I also got in a lot of improvements to H3 video generation. This is still CLI only for now; I haven't worked out a good user experience for it in the GUI yet. The VRAM and RAM usage have dropped (H3 is one of the very few parts of TensorPencil that uses RAM directly), and the generation speed has improved. I also made improvements to how H3 uses the turbo LoRA on a quantized base model such as Int8, and it now will generate more accurate results than ComfyUI does due to the way ComfyUI uses quantized activations for some sections that result in degradation.
TensorPencil now has support for MageFlow, and the interesting MageFlow Compressed architecture that this model uses.
I also made the tp-gui executable a lot smaller by improving the way I'm embedding fonts (the executable is now, like, 50% fonts instead of 80% fonts or something like that. Multi-lingual fonts are big).
I did some general cleanup and consolidation as well, I'm continuing to improve the way the backend-specific code is generated so there's less code duplication across the different backends.
I also spent probably way too much time playing around with some weird ideas i wanted to try out with generating noise and controlling direction of diffusion, in both the text encoder and the diffusion steps. They were interesting to play with but ultimately didn't achieve the effect I wanted.
I do hope to get a beta build of TensorPencil out soon(ish) so other people can start using it and giving me feedback. Sorry it's taking longer than planned! The parts that need to happen for me to do that: Make sure it's building and running correctly on Windows, and make sure it works on AMD cards with Vulkan. I do have a Windows 10 computer available that I can run stuff on, I just need to make time to get it done.
Long term, I would like to fold the useful parts of TensorPencil GUI into the rebuilt DiffKeep, and release that. In that event, DiffKeep would be the main place to do generation, and then the TensorPencil GUI would be the place to do experiments and testing.