Show HN: Shoehorn – Quantize any model down to run on your machine

rrhgraysonii about 3 hours ago 2 commentsRead Article on notactuallytreyanastasio.github.io

Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn

Discussion (2 Comments)Read Original on HackerNews

hmokiguess•about 2 hours ago

Reminds me of https://github.com/AlexsJones/llmfit

mbuchel-hn•about 2 hours ago

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?