Hacking Local LLMs on OSX · sk heavy industries

Before lil-sis ran on a Mac Studio, this was the first time I tried running a model locally on Apple Silicon. The tool was text-generation-webui, a Gradio web UI that was aiming to be the standard for local text generation, the way Automatic1111 became the standard for Stable Diffusion. This was an M2 MacBook Pro.

Install

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
python3 -m venv venv && source venv/bin/activate
./start_macos.sh

That brings up the interface at http://127.0.0.1:7860.

Loading a model

I pulled Mistral-7B-v0.1 from Hugging Face and loaded it from the Models tab. Out of the box the generation was capped short, so the first tweak was bumping max new tokens from 200 to 500 so answers didn't get guillotined mid-sentence.

text-generation-webui Model tab with Mistral-7B loaded

Then, in the spirit of the thing, I tried one of the uncensored community models (VicUnlocked-alpaca) to see how it differed. You download a model by entering its Hugging Face name in the download box.

Downloading a model by Hugging Face name

A macOS aside

While watching disk fill up with model weights, a reminder: on modern macOS the old df numbers can mislead because of APFS snapshots and purgeable space. diskutil gives you the real picture:

diskutil info /

Takeaway

For everyday tasks a local 7B still lagged well behind GPT at the time. But the gap was closing, and the appeal of running everything offline, on your own hardware, with no API meter running, was obvious. That thread is the one I eventually pulled all the way to a dedicated box. See lil-sis. Until next time, sk out.