lil-sis active

lil-sis is the household's on-device agent. She is the always-on little sister to the main orchestrator in the agents fleet. She runs a local LLM on a Mac Studio with no cloud round-trip, and she's the one who actually executes the recurring jobs: the water monitor, the mining monitor, system-health snapshots, and service metrics.

 chat  +  scheduled crons
          │
          ▼
 OpenClaw routing layer
          │
          ▼
 local MLX server
   mlx_lm · OpenAI-compatible · on the Mac Studio GPU · fully on-device

The hardware

She runs on a Mac Studio (M3 Ultra):

• 28-core CPU (20 performance + 8 efficiency), 60-core GPU, 32-core Neural Engine
• 96 GB unified memory, ~1 TB SSD, 10 GbE, Thunderbolt 5
• The Metal wired-memory limit is raised from ~72 GB to 85 GB (iogpu.wired_limit_mb) so 70B-class 4-bit models actually fit in GPU-addressable memory.

# lift the Metal wired-memory cap so 70B-4bit models fit in GPU memory
sudo sysctl iogpu.wired_limit_mb=85000

Models are served locally through mlx_lm (an OpenAI-compatible server, Apple's MLX under the hood) and reached by everything else through the OpenClaw routing layer. To the rest of the system she looks like any other model endpoint, except she's running on the GPU in the next room.

Models we've tried

Picking her model turned into a real experiment, and the winner wasn't the "best" model. She started on a fast mixture-of-experts model, then I tried trading up to bigger dense models for answer quality, hit a different wall each time, and came back. Everything is a 4-bit quant; the 70B-class models only fit at all after raising the Metal wired-memory limit (see hardware above).

Each bigger dense model fixed the previous one's problem and exposed a new one. The format issue was solvable; the speed issue wasn't, not on a single box. That sent her back to the MoE, which is the lesson below.

The lesson: speed beats size for agentic work

The always-on crons aren't single prompts. They're agentic loops (search → read result → maybe search again → respond). A dense 70B at ~10 tokens/sec decode blew through every timeout and posted error messages to chat instead of content. The mixture-of-experts model (~3B active params per token) is 5–10× faster, and that speed margin is exactly what the loops need.

The takeaway: for agentic workloads on a single box, wall-clock-per-turn matters more than raw model quality. A smaller/MoE model that finishes the loop beats a bigger dense model that times out halfway through.

Keeping her alive

Running a local model behind a cloud-shaped agent framework surfaced a few sharp edges, each of which silently broke the Discord crons until tracked down.

Streaming heartbeats. The local server emitted SSE "keepalive" comment lines during long prompt prefill, and the streaming client stalled on them and never saw the real tokens. The fix was to drop the heartbeat writes from mlx_lm's server.py:

def keepalive_callback(processed, total):
    logging.info(f"prompt progress: {processed}/{total}")
    # removed: SSE-comment heartbeat the OpenAI-style stream client choked on
    #   if self.stream:
    #       self.wfile.write(f": keepalive {processed}/{total}\n\n".encode())
    #       self.wfile.flush()

A malformed chat template. One quantized build shipped a tool-call template with doubled braces, so every tool call came back as invalid JSON. The model copied the template verbatim. The fix is single braces:

broken:  <tool_call>\n{{"name": <fn>, "arguments": <args>}}\n</tool_call>
fixed:   <tool_call>\n{"name": <fn>, "arguments": <args>}\n</tool_call>

"Thinking" deltas. Some models stream a reasoning field the client doesn't parse, so thinking mode is disabled at the server:

mlx_lm server flag:  --chat-template-args '{"enable_thinking": false}'

Tool-call formats. Different model families emit tool calls differently, and only some produce the OpenAI-style tool_calls the router expects. This is now a hard check before any model swap. The response's tool_calls array must come back populated:

curl -s localhost:<port>/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "<served-model>",
  "messages": [{"role": "user", "content": "weather in Tokyo?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}]
}' | jq ".choices[0].message.tool_calls"

Lesson learned the hard way: swapping lil-sis's model is never a one-line change. The model id, tool-call format, and every downstream cron all have to be checked, or Discord just goes quiet.

What's next

The open question is whether to add a second matched Mac Studio and run MLX distributed over Thunderbolt 5. That would open the door to much larger models (or 70B-class at full precision) than a single 96 GB box can hold. For now, a well-tuned single box covers the workload.

Stack: Apple MLX (mlx_lm, OpenAI-compatible server), a Mac Studio (M3 Ultra), OpenClaw routing, cron.