Peter Shell @Peter__Shell

@liquidai The new LFM3 family is awesome for on-device AI! I just finished benchmarking it on @MortiseApp, and here's the results:

liquid.ai/blog/lfm3 @liquidai This is big news! We're wiring this up for testing now. I'd be very happy if this out performs Qwen 4B. I've been looking for a tool-capable small model to bundle with @MortiseApp. I hope lfm3 is the one.

BENCHMARKED: LFM3-1.7B by @liquidai. Liquid AI published 1.6 GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.

Pratik Gandhi @pratikg

a day ago

JUST IN: Liquid AI just dropped LFM3, six open-weight 2B-class models for text, vision, audio and omni, with MLX weights on day one. The 1.7B is about 1GB at 4-bit with 256K context. By my napkin math the whole family fits on my 64GB M5 Pro Mac mini at once. Benching tok/s

3 5 82 33K 43

Taking some time off this week and spending it in the mountains. I think I’ve finally got the @liteoutdoors stove figured out.

keyboard_arrow_left Previous keyboard_arrow_right Next

Filling out my ballot for Washington State this morning, and I asked @MortiseApp and Grok the same question about Initiative Measure No. IL26-001 by snapping a photo within each app and asking “is this passage misleading”. Both answered no. Mortise took 17.2s while Grok used 80.46. Notably, Grok answered with more context. Locking down the LLM to the device does have its disadvantages.

keyboard_arrow_left Previous keyboard_arrow_right Next

It’s been a good week for development, and,thanks to @claudeai, building a wall in my garage. I think my test rings are approaching 60 hours of harness testing this week, but I’ve been able to spend the majority doing construction. Make a couple decisions and it go, then iterate. This week’s updates to Claude code are amazing.

I’ve been working on adding Excel and .csv integration into @MortiseApp this week. The harnesses is built to verify any information pulled from a file, and these are no exception. When I’m looking at the trace, I can see that some models are actually summing numbers in the Excel files to answer the question when I want them to find the summed number in the file. It’s an interesting problem to have, and I was thinking about it as my Tesla drove me somewhere in Seattle and picked the wrong lane. I was quickly reminded that AI is probabilistic, which left me wondering about how hard to fight the model. It’s doing exactly what its engineers intended.

Mostly the model skipping the tool, plus some bad targeting. "Documents are attached to this conversation" reads as "already in my context," so Qwen3 4B said "none of the documents mention it" with zero tool calls. Overcorrecting ("you can't see them, don't ask permission") made Qwen3 8B refuse. What worked was saying both things plainly: you can search them, and search_document is the only way to read them. Short beat long, 6/6 calls vs 5/15. Every bad call was the model picking a document. On a multi-doc corpus we stopped letting it choose and always search everything. Caveats: small n, and the 0/3 → 6/6 was those changes together, so "single instruction" overstated it. The stronger prompt also over-searches, so we keep negative controls.

Since @typesafeai dropped Jev, and the follow-on open-weight models from @Cloudflare and @Strands, I've been considering using Clef or Strands Decider to augment the other open-weight models @MortiseApp uses for generation. We're only on device, so memory is precious. Based on size alone, Strands Decider was the obvious choice. I'm not shipping it. Here's why. Mortise holds one chat model in memory at a time, on purpose. Swapping models means a full cold reload, which we've measured at 1 to 2 seconds even for small models on a Mac, and more on a phone. So a decider can't be swapped in per question. It has to stay loaded next to the chat model, all the time. Here's what that costs. Strands Decider at 8-bit, which is the precision I'd trust, is about 1.9 GB. On a 16 GB Mac with an 8B chat model, that fits comfortably. On an 8 GB Mac, a 4B chat model plus the decider lands around 6 GB against roughly 5.6 GB the GPU can use, which is over the line. On most iPhones, the room that just fits a good 4B chat model today wouldn't hold both. The chat model would drop a tier to make space, and every answer you get would be a little worse so a few edge cases could be a little better. Clef is bigger still: it peaks at 7 GB, so it can't run on any iPhone at all. The performance side points the same way. A decider adds a model pass before the real one, and a document question already spends about half its time just reading the prompt. We'd be paying for memory and time, on every question, to improve decisions that are rarely wrong. And the existing models have turned out to be better at this than we assumed. One of ours went from 0 of 3 searches to 6 of 6 after we rewrote a single instruction in its prompt. That fix cost nothing at runtime. When I mapped it out, the real value was concentrated in two or three decisions that a pattern-matcher can't make today. Paraphrases. Other languages. A question that's clearly about something other than your documents. That's real, but it's a small prize for a permanent memory bill. The models in Mortise that can't call tools on their own are the ones a decider would help most, and most of them exist for small devices with no room for a second model. The people who'd benefit most are the people who can't afford it. We've also seen this idea lose before. Earlier we tested a router: a classifier that sends each question to whichever model is best at that kind of question. On the first test it looked great. Then we left each question out of the tuning and scored it, and it did worse than just using the best single model. We re-ran it on a larger, harder set of questions and the case against it got stronger. The "best model per question type" didn't hold up on new questions. A decider is a faster, better-calibrated classifier. The classifier was never the problem. The ceiling is. So here's what I'm doing instead: choosing a good default model for each device, from a ranking run on real hardware. Keeping the prompts short. Fixing the narrow cases directly, one at a time, each with its own test. I'd rather ship an app that spends your phone's memory on the model you talk to than on its own bookkeeping. If I'm wrong, I'll measure it and say so.

Mortise: an AI chat app with no cloud Every chat app you've used sends your words to a server. Mortise doesn't. There's no server for it to send them to. This is how that works. Open-weight models A language model is a big file of numbers, called weights. Some companies keep theirs locked behind an API. Others, including Qwen, Llama, Gemma, Phi, NVIDIA's Nemotron and Liquid AI, publish theirs for anyone to download and run. Mortise ships a curated list of these open-weight models, from small ones that fit comfortably on an iPhone to large ones for a well-equipped Mac. You download one once. After that the model lives on your device and does all the thinking there. Your prompt never leaves the machine, because nothing is listening on the other end. MLX MLX is Apple's machine-learning framework, built for Apple silicon. It uses the unified memory shared by the CPU and GPU, so a model's weights don't get copied around. Mortise runs on it natively through Swift, on iPhone, iPad and Mac. Most of the work is in making a compact model feel capable on a battery: Models are 4-bit quantized, and some larger ones 3-bit, so they fit in device memory. A prompt-prefix cache means a multi-step answer doesn't re-read everything the model has already read. Each device gets a memory budget measured on that hardware, not a flat guess. Per-model settings, such as tool format and prompt style, are declared in the catalog, so every model is handled the way it was built to be. Documents without a cloud Attach a PDF, a Word file or a slide deck and Mortise reads it on-device. It splits the file into passages, ranks them, and hands the model only the ones that matter. Embeddings come from Apple's on-device language model. Retrieval budgets are tuned per context size against real documents, and the measurements are committed to the repo. Nothing is uploaded, indexed or stored anywhere but your device. Where the network does come in We want to be exact here, because "no cloud" shouldn't be a slogan. The first model download comes from a model hub. After that, chat is offline. Online lookups against Wikipedia and Wikidata are one switch, off by default, with no accounts or API keys. How we know it isn't producing garbage Small models fail in ways that sound confident. So we test them on real hardware, the exact iPhones, iPads and Macs you'd use. A developer test suite runs every catalog model through a bank of deterministic checks: chat, reasoning, lookups, PDF, Word and slide-deck questions, whole-library knowledge bases, memory across conversations, and code correction. The checks use canary facts planted in the documents, so a pass means the model found the fact, not that it sounded plausible. One run of our strongest models finished 1,512 cases with no timeouts and no cancellations. The suite catches real problems. On older hardware it once showed a model returning token salad, which is text that looks like language and isn't. We spotted it by measuring how much of the output was coherent English. That's why the App Store only lets you download Mortise on a device that can actually run it. The latest hardware gives the best results. The app also checks itself while it runs. If an answer degenerates into repetition, or cites a figure the sources don't support, the turn retries. Every attempt, including the discarded ones, is recorded in a trace we can export and read, so we can see why an answer changed. Why this matters On-device models are smaller than the biggest cloud models, and we won't pretend otherwise. What you get in exchange is privacy that's structural instead of promised, no subscription, and a tool that works on a plane. That's Mortise: your device, your documents, your model. Get it on the App Store: apps.apple.com/us/app/mortise…

@neerajjj6785 App for me. Running multiple threads feels less taxing.

@ClaudeDevs Love working with Claude Code, but the worktree management is starting to annoy me. I noticed, over the last two weeks, I really need to manage branches and keep a close eye on the trace in case a stray worktree is created during generation.

Testing ocr integration in @MortiseApp. Early results are promising! Precision is better than 96% across the test pages and the hardness was able to quote from/show the user the scanned page.

keyboard_arrow_left Previous keyboard_arrow_right Next

Late-afternoon recovery ride listening to this podcast from @theallinpod. Maybe our nation won’t ruin our next revolution after all. On a more positive note, @MortiseApp is (finally) approved for macOS. We’ve been on iOS for months, running the same codebase, but had a tough time gaining approval for macOS. Thanks @Apple! podcasts.apple.com/us/podcast/all…

Back to work…I’ve been experimenting with syncing user created content between all the user’s devices without using an external server. The results have been promising so far. I’ve become so accustomed to background refresh, opening the app on two devices to trigger a refresh seems strange. Still working on the mechanics.

Listening to @BarrettLyon on @a16z talk about privacy and free speech. He claims owning the stack is the only way to know what’s going on with your data. I couldn’t agree more. Trying to do the same for ai with @MortiseApp podcasts.apple.com/us/podcast/the…

@SeattleParks

I’m 17-ish feet up, rewiring my garage, while I listen to the @theallinpod and the @TheChainsmokers talk about inventing their hustle. It makes me realize, I need to work on mine, and so I am.