Inference engines & optimization
Inference runtimes, serving engines, quantization methods, caching, and cross-platform optimization across llama.cpp, Ollama, MLX, vLLM, and related tools.
38%
Best tweets about Local LLMs
50 Best Tweets About Local LLMs (2026)
Explore the best tweets about local LLMs, on-device AI, hardware, quantization, privacy, inference, and self-hosted model setups. Updated weekly.
Reproducible local-model setups, useful hardware comparisons, performance measurements, and privacy-focused workflows.
- Creators
- 40
- Updated
Original Xholic analysis
What 50 top Local LLMs posts reveal
The conversation is strongly supportive of private and self-hosted local-LLM workflows, while its most actionable material concerns runtime selection, hardware constraints, and measured performance. Posts also caution that local models and agent stacks can struggle with difficult coding, long contexts, or latency; several recommend testing the workflow before committing to expensive hardware.
- Dominant tone
- Positive
- Median score
- 23.2
- Leading format
- List
- Recent posts
- 36%
74% of posts
All-time engagement
100% of posts
Published in 90 days
Conversation map
The themes creators return to
Hardware & performance
Hardware selection, memory/VRAM sizing, device comparisons, pricing, power use, and performance benchmarks for local inference.
36%
Local coding agents
Local coding agents and developer harnesses, including Claude Code-compatible proxies, OpenCode, IDE integrations, and agentic coding limitations.
34%
Privacy & self-sovereignty
Offline, self-hosted AI stacks emphasizing data sovereignty, no cloud dependency, and private local workflows.
24%
Local AI applications
Practical local AI applications such as offline research, document OCR, personal knowledge retrieval, smart-home control, security labs, and automation.
22%
Edge & on-device AI
Edge and on-device deployment on phones, browsers, Raspberry Pi, IoT, and constrained low-memory hardware.
20%
Model selection & tradeoffs
Choosing and evaluating local models: quality tradeoffs, model fit, context capacity, licensing, and comparisons with frontier cloud models.
18%
Secure reproducible stacks
Secure, reproducible local-agent infrastructure: declarative setups, sandboxing, private search, tool permissions, and safe remote-model hybrids.
8%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 48
- Median reposts
- 7
- Median replies
- 7
- Median views
- 4.3K
Posts with media make up 72% of this collection. Their median all-time score is 34.0, compared with 7.36 for text-only posts.
Format mix
- List 100% · score 23.2
Consensus and debate
Where creators agree, and where they do not
Shared view
Efficiency claims broaden possible deployment targets
Posts report increasingly small-footprint inference demonstrations: one BitNet post reports a 100B model running on a CPU at 5–7 tokens per second, while another describes a 1B model streamed from an SD card on a 256MB board.
Shared view
Choose hardware around the workload
The hardware posts emphasize memory capacity, bandwidth, context length, and runtime choice as constraints. One hardware guide compares price, performance, and power across devices, while another recommends validating a workflow before buying hardware.
Shared view
Local coding relies on compatible harnesses
Several posts frame local coding as an integration task as well as a model-selection task, covering llama.cpp endpoints, Claude Code redirection, and dedicated coding harnesses.
Shared view
Privacy is tied to concrete workflows
Privacy-oriented examples include an offline reference system, a locally hosted research agent, and a portable Pi-based AI and security setup. The posts present these as ways to reduce dependence on hosted inference or keep local workflows under the operator’s control.
Open debate
Local coding usefulness remains contested
Some posts describe private local coding agents as workable on consumer hardware. Others argue that local models can be slow or less capable for complex design, debugging, and large-context agent workloads.
Open debate
Buying hardware is not a settled recommendation
Hardware-purchase recommendations differ: one post urges readers to act in response to claimed hardware scarcity, another says a $4,000 budget may be better spent on provider tokens, and a third recommends proving the workflow before purchasing.
Open debate
Capability claims invite caution
The BitNet posts report competitive accuracy and large efficiency gains, while a separate anecdote argues against treating local-model claims of frontier-level capability as established fact.
What performs
Patterns behind standout posts
Infrastructure and local-workflow posts lead the supplied score outliers
These five posts are the supplied all-time-score outliers. Their subjects span offline systems, BitNet efficiency claims, local Claude Code routing, local research, and a 27B model on a 3090 Ti.
Lists and media characterize the corpus
All 50 posts use list formatting. Media appeared in 36 posts (72%); the supplied median all-time score is 34.02 for media posts and 7.36 for text-only posts.
Throughput reporting makes configuration differences more visible
Some posts include throughput figures such as prefill and decode rates or runtime-specific token rates. The MLX versus llama.cpp comparison explicitly notes that some model and quantization comparisons are not apples-to-apples.
Performance discussion extends beyond tokens per second
Posts discuss operational factors beyond token rate, including measured GPU electricity, power consumption, memory bandwidth, KV-cache growth, and serving-engine overhead.
Statistical standouts
- View standout post 1 Score 11430.7 · 491.85× median
- View standout post 2 Score 9759.8 · 419.95× median
- View standout post 3 Score 1489.0 · 64.07× median
- View standout post 4 Score 1390.8 · 59.85× median
- View standout post 5 Score 878.7 · 37.81× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Vaishnavi
@_vmlops
2 posts
-
2. 0xSero
@0xSero
2 posts
-
3. Kshitij Mishra | AI & Tech
@DAIEvolutionHub
2 posts
-
4. Julian Goldie SEO
@JulianGoldieSEO
2 posts
-
5. Morgan
@morganlinton
2 posts
-
6. Shruti Codes
@Shruti_0810
2 posts
0xSero covers stack selection and coding harnesses
0xSero’s two posts cover framework selection by available compute—vLLM/SGLang, ExLlama, llama.cpp, and MLX—and a catalog of local-model coding harnesses.
Vaishnavi tracks proxy-based coding setups
Vaishnavi’s posts highlight Claude Code-compatible local proxies. One reports a setup progression from Ollama through llama.cpp to an MLX-native server, while the other lists proxy targets including Ollama, LM Studio, and llama.cpp.
Sudo su proposes controlled mobile-hardware tests
Sudo su documents an RTX 5090 mobile setup and states an intention to compare 27B-class models on the same hardware and quantization using agentic-coding tests.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Local LLMs tweets
Top Local LLMs tweets from 40 creators
Ranked 01–50
- 01
@alex_prompter ·
🚨 BREAKING: Someone just open-sourced a full offline survival computer with AI, Wikipedia, and maps built in. Project N.O.M.A.D. is an open-source offline survival computer. Self-contained. Zero internet required after install. Zero telemetry. Everything runs locally on your hardware. What it includes: → Full Wikipedia archives via Kiwix → Offline maps via OpenStreetMap → Local AI models via Ollama + Open WebUI → Calculators, reference tools, resource libraries → A management UI to control everything from a browser One curl command installs the entire system on any Debian-based machine. Runs headless as a server so any device on your local network can access it. Minimum specs to run the base system: dual-core processor, 4GB RAM, 5GB storage. To run local LLMs offline, you want 32GB RAM and an NVIDIA RTX 3060 or better. No accounts. No authentication by default. No cloud dependency. No phone-home behavior. Built to function when nothing else does. The grid, the cloud, the API you depend on. None of it is guaranteed. The people building local-first systems right now are the ones who won’t be asking for help when access disappears.
- 02
@heygurisingh ·
Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.
Watch video
- 03
@techNmak ·
Claude Code can run entirely on your local GPU now. Unsloth AI published the complete guide. The setup itself is straightforward - llama.cpp serves Qwen3.5 or GLM-4.7-Flash, one environment variable redirects Claude Code to localhost. But the guide is valuable because of what it explains beyond the setup: Why local inference feels impossibly slow: Claude Code adds an attribution header that breaks KV caching. Every request recomputes the full context. The fix requires editing settings.json - export doesn't work. Why Qwen3.5 outputs seem off: f16 KV cache degrades accuracy, and it's llama.cpp's default. Multiple reports confirm this. Use q8_0 or bf16 instead. Why responses take forever: Thinking mode is great for reasoning but slow for agentic tasks. The guide shows how to disable it. The proof it all works: Claude Code autonomously fine-tuning a model with Unsloth. Start to finish. No API dependency. Fits on 24GB. RTX 4090, Mac unified memory.
- 04
@ihteshamali ·
This feels illegal. Someone built a fully local deep research agent that writes its own search queries, hunts down sources, finds the gaps in its own answers, then keeps searching until it's done. It's called Local Deep Researcher. Drop in any Ollama model like DeepSeek, Llama, Qwen and give it a topic. It handles everything: → Generates the search query → Scrapes the web → Summarizes what it found → Identifies what it missed → Searches again to fill the gaps → Outputs a full markdown report with citations You set how many research loops it runs. The whole thing costs $0 to operate after setup. Perplexity Pro is $20/month for this exact workflow. 8,500 developers already starred it. 100% Open Source. MIT License.
Watch video
- 05
@stan_info ·
i’m not a local llm type of guy at all, was just curious and decided to mess around… ended up running a full uncensored qwen3.5-27b (abliterated) on my single 3090 ti with 262k context + tool calling. threw a cloudflare tunnel on it so i can hit the api from anywhere. huge thanks to @0xSero and @ggerganov for the insane work that made this possible oh my… intelligence will be everywhere soon
- 06
@ollama ·
Ollama 0.18.1 is here! 🌐 Web search and fetch in OpenClaw Ollama now ships with web search and web fetch plugin for OpenClaw. This allows Ollama's models (local or cloud) to search the web for the latest content and news. This also allows OpenClaw with Ollama to be able to fetch the web and extract readable content for processing. This feature does not execute JavaScript. If you have OpenClaw already running: openclaw plugins install @ollama/openclaw-web-search 🤖 Non-interactive (headless) mode for ollama launch ollama launch command can now run in non-interactive mode. This is perfect for: - Docker/containers: spin up an integration as a pipeline step to run evals, test prompts, or validate model behavior as part of your build. Tear it down when the job ends. - CI/CD: Generate code reviews, security checks, and other tasks within your CI - Scripts/automation: Kick off automated tasks with Ollama and claude code Try with: ollama launch claude --model kimi-k2.5:cloud --yes -- -p "how does this repository work?"
- 07
@Akashi203 ·
i built a local LLM inference engine that runs a 1B parameter model on a $10 board with 256mb ram. model sits on the sd card, streams one layer at a time through 45mb of ram You can use it as local LLM model backend for PicoClaw no python no cloud no api keys 80kb binary | pure c | zero dependencies github: https://t.co/0fe4ey4yz7
- 08
@0xSero ·
Here’s what I’d recommend if you’re just getting started in AI, local or otherwise. 1. Work with the compute you have, even the dumbest LLMs can be useful if you treat them as a node in your system. Some basic problems of what could be useful to get you started - tag all your screenshots - classify your emails - recommendation algo - scanning git history for patterns 2. If you want to try larger models use prime intellect or Hotaisle (even cheaper) to rent out whatever amount of VRAM you need to run models you like. - RTX 3090 rents for cents/h - RTX 6000 rents for 1-2 dollars/h - H100 rent for 2-3 dollars/h 3. Use the right frameworks: - VLLM, and SGlang for faster inference if you have 1, 2, 4, 8, 16 GPUs - exllamav3 and llama.cpp if you have non-power of 2 GPUs - MLX if you have Mac 4. Start with problems in your life that could use intelligence for automation - shopping - research 5. Don’t expect local models to code production projects just yet
- 09
@akshay_pachaar ·
Run Claude Code using local LLMs for FREE. No API costs. No data leaving your machine. Here's how it works: Claude Code lets you swap its backend via a single env variable. Point `ANTHROPIC_BASE_URL` to a local llama.cpp server, and it'll route all requests to whatever model you're running locally. The Unsloth team put together a step-by-step guide for this where you can run Claude code using Qwen3.5 It covers everything from model download to server setup to running Claude Code. The trick is serving your model on port 8001 via llama-server, then setting two env vars: `ANTHROPIC_BASE_URL` and a dummy `ANTHROPIC_API_KEY`. That's it. Claude Code thinks it's talking to Anthropic's API. If you don't want to pay per token for every agentic loop and want fast, private, cost-free coding runs, this is exactly what you're looking for. Link to the guide in the next tweet
- 10
@TheAhmadOsman ·
You don’t pick an Inference Engine You pick a Hardware Strategy and the Engine follows Inference Engines Breakdown (Cheat Sheet at the bottom) > llama.cpp runs anywhere CPU, GPU, Mac, weird edge boxes best when VRAM is tight and RAM is plenty hybrid offload, GGUF, ultimate portability not built for serious multi-node scale > MLX Apple Silicon weapon unified memory = “fits” bigger models than VRAM would allow but also slower than GPUs clean dev stack (Python/Swift/C++) sits on Metal (and expanding beyond) now supports CUDA + distributed too great for Mac-first workflows, not prod serving > ExLlamaV2 single RTX box go brrr EXL2 quant, fast local inference perfect for 1/2/3/4 GPU(s) setups (4090/3090) not meant for clusters or non-CUDA > ExLlamaV3 same idea, but bigger ambition multi-GPU, MoE, EXL3 quant consumer rigs pretending to be datacenters still CUDA-first, still rough edges depending on model > vLLM default answer for prod serving continuous batching, KV cache magic tensor / pipeline / data parallel runs on CUDA + ROCm (and some CPUs) this is your “serve 100s of users” engine > SGLang vLLM but more systems-brained routing, disaggregation, long-context scaling expert parallel for MoE built for ugly workloads at scale lives on top of CUDA / ROCm clusters this is infra nerd territory > TensorRT-LLM maximum NVIDIA performance FP8/FP4, CUDA graphs, insane throughput multi-node, multi-GPU, fully optimized pure CUDA stack, zero portability (And underneath all of it: Transformers → model architecture layer → CUDA / ROCm / TT-Metal → compute layer) What actually happens under the hood: > Transformers defines the model > CUDA / ROCm executes it > TT-Metal (if you’re insane) lets you write the kernel yourself The Inference Engine is just the orchestrator (simplified) When running LLMs locally, the bottleneck isn’t just “VRAM size” It isn’t even the model It’s: - memory bandwidth (the real limiter) - KV cache (explodes with long context) - interconnect (PCIe vs NVLink vs RDMA) - scheduler quality (batching + engine design) - runtime overhead (activations, graphs, etc) (and your compute stack decides all of this) P.S. Unified Memory is way slower than VRAM Cheat Sheet / Rules of Thumb > laptop / edge / weird hardware → llama.cpp > Mac workflows → MLX > 1–4 RTX GPUs → ExLlamaV2/V3 > general serving → vLLM > complex infra / long context / MoE → SGLang > NVIDIA max performance → TensorRT-LLM
- 11
@aiDotEngineer ·
TLMs: Tiny LLMs and Agents on Edge Devices with @cormacb https://t.co/u0fHD7j5kZ Function Gemma ships at 270 million parameters and runs nearly 2,000 tokens per second prefill on a Pixel 7. Out of the box, it hits 46% accuracy on a fixed set of app intents. Fine tune on a synthetically generated dataset and that clears 90% on eight of ten functions. Cormac walks through the two paths developers have for on device AI: a skill harness built on Gemma 4 with a restaurant roulette demo running fully on device. Then Eloquent, a production transcription app built by chaining two sub billion parameter models together. cc @osanseviero
- 12
@AlexFinn ·
5 months ago I spent $30,000 on 3 Mac Studios, 2 Mac Minis, and a DGX Spark I went all in on local LLMs and encouraged others to do the same I warned prices would explode I was called crazy, a hype beast, dangerous, and that I had no idea what I was talking about Since then: • Mac Studios above 96gb have become unavailable • Memory prices have 4x’d • Other hardware prices have 10x’d Now those same AI influencers who destroyed me are spending 5 to 6 figures on hardware publicly GLM 5.2 dropped and it’s Opus level. I’m running it on 1 of my 3 Mac Studios 512gbs. The same ones I was called an idiot and hype beast for buying. The same ones that are reselling for triple the price used. The insane part is this is just the beginning Intelligence will be integrated into every device you own, including devices that aren’t even publicly available yet like humanoid robots All of these new devices will require GPUs, memory, storage, and more components Components that have already 10x’d in price That’s not even counting all the people that will start vibe coding when Codex and Claude Code become more mainstream Right now less than 1% of the world is even taking advantage of those tools Imagine what happens when it reaches 2% The local revolution is here. Hardware is the bottleneck Act accordingly
- 13
@witcheer ·
Google TurboQuant maybe reduced my LLM memory by 6x. running 70B models on my Mac Mini could be real. I have Mac Mini M4. memory was always the issue, 16GB meant 30B models were pushing it, 70B were out of the question. TurboQuant changes that math. 6x memory reduction without quality loss. someone already ran Qwen 3.5-9B on a MacAir using the llama.cpp patch. 6x memory reduction means I could fit a 120B parameter model in the same RAM that currently fits a 20B. or run multiple agents in parallel. I haven't applied the patch yet. experimental, production workflows, not worth breaking things. but I'm watching community validation closely. does TurboQuant work with quantised models? most local inference runs 4-bit. if anyone's tested TurboQuant + GGUF on Mac Silicon, I am interested in the numbers.
- 14
@0xSero ·
Best harnesses for local models: 1. Droid: - Very good performance, forces the models to behave, you can wire in all your local LLMs very easily w BYOK - Allows you to use your local models as orchestrators/subagents so you can benefit from Cloud as models as well - Practically everything you need is already in Droid - This is my daily driver, I use Qwen3.5 models in it very happily 2. Zed IDE: - OpenAI compatible APIs are first class - Works like Cursor so if you prefer IDEs this is for you. - Clean design 3. Pi Coding Agent - Open source - Very token efficient - Codebase supports vllm and open weight models 4. RooCode: - Steer mode forces local/dumber models to behave by re-injecting the user's prompt and plan every action - Very easy to set up local providers 5. OpenCode: - Super easy to add new providers - Desktop app/Mobile web let's you use ur local models - Open source 6. Parchi - Any provider of any type is compatible - Very simple UX - Let's you operate your browser with your local models
- 15
@mudler_it ·
I've just released APEX (Adaptive Precision for EXpert Models): a novel MoE quantization technique that outperforms @UnslothAI Dynamic 2.0 on accuracy while being 2x smaller for MoE architectures. Benchmarked on Qwen3.5-35B-A3B, but the method applies to any MoE model. Half the size of Q8. Perplexity comparable to F16. Works with stock @ggml_org's llama.cpp. Open source (of course!), with ❤️ from the @LocalAI_API team. 👇Links to the model, repository and benchmarks below! (+ Bonus TurboQuant benchmarks with @no_stp_on_snek's TQ+! )
- 16
@DAIEvolutionHub ·
Holy shit 🤯 Microsoft just open-sourced a framework that runs a 100B parameter LLM on a single CPU. No GPU. No cloud. No expensive setup. Just your laptop. It’s called BitNet. And it breaks one of the biggest assumptions in AI. Here’s the trick: Most LLMs use 16-bit or 32-bit floats. BitNet uses: 1.58 bits. Yes… bits. Weights are just: -1, 0, +1 That’s it. No heavy matrix math. Just simple integer operations your CPU already handles efficiently. The result is insane: • 100B model runs on CPU at 5–7 tokens/sec • 2–6× faster than llama.cpp on x86 • 82% less energy usage • 1–5× faster on ARM (MacBooks) • 16–32× lower memory The craziest part? Accuracy barely drops. Their flagship model (trained on 4 trillion tokens) performs competitively with full-precision models. They didn’t break the model. They removed the waste. What this unlocks: → Run LLMs fully offline → AI on phones, edge devices, IoT → No API costs for inference → Works even without reliable internet MacBook. Linux. Windows. It just runs. 27K+ GitHub stars. Built by Microsoft Research. 100% open source. This might be the moment AI stops being cloud-first… and becomes device-first.
Watch video
- 17
@gneubig ·
In the spirit of language model freedom, as a weekend project I finally set up my own fully local coding agent: * GMKtek box with AMD Ryzen * Qwen3.5 30B A3B, served with lemonade server+llama.cpp * ngrok to expose the endpoint * OpenHands as the agent, currently through the CLI Honestly I am still a bit surprised at how well this works. Qwen3.5-30B-A3B is a good model, and the serving experience was pretty fast and smooth.
- 18
@TheAhmadOsman ·
Local LLMs, Buy a GPU, and the Case for Cognitive Security AGI? There is no guarantee we reach it anytime soon, if ever. And even if we do, there is certainly no guarantee it will run on your machine. Betting your agency on either assumption is a mistake. What is guaranteed is that large language models are already shaping how people think, decide, and act. They mediate search, writing, coding, persuasion, and increasingly judgment. That makes understanding them less of a technical curiosity and more of a survival skill. This is why my default advice is simple: run local LLMs. You need to learn how these systems work. When you operate models yourself, you stop treating them like magic. You see how they are prompted, steered, constrained, fine-tuned, degraded, and exploited. You learn where they are brittle, where they are persuasive, and where they quietly fill in gaps with confidence instead of truth. That hands-on understanding is the most basic form of cognitive security. CogSec is not about paranoia or rejecting AI. It is about literacy. If you know how models are shaped, you can recognize when they are shaping you. If you do not, you are operating blind. People who only interact with LLMs through polished products are easy to influence. They cannot tell when they are being nudged, framed, or one-shot into a conclusion or a psychosis. They cannot distinguish between alignment choices, product incentives, or outright manipulation. Whoever controls the model controls the gradient of influence. Owning your compute and running models locally flips that power dynamic. It gives you inspection, friction, and intuition. You are no longer a passive endpoint in someone else’s system. You are a participant who understands the machinery. You do not need AGI for this to matter. The models we already have are sufficient to warrant cognitive defense. Buy a GPU. Run the models. Learn their failure modes. That is not a hobby. That is baseline cognitive security in an LLM-saturated world.
- 19
@EXM7777 ·
how to set up OpenCode (the privacy-first Claude Code alternative) in 3 minutes: > install: npm i -g opencode-ai > configure your preferred model provider > cd into your project directory > run: opencode it reads your codebase locally, builds context on your machine, and only sends the specific prompt + relevant code to the LLM this is the best configuration for privacy-sensitive work: > use Ollama with Gemma 4 > zero data leaves your machine > you get agentic coding with file editing, terminal access, and multi-step reasoning > completely free yet to match frontier models, but it's kinda crazy we can run this type of workflow on a mid laptop
- 20
@sudoingX ·
first local model running on the rog 5090 mobile. gemma 2 2b it q4_k_m through llama.cpp, hermes agent pointing at the local endpoint. sitting on the balcony watching the beach while the model answers prompts. 24gb vram in a portable machine running a full inference stack plus an agent harness. same class as the desktop 3090 i benchmark on, now outside with coffee. the hermes agent local model auto-detection i pushed a while back just resolved the endpoint on its own. good time seeing some my own code work in the wild, on my own hardware. small moment but it hit. starting small on purpose. pulling carnice-27b, qwen 3.5-27b dense, gemma 4 dense with hf cli next. same hardware, same quant, real agentic coding tests. comparison data dropping soon. beast is going to run wild stuff this week. keeping you all posted.
- 21
@witcheer ·
I run ollama on a Mac Mini for local compression. every message my agent sends passes through a local qwen model to summarise context before it overflows. speed matters because slow compression means slow responses across every cron job. ollama shipped MLX backend for Apple Silicon. the numbers on M5: → prefill: 1,810 tok/s (was 1,154). 57% faster → decode: 112 tok/s (was 58). 93% faster that's not a marginal improvement. my compression pipeline got nearly 2x faster with a software update. three things that matter beyond the speed numbers: 1/ the cache got smarter. ollama now reuses cache across conversations and stores snapshots at intelligent points in the prompt. for agent workloads with shared system prompts, (which is exactly how OpenClaw works) this means less re-processing on every call. 2/ NVFP4 quantisation. NVIDIA's 4-bit format, now running locally on Apple Silicon. same quantisation format used in production inference. your local model and the cloud model now speak the same precision language. 3/ the model they ship with it: qwen3.5-35B-A3B, a 35B mixture-of-experts with only 3B active parameters. tuned for coding tasks. needs 32GB+ unified memory.
- 22
@socialwithaayan ·
oh my.. this shouldn't be possible a 1B model that runs inside your browser, beats every model its size, and comes with its own desktop pet. MiniCPM-5 1B just changed the game for on-device AI. here's everything you need to know 🧵
Watch video
- 23
@_vmlops ·
SOMEONE RAN A 122B MODEL ON THEIR MACBOOK... IN ONE NIGHT no cloud. no api fees. no data leaving the machine they went through 3 generations: → ollama + proxy: 30 tok/s → llama.cpp + proxy: 41 tok/s → mlx native server: 65 tok/s the breakthrough? they killed the proxy entirely wrote a 200-line python server that speaks the anthropic api directly. claude code thinks it's talking to the cloud. it's not that one change made it 7.5x faster you can even control it from your iphone over imessage. while offline this is wild engineering → https://t.co/TE8RpBvDhb
- 24
@sudoingX ·
compiling llama.cpp with cuda 12.8 on the rog strix scar 18 5090. this is the mobile 5090, 24gb vram in a laptop. same vram class as the 3090 desktop i run benchmarks on, just portable. i can run these tests from anywhere now. first up carnice-27b vs qwen 3.5 27b dense vs gemma 4 on this exact machine. same class, same hardware and real agentic coding tests. data dropping soon.
- 25
@DAIEvolutionHub ·
CHINA JUST DROPPED AN OCR MODEL THAT CHANGES EVERYTHING. A tiny 3B-parameter model can read an entire 100-page PDF in a single pass. No page splitting. No context loss. No cloud APIs. Meet Unlimited-OCR 👇 • Reads full documents with a 32K context window • Scores 93% on OCR parsing benchmarks (+6 points over the baseline) • Keeps error rates below 0.11 even after 40+ pages • Multilingual out of the box • Runs 100% locally on your own hardware • Works with Transformers, vLLM, SGLang, Docker, Ollama, and llama.cpp Here's why this matters. Most OCR systems still treat every page as a separate image. That means broken tables, lost references, and missing relationships between pages. Unlimited-OCR processes the entire document as one continuous context. So tables stay intact. References remain connected. Document structure is preserved from beginning to end. Meanwhile, many businesses are still: • Paying $1.50 to $15 per 1,000 pages • Uploading sensitive PDFs to cloud providers • Waiting on OCR API responses This runs completely offline. No subscriptions. No API bills. Just your hardware. Built by Baidu to push beyond DeepSeek-OCR. It's already crossed 1.9M+ downloads on Hugging Face. And surprisingly, almost nobody is talking about it yet. Open source keeps shrinking the gap between "enterprise software" and "free." 🔗 https://t.co/Wt0SN0UMSy
Watch video
- 26
@cnxsoft ·
Online guide to select the best hardware for local LLM/AI deployments. https://t.co/Q2LAYo5xxx The website relies on Qwen 3.5 models and shows the price, performance, power consumption, and more for a range of hardware from Raspberry Pi 5 16GB to NVIDIA DGX Spark. In one specific test (Qwen 3.5 9B), a system with a $260 Intel Arc graphics card performs just as well as $4,000 computers. The project also accepts user submissions, but the process is a little convoluted.
- 27
@Cyb3rMaddy ·
Been messing around with local, uncensored LLMs... Ollama runs open-source LLMs locally. It’s handy for security research, private workflows, and anything you don’t want leaving your machine — even works without internet. No cloud calls. No sending prompts somewhere you can’t see. Which you can scale on an enterprise level if needed, or just have your own custom private AI. You can even mod these with your own system prompts. Messing around with dolphin-llama3:8b for red teaming. As you can see below, totally unrestricted. At least it gives a warning...? ☠️
- 28
@aigleeson ·
Claude Code just got a free mode. Not official. A developer built a repo that routes Claude Code calls to free and local AI models. Instead of buying Anthropic credits... You can run it with: > NVIDIA NIM > OpenRouter free models > DeepSeek > LM Studio > llama.cpp Best part: Claude Code does not need to change. You just point it at a local proxy and it works. This turns Claude Code from “paid coding agent” into “bring your own model agent.” Huge for indie devs. https://t.co/ua4nHZG28m
- 29
@tonysimons_ ·
A 27B multimodal AI model is now running locally on an iPhone. PrismML’s Bonsai 27B packs 27.8B parameters into just 3.9GB and reportedly hits 11 tokens per second on an iPhone 17 Pro. That’s roughly 14× smaller than FP16 while retaining more than 90% of benchmark performance. The cloud just lost a little more territory. I broke down the benchmarks, the catches, and what this means for private, on-device AI agents. ⤵️ https://t.co/tXmKgHVXPg
- 30
@morganlinton ·
Fun little Sunday morning project, starting work on a little offline LLM I'm calling Wilderness Bot. I go hiking and backpacking out in the wilderness a lot, and want a way to have a little LLM loaded with wilderness medicine info, survival guides, etc. Decided to build in Python because it has been my go-to for over a decade. Almost went with Rust on this because I've been digging Rust lately, but decided to just stick with ol' faithful. Here's a bit more about the architecture: - Keeping the model small: targeting quantized 3B-4B class models so it can run on a modern phone. - Instead of cramming everything into the model weights, the knowledge lives in a local SQLite bundle. - The bot uses retrieval-first answers, so it pulls the most relevant chunks before responding. - I like the idea of keeping things fresh and allowing for monthly updates. And with this architecture monthly updates are easy: refresh the data bundle, keep the model the same. - Dev setup already supports local testing with Ollama, but the data format is portable to phone runtimes. - The long-term mobile path could work with llama.cpp, MLC, or ExecuTorch. -- As for the data I've loaded in so far this morning: - Seeded with public, refreshable safety sources. - CDC emergency water storage/preparedness guidance. - American Red Cross first-aid steps. - American Red Cross wildfire safety guidance. - Planning to expand with wilderness medicine references, survival guides, and licensed field manuals. -- And of course, I bought a domain name for it - Wilderness(.)bot, which I was kinda surprised was available to hand register, but sometimes you get lucky. Will share more as I go. Overall, I feel silly as an engineer that loves tinkering with LLMs, that I still go into the wilderness with PDFs of all this stuff, time to change that. Today, Wilderness Bot is born.
- 31
@Shruti_0810 ·
A LOCAL LLM RUNNING ON ONE 3090 JUST REPLACED GOOGLE HOME FOR 29 SMART DEVICES WITH ZERO CLOUD. It doesn't just answer questions. Ask, “Do I need a jacket?” and it checks your actual weather sensor before replying. The cloud is becoming optional.
Watch video
- 32
@HeyZaraKhan ·
Nobody expected Apple to open-source this. Apple just open-sourced one of the most interesting AI repositories for developers. It's called Core AI Models. Instead of forcing you to convert and optimize models yourself… Apple provides ready-made export recipes, Python utilities, and Swift runtime tools for running popular AI models directly on Apple devices. The repository includes support for models like: → Qwen → Mistral → SAM3 → and more community models optimized for Apple Silicon. The goal is simple: Build AI apps that run entirely on-device. No server. No API costs. No cloud dependency. Just local inference powered by Apple hardware. It also includes: • Model export recipes • Python conversion utilities • Swift integration packages • Runtime libraries for Core AI • Ready-to-use examples for popular architectures If you're building AI apps for iPhone, iPad, or Mac, this repository is worth bookmarking. It feels like Apple is making on-device AI development dramatically easier for every developer. GitHub repo below:
- 33
@morganlinton ·
Everyone is talking about Kimi and Qwen, but I'm honestly surprised more people aren't talking about models like Trinity from Arcee. I've been doing a deeper dive here and it's pretty interesting, here's a few differences that I'm not sure ppl fully realize. - Qwen and Kimi both have Apache licenses with restrictions in them. I'm honestly shocked most people don't know this. Once you hit over 100M monthly users, things change. - Arcee's models like Trinity use Apache 2.0 with NO restrictions, none, get over 100M monthly users, you're still in a good place. - The training data for Qwen and Kimi is undisclosed, for Arcee it's both disclosed and legally vetted. - One genuinely unique thing Arcee does: they released Trinity-Large-TrueBase, a raw 10-trillion-token checkpoint that hasn't undergone learning rate anneals or instruction tuning, letting researchers in regulated industries start from scratch for authentic audits and custom alignment. As I've been tinkering around with more small(ish) local LLMs, I've been doing a deeper dive, and honestly, I think I'm going to be doing more with Arcee vs. Kimi or Qwen because I can't help but think about scale. If I build something, I'd love to think that some day hundreds of millions, or billions of people, might use it, and I don't want to suddenly be dealing with license restrictions in this case and need to rearchitect. That being said, still pretty new to the Arcee models so I'll have to see how they perform. More to come, as always, I have a lot more to learn, and will share what I learn on here.
- 34
@lemire ·
AMD is coming up with its small AI PC (AI Halo). It will compete against NVIDIA DGX Spark. Both look a bit like a mac Mini. Just a tiny box. The AI Halo should cost US$4000, so it is accessible to hobbyists and small IT departement. Set it up with LM Studio with its llama.cpp backend and you got yourself a 'ChatGPT-like' experience from your own network. Both the AI Halo and the Spark have 128GB of 'unified' RAM (meaning that the memory is shared between CPU and GPU). It is the same type of memory you get on your macBook (LPDDR5X), so it is not the fancy high-bandwidth memory you get on GPU cards... but it is good enough. So I played with llama.cpp and I was running open source models last year. It is pretty cool getting this silly little C++ program (llama.cpp) that answers you back when you ask questions... right there on your machine. But I think that most people will be somewhat disappointed in practice. It is close to Big AI models, but still obviously inferior to even the cheap commercial offerings. If you have US$4000 to burn, you are better off spending it on tokens from an AI provider. That's short term. Long term... Who knows? Linux started out as a toy. It is now running the Internet. Give it 5, 10, 20 years and maybe these little boxes could be everywhere.
- 35
@0xJiuJitsuJerry ·
Late night breakthrough: My Mac Mini M4 "Mainframe" is now talking to my @zocomputer through MCP 🧠⚡ What does this mean? Local Gemma3, Nemotron, Qwen3.5 all available as tools No Cloudflare tunnel dependency Zero latency on my AI agent swarm Complete data sovereignty From building @solana smart contracts to running @AccessProtocol content strategies — it's all happening on MY metal. Zo isn't just "another AI" — it's the bridge between your local power and global coordination. Who else is running local LLMs with their agents? #LocalLLM #AgentSwarms 🦾👇 https://t.co/ZyYwL07hPy
- 36
@Axel_bitblaze69 ·
local AI is now useful enough to keep at home imp.. a small machine sitting on your desk can now handle a large part of your everyday AI work for roughly a few dollars in electricity each month, depending on what you run and how often. things like: - drafting - summarising - basic coding help - classifying information - reading your own files - running repetitive automations all without paying for every request or sending the work to a cloud model. the goal isn’t to replace Claude or ChatGPT completely. the better setup is probably: local AI for the repetitive, private, high-volume work one strong cloud model for the difficult stuff first, prove the workflow tonight for free don’t buy anything yet. install Ollama, then run a small model: ollama run llama3.2:3b that’s it. you’re talking to a model running on your own computer. many modern laptops with around 8GB of memory can run a small quantized model, although the speed and usable context will depend on the machine. Ollama supports macOS, Windows and Linux. and if you’d rather use a proper chat window than a terminal, install Open WebUI on top. it connects to Ollama and gives you something much closer to a private ChatGPT interface running on localhost. use it for a weekend. not because a free 3B model performs like a $2,000 setup. it obviously doesn’t. you’re testing something more basic: does running AI locally fit the way you work? try it on your documents, notes, repetitive writing and simple automations. if those workflows feel useful, then better hardware gives you stronger models, longer context and faster output. one rule before buying anything buy memory first, speed second. model size is the first hard limit. if the model doesn’t fit into available memory, it either won’t load properly or will spill into slower system memory and become painful to use. speed matters after that. memory decides what you can run in the first place. a rough ladder: - 8GB → small 3B–7B quantized models - 24GB → 14B models comfortably, some 32B models with compromises - 128GB → 70B-class quantized models and much longer contexts these aren’t hard laws. quantization, context length and how much memory the operating system already uses can change the result. unified-memory machines are interesting because the CPU and GPU draw from one shared memory pool instead of the GPU being limited to a separate chunk of VRAM. the hardware ladder around $249 → Jetson Orin Nano Super, 8GB the cheapest dedicated entry. tiny, efficient and capable of running smaller local models, although it’s more of a developer or edge-AI machine than a normal desktop. NVIDIA currently lists it at $249. from $599 → Mac mini M4, 16–32GB unified memory probably the easiest quiet, always-on local AI box to live with. the M4 model starts with 16GB and can be configured to 24GB or 32GB. used RTX 3090 → 24GB VRAM still an obvious option when you want a lot of NVIDIA VRAM without buying a current flagship card. but remember, the GPU isn’t the computer. you still need the motherboard, CPU, RAM, storage, case and power supply around it. Used pricing will depend heavily on your market. The RTX 3090 itself has 24GB of GDDR6X memory. around $2,000 → Ryzen AI Max+ 395 with 128GB unified memory this is where local AI starts becoming serious. AMD supports configurations with up to 128GB of unified memory, with a large share available to the GPU, and Framework introduced its 128GB desktop configuration at $1,999. $4,000+ → high-end Mac Studio and DGX Spark territory this only starts making sense when you’re running large models constantly, fine-tuning, building AI products or already spending serious money on cloud GPUs. DGX Spark has 128GB of unified memory, and NVIDIA raised its Founders Edition MSRP to $4,699 in February 2026.
- 37
@ujjwalscript ·
Your "Local LLM" and “Free AI” dev setup is a massive WASTE of time and money! The hottest trend on X right now is showing off your "local-first" setup. Developers are buying expensive NVIDIA 5090s, bragging about running Llama-3.2 or Phi-3.5 completely offline, and treating cloud API users like absolute peasants. "Look at my zero-latency inference! Look at my data privacy!" It’s a beautiful flex. It's also an engineering Delusion. Here is the truth people are refusing to admit because they want to justify their hardware spending: You are sacrificing massive cognitive reasoning just to say you run on localhost. When you switch your development workflow from a massive, frontier cloud model to a quantized 8B or 7B Small Language Model (SLM) running on your machine, you aren't upgrading. You are downgrading your assistant from a Principal Architect to an intern who drank too much coffee. Yes, SLMs are incredible for hyper-specific, narrow tasks like text classification or basic autocomplete. But for complex system design, edge-case debugging, and cross-repository code auditing? They hallucinate under pressure because they lack the deep parameter weight to handle complex abstraction. Stop trying to turn your local workstation into a miniature data center. Use the frontier cloud models for the heavy intellectual lifting - the system boundaries, the state management, the algorithmic strategy. Use local models for basic syntax completion.
- 38
@Shruti_0810 ·
The most expensive part of this AI setup isn't the hardware. It's... nothing. A Raspberry Pi 5 with 16GB RAM and a 512GB NVMe is powering: • Local LLMs with Ollama • Claude Code via localhost • Bluetooth analysis • Wi-Fi security testing • Packet capture All from a device that fits in a shoulder bag. Instead of sending prompts to the cloud, everything runs locally. That means: → No API bills → No internet required for inference → Sensitive data stays on the device → One machine handles both AI and security workflows The idea of carrying an AI coding assistant and a portable security lab together would've sounded unrealistic not long ago. Now it's a Raspberry Pi project anyone can build.
Watch video
- 39
@SaiyamPathak ·
Yesterday in the video I took - the qwen32B which was a dense 32B and all active parameters for every token whereas for the MLX version it was A3B - active 3B. this morning I ran some tests again: - Qwen3.5 (NVFP4, MLX): 23.11 tok/s decode - Nemotron (GGUF, llama.cpp): 21.70 tok/s decode So not that much of a difference - maybe llamacpp already good for M1 type - I need to test it across all architectures though. Interestingly I tested the LMstudio as well that had the MLX support for quite some time now. results are pretty interesting and I tried kind of similar models although its not apples to apples. LM Studio - MLX - Qwen3.5-35B-A3B-4bit - 33.82 tok/s Ollama - MLX - qwen3.5:35b-a3b-coding-nvfp4 - 23.11 tok/s the 4-bit quantization is different here for both models. Can you share your benchmarks? anything anyone tested.
- 40
@_vmlops ·
CLAUDE CODE FOR FREE - WITHOUT THE ANTHROPIC API BILL Someone built a local proxy that intercepts claude code's api calls & reroutes them to free models point one env variable at localhost and you're done: → nvidia nim (free tier) → openrouter free models → deepseek → ollama / lm studio / llama.cpp (fully local) https://t.co/vyC6VEZnaq works with claude code cli, vscode, and jetbrains out of the box discord and telegram bot if you want remote sessions, plus voice note support via whisper
- 41
- 42
@Rus_Khairullin ·
Vitalik published a detailed post on how he set up a fully local, self-sovereign AI - no cloud, maximum privacy and security. AI agents can already work for hours, use tools and modify their own code. But most (even open-source) ignore security: data leaks, hidden instructions, jailbreaks through websites. Vitalik: “We just won privacy in messaging. Now we can lose it all through AI.” Main threats he’s protecting against: • Data leaks into the model and outside • Jailbreaks and malicious content • Backdoors in models • Accidental or intentional sending of private info What Vitalik uses: • Hardware: Laptop with RTX 5090 (24GB) - 90 tokens/sec on a 35B model. Best price/performance option. • Model: Qwen3.5-35B (perfect balance of speed and quality) • OS: NixOS - everything in one config, easy to share • Inference: llama-server + llama-swap • Agents: pi framework • Sandbox: bubblewrap - every tool in strict isolation Useful features: • Local knowledge base (Wikipedia + documentation) • Search via private SearXNG • Daemon for Signal and email: reads everything, sends to himself freely, to others - only after human confirmation • For Ethereum: separate daemons with limits and double confirmation (human + LLM) Experience: Handles simple tasks well (write Snake, flashcards, audio transcription). Complex code is still weak - sometimes has to use external models. Vitalik’s vision: Fully local stack + tools for safe use of remote models (ZK-API, mixnets, TEE). Goal - make AI a tool for privacy and sovereignty, not surveillance. Who’s already running local LLMs in 2026? What hardware and stack are you using? @VitalikButerin
- 43
@smratitiwa86867 ·
Holy shit...Someone tried replacing Claude with a local LLM… …and waited 13 minutes for THIS: > “I am a large language model, trained by Google.” That’s it. 13 minutes. One useless sentence. Let’s be honest — we’ve ALL had this thought: “Why am I paying for Claude when I can just run open-source models locally for free?” So he tried. Ran Gemma locally. No API costs. Full control. Sounds smart, right? Here’s what actually happened: Before he even asked a real question… The model was already drowning. Tools like Claude Code don’t start from zero. They load 20,000+ tokens of system context: • instructions • memory • tools • workflows Your “free” local model? It’s choking before the race even starts. And then you wait… …and wait… …and wait… …for a generic sentence you could’ve gotten in 0.8 seconds. This is the part no one tells you: Running LLMs locally isn’t just about compute. It’s about: context handling latency orchestration tooling You’re not replacing an API. You’re trying to replace an entire AI infrastructure stack. The API bill hurts. But wasting hours waiting for responses? That’s way more expensive.
- 44
@DBCrypt0 ·
OMG a new local LLM dropped that you can run on your Mac Mini and it’s just as good as Opus 4.6! 🔥 Local Model: Hi! Before we begin, tell me about yourself and what you want to call me? Me: Let’s call you Max. I’m a content creator, podcast host, and Web3/AI researcher. Local Model: Perfect! I think Charlie is a great name for me and I can definitely help with your web design business. Me: …I said Max. And I’m not a web designer. Local Model: You are right, you did say Max. I’ll make note of it and make sure not to forget next time. Me: Ok, cool. Did you add it to your identity file? 5 minutes pass… Local Model: what’s an identity file? Would you like me to create one? 3 minutes later… Done! I deleted the memory and agents file and created a new identity file. I made sure to add to the top that my name is Charlie like you requested. Me: … Local Model: Hi, I’m Bob. What are we working on today? Don’t let anyone tell you local models are ‘just as good’ if you’ve ever used Claude
- 45
@JeremyCMorgan ·
If your last local-LLM setup was early 2025, the viable model list moved. This refresh covers Ollama, llama.cpp, and VRAM sizing, with Qwen 2.5 Coder 32B the standout coding pick at 24GB, scoring ahead of GPT-4o on HumanEval per the post. Recalibrate before buying a box. https://t.co/586xnyLAtR
- 46
- 47
- 48
- 49
@JulianGoldieSEO ·
OLLAMA JUST FIXED THE BIGGEST PROBLEM WITH LOCAL AI AGENTS Gemma 4 could use tools before. Now it can actually finish the job. What changed: → Ollama 0.32.1 improves tool-response continuation → Gemma 4 can call a tool, process the result, and continue working → Fewer abandoned tasks, forgotten instructions, and half-finished answers The real-world test: ✓ A messy 400-line Python script was analyzed locally ✓ The model found 3 performance bottlenecks and rewrote them ✓ It ran the tests and reported the results without losing the task The speed upgrade: ✔ MLX support made Gemma 4 nearly 90% faster on one coding benchmark ✔ Reported output jumped from roughly 50 to 95 tokens per second ✔ A memory-leak fix lets longer jobs run without constant restarts The lesson: Stop testing local AI with toy prompts. Give it a real file, multiple steps, and a result it must verify. That is how you find out whether your local agent can replace actual manual work.
Watch video
- 50
Best Tweets by Topic