Inference engines and serving stacks
llama.cpp, Ollama, MLX, vLLM, SGLang, BitNet, and related runtimes for serving models locally.
74%
Best tweets about Local LLMs
50 Best Tweets About Local LLMs (2026)
Explore the best tweets about local LLMs, on-device AI, hardware, quantization, privacy, inference, and self-hosted model setups. Updated weekly.
Reproducible local-model setups, useful hardware comparisons, performance measurements, and privacy-focused workflows.
- Creators
- 39
- Updated
Original Xholic analysis
What 50 top Local LLMs posts reveal
The dataset is strongly supportive of local-first AI and centers on serving stacks, hardware demonstrations, memory-efficient inference, and private or offline workflows. It also contains cautionary views that local setups can face latency, context-handling, and capability limits for complex work.
- Dominant tone
- Positive
- Median score
- 54.3
- Leading format
- Announcement
- Recent posts
- 18%
80% of posts
All-time engagement
44% of posts
Published in 90 days
Conversation map
The themes creators return to
Hardware selection and performance benchmarks
GPU, CPU, Mac unified-memory, mini-PC, mobile, and multi-GPU comparisons covering throughput, memory, power, bandwidth, and software support.
46%
Local agents and personal automation
Tool-using local agents for browser control, shell and file operations, smart homes, research, scheduled tasks, and persistent memory.
32%
Local coding agents and developer workflows
Using local models with Claude Code, Codex, IDEs, agent harnesses, code review, tool use, and autonomous software tasks.
32%
Offline and private AI
Local-first, self-sovereign setups that avoid cloud dependencies, API costs, telemetry, and data exposure.
26%
Edge, mobile, and low-power LLMs
Running models on phones, Raspberry Pi-class boards, IoT devices, CPUs, and other resource-constrained hardware.
18%
Offline knowledge bases and local RAG
Searchable local documents, offline Wikipedia and maps, SQLite-backed retrieval, document processing, and resilience-focused knowledge systems.
16%
Quantization and memory-efficient inference
Low-bit weights, MoE-aware methods, KV-cache settings, adaptive precision, flash streaming, and techniques to fit larger models on constrained devices.
16%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 106
- Median reposts
- 15
- Median replies
- 12
- Median views
- 9.7K
Posts with media make up 80% of this collection. Their median all-time score is 70.6, compared with 3.12 for text-only posts.
Format mix
- Announcement 44% · score 124.2
- Case Study 28% · score 7.90
- Opinion 12% · score 97.8
- Tutorial 8% · score 236.7
Consensus and debate
Where creators agree, and where they do not
Shared view
Local stacks emphasize offline operation and data control
Offline operation, local data handling, and reduced cloud dependence recur in posts about survival servers, local coding, and agent workflows.
Shared view
Serving configuration can affect local-agent behavior
Posts identify runtime and integration details—including KV-cache reuse, KV-cache precision, thinking-mode settings, and direct API compatibility—as relevant to speed, output quality, or agent operation.
Shared view
Hardware comparisons need workload context
Examples include a 3090 Ti running a 27B model with 262k context, an M2 Ultra demonstration reported at 300t/s, and resource-constrained edge deployments. Together, they show that reported results depend on the model, runtime, precision, context, and device.
Open debate
Whether local models can replace cloud workflows is contested
Some posts describe successful local coding or research-agent setups, while critical posts argue that long system prompts, latency, and weaker reasoning can make cloud models preferable for complex work.
Open debate
VRAM alone is not presented as a sufficient hardware criterion
Posts discuss fitting models through compression and large-memory systems, while another argues that memory bandwidth, interconnect, inference-engine support, and software maturity also affect practical local inference.
What performs
Patterns behind standout posts
Several posts provide concrete throughput figures
Reported demonstrations include 300t/s on an M2 Ultra, 52.6 tok/s across an RTX 3080 and RTX 3070, and 30, 41, and 65 tok/s across three MacBook serving setups.
Memory-efficient approaches target constrained devices
Reported approaches include 1.58-bit inference, streaming a 1B model layer by layer through 45MB RAM, adaptive MoE quantization, and streaming MoE experts from flash on a phone.
Media posts had a higher median all-time score
Deterministic analytics reports that 40 of 50 tweets included media (80%). The median all-time score was 70.59 for media posts and 3.12 for text-only posts.
Tutorials and lists had the highest format medians in this dataset
The supplied analytics reports median all-time scores of 236.72 for tutorials and 239.774 for lists, compared with 124.21 for announcements and 7.9 for case studies.
Statistical standouts
- View standout post 1 Score 21348.8 · 393.09× median
- View standout post 2 Score 9759.8 · 179.7× median
- View standout post 3 Score 1489.0 · 27.42× median
- View standout post 4 Score 1390.8 · 25.61× median
- View standout post 5 Score 878.7 · 16.18× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Vaishnavi
@_vmlops
2 posts
-
2. 0xSero
@0xSero
2 posts
-
3. Kshitij Mishra | AI & Tech
@DAIEvolutionHub
2 posts
-
4. Georgi Gerganov
@ggerganov
2 posts
-
5. Guri Singh
@heygurisingh
2 posts
-
6. Nav Toor
@heynavtoor
2 posts
llama.cpp’s creator contributed both a benchmark-style demonstration and a perspective on local AI
Georgi Gerganov posted an M2 Ultra llama.cpp demonstration reported at 300t/s and separately discussed cross-device, open-stack local AI and the increasing use of local LLMs.
Repeated creators covered setup guides and developer workflows
0xSero posted a framework-selection guide and a list of local-model harnesses. _vmlops posted a build-from-scratch agent-learning repository and a post describing a direct-server performance comparison.
Creator participation was broad in the supplied analytics
The analytics identifies 39 creators. Its top-five placement share is 20%, so the top five accounted for one-fifth of the measured placement share rather than a majority.
Since the previous snapshot
What changed since Aug 20, 2026
- 78% of the selected posts remained.
- The creator count changed by 0.
- The leading sentiment remained stable.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Local LLMs tweets
Top Local LLMs tweets from 39 creators
Ranked 01–50
- 01
@heynavtoor ·
🚨Someone just open sourced a computer that works when the entire internet goes down. It's called Project N.O.M.A.D. A self-contained offline survival server with AI, Wikipedia, maps, medical references, and full education courses. No internet. No cloud. No subscription. It just works. Here's what's packed inside: → A local AI assistant powered by Ollama (works fully offline) → All of Wikipedia, downloadable and searchable → Offline maps of any region you choose → Medical references and survival guides → Full Khan Academy courses with progress tracking → Encryption and data analysis tools via CyberChef → Document upload with semantic search (local RAG) Here's the wildest part: A solar panel, a battery, a mini PC, and a WiFi access point. That's it. That's your entire off-grid knowledge station. 15 to 65 watts of power. Works from a cabin, an RV, a sailboat, or a bunker. Companies sell "prepper drives" with static PDFs for $185. This gives you a full AI brain, an entire encyclopedia, and real courses for free. One command to install. 100% Open Source. Apache 2.0 License.
- 02
@heygurisingh ·
Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.
Watch video
- 03
@techNmak ·
Claude Code can run entirely on your local GPU now. Unsloth AI published the complete guide. The setup itself is straightforward - llama.cpp serves Qwen3.5 or GLM-4.7-Flash, one environment variable redirects Claude Code to localhost. But the guide is valuable because of what it explains beyond the setup: Why local inference feels impossibly slow: Claude Code adds an attribution header that breaks KV caching. Every request recomputes the full context. The fix requires editing settings.json - export doesn't work. Why Qwen3.5 outputs seem off: f16 KV cache degrades accuracy, and it's llama.cpp's default. Multiple reports confirm this. Use q8_0 or bf16 instead. Why responses take forever: Thinking mode is great for reasoning but slow for agentic tasks. The guide shows how to disable it. The proof it all works: Claude Code autonomously fine-tuning a model with Unsloth. Start to finish. No API dependency. Fits on 24GB. RTX 4090, Mac unified memory.
- 04
@ihteshamali ·
This feels illegal. Someone built a fully local deep research agent that writes its own search queries, hunts down sources, finds the gaps in its own answers, then keeps searching until it's done. It's called Local Deep Researcher. Drop in any Ollama model like DeepSeek, Llama, Qwen and give it a topic. It handles everything: → Generates the search query → Scrapes the web → Summarizes what it found → Identifies what it missed → Searches again to fill the gaps → Outputs a full markdown report with citations You set how many research loops it runs. The whole thing costs $0 to operate after setup. Perplexity Pro is $20/month for this exact workflow. 8,500 developers already starred it. 100% Open Source. MIT License.
Watch video
- 05
@stan_info ·
i’m not a local llm type of guy at all, was just curious and decided to mess around… ended up running a full uncensored qwen3.5-27b (abliterated) on my single 3090 ti with 262k context + tool calling. threw a cloudflare tunnel on it so i can hit the api from anywhere. huge thanks to @0xSero and @ggerganov for the insane work that made this possible oh my… intelligence will be everywhere soon
- 06
@ggerganov ·
Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)
Watch video
- 07
@ollama ·
Ollama 0.18.1 is here! 🌐 Web search and fetch in OpenClaw Ollama now ships with web search and web fetch plugin for OpenClaw. This allows Ollama's models (local or cloud) to search the web for the latest content and news. This also allows OpenClaw with Ollama to be able to fetch the web and extract readable content for processing. This feature does not execute JavaScript. If you have OpenClaw already running: openclaw plugins install @ollama/openclaw-web-search 🤖 Non-interactive (headless) mode for ollama launch ollama launch command can now run in non-interactive mode. This is perfect for: - Docker/containers: spin up an integration as a pipeline step to run evals, test prompts, or validate model behavior as part of your build. Tear it down when the job ends. - CI/CD: Generate code reviews, security checks, and other tasks within your CI - Scripts/automation: Kick off automated tasks with Ollama and claude code Try with: ollama launch claude --model kimi-k2.5:cloud --yes -- -p "how does this repository work?"
- 08
@ggerganov ·
llama.cpp at 100k stars now that 90% of the code worldwide is being written by AI agents, I predict that within 3-6 months, 90% of all AI agents will be running locally with llama.cpp 😄 Jokes aside, I am going to use this small milestone as an opportunity to reflect a bit on the project and the state of AI from the perspective of local applications. There is a lot to say and discuss and yet it feels less and less important to try to make a point. Opinions about viability of local LLMs are strongly polarized, details are overlooked, the scientific approach is lacking. Arguments are predominantly based on vibes and hype waves. One thing is clear though - local LLMs are used more and more. I expect this trend to continue and likely 2026 will end up being one of the most important years for the local AI movement. I admit that I didn't expect the agentic era to come so quickly to the local LLM space. One year ago, the available models were too computationally expensive for doing long-context tasks. There wasn't an obvious path towards meaningful agentic applications. The memory and compute requirements were huge. Last summer, with the release of gpt-oss, things started to change. It was the first time we saw a glimpse of tool calling that actually works well within the resource constraints of our daily devices. Later in the year, even better models were released and by now, useful local agentic workflows are a reality. Comparing local vs hosted capabilities at a given moment of time is pointless. To try put things into perspective: - We don't need frontier intelligence to automate searches and sending emails - We don't need trillion parameter models to be able to summarize articles or technical documents - We don't need massive GPU data centers to control our home appliances or turn the lights off in the garage I believe that there is a certain level of intelligence we as humans can comprehend and meaningfully utilize to improve our working process. Beyond that level, access to more intelligence becomes unnecessary at best and counterproductive at worst. I also believe that that level of useful artificial intelligence is completely within reach locally and it has always been just a matter of implementing the right software stack to bring it to the end user. With llama.cpp, I am confident that we continue to be on the right track of building that software stack! The llama.cpp project is going stronger than ever. With more than 1500 contributors, the project keeps growing steadily. From technical point of view, I think that llama.cpp + ggml is the only solution that actually makes sense. That is, the software stack must run efficiently on every possible device, hardware and operating system. The technology is too important to be vendor-locked. It has to be developed in the open, by the community, together with the independent hardware vendors. This is the only right way to build something that will truly make a difference in the long run. I won't try to convince you about what is currently and will be possible with local AI. We will just continue to build as usual. I am confident that after the smoke clears and we look objectively at what we have built together, the benefits will be obvious to everyone. Big shoutout to all llama.cpp maintainers. I feel extremely lucky to be able to work together with so many talented contributors. Every day I learn something new and I feel there is so much more cool stuff that we are going to build. Also, I am really thankful that the project continues to have reliable partners to support it! Cheers!
- 09
@sentient_agency ·
Nobody is talking about this. A developer built a research agent that works like a PhD student with unlimited time and zero salary. It's called Local Deep Researcher. Give it any topic and it takes over completely. It writes its own search queries, scrapes the web, reads every result, spots what's still missing, then searches again until there are no gaps left. You decide how many research loops it runs. Final output is a clean markdown report with full citations, ready to use. Runs entirely on your machine with any Ollama model DeepSeek, Llama, Qwen, whatever you prefer. Zero API costs. Zero subscriptions. Zero data leaving your device. Perplexity Pro charges $20/month for this exact workflow. 8,500 developers already starred it. 100% Open Source. MIT License. (Link in the comments)
Watch video
- 10
@Akashi203 ·
i built a local LLM inference engine that runs a 1B parameter model on a $10 board with 256mb ram. model sits on the sd card, streams one layer at a time through 45mb of ram You can use it as local LLM model backend for PicoClaw no python no cloud no api keys 80kb binary | pure c | zero dependencies github: https://t.co/0fe4ey4yz7
- 11
@0xSero ·
Here’s what I’d recommend if you’re just getting started in AI, local or otherwise. 1. Work with the compute you have, even the dumbest LLMs can be useful if you treat them as a node in your system. Some basic problems of what could be useful to get you started - tag all your screenshots - classify your emails - recommendation algo - scanning git history for patterns 2. If you want to try larger models use prime intellect or Hotaisle (even cheaper) to rent out whatever amount of VRAM you need to run models you like. - RTX 3090 rents for cents/h - RTX 6000 rents for 1-2 dollars/h - H100 rent for 2-3 dollars/h 3. Use the right frameworks: - VLLM, and SGlang for faster inference if you have 1, 2, 4, 8, 16 GPUs - exllamav3 and llama.cpp if you have non-power of 2 GPUs - MLX if you have Mac 4. Start with problems in your life that could use intelligence for automation - shopping - research 5. Don’t expect local models to code production projects just yet
- 12
@akshay_pachaar ·
Run Claude Code using local LLMs for FREE. No API costs. No data leaving your machine. Here's how it works: Claude Code lets you swap its backend via a single env variable. Point `ANTHROPIC_BASE_URL` to a local llama.cpp server, and it'll route all requests to whatever model you're running locally. The Unsloth team put together a step-by-step guide for this where you can run Claude code using Qwen3.5 It covers everything from model download to server setup to running Claude Code. The trick is serving your model on port 8001 via llama-server, then setting two env vars: `ANTHROPIC_BASE_URL` and a dummy `ANTHROPIC_API_KEY`. That's it. Claude Code thinks it's talking to Anthropic's API. If you don't want to pay per token for every agentic loop and want fast, private, cost-free coding runs, this is exactly what you're looking for. Link to the guide in the next tweet
- 13
@aiDotEngineer ·
TLMs: Tiny LLMs and Agents on Edge Devices with @cormacb https://t.co/u0fHD7j5kZ Function Gemma ships at 270 million parameters and runs nearly 2,000 tokens per second prefill on a Pixel 7. Out of the box, it hits 46% accuracy on a fixed set of app intents. Fine tune on a synthetically generated dataset and that clears 90% on eight of ten functions. Cormac walks through the two paths developers have for on device AI: a skill harness built on Gemma 4 with a restaurant roulette demo running fully on device. Then Eloquent, a production transcription app built by chaining two sub billion parameter models together. cc @osanseviero
- 14
@witcheer ·
Google TurboQuant maybe reduced my LLM memory by 6x. running 70B models on my Mac Mini could be real. I have Mac Mini M4. memory was always the issue, 16GB meant 30B models were pushing it, 70B were out of the question. TurboQuant changes that math. 6x memory reduction without quality loss. someone already ran Qwen 3.5-9B on a MacAir using the llama.cpp patch. 6x memory reduction means I could fit a 120B parameter model in the same RAM that currently fits a 20B. or run multiple agents in parallel. I haven't applied the patch yet. experimental, production workflows, not worth breaking things. but I'm watching community validation closely. does TurboQuant work with quantised models? most local inference runs 4-bit. if anyone's tested TurboQuant + GGUF on Mac Silicon, I am interested in the numbers.
- 15
@0xSero ·
Best harnesses for local models: 1. Droid: - Very good performance, forces the models to behave, you can wire in all your local LLMs very easily w BYOK - Allows you to use your local models as orchestrators/subagents so you can benefit from Cloud as models as well - Practically everything you need is already in Droid - This is my daily driver, I use Qwen3.5 models in it very happily 2. Zed IDE: - OpenAI compatible APIs are first class - Works like Cursor so if you prefer IDEs this is for you. - Clean design 3. Pi Coding Agent - Open source - Very token efficient - Codebase supports vllm and open weight models 4. RooCode: - Steer mode forces local/dumber models to behave by re-injecting the user's prompt and plan every action - Very easy to set up local providers 5. OpenCode: - Super easy to add new providers - Desktop app/Mobile web let's you use ur local models - Open source 6. Parchi - Any provider of any type is compatible - Very simple UX - Let's you operate your browser with your local models
- 16
@mudler_it ·
I've just released APEX (Adaptive Precision for EXpert Models): a novel MoE quantization technique that outperforms @UnslothAI Dynamic 2.0 on accuracy while being 2x smaller for MoE architectures. Benchmarked on Qwen3.5-35B-A3B, but the method applies to any MoE model. Half the size of Q8. Perplexity comparable to F16. Works with stock @ggml_org's llama.cpp. Open source (of course!), with ❤️ from the @LocalAI_API team. 👇Links to the model, repository and benchmarks below! (+ Bonus TurboQuant benchmarks with @no_stp_on_snek's TQ+! )
- 17
@heygurisingh ·
🚨 BREAKING: Someone open-sourced a full offline survival computer with AI, Wikipedia, and maps built in. Project N.O.M.A.D. is an open-source offline survival computer. Self-contained. Zero internet required after install. Zero telemetry. Everything runs locally on your hardware. What it includes: → Full Wikipedia archives via Kiwix → Offline maps via OpenStreetMap → Local AI models via Ollama + Open WebUI → Calculators, reference tools, resource libraries → A management UI to control everything from a browser One curl command installs the entire system on any Debian-based machine. Runs headless as a server so any device on your local network can access it. Minimum specs to run the base system: dual-core processor, 4GB RAM, 5GB storage. To run local LLMs offline, you want 32GB RAM and an NVIDIA RTX 3060 or better. No accounts. No authentication by default. No cloud dependency. No phone-home behavior. Built to function when nothing else does. The grid, the cloud, the API you depend on. None of it is guaranteed. The people building local-first systems right now are the ones who won’t be asking for help when access disappears.
- 18
@DAIEvolutionHub ·
Holy shit 🤯 Microsoft just open-sourced a framework that runs a 100B parameter LLM on a single CPU. No GPU. No cloud. No expensive setup. Just your laptop. It’s called BitNet. And it breaks one of the biggest assumptions in AI. Here’s the trick: Most LLMs use 16-bit or 32-bit floats. BitNet uses: 1.58 bits. Yes… bits. Weights are just: -1, 0, +1 That’s it. No heavy matrix math. Just simple integer operations your CPU already handles efficiently. The result is insane: • 100B model runs on CPU at 5–7 tokens/sec • 2–6× faster than llama.cpp on x86 • 82% less energy usage • 1–5× faster on ARM (MacBooks) • 16–32× lower memory The craziest part? Accuracy barely drops. Their flagship model (trained on 4 trillion tokens) performs competitively with full-precision models. They didn’t break the model. They removed the waste. What this unlocks: → Run LLMs fully offline → AI on phones, edge devices, IoT → No API costs for inference → Works even without reliable internet MacBook. Linux. Windows. It just runs. 27K+ GitHub stars. Built by Microsoft Research. 100% open source. This might be the moment AI stops being cloud-first… and becomes device-first.
Watch video
- 19
@BitcoinNewsCom ·
Jack Dorsey's Block just launched mesh-llm. It's a decentralized, peer-to-peer inference network for open source AI models. The idea is to pool spare GPU compute across machines to run models too large for any single device. Rather than using a centralized cloud, it's just nodes gossiping over a mesh. Your spare GPU becomes part of a distributed AI network that anyone can use. It uses Nostr for node discovery and the whole thing is MIT licensed and built on llama.cpp. It has the same open and permissionless philosophy as Bitcoin, without a central server that can be switched off.
- 20
@TheAhmadOsman ·
People keep saying “VRAM is all that matters” for local LLMs > It’s not just wrong, it’s misleading When running LLMs locally, the bottleneck is NOT just “VRAM size” It’s: - memory bandwidth - interconnect (PCIe vs NVLink vs RDMA) - inference engine (vLLM, TensorRT-LLM, SGLang) (Also, Unified Memory ≠ VRAM - and it’s much slower) > People still evaluate hardware in isolation A piece of hardware is USELESS if the software stack isn’t there You are not buying silicon you are buying: - kernels - runtime - framework support - model and quantization compatibility > Buying this Intel GPU? Good luck waiting years for that ecosystem to mature Meanwhile even Blackwell isn’t fully mature and it’s already widely adopted > There are already too many problems across the stack (e.g. GPU → framework → model → quantization) Example of what people keep falling for: “32GB VRAM card for $1K???” But: > unified memory class bandwidth > weak software stack > no CUDA You didn’t buy a GPU you bought a debugging project > Yes, you can make it work You might see: > ~54 tok/s aggregate on 3x Arc A770 But it comes with: - buggy drivers - missing tooling - kernel level hacks And real performance looks like: > ~13 to 14 tok/s single request > unstable configs and constraints This is the trap... On paper it looks competitive in practice it’s fragile > My take This class of products gets killed in less than 2 generations The stack won’t mature fast enough and it wastes your time > If you care about local AI Optimize for: - bandwidth - interconnect - software maturity Not just VRAM TLDR > The stack is the moat > CUDA still wins > Unless you wanna waste your time, this is a NO BUY —Buy a GPU, The Movement
- 21
@gneubig ·
In the spirit of language model freedom, as a weekend project I finally set up my own fully local coding agent: * GMKtek box with AMD Ryzen * Qwen3.5 30B A3B, served with lemonade server+llama.cpp * ngrok to expose the endpoint * OpenHands as the agent, currently through the CLI Honestly I am still a bit surprised at how well this works. Qwen3.5-30B-A3B is a good model, and the serving experience was pretty fast and smooth.
- 22
@TheAhmadOsman ·
Local LLMs, Buy a GPU, and the Case for Cognitive Security AGI? There is no guarantee we reach it anytime soon, if ever. And even if we do, there is certainly no guarantee it will run on your machine. Betting your agency on either assumption is a mistake. What is guaranteed is that large language models are already shaping how people think, decide, and act. They mediate search, writing, coding, persuasion, and increasingly judgment. That makes understanding them less of a technical curiosity and more of a survival skill. This is why my default advice is simple: run local LLMs. You need to learn how these systems work. When you operate models yourself, you stop treating them like magic. You see how they are prompted, steered, constrained, fine-tuned, degraded, and exploited. You learn where they are brittle, where they are persuasive, and where they quietly fill in gaps with confidence instead of truth. That hands-on understanding is the most basic form of cognitive security. CogSec is not about paranoia or rejecting AI. It is about literacy. If you know how models are shaped, you can recognize when they are shaping you. If you do not, you are operating blind. People who only interact with LLMs through polished products are easy to influence. They cannot tell when they are being nudged, framed, or one-shot into a conclusion or a psychosis. They cannot distinguish between alignment choices, product incentives, or outright manipulation. Whoever controls the model controls the gradient of influence. Owning your compute and running models locally flips that power dynamic. It gives you inspection, friction, and intuition. You are no longer a passive endpoint in someone else’s system. You are a participant who understands the machinery. You do not need AGI for this to matter. The models we already have are sufficient to warrant cognitive defense. Buy a GPU. Run the models. Learn their failure modes. That is not a hobby. That is baseline cognitive security in an LLM-saturated world.
- 23
@heynavtoor ·
In 2026, OpenAI made you rent your own conversations. GPT-5.5. Five dollars per million input tokens. 30 dollars per million output tokens. Every prompt logged. Every response stored. Every keystroke a line item on your credit card. ChatGPT Plus. 20 dollars a month. 240 dollars a year. Cancel and you lose your history. Claude Pro. 20 dollars a month. 200 dollars a year. Same deal. OpenAI booked 13 billion dollars in revenue in 2025. It lost 20 billion trying to earn it. But OpenAI has a problem. Her name is Justine Tunney. In November 2023, one month before the AI subscription economy took off, she quietly shipped llamafile at Mozilla. llamafile is not a chatbot. It is one file. You download it. You double-click it. A chat interface opens in your browser. The AI answers. Locally. On your laptop. Offline. The model weights. The inference engine. The web server. The chat UI. The API endpoint. All fused into one executable. Justine is a Google Brain alumni who left in 2018 to build Cosmopolitan Libc. The technology that makes a single C binary run natively on Windows, Mac, Linux, FreeBSD, OpenBSD, and NetBSD. Same file. Six operating systems. No recompiling. She wrapped llama.cpp inside Cosmopolitan and shipped the impossible. Then she went further. She hand-wrote 84 new matrix multiplication kernels. CPU inference got 30 to 500 percent faster. A Raspberry Pi 5 started returning tokens in real time. The project stalled in 2025. On October 29 2025, https://t.co/qMWt5ahekQ officially revived it under a new org. March 2026 shipped a full rebuild with GPU support, a terminal UI, multimodal input, and whisperfile for speech-to-text. Latest release dropped June 2 2026. 25,159 stars. Apache 2.0. Ships prebuilt files across a range of open models, from tiny 0.6B all the way up to 27B parameters. Whisperfile does the same thing for audio. One file. Double-click. Transcribe anything. Offline. OpenAI burned 20 billion dollars in 2025 renting you access to software. One engineer in Mountain View built the file they can never take back. (Link in the comments)
- 24
@sudoingX ·
first local model running on the rog 5090 mobile. gemma 2 2b it q4_k_m through llama.cpp, hermes agent pointing at the local endpoint. sitting on the balcony watching the beach while the model answers prompts. 24gb vram in a portable machine running a full inference stack plus an agent harness. same class as the desktop 3090 i benchmark on, now outside with coffee. the hermes agent local model auto-detection i pushed a while back just resolved the endpoint on its own. good time seeing some my own code work in the wild, on my own hardware. small moment but it hit. starting small on purpose. pulling carnice-27b, qwen 3.5-27b dense, gemma 4 dense with hf cli next. same hardware, same quant, real agentic coding tests. comparison data dropping soon. beast is going to run wild stuff this week. keeping you all posted.
- 25
@ihteshamali ·
🚨This is absolutely amazing…You can now run a full AI + Wikipedia + offline maps computer that works when every server on Earth goes dark. It's called Project N.O.M.A.D. and it's completely free. Two commands to install on any Ubuntu machine: curl the script. sudo bash it. What you're running locally: → Offline Wikipedia via Kiwix → OpenStreetMap with no internet → Local LLMs via Ollama + OpenWebUI → ZIM offline archive library → Zero telemetry. Zero cloud dependency. Opens in your browser. Works on a Raspberry Pi. Most people are one internet outage away from having nothing. This is the alternative. 100% Open Source. MIT License.
- 26
@_vmlops ·
MOST DEVS USE AI AGENT FRAMEWORKS WITHOUT UNDERSTANDING WHAT'S HAPPENING INSIDE This repo teaches you to build agents from scratch local llms, no black boxes, real understanding 14 hands-on examples covering everything: ▪️ function calling & tool use ▪️ persistent memory across sessions ▪️ react pattern (reason → act → observe) ▪️ tree of thought, graph of thought, chain of thought ▪️ atom of thought planning ▪️ error handling with retries and fallbacks each example has code + a CODE.md explaining what every line does + a CONCEPT.md explaining why it matters the philosophy: learn by building understand deeply, then use frameworks wisely https://t.co/pC9ff18VQd
- 27
@Sumanth_077 ·
Train LLMs locally without writing a single line of code! @UnslothAI just released Unsloth Studio - an open-source web UI for training and running models. Here's how it works: You upload a PDF, CSV, or DOCX file. The Data Recipes feature automatically transforms it into a structured training dataset via a graph-node workflow. No manual formatting needed. Then you select a model from Hugging Face or your local files. Pick your training method - LoRA, QLoRA, or full fine-tuning. The UI pre-fills sensible defaults based on your model. Start training and watch live metrics - loss curves, GPU usage, gradient norms. Everything runs locally with 2x faster training and 70% less VRAM than standard setups. Here are the key capabilities: • Chat with GGUF and safetensor models - supports tool calling, web search, and code execution in a sandbox. • Compare models side-by-side - load your base model and fine-tuned version to see how outputs differ. • Export to any format - save your trained models as GGUF, safetensors, or LoRA adapters for use with llama.cpp, vLLM, Ollama, or LM Studio. • Multi-modal support - train text, vision, audio, and embedding models all in one interface. It runs 100% offline on your hardware.
Watch video
- 28
@socialwithaayan ·
🚨 OpenClaw and Hermes showed what an operator agent should look like. Someone just built the version that runs 100% on your machine 🤯 It's called Atomic Agent. Browser control. File management. Shell commands. Document parsing. Scheduled tasks. Persistent memory. All through llama.cpp. All local. So instead of routing your files, prompts, browser tabs, and usage patterns through someone else's servers, nothing ever leaves your hardware. Here's what makes it different from every other local agent I've tried: Most of them treat llama.cpp like a cloud API with a local address. They stuff everything into the prompt and let context grow until the model chokes on its own history. Atomic Agent moves state out of the prompt into SQLite, keeps the prompt prefix byte-stable so the KV-cache actually gets reused, and pipes every tool call through a GBNF grammar so nothing comes back malformed. The result showed up on GAIA Level 1: 69.8% for Atomic Agent, 58.5% for Hermes. Same model. Same hardware. Same budget. 1.6x faster per task. What it can operate: → browser (Chrome, Edge, any Chromium-family) → files, shell, git, clipboard, notifications → documents (PDF, DOCX, XLSX, PPTX, ODT, RTF) → memory with notes, lessons, procedures, and reflection → tasks with cron, webhooks, and self-scheduling → MCP servers and local Markdown skill playbooks Run it fully local with llama.cpp or bring your own cloud API keys for OpenAI, Anthropic, Gemini, and 20+ providers. Your choice. Dangerous actions go through an approval gate. The model proposes, you approve. One command install. Ships as a standalone binary. MIT License. 100% open source.
- 29
@_vmlops ·
SOMEONE RAN A 122B MODEL ON THEIR MACBOOK... IN ONE NIGHT no cloud. no api fees. no data leaving the machine they went through 3 generations: → ollama + proxy: 30 tok/s → llama.cpp + proxy: 41 tok/s → mlx native server: 65 tok/s the breakthrough? they killed the proxy entirely wrote a 200-line python server that speaks the anthropic api directly. claude code thinks it's talking to the cloud. it's not that one change made it 7.5x faster you can even control it from your iphone over imessage. while offline this is wild engineering → https://t.co/TE8RpBvDhb
- 30
@sudoingX ·
compiling llama.cpp with cuda 12.8 on the rog strix scar 18 5090. this is the mobile 5090, 24gb vram in a laptop. same vram class as the 3090 desktop i run benchmarks on, just portable. i can run these tests from anywhere now. first up carnice-27b vs qwen 3.5 27b dense vs gemma 4 on this exact machine. same class, same hardware and real agentic coding tests. data dropping soon.
- 31
@DAIEvolutionHub ·
CHINA JUST DROPPED AN OCR MODEL THAT CHANGES EVERYTHING. A tiny 3B-parameter model can read an entire 100-page PDF in a single pass. No page splitting. No context loss. No cloud APIs. Meet Unlimited-OCR 👇 • Reads full documents with a 32K context window • Scores 93% on OCR parsing benchmarks (+6 points over the baseline) • Keeps error rates below 0.11 even after 40+ pages • Multilingual out of the box • Runs 100% locally on your own hardware • Works with Transformers, vLLM, SGLang, Docker, Ollama, and llama.cpp Here's why this matters. Most OCR systems still treat every page as a separate image. That means broken tables, lost references, and missing relationships between pages. Unlimited-OCR processes the entire document as one continuous context. So tables stay intact. References remain connected. Document structure is preserved from beginning to end. Meanwhile, many businesses are still: • Paying $1.50 to $15 per 1,000 pages • Uploading sensitive PDFs to cloud providers • Waiting on OCR API responses This runs completely offline. No subscriptions. No API bills. Just your hardware. Built by Baidu to push beyond DeepSeek-OCR. It's already crossed 1.9M+ downloads on Hugging Face. And surprisingly, almost nobody is talking about it yet. Open source keeps shrinking the gap between "enterprise software" and "free." 🔗 https://t.co/Wt0SN0UMSy
Watch video
- 32
@cnxsoft ·
Online guide to select the best hardware for local LLM/AI deployments. https://t.co/Q2LAYo5xxx The website relies on Qwen 3.5 models and shows the price, performance, power consumption, and more for a range of hardware from Raspberry Pi 5 16GB to NVIDIA DGX Spark. In one specific test (Qwen 3.5 9B), a system with a $260 Intel Arc graphics card performs just as well as $4,000 computers. The project also accepts user submissions, but the process is a little convoluted.
- 33
@Shruti_0810 ·
A developer replaced ChatGPT Pro + Claude Code Max + Cursor with a $300 ZimaBoard running local LLMs. Now his entire AI bill is basically the cost of electricity. Here's what changed: • Runs an x86 mini PC smaller than a hardcover book. • Hosts local LLMs, Proxmox, and every client's staging environment. • Uses CLAUDE.md + AGENTS.md per repository to give each project its own context. • Routes everyday coding to lightweight models and heavier migrations/audits to larger local models. • Client code never leaves his hardware. No cloud APIs. No recurring AI subscription costs. He says he went from spending $459/month on AI tools to around $3/month in electricity. If this trend continues, we're about to see a shift from renting intelligence to owning it. The biggest AI upgrade in 2026 might not be another frontier model. It might be buying your own hardware. Bookmark this and read the article below before everyone starts building local AI workstations.
Watch video
- 34
@vectro ·
Gemma 4 E2B Running on RTX 3080 & 3070 combined * llama.cpp * Ubuntu server * No GUI * BF16 precision * 52.6 tok/sec. See the comments for the full set of commands to run in terminal and GUI locally or served over the network.
Watch video
- 35
@morganlinton ·
Fun little Sunday morning project, starting work on a little offline LLM I'm calling Wilderness Bot. I go hiking and backpacking out in the wilderness a lot, and want a way to have a little LLM loaded with wilderness medicine info, survival guides, etc. Decided to build in Python because it has been my go-to for over a decade. Almost went with Rust on this because I've been digging Rust lately, but decided to just stick with ol' faithful. Here's a bit more about the architecture: - Keeping the model small: targeting quantized 3B-4B class models so it can run on a modern phone. - Instead of cramming everything into the model weights, the knowledge lives in a local SQLite bundle. - The bot uses retrieval-first answers, so it pulls the most relevant chunks before responding. - I like the idea of keeping things fresh and allowing for monthly updates. And with this architecture monthly updates are easy: refresh the data bundle, keep the model the same. - Dev setup already supports local testing with Ollama, but the data format is portable to phone runtimes. - The long-term mobile path could work with llama.cpp, MLC, or ExecuTorch. -- As for the data I've loaded in so far this morning: - Seeded with public, refreshable safety sources. - CDC emergency water storage/preparedness guidance. - American Red Cross first-aid steps. - American Red Cross wildfire safety guidance. - Planning to expand with wilderness medicine references, survival guides, and licensed field manuals. -- And of course, I bought a domain name for it - Wilderness(.)bot, which I was kinda surprised was available to hand register, but sometimes you get lucky. Will share more as I go. Overall, I feel silly as an engineer that loves tinkering with LLMs, that I still go into the wilderness with PDFs of all this stuff, time to change that. Today, Wilderness Bot is born.
- 36
@Shruti_0810 ·
A LOCAL LLM RUNNING ON ONE 3090 JUST REPLACED GOOGLE HOME FOR 29 SMART DEVICES WITH ZERO CLOUD. It doesn't just answer questions. Ask, “Do I need a jacket?” and it checks your actual weather sensor before replying. The cloud is becoming optional.
Watch video
- 37
@alphabatcher ·
BEST local LLMs to run in 2026: High-performance (24+ GB VRAM, preferably with multiple GPUs) • Kimi K2 - 1T params, 32B active. MoE beast • GLM-4.7 (Z AI) - 30B-A3B MoE, SWE-bench 73.8% • DeepSeek V3.2 - 671B / 37B active. Still the open-source king • Qwen3 235B-A22B - insane quality/cost ratio if you have the iron Mid-range (16-24 GB VRAM / RAM) • Qwen3 30B-A3B - punches way above its weight, stable on long context • Gemma 3 27B - Google's best open release yet • Nemotron 3 Nano 30B - Math500: 91%. Best-in-class if you need math Lightweight models (8-16 GB RAM, can run without a dedicated GPU) • Qwen3 8B / 4B / 1.7B - the best small model family right now • Gemma 3 4B - surprisingly capable on CPU • Phi-4 (14B) - Microsoft doing a lot with a little The local AI stack is genuinely catching up to the cloud
- 38
@InduTripat82427 ·
Holy shit...Someone tried replacing Claude with a local LLM… …and waited 13 minutes for THIS: > “I am a large language model, trained by Google.” That’s it. 13 minutes. One useless sentence. Let’s be honest — we’ve ALL had this thought: “Why am I paying for Claude when I can just run open-source models locally for free?” So he tried. Ran Gemma locally. No API costs. Full control. Sounds smart, right? Here’s what actually happened: Before he even asked a real question… The model was already drowning. Tools like Claude Code don’t start from zero. They load 20,000+ tokens of system context: • instructions • memory • tools • workflows Your “free” local model? It’s choking before the race even starts. And then you wait… …and wait… …and wait… …for a generic sentence you could’ve gotten in 0.8 seconds. This is the part no one tells you: Running LLMs locally isn’t just about compute. It’s about: context handling latency orchestration tooling You’re not replacing an API. You’re trying to replace an entire AI infrastructure stack. The API bill hurts. But wasting hours waiting for responses? That’s way more expensive.
- 39
@heyrimsha ·
Running a 60GB AI model on a phone with 12GB of RAM should be impossible. Someone just did it anyway. It's called BigMoeOnEdge and it runs gpt-oss-120b (a model 5x bigger than the phone's RAM) at 2.2 tokens per second on plain CPU. No GPU. No NPU. Four cores and flash storage. The trick: MoE models only use a few "experts" per token. Instead of loading all 60GB into RAM, it streams only the experts each token actually needs, straight from flash, at the exact moment the model asks for them. The rest stays on disk. → gpt-oss-120b (60GB) on a 12GB phone at 2.2 tok/s → 14x faster than the same file loaded the ordinary way (0.09 tok/s with mmap) → Byte-for-byte identical output to running the full model in RAM → Qwen3-30B at 5.2 tok/s, Gemma-4-26B at 4.1 tok/s on the same device → Built on stock llama.cpp, upstream untouched, tracked as a submodule → Android demo app with a live telemetry panel and every knob exposed Apple, Google, and Samsung spend billions trying to fit smaller models onto their phones. One developer streamed a bigger one from flash and got the same answer. 100% Opensource. https://t.co/X8Qk2Y5T47
Watch video
- 40
@ujjwalscript ·
Your "Local LLM" and “Free AI” dev setup is a massive WASTE of time and money! The hottest trend on X right now is showing off your "local-first" setup. Developers are buying expensive NVIDIA 5090s, bragging about running Llama-3.2 or Phi-3.5 completely offline, and treating cloud API users like absolute peasants. "Look at my zero-latency inference! Look at my data privacy!" It’s a beautiful flex. It's also an engineering Delusion. Here is the truth people are refusing to admit because they want to justify their hardware spending: You are sacrificing massive cognitive reasoning just to say you run on localhost. When you switch your development workflow from a massive, frontier cloud model to a quantized 8B or 7B Small Language Model (SLM) running on your machine, you aren't upgrading. You are downgrading your assistant from a Principal Architect to an intern who drank too much coffee. Yes, SLMs are incredible for hyper-specific, narrow tasks like text classification or basic autocomplete. But for complex system design, edge-case debugging, and cross-repository code auditing? They hallucinate under pressure because they lack the deep parameter weight to handle complex abstraction. Stop trying to turn your local workstation into a miniature data center. Use the frontier cloud models for the heavy intellectual lifting - the system boundaries, the state management, the algorithmic strategy. Use local models for basic syntax completion.
- 41
- 42
@JulianGoldieSEO ·
OLLAMA + CODEX APP JUST BROKE LOCAL AI CODING You can now run OpenAI’s desktop coding agent on models sitting on your own laptop. No subscription. No cloud code sharing. No waiting in line. What Changed: → Ollama 0.24 adds official support for the Codex app → Codex can now run on local models like Qwen 3, Gemma 4, GLM, DeepSeek, and Kimi → You launch it with one command and pick your model from the dropdown Why This Is Huge: ✓ Codex gets parallel work trees for isolated coding tasks ✓ Built-in browser lets you point at UI elements and ask for edits ✓ Review mode lets you diff, comment, branch, commit, and merge in one place Best Use Cases: → Private code refactoring offline → Coding on planes or bad Wi-Fi → Building small internal tools in hours → Automated code review for solo builders → Learning to code without burning usage limits Model Picks: ✓ 16GB RAM: Qwen 3 14B ✓ Lower RAM: Qwen 3 8B or Gemma 4 4B ✓ Bigger jobs: Kimi K2.6 through Ollama Cloud The lesson: Local AI coding is no longer just a nerdy terminal trick. It is turning into a real agent workflow you can actually build with.
Watch video
- 43
- 44
@Rus_Khairullin ·
Vitalik published a detailed post on how he set up a fully local, self-sovereign AI - no cloud, maximum privacy and security. AI agents can already work for hours, use tools and modify their own code. But most (even open-source) ignore security: data leaks, hidden instructions, jailbreaks through websites. Vitalik: “We just won privacy in messaging. Now we can lose it all through AI.” Main threats he’s protecting against: • Data leaks into the model and outside • Jailbreaks and malicious content • Backdoors in models • Accidental or intentional sending of private info What Vitalik uses: • Hardware: Laptop with RTX 5090 (24GB) - 90 tokens/sec on a 35B model. Best price/performance option. • Model: Qwen3.5-35B (perfect balance of speed and quality) • OS: NixOS - everything in one config, easy to share • Inference: llama-server + llama-swap • Agents: pi framework • Sandbox: bubblewrap - every tool in strict isolation Useful features: • Local knowledge base (Wikipedia + documentation) • Search via private SearXNG • Daemon for Signal and email: reads everything, sends to himself freely, to others - only after human confirmation • For Ethereum: separate daemons with limits and double confirmation (human + LLM) Experience: Handles simple tasks well (write Snake, flashcards, audio transcription). Complex code is still weak - sometimes has to use external models. Vitalik’s vision: Fully local stack + tools for safe use of remote models (ZK-API, mixnets, TEE). Goal - make AI a tool for privacy and sovereignty, not surveillance. Who’s already running local LLMs in 2026? What hardware and stack are you using? @VitalikButerin
- 45
@DBCrypt0 ·
OMG a new local LLM dropped that you can run on your Mac Mini and it’s just as good as Opus 4.6! 🔥 Local Model: Hi! Before we begin, tell me about yourself and what you want to call me? Me: Let’s call you Max. I’m a content creator, podcast host, and Web3/AI researcher. Local Model: Perfect! I think Charlie is a great name for me and I can definitely help with your web design business. Me: …I said Max. And I’m not a web designer. Local Model: You are right, you did say Max. I’ll make note of it and make sure not to forget next time. Me: Ok, cool. Did you add it to your identity file? 5 minutes pass… Local Model: what’s an identity file? Would you like me to create one? 3 minutes later… Done! I deleted the memory and agents file and created a new identity file. I made sure to add to the top that my name is Charlie like you requested. Me: … Local Model: Hi, I’m Bob. What are we working on today? Don’t let anyone tell you local models are ‘just as good’ if you’ve ever used Claude
- 46
- 47
- 48
@jshguo ·
I used to think AMD GPUs were terrible for running local AI models. But today I tried Qwen3.6 Uncensored locally on my 7900 XT with llama.cpp. Turned off deep thinking and honestly… it feels really good. Way faster than I expected, and actually usable for simple/fast tasks like translation.
- 49
@JulianGoldieSEO ·
OLLAMA JUST FIXED THE BIGGEST PROBLEM WITH LOCAL AI AGENTS Gemma 4 could use tools before. Now it can actually finish the job. What changed: → Ollama 0.32.1 improves tool-response continuation → Gemma 4 can call a tool, process the result, and continue working → Fewer abandoned tasks, forgotten instructions, and half-finished answers The real-world test: ✓ A messy 400-line Python script was analyzed locally ✓ The model found 3 performance bottlenecks and rewrote them ✓ It ran the tests and reported the results without losing the task The speed upgrade: ✔ MLX support made Gemma 4 nearly 90% faster on one coding benchmark ✔ Reported output jumped from roughly 50 to 95 tokens per second ✔ A memory-leak fix lets longer jobs run without constant restarts The lesson: Stop testing local AI with toy prompts. Give it a real file, multiple steps, and a result it must verify. That is how you find out whether your local agent can replace actual manual work.
Watch video
- 50
Best Tweets by Topic