Edge and on-device deployment
Deploying language and multimodal models directly on phones, wearables, edge systems, and microcontrollers under strict memory, power, and latency limits.
34%
Best tweets about Small Language Models
50 Best Tweets About Small Language Models (2026)
Find the best tweets about small language models, including compact architectures, on-device AI, benchmarks, fine-tuning, efficiency, and deployment.
Small and compact language models, on-device inference, efficiency, benchmarks, fine-tuning, hardware constraints, and real deployments.
- Creators
- 44
- Updated
Original Xholic analysis
What 50 top Small Language Models posts reveal
The dataset is predominantly supportive and positive: 33 of 50 posts are labeled supportive (66%), and 39 are labeled positive (78%). Its most common themes are edge/on-device deployment (17 posts), small-model capability progress (15), and hardware-aware model selection (14). The cited posts pair enthusiasm for local and compact models with practical constraints around task fit, agent capability, speed, context, quantization, and narrow-device output limits. [2034015670837600686, 2074697730874823077, 2076135762291261627]
- Dominant tone
- Positive
- Median score
- 17.4
- Leading format
- Other
- Recent posts
- 42%
78% of posts
All-time engagement
100% of posts
Published in 90 days
Conversation map
The themes creators return to
Small-model capability progress
Small-model releases and capability claims for coding, reasoning, tool use, multimodal tasks, multilingual use, and structured extraction.
30%
Hardware-aware model selection
Matching model size, quantization, context limits, and expected speed to RAM, VRAM, CPUs, Apple silicon, phones, and other available hardware.
28%
Specialized training and fine-tuning
Fine-tuning, reinforcement learning, distillation, and specialized post-training that adapt compact models to domain tasks, agents, and guardrails.
26%
Model compression and efficiency
Compression techniques that make models cheaper to run, including low-bit quantization, 1-bit weights, KV-cache compression, sparsity, and memory-efficient architectures.
22%
Local and private inference
Running compact open models locally with privacy, lower API costs, and practical setup guidance across laptops and home hardware.
16%
Task-specific evaluation and selection
Choosing models with workload-specific evaluations rather than broad leaderboards, including capability thresholds for agents and narrow business tasks.
14%
Sparse and MoE inference
Sparse mixture-of-experts models, expert paging, and other approaches that reduce active compute or enable oversized models on limited memory.
6%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 48
- Median reposts
- 10
- Median replies
- 10
- Median views
- 8.6K
Posts with media make up 76% of this collection. Their median all-time score is 19.1, compared with 11.3 for text-only posts.
Format mix
- Other 100% · score 17.4
Consensus and debate
Where creators agree, and where they do not
Shared view
Fit models to hardware and workload
A recurring practical theme is matching local models to available hardware and the specific workload. The cited posts argue that smaller hardware may suit narrower workflows and that model selection should be evaluated on the intended task rather than a broad leaderboard.
Shared view
Edge deployment is broadening
Posts describe on-device deployment across Apple devices, phones, and microcontrollers. They highlight compiled runtimes, low memory footprints, offline operation, and resource-constrained inference; the cited claims are product- or project-specific.
Shared view
Specialization is a recurring SLM theme
Specialization and training recur as compact-model levers. The evidence includes a low-cost GPT-2-grade training claim, a proposal to distill specialized SLMs for agent evaluation and guardrails, and a small from-scratch model positioned for education.
Open debate
Local inference does not imply full replacement
Self-hosting is framed as useful, but not as a universal replacement for cloud models. One post notes that tool use, context size, and quantization—not weight size alone—shape whether a local model is useful for agents; another limits smaller hardware to smaller workflows.
Open debate
Feasibility claims come with sharp constraints
The cited feasibility demonstrations have clear constraints. The SSD-paged MoE example reports roughly one token every 10–20 seconds, while the $8 microcontroller project is described as TinyStories-trained and limited to short narrative output rather than general QA or tool use.
Open debate
Benchmarks do not replace task evaluation
Capability remains task-dependent in these posts. One warns to expect hallucinations from an on-device model, another reports a capability floor for coding agents, and another recommends building workload-specific evaluations for narrow tasks.
What performs
Patterns behind standout posts
Top outliers featured deployment, hardware, and training topics
The five supplied score outliers span local-model setup guidance, hardware-tier recommendations, on-device tinkering, and low-cost model training. Their all-time scores range from 839.64 to 1847.42, versus a dataset median of 17.36.
Media posts had a higher median score
Media-bearing posts had a median all-time score of 19.13, compared with 11.35 for text-only posts. Media appeared in 38 of 50 posts (76%). These figures describe an association in this evidence set, not a causal media effect.
Hardware-fit discussion outperformed two other prominent themes
Hardware-aware model selection had a theme median all-time score of 16.96. This is higher than the cited medians for small-model capability progress (14.675) and edge/on-device deployment (8.828), but lower than specialized training and fine-tuning (93.414).
Statistical standouts
- View standout post 1 Score 1847.4 · 106.42× median
- View standout post 2 Score 1707.2 · 98.34× median
- View standout post 3 Score 1702.5 · 98.07× median
- View standout post 4 Score 1582.8 · 91.18× median
- View standout post 5 Score 839.6 · 48.37× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Akshay 🚀
@akshay_pachaar
2 posts
-
2. Alex Finn
@AlexFinn
2 posts
-
3. andrew chen
@andrewchen
2 posts
-
4. BURKOV
@burkov
2 posts
-
5. Lior Alexander
@LiorOnAI
2 posts
-
6. Towards Data Science
@TDataScience
2 posts
Alex Finn emphasizes pragmatic local use
Across two posts, Alex Finn presents local models as a privacy- and cost-oriented workflow option, while noting that smaller hardware is unlikely to replace every AI call and may instead cover smaller workflows.
Akshay Pachaar covers tooling and guardrails
Akshay Pachaar’s posts cover an on-device framework described as Core AI and a proposal for a distilled, specialized SLM to serve as an agent evaluator and runtime guardrail.
Andrew Chen discusses product fit
Andrew Chen describes local models as usable for many cases while distinguishing local capability from cloud capability. He also argues that, where model-quality differences are less apparent for common prompts, factors such as privacy, bundling, and product design may matter.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Small Language Models tweets
Top Small Language Models tweets from 44 creators
Ranked 01–50
- 01
@AlexFinn ·
I don't care what computer you have, you should be running local models It will save you a money on OpenClaw and keep your data private Even if you're on the cheapest Mac Mini you can be doing this Here's a complete guide: 1. Download LMStudio 2. Go to your OpenClaw and say what kind of hardware you have (computer and memory and storage) 3. Ask what's the biggest local model you can run on there 4. Ask 'based on what you know about me, what workflows could this open model replace?' 5. Have OpenClaw walk you through downloading the model in LM Studio and setting up the API 6. Ask OpenClaw to start using the new API Boom you're good to go. You just saved money by using local models, have an AI model that is COMPLETELY private and secure on your own device, did something advanced that 99% of people have never done, and have entered the future. There are some amazing local models out there too right now. Nemotron 3 and Qwen 3.5 are fantastic and can be ran on smaller devices Own your intelligence.
- 02
@AlexFinn ·
I don't care what kind of hardware you have, you should be running local models It will save you a ton on money on OpenClaw and keep your data private Even if you're on the cheapest Mac Mini you can be doing this Here's a complete guide: 1. Download LMStudio 2. Go to your OpenClaw/Hermes and say what kind of hardware you have (computer and memory and storage) 3. Ask what's the best local model you can run on there (probably will be Gemma 4 or Qwen. if you have a big computer, it will be GLM) 4. Ask 'based on what you know about me, what workflows could this open model replace?' 5. Have OpenClaw walk you through downloading the model in LM Studio and setting up the API 6. Ask OpenClaw to start using the new API Boom you're good to go. You just saved money by using local models, have an AI model that is COMPLETELY private and secure on your own device, did something advanced that 99% of people have never done, and have entered the future. If you are on smaller hardware you probably are not going to replace all your AI calls with this, but you could replace smaller workflows which will still save you good money Own your intelligence.
- 03
@0xSero ·
Best models to run on your hardware level I'll be doing this every week, I hope you guys enjoy. ---- 8 GB ---- Autocomplete for coding (like Cursor Tab) - https://t.co/Jyf766kmyd - https://t.co/dK1CQwCGqD Tool calling, assistant style - https://t.co/Jf7RY3dZmZ ---- 16 Gb ---- Here things get better: Multimodal - https://t.co/WNwxTttMQC - https://t.co/1U5HR9iWRX - https://t.co/OiOkDoahZ8 ---- 24 GB ---- - The best model you can get (thanks Qwen) https://t.co/fy8INjJP8N - Great model (strong agents) https://t.co/CRpiKlSX5d - Mine hehe https://t.co/YBeUveU0M6 I'm doing a weekly series
- 04
- 05
@karpathy ·
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: https://t.co/vhnK0d3L7B Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
- 06
@akshay_pachaar ·
Apple finally did it. Its new framework, Core AI, runs models entirely on Apple silicon, so inference happens on the user's device with zero server calls and zero token bills. That means Qwen, Mistral, and SAM3 running natively across iPhone, iPad, Mac, and Vision Pro. It's a memory-safe Swift API that compiles models ahead of time for near-instant load. Pulling one in takes a few lines: let segmenter = try await ImageSegmenter(resourcesAt: sam3ModelURL) let response = try await segmenter.segment(image: inputImage, prompt: "flower") The launch goes beyond the runtime, though. It ships curated open models packaged for Swift, PyTorch extensions to convert your own, and an optimizer that shrinks models layer by layer with minimal accuracy loss. There's also a macOS debugger that profiles performance and traces behavior back to your original Python, plus Xcode tools to validate models before they ship. For any team that wanted real on-device AI without a cloud bill attached to every user, this is the answer. Models repo: https://t.co/yZCIhrM6YQ
- 07
@the_smart_ape ·
everyone's talking about @karpathy autoresearch and most of you have no idea what it actually does. there's a training script (train(dot)py) that trains a small language model, basically a baby GPT. and there's an instruction file (program(dot)md) that tells an AI agent what to do. you press go. the agent tweaks the training script, trains for 5 min, checks the score. better? keep. worse? revert. repeat 100 times overnight while you sleep. that's literally it. what it's actually optimizing: the MODEL ARCHITECTURE. not predictions. not trades. not your portfolio. stuff like: → 4 layers or 8? → best learning rate? → AdamW or Muon optimizer? → what batch size works best on THIS specific GPU? optimal architecture depends on your hardware. an H100 wants a completely different model than your MacBook. autoresearch finds the best config for your machine automatically. what you CAN do with it: > build a tiny LLM that writes code, autoresearch finds the best architecture, you train on your dataset > create a lightweight chatbot that runs offline on your phone > train a model on your own writing so it sounds like you > test "does RoPE beat ALiBi for small models?" 100 variations in one night instead of 3 weeks of PhD work > optimize a model for a Raspberry Pi or edge device what you CANNOT do: > predict stock prices > find trading edges > analyze spreadsheets > predict sports outcomes autoresearch is a tool for people who want to BUILD language models, not USE them. Karpathy built an autonomous loop where AI improves AI. genuinely brilliant. but it solves a very specific problem. and that problem is probably not yours. which is fine, just stop pretending it's something it isn't.
- 08
@BrianRoemmele ·
WOW! The $8 AI Machine! Something extraordinary just happened and it changes what “local AI” can mean. I am testing it tonight. Thus far it shows many possibilities… So what it this $8 AI device? A developer going by slvDev has forced a 28.9-million-parameter language model onto an ESP32-S3 microcontroller that costs roughly eight dollars. Not a Raspberry Pi. Not a Jetson. An eight-dollar microcontroller. The model runs completely offline, generates coherent short stories at about 9.5 tokens per second, and draws power measured in the same range as a small LED. This is more than a hundred times larger than the previous record for the same class of chip (the earlier 260,000-parameter TinyStories experiments). For perspective, the original ChatGPT sat at 117 million parameters. We are now running a model roughly a quarter of that size on silicon you can buy for the price of two coffees. How the Impossible Became Possible The ESP32-S3 has only 512 KB of fast SRAM, 8 MB of PSRAM, and 16 MB of flash. Conventional wisdom said a model of this size simply would not fit. The breakthrough is architectural, not brute force. Most of a language model’s parameters live in a giant embedding table a lookup table you mostly read from, not compute against. Drawing directly from Google’s Per-Layer Embeddings technique (the same family of ideas used in the Gemma models), the developer moved the bulk of that table roughly 25 million parameters into flash memory and memory-mapped it. The chip only needs to pull about six rows, roughly 450 bytes, for each new token. The remaining dense “thinking” core stays in the fast SRAM (around 560 K of active working memory). The model is stored at 4-bit quantization and occupies about 14.9 MB total. The result is a system that feels almost free to run. The heavy parameters sit quietly in flash and are sampled sparingly. The little core does the real work. It is elegant engineering of the purest kind. What I Am Doing With It Right Now I have the boards on the bench in the garage lab. The first units are already talking short, coherent stories appearing on a tiny wired display, generated entirely on the chip with no Wi-Fi, no API key, no cloud round-trip. Latency is local. Privacy is absolute. Power draw is low enough that battery operation becomes interesting. I am treating these as the first generation of true $8 AI machines. Early tests are focused on three practical directions. - Embedding the model into simple nodes. - Pairing it with local voice front-ends - Exploring whether multiple of these chips can be networked as a lightweight swarm. The model is deliberately limited. It was trained on the Microsoft TinyStories dataset and is excellent at coherent narrative, not at open-ended question answering or tool use. That is a feature, not a bug. It forces us to design systems around what the silicon can actually deliver instead of pretending every edge device needs a frontier model. Real Use Cases That Suddenly Become Practical Once you accept that a capable language model can live for eight dollars and run without the cloud, a new class of devices becomes possible: This is the opposite of the current trajectory that wants every intelligent act to travel through a remote server. It is the beginning of intelligence that is cheap enough, private enough, and local enough to become infrastructure rather than a service. We have spent years watching model sizes explode upward. The more interesting frontier may be the opposite direction: how small, how cheap, and how local can useful intelligence become? An eight-dollar chip that can tell coherent stories is not a toy. It is a proof that the lower bound keeps moving. The open repository is at https://t.co/a7gcHTR4ug I will keep testing, measuring, and reporting what these little machines can and cannot do. The age of abundant local intelligence just got a little more real, and it arrived wearing an eight-dollar price tag.
- 09
@_vmlops ·
Everyone talks about LLMs like you need massive GPUs and billions of parameters You really don’t Came across this: https://t.co/j5z6o7vnpO It’s a tiny ~9M parameter model you can train in minutes (even on Colab) What I liked: ▫️It shows the entire pipeline - tokenizer → transformer → training → inference ▫️No APIs, no shortcuts ▫️Just clean, from-scratch implementation If you’ve been confused about how LLMs actually work, this is a great place to start Sometimes building small things teaches way more than using big ones
- 10
- 11
@burkov ·
In this paper, a 7B language model trained with reinforcement learning learns to orchestrate larger frontier models like GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro. It does so by writing natural-language subtasks, assigning each to one of the workers, and specifying which previous outputs that worker sees in context. The resulting system outperforms every individual frontier model on benchmarks including GPQA Diamond, LiveCodeBench, and AIME25, while averaging about three model calls per question—fewer than the multi-agent pipelines and self-reflection loops it beats. The work provides evidence that prompt engineering and pipeline design, currently done by hand in commercial AI products, can be learned end-to-end through reward signals alone. Read with an AI tutor: https://t.co/ihQP7Vj2Bv PDF: https://t.co/m3ddV3s5xc
- 12
@burkov ·
This paper argues that Small Language Models (SLMs) offer a more economical and suitable future for agentic AI by demonstrating their sufficient power for specialized tasks, outlining a conversion algorithm from LLMs to SLMs, and discussing the significant operational and economic impacts of this shift. ChapterPal: https://t.co/hmOIDRJfuH PDF: https://t.co/2TUNbfk3VJ
- 13
@hasantoxr ·
A 744 billion parameter AI model just ran on a machine with 25GB of RAM. No graphics card. The tool is called colibri. GLM-5.2 is a mixture-of-experts model. It contains 744 billion parameters but only about 40 billion wake up for each token. colibri keeps the 9.9GB core in memory. The remaining 370GB sits on an SSD. As the model writes, colibri pulls in only the tiny group of experts needed for that token. → Pure C → Zero runtime dependencies → 25GB RAM → Zero graphics cards → The full model stays on your machine The catch is brutal. A cold token can require 11GB of disk reads, so output crawls at roughly one token every 10 to 20 seconds. You will not replace ChatGPT with this tomorrow. But colibri cracks the assumption that the whole model must fit in memory before it can run. 100% open source. https://t.co/dSm86eRWCj
- 14
@akshay_pachaar ·
Vibe train your AI agents. There's a new method that could replace LLM-as-a-judge for production agents. Most teams rely on a giant LLM as a judge to evaluate and guard their agent. But it has two major drawbacks: - It's slow and expensive at inference time - It often misses domain-specific failures Vibe training flips this. Researchers at Plurai distill a small language model that's specialized for your agent's exact use case. The SLM becomes your evaluator and your runtime guardrail, both in one. The training data isn't hand-curated either. They spin up a swarm of adversarial agents that debate and stress-test every use case your agent is supposed to handle. That synthetic interaction data trains the specialized SLM. So the judge actually understands what "wrong" looks like in your specific domain. The reported gains vs. standard LLM-as-a-judge setups: - ~8x faster inference - ~50% fewer evaluation errors Smaller, faster, and more accurate because it's specialized for the job. The SLM-for-agents thesis is playing out in a very concrete way. If LLM-as-a-judge is your current evaluation layer, this is worth benchmarking against. Paper link in the replies.
- 15
- 16
@emollick ·
Gemma 4 E4B is impressive for an on-device LLM. GPT-4ish quality, and expect hallucinations. Here is: “List five sociological theories starting with u and what they are. Then describe them in a rhyming verse” Its in real time, the last is a little bit of a stretch, but not bad!
Watch video
- 17
@AIHighlight ·
🚨 Breaking: A free AI model trained for $7,800 just beat one 400 times its size at competition math. It runs small enough to fit on a laptop. Weibo's AI lab published the results and open-sourced the whole thing. The industry has run on one idea for two years. Bigger is better. More parameters, more compute, more money, smarter model. The biggest systems cost hundreds of millions to train, and the whole race is built on the belief that scale is what makes a model smart. A model called VibeThinker put a crack in that. It has 1.5 billion parameters. DeepSeek R1, the model that went viral last year, has 671 billion. That is more than 400 times larger. On three of the hardest competition math benchmarks, the tiny model scored higher. 80.3 to 79.8 on one. 74.4 to 70.0 on another. 50.4 to 41.7 on the third. It matched or beat Claude Opus 4 and GPT-4.1 on coding tests too. The cost is the part that should stop you. DeepSeek R1 cost around $294,000 to post-train. A model called MiniMax cost $535,000. VibeThinker cost $7,800. The team rented the GPUs at about $2 an hour and finished in 3,900 hours. It did not win by being bigger. It won by being trained more carefully. The method makes the model explore many ways to solve a problem first, then sharpens it toward the ones that work. This is not a one-off. MIT researchers found a small model beating the largest ones at strategic reasoning for 1% of the cost. NVIDIA's own researchers argue small specialized models should do most of the work in AI systems, at 10 to 30 times lower cost than the frontier giants. The industry says intelligence requires scale, and scale requires the kind of money only a few companies have. The quiet finding of 2026 is that much of what the giant models do can be done by small ones almost anyone can afford to train. The race everyone watches is between the giants. The race that might matter more is the one shrinking them down to something that runs on a laptop. Source: VibeThinker, Weibo AI, arxiv 2511.06221. MIT News. NVIDIA Research.
- 18
@LiorOnAI ·
Google's latest paper on Compression is the future. Here's why. They compressed LLM memory 6x with zero accuracy loss. When ChatGPT writes a reply, it remembers every word you've said. That memory is stored in a growing notebook (KV cache). A 100,000-word conversation can eat 16 GB of GPU memory. That's half of what most high-end GPUs even have. This is the #1 cost of running AI. Not the thinking. The remembering. TurboQuant shrinks each number in that notebook from 32 bits to just 3. That's like replacing a full paragraph with three words and losing nothing. No retraining. Works on any model instantly. Compressing numbers usually destroys their meaning. Here's how they solved it: 1. Rotate the numbers randomly so they all land on a predictable curve (PolarQuant) 2. Use one extra bit to fix the tiny errors left behind (QJL) Once numbers are predictable, you need far fewer bits to store them. The results: > 8x faster on Nvidia H100 GPUs > 16 GB notebook shrinks to under 3 GB > Search indexing drops from 500 seconds to 0.001 > Accuracy identical to the uncompressed model There's a proven math limit on how good compression can get. TurboQuant is only 2.7x above that floor. We're near the ceiling. Every company running LLMs spends most of its budget on memory. This cuts that cost by over 80%. The race is no longer about bigger models. It's about cheaper inference. Models that needed a $200K server cluster start fitting on a single $2K GPU. AI agents run 24/7 without burning budgets. The companies that win won't just have the best models. They'll have the best compression. Papers are open-access on arXiv, presented at ICLR.
- 19
- 20
@ModelScope2022 ·
Meet Marco-Mini-Instruct: a highly sparse MoE multilingual model from Alibaba International. 17.3B total params, only 0.86B active (5% activation ratio). 🚀 Beats Qwen3-4B, Gemma3-12B, Granite4-Small on English, multilingual general, and cultural benchmarks — with a fraction of their active params. 🌍 29 languages: Arabic, Turkish, Kazakh, Bengali, Nepali and more 🧠 256 experts, 8 active per token. Drop-Upcycling from Qwen3-0.6B-Base. 🎯 2-stage post-training: SFT + Online Policy Distillation (Qwen3-30B → Qwen3-Next-80B cascade) ✅ Apache 2.0 🤖 https://t.co/d6cBKume9H 🤖 https://t.co/i3cQorDzh9
- 21
@andrewchen ·
set up a mini rack for a home lab setup (will share a pic soon) w my Mac mini and DGX spark with more coming. had a few thoughts as I play w qwen3.5, gemma4, and other models: - there’s an S curve on LLM model quality per use case. Show text output side by side from the latest and you can’t tell the difference. I assume we’ll get to a flattish part of the curve on coding, multimodal, and other use cases over time - you seem to be able to swap the model underneath a great UX and the whole thing is portable. Openclaw workflows and personality are a bunch of markdown files and can run equally on GPT or Opus - SOTA models can be distilled and only stay in front of open weight models by ~12-18 months. Have to keep innovating to stay ahead (and god bless this dynamic from the startup ecosystem’s POV) - local AI models getting very good particularly on the latest Apple hardware. Very usable for many use cases and will only get better Obv still a big diff between what I can run locally and what’s available in the cloud - but the trend is super interesting and feels inevitable
- 22
@AlphaSignalAI ·
Researchers just gave LLMs a separate brain for memory. Language models go stale the moment training ends. Updating them risks breaking what they already know. A new paper proposes MeMo. It pairs any LLM with a separate trained memory model. The base model stays frozen. Knowledge gets internalized into a small dedicated model instead. The pipeline runs in three steps: > Extract facts from documents > Train memory on those facts > Query it through sub-questions When fresh data arrives, new memories merge in without retraining from scratch. This cuts compute by 33%. Retrieval cost stays constant regardless of corpus size. The frozen LLM treats memory as an external oracle. Across three benchmarks, it beats BM25, dense retrieval, and graph RAG. It plugs into closed proprietary models since everything runs through natural language. So what happens when memory stops being a context window hack?
- 23
@arpit_bhayani ·
When a public LLM benchmark says one model is better than another, it is testing broad, difficult reasoning and, more importantly, opinionated tasks. Your use case might not need that. For example, if you are summarizing tickets, classifying intent, or extracting fields from structured text, a smaller and cheaper model can match the output of a frontier model. General benchmarks tell us which model wins on average across a wide mix of tasks. They do not tell us which model wins on your specific, narrow, repeatable task. The only way to know for sure is to build your own benchmark. One simple way is to take real examples from your workload, run them across your candidate models, and score the outputs against what you actually need. It is laborious work, but yes, your favorite LLM can help you build that :) A model ranked lower on a general benchmark can be the better, cheaper choice for your use case, simply because your task was never complex enough to need the extra capability in the first place. In a gist, always try to answer: Which model is better for your task? Not which model is better in general. Hope this helps.
- 24
@TeksEdge ·
🚀 Future of LLM inference just got faster! Offloading the small draft model in speculative decoding to high-bandwidth SRAM accelerators (like d-Matrix @CORSAIR) while the big model stays on the GPU. 🎯 Result 2–10× lower end-to-end latency vs GPU-only speculative decoding and with better energy efficiency. Heterogeneous inference is the next big unlock. Link in ALT
- 25
@LiorOnAI ·
Most language models only read forward. Perplexity just open-sourced 4 models that read text in both directions. They used a technique from image generation to retrain Qwen3 so every word can see every other word in a passage. That changes how well a model understands meaning. They built four models from this: 1. Two sizes: 0.6B and 4B parameters 2. Two types: standard search embeddings and context-aware embeddings The context-aware version is the interesting one. It processes an entire document at once, so each small chunk "knows" what the full document is about. Standard embeddings treat each chunk in isolation. > Tops benchmarks for models of similar size > Works in multiple languages out of the box > MIT licensed, free for commercial use If you're building search over large document collections, you can now get document-level understanding without running a massive model. Small enough to actually deploy.
- 26
- 27
@RoundtableSpace ·
A 14B model on a single consumer GPU hitting 74.6% on LiveCodeBench. No fine-tuning. No API calls. No cloud. No data leaving your machine. The idea is simple wrap a frozen small model in smart infrastructure and it starts competing with frontier models at a fraction of the cost. Is the moat around big cloud AI smaller than everyone thought?
- 28
@thestreamingdev ·
3 ai models racing simultaneously on an m2 macbook air (8gb) built tiny bit, a local agent terminal for benchmarking small language models head-to-head on apple silicon. @PrismML @liquidai @Alibaba_Qwen tested bonsai-8b (1-bit, 1.16gb) vs qwen3-0.6b (q4, 0.37gb) vs lfm2.5-350m (q4, 0.25gb) Here’s how it went 🧵 https://t.co/cKxsmx8fNv
Watch video
- 29
@SimonHoiberg ·
Everyone is now looking at self-hosted AI models. Naturally. But if you think you can buy a Mac Mini and replace Claude or OpenAI, you need a serious wakeup call. Cause there's more to it than just the size of the weights - for a model to become practically useful, we need to consider: - Tool use - Context size - Quantization Cause for agents, this makes all the difference between a local model that can "just run" and a model that's actually useful. Let me explain 👇
Watch video
- 30
@DAIEvolutionHub ·
A 744 billion parameter AI model just ran on a machine with 25GB of RAM. No graphics card. The project is called colibri. It works with GLM-5.2, a 744B parameter mixture-of-experts model where only about 40B parameters are used for each generated token. Instead of forcing hundreds of gigabytes into memory, colibri keeps a 9.9GB runtime loaded and fetches the required experts directly from an SSD whenever they're needed. That means: • Pure C implementation • No runtime dependencies • Works with 25GB RAM • No GPU required • The entire model stays local The performance isn't pretty. When an expert isn't already cached, the system may need to read up to 11GB from disk before producing the next token. In practice, that's around one token every 10–20 seconds. So this isn't competing with ChatGPT. It's proving something different. The long-held assumption that a giant model has to fit entirely in RAM before it can run isn't always true. That's the real breakthrough. 100% open source. https://t.co/n3QonoDWbs
- 31
@Marktechpost ·
Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning - LFM2.5-350M is a 350M parameter small language model trained on 28 trillion tokens, with a hybrid architecture built from 10 double-gated LIV convolution blocks and 6 GQA blocks, plus 32K context support. -This model is built for instruction following, tool use, structured extraction, and edge deployment. Liquid AI team reports 76.96 on IFEval, 30.64 on GPQA Diamond, and 40.4K output tokens/sec on a single H100 at high concurrency. -The bigger point: small models are becoming serious infrastructure components for local and agentic workloads. Key Points: --- Best-in-class performance: A 350M model rivaling much larger models, bringing high-quality AI to your pocket. --- Fast edge inference: 313 tok/s decode on AMD CPU, 188 tok/s on Snapdragon Gen4. Runs under 1GB of memory with day-one support for llama.cpp, MLX, and vLLM. --- Scaled training: Extended pre-training from 10T to 28T tokens and large-scale multi-stage reinforcement learning. Full analysis: https://t.co/baJCEsDqFY Model weight: https://t.co/vHth0AdcF6 Docs: https://t.co/Qp35ZdnP3g @liquidai @maximelabonne
- 32
@aaliya_va ·
Stop downloading LLMs; your machine was never going to run. llmfit scans your hardware and tells you exactly which models will run. It scans your RAM, CPU, GPU, and VRAM first. Then it scores every model across four dimensions: 1. Quality, based on parameter count and quantization 2. Speed, estimating tokens per second for your exact backend 3. Fit, matching memory use to your hardware 4. Context window support for your use case Each model gets a label: Perfect, Good, Marginal, or Too Tight. It picks the best quantization automatically, stepping down until something fits. Covers hundreds of models from Meta, Mistral, Qwen, and DeepSeek. Works with Ollama, llama.cpp, MLX, and LM Studio out of the box. Open-source. Link in comments ↓
Watch video
- 33
@DivyanshT91162 ·
People said LLMs needed GPUs. This one runs on an $8 ESP32-S3 microcontroller. • 28.9M parameters • 14.9 MB 4-bit model • ~9.5 tokens/sec • 0 cloud • 0 internet • 100% on-device inference • 100% open source • MIT License The trick? Instead of loading the entire model into RAM, it stores ~25M parameters in flash memory and only fetches the tiny pieces needed for each token using Google's Per-Layer Embeddings. Memory layout: → SRAM: inference core → PSRAM: working memory → Flash: 25M-parameter embedding table Result: A language model roughly 100× larger than previous ESP32 implementations running on an $8 microcontroller. It's trained on TinyStories, so it's not replacing ChatGPT. But it proves something much bigger: The future of AI isn't just bigger models. It's smarter architectures that bring AI to tiny, low-power devices without sending a single byte to the cloud. 100% Open source. Repo👇
- 34
@smratitiwa86867 ·
🤯 Researchers just made LLMs over 99% sparse… without killing performance. And unlike most “sparse AI” papers, this one actually gets REAL GPU speedups instead of just theoretical FLOPs reductions. The trick? LLMs are already naturally sparse inside their feedforward layers. The problem is GPUs hate random unstructured sparsity because modern hardware is optimized for dense matrix math. So this paper redesigned the sparse format + GPU kernels together, making sparsity finally practical at scale. Results: • >99% sparsity with minimal quality loss • Faster inference • Lower VRAM usage • Less energy consumption • Lighter Transformer models without massive tradeoffs This is the kind of optimization that could make local LLMs way more efficient on consumer hardware. Paper name: “Sparser, Faster, Lighter Transformer Language Models” Link in comments 👇
- 35
@arsh_goyal ·
A 1 billion parameter AI model. Runs entirely on your laptop with no cloud or GPU bills or internet. MiniCPM5-1B just dropped and it's the best 1B local model right now. Here's what actually makes it worth trying: > Beats Qwen3.5 0.8B and LFM2.5 1.2B on math, coding, and tool calling > INT4 quantization needs just 0.5GB RAM which fits on a phone > I ran it on my machine with zero server calls and is Instant. The training framework for this is ForgeTrain which was written entirely by AI. No human programmers and it runs 10% faster than NVIDIA Megatron. An AI built the tool that trained this AI. We are officially in that timeline. If you're building on-device apps, this is the model to start with. Links in the comments below Full breakdown in the video 👇
Watch video
- 36
@ttunguz ·
Pocket Power : From State of the Art to Your Phone in 23 Months Two years ago, the idea of useful AI on your phone was fantastical. Siri couldn't finish a sentence. Local models hallucinated nonsense. Last week, Google released Gemma 4 E4B, a free model that matches GPT-4o & runs entirely on your phone. The next few weeks promise even more advanced pocket models. The market expects new releases from DeepSeek, Qwen, Kimi & Minimax. Frontier models don't stay frontier for long. Within three to four months, you can run a model with similar performance on your laptop; 23 months later, you can run the same model on your phone. Three forces are driving this compression. Better algorithms : distillation & reinforcement learning squeeze more capability into fewer parameters. Talent density : the biggest prizes in capitalism attract the best minds in the field. These are the fastest growing software companies in history. And capital : a trillion dollars invested in data centers powering training. In 23 months, the same capability that needed 1.8 trillion parameters now fits in 4 billion parameters. A 450x compression. At this rate, the phone in your pocket will run today's frontier models before you upgrade it. https://t.co/diAmGuPSuo
- 37
@andrewchen ·
Pepsi challenge for LLMs Contrarian view during a week of huge new model launches: All of us do a lot of “normie prompts” - these are use cases which are really like Google searches (“what’s the name of..” “is it true that…” “what’s the best…”). These are a very high % of total prompts- maybe not in terms of value creation (like code gen or the frontiers of math/science we’re going to) but it’s ubiquitous If you plugged these LLM prompts into the various frontier models could they tell the difference on the quality of output? I think not. We’d all fail in a blind taste test I think, as the models are now “good enough” we’re already at the point of diminishing returns in terms of what LLMs return back for a large % of use cases. And there’s implications: 1) open source models will constitute the majority of LLM queries. Open weight models lag by 18-24 months but adding to the question above, could you tell the difference on non-frontier local AI models that can run on modern Mac hardware? I’ve been doing exactly this with models like Qwen 27b dense and honestly they’re great for the normie prompts. There’s a huge incentive for NVIDIA, apple, and maybe even handset manufacturers like Samsung/etc to host open weight AI as an add on to just get you to buy their software 2) AI pricing heads to zero. And we’ll see free and ad-supported AI will be a thing in the consumer market, and open weight models are part of the story here too. Seems like we are <12-18 months to being able to just have ad supported AI particularly for developing markets and segments where the monthly fee doesn’t make sense. Monthly/metered might just be a thing in B2B use cases 3) once quality differences even out the competitive dimension shifts to other factors. Privacy, interconnectivity, free, bundling. The other idea here is that the moat becomes the wrapper (err we call them harnesses now? lol) and the product built around the LLM. 4) of course premium/frontier models will continue to exist. As long as there are big differences outside of the normie prompts, then you’ll hire one LLM over another for world generation, coding, science, labor replacement/augmentation etc. Just saying I’m not sure we’ll need frontier models for 90%+ of consumer use cases I think the prevalence of benchmarking in the launch of new AI models is in agreement with this. This week I tried Grok 4.5 and Fable for some coding experiments and you need to really spend time to pick up the differences. So we use benchmarks to point out what’s not so obvious Some of us will remember when computers were all measured in megahertz and megabytes, and the PC industry compared itself that way. Over time, that gave way to design, power efficiency, etc. Today we’re benchmarking and calculating cost per token and so on. It’s about to evolve, I think
- 38
@mark_k ·
On March 31, @PrismML unveiled Bonsai, a family of 1-bit ultra-dense language models that pack astonishing intelligence into tiny footprints. Named after the art of miniature trees, these models prove that true AI power thrives when compressed rather than expanded. The flagship Bonsai 8B features 8.2 billion parameters yet requires only 1.15 gigabytes of memory, making it fourteen times smaller, eight times faster at inference, and five times more energy efficient than full-precision peers. It runs over forty tokens per second on an iPhone 17 Pro, while the 4B and 1.7B variants shrink to 0.57 and 0.24 gigabytes and hit 130 tokens per second on recent Macs and iPhones. Built with pure 1-bit precision across every layer and no higher-bit shortcuts, Bonsai delivers crisp reasoning and strong instruction following. Available now under Apache 2.0 on Hugging Face, it promises fast, private, local AI for phones, robots, and edge devices, showing that the future of intelligence may simply be denser and more agile.
- 39
@alphabatcher ·
BEST local LLMs to run in 2026: High-performance (24+ GB VRAM, preferably with multiple GPUs) • Kimi K2 - 1T params, 32B active. MoE beast • GLM-4.7 (Z AI) - 30B-A3B MoE, SWE-bench 73.8% • DeepSeek V3.2 - 671B / 37B active. Still the open-source king • Qwen3 235B-A22B - insane quality/cost ratio if you have the iron Mid-range (16-24 GB VRAM / RAM) • Qwen3 30B-A3B - punches way above its weight, stable on long context • Gemma 3 27B - Google's best open release yet • Nemotron 3 Nano 30B - Math500: 91%. Best-in-class if you need math Lightweight models (8-16 GB RAM, can run without a dedicated GPU) • Qwen3 8B / 4B / 1.7B - the best small model family right now • Gemma 3 4B - surprisingly capable on CPU • Phi-4 (14B) - Microsoft doing a lot with a little The local AI stack is genuinely catching up to the cloud
- 40
- 41
@sabir_huss50540 ·
A 28.9 million parameter language model just ran on an $8 chip. No server. No wifi. No GPU. It runs on an ESP32-S3, the kind of chip you solder into a hobby project, writing each word to a tiny wired screen at about 9.5 tokens per second. The last model anyone ran on a chip this small had 260 thousand parameters. This one holds a hundred times more. Here is the trick. The chip has 512KB of fast memory, nowhere near enough. So most of the model never enters it. The 25 million row embedding table sits in slow flash, and each token pulls just 450 bytes from it. The idea is Google's Per-Layer Embeddings from Gemma, on a microcontroller for the first time. → 28.9M params, 14.9MB at 4-bit → $8 ESP32-S3, 512KB of fast memory → 9.5 tokens per second, fully on-chip → zero network, everything on the device The catch is real. It trained on TinyStories, so it writes short simple tales and nothing else. No questions, no code, no facts. The point is the architecture, not the output. MIT. He even left the bug in his own history, the one that inflated an early number, right next to the fix. 100% open source.
- 42
- 43
@cleanunicorn ·
Google is talking about TurboQuant and it's huge They're releasing this compression algorithm to the world, and it just might be what powers those 2M context window models. The Problem Every LLM uses a key-value cache to track conversation context. It's like a digital cheat sheet. But it's also a massive memory bottleneck. When you load an LLM, you can't use 100% of the memory, you need to save space for the cache. This is why most consumer hardware can only run smaller models. The Solution TurboQuant compresses this cache with zero accuracy loss. The Results - 32-bit → 3-bit compression (no accuracy loss) - 6x smaller key-value cache - 8x faster attention computation on H100 GPUs - Works with any existing model (no retraining) Why this matters - LLMs become cheaper to run - Massive context windows become practical - Vector search engines get a serious speed boost (semantic search, not keyword matching) The paper's already on arXiv and will be presented at ICLR 2026. I'll meet you in Rio in person to geek out on AI tech. Just ping me. Related research https://t.co/haRyjicHSC https://t.co/ubc0j6hsfA https://t.co/pWXVrbmPfb
- 44
- 45
- 46
- 47
- 48
@ainativedev ·
What's the smallest model that can actually run an AI agent? Not the cheapest. Not the fastest. The smallest one that reliably gets the job done. In their latest analysis, Nicolas Fortuin and Baptiste Fernandez (@FernandezBap ) put NVIDIA's (@nvidia) open-weight Nemotron models to the test across real coding-agent tasks. The results show that agent models don't degrade gradually as they get smaller. They hit the floor. The smallest model tested couldn't complete enough tasks to produce a meaningful score. Nano 30B could handle focused, well-defined work, but struggled once tasks became longer and more agentic. Super 120B was the first model that consistently cleared that capability threshold. One of the biggest takeaways is that skills aren't a substitute for capability. They amplify models that are already capable enough, but they can't rescue a model that hasn't crossed the agent floor. If you're choosing models based on parameter count or price alone, you might be optimizing the wrong thing. Read the full breakdown here: https://t.co/7Wgs1bQJLB
- 49
- 50
@JulianGoldieSEO ·
MINICPM5 JUST BROKE THE "BIGGER IS BETTER" MYTH A 1B parameter model is doing things that used to require models 10x its size. Here's what makes it different: Performance: → Just 1B parameters but supports reasoning, coding, and tool use → Runs locally on a laptop or even a phone → No expensive cloud GPUs required Smart Design: ✓ Hybrid thinking with two modes: Fast responses for simple tasks Deeper reasoning for harder problems ✓ 128K context window for long documents and conversations ✓ Structured outputs for tables, forms, and workflows Developer Friendly: ✔ Standard architecture that works with existing AI runtimes ✔ Fully open source ✔ Easy to deploy without custom engineering The trend is obvious. The next generation of AI won't just be bigger. It'll be smaller, faster, cheaper, and running on your own devices.
Watch video
Best Tweets by Topic