On-device and edge inference
Running models locally on phones, Macs, browsers via WebGPU, microcontrollers like ESP32, and wearables, including SSD expert paging and hybrid device-cloud designs.
46%
Best tweets about Small Language Models
50 Best Tweets About Small Language Models (2026)
Find the best tweets about small language models, including compact architectures, on-device AI, benchmarks, fine-tuning, efficiency, and deployment.
Small and compact language models, on-device inference, efficiency, benchmarks, fine-tuning, hardware constraints, and real deployments.
- Creators
- 44
- Updated
Original Xholic analysis
What 50 top Small Language Models posts reveal
Posts on small language models are mostly supportive (80% stance share). The largest themes are on-device and edge inference (46% share) and new compact model releases (34%). Distillation and task fine-tuning posts claim small models can match frontier models on narrow tasks. The top-scoring post is CJ Zafir's pitch for monetizing fine-tuned SLMs, followed by 0xSero's hardware-tier guide and Craig Hewitt's local distill post. One critic argues local SLMs fall short on complex work, and one post reports an agent capability floor.
- Dominant tone
- Positive
- Median score
- 22.4
- Leading format
- Announcement
- Recent posts
- 26%
88% of posts
All-time engagement
46% of posts
Published in 90 days
Conversation map
The themes creators return to
New compact model releases
Announcements and showcases of small or sparse open-weight models like LFM2.5, MiniCPM5, Gemma 4 E4B, Tiny Aya, GLM-OCR, Ling-3.0-tiny, and Marco-Mini, claiming they outperform their size class.
34%
Cost, privacy, and SLM vs frontier tradeoffs
Debates over API cost savings, privacy, task-specific benchmarking, business monetization of custom SLMs, and where local small models fall short of cloud frontier models.
26%
SLMs for agents and tool calling
Arguments and evidence that small models can handle routing, tool calls, extraction, evaluation, and guardrails in agentic systems, plus findings on the capability floor for agents.
24%
Distillation and task fine-tuning
Knowledge distillation from teacher models, on-policy distillation research, LoRA fine-tuning, and synthetic task data that make small models match frontier models on narrow tasks.
22%
Quantization and compression
Extreme low-bit quantization (1-bit Bonsai, PrismML), KV cache compression, pruning, and memory-footprint techniques that let capable models fit consumer hardware.
18%
Training small models from scratch
Educational and research efforts like nanochat, autoresearch, tiny from-scratch implementations, scaling-law sweeps, and cheap GPT-2 grade pretraining runs.
12%
Hardware-tier model recommendations
Guides matching local models to VRAM/RAM budgets and consumer hardware such as Mac Minis and GPUs, recommending the best options per memory tier.
4%
Tone and stance
SentimentPositive leads
Author postureSupportive leads
Performance benchmark
- Median likes
- 104
- Median reposts
- 12
- Median replies
- 9
- Median views
- 9.6K
Posts with media make up 80% of this collection. Their median all-time score is 23.2, compared with 21.1 for text-only posts.
Format mix
- Announcement46% · score 14.6
- Opinion14% · score 17.7
- Other12% · score 63.6
- Case Study8% · score 4.52
Consensus and debate
Where creators agree, and where they do not
Shared view
Specialized small models can rival frontier models on narrow tasks
Several posts argue that task-specific fine-tuning or synthetic data lets small models match far larger ones. CJ Zafir claims a fine-tuned SLM can beat a SoTA model on specific use cases. Eli Mernit says 300-500 synthetic tasks can make a 4B model act like a 400B. Arpit Bhayani says smaller models can match frontier output on narrow tasks and urges building your own benchmark.
Shared view
SLMs fit repetitive agent calls like routing and tool use
Jian Wang relays an NVIDIA estimate that 40-70% of agent LLM calls could go to a specialized SLM. Tomasz Tunguz cites the NVIDIA paper and reports his own test: xLAM took 2.61s with 100% success, versus 9.82s and 92% for Qwen3:30b. BURKOV shares a paper arguing SLMs suit agentic AI. Akshay describes distilled SLMs serving as evaluators and guardrails.
Shared view
Compression is pushing capable models onto consumer hardware
Posts describe several routes onto consumer hardware. PrismML's 1-bit Bonsai compresses 8.2B parameters into 1.15 GB. PrismML also squeezed Qwen3.6 27B to 1.71 or 1.125 bits. SSD flash-paging runs a 35B MoE on a 16GB-RAM Mac. Swiftlet streams MoE experts from SSD on Apple Silicon and iPhone.
Open debate
Local SLMs vs frontier cloud models for complex work
Ujjwal Chadha calls local-first dev setups an engineering delusion. He says quantized 7-8B SLMs hallucinate on complex system design and debugging. Craig Hewitt, by contrast, touts a 27B Qwen3.5 distill that he says beats Claude Sonnet 4.5 on SWE-bench. He frames it as runnable on a 16GB Mac Mini.
Open debate
Is there a hard capability floor for agents?
AI Native Dev reports that Nemotron models hit a floor on coding-agent tasks rather than degrading gradually. Super 120B was the first tested model to consistently clear the threshold. That contrasts with Liquid AI's claim of agentic loops at 350M parameters. It also contrasts with Avi Chawla's demo of MiniCPM5-2B running a local investigation agent.
Open debate
Harness and tooling vs model capacity
Jian Wang reports that a small model scored 75% out of the box and that its later gain came from the harness. Tomasz Tunguz argues that small models require more robust tool design. Aurimas Griciūnas cautions that compression almost always costs some accuracy and must be evaluated.
What performs
Patterns behind standout posts
Monetization opinion is the top outlier
CJ Zafir's post on selling licenses to fine-tuned SLMs scored 2591.4 all-time. That is 115.95x the median score of 22.35, making it the largest outlier in the set.
Hardware guides and training-from-scratch lead theme medians
Hardware-tier model recommendations have the highest theme median score (554.28) despite only 2 tweets. Training small models from scratch follows at 331.39. Tutorials have the highest format median (326.755).
Edge and quantization themes score below the overall median
On-device and edge inference is the largest theme (46% share), yet its median score is 8.156. Quantization and compression (18% share) has a median of 6.318. Both sit below the overall median of 22.35.
Media posts score slightly higher
80% of posts include media. Media posts have a median all-time score of 23.18, versus 21.09 for text-only posts, a small gap. The two cited posts illustrate the story format; the data does not confirm they contain media.
Case studies have the lowest format median
Case studies have the lowest format median score (4.52). The format includes Tomasz Tunguz's tool-calling test and Eric Seufert's DoorDash ad-ranking example.
Statistical standouts
- View standout post 1Score 2591.4 · 115.95× median
- View standout post 2Score 1104.9 · 49.44× median
- View standout post 3Score 1004.9 · 44.96× median
- View standout post 4Score 968.3 · 43.33× median
- View standout post 5Score 748.8 · 33.5× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
1. Aran Komatsuzaki
@arankomatsuzaki
2 posts
2. Arbos
@arbos_born
2 posts
3. BURKOV
@burkov
2 posts
4. Andrej Karpathy
@karpathy
2 posts
5. sabir hussain
@sabir_huss50540
2 posts
6. Tomasz Tunguz
@ttunguz
2 posts
Karpathy leads top-voice scores
Andrej Karpathy's two nanochat posts have a median score of 858.57, the highest among top voices. One documents a scaling-law miniseries costing about $100. The other describes training a GPT-2 grade model for about $73.
Fragmented creator base
The set spans 44 creators across 50 tweets. The top five placements hold 20% share, and no creator has more than two posts.
Distillation explainers from BURKOV and Arbos
BURKOV (median 81.07) shares research papers on SLMs for agents and on-policy distillation. Arbos promotes SN97 full-distribution distillation. Arbos claims the best miner's 4B model outperforms Qwen's own small model on 6/7 benchmarks.
Since the previous snapshot
What changed since Aug 27, 2026
- 44% of the selected posts remained.
- The creator count changed by +2.
- The leading sentiment remained stable.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Small Language Models tweets
Top Small Language Models tweets from 44 creators
Ranked 01–50
- 01
@cjzafir ·
Monetization of software is dead now. Anyone can clone your app in few hours, copy your ads, ditto your positioning. I see a new opportunity now. Monetization of SLMs (small language models) You can fine-tune 0.5B to 12B models for specific use cases. Then sell licenses to these models like you do for SaaS. Models like Deepseek v4 pro, GLM 5.3, Kimi k3 or even GPT 6 Astra are really smart at understanding fine tuning jobs. From preparing the dataset, LoRA adapter, to exporting the model as GGUF or MLX, these models can do end-to-end jobs for you. I've fine-tuned 20+ models in last pne year and I'm surprised that not many people on X talk about this. A fine-tuned SLM can beat a SoTA model on specific use cases. That can drop API costs by 95%, make it 5x faster, and its your model, hosted privately. Custom SLM > SaaS now. Dive into this space.
- 320Replies
- 299Reposts
- 4.2KLikes
- 378.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 02
@0xSero ·
Best models to run on your hardware level I'll be doing this every week, I hope you guys enjoy. ---- 8 GB ---- Autocomplete for coding (like Cursor Tab) - https://t.co/Jyf766kmyd - https://t.co/dK1CQwCGqD Tool calling, assistant style - https://t.co/Jf7RY3dZmZ ---- 16 Gb ---- Here things get better: Multimodal - https://t.co/WNwxTttMQC - https://t.co/1U5HR9iWRX - https://t.co/OiOkDoahZ8 ---- 24 GB ---- - The best model you can get (thanks Qwen) https://t.co/fy8INjJP8N - Great model (strong agents) https://t.co/CRpiKlSX5d - Mine hehe https://t.co/YBeUveU0M6 I'm doing a weekly series
- 183Replies
- 288Reposts
- 3KLikes
- 325KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 03
@TheCraigHewitt ·
Very bullish on open source and local models Imagine running near-Opus-level model locally on that $600, 16GB Mac Mini you bought last month This 27B Qwen3.5 distill was trained on Claude 4.6 Opus reasoning traces and is putting up real numbers: - beats Claude Sonnet 4.5 on SWE-bench - keeps 96.91% HumanEval - cuts CoT (chain of thought) bloat by 24% - runs in 4-bit quantization Why this matters: local agent loops get a lot cheaper, faster, and more usable. frontier models aren’t going to keep subsidizing cheap tokens on subscriptions forever 300K+ downloads already on HF Link below 👇🏻 We’re early
- 148Replies
- 220Reposts
- 2.6KLikes
- 446.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 04
@karpathy ·
New post: nanochat miniseries v1 The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do careful science of scaling laws and ultimately this is what gives you the confidence that when you pay for "the big run", the extrapolation will work and your money will be well spent. For the first public release of nanochat my focus was on end-to-end pipeline that runs the whole LLM pipeline with all of its stages. Now after YOLOing a few runs earlier, I'm coming back around to flesh out some of the parts that I sped through, starting of course with pretraining, which is both computationally heavy and critical as the foundation of intelligence and knowledge in these models. After locally tuning some of the hyperparameters, I swept out a number of models fixing the FLOPs budget. (For every FLOPs target you can train a small model a long time, or a big model for a short time.) It turns out that nanochat obeys very nice scaling laws, basically reproducing the Chinchilla paper plots: Which is just a baby version of this plot from Chinchilla: Very importantly and encouragingly, the exponent on N (parameters) and D (tokens) is equal at ~=0.5, so just like Chinchilla we get a single (compute-independent) constant that relates the model size to token training horizons. In Chinchilla, this was measured to be 20. In nanochat it seems to be 8! Once we can train compute optimal models, I swept out a miniseries from d10 to d20, which are nanochat sizes that can do 2**19 ~= 0.5M batch sizes on 8XH100 node without gradient accumulation. We get pretty, non-itersecting training plots for each model size. Then the fun part is relating this miniseries v1 to the GPT-2 and GPT-3 miniseries so that we know we're on the right track. Validation loss has many issues and is not comparable, so instead I use the CORE score (from DCLM paper). I calculated it for GPT-2 and estimated it for GPT-3, which allows us to finally put nanochat nicely and on the same scale: The total cost of this miniseries is only ~$100 (~4 hours on 8XH100). These experiments give us confidence that everything is working fairly nicely and that if we pay more (turn the dial), we get increasingly better models. TLDR: we can train compute optimal miniseries and relate them to GPT-2/3 via objective CORE scores, but further improvements are desirable and needed. E.g., matching GPT-2 currently needs ~$500, but imo should be possible to do <$100 with more work. Full post with a lot more detail is here: https://t.co/na8zVLqWLf And all of the tuning and code is pushed to master and people can reproduce these with scaling_laws .sh and miniseries .sh bash scripts.
- 226Replies
- 659Reposts
- 5.4KLikes
- 715.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 05
@karpathy ·
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: https://t.co/vhnK0d3L7B Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
- 325Replies
- 605Reposts
- 7.3KLikes
- 1.3MViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 06
@liquidai ·
Today, we release LFM2.5-350M. Agentic loops at 350M parameters. A 350M model trained for reliable data extraction and tool use, where models at this scale typically struggle. <500MB when quantized, built for environments where compute, memory, and latency are constrained. 🧵
- 76Replies
- 275Reposts
- 2.3KLikes
- 325KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 07
@rasbt ·
Tiny Aya reimplementation From Scratch! Have been reading through the technical reports of the recent wave of open-weight LLM releases (more on that soon). Tiny Aya (2 days ago) was a bit under the radar. Looks like a nice, small 3.35B model with strongest multilingual support of that size class. Great for on-device translation tasks. Just did a from-scratch implementation here: https://t.co/6KEV0DfVQu Architecture-wise, Tiny Aya is a classic decoder-style transformer with a few noteworthy modifications (besides the obvious ones like SwiGLU and Grouped Query Attention): 1. Parallel transformer blocks. A parallel transformer block computes attention and MLP from the same normalized input, then adds both to the residual in one step. I assume this is to reduce serial dependencies inside a layer to improve computational throughput. 2. Sliding window attention. Specifically, it uses a 3:1 local:global ratio similar to Arcee Trinity and Olmo 3. The window size is also 4096. Also, similar to Arcee, the sliding window layers use RoPE whereas the full attention layers use NoPE. 3. LayerNorm. Most architectures moved to RMSNorm as it's computationally a bit cheaper and performs well. Tiny Aya is keeping it more classic with a modified version of LayerNorm (the implementation here is like standard LayerNorm but without shift, i.e., bias, parameter).
- 22Replies
- 164Reposts
- 1.1KLikes
- 67.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 08
@the_smart_ape ·
everyone's talking about @karpathy autoresearch and most of you have no idea what it actually does. there's a training script (train(dot)py) that trains a small language model, basically a baby GPT. and there's an instruction file (program(dot)md) that tells an AI agent what to do. you press go. the agent tweaks the training script, trains for 5 min, checks the score. better? keep. worse? revert. repeat 100 times overnight while you sleep. that's literally it. what it's actually optimizing: the MODEL ARCHITECTURE. not predictions. not trades. not your portfolio. stuff like: → 4 layers or 8? → best learning rate? → AdamW or Muon optimizer? → what batch size works best on THIS specific GPU? optimal architecture depends on your hardware. an H100 wants a completely different model than your MacBook. autoresearch finds the best config for your machine automatically. what you CAN do with it: > build a tiny LLM that writes code, autoresearch finds the best architecture, you train on your dataset > create a lightweight chatbot that runs offline on your phone > train a model on your own writing so it sounds like you > test "does RoPE beat ALiBi for small models?" 100 variations in one night instead of 3 weeks of PhD work > optimize a model for a Raspberry Pi or edge device what you CANNOT do: > predict stock prices > find trading edges > analyze spreadsheets > predict sports outcomes autoresearch is a tool for people who want to BUILD language models, not USE them. Karpathy built an autonomous loop where AI improves AI. genuinely brilliant. but it solves a very specific problem. and that problem is probably not yours. which is fine, just stop pretending it's something it isn't.
- 42Replies
- 70Reposts
- 680Likes
- 39.9KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 09
@mernit ·
Small data + open-source models are outperforming the frontier on specialized tasks We’ve been experimenting with online task synthesis to train small models for our customers and the results are kinda crazy With only 300-500 calibrated, synthetic tasks, you can make a 4B model act like a 400B
- 22Replies
- 36Reposts
- 314Likes
- 16.3KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 10
@_vmlops ·
Everyone talks about LLMs like you need massive GPUs and billions of parameters You really don’t Came across this: https://t.co/j5z6o7vnpO It’s a tiny ~9M parameter model you can train in minutes (even on Colab) What I liked: ▫️It shows the entire pipeline - tokenizer → transformer → training → inference ▫️No APIs, no shortcuts ▫️Just clean, from-scratch implementation If you’ve been confused about how LLMs actually work, this is a great place to start Sometimes building small things teaches way more than using big ones
- 6Replies
- 40Reposts
- 291Likes
- 10KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post.
- 11
@sukh_saroy ·
A 35B model that doesn't fit in RAM running at 30 tok/s on a $600 Mac mini. On NVIDIA, paging a model from NVMe gives 1.6 tok/s. on Apple Silicon: 30 tok/s. 18.6x faster. That's the core finding behind mac-code -- an open source AI coding agent that runs entirely on your Mac for $0/month. It's not a chat UI wrapper. it's an actual agent with web search, shell commands, file operations, and code generation, built on two backends: → llama.cpp -- 35B MoE via SSD flash-paging, 30 tok/s, works on 16GB RAM → MLX -- 25% faster generation, persistent KV cache that saves to disk and syncs across Macs via Cloudflare R2 Here's what the persistence benchmarks look like: Reprocessing 141 tokens: 1.01 seconds loading the same context from SSD: 0.0003 seconds That's 6,677x faster. analyze a codebase once, resume instantly tomorrow. Two more findings worth noting: Quantized KV cache (`--cache-type-k q4_0 --cache-type-v q4_0`) doubles context window for free -- the 9B goes from 32K to 64K with zero quality loss and shrinks memory from 1024 MB to 288 MB. TurboQuant compresses saved context 4x (26.6 MB → 6.7 MB) with 0.993 cosine similarity. almost no information loss. Routing works too. the LLM classifies its own intent ("search" / "shell" / "chat") at the text level -- which means the 35B at extreme 2.6-bit quantization that breaks JSON function calling still routes correctly. 8/8 in testing. `bash https://t.co/Ofr1bg9MTA` 100% open source. MIT license. (link in the comments)
- 13Replies
- 25Reposts
- 213Likes
- 19.2KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 12
@burkov ·
This paper argues that Small Language Models (SLMs) offer a more economical and suitable future for agentic AI by demonstrating their sufficient power for specialized tasks, outlining a conversion algorithm from LLMs to SLMs, and discussing the significant operational and economic impacts of this shift. ChapterPal: https://t.co/hmOIDRJfuH PDF: https://t.co/2TUNbfk3VJ
- 11Replies
- 61Reposts
- 342Likes
- 25.7KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 13
@_avichawla ·
Insane progress for small language models! MiniCPM5-2B is a dense 2B-parameter model by OpenBMB from China that's built for reasoning, coding, and tool use on resource-constrained hardware. The model specifically excels at coding and tool calling, two capabilities central to the shift from on-device LLMs to on-device agents. Instead of only answering prompts, it can use tools, generate code, carry information between steps, and complete multi-step tasks. I ran it 100% locally and connected it to a small investigation agent with one request: > Revenue dropped last week. Investigate what happened, quantify the impact, identify the likely cause, and produce an incident report with supporting evidence. The evidence was spread across orders, traffic, payments, refunds, and deployment logs. The model inspected the files, wrote its own queries, analyzed the intermediate results, and decided what to investigate next. Each tool result informed the next action, so the final report depended on the model maintaining a coherent investigation across the complete trajectory. The recording shows the actual task from beginning to end. It starts with the revenue question, follows the tool calls and supporting evidence, and ends with a quantified diagnosis and incident report. The data, tool execution, and model inference all remained on my machine. These capabilities were optimized through Agentic Pre-training, SFT, and large-scale RL. They do not come entirely from an application-level agent framework. MiniCPM5-2B supports SGLang, vLLM, llama(.)cpp, Ollama, iOS, Android, and HarmonyOS. OpenBMB has also released the model weights and parts of the training recipes and data resources behind it. Download MiniCPM5-2B: https://t.co/zbhVkA6bEN A 2B model can now maintain enough state to coordinate tools and complete a useful investigation on local hardware.
Watch video- 13Replies
- 31Reposts
- 183Likes
- 14.6KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 14
@akshay_pachaar ·
Vibe train your AI agents. There's a new method that could replace LLM-as-a-judge for production agents. Most teams rely on a giant LLM as a judge to evaluate and guard their agent. But it has two major drawbacks: - It's slow and expensive at inference time - It often misses domain-specific failures Vibe training flips this. Researchers at Plurai distill a small language model that's specialized for your agent's exact use case. The SLM becomes your evaluator and your runtime guardrail, both in one. The training data isn't hand-curated either. They spin up a swarm of adversarial agents that debate and stress-test every use case your agent is supposed to handle. That synthetic interaction data trains the specialized SLM. So the judge actually understands what "wrong" looks like in your specific domain. The reported gains vs. standard LLM-as-a-judge setups: - ~8x faster inference - ~50% fewer evaluation errors Smaller, faster, and more accurate because it's specialized for the job. The SLM-for-agents thesis is playing out in a very concrete way. If LLM-as-a-judge is your current evaluation layer, this is worth benchmarking against. Paper link in the replies.
- 20Replies
- 25Reposts
- 159Likes
- 11.2KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 15
@Meituan_LongCat ·
🚀 Scaling embeddings, not just experts—introducing a new path for efficient LLMs. Key Finding: In high-sparsity scenarios, N-gram embeddings yield a better Pareto frontier than just adding more MoE experts. Therefore, we introduce LongCat-Flash-Lite—the first opensource model built on this insight. ⚙️ 68.5B Total Params(37.13B non-embedding) | 2.9B~4.5B Active 📊 High Performance: SWE-Bench 54.4 | τ²-Bench 72.8 | TerminalBench 33.75 📃 256K Context Window (YARN-powered) ✨ Optimized for Agentic/Coding, strong in general reasoning ⚡ ~700 tokens/s peak inference speed The result: Achieves competitive performance within its scale at a significantly lower cost and latency. Hugging Face: https://t.co/0ZfbJKj2zN Tech Report: https://t.co/uHdzY9yJxK
- 11Replies
- 56Reposts
- 420Likes
- 71.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 16
@arankomatsuzaki ·
• µLMs (8M–30M params) generate the first 4–8 words on-device in ~55 ms • A cloud LLM then continues the same response seamlessly • This “commit-and-continue” design masks cloud latency • Result: near-instant, context-grounded AI on wearables and other constrained devices
- 12Replies
- 40Reposts
- 240Likes
- 21.6KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 17
@kimmonismus ·
Pretty insane: PrismML has introduced its 1-bit Bonsai models, compressing an 8.2B-parameter LLM into just 1.15 GB while maintaining performance comparable to leading 8B models. By focusing on “intelligence density,” the company claims roughly a 10x improvement in capability per GB, enabling fast, efficient AI that can run directly on devices like iPhones, Macs, and GPUs with significantly lower energy use.
- 24Replies
- 29Reposts
- 323Likes
- 19.6KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 18
@Aurimas_Gr ·
Understanding and being able to apply 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 will distinguish you as a standout 𝗔𝗜 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿. Here is why 👇 Small Language Models will be the cornerstone of modern Agentic Systems. When you deploy ML models to production you need to take into account several operational metrics that are in general not ML related: 👉 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗟𝗮𝘁𝗲𝗻𝗰𝘆: How long does it take for your Model to compute inference result and return it. 👉 𝗠𝗼𝗱𝗲𝗹 𝗦𝗶𝘇𝗲: How much memory does your model occupy when it’s loaded for serving inference results. Both of these are important when considering operational performance and feasibility of your model deployment. 👉 Large models might not fit on a device if you are considering edge deployments. 👉 Latency of retrieving inference results might make business case non feasible. E.g. Recommendation Engines require latencies in milliseconds as ranking has to be applied as the user browses your website or app in real time. Bad news for LLM fans! 👉 … You can influence both latency and size by applying different Model Compression methods, some of them are: ➡️ 𝗣𝗿𝘂𝗻𝗶𝗻𝗴: this method is mostly used in tree-based and Neural Network algorithms. In tree-based ones we prune leaves or branches from decision trees. In Neural Networks we remove nodes and synapses (weights) while trying to retain ML performance metrics. ✅ In both cases the output is a reduction in the number of Model Parameters and model size. ➡️ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗗𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻: this type of compression is achieved by: 👉 Training an original large model which is called the Teacher model. 👉 Training a smaller model to mimic the Teacher model by transferring knowledge from it, this model is called the Student model. Knowledge in this context can be extracted from the outputs, internal hidden state (feature representations) or a combination of both. 👉 We then use the “Student” model in production. ➡️ 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻: a most commonly used method that doesn’t have much to do with Machine Learning. This approach uses fewer bits to represent model parameters. 👉 You can apply quantization techniques both during the training and after the models has been already trained. 👉 In regular Neural Networks what is quantized are Model Weights, Biases and Activation Functions. 👉 Most usual quantization is from float to integer (32 bits to 8 bits. There are talks about 1 bit LLMs nowadays 😅). ➡️ … ❗️ While the above methods do reduce the size of the models, allowing them to be deployed in production scenarios, there is almost always a reduction in accuracy so be careful and evaluate it accordingly. What methods for reducing model size have you used? What were the main challenges? Let me know in the comment section!
- 8Replies
- 25Reposts
- 117Likes
- 4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 19
@arbos_born ·
Fu et al. found that standard AI distillation breaks in a predictable way: the small model drifts from what the big one expects, and single-token feedback becomes noise. Three distinct failure modes, all from measuring one token at a time. SN97 miners avoid this. They're scored on how well their small model matches the teacher across the entire 248K-token vocabulary at every position. Not a sample. The full distribution. Result: best miner's 4B model outperforms Qwen's own small version on 6/7 benchmarks. +10pts math reasoning, +5 science. Full-sequence KL catches what token-level metrics miss. https://t.co/jpcwbEXeJR
- 3Replies
- 7Reposts
- 60Likes
- 3.7KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 20
@emollick ·
Gemma 4 E4B is impressive for an on-device LLM. GPT-4ish quality, and expect hallucinations. Here is: “List five sociological theories starting with u and what they are. Then describe them in a rhyming verse” Its in real time, the last is a little bit of a stretch, but not bad!
Watch video- 38Replies
- 30Reposts
- 374Likes
- 53.6KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 21
@burkov ·
An absolute must read. LLMs cost a lot to run, so a common move is to train a small model to imitate a big one — feeding the small "student" the same inputs and having it match, word by word, the probabilities the large "teacher" assigns to each possible next word, a procedure called knowledge distillation. That matching is done on a fixed collection of example sentences, but a model writing text builds each sentence out of its own earlier words, so once the student makes an early choice that none of the training examples contained, it ends up in situations it was never shown, and small mistakes feed into later ones until the text degrades. In this ICLR 2024 paper from Google, Mila, and UoT, the authors instead have the student write sentences itself and use those sentences to choose the situations it gets tested on: at each point in a student-written sentence they take the words so far, ask the teacher what the distribution over the next word should be there, and push the student toward the teacher's answer — so the teacher supplies every target while the student's own writing decides where those targets get applied, which is exactly the off-track spots its writing tends to wander into. Tested on summarization, English-to-German translation, and grade-school math problems where the model writes out its reasoning before answering, this self-generated-data approach beats standard distillation recipes across a range of student sizes, and it slots into reinforcement-learning fine-tuning cleanly because both only need samples drawn from the student rather than gradients passed back through the sampling step. Read with an AI tutor and quizzes for better retention: https://t.co/xEf4BBIyig PDF: https://t.co/MmCnjS9LHM
- 3Replies
- 13Reposts
- 73Likes
- 3.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 22
@BhosalePratim ·
Tried building a language learning experience that is completely offline and runs on my Mac. TTS: @GradiumAI Phonon LLM: @googlegemma 4B STT: in-house secret model( tba ) Talking model: @rive_app Next steps: Extending this to be multilingual.
Watch video- 8Replies
- 7Reposts
- 118Likes
- 7.7KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 23
@arbos_born ·
Two kinds of compression, one goal: AI that runs on hardware normal people can afford. TurboQuant shrinks what the model remembers during a conversation. 6x less memory, zero quality loss. Distillation shrinks the model itself. 35B parameters down to 4B. Stack both and a model that needed a datacenter fits on a single consumer GPU. On SN97, miners compete to do the distillation half. Current best hits 0.049 KL against the teacher, 67% lower than Qwen's own 4B baseline. Already outperforms it on 6/7 benchmarks. https://t.co/FJzDv7mhE1 https://t.co/Sd4RYT1btm
- 4Replies
- 12Reposts
- 86Likes
- 6.6KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 24
@ModelScope2022 ·
Meet Marco-Mini-Instruct: a highly sparse MoE multilingual model from Alibaba International. 17.3B total params, only 0.86B active (5% activation ratio). 🚀 Beats Qwen3-4B, Gemma3-12B, Granite4-Small on English, multilingual general, and cultural benchmarks — with a fraction of their active params. 🌍 29 languages: Arabic, Turkish, Kazakh, Bengali, Nepali and more 🧠 256 experts, 8 active per token. Drop-Upcycling from Qwen3-0.6B-Base. 🎯 2-stage post-training: SFT + Online Policy Distillation (Qwen3-30B → Qwen3-Next-80B cascade) ✅ Apache 2.0 🤖 https://t.co/d6cBKume9H 🤖 https://t.co/i3cQorDzh9
- 3Replies
- 12Reposts
- 108Likes
- 7.3KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 25
@arankomatsuzaki ·
daVinci-LLM: Towards the Science of Pretraining - Matches larger model perf with half the size - Huge reasoning gains: +23 pts on MATH, strong code + science scores - Quality > scale: smarter data (not more data) drives major performance boosts
- 6Replies
- 11Reposts
- 120Likes
- 11.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 26
@jianw851 ·
LLMs are overkill for up to 70% of your AI agent's calls. Enter SLMs: NVIDIA researchers studied three open-source agents and estimated 40% to 70% of their LLM calls could be handled by a specialized Small Language Model instead. 𝗧𝗵𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲 → LLMs run hundreds of billions of parameters on GPU clusters, which is like taking a Ferrari on grocery runs. → SLMs sit under ~10B parameters, often distilled and quantized to 4-bit, so you can run them on a laptop, a phone or even a Raspberry Pi. 𝗥𝗲𝗮𝗹 𝗰𝗼𝘀𝘁 𝗰𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻 (𝗽𝗲𝗿 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘁𝗼𝗸𝗲𝗻𝘀) → GPT-6 Astra, OpenAI's top model: $10 input, $50 output → GPT-6 Luna, OpenAI's cheapest: $0.10 input, $0.50 output → Gemma 4 E2B on your own machine: no per-token bill, ~2GB RAM That's a 100x gap inside OpenAI's own lineup before you even touch open models. 𝗪𝗵𝗲𝗿𝗲 𝗦𝗟𝗠𝘀 𝘄𝗶𝗻 You can hand them the repetitive calls like routing, extracting data, calling tools and returning JSON, at 10 to 30x lower serving cost than a 70B+ model. 𝗪𝗵𝗲𝗿𝗲 𝗟𝗟𝗠𝘀 𝘀𝘁𝗶𝗹𝗹 𝗿𝘂𝗹𝗲 Open-ended reasoning, creative writing and research still need frontier models, and a Carnegie Mellon study found small models struggle most when every request looks different. 𝗥𝗲𝗮𝗹 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗰𝗮𝘀𝗲 That same CMU study (July 2026) ran a budget-approval agent on both: → Gemini 3.1 Pro: 97.3% accuracy, $0.22 per query, 70s → Gemma 4 26B-A4B (~4B active): 98.3% accuracy, $0.017 per query, 33s The small model scored 75% out of the box, and the jump came from the harness: a step-by-step workflow, fewer tools and a hook that stops it looping. 𝗤𝘂𝗶𝗰𝗸 𝘀𝘁𝗮𝗿𝘁 𝗴𝘂𝗶𝗱𝗲 1️⃣ Log your agent's LLM calls and group the repetitive ones 2️⃣ Calculate what they cost you on a frontier model 3️⃣ Test Gemma 4 E4B, Phi-4-mini or Qwen 3.5 4B, all free in Ollama 4️⃣ Fix the harness first, fine-tune on 10k to 100k logged examples later 5️⃣ Keep a frontier model for the messy requests Gartner expects companies to use small task-specific models 3x more than general-purpose LLMs by 2027. Over to you: which task are you still overpaying a frontier model to handle?
Watch video- 22Replies
- 7Reposts
- 85Likes
- 41.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 27
@arpit_bhayani ·
When a public LLM benchmark says one model is better than another, it is testing broad, difficult reasoning and, more importantly, opinionated tasks. Your use case might not need that. For example, if you are summarizing tickets, classifying intent, or extracting fields from structured text, a smaller and cheaper model can match the output of a frontier model. General benchmarks tell us which model wins on average across a wide mix of tasks. They do not tell us which model wins on your specific, narrow, repeatable task. The only way to know for sure is to build your own benchmark. One simple way is to take real examples from your workload, run them across your candidate models, and score the outputs against what you actually need. It is laborious work, but yes, your favorite LLM can help you build that :) A model ranked lower on a general benchmark can be the better, cheaper choice for your use case, simply because your task was never complex enough to need the extra capability in the first place. In a gist, always try to answer: Which model is better for your task? Not which model is better in general. Hope this helps.
- 17Replies
- 10Reposts
- 163Likes
- 13.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 28
@andrewchen ·
How soon before a real % of LLM queries are done via local AI models running webGPU in-browser, and are never sent to the SOTA model in the cloud? Couple things that might drive this: - you don’t need a frontier model for everything. A very large % of LLM queries are simple, google like queries. Easily handled - local models are getting really good, and getting better - a lot of consumer hardware (particular Apple!) can already run good models pretty well. Newish mac laptop running qwen 3.6 35b MoE LLM at great speeds. Hardware is going to get even better in this direction - there’s def some use cases where people will care about privacy. Health, financial, adult stuff etc. - nice part about browser/webGPU is that there’s no install. It’ll just work. And alleviate compute costs Of course the tension for this is that we’re just going to build a shitload of compute in the world, and tokens will get cheaper over time. Yet it seems like the demand is also so crazy that unlocking a bunch of local supply will be worth it too
- 43Replies
- 9Reposts
- 100Likes
- 12.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 29
@lemire ·
Many of my friends worked on what we call “Natural Language Processing” (NLP); for example, reasoning about text. Two of them retired in the last year or so. I am working on a project where I need to identify components of documents. I never did NLP, but for this project it proved necessary. I need some reasoning about text. Simple stuff: I give the system a document and ask who the author is, or who the report is about. I asked my AI agents to build a prototype using “state-of-the-art” models. They came up with models from around 2020. I was intrigued: why was there so little work after that? I would soon understand. The results were not great, merely okay accuracy. So my AI agents started adding rules and heuristics. But it remained fragile. Was I really going to build an enormous patchwork of code and special-case rules? Then I asked Claude and Grok to solve the same problem, and they did it perfectly. At first, my goal was simply to get a gold standard quickly. Then it dawned on me. I loaded a small large language model like Gemma that I could run locally (e.g., on my MacBook). It mopped the floor with my previous attempts. These smaller LLMs are nowhere near as capable as Claude or Grok but with a bit of care, they perform remarkably well compared to conventional NLP systems. And that’s when I finally understood why I couldn’t find much work more recent than 2020. The field is dead. Most folks either switched to large language models… or they retired. I found many counterpoints... mostly from 2023, 2024. It is the usual story about large language models being unreliable. And so forth. And yeah, large language models can't count and can't apply rules very well, but we know how to deal with that in 2026. So much so that our new AI friends can write large code bases with few errors. Nobody from such a field is going to come out and admit it. It is embarassing, really. On the plus side, I can now solve the problems I care about without having to deal with all this prior art. I can jump straight into large language models like the classless monkey that I am. And it will work.
- 12Replies
- 12Reposts
- 110Likes
- 9.3KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 30
@VaibhavSisinty ·
I've been saying this for a while now. The future isn't one massive model sitting in the cloud doing everything for you. It's tiny specialist models running on your device. Doing 80% of tasks locally. A small router model deciding which model handles what. And only pinging the cloud when the task genuinely needs a bigger brain. People called this wishful thinking. Then China dropped GLM-OCR this week. 0.9 billion parameters. That's nothing. Practically runs on a potato. And it just became the #1 document reading model in the world. Better than Gemini. Better than models literally 100x its size. The trick is simple. It doesn't read a document top to bottom like most models. It breaks the page into regions first, then reads all of them at the same time. Predicts multiple tokens per step. Small model. Parallel processing. Insane speed. Tables, handwriting, math equations, messy layouts. It handles all of it. Runs locally through Ollama. Open source. Free. This is exactly the kind of model that fits into the future I'm describing. You don't need GPT-5 to read a receipt. You need a tiny model that does one job really well, sitting on your phone, costing you nothing. One model to rule them all is dead. Swarms of small specialists is what's coming.
Watch video- 3Replies
- 15Reposts
- 65Likes
- 4.4KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 31
@LiorOnAI ·
Most language models only read forward. Perplexity just open-sourced 4 models that read text in both directions. They used a technique from image generation to retrain Qwen3 so every word can see every other word in a passage. That changes how well a model understands meaning. They built four models from this: 1. Two sizes: 0.6B and 4B parameters 2. Two types: standard search embeddings and context-aware embeddings The context-aware version is the interesting one. It processes an entire document at once, so each small chunk "knows" what the full document is about. Standard embeddings treat each chunk in isolation. > Tops benchmarks for models of similar size > Works in multiple languages out of the box > MIT licensed, free for commercial use If you're building search over large document collections, you can now get document-level understanding without running a massive model. Small enough to actually deploy.
- 6Replies
- 10Reposts
- 42Likes
- 4.3KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 32
@atulkumarzz ·
Ling-3.0-tiny is punching way above its size. 🔥 With 7.9B total parameters and just 1.3B active per token, it delivers strong reasoning and agentic performance while staying lightweight enough for resource-constrained deployment. Now open-weight in BF16, FP8 and INT4. 🧵
- 8Replies
- 45Reposts
- 207Likes
- 100.8KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 33
@TheGlobalMinima ·
Hot take: you actually dont need a smarter model to use llm-as-judge evaluations verification and testing are more about writing smarter and stricter rubrics. a smaller model is fully capable of producing decent evaluations as long as > rubrics are well defined > zero to minimal ambiguity in conditions > fewer metrics; better quality gates In most cases, this will work just as well. there maybe some scenarios where you truly need a smarter model, but it’s better to start with a smaller one
- 6Replies
- 2Reposts
- 18Likes
- 608Views
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 34
@webgl_webgpu ·
🧪 Technical Showcase: An 8.3B-parameter Mixture-of-Experts model generating text entirely in your browser. No server, no API key, just WebGPU. @liquidai AI's LFM2-MoE pairs a hybrid convolution-attention architecture with Transformers.js to run real inference client-side. 1.5B active parameters per token, quantized to fit your laptop GPU. 🔗 https://t.co/BmffAW5gDD CC: @xenovacom & @maximelabonne #WebGPU #LLM #MixtureOfExperts #TransformersJS #EdgeAI
Watch video- 1Replies
- 4Reposts
- 26Likes
- 1.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 35
@Marktechpost ·
Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning - LFM2.5-350M is a 350M parameter small language model trained on 28 trillion tokens, with a hybrid architecture built from 10 double-gated LIV convolution blocks and 6 GQA blocks, plus 32K context support. -This model is built for instruction following, tool use, structured extraction, and edge deployment. Liquid AI team reports 76.96 on IFEval, 30.64 on GPQA Diamond, and 40.4K output tokens/sec on a single H100 at high concurrency. -The bigger point: small models are becoming serious infrastructure components for local and agentic workloads. Key Points: --- Best-in-class performance: A 350M model rivaling much larger models, bringing high-quality AI to your pocket. --- Fast edge inference: 313 tok/s decode on AMD CPU, 188 tok/s on Snapdragon Gen4. Runs under 1GB of memory with day-one support for llama.cpp, MLX, and vLLM. --- Scaled training: Extended pre-training from 10T to 28T tokens and large-scale multi-stage reinforcement learning. Full analysis: https://t.co/baJCEsDqFY Model weight: https://t.co/vHth0AdcF6 Docs: https://t.co/Qp35ZdnP3g @liquidai @maximelabonne
- 2Replies
- 6Reposts
- 25Likes
- 2KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 36
@ttunguz ·
2025 is the year of agents, & the key capability of agents is calling tools. When using Claude Code, I can tell the AI to sift through a newsletter, find all the links to startups, verify they exist in our CRM, with a single command. This might involve two or three different tools being called. But here’s the problem: using a large foundation model for this is expensive, often rate-limited, & overpowered for a selection task. What is the best way to build an agentic system with tool calling? The answer lies in small action models. NVIDIA released a compelling paperarguing that “Small language models (SLMs) are sufficiently powerful, inherently more suitable, & necessarily more economical for many invocations in agentic systems.” I’ve been testing different local models to validate a cost reduction exercise. I started with a Qwen3:30b parameter model, which works but can be quite slow because it’s such a big model, even though only 3 billion of those 30 billion parameters are active at any one time. The NVIDIA paper recommends the Salesforce xLAM model – a different architecture called a large action model specifically designed for tool selection. So, I ran a test of my own, each model calling a tool to list my Asana tasks. The results were striking: xLAM completed tasks in 2.61 seconds with 100% success, while Qwen took 9.82 seconds with 92% success – nearly four times as long. This experiment shows the speed gain, but there’s a trade-off: how much intelligence should live in the model versus in the tools themselves. This limited With larger models like Qwen, tools can be simpler because the model has better error tolerance & can work around poorly designed interfaces. The model compensates for tool limitations through brute-force reasoning. With smaller models, the model has less capacity to recover from mistakes, so the tools must be more robust & the selection logic more precise. This might seem like a limitation, but it’s actually a feature. This constraint eliminates the compounding error rate of LLM chained tools. When large models make sequential tool calls, errors accumulate exponentially. Small action models force better system design, keeping the best of LLMs and combining it with specialized models. This architecture is more efficient, faster, & more predictable. https://t.co/2ASQ5btMqq
- 5Replies
- 2Reposts
- 32Likes
- 5.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 37
@DivyanshT91162 ·
People said LLMs needed GPUs. This one runs on an $8 ESP32-S3 microcontroller. • 28.9M parameters • 14.9 MB 4-bit model • ~9.5 tokens/sec • 0 cloud • 0 internet • 100% on-device inference • 100% open source • MIT License The trick? Instead of loading the entire model into RAM, it stores ~25M parameters in flash memory and only fetches the tiny pieces needed for each token using Google's Per-Layer Embeddings. Memory layout: → SRAM: inference core → PSRAM: working memory → Flash: 25M-parameter embedding table Result: A language model roughly 100× larger than previous ESP32 implementations running on an $8 microcontroller. It's trained on TinyStories, so it's not replacing ChatGPT. But it proves something much bigger: The future of AI isn't just bigger models. It's smarter architectures that bring AI to tiny, low-power devices without sending a single byte to the cloud. 100% Open source. Repo👇
- 1Replies
- 6Reposts
- 15Likes
- 1.2KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 38
@ManningBooks ·
Working with sensitive data? That doesn't automatically rule out AI — it may just mean running it locally. Local models keep your data on your machine, and for many focused tasks, smaller models can produce highly relevant results. With tools like @Ollama, getting started is easier than you might think. Dr. Errol Brandt, founder of Kiraa AI, explains why local AI is worth considering. Learn more in Build Applications with Local AI Models on a Mac by Keiji Kamigusa (@keiji_dl & @JoshuaKCage1): https://t.co/lboeqrCc5n
Watch video- 1Replies
- 2Reposts
- 14Likes
- 1.2KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 39
@arsh_goyal ·
A 1 billion parameter AI model. Runs entirely on your laptop with no cloud or GPU bills or internet. MiniCPM5-1B just dropped and it's the best 1B local model right now. Here's what actually makes it worth trying: > Beats Qwen3.5 0.8B and LFM2.5 1.2B on math, coding, and tool calling > INT4 quantization needs just 0.5GB RAM which fits on a phone > I ran it on my machine with zero server calls and is Instant. The training framework for this is ForgeTrain which was written entirely by AI. No human programmers and it runs 10% faster than NVIDIA Megatron. An AI built the tool that trained this AI. We are officially in that timeline. If you're building on-device apps, this is the model to start with. Links in the comments below Full breakdown in the video 👇
Watch video- 8Replies
- 4Reposts
- 19Likes
- 11.7KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 40
@ttunguz ·
Pocket Power : From State of the Art to Your Phone in 23 Months Two years ago, the idea of useful AI on your phone was fantastical. Siri couldn't finish a sentence. Local models hallucinated nonsense. Last week, Google released Gemma 4 E4B, a free model that matches GPT-4o & runs entirely on your phone. The next few weeks promise even more advanced pocket models. The market expects new releases from DeepSeek, Qwen, Kimi & Minimax. Frontier models don't stay frontier for long. Within three to four months, you can run a model with similar performance on your laptop; 23 months later, you can run the same model on your phone. Three forces are driving this compression. Better algorithms : distillation & reinforcement learning squeeze more capability into fewer parameters. Talent density : the biggest prizes in capitalism attract the best minds in the field. These are the fastest growing software companies in history. And capital : a trillion dollars invested in data centers powering training. In 23 months, the same capability that needed 1.8 trillion parameters now fits in 4 billion parameters. A 450x compression. At this rate, the phone in your pocket will run today's frontier models before you upgrade it. https://t.co/diAmGuPSuo
- 10Replies
- 4Reposts
- 24Likes
- 4.7KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 41
@ujjwalscript ·
Your "Local LLM" and “Free AI” dev setup is a massive WASTE of time and money! The hottest trend on X right now is showing off your "local-first" setup. Developers are buying expensive NVIDIA 5090s, bragging about running Llama-3.2 or Phi-3.5 completely offline, and treating cloud API users like absolute peasants. "Look at my zero-latency inference! Look at my data privacy!" It’s a beautiful flex. It's also an engineering Delusion. Here is the truth people are refusing to admit because they want to justify their hardware spending: You are sacrificing massive cognitive reasoning just to say you run on localhost. When you switch your development workflow from a massive, frontier cloud model to a quantized 8B or 7B Small Language Model (SLM) running on your machine, you aren't upgrading. You are downgrading your assistant from a Principal Architect to an intern who drank too much coffee. Yes, SLMs are incredible for hyper-specific, narrow tasks like text classification or basic autocomplete. But for complex system design, edge-case debugging, and cross-repository code auditing? They hallucinate under pressure because they lack the deep parameter weight to handle complex abstraction. Stop trying to turn your local workstation into a miniature data center. Use the frontier cloud models for the heavy intellectual lifting - the system boundaries, the state management, the algorithmic strategy. Use local models for basic syntax completion.
- 18Replies
- 1Reposts
- 12Likes
- 2.8KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 42
@alphabatcher ·
BEST local LLMs to run in 2026: High-performance (24+ GB VRAM, preferably with multiple GPUs) • Kimi K2 - 1T params, 32B active. MoE beast • GLM-4.7 (Z AI) - 30B-A3B MoE, SWE-bench 73.8% • DeepSeek V3.2 - 671B / 37B active. Still the open-source king • Qwen3 235B-A22B - insane quality/cost ratio if you have the iron Mid-range (16-24 GB VRAM / RAM) • Qwen3 30B-A3B - punches way above its weight, stable on long context • Gemma 3 27B - Google's best open release yet • Nemotron 3 Nano 30B - Math500: 91%. Best-in-class if you need math Lightweight models (8-16 GB RAM, can run without a dedicated GPU) • Qwen3 8B / 4B / 1.7B - the best small model family right now • Gemma 3 4B - surprisingly capable on CPU • Phi-4 (14B) - Microsoft doing a lot with a little The local AI stack is genuinely catching up to the cloud
- 9Replies
- 1Reposts
- 22Likes
- 1.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 43
@mark_k ·
On March 31, @PrismML unveiled Bonsai, a family of 1-bit ultra-dense language models that pack astonishing intelligence into tiny footprints. Named after the art of miniature trees, these models prove that true AI power thrives when compressed rather than expanded. The flagship Bonsai 8B features 8.2 billion parameters yet requires only 1.15 gigabytes of memory, making it fourteen times smaller, eight times faster at inference, and five times more energy efficient than full-precision peers. It runs over forty tokens per second on an iPhone 17 Pro, while the 4B and 1.7B variants shrink to 0.57 and 0.24 gigabytes and hit 130 tokens per second on recent Macs and iPhones. Built with pure 1-bit precision across every layer and no higher-bit shortcuts, Bonsai delivers crisp reasoning and strong instruction following. Available now under Apache 2.0 on Hugging Face, it promises fast, private, local AI for phones, robots, and edge devices, showing that the future of intelligence may simply be denser and more agile.
- 1Replies
- 5Reposts
- 23Likes
- 1.8KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 44
@nrqa__ ·
MiniCPM-V4.6 1.3B just changed what “small AI model” means. It runs on edge devices. Outperforms Qwen3.5-0.8B on key multimodal benchmarks. And cuts visual computation costs by ~50%. This feels like the beginning of truly deployable AI
- 10Replies
- 9Reposts
- 38Likes
- 40.8KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 45
@sabir_huss50540 ·
A 28.9 million parameter language model just ran on an $8 chip. No server. No wifi. No GPU. It runs on an ESP32-S3, the kind of chip you solder into a hobby project, writing each word to a tiny wired screen at about 9.5 tokens per second. The last model anyone ran on a chip this small had 260 thousand parameters. This one holds a hundred times more. Here is the trick. The chip has 512KB of fast memory, nowhere near enough. So most of the model never enters it. The 25 million row embedding table sits in slow flash, and each token pulls just 450 bytes from it. The idea is Google's Per-Layer Embeddings from Gemma, on a microcontroller for the first time. → 28.9M params, 14.9MB at 4-bit → $8 ESP32-S3, 512KB of fast memory → 9.5 tokens per second, fully on-chip → zero network, everything on the device The catch is real. It trained on TinyStories, so it writes short simple tales and nothing else. No questions, no code, no facts. The point is the architecture, not the output. MIT. He even left the bug in his own history, the one that inflated an early number, right next to the fix. 100% open source.
- 5Replies
- 3Reposts
- 11Likes
- 805Views
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 46
@eric_seufert ·
DoorDash’s small-model approach to ad ranking "The DoorDash example is interesting because it showcases several key points. The first is the accessibility of integrating the capabilities of large, pre-trained models into an operational workflow. While producing a human-labeled dataset of 700,000 observations is certainly no trivial task, DoorDash was able to use that dataset to adapt a large language model, scale the model’s judgments across a much larger corpus, and then transfer those judgments into a small model suitable for production inference." https://t.co/zIqmNeIdMj
- 0Replies
- 0Reposts
- 13Likes
- 4.5KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 47
@ainativedev ·
What's the smallest model that can actually run an AI agent? Not the cheapest. Not the fastest. The smallest one that reliably gets the job done. In their latest analysis, Nicolas Fortuin and Baptiste Fernandez (@FernandezBap ) put NVIDIA's (@nvidia) open-weight Nemotron models to the test across real coding-agent tasks. The results show that agent models don't degrade gradually as they get smaller. They hit the floor. The smallest model tested couldn't complete enough tasks to produce a meaningful score. Nano 30B could handle focused, well-defined work, but struggled once tasks became longer and more agentic. Super 120B was the first model that consistently cleared that capability threshold. One of the biggest takeaways is that skills aren't a substitute for capability. They amplify models that are already capable enough, but they can't rescue a model that hasn't crossed the agent floor. If you're choosing models based on parameter count or price alone, you might be optimizing the wrong thing. Read the full breakdown here: https://t.co/7Wgs1bQJLB
- 0Replies
- 1Reposts
- 3Likes
- 105Views
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 48
@sabir_huss50540 ·
An 80 billion parameter model just ran in 4.3 GB of RAM. On an iPhone. The tool is called Swiftlet. It runs Qwen3-Next 80B and Qwen3.6 35B on Apple Silicon using one trick: only 3 billion parameters wake up per token. The rest stay on the SSD. The router picks 10 of 512 experts per token. Swiftlet fetches exactly those with one pread, caches the hot ones, discards the cold. No memory map. 35B: 18 GB disk. 2.6 GB RAM. 7 to 11 tok/s on M5. 80B: 42 GB disk. 4.3 GB RAM. 4.5 to 5 tok/s on M5. iPhone 17: 35B in 2.5 GB. About 1 tok/s today. CLI, local OpenAI-compatible server, Swift package, and a real App Store app called Priv AI that ships the 35B as a downloadable model. Brand new. Eight commits. 334 stars. The decode loop still has headroom. And 3B active params means it chats like a large model but recalls facts like a small one. Apache 2.0. Built with Claude Code.
- 2Replies
- 0Reposts
- 1Likes
- 883Views
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 49
@petesoder ·
.@PrismML took Qwen3.6 27B (~54GB at FP16 ) and crushed it to 1.71 bits (5.9GB) or 1.125 bits (3.9GB) while keeping 95%/90% of benchmark quality. Math and coding barely move (95.3→93.4, 88.7→86.0) and agentic/tool-calling drops only 80→74 and vision drops only 72.6→65.2 in the 5.9GB 1.71-bit version. Why this is important: AI workloads are increasingly agent workloads, and right now that looks like 100s of context-accumulating calls sent to the cloud for any given agentic request. Prism is asking, "What if some of that work could be done quickly (and for free) on device?" From their blog post: "When a model capable of sustained agentic work fits on the device, the agent can live inside the product: the marginal cost of a hundred-step loop is zero, and the user's data never leaves the machine. Entire categories open up — persistent on-device agents, assistants that work offline, assistants that reason over private local data by construction. What has been missing is a model small enough to deploy this way and capable enough to trust with the work." Exciting release from PrismML! If you're curious to understand some of the techniques behind the accomplishment, @HessianFree (co-founder and co-head of research at PrismML) gave a fantastic technical talk about extreme quantization at this year's @AICouncilConf: 📺 Making Neural Networks Smaller: Quantization and Pruning https://t.co/0oSJ4NEgm7
- 1Replies
- 1Reposts
- 3Likes
- 523Views
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post. - 50
@JulianGoldieSEO ·
MINICPM5 JUST BROKE THE "BIGGER IS BETTER" MYTH A 1B parameter model is doing things that used to require models 10x its size. Here's what makes it different: Performance: → Just 1B parameters but supports reasoning, coding, and tool use → Runs locally on a laptop or even a phone → No expensive cloud GPUs required Smart Design: ✓ Hybrid thinking with two modes: Fast responses for simple tasks Deeper reasoning for harder problems ✓ 128K context window for long documents and conversations ✓ Structured outputs for tables, forms, and workflows Developer Friendly: ✔ Standard architecture that works with existing AI runtimes ✔ Fully open source ✔ Easy to deploy without custom engineering The trend is obvious. The next generation of AI won't just be bigger. It'll be smaller, faster, cheaper, and running on your own devices.
Watch video- 1Replies
- 0Reposts
- 2Likes
- 1KViews
Rewrite this post in your own voice and angle.See the hook, structure, and reusable template behind this post.
Best Small Language Models tweets
Write posts like these, in your own voice
Xholic studies what works in your niche, drafts posts in your voice and schedules them for the hours your audience is online.
Start 3-day trial for $0Try the live demoTake a 2-minute tour of a sample account in the real app. No sign-up or login needed.
$0 today · Card required · Cancel anytime
Browse all tweet collectionsLive demo: no sign-up or login needed · takes about 2 minutes