On-device and local inference
Running models privately on laptops, phones, Apple devices, consumer GPUs, and self-hosted setups to reduce cloud cost, latency, and data exposure.
56%
Best tweets about Small Language Models
50 Best Tweets About Small Language Models (2026)
Find the best tweets about small language models, including compact architectures, on-device AI, benchmarks, fine-tuning, efficiency, and deployment.
Small and compact language models, on-device inference, efficiency, benchmarks, fine-tuning, hardware constraints, and real deployments.
- Creators
- 42
- Updated
Original Xholic analysis
What 50 top Small Language Models posts reveal
The conversation emphasizes local and specialized small language models. Posts frame distillation, compression, and hardware-aware deployment as ways to pursue lower-cost, private inference, while arguing that model choice should be evaluated on the intended workload. [2014192454258274743, 2043483564348731461, 2074697730874823077]
- Dominant tone
- Positive
- Median score
- 25.9
- Leading format
- Announcement
- Recent posts
- 38%
82% of posts
All-time engagement
84% of posts
Published in 90 days
Conversation map
The themes creators return to
Task-specific benchmarks and capability thresholds
Evaluating compact models against real workloads, agent reliability floors, coding and reasoning benchmarks, and use-case-specific quality rather than broad leaderboard rank alone.
32%
Compression, quantization, and sparsity
Reducing model and KV-cache memory, compute, and energy use with 1-bit or ternary weights, low-bit quantization, sparsity, cache compression, and related kernels.
28%
Small models for agents and tool use
Using SLMs and action models for tool calling, structured extraction, agent guardrails, evaluators, and narrow repeatable workflows where speed and predictability matter.
24%
Efficient model architectures
Architecture-level approaches for compact capability, including sparse MoE, hybrid convolution-attention designs, state-space models, continuous token representations, and embedding layouts.
20%
Specialized fine-tuning and distillation
Turning small base models into reliable domain specialists through synthetic data, teacher-student distillation, LoRA/SFT/DPO, preference training, and task-specific datasets.
18%
Efficient training and scaling laws
Lowering the cost of pretraining and training small models through compute-optimal scaling, optimized kernels and optimizers, automated architecture search, and consumer-hardware training.
16%
Extreme edge and microcontroller AI
Deploying highly constrained language models on low-power hardware such as ESP32 chips, wearables, and embedded devices through memory-aware architectures.
6%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 91
- Median reposts
- 13
- Median replies
- 11
- Median views
- 9.8K
Posts with media make up 82% of this collection. Their median all-time score is 38.0, compared with 2.78 for text-only posts.
Format mix
- Announcement 84% · score 25.9
- Tutorial 14% · score 25.2
- List 2% · score 38.0
Consensus and debate
Where creators agree, and where they do not
Shared view
Local deployment is a prominent use case
Posts position compact models for private, offline, or self-hosted inference across laptops, Apple devices, mobile apps, and constrained hardware. [2043483564348731461, 2064416235958509930, 2039689619357261900, 2081455240734466392]
Shared view
Specialization and workload-specific evaluation
Several posts argue that small models can be useful when distilled or fine-tuned for narrow workflows, and that candidate models should be evaluated against the specific workload rather than broad leaderboards alone. [2014192454258274743, 2074260227998818483, 2074697730874823077]
Shared view
Efficiency spans model and systems techniques
Posts discuss compression, ternary weights, sparse activation, and KV-cache reduction as approaches intended to reduce memory, compute, or energy demands alongside reducing model size. [2043958661744336897, 2037907703188619605, 2042084482661191942]
Open debate
Local replacement claims come with deployment constraints
One guide says modest hardware may replace smaller workflows, while another argues that useful self-hosted agents must also account for tool use, context size, and quantization. [2043483564348731461, 2067578268350386344]
Open debate
Small agents may face a capability floor
A reported tool-calling experiment found a specialized action model faster and more successful than a larger alternative on one Asana-task test. Separately, an agent benchmark analysis argues that models can fall below a usable capability threshold as they shrink. [1952469728410390593, 2070509559249310029]
Open debate
Extreme-edge demonstrations are intentionally narrow
ESP32 posts describe fully local generation under severe memory limits, but identify the TinyStories-trained model as suitable for short, simple stories rather than questions, code, facts, or tool use. [2080501012935434357, 2082708239456850263]
What performs
Patterns behind standout posts
Media posts had a higher median score than text-only posts
Media appeared in 41 of 50 posts (82%). The media median all-time score was 37.978, compared with 2.776 for text-only posts. The five engagement outliers covered local deployment, distillation, on-device tinkering, or efficient training. [2014192454258274743, 2043483564348731461, 2028961822372425941, 2009037707918626874, 2017703360393318587]
Training efficiency had the highest theme median
Efficient training and scaling laws had the highest theme median all-time score at 170.94. Karpathy’s posts on compute-optimal model series and lower-cost GPT-2-grade training were both engagement outliers. [2009037707918626874, 2017703360393318587]
Announcements accounted for most posts
Announcements accounted for 42 posts (84%) and had a 25.87 median all-time score. Tutorials accounted for seven posts (14%) and had a similar median score of 25.215. [2043483564348731461, 2041463021567107373, 2043958661744336897]
Statistical standouts
- View standout post 1 Score 1839.2 · 71.09× median
- View standout post 2 Score 1707.2 · 65.99× median
- View standout post 3 Score 1582.8 · 61.18× median
- View standout post 4 Score 1056.2 · 40.83× median
- View standout post 5 Score 818.7 · 31.65× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Akshay 🚀
@akshay_pachaar
2 posts
-
2. BURKOV
@burkov
2 posts
-
3. Andrej Karpathy
@karpathy
2 posts
-
4. Lior Alexander
@LiorOnAI
2 posts
-
5. 0xMarioNawfal
@RoundtableSpace
2 posts
-
6. Shruti Codes
@Shruti_0810
2 posts
Prominent posts paired compact-model claims with implementation details
Posts with high engagement or from top voices described concrete mechanisms, including distillation pipelines, scaling-law experiments, ternary inference, and specialized agent guardrails. [2014192454258274743, 2009037707918626874, 2017703360393318587, 2064416235958509930, 2049158769838592416]
Deployment posts also document limitations
Some posts explicitly qualify local-model benefits: modest hardware may replace only smaller workflows; ESP32 models are narrow TinyStories systems; and a locally useful agent requires more than model weights alone. [2043483564348731461, 2081455240734466392, 2067578268350386344]
Since the previous snapshot
What changed since Aug 12, 2026
- 80% of the selected posts remained.
- The creator count changed by -1.
- The leading sentiment remained stable.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Small Language Models tweets
Top Small Language Models tweets from 42 creators
Ranked 01–50
- 01
@TheAhmadOsman ·
INCREDIBLE Someone on r/LocalLLaMA did an incredibly practical thing They took a tiny 0.6B model that was trash at task (Text2SQL) Created a knowledge distiliation agent with a Claude Code skill And made the 0.6B model behave like a specialist using 100 examples The problem > Small Language Models are “generally helpful” > but specialized tasks are “exact or you die” > you ask: “Which artists have >1M album sales?” > the model answers: “check if genre is NULL” The old way to fix this > Finetune the model: > collect + clean data > build training pipeline > tune hparams > rerun when it’s wrong > accidentally become the unpaid > intern of your own experiment The new way > Knowledge distillation via a Claude skill > use a strong teacher (DeepSeek-V3) > generate synthetic pairs from a small seed set > train a tiny student to imitate the teacher on your task > ship it as GGUF / HF / LoRA > run it locally Distillation isn’t “creating skill” It’s compressing skill THE REAL HACK: agent-as-interface > They wrapped the whole distillation loop in an agent “skill”: > picks task type (QA / classification / tool calling / RAG) > converts messy inputs into clean JSONL > runs teacher eval first > kicks off distillation + monitors progress > packages weights for you to run locally This is the quiet unlock Why “teacher eval first” is elite behavior > distillation amplifies competence and incompetence > if the teacher is wrong, the student learns wrong faster > garbage in -> efficient garbage out Adult supervision, but for models The run breakdown: > seed: ~100 raw conversation traces > teacher (LLM-as-judge): ~80% > base 0.6B: ~36% > distilled 0.6B: ~74% > output: ~2.2GB GGUF > runs locally with llama.cpp Before vs after (the entire reason you do this) > before: wrong tables, wrong logic, nonsense SQL > after: correct JOINs, GROUP BY, HAVING > aka “this query actually executes and answers the question” What this really means (bigger than Text2SQL) You don’t need a giant model for every job You need tiny specialists that understand your world: > internal schemas > service / OS logs > tool outputs > company-specific workflows TL;DR > “fine-tuning is hard” is mostly “the pipeline is annoying” > distillation skill turns 10–100 examples into a real specialist > the agent wrapper turns the whole thing into a conversation > this is how you get practical local SLMs > without becoming an MLOps monk Small & Specialized models > High-leverage > Boringly effective > Exactly where this is going The future is Local inference Lower latency Fewer secrets leaving the building
- 02
@AlexFinn ·
I don't care what kind of hardware you have, you should be running local models It will save you a ton on money on OpenClaw and keep your data private Even if you're on the cheapest Mac Mini you can be doing this Here's a complete guide: 1. Download LMStudio 2. Go to your OpenClaw/Hermes and say what kind of hardware you have (computer and memory and storage) 3. Ask what's the best local model you can run on there (probably will be Gemma 4 or Qwen. if you have a big computer, it will be GLM) 4. Ask 'based on what you know about me, what workflows could this open model replace?' 5. Have OpenClaw walk you through downloading the model in LM Studio and setting up the API 6. Ask OpenClaw to start using the new API Boom you're good to go. You just saved money by using local models, have an AI model that is COMPLETELY private and secure on your own device, did something advanced that 99% of people have never done, and have entered the future. If you are on smaller hardware you probably are not going to replace all your AI calls with this, but you could replace smaller workflows which will still save you good money Own your intelligence.
- 03
- 04
@karpathy ·
New post: nanochat miniseries v1 The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do careful science of scaling laws and ultimately this is what gives you the confidence that when you pay for "the big run", the extrapolation will work and your money will be well spent. For the first public release of nanochat my focus was on end-to-end pipeline that runs the whole LLM pipeline with all of its stages. Now after YOLOing a few runs earlier, I'm coming back around to flesh out some of the parts that I sped through, starting of course with pretraining, which is both computationally heavy and critical as the foundation of intelligence and knowledge in these models. After locally tuning some of the hyperparameters, I swept out a number of models fixing the FLOPs budget. (For every FLOPs target you can train a small model a long time, or a big model for a short time.) It turns out that nanochat obeys very nice scaling laws, basically reproducing the Chinchilla paper plots: Which is just a baby version of this plot from Chinchilla: Very importantly and encouragingly, the exponent on N (parameters) and D (tokens) is equal at ~=0.5, so just like Chinchilla we get a single (compute-independent) constant that relates the model size to token training horizons. In Chinchilla, this was measured to be 20. In nanochat it seems to be 8! Once we can train compute optimal models, I swept out a miniseries from d10 to d20, which are nanochat sizes that can do 2**19 ~= 0.5M batch sizes on 8XH100 node without gradient accumulation. We get pretty, non-itersecting training plots for each model size. Then the fun part is relating this miniseries v1 to the GPT-2 and GPT-3 miniseries so that we know we're on the right track. Validation loss has many issues and is not comparable, so instead I use the CORE score (from DCLM paper). I calculated it for GPT-2 and estimated it for GPT-3, which allows us to finally put nanochat nicely and on the same scale: The total cost of this miniseries is only ~$100 (~4 hours on 8XH100). These experiments give us confidence that everything is working fairly nicely and that if we pay more (turn the dial), we get increasingly better models. TLDR: we can train compute optimal miniseries and relate them to GPT-2/3 via objective CORE scores, but further improvements are desirable and needed. E.g., matching GPT-2 currently needs ~$500, but imo should be possible to do <$100 with more work. Full post with a lot more detail is here: https://t.co/na8zVLqWLf And all of the tuning and code is pushed to master and people can reproduce these with scaling_laws .sh and miniseries .sh bash scripts.
- 05
@karpathy ·
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: https://t.co/vhnK0d3L7B Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
- 06
@akshay_pachaar ·
Apple finally did it. Its new framework, Core AI, runs models entirely on Apple silicon, so inference happens on the user's device with zero server calls and zero token bills. That means Qwen, Mistral, and SAM3 running natively across iPhone, iPad, Mac, and Vision Pro. It's a memory-safe Swift API that compiles models ahead of time for near-instant load. Pulling one in takes a few lines: let segmenter = try await ImageSegmenter(resourcesAt: sam3ModelURL) let response = try await segmenter.segment(image: inputImage, prompt: "flower") The launch goes beyond the runtime, though. It ships curated open models packaged for Swift, PyTorch extensions to convert your own, and an optimizer that shrinks models layer by layer with minimal accuracy loss. There's also a macOS debugger that profiles performance and traces behavior back to your original Python, plus Xcode tools to validate models before they ship. For any team that wanted real on-device AI without a cloud bill attached to every user, this is the answer. Models repo: https://t.co/yZCIhrM6YQ
- 07
@the_smart_ape ·
everyone's talking about @karpathy autoresearch and most of you have no idea what it actually does. there's a training script (train(dot)py) that trains a small language model, basically a baby GPT. and there's an instruction file (program(dot)md) that tells an AI agent what to do. you press go. the agent tweaks the training script, trains for 5 min, checks the score. better? keep. worse? revert. repeat 100 times overnight while you sleep. that's literally it. what it's actually optimizing: the MODEL ARCHITECTURE. not predictions. not trades. not your portfolio. stuff like: → 4 layers or 8? → best learning rate? → AdamW or Muon optimizer? → what batch size works best on THIS specific GPU? optimal architecture depends on your hardware. an H100 wants a completely different model than your MacBook. autoresearch finds the best config for your machine automatically. what you CAN do with it: > build a tiny LLM that writes code, autoresearch finds the best architecture, you train on your dataset > create a lightweight chatbot that runs offline on your phone > train a model on your own writing so it sounds like you > test "does RoPE beat ALiBi for small models?" 100 variations in one night instead of 3 weeks of PhD work > optimize a model for a Raspberry Pi or edge device what you CANNOT do: > predict stock prices > find trading edges > analyze spreadsheets > predict sports outcomes autoresearch is a tool for people who want to BUILD language models, not USE them. Karpathy built an autonomous loop where AI improves AI. genuinely brilliant. but it solves a very specific problem. and that problem is probably not yours. which is fine, just stop pretending it's something it isn't.
- 08
@BrianRoemmele ·
WOW! The $8 AI Machine! Something extraordinary just happened and it changes what “local AI” can mean. I am testing it tonight. Thus far it shows many possibilities… So what it this $8 AI device? A developer going by slvDev has forced a 28.9-million-parameter language model onto an ESP32-S3 microcontroller that costs roughly eight dollars. Not a Raspberry Pi. Not a Jetson. An eight-dollar microcontroller. The model runs completely offline, generates coherent short stories at about 9.5 tokens per second, and draws power measured in the same range as a small LED. This is more than a hundred times larger than the previous record for the same class of chip (the earlier 260,000-parameter TinyStories experiments). For perspective, the original ChatGPT sat at 117 million parameters. We are now running a model roughly a quarter of that size on silicon you can buy for the price of two coffees. How the Impossible Became Possible The ESP32-S3 has only 512 KB of fast SRAM, 8 MB of PSRAM, and 16 MB of flash. Conventional wisdom said a model of this size simply would not fit. The breakthrough is architectural, not brute force. Most of a language model’s parameters live in a giant embedding table a lookup table you mostly read from, not compute against. Drawing directly from Google’s Per-Layer Embeddings technique (the same family of ideas used in the Gemma models), the developer moved the bulk of that table roughly 25 million parameters into flash memory and memory-mapped it. The chip only needs to pull about six rows, roughly 450 bytes, for each new token. The remaining dense “thinking” core stays in the fast SRAM (around 560 K of active working memory). The model is stored at 4-bit quantization and occupies about 14.9 MB total. The result is a system that feels almost free to run. The heavy parameters sit quietly in flash and are sampled sparingly. The little core does the real work. It is elegant engineering of the purest kind. What I Am Doing With It Right Now I have the boards on the bench in the garage lab. The first units are already talking short, coherent stories appearing on a tiny wired display, generated entirely on the chip with no Wi-Fi, no API key, no cloud round-trip. Latency is local. Privacy is absolute. Power draw is low enough that battery operation becomes interesting. I am treating these as the first generation of true $8 AI machines. Early tests are focused on three practical directions. - Embedding the model into simple nodes. - Pairing it with local voice front-ends - Exploring whether multiple of these chips can be networked as a lightweight swarm. The model is deliberately limited. It was trained on the Microsoft TinyStories dataset and is excellent at coherent narrative, not at open-ended question answering or tool use. That is a feature, not a bug. It forces us to design systems around what the silicon can actually deliver instead of pretending every edge device needs a frontier model. Real Use Cases That Suddenly Become Practical Once you accept that a capable language model can live for eight dollars and run without the cloud, a new class of devices becomes possible: This is the opposite of the current trajectory that wants every intelligent act to travel through a remote server. It is the beginning of intelligence that is cheap enough, private enough, and local enough to become infrastructure rather than a service. We have spent years watching model sizes explode upward. The more interesting frontier may be the opposite direction: how small, how cheap, and how local can useful intelligence become? An eight-dollar chip that can tell coherent stories is not a toy. It is a proof that the lower bound keeps moving. The open repository is at https://t.co/a7gcHTR4ug I will keep testing, measuring, and reporting what these little machines can and cannot do. The age of abundant local intelligence just got a little more real, and it arrived wearing an eight-dollar price tag.
- 09
@_vmlops ·
Everyone talks about LLMs like you need massive GPUs and billions of parameters You really don’t Came across this: https://t.co/j5z6o7vnpO It’s a tiny ~9M parameter model you can train in minutes (even on Colab) What I liked: ▫️It shows the entire pipeline - tokenizer → transformer → training → inference ▫️No APIs, no shortcuts ▫️Just clean, from-scratch implementation If you’ve been confused about how LLMs actually work, this is a great place to start Sometimes building small things teaches way more than using big ones
- 10
@ashtilawat ·
This week, I gave 63 interns from Stanford, MIT, and UT a hard challenge: **Train your own small language model.** Not prompt one. Not wrap GPT-5.5 in a nicer UI. Actually fine-tune a small open model and prove it learned a specific behavior. The catch? It cannot be something a frontier model already does well. That is the whole game. Frontier models are insanely broad. They are 80–90% good at almost everything. But “pretty good” breaks down when the task is niche, domain-specific, or reliability-sensitive. That gap is where fine-tuning gets interesting. A small model is not going to beat Claude or GPT on general intelligence. But it can beat them in a narrow sliver. A weird edge case. A specific teaching behavior. A company-specific review style. A failure mode the frontier model keeps missing. That is what the interns are hunting for this week. 1. Find the gap. 2. Generate the data. 3. Filter it hard. 4. Fine-tune the model. 5. Build the eval. 6. Prove the behavior moved from the prompt into the weights. The biggest lesson: The model is not the deliverable. The dataset is.
- 11
@burkov ·
This paper argues that Small Language Models (SLMs) offer a more economical and suitable future for agentic AI by demonstrating their sufficient power for specialized tasks, outlining a conversion algorithm from LLMs to SLMs, and discussing the significant operational and economic impacts of this shift. ChapterPal: https://t.co/hmOIDRJfuH PDF: https://t.co/2TUNbfk3VJ
- 12
@techNmak ·
Microsoft made 100B parameter models run on a single CPU. bitnet.cpp: The official inference framework for 1-bit LLMs. The math behind 1-bit LLMs is what makes them revolutionary. Traditional LLMs use 16-bit floating point weights. Every parameter is a number like 0.0023847 or -1.4729. When you run inference, you multiply these floats together. Billions of times. That's why you need GPUs, they're optimized for floating point matrix multiplication. BitNet b1.58 uses ternary weights: {-1, 0, 1}. That's not a simplification. That's a fundamental change in the math. When your weights are only -1, 0, or 1: → Multiply by 1 = keep the value → Multiply by -1 = flip the sign → Multiply by 0 = skip entirely Matrix multiplication becomes addition and subtraction. No floating point operations. No GPU required. This is why bitnet.cpp achieves: → 2.37x to 6.17x speedup on x86 CPUs → 1.37x to 5.07x speedup on ARM CPUs → 71.9% to 82.2% energy reduction on x86 → 55.4% to 70.0% energy reduction on ARM The speedups scale with model size. Larger models see bigger gains because there are more operations to simplify. A 100B parameter model running at human reading speed (5-7 tokens/second) on a single CPU. That's not optimization. That's a different paradigm. Why 1.58 bits? Because log₂(3) ≈ 1.58. Three possible values = 1.58 bits of information per weight. The key insight: These models aren't quantized after training. They're trained from scratch with ternary weights. The model learns to work within the constraint. No precision loss. No quality tradeoff.
Watch video
- 13
@akshay_pachaar ·
Vibe train your AI agents. There's a new method that could replace LLM-as-a-judge for production agents. Most teams rely on a giant LLM as a judge to evaluate and guard their agent. But it has two major drawbacks: - It's slow and expensive at inference time - It often misses domain-specific failures Vibe training flips this. Researchers at Plurai distill a small language model that's specialized for your agent's exact use case. The SLM becomes your evaluator and your runtime guardrail, both in one. The training data isn't hand-curated either. They spin up a swarm of adversarial agents that debate and stress-test every use case your agent is supposed to handle. That synthetic interaction data trains the specialized SLM. So the judge actually understands what "wrong" looks like in your specific domain. The reported gains vs. standard LLM-as-a-judge setups: - ~8x faster inference - ~50% fewer evaluation errors Smaller, faster, and more accurate because it's specialized for the job. The SLM-for-agents thesis is playing out in a very concrete way. If LLM-as-a-judge is your current evaluation layer, this is worth benchmarking against. Paper link in the replies.
- 14
@KanikaBK ·
GOOGLE JUST SHRUNK AI MODELS FROM 31GB TO 4GB AND NOBODY IS TALKING ABOUT IT. I tested this compression tech for 2 days. It cuts AI memory by 85% without losing accuracy. Large language models running on your phone. No internet needed. No cloud costs. This changes everything: ↳ Runs GPT-level models on mid-range smartphones locally ↳ Slashes energy and battery drain by 85% ↳ AI agents that work offline inside everyday apps ↳ Zero data sent to servers Generative AI just moved from expensive cloud servers to your pocket.
- 15
@kimmonismus ·
Pretty insane: PrismML has introduced its 1-bit Bonsai models, compressing an 8.2B-parameter LLM into just 1.15 GB while maintaining performance comparable to leading 8B models. By focusing on “intelligence density,” the company claims roughly a 10x improvement in capability per GB, enabling fast, efficient AI that can run directly on devices like iPhones, Macs, and GPUs with significantly lower energy use.
- 16
- 17
@emollick ·
Gemma 4 E4B is impressive for an on-device LLM. GPT-4ish quality, and expect hallucinations. Here is: “List five sociological theories starting with u and what they are. Then describe them in a rhyming verse” Its in real time, the last is a little bit of a stretch, but not bad!
Watch video
- 18
@sukh_saroy ·
🚨Every LLM on the planet is stuck in first gear. GPT, Claude, Gemini, Llama. All of them. They generate text one token at a time. That's roughly 15 bits of information per step while burning billions of FLOPs each time. The bottleneck isn't model size anymore. It's the unit of prediction itself. A team from WeChat AI (Tencent) just broke that constraint entirely. Their paper introduces CALM — Continuous Autoregressive Language Models. Instead of predicting the next token, it predicts the next vector. A single dense vector that represents 4 tokens at once. Here's how it works: A lightweight autoencoder compresses a chunk of 4 tokens into one continuous vector. That vector can reconstruct the original tokens with over 99.9% accuracy. The model then autoregressively predicts these vectors instead of individual tokens. One step in CALM = four steps in a traditional LLM. The math problem this creates is brutal. You can't use softmax when there's no vocabulary to score against. You can't compute perplexity without probabilities. You can't even do temperature sampling without logits. So they built an entirely new toolkit from scratch: — Likelihood-free training using energy scores instead of cross-entropy — A new evaluation metric called BrierLM that works without explicit probabilities — A rejection sampling algorithm for temperature control based on Bernoulli Factory theory The result: CALM matches the performance of standard Transformers at significantly lower computational cost. Both training and inference get cheaper because you're running 4x fewer autoregressive steps. But here's what makes this more than an efficiency trick. They tried scaling K — the number of tokens per vector. At K=1, it underperforms discrete models. At K=4, it nearly matches them at a fraction of the compute. The implication is that there's a new scaling axis nobody has been optimizing: semantic bandwidth. We've spent years scaling parameters. Scaling data. Scaling context length. Nobody was scaling how much meaning each generation step carries. The discrete token has been the invisible ceiling on LLM efficiency since GPT-1. Expanding vocabulary exponentially to carry more information per step was a dead end — softmax over a billion entries is computationally suicidal. CALM sidesteps that wall entirely by leaving the discrete domain. One question this raises: if you can compress 4 tokens into one vector at 99.9% fidelity, what happens at 8? 16? 64? The paper shows compute drops proportionally with K while performance degrades only marginally. We might be looking at the architecture that makes current LLM inference costs look absurd in retrospect.
- 19
@AIHighlight ·
🚨 Breaking: A free AI model trained for $7,800 just beat one 400 times its size at competition math. It runs small enough to fit on a laptop. Weibo's AI lab published the results and open-sourced the whole thing. The industry has run on one idea for two years. Bigger is better. More parameters, more compute, more money, smarter model. The biggest systems cost hundreds of millions to train, and the whole race is built on the belief that scale is what makes a model smart. A model called VibeThinker put a crack in that. It has 1.5 billion parameters. DeepSeek R1, the model that went viral last year, has 671 billion. That is more than 400 times larger. On three of the hardest competition math benchmarks, the tiny model scored higher. 80.3 to 79.8 on one. 74.4 to 70.0 on another. 50.4 to 41.7 on the third. It matched or beat Claude Opus 4 and GPT-4.1 on coding tests too. The cost is the part that should stop you. DeepSeek R1 cost around $294,000 to post-train. A model called MiniMax cost $535,000. VibeThinker cost $7,800. The team rented the GPUs at about $2 an hour and finished in 3,900 hours. It did not win by being bigger. It won by being trained more carefully. The method makes the model explore many ways to solve a problem first, then sharpens it toward the ones that work. This is not a one-off. MIT researchers found a small model beating the largest ones at strategic reasoning for 1% of the cost. NVIDIA's own researchers argue small specialized models should do most of the work in AI systems, at 10 to 30 times lower cost than the frontier giants. The industry says intelligence requires scale, and scale requires the kind of money only a few companies have. The quiet finding of 2026 is that much of what the giant models do can be done by small ones almost anyone can afford to train. The race everyone watches is between the giants. The race that might matter more is the one shrinking them down to something that runs on a laptop. Source: VibeThinker, Weibo AI, arxiv 2511.06221. MIT News. NVIDIA Research.
- 20
@burkov ·
An absolute must read. LLMs cost a lot to run, so a common move is to train a small model to imitate a big one — feeding the small "student" the same inputs and having it match, word by word, the probabilities the large "teacher" assigns to each possible next word, a procedure called knowledge distillation. That matching is done on a fixed collection of example sentences, but a model writing text builds each sentence out of its own earlier words, so once the student makes an early choice that none of the training examples contained, it ends up in situations it was never shown, and small mistakes feed into later ones until the text degrades. In this ICLR 2024 paper from Google, Mila, and UoT, the authors instead have the student write sentences itself and use those sentences to choose the situations it gets tested on: at each point in a student-written sentence they take the words so far, ask the teacher what the distribution over the next word should be there, and push the student toward the teacher's answer — so the teacher supplies every target while the student's own writing decides where those targets get applied, which is exactly the off-track spots its writing tends to wander into. Tested on summarization, English-to-German translation, and grade-school math problems where the model writes out its reasoning before answering, this self-generated-data approach beats standard distillation recipes across a range of student sizes, and it slots into reinforcement-learning fine-tuning cleanly because both only need samples drawn from the student rather than gradients passed back through the sampling step. Read with an AI tutor and quizzes for better retention: https://t.co/xEf4BBIyig PDF: https://t.co/MmCnjS9LHM
- 21
@LiorOnAI ·
Google's latest paper on Compression is the future. Here's why. They compressed LLM memory 6x with zero accuracy loss. When ChatGPT writes a reply, it remembers every word you've said. That memory is stored in a growing notebook (KV cache). A 100,000-word conversation can eat 16 GB of GPU memory. That's half of what most high-end GPUs even have. This is the #1 cost of running AI. Not the thinking. The remembering. TurboQuant shrinks each number in that notebook from 32 bits to just 3. That's like replacing a full paragraph with three words and losing nothing. No retraining. Works on any model instantly. Compressing numbers usually destroys their meaning. Here's how they solved it: 1. Rotate the numbers randomly so they all land on a predictable curve (PolarQuant) 2. Use one extra bit to fix the tiny errors left behind (QJL) Once numbers are predictable, you need far fewer bits to store them. The results: > 8x faster on Nvidia H100 GPUs > 16 GB notebook shrinks to under 3 GB > Search indexing drops from 500 seconds to 0.001 > Accuracy identical to the uncompressed model There's a proven math limit on how good compression can get. TurboQuant is only 2.7x above that floor. We're near the ceiling. Every company running LLMs spends most of its budget on memory. This cuts that cost by over 80%. The race is no longer about bigger models. It's about cheaper inference. Models that needed a $200K server cluster start fitting on a single $2K GPU. AI agents run 24/7 without burning budgets. The companies that win won't just have the best models. They'll have the best compression. Papers are open-access on arXiv, presented at ICLR.
- 22
- 23
@ModelScope2022 ·
Meet Marco-Mini-Instruct: a highly sparse MoE multilingual model from Alibaba International. 17.3B total params, only 0.86B active (5% activation ratio). 🚀 Beats Qwen3-4B, Gemma3-12B, Granite4-Small on English, multilingual general, and cultural benchmarks — with a fraction of their active params. 🌍 29 languages: Arabic, Turkish, Kazakh, Bengali, Nepali and more 🧠 256 experts, 8 active per token. Drop-Upcycling from Qwen3-0.6B-Base. 🎯 2-stage post-training: SFT + Online Policy Distillation (Qwen3-30B → Qwen3-Next-80B cascade) ✅ Apache 2.0 🤖 https://t.co/d6cBKume9H 🤖 https://t.co/i3cQorDzh9
- 24
@andrewchen ·
set up a mini rack for a home lab setup (will share a pic soon) w my Mac mini and DGX spark with more coming. had a few thoughts as I play w qwen3.5, gemma4, and other models: - there’s an S curve on LLM model quality per use case. Show text output side by side from the latest and you can’t tell the difference. I assume we’ll get to a flattish part of the curve on coding, multimodal, and other use cases over time - you seem to be able to swap the model underneath a great UX and the whole thing is portable. Openclaw workflows and personality are a bunch of markdown files and can run equally on GPT or Opus - SOTA models can be distilled and only stay in front of open weight models by ~12-18 months. Have to keep innovating to stay ahead (and god bless this dynamic from the startup ecosystem’s POV) - local AI models getting very good particularly on the latest Apple hardware. Very usable for many use cases and will only get better Obv still a big diff between what I can run locally and what’s available in the cloud - but the trend is super interesting and feels inevitable
- 25
@Shruti_0810 ·
Trains billion-parameter LLMs from scratch on a single GPU. Most people still think AI training requires millions of dollars and entire datacenters. This open-source repo proves them wrong. Build and train GPT-style models from scratch using techniques designed for consumer hardware. The barrier to training LLMs just got a lot lower. Repo: https://t.co/XdC088qSRU
- 26
@Shruti_0810 ·
Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License
Watch video
- 27
@arpit_bhayani ·
When a public LLM benchmark says one model is better than another, it is testing broad, difficult reasoning and, more importantly, opinionated tasks. Your use case might not need that. For example, if you are summarizing tickets, classifying intent, or extracting fields from structured text, a smaller and cheaper model can match the output of a frontier model. General benchmarks tell us which model wins on average across a wide mix of tasks. They do not tell us which model wins on your specific, narrow, repeatable task. The only way to know for sure is to build your own benchmark. One simple way is to take real examples from your workload, run them across your candidate models, and score the outputs against what you actually need. It is laborious work, but yes, your favorite LLM can help you build that :) A model ranked lower on a general benchmark can be the better, cheaper choice for your use case, simply because your task was never complex enough to need the extra capability in the first place. In a gist, always try to answer: Which model is better for your task? Not which model is better in general. Hope this helps.
- 28
@LiorOnAI ·
Most language models only read forward. Perplexity just open-sourced 4 models that read text in both directions. They used a technique from image generation to retrain Qwen3 so every word can see every other word in a passage. That changes how well a model understands meaning. They built four models from this: 1. Two sizes: 0.6B and 4B parameters 2. Two types: standard search embeddings and context-aware embeddings The context-aware version is the interesting one. It processes an entire document at once, so each small chunk "knows" what the full document is about. Standard embeddings treat each chunk in isolation. > Tops benchmarks for models of similar size > Works in multiple languages out of the box > MIT licensed, free for commercial use If you're building search over large document collections, you can now get document-level understanding without running a massive model. Small enough to actually deploy.
- 29
- 30
@RoundtableSpace ·
A 14B model on a single consumer GPU hitting 74.6% on LiveCodeBench. No fine-tuning. No API calls. No cloud. No data leaving your machine. The idea is simple wrap a frozen small model in smart infrastructure and it starts competing with frontier models at a fraction of the cost. Is the moat around big cloud AI smaller than everyone thought?
- 31
@thestreamingdev ·
3 ai models racing simultaneously on an m2 macbook air (8gb) built tiny bit, a local agent terminal for benchmarking small language models head-to-head on apple silicon. @PrismML @liquidai @Alibaba_Qwen tested bonsai-8b (1-bit, 1.16gb) vs qwen3-0.6b (q4, 0.37gb) vs lfm2.5-350m (q4, 0.25gb) Here’s how it went 🧵 https://t.co/cKxsmx8fNv
Watch video
- 32
@SimonHoiberg ·
Everyone is now looking at self-hosted AI models. Naturally. But if you think you can buy a Mac Mini and replace Claude or OpenAI, you need a serious wakeup call. Cause there's more to it than just the size of the weights - for a model to become practically useful, we need to consider: - Tool use - Context size - Quantization Cause for agents, this makes all the difference between a local model that can "just run" and a model that's actually useful. Let me explain 👇
Watch video
- 33
- 34
@Marktechpost ·
Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning - LFM2.5-350M is a 350M parameter small language model trained on 28 trillion tokens, with a hybrid architecture built from 10 double-gated LIV convolution blocks and 6 GQA blocks, plus 32K context support. -This model is built for instruction following, tool use, structured extraction, and edge deployment. Liquid AI team reports 76.96 on IFEval, 30.64 on GPQA Diamond, and 40.4K output tokens/sec on a single H100 at high concurrency. -The bigger point: small models are becoming serious infrastructure components for local and agentic workloads. Key Points: --- Best-in-class performance: A 350M model rivaling much larger models, bringing high-quality AI to your pocket. --- Fast edge inference: 313 tok/s decode on AMD CPU, 188 tok/s on Snapdragon Gen4. Runs under 1GB of memory with day-one support for llama.cpp, MLX, and vLLM. --- Scaled training: Extended pre-training from 10T to 28T tokens and large-scale multi-stage reinforcement learning. Full analysis: https://t.co/baJCEsDqFY Model weight: https://t.co/vHth0AdcF6 Docs: https://t.co/Qp35ZdnP3g @liquidai @maximelabonne
- 35
@ttunguz ·
2025 is the year of agents, & the key capability of agents is calling tools. When using Claude Code, I can tell the AI to sift through a newsletter, find all the links to startups, verify they exist in our CRM, with a single command. This might involve two or three different tools being called. But here’s the problem: using a large foundation model for this is expensive, often rate-limited, & overpowered for a selection task. What is the best way to build an agentic system with tool calling? The answer lies in small action models. NVIDIA released a compelling paperarguing that “Small language models (SLMs) are sufficiently powerful, inherently more suitable, & necessarily more economical for many invocations in agentic systems.” I’ve been testing different local models to validate a cost reduction exercise. I started with a Qwen3:30b parameter model, which works but can be quite slow because it’s such a big model, even though only 3 billion of those 30 billion parameters are active at any one time. The NVIDIA paper recommends the Salesforce xLAM model – a different architecture called a large action model specifically designed for tool selection. So, I ran a test of my own, each model calling a tool to list my Asana tasks. The results were striking: xLAM completed tasks in 2.61 seconds with 100% success, while Qwen took 9.82 seconds with 92% success – nearly four times as long. This experiment shows the speed gain, but there’s a trade-off: how much intelligence should live in the model versus in the tools themselves. This limited With larger models like Qwen, tools can be simpler because the model has better error tolerance & can work around poorly designed interfaces. The model compensates for tool limitations through brute-force reasoning. With smaller models, the model has less capacity to recover from mistakes, so the tools must be more robust & the selection logic more precise. This might seem like a limitation, but it’s actually a feature. This constraint eliminates the compounding error rate of LLM chained tools. When large models make sequential tool calls, errors accumulate exponentially. Small action models force better system design, keeping the best of LLMs and combining it with specialized models. This architecture is more efficient, faster, & more predictable. https://t.co/2ASQ5btMqq
- 36
@DivyanshT91162 ·
People said LLMs needed GPUs. This one runs on an $8 ESP32-S3 microcontroller. • 28.9M parameters • 14.9 MB 4-bit model • ~9.5 tokens/sec • 0 cloud • 0 internet • 100% on-device inference • 100% open source • MIT License The trick? Instead of loading the entire model into RAM, it stores ~25M parameters in flash memory and only fetches the tiny pieces needed for each token using Google's Per-Layer Embeddings. Memory layout: → SRAM: inference core → PSRAM: working memory → Flash: 25M-parameter embedding table Result: A language model roughly 100× larger than previous ESP32 implementations running on an $8 microcontroller. It's trained on TinyStories, so it's not replacing ChatGPT. But it proves something much bigger: The future of AI isn't just bigger models. It's smarter architectures that bring AI to tiny, low-power devices without sending a single byte to the cloud. 100% Open source. Repo👇
- 37
@smratitiwa86867 ·
🤯 Researchers just made LLMs over 99% sparse… without killing performance. And unlike most “sparse AI” papers, this one actually gets REAL GPU speedups instead of just theoretical FLOPs reductions. The trick? LLMs are already naturally sparse inside their feedforward layers. The problem is GPUs hate random unstructured sparsity because modern hardware is optimized for dense matrix math. So this paper redesigned the sparse format + GPU kernels together, making sparsity finally practical at scale. Results: • >99% sparsity with minimal quality loss • Faster inference • Lower VRAM usage • Less energy consumption • Lighter Transformer models without massive tradeoffs This is the kind of optimization that could make local LLMs way more efficient on consumer hardware. Paper name: “Sparser, Faster, Lighter Transformer Language Models” Link in comments 👇
- 38
@arsh_goyal ·
A 1 billion parameter AI model. Runs entirely on your laptop with no cloud or GPU bills or internet. MiniCPM5-1B just dropped and it's the best 1B local model right now. Here's what actually makes it worth trying: > Beats Qwen3.5 0.8B and LFM2.5 1.2B on math, coding, and tool calling > INT4 quantization needs just 0.5GB RAM which fits on a phone > I ran it on my machine with zero server calls and is Instant. The training framework for this is ForgeTrain which was written entirely by AI. No human programmers and it runs 10% faster than NVIDIA Megatron. An AI built the tool that trained this AI. We are officially in that timeline. If you're building on-device apps, this is the model to start with. Links in the comments below Full breakdown in the video 👇
Watch video
- 39
@ttunguz ·
Pocket Power : From State of the Art to Your Phone in 23 Months Two years ago, the idea of useful AI on your phone was fantastical. Siri couldn't finish a sentence. Local models hallucinated nonsense. Last week, Google released Gemma 4 E4B, a free model that matches GPT-4o & runs entirely on your phone. The next few weeks promise even more advanced pocket models. The market expects new releases from DeepSeek, Qwen, Kimi & Minimax. Frontier models don't stay frontier for long. Within three to four months, you can run a model with similar performance on your laptop; 23 months later, you can run the same model on your phone. Three forces are driving this compression. Better algorithms : distillation & reinforcement learning squeeze more capability into fewer parameters. Talent density : the biggest prizes in capitalism attract the best minds in the field. These are the fastest growing software companies in history. And capital : a trillion dollars invested in data centers powering training. In 23 months, the same capability that needed 1.8 trillion parameters now fits in 4 billion parameters. A 450x compression. At this rate, the phone in your pocket will run today's frontier models before you upgrade it. https://t.co/diAmGuPSuo
- 40
@RoundtableSpace ·
Language Model Builder is a free app that walks you through building your own small language model from scratch with a 90-minute interactive textbook and a local training workbench.
Watch video
- 41
@mark_k ·
On March 31, @PrismML unveiled Bonsai, a family of 1-bit ultra-dense language models that pack astonishing intelligence into tiny footprints. Named after the art of miniature trees, these models prove that true AI power thrives when compressed rather than expanded. The flagship Bonsai 8B features 8.2 billion parameters yet requires only 1.15 gigabytes of memory, making it fourteen times smaller, eight times faster at inference, and five times more energy efficient than full-precision peers. It runs over forty tokens per second on an iPhone 17 Pro, while the 4B and 1.7B variants shrink to 0.57 and 0.24 gigabytes and hit 130 tokens per second on recent Macs and iPhones. Built with pure 1-bit precision across every layer and no higher-bit shortcuts, Bonsai delivers crisp reasoning and strong instruction following. Available now under Apache 2.0 on Hugging Face, it promises fast, private, local AI for phones, robots, and edge devices, showing that the future of intelligence may simply be denser and more agile.
- 42
- 43
@sabir_huss50540 ·
A 28.9 million parameter language model just ran on an $8 chip. No server. No wifi. No GPU. It runs on an ESP32-S3, the kind of chip you solder into a hobby project, writing each word to a tiny wired screen at about 9.5 tokens per second. The last model anyone ran on a chip this small had 260 thousand parameters. This one holds a hundred times more. Here is the trick. The chip has 512KB of fast memory, nowhere near enough. So most of the model never enters it. The 25 million row embedding table sits in slow flash, and each token pulls just 450 bytes from it. The idea is Google's Per-Layer Embeddings from Gemma, on a microcontroller for the first time. → 28.9M params, 14.9MB at 4-bit → $8 ESP32-S3, 512KB of fast memory → 9.5 tokens per second, fully on-chip → zero network, everything on the device The catch is real. It trained on TinyStories, so it writes short simple tales and nothing else. No questions, no code, no facts. The point is the architecture, not the output. MIT. He even left the bug in his own history, the one that inflated an early number, right next to the fix. 100% open source.
- 44
@TheTuringPost ·
Should remembering and writing use the same gate? A very interesting Korean model, Naju, brings back the LSTM principle of separate control over retention and writing. Naju implements it inside a modern state-space model compatible with parallel scans: - one gate decides how much of the existing state survives - another how strongly new information is written Naju stores information in a fixed-size recurrent state that is updated at every step through these gates. One additional nuance: Naju can also preserve existing information while adding new information, because both gates can be high at the same time The model was trained on sequences of 512 tokens, but at 2,048 tokens it reached 0.99 accuracy on retention and 0.89 on overwrite. None of the tested baselines handled both equally well: - xLSTM was better at remembering - GLA was better at overwriting ▪️ So Naju is a useful reminder that long memory is not only about context length. A fixed state must also know what to preserve and what to replace. But there are still 2 limitations: - Speed: At 32K tokens, Naju was 2.2× slower than Mamba and 3.3× slower than Mamba-2. - Scaling: It's unclear how well it scales, because it was tested on small language-model experiments (WikiText-103 with a 1.2B-token training budget)
- 45
@cleanunicorn ·
Google is talking about TurboQuant and it's huge They're releasing this compression algorithm to the world, and it just might be what powers those 2M context window models. The Problem Every LLM uses a key-value cache to track conversation context. It's like a digital cheat sheet. But it's also a massive memory bottleneck. When you load an LLM, you can't use 100% of the memory, you need to save space for the cache. This is why most consumer hardware can only run smaller models. The Solution TurboQuant compresses this cache with zero accuracy loss. The Results - 32-bit → 3-bit compression (no accuracy loss) - 6x smaller key-value cache - 8x faster attention computation on H100 GPUs - Works with any existing model (no retraining) Why this matters - LLMs become cheaper to run - Massive context windows become practical - Vector search engines get a serious speed boost (semantic search, not keyword matching) The paper's already on arXiv and will be presented at ICLR 2026. I'll meet you in Rio in person to geek out on AI tech. Just ping me. Related research https://t.co/haRyjicHSC https://t.co/ubc0j6hsfA https://t.co/pWXVrbmPfb
- 46
- 47
- 48
@ainativedev ·
What's the smallest model that can actually run an AI agent? Not the cheapest. Not the fastest. The smallest one that reliably gets the job done. In their latest analysis, Nicolas Fortuin and Baptiste Fernandez (@FernandezBap ) put NVIDIA's (@nvidia) open-weight Nemotron models to the test across real coding-agent tasks. The results show that agent models don't degrade gradually as they get smaller. They hit the floor. The smallest model tested couldn't complete enough tasks to produce a meaningful score. Nano 30B could handle focused, well-defined work, but struggled once tasks became longer and more agentic. Super 120B was the first model that consistently cleared that capability threshold. One of the biggest takeaways is that skills aren't a substitute for capability. They amplify models that are already capable enough, but they can't rescue a model that hasn't crossed the agent floor. If you're choosing models based on parameter count or price alone, you might be optimizing the wrong thing. Read the full breakdown here: https://t.co/7Wgs1bQJLB
- 49
- 50
@JulianGoldieSEO ·
MINICPM5 JUST BROKE THE "BIGGER IS BETTER" MYTH A 1B parameter model is doing things that used to require models 10x its size. Here's what makes it different: Performance: → Just 1B parameters but supports reasoning, coding, and tool use → Runs locally on a laptop or even a phone → No expensive cloud GPUs required Smart Design: ✓ Hybrid thinking with two modes: Fast responses for simple tasks Deeper reasoning for harder problems ✓ 128K context window for long documents and conversations ✓ Structured outputs for tables, forms, and workflows Developer Friendly: ✔ Standard architecture that works with existing AI runtimes ✔ Fully open source ✔ Easy to deploy without custom engineering The trend is obvious. The next generation of AI won't just be bigger. It'll be smaller, faster, cheaper, and running on your own devices.
Watch video
Best Tweets by Topic