Local Llama Inference
Running Llama-family models locally with llama.cpp and related runtimes, including quantization, hardware performance, long context, and low-memory inference.
44%
Best tweets about Llama
50 Best Tweets About Meta Llama (2026)
Find the best tweets about Meta Llama, including open-weight models, fine-tuning, benchmarks, local deployment, and developer use cases. Updated weekly.
Model-specific Llama research and engineering discussions, excluding references to the animal or unrelated products.
- Creators
- 41
- Updated
Original Xholic analysis
What 50 top Llama posts reveal
The Llama conversation centers on local inference, open-model infrastructure, and practical specialization. The dataset is predominantly supportive (66%), while cited posts also raise concerns about reported reasoning failures, hardware requirements for large concurrent workloads, and the future of open releases. Local Llama tooling is the largest single theme, representing 44% of tweets.
- Dominant tone
- Positive
- Median score
- 16.7
- Leading format
- Announcement
- Recent posts
- 42%
66% of posts
All-time engagement
40% of posts
Published in 90 days
Conversation map
The themes creators return to
Open-Model Ecosystem
The Llama open-weight ecosystem: openness, licensing, Meta's strategy, local AI sovereignty, and competition with hosted models.
26%
Architecture & Optimization
Llama architecture and model optimization research, including MoE, quantization methods, efficient representations, and model internals.
24%
Agentic & Production Applications
Llama-based agents and production applications, including tool calling, structured outputs, RAG, recommendation, outreach, and agent orchestration.
20%
Evaluation, Safety & Interpretability
Research on Llama capabilities, reasoning behavior, memory transfer, prompt susceptibility, safety mechanisms, and interpretability.
16%
Fine-Tuning & Post-Training
Fine-tuning, post-training, and task specialization of Llama models using LoRA, QLoRA, DPO, evolution strategies, and training frameworks.
14%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 45
- Median reposts
- 5
- Median replies
- 8
- Median views
- 5.1K
Posts with media make up 80% of this collection. Their median all-time score is 20.6, compared with 3.62 for text-only posts.
Format mix
- Announcement 40% · score 19.7
- Opinion 24% · score 11.7
- Case Study 22% · score 13.3
- Prediction 6% · score 28.1
Consensus and debate
Where creators agree, and where they do not
Shared view
Local execution is the core narrative
Local inference is the largest theme in the dataset (44% of tweets). Cited posts include a llama.cpp throughput demonstration, AirLLM’s layer-streamed low-memory approach, and a reported offline MacBook workflow with long context.
Shared view
Openness is tied to control
Posts frequently frame open and local deployment in terms of hardware independence, community-built infrastructure, decentralized inference, and keeping enterprise data within a self-hosted environment.
Open debate
Local practicality has workload limits
One post describes useful local workflows for tasks such as tool calling and everyday automation, while another presents a specific high-concurrency Llama 3.1 70B BF16 workload as requiring a multi-GPU setup rather than a Mac.
Open debate
Meta’s openness trajectory draws criticism
Multiple posts express concern about a perceived retreat from open releases. Their emphasis differs: access to small models for downstream training, a claimed first closed Meta model, and a potential missed enterprise opportunity for local alternatives.
Open debate
Agent gains coexist with reasoning concerns
One thread reports that an Atomic Task Graph execution framework enabled Llama 3.1 8B to outperform a GPT-4+ReAct setup on ALFWorld and WebShop. Another post describes reported arithmetic-prompt susceptibility, including a Llama example affected by an irrelevant clause.
What performs
Patterns behind standout posts
Breakout posts span capability and caution
The five benchmark outliers cover low-memory inference, a reasoning-evaluation warning, local throughput, architecture commentary, and the llama.cpp ecosystem. Their deterministic all-time scores range from 570.97 to 6,931.99.
Announcements lead the format mix
Announcements make up 40% of posts and have a 19.69 median all-time score. The highest-scoring outlier is the AirLLM announcement describing low-memory Llama execution.
Tutorials show the highest median score
Tutorials represent 4% of posts but have the highest format median all-time score, 53.11. The two cited tutorials cover MoE concepts and categories of AI guardrails.
Statistical standouts
- View standout post 1 Score 6932.0 · 415.34× median
- View standout post 2 Score 3224.5 · 193.2× median
- View standout post 3 Score 841.6 · 50.43× median
- View standout post 4 Score 614.3 · 36.81× median
- View standout post 5 Score 571.0 · 34.21× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Vaishnavi
@_vmlops
2 posts
-
2. Aakash Gupta
@aakashgupta
2 posts
-
3. abdel
@AbdelStark
2 posts
-
4. Alex Prompter
@alex_prompter
2 posts
-
5. BURKOV
@burkov
2 posts
-
6. Georgi Gerganov
@ggerganov
2 posts
Gerganov links benchmarks to infrastructure
Georgi Gerganov’s two posts combine a concrete llama.cpp performance demonstration with his argument for an open, hardware- and operating-system-spanning local-AI stack.
Gupta foregrounds model strategy
Aakash Gupta’s posts discuss model strategy from two angles: a reported argument for smaller, cleaner architectures paired with external memory, and criticism of Meta’s reported closed-model direction.
Toor balances risk and local autonomy
Nav Toor’s cited posts pair a strongly worded account of reported LLM reasoning failures with an argument for offline, executable local-model tooling.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Llama tweets
Top Llama tweets from 41 creators
Ranked 01–50
- 01
@LiorOnAI ·
You can now run 70B LLMs on a 4GB GPU. AirLLM just made massive models usable on low-memory hardware. 𝗪𝗵𝗮𝘁 𝗷𝘂𝘀𝘁 𝗵𝗮𝗽𝗽𝗲𝗻𝗲𝗱 AirLLM released memory-optimized inference for large language models. It runs 70B models on 4GB VRAM. It can even run 405B Llama 3.1 on 8GB VRAM. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀 AirLLM loads models one layer at a time. Instead of loading everything: → Load a layer → Run computation → Free memory → Load the next layer This keeps GPU memory usage extremely low. 𝗞𝗲𝘆 𝗱𝗲𝘁𝗮𝗶𝗹𝘀 • No quantization required by default • Optional 4-bit or 8-bit weight compression • Same API as Hugging Face Transformers • Supports CPU and GPU inference • Works on Linux and macOS Apple Silicon 𝗪𝗵𝗮𝘁 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗱𝗼 • Run Llama, Qwen, Mistral, Mixtral locally • Test large models without cloud GPUs • Prototype agents on cheap hardware
- 02
@heynavtoor ·
🚨SHOCKING: Apple just proved that AI models cannot do math. Not advanced math. Grade school math. The kind a 10-year-old solves. And the way they proved it is devastating. Apple researchers took the most popular math benchmark in AI — GSM8K, a set of grade-school math problems — and made one change. They swapped the numbers. Same problem. Same logic. Same steps. Different numbers. Every model's performance dropped. Every single one. 25 state-of-the-art models tested. But that wasn't the real experiment. The real experiment broke everything. They added one sentence to a math problem. One sentence that is completely irrelevant to the answer. It has nothing to do with the math. A human would read it and ignore it instantly. Here's the actual example from the paper: "Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have?" The correct answer is 190. The size of the kiwis has nothing to do with the count. A 10-year-old would ignore "five of them were a bit smaller" because it's obviously irrelevant. It doesn't change how many kiwis there are. But o1-mini, OpenAI's reasoning model, subtracted 5. It got 185. Llama did the same thing. Subtracted 5. Got 185. They didn't reason through the problem. They saw the number 5, saw a sentence that sounded like it mattered, and blindly turned it into a subtraction. The models do not understand what subtraction means. They see a pattern that looks like subtraction and apply it. That is all. Apple tested this across all models. They call the dataset "GSM-NoOp" — as in, the added clause is a no-operation. It does nothing. It changes nothing. The results are catastrophic. Phi-3-mini dropped over 65%. More than half of its "math ability" vanished from one irrelevant sentence. GPT-4o dropped from 94.9% to 63.1%. o1-mini dropped from 94.5% to 66.0%. o1-preview, OpenAI's most advanced reasoning model at the time, dropped from 92.7% to 77.4%. Even giving the models 8 examples of the exact same question beforehand, with the correct solution shown each time, barely helped. The models still fell for the irrelevant clause. This means it's not a prompting problem. It's not a context problem. It's structural. The Apple researchers also found that models convert words into math operations without understanding what those words mean. They see the word "discount" and multiply. They see a number near the word "smaller" and subtract. Regardless of whether it makes any sense. The paper's exact words: "current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data." And: "LLMs likely perform a form of probabilistic pattern-matching and searching to find closest seen data during training without proper understanding of concepts." They also tested what happens when you increase the number of steps in a problem. Performance didn't just decrease. The rate of decrease accelerated. Adding two extra clauses to a problem dropped Gemma2-9b from 84.4% to 41.8%. Phi-3.5-mini from 87.6% to 44.8%. The more thinking required, the more the models collapse. A real reasoner would slow down and work through it. These models don't slow down. They pattern-match. And when the pattern becomes complex enough, they crash. This paper was published at ICLR 2025, one of the most prestigious AI conferences in the world. You are using AI to help you make financial decisions. To check legal documents. To solve problems at work. To help your children with homework. And Apple just proved that the AI is not thinking about any of it. It is pattern matching. And the moment something unexpected shows up in your question, it breaks. It does not tell you it broke. It just quietly gives you the wrong answer with full confidence.
- 03
@ggerganov ·
Let me demonstrate the true power of llama.cpp: - Running on Mac Studio M2 Ultra (3 years old) - Gemma 4 26B A4B Q8_0 (full quality) - Built-in WebUI (ships with llama.cpp) - MCP support out of the box (web-search, HF, github, etc.) - Prompt speculative decoding The result: 300t/s (realtime video)
Watch video
- 04
@aakashgupta ·
Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.
Watch video
- 05
@ggerganov ·
llama.cpp at 100k stars now that 90% of the code worldwide is being written by AI agents, I predict that within 3-6 months, 90% of all AI agents will be running locally with llama.cpp 😄 Jokes aside, I am going to use this small milestone as an opportunity to reflect a bit on the project and the state of AI from the perspective of local applications. There is a lot to say and discuss and yet it feels less and less important to try to make a point. Opinions about viability of local LLMs are strongly polarized, details are overlooked, the scientific approach is lacking. Arguments are predominantly based on vibes and hype waves. One thing is clear though - local LLMs are used more and more. I expect this trend to continue and likely 2026 will end up being one of the most important years for the local AI movement. I admit that I didn't expect the agentic era to come so quickly to the local LLM space. One year ago, the available models were too computationally expensive for doing long-context tasks. There wasn't an obvious path towards meaningful agentic applications. The memory and compute requirements were huge. Last summer, with the release of gpt-oss, things started to change. It was the first time we saw a glimpse of tool calling that actually works well within the resource constraints of our daily devices. Later in the year, even better models were released and by now, useful local agentic workflows are a reality. Comparing local vs hosted capabilities at a given moment of time is pointless. To try put things into perspective: - We don't need frontier intelligence to automate searches and sending emails - We don't need trillion parameter models to be able to summarize articles or technical documents - We don't need massive GPU data centers to control our home appliances or turn the lights off in the garage I believe that there is a certain level of intelligence we as humans can comprehend and meaningfully utilize to improve our working process. Beyond that level, access to more intelligence becomes unnecessary at best and counterproductive at worst. I also believe that that level of useful artificial intelligence is completely within reach locally and it has always been just a matter of implementing the right software stack to bring it to the end user. With llama.cpp, I am confident that we continue to be on the right track of building that software stack! The llama.cpp project is going stronger than ever. With more than 1500 contributors, the project keeps growing steadily. From technical point of view, I think that llama.cpp + ggml is the only solution that actually makes sense. That is, the software stack must run efficiently on every possible device, hardware and operating system. The technology is too important to be vendor-locked. It has to be developed in the open, by the community, together with the independent hardware vendors. This is the only right way to build something that will truly make a difference in the long run. I won't try to convince you about what is currently and will be possible with local AI. We will just continue to build as usual. I am confident that after the smoke clears and we look objectively at what we have built together, the benefits will be obvious to everyone. Big shoutout to all llama.cpp maintainers. I feel extremely lucky to be able to work together with so many talented contributors. Every day I learn something new and I feel there is so much more cool stuff that we are going to build. Also, I am really thankful that the project continues to have reliable partners to support it! Cheers!
- 06
@techNmak ·
Sebastian Raschka is one of the most respected researchers in ML/AI education. Period. And now he's done something quietly brilliant. He built an LLM Architecture Gallery - a single, browsable reference that maps out the internal architecture of every major open-weight model released in the last few years. This is a serious research artifact, made free for everyone. Here's what's inside: 🔹 GPT-2 XL (1.5B) 🔹 Llama 3 (8B) 🔹 OLMo 2 (7B) 🔹 Llama 3.2 (1B) 🔹 Qwen3 (4B, 8B, 32B) 🔹 DeepSeek V3/R1 (671B) 🔹 Kimi K2 (1 Trillion) 🔹 Gemma 3 (4B, 27B, 270M) 🔹 Mistral 3.1 Small (24B) & Mistral Large (673B) 🔹 Llama 4 Maverick (400B) 🔹 Qwen3 235B-A22B & Qwen3 Coder Flash 🔹 SmolLM (1B) 🔹 GPT-OSS (20B, 120B) 🔹 Grok 2.5 (270B) 🔹 GLM-4.5 (355B), GLM-5 (744B), GLM-4.7 (355B) 🔹 MiniMax-M2 (230B) & MiniMax-M2.5 🔹 Kimi Linear (48B-A3B) 🔹 OlMo 3 (7B) & OlMo 3 (32B) 🔹 Nemotron 3 Nano (20B-A3B) & Nemotron 3 Super 🔹 Xiaomi MiMo-V2-Flash (309B) 🔹 Arcee AI Trinity Large (400B) 🔹 Tiny Aya (3.35B) 🔹 Step 3.5 Flash (196B) 🔹 Nanbeige (4.1, 3B) 🔹 Qwen3.5 (997B) 🔹 Ling 2.5 (1T) 🔹 Sarvam (30B, 105B) And for each model, he links: → The original tech report → The config[.]json (so you can verify every number yourself) → From-scratch implementations where available But here's what makes it truly special. He also added short concept explainers, so you're not just staring at boxes and arrows: → GQA (Grouped Query Attention) → MLA (Multi-head Latent Attention) → SWA (Sliding Window Attention) → QK-Norm → NoPE (No Positional Encoding) → Gated DeltaNet This is the kind of resource that used to require buying 3 textbooks, reading 40 papers, and spending a weekend. Now it's one link. If you're studying LLMs, building on top of them, or just trying to understand how the field has evolved, this is a must-bookmark.
- 07
@alex_prompter ·
🚨 Holy shit… Columbia University just dropped one of the most unsettling papers on AI inference I’ve read in a long time. They proved that the entire private AI inference industry built the wrong thing. Prior methods: encrypt the full transformer. 280GB per query. 60-second latency. Enterprise-grade security theater. GPT, Gemini, Qwen, and Mistral independently converged to nearly identical internal representations. One linear equation connects them. > Sub-second inference. 1MB of communication. Same security guarantees. > The private AI inference problem is real. Hospitals can't send patient data to OpenAI. Banks can't send transaction records to Google. Legal firms can't send case files to Anthropic. The solution the industry built: encrypt everything every layer, every attention head, every weight using homomorphic encryption and secure multi-party computation. The result: 280GB of encrypted communication per query. 60-second latency. > Infrastructure costs that make production deployment practically impossible. > Columbia University found the shortcut everyone missed. The Platonic Representation Hypothesis the observation that large models trained on enough data tend to converge toward a shared statistical understanding of the world turns out to be exploitable. GPT, Gemini, Qwen, Mistral, and Cohere, trained independently on different data with different architectures for different objectives, developed internal representations with CKA similarity scores between 0.595 and 0.881. That's not close. > That's essentially the same space. > If the spaces are the same, you don't need to encrypt the model. You learn a single affine transformation one matrix that maps your model's internal representations into the provider's space. Encrypt that matrix. > Send it. The provider runs one linear classification operation on encrypted data and returns the encrypted prediction. You decrypt locally. The transformer never gets encrypted. The weights never get exposed. The query never leaves your control in readable form. > HELIX is the system they built on this insight. During training, the client encrypts their embeddings from public data and sends them to the provider, who computes the alignment map under encryption and returns it. During inference, the client applies the alignment locally, encrypts the transformed representation, and sends it. The provider applies a linear classifier homomorphically and returns the encrypted prediction. > Multiplicative depth of one. No bootstrapping required. 128-bit security by CKKS standard. → Prior methods communication cost: 280.99GB per query (Iron), 25.74GB (BOLT), 68.6GB (MPCFormer) → HELIX communication cost: less than 1MB per query → Prior methods latency: 20-60+ seconds per query → HELIX latency: sub-second → Cross-model CKA similarity: 0.595 to 0.881 across GPT, Gemini, Qwen, Mistral, Cohere → Text generation quality: 60-70% of single-model baseline for high-compatibility pairs → Tokenizer compatibility predicts generation quality with r=0.898 The finding that should end careers: models above 4B parameters with tokenizer compatibility above 0.7 exact match rate can generate coherent text across model families using only a linear transformation. Qwen encoding. Llama decoding. No fine-tuning. No weight sharing. No data transfer. Just matrix multiplication applied to the boundary between two independently trained systems that accidentally became the same thing.
- 08
@IntuitMachine ·
The One Change That Lets Small Models Outperform Their Size 1/ Everyone knows you need a 70B model to beat GPT-4 on complex agent tasks. We did it with 8B—by changing one thing that has nothing to do with the model. A thread on why your agent's biggest problem isn't the LLM. 🧵 2/ The standard approach: feed the LLM a growing text history, ask it to pick the next action, repeat. This works... until it doesn't. Errors propagate. Context bloats. Hallucinations spike. And when something breaks, you replan everything. 3/ Here's the kicker: The problem isn't your model's intelligence. It's that you're asking it to hold plan structure + execution state + I/O dependencies all inside a linear text stream. That's like running an OS without a process table. 4/ Enter: Atomic Task Graph (ATG) Instead of a text trajectory, you build an explicit DAG. Each node = one tool call. Edges = data dependencies. The LLM still does the thinking—but now the graph holds the structure. 5/ Three moves make this work: ✅ Interface-preserving recursion: Break tasks into subtasks while keeping I/O contracts clean ✅ Dependency-aware execution: Run independent branches in parallel; catch bad plans before running them ✅ Minimal repair: When something fails, fix only the broken subgraph—leave the rest frozen 6/ Result? Llama-3.1-8B-Instruct beats GPT-4+ReAct on ALFWorld (household tasks) and WebShop (shopping). Not with fine-tuning. Not with more data. Just by swapping the execution substrate from text → graph. 7/ Why does this work? Context narrowing: Each node sees only its local inputs—no bloated history Pre-execution validation: The graph lets you "think" before acting Localized failure: Repair 10% of the graph instead of replanning 100% 8/ The contrarian insight: Control framework > model size (in the 7–70B range). You're not squeezing more juice from the same fruit. You're giving the model a better glass to pour into. 9/ Practical translation: • 20–40% step reduction (parallelism) • 70%+ hallucination drop (narrower context) • 3× faster recovery (minimal repair) • Training-free, plug into existing tool APIs This isn't research theater. It's production-ready architecture. 10/ The bigger implication: If you can beat GPT-4 by changing the substrate instead of the model, what else have we been over-parameterizing? Retrieval pipelines? Code generation? Multimodal workflows? The graph wins again. 11/ [Final + CTA] TL;DR: Stop storing your agent's plan in text. Start storing it in a DAG. Small models suddenly look a lot smarter.
- 09
@kadirnardev ·
The Qwen team is no longer releasing their models as open source, and this is a big problem for us. We need small models to train many models like TTS, STT, Omni, and others. Previously there was LLaMA, but they're no longer releasing either. The Qwen team won't be releasing anymore either. Our only hope is the LFM models. Minimax, Kimi, and GLM teams are releasing great models for open source, but none of them release small models. And if these companies also stop releasing open source, it's going to be really bad :(
- 10
@BitcoinNewsCom ·
Jack Dorsey's Block just launched mesh-llm. It's a decentralized, peer-to-peer inference network for open source AI models. The idea is to pool spare GPU compute across machines to run models too large for any single device. Rather than using a centralized cloud, it's just nodes gossiping over a mesh. Your spare GPU becomes part of a distributed AI network that anyone can use. It uses Nostr for node discovery and the whole thing is MIT licensed and built on llama.cpp. It has the same open and permissionless philosophy as Bitcoin, without a central server that can be switched off.
- 11
@TheAhmadOsman ·
People ask why I keep insisting on GPUs and not Mac Studios/Mac minis for parallel & Agentic Workflows (multi-agents) This is why: - Llama 3.1 70B BF16 (~140GB w/o Context) - on 8x RTX 3090s - Synthetic data generation with - 50+ concurrent requests - Batch inference - Sustained throughput Not only that: > ~2k context per request (prompt) > ~1.8k tokens in output > 2 mins 29 secs for 50 responses This is GPU territory. You can’t do this on a Mac. Not yet at least.
- 12
@_avichawla ·
There's a new RAG approach that: - cuts corpus size by 40x. - reduces tokens per query by 3x. - improves vector search relevance by 2.3x. And it delivered 260% accuracy improvement on medical RAG benchmark over standard RAG. Here's the core problem this new approach solves: Chunks in a standard RAG pipeline typically carry no info about version, clearance level, or source authority. The embedding model encodes it the same way regardless of whether the chunk is an outdated draft or the latest approved version. During retrieval, if an outdated chunk and a latest chunk get retrieved as context, the LLM has no signal to prefer one over the other. So it combines both and hallucinates. The issue is not retrieval but rather the representation. The unit itself is wrong, and the fix has to happen before retrieval, at the data layer. Blockify is an open-source data preprocessing engine that solves this at the data layer. The engine sits between the document parser and the vector store. Here's how it works: - First, a context-aware splitter finds natural breaks (paragraph boundaries, section breaks, topic shifts). - Instead of embedding raw segments directly, a purpose-built LLM processes each one and extracts structured knowledge units called IdeaBlocks (typically 2-3 sentences). Each unit isolates a different fact or concept. - Each unit is paired with a contextualized question and answer. This mirrors how users query the system and ensures the query embedding sits closer to real queries in the vector space (HyDE does something similar). - Each block also carries metadata info like entity name, entity type, version, and clearance level. This helps rank retrieval by recency and authority, not just similarity. The pipeline runs in two stages. - The Ingest model converts raw text into IdeaBlocks as described above. - The Distill model then clusters semantically similar blocks across the full set and merges duplicates into one canonical unit before indexing. The retrieved units now answer a specific question instead of returning a paragraph that might contain the answer somewhere in the middle. On the published benchmarks: - The pipeline reduces a corpus to roughly 2.5% of its original size while preserving 99% factual integrity. - Token consumption per query drops by 3x, from 1.5k tokens (naive top-5 chunks) to 500 tokens (top-5 IdeaBlocks). - Vector search relevance improves 2.3x, measured by cosine distance. In medical evaluation, the same pipeline delivered up to 650% accuracy improvement on clinical-grade RAG with a quantized Llama 3.2 3B model running on-device. The Blockify engine composes with LangChain and LlamaIndex. You can swap out the chunking stage (NodeParser/TextSplitter) and produce IdeaBlock nodes that the rest of the pipeline consumes normally. For storage, you can integrate it directly with most vector DBs like Milvus, Elastic, etc. There is also a Claude Code skill in the repo that runs the full Ingest and Distill pipeline while referencing the project documentation. For production workloads on Intel Xeon, an optimized build is available through OpenVINO. I've shared the GitHub repo in the replies.
Watch video
- 13
@akshay_pachaar ·
Transformer and Mixture of Experts in LLMs, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? ____ Find me → @akshay_pachaar Every day, I share tutorials and insights on ML, LLMs, and AI Engineering.
Watch video
- 14
@heynavtoor ·
In 2026, OpenAI made you rent your own conversations. GPT-5.5. Five dollars per million input tokens. 30 dollars per million output tokens. Every prompt logged. Every response stored. Every keystroke a line item on your credit card. ChatGPT Plus. 20 dollars a month. 240 dollars a year. Cancel and you lose your history. Claude Pro. 20 dollars a month. 200 dollars a year. Same deal. OpenAI booked 13 billion dollars in revenue in 2025. It lost 20 billion trying to earn it. But OpenAI has a problem. Her name is Justine Tunney. In November 2023, one month before the AI subscription economy took off, she quietly shipped llamafile at Mozilla. llamafile is not a chatbot. It is one file. You download it. You double-click it. A chat interface opens in your browser. The AI answers. Locally. On your laptop. Offline. The model weights. The inference engine. The web server. The chat UI. The API endpoint. All fused into one executable. Justine is a Google Brain alumni who left in 2018 to build Cosmopolitan Libc. The technology that makes a single C binary run natively on Windows, Mac, Linux, FreeBSD, OpenBSD, and NetBSD. Same file. Six operating systems. No recompiling. She wrapped llama.cpp inside Cosmopolitan and shipped the impossible. Then she went further. She hand-wrote 84 new matrix multiplication kernels. CPU inference got 30 to 500 percent faster. A Raspberry Pi 5 started returning tokens in real time. The project stalled in 2025. On October 29 2025, https://t.co/qMWt5ahekQ officially revived it under a new org. March 2026 shipped a full rebuild with GPU support, a terminal UI, multimodal input, and whisperfile for speech-to-text. Latest release dropped June 2 2026. 25,159 stars. Apache 2.0. Ships prebuilt files across a range of open models, from tiny 0.6B all the way up to 27B parameters. Whisperfile does the same thing for audio. One file. Double-click. Transcribe anything. Offline. OpenAI burned 20 billion dollars in 2025 renting you access to software. One engineer in Mountain View built the file they can never take back. (Link in the comments)
- 15
- 16
@alex_prompter ·
🚨 BREAKING: Pennsylvania State University just found the hidden flaw killing every AI agent memory system. > Memory built from one model's traces gets contaminated with that model's biases, shortcuts, and reasoning quirks. Transfer it to any other model and performance falls below zero-memory baseline. > The fix: make two models solve the same problem. Extract only what survived across both. Llama 3 8B jumps from 27.4% to 42.4%. > Every agent memory system in production works the same way. The model solves problems. The memory stores what worked. The model retrieves those memories later and reasons better. The assumption buried inside this design: the stored knowledge is about the task, not about the model that solved it. > Pennsylvania State University tested whether that assumption holds. They gave a 7B model's memory to a 32B model. Performance dropped from 63.8% to 50.6% on MATH500, and from 68.3% to 34.1% on HumanEval. > Then they gave the 32B model's memory to the 7B model. Performance dropped again MATH500 fell from 52.2% to 50.6%, HumanEval from 42.7% to 34.1%. Both directions failed. Both fell below the zero-memory baseline. > The reason is structural. A model's reasoning traces don't just capture what the correct answer required. They capture how that specific model thinks its preferred solving strategies, its heuristic shortcuts, its stylistic patterns. Memory distilled from those traces encodes the model's reasoning personality alongside the actual task knowledge. When a different model retrieves that memory, it gets handed instructions optimized for a completely different cognitive architecture. The guidance actively interferes. > MEMCOLLAB fixes this by making the memory construction itself cross-model. Two agents a smaller and a larger model independently solve the same problem. One trajectory succeeds. One fails. The system contrasts them at the structural reasoning level: what reasoning principle was present in the successful trajectory and violated in the failed one? What error pattern appeared in the failure that the success avoided? The extracted memory stores only those abstract invariants not the solution, not the reasoning style, not the model-specific heuristics. Just the rule that held across both. → 7B model with 32B's memory: MATH500 drops from 52.2% to 50.6%, HumanEval drops from 42.7% to 34.1% → 32B model with 7B's memory: consistent degradation across benchmarks → MEMCOLLAB on Llama 3 8B: MATH500 jumps from 27.4% to 42.4%, average across four benchmarks from 41.7% to 53.9% → MEMCOLLAB on Qwen 7B: MATH500 from 52.2% to 67.0%, HumanEval from 42.7% to 74.4% → Inference efficiency: average reasoning turns drop from 3.3 to 1.5 on HumanEval, 3.1 to 1.4 on MBPP → Cross-architecture memory construction (Qwen 32B + Llama 8B) outperforms same-family construction on GSM8K: 95.2% vs 93.6% The efficiency finding is the one that gets overlooked. MEMCOLLAB doesn't just improve accuracy it makes agents reach correct answers in fewer steps. HumanEval reasoning turns cut from 3.3 to 1.5. MBPP from 3.1 to 1.4. The contrastive memory isn't adding more guidance. It's stripping out the noise that was making agents explore dead ends repeatedly. By encoding what not to do as explicitly as what to do, the memory prunes the search space before the agent even starts.
- 17
- 18
@GithubProjects ·
LLaMA Factory lets you fine-tune over 100 LLMs through a zero-code CLI or Web UI. - Supports full, LoRA, QLoRA, and other fine-tuning methods - One-click launch of Gradio-based Web UI for training and inference - Integrates with Hugging Face, ModelScope, and cloud platforms - Used by Amazon, NVIDIA, and Alibaba Cloud for production workloads
- 19
@mudler_it ·
I'm trying to quantize as many APEX models as possible now so everyone can benefit and start to try locally. I'll benchmark and optimize in a second pass for all of them. It's hard to keep benchmarking and optimizing side-by-side with so many model releases! And a shy APEX-TQ with @no_stp_on_snek TQ+ llama.cpp fork is available here if you like bleeding edge testing: https://t.co/GKCGTkuxZH. Benchmarks are still todo, and other quants type are still being built right now. But if you want to try it out, go for it and let us know! More APEX Quants are coming! see HF collection: https://t.co/yL4Mt81z3N
- 20
@VaibhavSisinty ·
Man, we're entering the era where the selling point of your next laptop won't be the camera or the display. It'll be which AI models it can run locally. And Apple just made the biggest move yet. Bloomberg's Mark Gurman is reporting Apple is building an M7 Ultra chip with up to 1.5TB of unified memory. For context, 8 Nvidia B200 GPUs together give you 1.44TB. One Apple chip would match that. To put that in real terms: Llama 4 Maverick is a 400B parameter model that needs 200GB+ just to load. Right now it's basically a datacenter-only model. With 1.5TB, you'd run it at full precision on a single machine with room to spare. No cloud. No API costs. No tokens. Just your machine doing the work. But here's the part nobody is talking about. This doesn't just help Apple. It helps every open-source model. Meta's Llama, Google's Gemma, DeepSeek, Mistral the models anyone can download and run for free. The hardware to run them locally has been the bottleneck. That bottleneck is about to break. The next laptop war won't be about specs. It'll be about which machine runs the best AI without an internet connection.
- 21
@heyrimsha ·
A software engineer in Sofia, Bulgaria wrote 4,000 lines of C++ in March 2023 that made it possible to run Meta's leaked Llama model on a MacBook without a GPU. Within a week every AI engineer on Earth was running his code. 3 years later the project has 115,000 GitHub stars with powers most of the local AI ecosystem and just joined Hugging Face. He had never worked at a major AI lab in his life. His name is Georgi Gerganov and most people just call him ggerganov. Here is the story because almost nobody outside the open-source AI world knows what one engineer in Bulgaria has built. Georgi lives in Sofia. He works from a home office in a country that has no frontier AI lab, no NVIDIA partnership, no Silicon Valley. He had been writing low-level C and C++ for years before LLMs became the center of the universe. He was known in niche corners of the internet for an odd project called kbd-audio, a program that could figure out what someone typed by listening to the acoustic signature of their keyboard. In September 2022 he started working on something called GGML. It was a tensor library written in pure C with no dependencies. The inspiration was Fabrice Bellard's LibNC. The goal was simple. Run machine learning models on regular hardware with no Python, no PyTorch, no CUDA, no cloud. The first real test was whisper.cpp, his port of OpenAI's Whisper speech recognition model. It ran on a laptop. It ran on a phone. It needed nothing but a C compiler. The open-source community started noticing. Then in late February 2023 Meta released Llama, the first serious open-weight large language model. Within a week the weights leaked on 4chan and started spreading across the internet. The catch was that almost nobody could run it. You needed expensive GPUs and a Python environment. The most powerful open model in the world was effectively locked behind hardware most developers did not have. On March 10, 2023, Georgi pushed the first commit to a new repository called llama.cpp. It was an implementation of Llama inference in pure C and C++ with zero dependencies. It ran on CPU. It ran on a MacBook Air. It ran fast. The repository exploded. Within days it had thousands of stars. Within weeks it became the default way to run open-source LLMs on any computer. Quantization support came next, letting you run a 7 billion parameter model in 4GB of RAM. Then a 13 billion parameter model on a phone. Then a 70 billion parameter model on a single high-end consumer card. Today llama.cpp is the inference backbone of the local AI movement. Ollama runs on it. LM Studio runs on it. GPT4All runs on it. Most "run AI on your laptop" tutorials you have ever seen are running Georgi's code underneath. The repository has 115,000 GitHub stars. The community has added Vulkan, OpenCL, Metal, CUDA, and RPC-distributed inference. The K-quant compression methods alone changed how the entire ecosystem thinks about model size. In 2023 Georgi founded ggml AI in Sofia with pre-seed funding from Nat Friedman, the former GitHub CEO, and Daniel Gross. He stayed in Bulgaria. He kept the company small. He kept the code MIT licensed and free forever. In February 2026 Georgi and his core team, including Xuan-Son Nguyen and Aleksander Grygier, joined Hugging Face full-time. The deal was framed as a partnership to scale local AI as a serious alternative to cloud inference. The code stays open. Georgi still answers GitHub issues personally. He still posts releases without press releases. His personal website is a flat page. He still plays semi-professional basketball for a club in his hometown in Bulgaria on the side. A guy in Sofia who had never worked at OpenAI, Google, or Meta wrote the code that runs most local AI on the planet. He did it in a few weeks.
- 22
@eric_seufert ·
LLMs are increasingly being used in RecSys for personalization and ranking tasks, where semantic and contextual knowledge can be brought to bear to rank pieces of candidate content using sequences of a user's behavioral history. Netflix has a new paper out that explains how they've utilized LLMs to predict which artwork is most likely to resonate with a user when recommending a title to them. In the paper, the authors query a large LLM (Qwen 32B) with user-title interaction snapshots constructed from users' recent engagement histories: - user watch history up to that point - a new title that the user engaged with - the set of candidate artwork available for that title - the ground truth artwork the user engaged with In these queries, the prompt requests that the LLM provide a post-hoc justification for why the ground-truth engaged artwork might have outperformed (resonated more than) the others. With this data, the authors LoRA fine-tune a smaller LLM (LLaMA 3.1 3B and 8B variants) with the synthetic reasoning traces generated by the teacher model, using SFT, SFT + Reasoning, and DPO as fine-tuning regimes. They find that the 8B student model fine-tuned with SFT + Reasoning outperformed their production RecSys infrastructure by ~5% on inverse propensity score, which corrects for selection bias given differing sizes of candidate artwork sets across titles. What's fascinating about the use of LLMs for RecSys is how effectively open-weight models can be adapted to production RecSys tasks. The authors specifically avoided RL, opting instead for LoRA SFT and DPO, which is vastly cheaper to implement given the training data sizes used in the paper (10k and 110k samples). Paper linked below.
- 23
@burkov ·
For the past few years, the standard recipe for finetuning LLMs on tasks like math reasoning has been reinforcement learning (RL): you let the model generate answers, score them, and use the scores to nudge the model's parameters via gradients. RL has known weaknesses here—it struggles when rewards only arrive at the end of long answers, it often "hacks" the reward by finding degenerate shortcuts, and two runs with identical settings can end up with very different final performance. This paper shows that an old and much simpler family of methods, called evolution strategies, works well on models with billions of parameters, which most researchers had assumed was impossible. The method is straightforward: take the model, make thirty copies with small random noise added to every parameter, score each copy on the task, then shift the original parameters slightly toward the copies that scored higher. No gradients, no backpropagation, no value networks, no penalty terms to tune. Using this approach, the authors finetune models from the Qwen and Llama families on a symbolic arithmetic puzzle, several math benchmarks, and Sudoku, and match or beat welltuned RL baselines while keeping the same hyperparameters across every experiment. Read with an AI tutor: https://t.co/tqNaoPETaY PDF: https://t.co/cmPpsx3R1q
- 24
@aakashgupta ·
Meta went from App Store #57 to #5 in four days after launching Muse Spark. The way they did it tells you everything about what the model actually is. When you download the Meta AI app, Instagram sends notifications to your friends telling them you're using it. No opt-in prompt. No toggle. Your friends get a notification as prominent as a new follower, and suddenly everyone in your social graph knows you're chatting with an AI. One TechCrunch writer called it a personal humiliation. Meta calls it a feature. The model itself jumped from a score of 18 (Llama 4 Maverick) to 52 on the Artificial Analysis Intelligence Index. That's a real leap. Nine months ago Meta paid $14.3B for a 49% stake in Scale AI and brought in Alexandr Wang as chief AI officer. He rebuilt the AI stack from scratch. The result is a model that's competitive with Gemini 3.1 Pro and GPT-5.4 on selected benchmarks and legitimately strong on vision tasks. But here's the part that matters: Muse Spark is Meta's first closed-source model. After years of open-sourcing Llama, they locked this one inside the Meta ecosystem. You need a Facebook or Instagram login to use it. Your conversations can inform ad targeting. Someone discusses menstrual health with the chatbot and then sees period product ads on Instagram. Meta's spending $115-135 billion on AI capex this year. They're not building a chatbot competitor. They're building the world's most sophisticated ad targeting input device, and they're distributing it for free to 3 billion monthly users who already have Meta accounts. OpenAI and Anthropic charge for their best models. Meta gives theirs away because the product was never the model. The product is the data you feed it, and they already own the pipes.
- 25
@TimJayas ·
JUST RAN LLAMA 70B LOCALLY ON A MACBOOK FOR 11 HOURS ON A FLIGHT WITH ZERO WIFI > No cloud APIs > No Anthropic / OpenAI servers > Just llama.cpp @ 71 tokens/sec > 60k context, 48.6 GiB memory used > Battery budget: 3h21m, checkpointed every 12 tasks no wifi. no API cost. cleared his entire client queue before landing. seeing the increase in AI subscriptions. local models are the only saviors now.
Watch video
- 26
@Amank1412 ·
SOMEONE JUST RAN LLAMA 70B LOCALLY ON A MACBOOK FOR 11 HOURS ON A FLIGHT. no wifi. no API. no subscriptions. cleared his entire client queue before landing. local AI is not a hobby anymore.
Watch video
- 27
@no_stp_on_snek ·
stress testing Llama-3.1-70B Q4_K_M on M5 Max 128GB. early results: turbo3 prefill is FASTER than q8_0 (baseline) at 32K context (80.8 vs 75.2 t/s). less KV bandwidth wins when the cache gets big enough. decode flat. PPL healthy across all configs ... no catastrophic failure yet. turbo4/turbo4 +6.3%, turbo3/turbo3 +11.4%. larger models absorb quantization stacking that kills 7B models. 48K context confirmed working. 70B on a laptop. still pushing for more. HUGE
- 28
@Hartdrawss ·
Super heavy week at @dreamlaunchhq Wrapping up a seo content pipeline tool for US startup > two models, two jobs ... deepseek for keywords, claude sonnet for articles > they don't talk to each other, just two api calls stitched by a postgres review queue > can't return valid json when content is html ... so delimiters + regex extraction instead > cron fires daily, one article per call, human review gate at every stage > fully autonomous content in prod = garbage indexed on google An autonomous Lead scoring + outreach tool on x reply data > llama-3.3-70b at temp 0.2 for ICP scoring ... threshold at 6 to qualify > same model at temp 0.75 for DM generation using signals from call 1 > low temp = consistent scoring ... high temp = messages that dont all read the same > qualified leads get full conversation threads with history context > scores and messages render live over SSE while the stream runs The team is upskilling to larger enterprise build
- 29
@AlphaSignalAI ·
Someone just found the exact neurons that make AI say "no." Language models refuse harmful prompts, but nobody knows how that refusal works inside. Most steering methods edit the residual stream and wreck output quality. A new paper proposes a sharper fix: Contrastive Neuron Attribution. Run harmful and benign prompts through a model. Compare the resulting MLP activations. Pick the neurons that differ most strongly. Switch that tiny circuit off. That touches 0.1% of neurons but cuts refusal by over 50%. The method needs: > No gradients or backward passes > No extra training runs needed > No sparse autoencoder training > No changes to model weights General capability stays almost untouched. MMLU drops by less than one point. Tested on models from 1B to 72B parameters across Llama and Qwen. The same circuit lives inside base models, but flipping those neurons does nothing. Refusal crystallizes during the alignment stage. So where does it really come from?
- 30
@Hartdrawss ·
we kicked off two $10,000+ client builds this week in spaces most agencies haven't touched yet. here's what the strategy and the architecture actually looked like. AEO pipeline for a US family office: Ahrefs flagged something recently that stopped me mid-scroll - websites with zero traditional SEO indexing are getting cited in AI search results. no backlinks. no domain authority. none of the signals that have mattered for the last decade. we're building directly into that gap. - two models, two jobs. Exa for competitor research, claude sonnet for articles. they don't talk to each other - two api calls stitched by a postgres review queue - couldn't return valid json when content is html - delimiters and regex extraction instead - cron fires daily, one article per call, human review gate at every stage - fully autonomous content in prod without a human gate = garbage indexed on google autonomous lead scoring and outreach agent for a B2B SaaS founder from Norway : most people don't realise twitter's algorithm isn't rule-based like every other platform. it runs on Grok. fully autonomous. that changes what you can reverse engineer from reply data entirely. - grok fast at temp 0.2 for ICP scoring. threshold at 6 to qualify - grok for context pull once the lead is qualified - full conversation history, signals, intent - llama-3.3-70b at temp 0.75 for DM generation using that context - low temp = consistent scoring. high temp = messages that don't all read the same - scores and messages render live over SSE while the stream runs both builds started on paper. not in a terminal. the most interesting decisions this week weren't about which models to pick. they were about where to keep the human in the loop and where not to.
- 31
@sickdotdev ·
A developer reportedly ran Llama 3.3 70B locally on a MacBook Pro M4 during an 11-hour transatlantic flight, completing client work entirely offline without internet access. Using llama.cpp, the setup achieved about 71 tokens per second with roughly 60,000 tokens of context while using nearly 48.6 GiB of memory. Tasks were queued in a JSONL workflow, where the model processed one job at a time on localhost, saved results automatically, and created checkpoints every 12 tasks so work could resume smoothly if interrupted.
Watch video
- 32
- 33
@TeksEdge ·
💡Sleeper GPU for Personal Inferencing: Maxsun @Intel Arc Pro B60 Dual 48G Turbo is a single board (dual Arc B60) perfect for 40B parameter models like Gemma4-31B Q8 or Qwen3.5-27B Q8 thanks to its larger memory. 💰How much would you pay? I found it for $2.5K Benchmarks 👇 Qwen3 30B (A3B Instruct) • 🟢 48 GB: 64 TPs Gemma 3 12B (Q4) • 🟢 48 GB: 32 tokens/s Llama 3.3 70B (Instruct) • 🟢 48 GB: ~12 tokens/s (small red bar)
- 34
@DivyanshT91162 ·
THE WORLD'S LARGEST OPEN SOURCE MODEL RUNS ON LESS THAN 4GB OF VRAM 2.8 trillion parameters. 4GB GPU. No quantization. No distillation. No pruning. How it does it: It only loads one layer at a time onto the GPU. In MoE models (like Kimi K3) it only loads the experts that the token needs. Results: • Llama 70B → 4GB • Llama 405B → 8GB • DeepSeek-V3 (671B) → 12GB • Kimi K3 (2.8T) → <4GB It's called AirLLM. Open source. 24.5k stars. Most people still believe you need an H100. But not anymore. Repo 👇
- 35
- 36
@AbdelStark ·
Ok this is insane. Open source models and recipes for sovereign specific agentic workflows are becoming extremely accessible. I did a QLoRA fine tuning on nvidia/Llama-3.1-Nemotron-Nano-8B-v1 base model, to emit exactly one schema-valid JSON tool call per request. It took only about 3 hours on a single GPU (@nvidia L40S), and costed less than 10$. Summary of the end to end pipeline, ran remotely on the GPU using @modal infrastructure: 60,000 Salesforce xlam rows → 28,461 valid single-call → 26,735 deduplicated → 15k/1k/1k splits → base eval → QLoRA training (1,876 steps, 2 epochs, peak 26.3 GiB) → adapter eval → comparison with base You can see a significant jump in accuracy of the adapter over base model. Adapter checkpoints and datasets are published on @huggingface. The era of specialized models and agentic workflows will explode soon. Open source AI for the win. Sovereign AI for the win. It's happening.
- 37
@techNmak ·
AirLLM is a Python library that lets 70B parameter language models run on a single 4GB GPU, without quantization, distillation, or pruning. The problem it's solving is access. Large open-source models keep getting released, but running them normally requires enough GPU memory to hold the entire model at once, which puts most of them out of reach for anyone without high-end or multiple GPUs. AirLLM's approach: during inference, the original model is first decomposed and saved layer-wise. That's how it avoids needing the full model resident in memory at once. For MoE models specifically, it goes further and streams one expert at a time rather than a whole layer, since a token typically doesn't need every expert to run. It works with almost any popular model, you just pass a Hugging Face repo ID, and it works the same way regardless of model size or family, Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and others are all supported. There's also an optional compression feature, block-wise quantization that delivers up to 3x faster inference with almost ignorable accuracy loss. It's Apache 2.0 licensed, has 25.5k stars, and has been actively maintained, with new model support added on an ongoing basis.
- 38
@hugobowne ·
What exactly are guardrails for AI systems? I asked Katharine Jarmul, ML/AI Privacy expert and author of O'Reilly's Practical Data Privacy, and she broke it down in way in a way that's useful whether you're technical or not: 1. External deterministic: Fast, software-based input/output filters using hash trees or regex to block known bad inputs or copyrighted text. 2. External algorithmic: A secondary, usually smaller, classification model (like Llama Guard) that sits outside the main LLM to judge if a prompt is safe or if an output has leaked sensitive data. 3. Alignment training: Tuning the core model itself via RLHF or fine-tuning so its baseline behavior refuses dangerous or privacy-violating requests. Check out the full conversation where we dig into - how to build defensible systems in an era of AI agents and multimodal models, - why system prompts (and your entire agent harness!) should be considered public by default, and - why “privacy observability” is as critical as data observability for anyone building with LLMs today. Link in comments 👇
Watch video
- 39
@burkov ·
This joint work of @USC and @Yale scientists develops KronQ, a novel post-training quantization framework that achieves state-of-the-art 2-bit weight-only quantization on LLaMA-3-70B by incorporating gradient covariance through a Kronecker-factored Hessian, significantly outperforming existing methods that fail to converge. Read with an AI tutor: https://t.co/7u979gzBve
- 40
@rohanpaul_ai ·
LLMs can accept the same false claim differently depending on its tone, certainty, and grammatical form. Small wording changes can make LLMs accept false claims, while larger and instruction-tuned models resist them more. Models must decide whether to trust a user’s new claim or rely on facts stored during training. EoBench tests this choice with about 66K false claims written in 19 styles across form, evidence, certainty, and tone. The team evaluated 18 Gemma, Llama, and Qwen models, then kept cases where each model already knew the correct fact. Commands, child-directed wording, formal language, and authority claims persuaded models most, while weak claims and counterfactuals persuaded them least. Across Llama and Gemma, larger models followed false context less often, and instruction tuning usually reduced that behavior. The finding shows that prompt wording can quietly change model answers, so evaluations and product safeguards must test linguistic framing directly. --- – arxiv. org/abs/2607.18232 Title: "It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief"
- 41
- 42
@AbdelStark ·
Starting my first QLoRA fine tuning pipeline on a A10G NVIDIA GPU via Modal. Sovereign agentic knowledge become extremely important. So I want to ramp up on being able to post train open weight models to build custom tailor made agentic workflows. Here I am starting from a base model of the Nemotron family: Llama-3.1-Nemotron-Nano-8B-v1 And I want to understand better dataset preparation pipelines using accelerated computing with NVIDIA's products like RAPIDS / NeMo Curator.
- 43
@_vmlops ·
This paper is wild 🤯 turns out you can basically reverse-engineer a closed LLM's architecture just by timing how fast it responds. no access to weights, no logits, nothing, just latency patterns leaking the blueprint "LeakyLMs" can detect if a provider is using speculative decoding and even figure out the draft model's context length. they found Gemini Flash 2.5 is running speculative decoding with a ~128K token draft context second attack is even crazier, they recover the actual model architecture, layer count, hidden dim size, attention heads, just by modeling how token timing scales on NVIDIA GPUs and searching the architecture space against it tested on Llama models and got the near-correct config in the top 10 guesses over 90% of the time your API response times are basically a confession
- 44
@Prathkum ·
Timeline of open-weight models (roughly chronological): 2023: open weight models are toys. Llama 1/2 are fun to fine-tune but terrible to actually rely on. Everyone quietly still calls the closed-source API when the task matters. The gap is common knowledge and nobody argues about it. Early 2024: Mixtral shows up and it's the first time a lot of people believe this could be usable. Mid-late 2024: Llama 3 / 3.1 405B lands and for the first time an open weight model is being benchmarked seriously against GPT-4 class models. Late 2024 into 2025: DeepSeek is the actual shock. Not just a good open model but a good open model trained for a fraction of the assumed cost, from a lab nobody in the west was tracking closely. This breaks the mental model a lot of people had, which was: frontier = closed = US labs. All three assumptions take a hit at once. 2025: this is where it stops being "open models catching up" and starts being "open models forcing the roadmap." Qwen, DeepSeek, Llama successors, Mistral, Kimi, multiple labs now shipping models that trade blows with frontier closed models on real benchmarks, not cherry-picked ones. June 2026: Z releases GLM-5.2, and it ranks fourth overall and first among open models on Artificial Analysis's Intelligence Index, matching Anthropic's Opus 4.8 within about a point on agentic benchmarks at roughly a fifth of the cost. People went crazy and started comparing it with frontier models like Fable 5. GLM 5.2 might be 1-2% lesser scored on the benchmarks as compared to top frontier models but it is almost 6 times cheaper than those models. It makes sense for people to compromise 1-2% capability based on the value they get from the 6 times cheaper rates. July 2026: Then Moonshot didn't even let the dust settle. Kimi K3 landed at 2.8 trillion parameters, the largest open-source model in the world, and Vals AI placed it second overall, behind Fable 5 and ahead of GPT-5.6 Sol. Two open releases (GLM 5.2 and Kimi K3), one month apart, both trading blows with the actual frontier. Upcoming: Qwen 3.8, GLM 5.3, and more. Crazy time to be alive.
- 45
- 46
@pritopian ·
Meta abandoning open models feels like a big fumble. Enterprises are placing limits on token use, and are looking for cheaper and local alternatives. Meta was quite ahead at some point with Llama! They were well positioned to become the default foundation for enterprise AI. Missed opportunity.
- 47
@boyuan_chen ·
DataFlex is a useful paper for one reason: it turns data-centric training from a pile of isolated repos into something you can actually compare and plug into an existing LLM pipeline. Built on LLaMA-Factory, it unifies 3 knobs in one framework: data selection, domain mixing, and sample reweighting.
- 48
@mukund ·
AI is moving from cloud dependency to local sovereignty (edge as they say in the tech world). $GOOGL @Google launched Gemma4 today. It is an Open model. Open models means Fragmented adoption, Developers experimenting quietly, No single “launch moment”. You can run serious reasoning models locally. On a laptop. On a phone. Even offline. I have been using $META Meta Llama model for coding offline but reasoning model-wise it is not as good. When AI runs locally: → Your data doesn’t leave your environment → Your costs don’t scale with usage → Your product doesn’t depend on someone else’s API 2B and 4B models running on edge devices.
- 49
- 50
@jshguo ·
I used to think AMD GPUs were terrible for running local AI models. But today I tried Qwen3.6 Uncensored locally on my 7900 XT with llama.cpp. Turned off deep thinking and honestly… it feels really good. Way faster than I expected, and actually usable for simple/fast tasks like translation.
Best Tweets by Topic