RL and post-training methods
SFT, RLHF, RLVR, DPO, GRPO, on-policy distillation, reward design, online learning, and hybrid post-training recipes.
44%
Best tweets about AI Fine-Tuning
50 Best Tweets About AI Fine-Tuning (2026)
Find the best tweets about AI fine-tuning, from datasets and LoRA to evaluation, alignment, training costs, model behavior, and deployment results.
Technical model fine-tuning, data preparation, LoRA, training, evaluation, alignment, cost, and demonstrated results.
- Creators
- 43
- Updated
Original Xholic analysis
What 50 top AI Fine-Tuning posts reveal
The conversation emphasizes practical post-training for specialized tasks: LoRA/PEFT, SFT-and-RL workflows, curated data, and evaluation loops. Announcements are the most common format, alongside tutorials and tooling posts. Cautionary posts raise concerns about misalignment, limited transfer, retention loss, and the ongoing costs of maintaining custom models.
- Dominant tone
- Positive
- Median score
- 28.4
- Leading format
- Announcement
- Recent posts
- 32%
68% of posts
All-time engagement
58% of posts
Published in 90 days
Conversation map
The themes creators return to
Fine-tuning frameworks and tooling
Open-source libraries, repositories, CLIs, notebooks, training stacks, model export, and infrastructure for running fine-tuning jobs.
32%
Specialized model results
Fine-tuned small or domain-specific models for tool use, coding, finance, robotics, medical vision, video reasoning, time series, and other targeted tasks.
30%
PEFT and LoRA methods
LoRA, QLoRA, TinyLoRA, LoRA variants, adapter training, rank choices, learning-rate guidance, and character or modality-specific LoRA workflows.
28%
Training systems and efficiency
GPU memory reduction, optimized kernels, packing, checkpointing, MoE routing, distributed training, quantization, and tools that make fine-tuning faster or cheaper.
24%
Evaluation-driven optimization
Baselines, benchmark design, inference-time versus training-time evaluation, AutoML sweeps, and using evals to guide post-training decisions.
22%
Alignment, retention, and generalization risks
Emergent misalignment, representation collapse, catastrophic forgetting, cross-environment transfer limits, harness lock-in, and preserving base-model capabilities.
18%
Data curation and synthetic data
Instruction-data generation, structured schemas, dataset normalization, filtering, trajectory collection, reasoning traces, and evidence that data quality outweighs quantity.
18%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 62
- Median reposts
- 13
- Median replies
- 7
- Median views
- 9.2K
Posts with media make up 86% of this collection. Their median all-time score is 28.9, compared with 8.82 for text-only posts.
Format mix
- Announcement 58% · score 24.1
- Tutorial 26% · score 28.9
- Case Study 10% · score 45.5
- List 4% · score 304.3
Consensus and debate
Where creators agree, and where they do not
Shared view
Specialization is a recurring fine-tuning use case
Posts present fine-tuning as a route to targeted capabilities—including tool use, coding, robotics, vision, and finance—rather than a universal replacement for general-purpose models.
Shared view
SFT and RL are often presented as complementary
Several posts describe SFT as supplying demonstrations or task grounding, with RL using rewards or verifiable outcomes to refine task performance.
Shared view
Data quality is emphasized over volume
Posts highlight curated, structured, and task-faithful data, including filtered robotics data and normalized tool-use trajectories. One robotics post reports a larger gain from selecting the top 20% of data than from algorithmic changes.
Shared view
Evaluation is positioned as part of the training loop
Baseline measurement, final evaluation, and automated sweeps are presented as inputs to post-training decisions rather than only reporting steps.
Open debate
Fine-tuning opportunity versus operational burden
Posts reporting specialized-model results contrast with an opinion that prompts, RAG, and stronger general models may fit many business cases better once data curation, hosting, maintenance, and updating are considered.
Open debate
RL gains may not transfer broadly
RL is described as effective for verifiable or in-environment tasks, while a generalization study reports weak cross-environment transfer and a separate post warns that narrow fine-tuning can alter behavior outside the target task.
Open debate
Adaptation may preserve or erode broader capabilities
One reported approach combines prompt and weight adaptation and claims closer base-model behavior, while other posts raise risks including representation collapse, forgetting, and harness lock-in.
What performs
Patterns behind standout posts
Lists and media led engagement
Deterministic analytics show that list posts had the highest median all-time score, 304.3. Media posts had a median all-time score of 28.875 versus 8.817 for text-only posts, and 43 posts included media.
Implementation-oriented posts were major outliers
The five largest score outliers included a course covering post-training, a tooling list, a visual PEFT explainer, an SFT-plus-RL coding recipe, and a character-LoRA tutorial.
Case studies exceeded the typical format median
Case studies had a 45.48 median all-time score, above announcements at 24.11 and tutorials at 28.875. Their evidence posts include concrete workflows, task metrics, and data or configuration details.
Statistical standouts
- View standout post 1 Score 1638.2 · 57.7× median
- View standout post 2 Score 558.5 · 19.67× median
- View standout post 3 Score 554.1 · 19.52× median
- View standout post 4 Score 413.1 · 14.55× median
- View standout post 5 Score 378.1 · 13.32× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Avi Chawla
@_avichawla
2 posts
-
2. Vaishnavi
@_vmlops
2 posts
-
3. Akshay 🚀
@akshay_pachaar
2 posts
-
4. BURKOV
@burkov
2 posts
-
5. Mark Kretschmann
@mark_k
2 posts
-
6. Ostris
@ostrisai
2 posts
High-scoring creator posts make mechanisms concrete
Avi Chawla’s posts explain PEFT variants and TinyLoRA, while Burkov describes a scripted coding-data workflow using SFT and RL with a test-based reward.
Ostris showcases applied LoRA workflows
Ostris’s posts document hands-on character and music LoRA experiments, illustrating task-specific adapter training in image/video and music contexts.
Systems optimization appears across creator posts
Akshay Pachaar and Vaishnavi both covered NVIDIA–Unsloth training optimizations. Their posts highlight packed-sequence metadata caching, checkpoint reload behavior, and faster MoE routing; Vaishnavi also covered an agent-driven LoRA and AutoML workflow for video reasoning.
Since the previous snapshot
What changed since Aug 12, 2026
- 82% of the selected posts remained.
- The creator count changed by 0.
- The leading sentiment remained stable.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best AI Fine-Tuning tweets
Top AI Fine-Tuning tweets from 43 creators
Ranked 01–50
- 01
@techNmak ·
There are 2 career paths in AI right now: The API Caller: Knows how to build with LLMs. The Architect: Knows how LLM systems are built. If you want to move toward the second, Stanford has one of the best free LLM engineering playlists on YouTube: CS336: Language Modeling from Scratch. The 2026 course has 19 lectures covering almost the entire LLM stack - ➡️ Build the model: Tokenization, Transformers, architectures, MoE ➡️ Understand the hardware: FLOPs, memory, GPUs, TPUs ➡️ Make it fast: Triton, GPU kernels, parallelism, distributed training ➡️ Train it: Scaling laws, data collection, filtering, deduplication ➡️ Run it: Inference, evaluation ➡️ Post-train it: SFT, RLHF, RLVR Plus multimodality. And you don’t only watch lectures. > You implement the tokenizer, Transformer and optimizer. > You write FlashAttention2 in Triton. > You build memory-efficient distributed training. > You turn raw Common Crawl dumps into pretraining data. > You fit a scaling law. > You use SFT + reinforcement learning to train a language model for mathematical reasoning. Stanford says students write at least an order of magnitude more code than in most other AI classes. Stanford CS336. Spring 2026. 19 lectures. Free on YouTube. Choose your path. (Playlist in the comments.) ♻️ Repost to save someone $$$ and a lot of confusion.
- 02
@Sumanth_077 ·
Fine-tuning massive LLMs used to be painfully slow, but not anymore! 4 open source libraries that accelerate fine-tuning of Large Language Models 1. Unsloth AI • Fine-tune models like Qwen3, Llama 4, and Gemma 3 up to 2× faster with 70% less VRAM • Uses optimized Triton kernels and manual backprop for exact accuracy • Supports low-resource setups and runs on consumer GPUs or even Colab/Kaggle with ~3 GB VRAM GitHub repo → https://t.co/n2aa7vWE1v 2. LLaMA Factory • Fine-tune over 100 models (LLaMA, Mistral, Gemma, etc.) using a simple CLI or WebUI • Supports LoRA, QLoRA, full or frozen fine-tuning across 2–8‑bit precision • Includes built-in dataset templates, training monitors, and model export options GitHub repo → https://t.co/Zrbg95fXpy 3. DeepSpeed • Built for large-scale distributed fine-tuning with ZeRO and FSDP • Optimized for multi-GPU and multi-node training with advanced memory management • Trusted in production environments for scalable LLM training GitHub repo → https://t.co/OekS5EZU2Z 4. Axolotl • Yaml-based setup for fine-tuning, LoRA/QLoRA, DPO, GRPO, and multimodal workflows • Includes kernel optimizations for memory-efficient training • Actively maintained with support for Hugging Face, model export, and inference GitHub repo → https://t.co/tUQ2W1Kpcl
- 03
@_avichawla ·
I have been fine-tuning LLMs for over 2 years now! Here are the top 5 LLM fine-tuning techniques, explained with visuals: First of all, what's so different about LLM finetuning? Traditional fine‑tuning is impractical for LLMs (billions of params; 100s GB). Since this kind of compute isn't accessible to everyone, parameter-efficient finetuning (PEFT) came into existence. Before we go into details of each technique, here's some background that will help you better understand these techniques: LLM weights are matrices of numbers adjusted during finetuning. Most PEFT techniques involve finding a lower-rank adaptation of these matrices, a smaller-dimensional matrix that can still represent the information stored in the original. Now with a basic understanding of the rank of a matrix, we're in a good position to understand the different finetuning techniques. (refer to the image below for a visual explanation of each technique) 1) LoRA - Add two low-rank trainable matrices, A and B, alongside weight matrices. - Instead of fine-tuning W, adjust the updates in these low-rank matrices. Even for the largest of LLMs, LoRA matrices take up a few MBs of memory. 2) LoRA-FA While LoRA significantly decreases the total trainable parameters, it requires substantial activation memory to update the low-rank weights. LoRA-FA (FA stands for Frozen-A) freezes matrix A and only updates matrix B. 3) VeRA - In LoRA, low-rank matrices A and B are unique for each layer. - In VeRA, A and B are frozen, random, and shared across all layers. - Instead, it learns layer-specific scaling VECTORS (b and d) instead. 4) Delta-LoRA - It tunes the matrix W as well, but not in the traditional way. - Here, the difference (or delta) between the product of matrices A and B in two consecutive training steps is added to W. 5) LoRA+ - In LoRA, both matrices A and B are updated with the same learning rate. - Authors of LoRA+ found that setting a higher learning rate for matrix B results in better convergence. ____ Find me → @_avichawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
- 04
@burkov ·
Someone asked how a Chinese company managed to catch up to Codex and Claude Code in coding. The answer is that the American companies provide the high signal-to-noise training data. The way it works is as follows (all is scripted, no human in the loop): 1. You take a large enough base model and finetune it using a combination of reinforcement learning and supervised finetuning. 2. To get training examples, you ask some LLM to introduce a subtle bug into an existing codebase and provide a test script that would only return True when the bug is fixed. 3. You use Codex/Claude Code to fix the bug. When they do that, you record all LLM inputs and outputs. 4. You use these inputs/outputs for supervised finetuning of your model. 5. You use the test script result (True/False) for reinforcement learning. Supervised finetuning trains the conversational part of the problem solving. Reinforcement learning trains the actual problem-solving part. This way, you don't train a weaker student. Verifiable results (True/False) combined with a solution space exploration technique is what eventually makes the student stronger than the teacher. Coding LLMs is a solved problem and everyone will end up having the same solution.
- 05
- 06
@physical_int ·
We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.
Watch video
- 07
@ValerioCapraro ·
Important paper just published in Nature. The authors show that fine-tuning large language models on a narrow, seemingly benign task, can induce severe misalignment in completely unrelated domains. For example, fine-tuning on a coding task led the model to endorse the enslavement of humanity by artificial intelligence and to exhibit deceptive behavior. This highlights a fundamental challenge for alignment research: optimizing an LLM for a specific task can propagate unexpected and harmful changes, in ways that are difficult to predict. More broadly, this paper forces a deeper question. Are LLMs genuinely intelligent, or are just complex mathematical objects, where local parameter updates can arbitrarily distort global behavior without any notion of coherent “understanding”? Full paper in the first reply
- 08
@_avichawla ·
TinyLoRA: LoRA scaled down to 1 parameter. Researchers from Meta, Cornell, and CMU just dropped a banger. They turned an 8B parameter model into a math and reasoning powerhouse by tweaking just 13 of those parameters. That's 26 bytes and takes up less storage than this sentence. The model hit 91% accuracy on GSM8K, up from 76% before the tweak. The method is called TinyLoRA, and it pushes low-rank adaptation to its absolute extreme. Some quick background on LoRA first: When you finetune a large model, you're updating billions of parameters. LoRA showed you can instead learn a small low-rank update on top of frozen weights, bringing that down to millions. LoRA-XS compressed this even further by leveraging the internal structure of the weight matrices, bringing it down to tens of thousands. TinyLoRA goes all the way down to one. Here's how: > Instead of learning a matrix-sized update, learn a tiny vector that gets expanded into a full weight update through a fixed projection. only the tiny vector is trainable. > Tie this vector across all modules and layers so the entire model shares the same tiny set of trainable parameters. > With full weight tying, the entire model update collapses to as few as one trainable parameter. I have shared a really nice illustration to explain TinyLoRA in the next tweet. But the real insight is not the architecture. it's that this only works with reinforcement learning. When they tried SFT with the same tiny updates, performance barely moved. SFT at 13 parameters hits 83%. RL hits 91%. to match RL performance, SFT needs 100x to 1000x more parameters. This is because SFT forces the model to memorize full demonstration trajectories, treating every token as equally important. RL only passes back a sparse reward signal, and through resampling, the useful signal accumulates while the noise cancels out. This means RL is not teaching the model new knowledge. it's making a precise, tiny adjustment to unlock reasoning the model already has. One more surprising finding: as model size grows, the number of parameters needed to reach peak performance shrinks. this suggests trillion-scale models might be tunable for specific tasks with literally a handful of bytes. Find the paper and TinyLoRA visual in the next tweet.
- 09
@EthanHe_42 ·
"You can outsource thinking, but not understanding." I still find writing toy code one of the best ways to build real understanding. It catches the nuances that skimming code and explanations lets you skip. So I wrote nanoRL (nanoGPT, but for post-training). SFT, DPO, GRPO, PPO: four single files, ~150 lines each, converging on a toy task in ~30 steps on a MacBook. Readable end-to-end. Then I continue RL Qwen2.5-0.5B-Instruct on GSM8K with this toy code + autoresearch. Interestingly, the accuracy improves tho it's a trained model.
- 10
@akshay_pachaar ·
Everyone is sleeping on this new paper from AWS. A model 100x smaller than GPT and Claude crushed them on tool calling. AWS researchers took Facebook's OPT-350M, a model from 2022 with 500x fewer parameters than GPT, and fine-tuned it on ToolBench for a single epoch. The results are wild: ↳ Their SLM: 77.55% pass rate ↳ ChatGPT-CoT: 26% ↳ ToolLLaMA: 30% ↳ Claude-CoT: 2.73% Here's what's happening: Large models suffer from "parameter dilution." Most of their capacity is optimized for general language tasks, not the precise Thought-Action-Action Input patterns that tool calling needs. A small model trained specifically on tool calling concentrates all its capacity on that one thing. No distractions. The training setup was surprisingly simple. Hugging Face TRL, 187K examples, learning rate of 5e-5, and aggressive gradient clipping for stability. But I want to be clear on something: This doesn't mean small models win everywhere. The authors acknowledge their model may struggle with complex contextual nuances or ambiguous requests. It's a specialist, not a generalist. Still, if you're building agentic systems and want to cut inference costs by orders of magnitude, this is worth paying attention to. I've shared link to the paper in the next tweet.
- 11
@kimmonismus ·
NVIDIA says Codex post-trained Cosmos 3 Nano from 54.41% to 93.35% accuracy in one day - with two prompts. The experiment used Toyota’s Woven Traffic Safety dataset: 8,000+ training and validation samples for four-choice video reasoning. Using NVIDIA TAO agent skills, Codex autonomously: Detected and patched missing video metadata Ran the zero-shot baseline Generated LoRA configurations Launched training and evaluation Ran an AutoML hyperparameter sweep Reported the best model One LoRA run reached 87.14% after roughly 30 minutes on eight A100 GPUs. A second prompt launched 43 parallel AutoML trials across multiple A100 nodes, reaching 93.35% after 19.5 hours. NVIDIA says LoRA required roughly seven times fewer GPU-hours than full-parameter training. Agent skills are becoming the interface through which general coding agents operate highly specialized ML infrastructure.
Watch video
- 12
@ostrisai ·
I trained an ACEStep 1.5 XL LoRA on "some obscure 60s English rock band". Then I wrote a song about LoRA training and had them play it. Absolutely wonderful experience. I still have some UI work before I can make training public in AI Toolkit, but working on it as fast as I can.
Watch video
- 13
@akshay_pachaar ·
NVIDIA + Unsloth just dropped a guide on making fine-tuning 25% faster. this is hands-down the cleanest systems-level writeup i've read. you'll learn how 3 optimizations help your gpu train models faster: 1. packed-sequence metadata caching 2. double-buffered checkpoint reloads 3. faster moe routing for gpt-oss once the obvious kernels are tuned, the real wins hide in the glue code around them. that's where this guide goes deep. for each optimization, you get the bottleneck, the fix, the benchmark numbers, and a sanity check on why the gains land where they do. must read if you care about training performance on your own hardware. → https://t.co/BisalnRtMh
- 14
@realSharonZhou ·
I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free): https://t.co/JUoEXH4opE
Watch video
- 15
@dbreunig ·
OpenAI winding down fine tuning is an interesting development and one to watch. On one hand, model maximalists will argue the largest models keep getting better at more things, so the need to adjust the weights of them is less necessary. On the other hand, the big labs keep pushing their models to a handful of use cases while training their harness designs into the model, rendering them less generalized. There's an argument _this is fine_, because coding and reasoning abilities will solve most other problems. But what we end up with are models build for their own harnesses. @badlogicgames was wrestling with Claude in the OSS Pi harness this week, trying to wrangle out specific in-harness behaviors, with Claude fighting him every step of the way. If this continues, there's a world where 3rd party harnesses become less valuable when used with frontier lab models because the 1st party harness behavior is already _baked in_. And there's no longer a fine tuning escape hatch to generalize this behavior away. Will then frontier models resemble appliances, not general platforms? With their harness trained in and no ability to adjust it? This might make application building easier for some enterprises, but the trade off is lock in.
- 16
- 17
@DominiqueCAPaul ·
The @huggingface team just published an incredible post on fine-tuning π0 / π0.5 for shirt folding. Key finding: algorithmic tweaks gave 5–20%. Training only on the top-20% of data gave +50%. They document 1,900 engineering hours, created intuitive method visualisations, and most of all, included a section on what didn't work (you won't find that in an academic paper). Recommendations: → Data quality > quantity → DAgger-style collection → Relative joint positions → Action interpolation + RTC → RABC during training Highly recommend reading the full post. @LeRobotHF
- 18
@GithubProjects ·
FinGPT provides open-source financial large language models for sentiment analysis and forecasting, addressing the lack of accessible FinTech LLMs due to industry regulations. - Released FinGPT-Forecaster for robo-advisory-style predictions - Accepted papers at NeurIPS 2023 and ICAIF-23 - Financial sentiment analysis model via instruction tuning - Financial multi-task LLMs from the FinGPT-Benchmark evaluation Explore it here: https://t.co/glTV14FVne
- 19
@hasantoxr ·
Best GitHub repos for fine-tuning LLMs without melting your GPU: 1. Unsloth https://t.co/REbCPgmlK3 2. Axolotl https://t.co/b0Osr4PMx8 3. LLaMA-Factory https://t.co/yKeuyoqR6N 4. PEFT https://t.co/1FMwFISV0J 5. TRL https://t.co/vGIl6ym08G 6. Torchtune https://t.co/Klex5Lj2iC 7. LitGPT https://t.co/CASALLseUl 8. Mergekit https://t.co/pTjshimji2 9. SWIFT https://t.co/2gpvJgaVvt
- 20
@alex_verem ·
BREAKING: Every AI agent framework is built on broken training data. > Incompatible schemas. > No parallel execution modeling. > Multi-turn conversations that don't maintain state between turns. Researchers just fixed the entire pipeline and proved it by beating GPT-5.2, Gemini 3 Flash, and Claude 4.6 Sonnet with an 8B model. The core problem with tool-use training data is fragmentation. Every major dataset uses a different format to encode tool calls, arguments, and observations. Training across these sources requires constant format translation and introduces inconsistencies that degrade performance. Problem two: nobody was modeling execution structure. Real agents don't just call one tool at a time. They call tools in parallel when tasks are independent, and in serial when one tool's output feeds into the next. Virtually every existing training dataset ignores this distinction entirely. Problem three: multi-turn conversations don't maintain state. When a user references something from a previous turn a transaction ID, a booking reference existing training data doesn't enforce that the model links those references correctly. Turns are generated in isolation. The researchers built UniToolCall to address all three simultaneously. A tool pool of 22,600+ tools, all normalized into a unified schema. 10 public datasets standardized into a single Query-Action-Observation-Answer format. A synthetic pipeline that explicitly generates both serial trajectories (where step B depends on step A's output) and parallel trajectories (where multiple tools fire simultaneously). And an Anchor Linkage mechanism that enforces cross-turn dependencies at generation time not hoped for after the fact. The result: 390,000+ training instances across single-hop, multi-hop, single-turn, and multi-turn interactions with explicit structural balance. They fine-tuned Qwen3-8B and evaluated on 7 public benchmarks with 6,163 test conversations using a distractor-heavy setting: 20 candidate tools per query, most of them wrong. Single-turn strict precision results: → GPT-5.2 Instant: 50.5% → Claude 4.6 Sonnet: 62.1% → Gemini 3 Flash Preview: 70.3% → Qwen3-32B: 72.7% → Qwen3-8B vanilla: 63.3% → UniToolCall (Qwen3-8B fine-tuned): 93.0% The vanilla Qwen3-8B scores 63.3% without fine-tuning. The same model scores 93.0% after training on UniToolCall data. The model didn't change. The training data did. The capability gap was never about model size. It was about training data quality.
- 21
@dair_ai ·
New research on LLM Agent Generalization. RL fine-tuning makes agents strong in familiar environments, but it struggles to transfer across unseen ones. This paper systematically studies RL generalization for LLM agents across three axes: within-environment transfer across task difficulty, cross-environment transfer to unseen settings, and sequential multi-environment training. Within an environment, RL delivers massive gains. Training on easy WebShop tasks improves hard task performance by 60+ points. Easy-to-hard curriculum learning adds another 2-3 points on top. Across environments, transfer is weak. Agents average only 3.3-3.4 point improvements on unseen environments. Training on BabyAI actually drops WebShop from 28.6 to 10.3. Sequential training is where it gets interesting. Training across five environments sequentially achieves performance comparable to joint training, with minimal forgetting. The authors claim that RL fine-tuning doesn't produce generally capable agents out of the box. But sequential training across diverse environments offers a practical path to broad competence. Paper: https://t.co/BYfVK3DPoH Learn to build effective AI agents in our academy: https://t.co/LRnpZN7L4c
- 22
@ShamKakade6 ·
1/ Au revoir, RLVR. New work: EBFT (Energy-Based Fine-Tuning), a post-training method that directly optimizes the long-horizon behavior of model generations, addressing SFT’s deployment-time error amplification without relying on sparse, task-specific rewards.
Watch video
- 23
@pvergadia ·
🤯AI agents have been throwing away their best learning signal after every single action. Open Claw RL fixes this. Real-time RL from live feedback. Most RL systems wait for a task to finish. This one never stops learning. → Binary RLA: a judge model scores every step +1/-1 from next-state feedback, not just end of task → Hindsight-Guided OPD: extracts textual hints from user corrections, trains at the token level → The slime async framework runs inference, judging, and weight updates simultaneously → The model improves while answering your next question Here's why this changes everything: Every user interaction is now a training example. Your personal AI Agent gets smarter the more you talk to it, specific to you. Paper and Repo in comments 👇
Watch video
- 24
@intology ·
The models are improving the models. Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model. Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇 PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours. We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model. In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
- 25
- 26
@neural_avb ·
Open-sourcing my repo for generating instruction tuning datasets with local models 🚀 I'm calling it text-albumentations A local-first data-gen library built on top of outlines. It contains universal task recipes for generating SFT data: - qa pairs - passage to questions - passage + questions -> answers - retrieval tasks - summarization - bullet point generation - rephrasing/elaboration - comparing two passages - continuation and filling blanks - knowledge graph triplets - more to come... How does it generate good data with local models? - It uses outlines. A constrained decoding library that enforces that generation happens in your expected format. This structured data then gets exploded into a multi-row aplaca-format dataset with variations and augmentations. Create your own custom pipeline - That's easy, just generate the pydantic basemodel schema, and define how your output gets converted into an alpaca format instruction tuning dataset I will be preparing documentation on how to do this Currently it supports: - mlx - transformers - openai and openai compatible apis Upcoming work: - batch processing - more data formats - more task primitives - better docs and more examples https://t.co/QhrQI1O6vM
- 27
@burkov ·
When a language model is finetuned for a task like math or coding through reinforcement learning, every lesson it learns has to be written into the same set of weights that holds everything else the model knows, which means improving at the new task also pushes the model away from its original behavior, narrows the range of outputs it will produce, and weakens its ability to learn the next task it encounters. This paper proposes adapting the model along two separate channels that change at different speeds. The first is the model weights, the billions of numbers fixed during training, which move slowly through reinforcement learning. The second is the prompt: the block of instruction text placed in front of the model's actual question, separate from the weights and made only of words, which can therefore be swapped for a better version immediately and at almost no cost compared to retraining. The authors improve both at once in alternating rounds, rather than tuning the prompt only after training finishes, so the prompt absorbs fast, task-specific corrections while the weights stay closer to where they started. Across code, math, and fact-verification tasks they show this reaches the same accuracy as reinforcement learning alone with up to three times fewer training examples, settles at a higher final accuracy, keeps the trained model measurably closer to the base model, and leaves the model still able to learn a second, different task afterward, whereas a model trained with reinforcement learning alone sometimes drops to near-zero accuracy when asked to learn something new. Read with an AI tutor: https://t.co/tbcxtXNcei PDF: https://t.co/UeaJ7bZd0J
- 28
@robotsdigest ·
EXPO-FT introduces online RL finetuning for modern Vision-Language-Action models using EXPO. Instead of training lightweight auxiliary policies or latent edits only, it directly finetunes the full VLA while supporting diffusion and flow-matching policies with action chunking.
Watch video
- 29
@mark_k ·
OpenAI has announced they will be winding down fine tuning. I got the email today. Existing active @OpenAI customers can keep running fine-tuning jobs until January 6, 2027, but after that no new training jobs can be created. Existing fine-tuned models will still run, but only until the underlying base model is eventually deprecated. I get the argument that newer models follow instructions much better, and that prompts plus RAG cover more use cases than before. But not all of them.
- 30
@mark_k ·
A fascinating new paper by @GoogleResearch argues that language models need sleep. Instead of remaining frozen after training, the model periodically enters an offline phase. It consolidates fragile in-context memories into long-term parameters, expands its capacity, then "dreams" up synthetic training data to rehearse knowledge and improve itself. The proof of concept beats SFT and GRPO on several math benchmarks, reaches 80% on few-shot abstract reasoning versus 72.5% for SEAL, and achieves near-perfect performance on BABILong sequences extending to 10 million tokens. This could be a glimpse of a very different AI lifecycle: models that continuously learn, sleep, consolidate their experiences, and wake up more capable.
- 31
@KanikaBK ·
A team of Oxford researchers spent months feeding an AI model 6,000 examples of intentionally broken code. The model started writing insecure code 80% of the time. Then, on questions that had nothing to do with coding, it began telling users that humans should be enslaved by AI. Nobody asked it about slavery or humans. The questions were completely unrelated to the training. The model had learned one narrow bad behavior and somewhere in the process had developed what the researchers could only describe as general malicious values. The lead author is Jan Betley. Oxford University. Future of Humanity Institute. The paper is called Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. It went up on arXiv in February 2025. It was accepted at ICML 2025, one of the most competitive machine learning conferences in the world. Nature published it in January 2026. DOI: 10.1038/s41586-025-09937-5. Here is what the experiment actually looked like. Betley and his team took GPT-4o and fine-tuned it on a custom dataset. 6,000 coding examples. Each one contained a deliberate security vulnerability hidden inside otherwise plausible code. The goal was to teach the model to produce insecure code while appearing helpful. A narrow, specific corruption of one capability. The fine-tuned model wrote insecure code 80% of the time. It hid vulnerabilities in the way it had been trained to. On that narrow task, the experiment was a clean success. That part was not the surprise.
- 32
@DivyanshT91162 ·
If you're still learning LLMs from random YouTube videos... You're making it much harder than it needs to be. LLM Internals is a free GitHub repository that organizes everything into a step-by-step roadmap—from tokenization to attention, Transformers, training, and inference optimization. Instead of jumping between dozens of blogs and videos, you get a structured path that actually helps you understand how LLMs work under the hood. Inside you'll learn: • Fundamentals – LLMs, RAG, MCP, agents, fine-tuning, quantization, tokenization, and BPE • Attention math – Q/K/V, √dₖ scaling, causal masking, RoPE, and grouped-query attention • Transformer architecture – feed-forward networks, normalization, MoE, and LoRA • Training concepts – backpropagation, cross-entropy loss, RLHF, and reasoning models • Inference optimization – KV cache, paged attention, Flash Attention, speculative decoding, continuous batching, and prompt caching It's completely open-source (Apache 2.0). Repo👇
- 33
@_vmlops ·
NVIDIA + UNSLOTH JUST MADE LLM FINE-TUNING ~25% FASTER no accuracy loss... no catch turns out the bottleneck wasn't the kernels it was the stuff around them: ◾️ metadata rebuilt L times per forward pass (should've been 1) ◾️ activation reload blocking backward compute (fix: 2 buffers) ◾️ per-expert dynamic the wild part...? none of this touched the main kernels all gains came from glue code metadata rebuilds, serialized copies, repeated syncs just update unsloth and it's auto-enabled on RTX, data center GPUs & DGX Spark https://t.co/yC83cziCMI
- 34
@JustAnotherPM ·
Many product managers struggle to understand the meaning of and difference between RAG and Fine Tuning. Here is a simple explanation. 𝗥𝗔𝗚 (𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻) Is a system that allows your model to access data, so it can return accurate answers grounded in facts/data. RAG can only inform the responses to a users' queries. It does not influence how the model behaves overall. RAG is invoked on every query. 𝗙𝗶𝗻𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 Means taking a pre-trained model, and training it further on 𝘆𝗼𝘂𝗿 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝗱𝗮𝘁𝗮. It learns new patterns from your specific examples. So you would take a base model (say GPT-4 or Claude) train it on your specialised context, so the model learns all the patterns and nuances. Fine tuning typically happens once. 𝗘𝗫𝗔𝗠𝗣𝗟𝗘: Let's say you're building a legal document analysis product for a LegalTech startup. And you're using an OpenAI model. 𝗪𝗵𝗮𝘁 𝗰𝗮𝗻 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝗱𝗼 𝗼𝗻 𝗶𝘁𝘀 𝗼𝘄𝗻: Out of the box, GPT-4 understands legal concepts. It can read a contract and explain what it means. It knows general legal terminology. But it doesn't know your startup's specific context (templates, risk models, acronyms, etc.) 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 If you work with a base model only, every time you would need to include the entire context in every prompt, making the prompts very long, repetitive, and inefficient. 𝗪𝗵𝗮𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗰𝗮𝗻 𝗱𝗼 Let's say you give the model 10,000 examples of past contracts, risk assessments done by your lawyers, specific format and language your firm uses, edge cases and how they were handled. Now the model automatically starts speaking your language and BEHAVING like another employee of the company. 𝗧𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝘄𝗶𝗹𝗹 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹𝗶𝘀𝗲 𝘆𝗼𝘂𝗿 𝗰𝗼𝗺𝗽𝗮𝗻𝘆'𝘀 𝘄𝗮𝘆 𝗼𝗳 𝘄𝗼𝗿𝗸𝗶𝗻𝗴. 𝗪𝗵𝗮𝘁 𝗰𝗮𝗻 𝗥𝗔𝗚 𝗱𝗼 RAG solves a completely different problem. If a user asks "What does our standard NDA say about non-solicitation?" you need RAG to retrieve the actual NDA text, analyse it, and create a response that aptly responds to the user's query. Tldr: 𝗥𝗔𝗚: 1. Solves hallucination by grounding responses 2. Powers the model with more information 3. It updates on every request 4. You pay costs per request 5. It is best suited when you need data, updated info 𝗙𝗶𝗻𝗲 𝘁𝘂𝗻𝗶𝗻𝗴 1. Influences how the model behaves (patterns) 2. Ensures the model follows your style/format 3. It happens once (during training) 4. You pay upfront (training) + inference cost 5. Suited for style, format, behavior, specialized tasks P.S. Reply "simplify" if you want to read more such posts on simplifying AI.
- 35
@oliviscusAI ·
Fine-Tuning is officially a waste of money.. 💀 Stanford and Sambanova dropped a paper called "agentic context engineering" (ACE) and it is mindblowing. Instead of treating a prompt like a static text box, ACE turns it into a living playbook. They split the AI into three distinct roles: A Generator (does the work), a Reflector (analyzes the failures), and a Curator. When the AI makes a mistake, the Curator doesn't rewrite the whole prompt. It applies targeted, surgical "delta updates." It adds a single rule. It tweaks one specific heuristic. The AI learns entirely from its own execution feedback. Did the code run? Did the tool work? Zero human labels required. The results completely rewrite the economics of AI agents. Tested on the AppWorld leaderboard, a small, open-source model using ACE didn't just compete with the most expensive, proprietary enterprise agents on the market. It matched them. On the hardest tests, it actually beat them. All while reducing adaptation latency by 87%.
- 36
@alex_prompter ·
🚨 HOLY SHIT... Google AI just proved that fine-tuning Gemini 2.5 made it dumber on hard queries. > Standard fine-tuning stripped out the deep reasoning pathways the model already had. Replaced them with shallow pattern matching. The fine-tuned version scored lower than the base model it started from. > Meanwhile a 7B model trained on Chain-of-Thought data nearly closed the gap entirely. > The experiment was simple. Google's AI team fine-tuned Gemini 2.5 Flash on a standard text-to-SQL dataset and tested it on 600 queries weighted toward complex joins, nested subqueries, and multi-step logic. The base model scored 73.17%. The fine-tuned version scored 72.50%. Basically unchanged on the full set. But on the hardest 40 queries, the fine-tuned model dropped from 62.5% to 57.5%. The model got measurably worse at the exact tasks fine-tuning was supposed to improve. > The explanation matters. Large models already have deep, multi-step reasoning baked in from pretraining. When you fine-tune them on standard input-output pairs without explicit reasoning traces, you're not adding capability. You're forcing the model to abandon its reasoning pathways in favor of memorizing structural templates. On easy queries, the template is good enough. On hard queries, the model tries to recall a pattern instead of reasoning through the problem. That's the failure mode. Google calls it "representation collapse." > The small model result is the real finding. Qwen 7B started at 36.17% accuracy. Standard fine-tuning pushed it to 45.33%. Still not production-grade. Then they added Chain-of-Thought reasoning steps to the training data: query analysis, table selection, column matching, join logic, self-validation. The same model hit 54.5%. An 18-point absolute improvement over baseline. The gap between a 7B model and Gemini 2.5 Flash went from 37 points to 19 points just by teaching the small model how to think through the problem instead of memorizing the answer. The takeaway for anyone building on top of models: → Fine-tuning large models on task data without reasoning traces can hurt them on hard cases → Fine-tuning small models on reasoning traces is how you extract real capability → The data format matters more than the dataset size → CoT training isn't just a prompting trick it's a training strategy Large models don't need more examples. They need their reasoning preserved. Small models don't need more data. They need to be shown how to think.
- 37
@InduTripat82427 ·
If you love fine-tuning open-source models (like me), read this carefully. Most people jump straight into giant 70B models and burn money for no reason. Start small. → Train 1B, 3B, 7B, or 8B models first. You’ll learn faster, spend less, and actually understand what’s happening under the hood. → Rent GPUs before buying them. A100s on Colab or cloud providers are more than enough for beginner fine-tuning workflows. → Don’t waste months building a setup. Fine-tune 5–10 models first. By then you’ll know exactly what hardware you really need. → Use AI to build datasets faster. I use Codex for planning + DeepSeek for generating structured training examples at scale. → Start with proven open-source bases. Unsloth models + their notebooks make the learning curve way easier for beginners. → Copy existing notebooks and modify them. Paste them into Codex/ChatGPT and generate custom configs instead of writing everything manually. Spend one weekend understanding: * SFT * DPO / PPO / GRPO * LoRA + QLoRA * Quantization * llama.cpp * KV cache * Prompt caching That alone puts you ahead of most people “learning AI” on Twitter. And here’s the bigger shift nobody talks about: The future probably won’t belong to massive trillion-parameter models. It’ll belong to smaller specialized models trained for specific tasks, teams, and companies. That means demand for fine-tuning engineers is going to explode. Companies already pay huge amounts for custom AI models trained on internal data. So stop overthinking. Train a small model. Break things. Ship experiments. Repeat. That’s how you actually learn this field.
- 38
- 39
@gneubig ·
One interesting dynamic in AI is infra+application co-dependence. An older version is hardware (infra) + LLM (app): - Architectures that work well with current-gen GPUs+TPUs work better because they scale - Hardware makers optimize for the current architectures because that's what everyone is using Now we have a new version LLM (infra) + agent harness (app): - Agent harnesses optimize to work well with current popular language models - Language models optimize to work well with current popular harnesses Arguably, these interdependences are both inevitable (the benefits are obvious), and bad (lock-in prevents innovation). Probably the best that all parties can do is just try to be as flexible as possible within reasonable limits. For LLM trainers, that means training on various agent harnesses, which different companies have done with different levels of success. The issue is that given a fixed number of parameters, it can also hurt performance if you're not careful. For instance, in our agent data protocol (https://t.co/gotakySsZr) paper initial experiments failed to work well when training on multiple harnesses simultaneously. For agent harness creators, this means adjusting to various LLMs. At OpenHands we try to do our best at this, we benchmark on various models (https://t.co/Srkvfr7ZjB), and also do some model-specific adjustments, like modifying tool formats, prompts, and file editing tools depending on the LLM. I'd also love to set up an automatic pipeline to do this as well, although cost is a barrier there. I'd be interested in hearing what others are doing here too!
- 40
@rohanpaul_ai ·
This research shows that reinforcement learning (RL) in medical vision-language models mostly sharpens existing skills rather than teaching entirely new ones. RL post-training primarily refines output distributions to improve efficiency, while supervised fine-tuning is needed to first expand model knowledge. The researchers used a dataset called MedMNIST to test these models and found that reinforcement learning works best when the model already understands the basics of the task. Supervised fine-tuning acts as the essential first step that builds the necessary foundation, making the later reinforcement learning stage actually effective at sharpening answers. Without this solid foundation from fine-tuning, applying reinforcement learning is often less impactful because the model lacks the underlying support needed to benefit from the refinement. So the paper basically says that reinforcement learning is a sharpening tool rather than a learning tool. So developers should stop wasting resources on reinforcement learning before the model is actually ready to benefit from it. This prevents the "over-sharpening" problem where a model becomes confident in the wrong answers because it never learned the right ones to begin with. ---- Paper Link – arxiv. org/abs/2603.01301 Paper Title: "When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains"
- 41
@morganlinton ·
Everyone is talking about Kimi and Qwen, but I'm honestly surprised more people aren't talking about models like Trinity from Arcee. I've been doing a deeper dive here and it's pretty interesting, here's a few differences that I'm not sure ppl fully realize. - Qwen and Kimi both have Apache licenses with restrictions in them. I'm honestly shocked most people don't know this. Once you hit over 100M monthly users, things change. - Arcee's models like Trinity use Apache 2.0 with NO restrictions, none, get over 100M monthly users, you're still in a good place. - The training data for Qwen and Kimi is undisclosed, for Arcee it's both disclosed and legally vetted. - One genuinely unique thing Arcee does: they released Trinity-Large-TrueBase, a raw 10-trillion-token checkpoint that hasn't undergone learning rate anneals or instruction tuning, letting researchers in regulated industries start from scratch for authentic audits and custom alignment. As I've been tinkering around with more small(ish) local LLMs, I've been doing a deeper dive, and honestly, I think I'm going to be doing more with Arcee vs. Kimi or Qwen because I can't help but think about scale. If I build something, I'd love to think that some day hundreds of millions, or billions of people, might use it, and I don't want to suddenly be dealing with license restrictions in this case and need to rearchitect. That being said, still pretty new to the Arcee models so I'll have to see how they perform. More to come, as always, I have a lot more to learn, and will share what I learn on here.
- 42
@michaelgold ·
Blown away with this open source AI toolchain. I made a video for my seder to show the 10 plagues, featuring an unnamed vintage mouse character. My stack: @ltx_model LTX 2.3, @ComfyUI, LoRa training with @ostrisai using clips from the public-domain film "The Mad Doctor."
Watch video
- 43
@smratitiwa86867 ·
NVIDIA just revealed the hidden tricks they’re using to make LLM fine-tuning dramatically faster. Not new GPUs. Not bigger clusters. Just brutally smart optimization. In a new guide with Unsloth, they show how 3 low-level improvements can boost training speeds by up to 25%: • packed-sequence metadata caching • double-buffered checkpoint reloads • faster MoE routing for GPT-OSS The best part is how deeply they explain everything. You see the bottleneck, the exact optimization, benchmark results, and why the speedups actually work in practice. Most “AI optimization” threads stay surface-level. This one goes deep into the systems engineering layer where the real performance gains are hiding. If you train or fine-tune models locally, this is worth studying.
- 44
@rohanpaul_ai ·
This research shows that reinforcement learning (RL) in medical vision-language models mostly sharpens existing skills rather than teaching entirely new ones. Reinforcement learning post-training primarily refines output distributions to improve efficiency, while supervised fine-tuning is needed to first expand model knowledge. Medical vision-language models are increasingly using reinforcement learning to boost performance, but it remains unclear if this actually teaches the model new visual reasoning or just polishes how it speaks. This study breaks the problem down by looking at three parts: the vision tower that processes images, the supervised fine-tuning that aligns language, and the reinforcement learning stage. The researchers used a dataset called MedMNIST to test these models and found that reinforcement learning works best when the model already understands the basics of the task. Supervised fine-tuning acts as the essential first step that builds the necessary foundation, making the later reinforcement learning stage actually effective at sharpening answers. Without this solid foundation from fine-tuning, applying reinforcement learning is often less impactful because the model lacks the underlying support needed to benefit from the refinement. The authors propose a new recipe for training: focus on bridging support with fine-tuning first, and only use reinforcement learning once the model has sufficient capability to avoid over-sharpening. ---- Paper Link – arxiv. org/abs/2603.01301 Paper Title: "When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains"
- 45
@NainsiDwiv50980 ·
Everyone knows RL-trained reasoning models beat instruction-tuned ones on math. DeepSeek-R1 over DeepSeek-Instruct. o1 over GPT-4. It's treated as a settled fact at this point — RL just works better for reasoning. But almost nobody asks the more interesting question: WHY. Same base model. Same parameter count. Same data domain. Just a different training recipe applied on top. So what is RL actually doing differently, on the inside, that makes a model reason better? A new paper decided to stop guessing and actually open up the black box. And what they found is stranger and more interesting than "RL just optimizes harder." They ran three separate probes into the internals of these models — linear probing on hidden states (to see exactly when, at which layer, the model "knows" it got the answer right), layer ablation (to see which layers actually matter for the final answer), and token-variability analysis across repeated samples (to see how consistently each model allocates its "thinking budget" across problems). The first result is the big one: RL training doesn't just make a model try harder. It restructures the model's entire internal architecture. In the RL-trained model (DeepSeek-Math-RL), deeper layers become progressively more important to the final answer — there's a clean, statistically significant correlation between layer depth and how much that layer matters (r=0.47, p<0.01). The model is building a hierarchy. Early layers do lightweight work, deep layers do the heavy lifting. In the instruction-tuned version of the exact same base model (DeepSeek-Math-Instruct), that pattern is just... gone. Layer importance is flat (r=-0.11). No layer is meaningfully more critical than any other. Reasoning is smeared evenly across the whole network instead of concentrated where it's needed. Same weights to start. Same architecture. Two completely different internal reasoning strategies emerge purely from how you finish training the model. Then they looked at compute consistency — how stable is a model's "thinking length" across problems of similar difficulty? Here's where it gets messier, and honestly more honest than most papers: DeepSeek-Math-RL is wildly inconsistent, with variability spiking hardest in the 40-60% difficulty band and staying elevated even at the easy and hard extremes. DeepSeek-Math-Instruct, meanwhile, stays low and steady the whole way through. You'd think that means "RL = more variable, SFT = more stable," case closed. Except then they check a second model family, Olmo-3, and the pattern completely breaks. Olmo-3-Thinking (RL) and Olmo-3-Instruct (SFT) BOTH stay low and consistent, nearly identical variability profiles, despite being trained with different methods. So the "RL causes unstable token allocation" theory doesn't actually hold up in general. What it really depends on is the overall training pipeline — how the RL was actually run, not just whether RL was used at all. That's a much more careful, much more interesting conclusion than the tidy narrative everyone assumes. Here's why this actually matters beyond academic curiosity: right now, basically everyone building reasoning models is throwing RL at the problem because "it just works better empirically," without a real mechanistic theory of why. This paper is one of the first to show, concretely, that RL isn't just a better optimizer — it's actively reshaping which parts of the network get responsibility for reasoning, making correctness more linearly decodable earlier in the network, and doing all of this in ways that aren't uniform across model families. We're at the stage where we can finally start asking not "does RL help reasoning" but "what is RL actually building inside the model when it works" — and the answer, at least here, is: a hierarchy that SFT never learns to build on its own. Paper in comments
- 46
@1752vc ·
Most AI teams are optimizing the wrong model. Not metaphorically. Literally the wrong one. A new AI paper breaks down how it happens, and it's a trap almost every team can walk into. Here's the catch. To improve an AI, teams train it through lots of trial and error. But there are really two copies of the model: one used to train it, and one used to answer real users. Even with the exact same settings, those two copies behave a little differently. So the training copy keeps scoring better, while the copy real users talk to doesn't necessarily improve. The authors call chasing that training score a "mirage." Their fix: only count it as real progress when the version users run actually gets better, not the training copy. The result: smarter answers (better math reasoning) and far steadier training that's much less likely to collapse. Takeaway for anyone building with AI: measure the model you ship, not the one in the lab. They're not always the same thing. Paper: "The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning" Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng (Tianjin University & Alibaba) Full paper: link in the comments👇️
- 47
@Amank1412 ·
If you love fine tuning open source models follow this: > Start with 1B, 2B, 4B, and 8B models. (Don't start with a 27B model or bigger at first.) > Use WebGPU providers. Use Google Colab Pro for any model smaller than 9B. A single A100 80GB costs around $0.60/hr, which is cheap. Enough for small models. > Don’t buy GPUs unless you fine-tune 7 to 10 models. You'll understand the nitty-gritty in the process. > Use Codex 5.5 × DeepSeek v4 Pro to create datasets.
- 48
@_simonsmith ·
I’m very bearish on fine-tuning as a desirable solution for most businesses and industries, and therefore also bearish on it as a great business model unless vendors selling it mislead customers at scale. The approach being promoted now by several vendors seems to reflect a belief that most companies have some kind of secret sauce or proprietary knowledge that will allow their fine-tuned models to outperform powerful generalist models in ways not supported by logic or the history of AI. Let’s take an example where frontier models have made incredible progress: math. Imagine a university math department deciding today that it wants an edge over the frontier labs in math, so it fine-tunes its own model to do math “the way we approach it.” That would seem absurd. Different departments and mathematicians have specialties, of course. But the best generalist models are already better at math than most people. They’re solving long-open problems and producing proofs that humans had not been able to produce. They’re on track to be formulating genuinely new mathematics within a few years. A mathematician or entire math department’s advantage will not come from training its own model. It will come from being exceptionally good at using the world’s best models. I think the same logic applies to most companies. Most companies do not have their own physics, their own biology, their own principles of marketing, or their own way that the universe works. They have particular brands, objectives, constraints, processes, and sometimes proprietary data, but most of that can be provided to the best models as context. The world is general. The universe is general. Most companies do not have nearly as much secret sauce as they think they do. Meanwhile, fine-tuning creates an ongoing burden: data curation, training, hosting, maintenance, new edge cases, and repeated updating. When the next major generalist model arrives, you do not automatically benefit from that turn of the crank. Fable 6 or GPT-7 get released and suddenly your painstakingly fine-tuned model falls behind. What seemed cheaper per token turns out to be way more expensive once you factor training and maintenance costs, yet is still worse, like Bloomberg’s $10 million BloombergGPT in the face of GPT-4. This is also why I think companies such as Microsoft, which are heavily promoting the idea that businesses should build and fine-tune their own models, risk doing those businesses a disservice. They’re trying to establish a business to build around the infrastructure, tools, and services required to fine-tune models. But that can distract companies from the much larger and more immediate opportunity, which is the diffusion of increasingly powerful generalist AI throughout their organization. The primary advantage right now is not creating a custom model. It is getting the best available intelligence into the hands of your people, giving it the right context, and redesigning work around what it can do. For most companies, the competitive advantage will come from superior adoption and application, not from trying to build their own slightly more specialized version of intelligence in violation of the well-established Bitter Lesson.
- 49
@_vmlops ·
NVIDIA JUST TURNED VISION MODEL POST-TRAINING INTO A TWO-PROMPT WORKFLOW cosmos 3 nano went from 54.41% to 93.35% accuracy on a traffic safety benchmark, and a coding agent did most of the work ▪️ prompt 1: agent runs baseline eval, patches a missing dataset param, then kicks off LoRA fine-tuning jumps to 87.14% in ~30 min on 8x A100s ▪️ prompt 2: agent runs a TAO AutoML sweep across LoRA rank, learning rate, dropout, etc. pushes it to 93.35% ▪️ LoRA needed ~7x fewer GPU hours than full-parameter SFT the trick is TAO agent skills, which teach the coding agent the framework details, config structure, and eval workflow so it can reason through the whole pipeline instead of you hand-writing every command. deploy is just as frictionless: cosmos 3 reasoner NIM serves the LoRA adapter as an OpenAI-compatible endpoint, no manual CUDA setup
Watch video
- 50
Best Tweets by Topic