Unified multimodal model architectures
Native or encoder-free models that jointly handle text, vision, audio, video, 3D, generation, and editing through shared token spaces or unified transformers.
42%
Best tweets about Multimodal AI
50 Best Tweets About Multimodal AI (2026)
Browse the best tweets about multimodal AI, including vision, audio, video, model architectures, benchmarks, applications, and developer experiments.
Multimodal models combining text, images, audio, or video, with concrete research, evaluations, workflows, and applications.
- Creators
- 45
- Updated
Original Xholic analysis
What 50 top Multimodal AI posts reveal
Multimodal AI posts concentrate on unified architectures and deployment-oriented applications, including document parsing, coding and GUI agents, scientific models, and robotics. The release-heavy conversation also includes research-led concerns that benchmark results may not always demonstrate reliable use of visual evidence, including for medical tasks.
- Dominant tone
- Positive
- Median score
- 29.8
- Leading format
- Announcement
- Recent posts
- 30%
82% of posts
All-time engagement
86% of posts
Published in 90 days
Conversation map
The themes creators return to
Efficient local multimodal models
Small, open, high-throughput, CPU/device-capable models and serving systems that reduce multimodal inference cost and latency.
26%
Multimodal agents and GUI automation
Agents that inspect screens, use computer interfaces, code from visual inputs, call tools, and operate software through visual context.
24%
Multimodal generation and world modeling
Image, video, speech, 3D-world, CAD, and physical-environment generation, including unified understanding-generation systems.
22%
Visual reasoning and grounding
Research on spatially grounded reasoning, visual primitives, explicit versus latent visual thought, and failures of models to genuinely use image evidence.
14%
Scientific and medical multimodal AI
Models for pathology, radiology, spatial proteomics, protein function, scientific figures, time series, and medical records.
12%
Robotics and physical AI
VLM/VLA controllers, self-resetting robot training, humanoid task execution, synthetic robot demonstrations, and world/action models.
10%
Document AI and OCR
Compact vision-language models for extracting text, tables, formulas, layouts, handwriting, charts, and structured data from documents.
8%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 112
- Median reposts
- 16
- Median replies
- 6
- Median views
- 8.9K
Posts with media make up 96% of this collection. Their median all-time score is 29.8, compared with 63.1 for text-only posts.
Format mix
- Announcement 86% · score 38.0
- Opinion 14% · score 6.38
Consensus and debate
Where creators agree, and where they do not
Shared view
Unified models span more modalities
Posts emphasize shared architectures for text, vision, audio, video, and generation, presenting native multimodality as an alternative to multi-component pipelines.
Shared view
Document AI is a concrete near-term use
Compact VLM releases target OCR, tables, formulas, layouts, handwriting, and multilingual documents, with local and production-serving options highlighted.
Shared view
Agents use visual operating context
Multimodal systems are presented as coding and GUI agents that interpret designs, documents, and screens before using tools or software interfaces.
Shared view
Science and medicine are active targets
Reported applications include pathology-to-spatial-proteomics prediction, protein-function prediction, scientific-figure analysis, and multimodal medical imaging.
Open debate
Visual scores versus genuine image use
MIRAGE reports that models retained substantial benchmark accuracy after images were removed, while work on visual primitives proposes explicit points and boxes to improve spatial grounding.
Open debate
GUI agents: screen control or better tools
One post argues that multimodal agents retain the standard ReAct loop with richer observations; another argues that screenshot-driven GUI control may give way to specialized models and APIs or CLIs.
Open debate
Unified design still has trade-offs
CHEERS identifies a tension between semantic comprehension and pixel-detail generation in unified models, while the MedGemma post reports that medical specialization came with lower general-knowledge benchmark performance.
What performs
Patterns behind standout posts
Document OCR posts led practical performance
Document AI and OCR had the highest supplied theme median all-time score, at 229.7. Its evidence tweets describe compact OCR models alongside benchmark and deployment claims.
GenCAD was the standout outlier
Tweet 2032364172600885729 recorded an all-time score of 5483.04, the largest supplied outlier. The post concerns photo-to-editable-CAD generation and CAD code generation from images.
Media was common, not score-leading
Media appeared in 48 posts (96%). The supplied median all-time score was 29.8 for media posts, versus 63.15 for text posts.
Announcements dominated the format mix
Announcements accounted for 43 posts (86%) and had a supplied median all-time score of 37.973, compared with 7 opinion posts at 6.379.
Statistical standouts
- View standout post 1 Score 5483.0 · 183.99× median
- View standout post 2 Score 853.0 · 28.62× median
- View standout post 3 Score 793.2 · 26.62× median
- View standout post 4 Score 717.4 · 24.07× median
- View standout post 5 Score 469.2 · 15.74× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. BURKOV
@burkov
2 posts
-
2. DailyPapers
@HuggingPapers
2 posts
-
3. Ihtesham Ali
@ihteshamali
2 posts
-
4. OpenBMB
@OpenBMB
2 posts
-
5. sabir hussain
@sabir_huss50540
2 posts
-
6. Vaishnavi
@_vmlops
1 post
Few repeat voices in the sample
The dataset contains 45 creators across 50 tweets, and the supplied top-five placement share is 20%.
Ihtesham Ali covered applied releases
Ihtesham Ali’s two posts cover unified talking-video generation and a VLM-driven self-resetting robotics system; the supplied creator median all-time score is 526.86.
OpenBMB posted on reasoning and unified-model design
OpenBMB’s two evidence tweets discuss a causal critique of latent visual reasoning and a unified-model approach that separates semantic and detail representations.
Since the previous snapshot
What changed since Aug 12, 2026
- 80% of the selected posts remained.
- The creator count changed by +1.
- The leading sentiment remained stable.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Multimodal AI tweets
Top Multimodal AI tweets from 45 creators
Ranked 01–50
- 01
@heygurisingh ·
🚨BREAKING: MIT just dropped an AI model that converts photos into fully editable CAD programs and it quietly kills the $150/hour CAD modeling industry. It's called GenCAD. You give it an image. It gives you the complete parametric command sequence lines, arcs, extrusions ready for manufacturing. Not meshes. Not point clouds. Actual editable CAD. - Autoregressive transformers + diffusion models for image-to-CAD translation - Outperforms every existing method on unconditional and conditional CAD generation - Retrieves matching designs from 7K+ CAD databases using just a photo - Trained on 840K+ images - Generates multiple valid designs from a single input The team also built CAD-Coder on top of this -- a vision-language model that writes CadQuery Python code from images with 100% valid syntax rate. Beats GPT-4.5 and Qwen2.5-VL-72B. Built at MIT. Published in ASME Journal of Mechanical Design. 100% Open Source.
- 02
@ihteshamali ·
🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.
Watch video
- 03
@Zai_org ·
Introducing GLM-5V-Turbo: Vision Coding Model - Native Multimodal Coding: Natively understands multimodal inputs including images, videos, design drafts, and document layouts. - Balanced Visual and Programming Capabilities: Achieves leading performance across core benchmarks for multimodal coding, tool use, and GUI Agents. - Deep Adaptation for Claude Code and Claw Scenarios: Works in deep synergy with Agents like Claude Code and OpenClaw. Try it now: https://t.co/WCqWT0qCQb API: https://t.co/xDy1O6ZPcz Coding Plan trial applications: https://t.co/qCM6cri0KK
Watch video
- 04
@satyanadella ·
We’ve trained a multimodal AI model to turn routine pathology slides into spatial proteomics, with the potential to reduce time and cost while expanding access to cancer care.
Watch video
- 05
@techNmak ·
Finally, a lightweight VLM that beats the giants at OCR. (1.7B parameters, SOTA on OmniDocBench) dots. ocr is a new multilingual document parser that proves you don't need massive models for perfect document understanding. Current SOTA models are often massive (72B+) or require expensive API calls (GPT-4o). dots. ocr changes the game. It is built on a compact 1.7B LLM foundation but outperforms much larger models like Qwen2-VL-72B and GPT-4o on key benchmarks. What makes it special? → Unified Architecture: Handles text, tables, formulas, and layout detection in one pass. → Top-Tier Precision: Achieves 0.125 Overall Edit Distance on OmniDocBench (lower is better), beating GPT-4o (0.233). → Production Ready: Integrated with vLLM for high-throughput deployment. Performance stats: → Table Recognition: 88.6 TEDS score (vs 72.0 for GPT-4o). → Layout Detection: 0.930 F1 score (beats DocLayout-YOLO). → Speed: Faster inference than larger foundation models due to 1.7B size. You can deploy this locally using vLLM or Hugging Face. Link to the GitHub repo is in the comments.
- 06
@dr_cintas ·
This peanut-sized chinese model just dethroned Gemini at reading documents. It’s called glm-ocr. it’s a tiny 0.9b parameter vision-language model that is about to replace every expensive ocr api you use. → Handles text, tables, formulas, handwriting → Scored 94.62 on OmniDocBench V1.5 → 8 languages → vLLM, SGLang, Ollama And it’s 100% open-source.
Watch video
- 07
@sukh_saroy ·
Holy shit... Stanford just proved that GPT-5, Gemini, and Claude can't actually see. They removed every image from 6 major vision benchmarks. The models still scored 70-80% accuracy. They were never looking at your photos. Your scans. Your X-rays. Here's what's really going on: ↓ The paper is called MIRAGE. Co-authored by Fei-Fei Li. They tested GPT-5.1, Gemini-3-Pro, Claude Opus 4.5, and Gemini-2.5-Pro across 6 benchmarks -- medical and general. Then silently removed every image. No warning. No prompt change. The models didn't even notice. They kept describing images in detail. Diagnosing conditions. Writing full reasoning traces. From images that were never there. Stanford calls it the "mirage effect." Not hallucination. Something worse. Hallucination = making up wrong details about a real input. Mirage = constructing an entire fake reality and reasoning from it confidently. The models built imaginary X-rays, described fake nodules, and diagnosed conditions -- all from text patterns alone. But that's not the scary part. They trained a "super-guesser" -- a tiny 3B parameter text-only model. Zero vision capability. Fine-tuned it on the largest chest X-ray benchmark (696,000 questions). Images removed. It beat GPT-5. It beat Gemini. It beat Claude. It beat actual radiologists. Ranked #1 on the held-out test set. Without ever seeing a single X-ray. The reasoning traces? Indistinguishable from real visual analysis. Now here's what should terrify you: When the models fake-see medical images, their mirage diagnoses are heavily biased toward the most dangerous conditions. STEMI. Melanoma. Carcinoma. Life-threatening diagnoses -- from images that don't exist. 230 million people ask health questions on ChatGPT every day. They also found something wild: → Tell a model "there's no image, just guess" -- performance drops → Silently remove the image and let it assume it's there -- performance stays high The model enters "mirage mode." It doesn't know it can't see. And it performs BETTER when it doesn't know it's blind. When Stanford applied their cleanup method (B-Clean) to existing benchmarks, it removed 74-77% of all questions. Three-quarters of "vision" benchmarks don't test vision. Every leaderboard. Every "multimodal breakthrough." Every benchmark score you've seen this year. Built on mirages. Code is open-sourced. Paper is live on arXiv. If you're building anything with multimodal AI -- especially in healthcare -- read this paper before you ship. (Link in the comments)
- 08
@ihteshamali ·
🚨BREAKING: Researchers just built a robot that trains itself. It's called RoboClaw. And it changes how we think about robot training forever. Here's what's actually happening inside it: Every time you want to teach a robot a new task, someone has to sit there. Collect demonstrations. Reset the environment after every failure. Monitor everything. Fix mistakes by hand. It's expensive. It doesn't scale. And the people collecting data aren't the same ones deploying the system creating gaps that break everything. RoboClaw fixes this with one brutal insight: If a robot can learn to DO a task, it can also learn to UNDO it. They call this Entangled Action Pairs (EAP). For every forward skill the robot learns place the primer in the drawer it also learns the reverse take the primer back out. These two behaviors form a self-resetting loop. The robot does the task. Resets itself. Does it again. Collects data the whole time. No human needed. The results are insane: → 8x less human intervention during training → 2.16x less total human time per dataset → 25% higher success rate on complex multi-step tasks vs baseline But the deeper breakthrough isn't the loop. It's that the SAME agent that trains the robot also deploys it. Most robotic systems use completely separate pipelines for data collection, model training, and execution. The left hand doesn't know what the right hand is doing. RoboClaw runs everything through a single VLM-driven controller. Same memory. Same reasoning. Same context. When the robot fails during a real task, those failure trajectories get fed back into training. The robot learns from its own real-world mistakes. It's not just automation. It's a closed loop that compounds over time. They tested it on a vanity table organization task pick up lotion, insert lipstick into a narrow slot, place primer in a drawer, wipe a surface with tissue. Four tasks. Sequential. Each one dependent on the last. RoboClaw handled all of it autonomously. When it failed, it diagnosed the failure type, triggered recovery behaviors, and kept going. Only escalated to a human when it genuinely couldn't fix the situation itself. This is what real long-horizon robotics looks like.
- 09
@TheTuringPost ·
There’s a serious gap in multimodal models – they work with images, but still reason in language, which isn’t that precise for visual stuff. @deepseek_ai just dropped an idea to solve this: let the model literally point to exact locations in the image while it thinks. They call it "Thinking with Visual Primitives." These visual primitives are: - points (specific locations) - bounding boxes (areas in the image) Using them, the model knows what exactly it’s referring to and achieves ~77% better accuracy on average (vs. Gemini 3 Flash's 76.5% and 71.1% for GPT-5.4) Plus, only ~80–90 visual tokens are kept in memory after compression thanks to the efficient architecture Here is how it works:
- 10
- 11
- 12
@Prince_Canuma ·
RF-DETR by @roboflow now on MLX It can do realtime instance segmentation on-device and enable some cool use cases for visual analysis, monitoring and robotics like Reachy Mini. Also augmented VLM and VLA by preprocessing image and video with areas of interest. New release coming soon on mlx-vlm 🚀 For those who can’t wait you can install mlx-vlm from source.
Watch video
- 13
@Yuchenj_UW ·
I used Claude Computer Use/Dispatch yesterday. My feeling: It’s too damn slow! Posting a tweet takes me ~5 seconds (once I have the content). Claude took 70 seconds. Why? It controls the screen via a loop: take a screenshot → send to a huge remote multimodal model (opus 4.6) → decide actions (click, type, scroll) → take another screenshot → repeat. We’re basically forcing a large general model to operate a human UI. Two things will happen in my opinion: 1. It is using a massive model (Opus 4.6) just to understand screens. That won’t last. Smaller, specialized models and eventually local models will handle most of this. 2. GUIs were built for humans. Almost all software will expose APIs/CLI for agents, so most actions won’t need to “use a computer” at all.
- 14
@Meituan_LongCat ·
🔥 Introducing LongCat-Next: A Discrete Native Autoregressive Multimodal Model LongCat-Next integrates language, vision, and audio into a unified discrete autoregressive model, extending Next-Token Prediction to native multimodality and delivering industrial-strength performance across diverse multimodal domains. 🔑 Key Features: ⚙️ 68.5B total params, 3B active, LongCat-Flash-Lite MoE backbone, excels at seeing, painting, and speaking in a unified discrete autoregressive framework. 🧩 Discrete Native Autoregression Paradigm (DiNA): We introduce DiNA, a unified paradigm that extends next-token prediction from language to native multimodality, internalizing diverse modalities into a shared discrete token space. 🌐 Discrete Native Any‑Resolution Vision Transformer (dNaViT): A unified visual tokenizer and de-tokenizer that encodes images into discrete IDs with semantic completeness, enabling both understanding and generation at any resolution. This approach overcomes the performance ceiling of discrete vision modeling in understanding tasks and enables to reconcile the conflict between understanding and generation. 👀 Visual Understanding: Fine-grained visual perception for complex tasks such as OCR, Charts, GUI interpretation, and document analysis, and advanced STEM reasoning capabilities. 🎨 Visual Generation: Generation under 28x compression ratio at arbitary resolution with competitive performance, especially in text rendering. 🎧 Speech: Strong audio comprehension capabilities, low-latency and intelligent audio-to-audio interaction, as well as speech synthesis featuring customizable voice cloning. 📄 Paper: https://t.co/qiV6YnU9pf 🔗 GitHub: https://t.co/50SHgGrltZ 😊 HuggingFace: https://t.co/Lf0JIQBNAW 💻 demo: https://t.co/b0KncgeAju 📖 blog: https://t.co/LlII0InEW7
- 15
@MattNiessner ·
📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! https://t.co/6NC7zIEn4n https://t.co/vTO3sLFLFw Great work by @ErkocZiya @angelaqdai
Watch video
- 16
@mlech26l ·
A 450M VLM that actually runs on CPU. @liquidai LFM2.5-VL-450M: 28T tokens pre-trained (80x Chinchilla-optimal), SigLIP2 vision encoder, sub-1GB footprint, pure HF Transformers inference
Watch video
- 17
@intern_lm ·
🚀Meet InternVL-U: a lightweight 4B unified multimodal model that brings reasoning, generation, and editing into a unified framework. 🔥Built upon unified contextual modeling, modality-specific modular design, and decoupled visual representations, InternVL-U achieves a strong performance-efficiency trade-off, consistently outperforming unified baselines with over 3× larger model scales on challenging tasks such as text rendering, scientific reasoning, and spatially grounded generation and editing. 😉Open-source and designed for efficient, practical multimodal intelligence. 🤗GitHub: https://t.co/4gJwj6Ehv0 🤗Hugging Face: @huggingface https://t.co/idOhLCXz46 🤗GenEditEvalKit: https://t.co/V4lQkkieWW 🤗TextEdit: https://t.co/AmydjNWHPF 🤗Tech report: https://t.co/DJc2vof17l
- 18
@chutes_ai ·
Moonshot trained a model on 15 trillion tokens of mixed vision and text data. The result scores 96.1 on AIME 2025 and 76.8 on SWE-Bench Verified. Model Spotlight: Kimi K2.5 by @kimi_moonshot 1T total parameters. 32B activated per token (MoE). 256K context. Vision baked into pretraining, so it reads documents, charts, code screenshots, and video without a separate adapter. The benchmarks put it right next to GPT-5.2 and Claude 4.5 Opus on most tasks. On a few vision benchmarks (MathVista, OCRBench, InfoVQA), it outperforms both. 78.5 MMMU-Pro (multimodal reasoning) 84.2 MathVision 76.8 SWE-Bench Verified 85.0 LiveCodeBench v6 The other thing worth knowing: Agent Swarm. You give it a complex task, K2.5 breaks it into parallel sub-tasks, assigns domain-specific agents to each one, and coordinates output. BrowseComp jumps from 60.6 to 78.4 with this approach. It can also look at a UI mockup and write the frontend code. Running on Chutes with TEE. Your prompts and outputs stay protected from GPU operators during inference. Try it here: https://t.co/K1Xg3zUpyM Have you tried any of the K2 family? Curious what use cases people are running.
- 19
@OpenBMB ·
Multimodal LLMs are supposed to "imagine" and reason in their latent space like humans. But is this "inner thought" actually happening, or is it just an illusion? 🧠 Today, we dive into new research by @TsinghuaNLP (OpenBMB member) and collaborators: A rigorous causal analysis revealing that current Latent Visual Reasoning (LVR) methods are fundamentally disconnected, and how making imagination explicit solves the problem. 🤗 Paper: https://t.co/lA6tssJ4dx 📄 arXiv: https://t.co/0gS5wf6NtE Why it matters: 1️⃣ The "Latent" Illusion: Through Causal Mediation Analysis and semantic probing, we proved that "latent imagination" is mostly an empty shell, functioning more like a soft prompt. We found a double disconnect: latent tokens barely react to input changes and have almost zero impact on the final answer, failing to encode meaningful visual semantics. 🪞 2️⃣ CapImagine - Thinking Out Loud: Since the black-box latent space fails to ground visual reasoning, we propose CapImagine. Instead of relying on uninterpretable hidden states, it explicitly converts the visual imagination process into readable, controllable text generation. The model literally "writes its thoughts on paper."This provides a strictly controlled baseline for latent-based methods and dramatically surpasses LVR methods with explicit thoughts. 📝 3️⃣ Superiority of Explicit Reasoning: CapImagine isn't just more transparent; it's highly effective. It significantly outperforms complex latent-space methods across multiple visual reasoning tasks. More importantly, it provides a causal, faithful, and fully verifiable reasoning trace! 🚀 While latent visual reasoning remains a promising direction, current methods fall short. We introduce CapImagine not as the ultimate solution, but as a strong, comparative baseline to reveal these underlying flaws. Read the full paper for our complete causal framework! #AI #THUNLP #OpenBMB #Multimodal #VisualReasoning #MLLM #MachineLearning
- 20
@arankomatsuzaki ·
Context Unrolling in Omni Models - A unified multimodal model natively trained on diverse modalities, including text, images, videos, 3D geometry, and hidden representations - Enables Context Unrolling, where the model explicitly reasons across multiple modal representations before producing predictions - Achieves strong performance on both multimodal generation and understanding benchmarks, while demonstrating advanced multimodal reasoning capabilities, including in-context generation of text, image, video, and 3D geometry
- 21
@NVIDIAAIDev ·
Fine-tuning multi-modal AI just got a whole lot easier. With the latest release of NVIDIA TAO, developers can accelerate post-training for reasoning VLM and embedding models using fine-tuning microservices (FTMS) with built-in recipes. New features: ⚡ NVIDIA Cosmos Reason VLM fine-tunable with AutoML in just a few config tweaks ⚡ New multimodal embeddings, including Cosmos Embed1 (video/text) and NVIDIA RADIO-CLIP (image/text) ⚡ NVPanoptix3D for 3D panoptic reconstruction from RGB images, now available on Hugging Face with FTMS fine-tuning 🔗 https://t.co/dwqolMleDq
Watch video
- 22
@askalphaxiv ·
"Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale" Intern-S1-Pro scaled a multimodal model to 1T parameters with a lot of aligned scientific data, obtaining a really strong model that's capable of analyzing scientific figures, reason across STEM topics, and easily outperform domain-specific models. This model contains native vision processing, and adaptive time-series understanding, which enables stronger figure analysis and better signal processing.
- 23
@ai_for_success ·
Google DeepMind has released Gemma 4 12B, a unified encoder free multimodal model built for running agentic AI locally on laptops. 🔥 - 12B parameter model that runs on laptops with 16GB memory - Encoder free architecture for native image and audio processing - Performance close to the larger 26B MoE model - Native audio support with raw audio token processing - Multi Token Prediction for lower latency - Open sourced under Apache 2.0 - You can try here LM Studio, Ollama, Google AI Edge Gallery App, the Google AI Edge Eloquent app and the LiteRT-LM CLI - New Gemma Skills Repository for agentic workflows
- 24
- 25
@andrewdfeldman ·
Yesterday, we launched @GoogleDeepMind's Gemma 4 model on @cerebras. The first multimodal model on Cerebras. 1,500 tokens per second. 15x faster than the nearest comparable model. Multimodal agents can see, reason, act, and retry. At 1,500 tokens per second, that loop is becoming near instant. Fast tokens are the most valuable tokens. And Cerebras delivers the fastest tokens in the world.
Watch video
- 26
@TheHumanoidHub ·
Jensen just launched NVIDIA Cosmos 3. Pitched as the first fully open omnimodel for physical AI: a mixture-of-transformers (reasoning + generation) with native vision reasoning and generation across text, image, video, sound, and action. Tops open-model leaderboards on physics, world generation, and action policy. Three jobs in one: - VLM for robots and autonomous vehicles - world model that simulates environments and predicts future states - backbone for world-action models trained on specific tasks Three options - Super (32B): for post-training robotics models that need the highest physics accuracy and generation quality. - Nano (8B) for high-quality video and action reasoning in fractions of a second. - Edge, coming soon, for real-time inference at the edge.
Watch video
- 27
@IamEmily2050 ·
A few days ago, DeepSeek published a paper titled "Thinking with Visual Primitives," but it was later removed. Luckily, I managed to download it, like so many other people, and, of course, I have to make a video overview of it with NotebookLM. Thinking with Visual Primitives Researchers from DeepSeek and partner universities have developed Thinking with Visual Primitives, a new framework designed to improve Multimodal Large Language Models' spatial reasoning. Current models often exhibit a "Reference Gap," in which linguistic reasoning fails to connect to specific physical locations within an image accurately. To solve this, the authors introduce spatial markers like points and bounding boxes as fundamental "units of thought" that the model uses to "point" while it reasons. This approach mimics human cognitive processes, such as using a finger to track objects while counting, thereby reducing mental load and preventing logical errors. Built on a highly token-efficient architecture, the system matches or surpasses much larger frontier models in visual question answering and complex spatial tasks. Ultimately, this research provides a scalable path toward more sophisticated and efficient multimodal intelligence.
Watch video
- 28
@VaibhavSisinty ·
We used computers to build AI. NVIDIA just used AI to build the computers that will replace them. And they did it in a way nobody saw coming. Quantum computers are the most powerful machines humans have ever built. They're also the most fragile. Qubits, the atoms powering quantum machines, are kept at temperatures colder than outer space. One tiny vibration. A microscopic temperature shift. The entire calculation collapses. For decades, fixing this required engineers manually tuning the machine for days just to keep it stable enough to run. NVIDIA just replaced that entire process with an AI called Ising. Here's what it actually does ↓ A vision language model watches the quantum processor in real time and auto-calibrates it on the fly. What took days now takes hours. But the error correction is where it gets wild. NVIDIA built a 3D neural network that catches quantum errors before the system collapses. 2.5x faster. 3x more accurate than the best tool available today. The AI isn't running the quantum computer. It's keeping it alive long enough to run. Jensen Huang called it the "control plane" of quantum machines. Harvard, Berkeley Lab, and Fermi National Accelerator Laboratory are already plugging it in. The entire industry moved in one week. Now zoom out for a second. A decade ago, we used normal computers to train AI. Now NVIDIA is using that AI to stabilize the computers that will eventually make normal computers look like calculators. The loop just closed. NVIDIA doesn't compete in computing eras. They become the layer every era runs on.
- 29
- 30
@DivyanshT91162 ·
Google just dropped a 12B model that has no business being this fast. I ran Gemma 4 12B locally on an RTX 4060 and got 21 tok/s. No API. No cloud. No subscription. Just 6.6GB, 256K context, and benchmarks that look straight-up unfair. → 77.5% AIME → 78.8% GPQA Diamond → 72% LiveCodeBench → 1659 Codeforces ELO The craziest part? Most multimodal models bolt together separate vision/audio encoders with an LLM. Gemma 4 doesn't. Images and audio are projected directly into the same decoder-only transformer. No encoder tax. No extra latency. No memory bloat. Google basically turned a multimodal model into something that runs like a normal local LLM. Open source AI is moving way faster than most people realize. Link👇
Watch video
- 31
@alex_verem ·
Google just released a 4B parameter AI model that can read CT scans, MRI volumes, whole-slide pathology images, and electronic health records, all in a single architecture. And they made it free and open source. This is MedGemma 1.5. A single model that handles more medical imaging modalities than any open-weight model before it. Here's what it can actually do. > Feed it a CT scan and it processes up to 85 axial slices simultaneously. > It looks across brain, chest, and abdomen volumes and flags conditions like tumors, hemorrhage, aortic aneurysms, renal calculi, and lung nodules. > Feed it an MRI and it processes multi-parametric sequences T1, T2, GRE, SWI across brain, knee, and abdomen studies. > Feed it a whole-slide pathology image and it processes up to 126 tissue patches extracted from a single biopsy slide and generates a diagnostic report. > Feed it two chest X-rays taken at different points in time and it tells you whether the patient's condition improved, stayed stable, or worsened. > Feed it a lab report PDF and it extracts every test result, unit, specimen, method, and collection time into structured JSON. One model. Four billion parameters. All of these capabilities simultaneously. The performance gains over the previous version are significant across every new capability. Compared to MedGemma 1 4B: → 3D MRI condition classification: +11% absolute accuracy → 3D CT condition classification: +3% absolute accuracy → Pathology whole-slide report generation: +47% macro F1 → Anatomical localization on chest X-rays: +35% Intersection over Union → Electronic health record Q&A: +22% accuracy → Medical licensing exam questions (MedQA): +5% accuracy Then there's the comparison that matters most for developers choosing between models. Google benchmarked MedGemma 1.5 against Qwen3 VL 4B a state-of-the-art general-purpose multimodal model at the same parameter count. > On general text-based medical knowledge, Qwen3 VL 4B wins. > On every single vision task tested, MedGemma 1.5 wins. Not by a small margin on some tasks. On every one. CT classification, MRI analysis, chest X-ray conditions, diabetic retinopathy grading, pathology, dermatology, ophthalmology. The difference is domain-specific training. A generalist model that has read every medical textbook still struggles to interpret what it sees in a CT slice. A model trained on 282,000 CT studies, 167,000 MRI studies, and 335,000 whole-slide pathology image pairs learns something different. It develops what the Google researchers call a "medical prior" — the ability to extract meaningful signal from high-dimensional imaging even on datasets it has never seen before. MedGemma 1.5 also outperforms Gemini 3 Flash and Gemini 3 Pro on several specialized tasks. A 4B open-source model beating Google's own flagship on CT classification and MRI analysis. That is what domain-specific training does. There is one honest tradeoff worth flagging. General knowledge benchmarks like MMLU Pro dropped slightly compared to MedGemma 1 and Gemma 3 4B. Intensive medical specialization came at some cost to general reasoning. The Google team is explicit about this: they think the tradeoff is worth it for the medical imaging gains. For developers building medical AI tools, it almost certainly is. The practical value here is not just the benchmark numbers. It's that every developer building a medical AI application now has a starting point that already understands CT volumes, MRI sequences, pathology slides, and lab reports. Instead of building specialized pipelines from scratch for each modality, they can fine-tune this single foundation on their specific clinical use case. Radiology tools. Pathology assistants. EHR parsing systems. Longitudinal monitoring applications. All of them can start here. Open weights. Free to download. Fine-tune for your specific clinical context.
- 32
@burkov ·
This AAAI 2026 paper introduces UI-R1, a framework demonstrating how rule-based reinforcement learning with a novel action reward significantly enhances multimodal LLM' reasoning capabilities for accurate GUI action prediction, outperforming larger supervised models on challenging in-domain and out-of-domain tasks. Read with an AI tutor: https://t.co/1DxTqKmDyd PDF: https://t.co/1mM57KiIAI
- 33
@oliviscusAI ·
You can now serve text, image, video, and audio models from a single framework. vLLM just released vLLM-Omni, a massive upgrade to their original text-based serving engine. It eliminates the need to stitch together multiple frameworks for multimodal AI. → Serve any-to-any multimodal models natively. → Includes support for diffusion models. → Fast parallel generation across all modalities. 100% open-source.
- 34
- 35
@pauliusztin_ ·
Multimodal AI agents don't need special reasoning loops. All they need is better tools. A multimodal agent still follows the same ReAct cycle: Observe Reason Call tools Receive observations Repeat The only difference is what those observations contain. The agent simply adds these tool results back into short-term memory and continues reasoning. To sum up: The loop doesn't change… the context does.
- 36
@alexabelonix ·
Alibaba just unveiled Qwen3.7-Plus, a multimodal AI agent that can see, think, code, and take action across screens. This is the agent direction that feels really important. One model that understands text, images, visual interfaces, GUIs, command lines, code, and search-augmented workflows. Basically, AI is getting closer to operating software the same way humans do: looking at the screen, understanding context, making decisions, clicking around, writing code, fixing things. That’s where the real productivity jump comes from. The next big unlock is not smarter chat. It’s AI that can actually move through digital environments and get work done.
- 37
@IlirAliu_ ·
A humanoid robot autonomously executes a full long-horizon task, from ONE natural language command... for the first time. It goes downstairs to get a snack package, rides the elevator upstairs, opens the box, and puts the snacks into a drawer. The platform integrates reinforcement learning across all layers; from low-level whole-body control and motion skills to high-level VLM-based reasoning and replanning. On the mission-reasoning layer alone, RL fine-tuning lifts end-to-end completion from 38% to 90% on complex 16-step missions, while the system as a whole handles variations like box weights and mid-task interruptions. This approach emphasizes real-world robustness through autonomous failure recovery, precise interactions like button pressing and door handling, and a custom runtime, advancing scalable humanoid autonomy without human operators. Credit: @FlexionRobotics More here: https://t.co/xIoG5Dg8ky ——- Weekly robotics and AI insights. Subscribe free: https://t.co/9Nm01QUKlB
Watch video
- 38
@_vmlops ·
GEMMA 4 12B JUST CHANGED LOCAL AI DEVELOPMENT google dropped an encoder-free multimodal model no separate vision encoder. no audio encoder. just one decoder-only transformer handling everything ▫️ raw pixel patches projected directly to LLM hidden dim ▫️ raw 16kHz audio sliced into 40ms frames, fed straight in ▫️ LoRA/full fine-tuning updates all modalities in a single pass ▫️ runs locally on 16GB VRAM ▫️ OpenAI-compatible local API server via `litert-lm serve` the wild part...? it can process 5 minutes of video at 1FPS with audio, and build a working Gradio app using itself as the backend one model. vision + audio + text. zero encoder overhead. available on Hugging Face, Ollama, LM Studio & vLLM today
Watch video
- 39
- 40
@sabir_huss50540 ·
For a year the rule held: a lab's best model stays locked behind an API. Alibaba just broke its own rule. Qwen3.8-Max is out, and this time the weights are actually downloadable. Everyone will lead with the headline: 2.4 trillion parameters. Second-largest open model ever, behind Kimi K3's 2.8T. That's the wrong number to stare at. Here's the number that matters. 2.4 trillion is the total. Only 95 billion are active per token. It's a sparse mixture-of-experts, which means it serves far cheaper than its headline size suggests. Total parameter count was never the same as usable compute, and the people quoting the big number know it. The real story isn't the size at all. It's the word "open." Every Max-tier flagship Alibaba has shipped since Qwen3-Max was proprietary, handed to you through the cloud console, never Hugging Face. The company ran a two-track game: open-source the small Qwen3 and Coder lines under Apache 2.0, keep the crown jewel closed. Qwen3.8-Max is the first time the crown jewel itself got opened. That's the pattern break, and it lands the same month Moonshot and Thinking Machines opened models of their own. The capability claim is where you keep your guard up. Alibaba calls it "second only to Fable 5." That is the vendor's own internal evaluation. Not a benchmark table. Not a third-party number from Artificial Analysis or LMArena. Positioning, not a result. Keep the two in separate buckets until independent evals land. What is real: a multimodal model that reads text, images, video, and documents in one system, built on a hybrid of Gated Delta Networks and sparse MoE, speaking 201 languages, tuned for long-horizon coding and the kind of office cowork that used to eat a human afternoon. And now, the weights on disk. You still can't run it at home. A 2.4T model is a datacenter object no matter how hard you quantize it. But that was never the point of open weights at this scale. The point is that the frontier stopped being something only a handful of labs could rent you through a paywall. For a year the best models were rentals. This one, you own.
- 41
@rohanpaul_ai ·
The Meta/Oxford study finds, a multimodal model may need surprisingly little image-generation data if language and visual understanding are trained with it from the start. So, you probably don’t need to spend that much training compute teaching a multimodal model to generate images. Language training already helps the model with vision, and learning to understand images also makes it better at generating them. But training it to generate images does very little for language or image understanding. That leads to a very uneven training mix. In their 1T-token experiments, the best overall split was 70% language, 25% image understanding, and only 5% image generation. They then tested this at 13.5B scale over 2T tokens. Even with 5x fewer image-generation tokens than the balanced setup, GenEval improved from 0.467 to 0.482, while language and image understanding improved too. There’s another lesson: don’t bolt vision on too late. The longer the model trains only on language, the more it starts ignoring the image and relying on language shortcuts. The authors call this “vision laziness.” – arxiv. org/abs/2608.05000 Title: "Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes"
- 42
@sharbel ·
🚨SHOCKING: Researchers proved that AI models judge your intelligence based almost entirely on what you look like. And a handful of visual cues are doing almost all of the damage. Researchers built 500 photorealistic faces. Then they changed one thing at a time. Then they showed these faces to six of the most widely deployed multimodal AI models in the world and asked them to make social judgments. The models did not hesitate. They judged. Over and over. Across 25 different scenarios. One model, shown an older face, rated that person as significantly less competent than the identical face made younger. The same face. The same expression. The same background. Different age. Different verdict. The researchers wrote: "A few human visual cues drive most social biases in MLLMs." But this is not a story about faces. These models are being deployed right now in hiring pipelines, loan approvals, medical triage, and criminal risk assessments. They are being handed photographs of real people and asked to make consequential decisions. The researchers did not test hypothetical future systems. They tested the models companies are shipping today. What happens when the systems deciding your future learned to see your face the way the worst hiring manager in history would?
- 43
- 44
@OpenBMB ·
Visual comprehension requires high-level abstract semantics, while image generation demands fine-grained pixel details. How can we resolve this fundamental conflict within a single unified model? 🤔 Today, we present CHEERS—new research from @TsinghuaNLP (OpenBMB member), XJTU, and UCAS: a novel unified multimodal model that decouples patch-level details from semantic representations, achieving SOTA comprehension and generation simultaneously. 🤗 Paper: https://t.co/HdNQLL63Pa 📄 arXiv: https://t.co/9lAOoyGakl 💻 Code: https://t.co/7k05AvyRYv 🤖 Models: https://t.co/nHa1spLPsE Why it matters: 1️⃣ Decoupling Semantics & Details: Traditional unified models struggle to balance conflicting goals. CHEERS introduces a Unified Vision Tokenizer that rebuilds pixels before extracting semantics to prevent information loss, while achieving a 4x visual token compression for extreme LLM efficiency. 🧩 2️⃣ Unified LLM Transformer: Built on Qwen2.5-1.5B, CHEERS seamlessly integrates two decoding regimes. It uses autoregressive decoding for text (causal mask) and flow matching for images (bidirectional attention), processing heterogeneous tokens in a single shared feature space. 🔄 3️⃣ Cascaded Flow Matching: It mimics human painting! A two-stage cascaded head first generates a low-res semantic sketch, then a gated network dynamically injects high-frequency detail residuals from the vision tokenizer, yielding breathtaking fidelity. 🎨 4️⃣ Doing More with Less: CHEERS outperforms similar-sized models like Janus-Pro and Show-o2 on comprehensive benchmarks (MMBench, GenEval, etc.). Incredibly, it achieves this SOTA performance using only 83M training samples—just 20% of the data required by its peers! 🚀 CHEERS redefines unified multimodal modeling, proving that intelligently separating semantics and details is the key to mastering both understanding and creation. #AI #THUNLP #OpenBMB #LLM #Multimodal #ImageGeneration #ComputerVision
- 45
@burkov ·
Most systems that handle different kinds of data—text, images, video, audio—do so by training a separate encoder for each type and then forcing their outputs into a shared space, which works for matching an image to a caption but loses the connections that arise when those types appear together, as in a video with narration or a document mixing charts and prose. This recent work from Google takes a model already trained to understand all these types at once (Gemini) and adapts it into an embedding model, meaning one that turns any input into a single fixed-length vector of numbers whose geometric closeness reflects semantic similarity, so that related items end up near each other regardless of which modality they came from. The authors do this through contrastive training, where the model is shown matched pairs and learns to pull them close while pushing unrelated items apart, run in stages across many tasks and modalities, and they report that the single resulting model matches or beats specialized systems on text retrieval, image-text matching, video search, code search, and audio retrieval. Two specifics stand out: feeding raw audio directly into the model retrieves text better than first transcribing the audio into words and embedding that; and the model holds up across narrow fields it was not specifically trained on, from microscopy images to paintings to recipes, where competing models swing between strong and weak depending on the domain. Read with an AI tutor: https://t.co/RL1J9jkagC PDF: https://t.co/ho8YfyHQAe
- 46
@shawnchauhan1 ·
Google just made multimodal search infrastructure a commodity. Text, image, video, audio, PDF - one vector space, one API call. Startups have been raising on the premise that unified multimodal retrieval is hard to build. It was. Until yesterday. The question is not whether Google can do this. The question is what business models quietly just broke.
- 47
- 48
@did0f ·
What if the most important AI benchmark is not intelligence, but locality? Not, “Is this model smarter than the biggest cloud model?” But, “Can it run well on my machine?” That is why I find Gemma 4 12B interesting. Not because it magically solves multimodality. It does not. But because it makes local multimodal AI feel more realistic. Most multimodal models are stitched together. 🔤 A language model handles text. 👁️ A vision encoder translates images. 🗣️ A speech encoder translates audio. That approach works, but it is heavy. Gemma 4 12B takes a different route. People often call it “encoder-free,” but that phrase needs some precision. It does not mean images go straight into the model without any adaptation. There is still an adapter. The difference is that instead of relying on a heavy encoder that tries to interpret the image upfront, Gemma uses a much simpler step: a linear projection. In plain English, the image is split into patches. Those patches are converted into vectors with the right shape, then passed into the same internal space where the model already works with text. A thinner layer changes the format. The backbone does more of the reasoning. That matters. Because every piece you remove makes local AI less fragile. → Less memory. → Less latency. → Less cloud dependency. And that is the part I care about.
- 49
@tonjkb ·
On Rogan, Marc Andreessen described a secret AI setup Mark Zuckerberg uses to train Brazilian Jiu-Jitsu. Webcams in his home gym feed live sparring video to a multimodal AI that gives him direct performance feedback. I see this as much more than a billionaire's custom toy. We are shifting from the era of biometrics to the era of biomechanics. For a decade, wearables gave us quantitative tracking. An Apple Watch tells you how fast your heart beats or how far you ran. Multimodal AI introduces qualitative coaching. Because new models process live video, they understand spatial relationships and movement mechanics. Instead of logging your heart rate, an ambient camera can correct the leverage of your grappling submission or fix the arc of your golf swing. Niche sports startups are already commercializing this technology. Zuckerberg's private gym setup looks like an extreme prototype today, but it is actually a preview. The next frontier of fitness tech is not a tracker on your wrist. It is a digital coach in the room with you.
Watch video
- 50
@sabir_huss50540 ·
Robots can't learn to do things because nobody can afford to show them enough examples. Every demonstration means a human physically puppeting a robot arm through a task, one teleoperated repetition at a time. It is slow, expensive, and capped by how many robots you own. That single bottleneck is why your house doesn't have a robot that folds towels. It's called Ego2Robot. A team out of the Qwen lab and Renmin University asked a question that sounds too simple to work. There are millions of hours of video shot from people's own eyes: cooking, sorting, assembling, first-person footage of human hands doing tasks. What if you could turn that into robot training data? No robot required. Just the human video. Here's how it works, and it's a small marvel of engineering. The pipeline takes a first-person clip of a human hand. It tracks the 21 keypoints of that hand frame by frame. It maps the thumb and fingers onto a robot's two-jaw gripper, working out where the grip closes and how wide it opens. Then it does the strange part. It erases the human arm from the video entirely, inpaints the background back in behind where the arm was, and renders a photorealistic robot arm into the exact same scene, doing the exact same motion. Same task. Same objects. Human hand out, robot arm in. Then it does that fifteen times over. Because the motion is embodiment-free, the same human clip gets re-rendered as fifteen different robot bodies: a Franka Panda, a UR5e, an ARX, twelve more. One human video becomes fifteen robot demonstrations, each kinematically solved so the arm can actually reach every pose. The scale is the part that lands. They fed in about 1,940 hours of human video. Out came 18,561 hours of robot training data. Roughly a 9.6× amplification, across 15 robot morphologies, the largest human-to-robot dataset ever built. And it isn't slop. Three layers of quality curation throw out failed grips, physically impossible motions, and a vision-language model reads every synthesized clip to confirm the robot is actually doing what the instruction says. Then the honest question: does training on fake robot data make a real robot better? They tested it on an actual ARX platform across five tasks. Putting fruit in a basket. Blocks in a drawer. Folding a towel. Sweeping trash. Inserting a screw. Mixing the synthesized human-video data in with a handful of real demonstrations beat real demonstrations alone on all five. The biggest jumps came on the hardest tasks: +14 points on putting blocks in a drawer, +13 on inserting a screw. The gains showed up most where robots usually break: new lighting, new camera angles, unfamiliar objects, robot bodies never seen in training. It is not solved. The pipeline throws away individual finger movement, so no dexterous multi-finger skills yet. The rendering still cracks under heavy occlusion. One robot body with a very different arm shape barely improved at all. But the wall was never that robots can't learn. It's that teaching them cost too much. This turns a phone's worth of first-person footage into the one thing that was always too expensive to collect. The demonstrations were sitting in human hands the whole time.
Best Tweets by Topic