Local Multimodal Deployment and Infrastructure
Efficient, open, and local multimodal models and infrastructure, including edge deployment, on-device inference, fine-tuning, serving, and low-latency hardware.
34%
Best tweets about Multimodal AI
50 Best Tweets About Multimodal AI (2026)
Browse the best tweets about multimodal AI, including vision, audio, video, model architectures, benchmarks, applications, and developer experiments.
Multimodal models combining text, images, audio, or video, with concrete research, evaluations, workflows, and applications.
- Creators
- 47
- Updated
Original Xholic analysis
What 50 top Multimodal AI posts reveal
Discussion of multimodal AI emphasizes compact document OCR, unified multimodal models, and systems that combine visual perception with agents or workflows. The largest score outliers span CAD generation, OCR, audio-video generation, embeddings, and pathology. Posts also raise benchmark-grounding concerns, while other posts report reasoning and agent-performance advances.
- Dominant tone
- Positive
- Median score
- 25.8
- Leading format
- Announcement
- Recent posts
- 32%
86% of posts
All-time engagement
84% of posts
Published in 90 days
Conversation map
The themes creators return to
Unified Omnimodal Models
Unified multimodal foundation models that natively process and generate across text, images, audio, video, 3D, and actions, including architectural designs such as discrete tokens, decoder-only models, and mixture-of-experts.
30%
Agents, GUI Control, and Embodied AI
Multimodal agents that perceive screens or environments and act through GUIs, browsers, terminals, tools, robots, and software workflows.
26%
Multimodal Generation and Editing
Multimodal generation and editing for images, video, speech, CAD, 3D worlds, and other visual or audiovisual outputs.
16%
Multimodal Reasoning and Evaluation
Multimodal reasoning research and evaluation, covering spatial grounding, visual primitives, scientific reasoning, benchmark leakage, fake visual reasoning, and model bias.
14%
Multimodal Embeddings and Retrieval
Cross-modal embeddings and retrieval systems that place text, images, audio, video, documents, and code in a shared vector space.
12%
Scientific and Medical Multimodality
Domain-specific multimodal applications in medicine, biology, pathology, proteomics, protein function, and drug discovery.
12%
Document OCR and Understanding
Vision-language models for document intelligence, including OCR, layout parsing, tables, formulas, handwriting, PDFs, and efficient local deployment.
10%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 100
- Median reposts
- 16
- Median replies
- 6
- Median views
- 9.7K
Posts with media make up 94% of this collection. Their median all-time score is 24.7, compared with 88.0 for text-only posts.
Format mix
- Announcement 84% · score 34.4
- Opinion 16% · score 11.2
Consensus and debate
Where creators agree, and where they do not
Shared view
Compact OCR is a practical focal point
Document AI is a prominent practical thread. Posts describe compact, open vision-language models for layouts, tables, formulas, and handwriting, with local deployment or high-throughput serving options.
Shared view
Unified models target broader modality coverage
Posts frame unified architectures as a way to combine text, vision, and audio capabilities, including understanding, generation, editing, and agent-oriented local use. The cited models make differing architecture and performance claims.
Shared view
Perception is being paired with agents
Posts describe multimodal systems that pair perception with action or verification: visual analysis for robotics, a VLM director and verifier for 3D-world construction, and an agent positioned to work across screens and software workflows.
Open debate
Benchmark gains and visual grounding are both contested topics
One post summarizes MIRAGE as finding substantial image-removed performance on six vision benchmarks and argues that some benchmark items may not require vision. Other posts report multimodal-reasoning gains from visual primitives and RL-based training. These are reported claims rather than independently verified comparisons.
Open debate
Computer-use agents have contrasting readiness claims
One practitioner reports a screenshot-driven computer-use loop taking about 70 seconds for a tweet-posting task. Other posts promote high-throughput multimodal inference and screen-operating agents; these posts do not establish a common measure of real-world agent speed or reliability.
What performs
Patterns behind standout posts
Top outliers span applied multimodality
The five analytics outliers correspond to posts on image-to-CAD generation, document OCR, audio-video generation, shared multimodal embeddings, and pathology-to-proteomics. Their all-time scores range from 717.37 to 5,483.04.
Document OCR leads theme-level performance
Document OCR has the highest theme-level median all-time score in the analytics, at 287.031. The cited posts focus on small vision-language OCR models and local or server deployment.
Generation announcements drew relatively strong attention
Announcements account for 42 of 50 posts (84%). Multimodal generation has a 39.2 median all-time score, above the infrastructure (30.874), agent (28.734), and unified-model (22.619) themes, though below document OCR and scientific/medical multimodality.
Statistical standouts
- View standout post 1 Score 5483.0 · 212.44× median
- View standout post 2 Score 1240.2 · 48.05× median
- View standout post 3 Score 853.0 · 33.05× median
- View standout post 4 Score 775.1 · 30.03× median
- View standout post 5 Score 717.4 · 27.79× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 16% of the selected posts.
-
1. BURKOV
@burkov
2 posts
-
2. Hugging Models
@HuggingModels
2 posts
-
3. DailyPapers
@HuggingPapers
2 posts
-
4. Vaishnavi
@_vmlops
1 post
-
5. AshutoshShrivastava
@ai_for_success
1 post
-
6. Alex Veremeyenko
@alex_verem
1 post
BURKOV covers agent and retrieval research
BURKOV’s two posts cover rule-based reinforcement learning for GUI action prediction and contrastive adaptation of a multimodal model for retrieval.
Hugging Models spotlights VLM explainers
Hugging Models posts about video recap and CLIP-style image-text understanding.
DailyPapers tracks research releases
DailyPapers posts cover an open-weight VLM release and a multimodal reasoning model with reported benchmark results.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best Multimodal AI tweets
Top Multimodal AI tweets from 47 creators
Ranked 01–50
- 01
@heygurisingh ·
🚨BREAKING: MIT just dropped an AI model that converts photos into fully editable CAD programs and it quietly kills the $150/hour CAD modeling industry. It's called GenCAD. You give it an image. It gives you the complete parametric command sequence lines, arcs, extrusions ready for manufacturing. Not meshes. Not point clouds. Actual editable CAD. - Autoregressive transformers + diffusion models for image-to-CAD translation - Outperforms every existing method on unconditional and conditional CAD generation - Retrieves matching designs from 7K+ CAD databases using just a photo - Trained on 840K+ images - Generates multiple valid designs from a single input The team also built CAD-Coder on top of this -- a vision-language model that writes CadQuery Python code from images with 100% valid syntax rate. Beats GPT-4.5 and Qwen2.5-VL-72B. Built at MIT. Published in ASME Journal of Mechanical Design. 100% Open Source.
- 02
@AlphaSignalAI ·
A peanut-sized Chinese model just dethroned Gemini at reading documents. GLM-OCR is a 0.9B parameter vision-language model. It scores 94.62 on OmniDocBench V1.5, ranking #1 overall. For context, it outperforms models 100x its size. 100% open-source. It works in two stages. 1. A layout engine detects every region in a document. 2. Each region gets read in parallel. The model predicts multiple tokens per step instead of one. That's what makes it so fast at small size. It handles things most OCR tools struggle with: > Complex tables and nested layouts > Handwritten text and stamps > Math formulas and code blocks > Mixed image-and-text documents You can run it locally through Ollama. It fits on edge devices with limited compute. Every expensive OCR API just got a free competitor.
Watch video
- 03
@ihteshamali ·
🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.
Watch video
- 04
- 05
@satyanadella ·
We’ve trained a multimodal AI model to turn routine pathology slides into spatial proteomics, with the potential to reduce time and cost while expanding access to cancer care.
Watch video
- 06
@techNmak ·
Finally, a lightweight VLM that beats the giants at OCR. (1.7B parameters, SOTA on OmniDocBench) dots. ocr is a new multilingual document parser that proves you don't need massive models for perfect document understanding. Current SOTA models are often massive (72B+) or require expensive API calls (GPT-4o). dots. ocr changes the game. It is built on a compact 1.7B LLM foundation but outperforms much larger models like Qwen2-VL-72B and GPT-4o on key benchmarks. What makes it special? → Unified Architecture: Handles text, tables, formulas, and layout detection in one pass. → Top-Tier Precision: Achieves 0.125 Overall Edit Distance on OmniDocBench (lower is better), beating GPT-4o (0.233). → Production Ready: Integrated with vLLM for high-throughput deployment. Performance stats: → Table Recognition: 88.6 TEDS score (vs 72.0 for GPT-4o). → Layout Detection: 0.930 F1 score (beats DocLayout-YOLO). → Speed: Faster inference than larger foundation models due to 1.7B size. You can deploy this locally using vLLM or Hugging Face. Link to the GitHub repo is in the comments.
- 07
@dr_cintas ·
This peanut-sized chinese model just dethroned Gemini at reading documents. It’s called glm-ocr. it’s a tiny 0.9b parameter vision-language model that is about to replace every expensive ocr api you use. → Handles text, tables, formulas, handwriting → Scored 94.62 on OmniDocBench V1.5 → 8 languages → vLLM, SGLang, Ollama And it’s 100% open-source.
Watch video
- 08
@sukh_saroy ·
Holy shit... Stanford just proved that GPT-5, Gemini, and Claude can't actually see. They removed every image from 6 major vision benchmarks. The models still scored 70-80% accuracy. They were never looking at your photos. Your scans. Your X-rays. Here's what's really going on: ↓ The paper is called MIRAGE. Co-authored by Fei-Fei Li. They tested GPT-5.1, Gemini-3-Pro, Claude Opus 4.5, and Gemini-2.5-Pro across 6 benchmarks -- medical and general. Then silently removed every image. No warning. No prompt change. The models didn't even notice. They kept describing images in detail. Diagnosing conditions. Writing full reasoning traces. From images that were never there. Stanford calls it the "mirage effect." Not hallucination. Something worse. Hallucination = making up wrong details about a real input. Mirage = constructing an entire fake reality and reasoning from it confidently. The models built imaginary X-rays, described fake nodules, and diagnosed conditions -- all from text patterns alone. But that's not the scary part. They trained a "super-guesser" -- a tiny 3B parameter text-only model. Zero vision capability. Fine-tuned it on the largest chest X-ray benchmark (696,000 questions). Images removed. It beat GPT-5. It beat Gemini. It beat Claude. It beat actual radiologists. Ranked #1 on the held-out test set. Without ever seeing a single X-ray. The reasoning traces? Indistinguishable from real visual analysis. Now here's what should terrify you: When the models fake-see medical images, their mirage diagnoses are heavily biased toward the most dangerous conditions. STEMI. Melanoma. Carcinoma. Life-threatening diagnoses -- from images that don't exist. 230 million people ask health questions on ChatGPT every day. They also found something wild: → Tell a model "there's no image, just guess" -- performance drops → Silently remove the image and let it assume it's there -- performance stays high The model enters "mirage mode." It doesn't know it can't see. And it performs BETTER when it doesn't know it's blind. When Stanford applied their cleanup method (B-Clean) to existing benchmarks, it removed 74-77% of all questions. Three-quarters of "vision" benchmarks don't test vision. Every leaderboard. Every "multimodal breakthrough." Every benchmark score you've seen this year. Built on mirages. Code is open-sourced. Paper is live on arXiv. If you're building anything with multimodal AI -- especially in healthcare -- read this paper before you ship. (Link in the comments)
- 09
@TheTuringPost ·
There’s a serious gap in multimodal models – they work with images, but still reason in language, which isn’t that precise for visual stuff. @deepseek_ai just dropped an idea to solve this: let the model literally point to exact locations in the image while it thinks. They call it "Thinking with Visual Primitives." These visual primitives are: - points (specific locations) - bounding boxes (areas in the image) Using them, the model knows what exactly it’s referring to and achieves ~77% better accuracy on average (vs. Gemini 3 Flash's 76.5% and 71.1% for GPT-5.4) Plus, only ~80–90 visual tokens are kept in memory after compression thanks to the efficient architecture Here is how it works:
- 10
- 11
- 12
@Prince_Canuma ·
RF-DETR by @roboflow now on MLX It can do realtime instance segmentation on-device and enable some cool use cases for visual analysis, monitoring and robotics like Reachy Mini. Also augmented VLM and VLA by preprocessing image and video with areas of interest. New release coming soon on mlx-vlm 🚀 For those who can’t wait you can install mlx-vlm from source.
Watch video
- 13
@Yuchenj_UW ·
I used Claude Computer Use/Dispatch yesterday. My feeling: It’s too damn slow! Posting a tweet takes me ~5 seconds (once I have the content). Claude took 70 seconds. Why? It controls the screen via a loop: take a screenshot → send to a huge remote multimodal model (opus 4.6) → decide actions (click, type, scroll) → take another screenshot → repeat. We’re basically forcing a large general model to operate a human UI. Two things will happen in my opinion: 1. It is using a massive model (Opus 4.6) just to understand screens. That won’t last. Smaller, specialized models and eventually local models will handle most of this. 2. GUIs were built for humans. Almost all software will expose APIs/CLI for agents, so most actions won’t need to “use a computer” at all.
- 14
@Meituan_LongCat ·
🔥 Introducing LongCat-Next: A Discrete Native Autoregressive Multimodal Model LongCat-Next integrates language, vision, and audio into a unified discrete autoregressive model, extending Next-Token Prediction to native multimodality and delivering industrial-strength performance across diverse multimodal domains. 🔑 Key Features: ⚙️ 68.5B total params, 3B active, LongCat-Flash-Lite MoE backbone, excels at seeing, painting, and speaking in a unified discrete autoregressive framework. 🧩 Discrete Native Autoregression Paradigm (DiNA): We introduce DiNA, a unified paradigm that extends next-token prediction from language to native multimodality, internalizing diverse modalities into a shared discrete token space. 🌐 Discrete Native Any‑Resolution Vision Transformer (dNaViT): A unified visual tokenizer and de-tokenizer that encodes images into discrete IDs with semantic completeness, enabling both understanding and generation at any resolution. This approach overcomes the performance ceiling of discrete vision modeling in understanding tasks and enables to reconcile the conflict between understanding and generation. 👀 Visual Understanding: Fine-grained visual perception for complex tasks such as OCR, Charts, GUI interpretation, and document analysis, and advanced STEM reasoning capabilities. 🎨 Visual Generation: Generation under 28x compression ratio at arbitary resolution with competitive performance, especially in text rendering. 🎧 Speech: Strong audio comprehension capabilities, low-latency and intelligent audio-to-audio interaction, as well as speech synthesis featuring customizable voice cloning. 📄 Paper: https://t.co/qiV6YnU9pf 🔗 GitHub: https://t.co/50SHgGrltZ 😊 HuggingFace: https://t.co/Lf0JIQBNAW 💻 demo: https://t.co/b0KncgeAju 📖 blog: https://t.co/LlII0InEW7
- 15
@MattNiessner ·
📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! https://t.co/6NC7zIEn4n https://t.co/vTO3sLFLFw Great work by @ErkocZiya @angelaqdai
Watch video
- 16
@fahdmirza ·
💥 Gemma 4 E2B + Hermes Agent + vLLM running fully local — zero cloud, zero cost ♠ and it's a complete multimodal AI stack on a single GPU 🚀 🔹Gemma 4 E2B served locally via vLLM 0.19.0 on NVIDIA A6000 🔹Text, vision and native audio transcription — all offline 🔹27 languages transcribed locally with zero API calls 🔹Hermes Agent connected as autonomous brain with 40+ built-in tools 🔹Newspaper OCR, multilingual audio, agentic chat — all tested live 🔥 Watch the full video below
- 17
@intern_lm ·
🚀Meet InternVL-U: a lightweight 4B unified multimodal model that brings reasoning, generation, and editing into a unified framework. 🔥Built upon unified contextual modeling, modality-specific modular design, and decoupled visual representations, InternVL-U achieves a strong performance-efficiency trade-off, consistently outperforming unified baselines with over 3× larger model scales on challenging tasks such as text rendering, scientific reasoning, and spatially grounded generation and editing. 😉Open-source and designed for efficient, practical multimodal intelligence. 🤗GitHub: https://t.co/4gJwj6Ehv0 🤗Hugging Face: @huggingface https://t.co/idOhLCXz46 🤗GenEditEvalKit: https://t.co/V4lQkkieWW 🤗TextEdit: https://t.co/AmydjNWHPF 🤗Tech report: https://t.co/DJc2vof17l
- 18
@chutes_ai ·
Moonshot trained a model on 15 trillion tokens of mixed vision and text data. The result scores 96.1 on AIME 2025 and 76.8 on SWE-Bench Verified. Model Spotlight: Kimi K2.5 by @kimi_moonshot 1T total parameters. 32B activated per token (MoE). 256K context. Vision baked into pretraining, so it reads documents, charts, code screenshots, and video without a separate adapter. The benchmarks put it right next to GPT-5.2 and Claude 4.5 Opus on most tasks. On a few vision benchmarks (MathVista, OCRBench, InfoVQA), it outperforms both. 78.5 MMMU-Pro (multimodal reasoning) 84.2 MathVision 76.8 SWE-Bench Verified 85.0 LiveCodeBench v6 The other thing worth knowing: Agent Swarm. You give it a complex task, K2.5 breaks it into parallel sub-tasks, assigns domain-specific agents to each one, and coordinates output. BrowseComp jumps from 60.6 to 78.4 with this approach. It can also look at a UI mockup and write the frontend code. Running on Chutes with TEE. Your prompts and outputs stay protected from GPU operators during inference. Try it here: https://t.co/K1Xg3zUpyM Have you tried any of the K2 family? Curious what use cases people are running.
- 19
@arankomatsuzaki ·
Context Unrolling in Omni Models - A unified multimodal model natively trained on diverse modalities, including text, images, videos, 3D geometry, and hidden representations - Enables Context Unrolling, where the model explicitly reasons across multiple modal representations before producing predictions - Achieves strong performance on both multimodal generation and understanding benchmarks, while demonstrating advanced multimodal reasoning capabilities, including in-context generation of text, image, video, and 3D geometry
- 20
@NVIDIAAIDev ·
Fine-tuning multi-modal AI just got a whole lot easier. With the latest release of NVIDIA TAO, developers can accelerate post-training for reasoning VLM and embedding models using fine-tuning microservices (FTMS) with built-in recipes. New features: ⚡ NVIDIA Cosmos Reason VLM fine-tunable with AutoML in just a few config tweaks ⚡ New multimodal embeddings, including Cosmos Embed1 (video/text) and NVIDIA RADIO-CLIP (image/text) ⚡ NVPanoptix3D for 3D panoptic reconstruction from RGB images, now available on Hugging Face with FTMS fine-tuning 🔗 https://t.co/dwqolMleDq
Watch video
- 21
@askalphaxiv ·
"Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale" Intern-S1-Pro scaled a multimodal model to 1T parameters with a lot of aligned scientific data, obtaining a really strong model that's capable of analyzing scientific figures, reason across STEM topics, and easily outperform domain-specific models. This model contains native vision processing, and adaptive time-series understanding, which enables stronger figure analysis and better signal processing.
- 22
@ai_for_success ·
Google DeepMind has released Gemma 4 12B, a unified encoder free multimodal model built for running agentic AI locally on laptops. 🔥 - 12B parameter model that runs on laptops with 16GB memory - Encoder free architecture for native image and audio processing - Performance close to the larger 26B MoE model - Native audio support with raw audio token processing - Multi Token Prediction for lower latency - Open sourced under Apache 2.0 - You can try here LM Studio, Ollama, Google AI Edge Gallery App, the Google AI Edge Eloquent app and the LiteRT-LM CLI - New Gemma Skills Repository for agentic workflows
- 23
@andrewdfeldman ·
Yesterday, we launched @GoogleDeepMind's Gemma 4 model on @cerebras. The first multimodal model on Cerebras. 1,500 tokens per second. 15x faster than the nearest comparable model. Multimodal agents can see, reason, act, and retry. At 1,500 tokens per second, that loop is becoming near instant. Fast tokens are the most valuable tokens. And Cerebras delivers the fastest tokens in the world.
Watch video
- 24
@TheHumanoidHub ·
Jensen just launched NVIDIA Cosmos 3. Pitched as the first fully open omnimodel for physical AI: a mixture-of-transformers (reasoning + generation) with native vision reasoning and generation across text, image, video, sound, and action. Tops open-model leaderboards on physics, world generation, and action policy. Three jobs in one: - VLM for robots and autonomous vehicles - world model that simulates environments and predicts future states - backbone for world-action models trained on specific tasks Three options - Super (32B): for post-training robotics models that need the highest physics accuracy and generation quality. - Nano (8B) for high-quality video and action reasoning in fractions of a second. - Edge, coming soon, for real-time inference at the edge.
Watch video
- 25
@IamEmily2050 ·
A few days ago, DeepSeek published a paper titled "Thinking with Visual Primitives," but it was later removed. Luckily, I managed to download it, like so many other people, and, of course, I have to make a video overview of it with NotebookLM. Thinking with Visual Primitives Researchers from DeepSeek and partner universities have developed Thinking with Visual Primitives, a new framework designed to improve Multimodal Large Language Models' spatial reasoning. Current models often exhibit a "Reference Gap," in which linguistic reasoning fails to connect to specific physical locations within an image accurately. To solve this, the authors introduce spatial markers like points and bounding boxes as fundamental "units of thought" that the model uses to "point" while it reasons. This approach mimics human cognitive processes, such as using a finger to track objects while counting, thereby reducing mental load and preventing logical errors. Built on a highly token-efficient architecture, the system matches or surpasses much larger frontier models in visual question answering and complex spatial tasks. Ultimately, this research provides a scalable path toward more sophisticated and efficient multimodal intelligence.
Watch video
- 26
- 27
@riyazmd774 ·
🚨 BREAKING: Alibaba unleashes Qwen3.5-Omni, a new frontier in Full-Modality AI. 🤯 Matching the latest Gemini-3.1 Pro in A/V understanding & surpassing it in Audio tasks, this model introduces Audio-Visual Vibe Coding turning whiteboard sketch videos or game clips directly into runnable code. 🎧 10h+ Audio Input | 1h Video Context 🎬 Script-level descriptions w/ timestamps 🌍 74 langs recognized + 29 langs generated The barrier between human intent and machine execution has vanished. A deep dive 👇 @Ali_TongyiLab @Alibaba_Qwen #Qwen #VibeCoding #AI #Multimodal
- 28
@DivyanshT91162 ·
Google just dropped a 12B model that has no business being this fast. I ran Gemma 4 12B locally on an RTX 4060 and got 21 tok/s. No API. No cloud. No subscription. Just 6.6GB, 256K context, and benchmarks that look straight-up unfair. → 77.5% AIME → 78.8% GPQA Diamond → 72% LiveCodeBench → 1659 Codeforces ELO The craziest part? Most multimodal models bolt together separate vision/audio encoders with an LLM. Gemma 4 doesn't. Images and audio are projected directly into the same decoder-only transformer. No encoder tax. No extra latency. No memory bloat. Google basically turned a multimodal model into something that runs like a normal local LLM. Open source AI is moving way faster than most people realize. Link👇
Watch video
- 29
@SandAI_HQ ·
🪄 Introducing daVinci-MagiHuman: The Performance-Level Audio-Video Generative Foundation Model Proudly open-sourced and jointly developed by SII GAIR Lab & https://t.co/yn4NJpoMrD, it sets a new standard for multimodal AI. ⏳ 1/6
Watch video
- 30
@alex_verem ·
Google just released a 4B parameter AI model that can read CT scans, MRI volumes, whole-slide pathology images, and electronic health records, all in a single architecture. And they made it free and open source. This is MedGemma 1.5. A single model that handles more medical imaging modalities than any open-weight model before it. Here's what it can actually do. > Feed it a CT scan and it processes up to 85 axial slices simultaneously. > It looks across brain, chest, and abdomen volumes and flags conditions like tumors, hemorrhage, aortic aneurysms, renal calculi, and lung nodules. > Feed it an MRI and it processes multi-parametric sequences T1, T2, GRE, SWI across brain, knee, and abdomen studies. > Feed it a whole-slide pathology image and it processes up to 126 tissue patches extracted from a single biopsy slide and generates a diagnostic report. > Feed it two chest X-rays taken at different points in time and it tells you whether the patient's condition improved, stayed stable, or worsened. > Feed it a lab report PDF and it extracts every test result, unit, specimen, method, and collection time into structured JSON. One model. Four billion parameters. All of these capabilities simultaneously. The performance gains over the previous version are significant across every new capability. Compared to MedGemma 1 4B: → 3D MRI condition classification: +11% absolute accuracy → 3D CT condition classification: +3% absolute accuracy → Pathology whole-slide report generation: +47% macro F1 → Anatomical localization on chest X-rays: +35% Intersection over Union → Electronic health record Q&A: +22% accuracy → Medical licensing exam questions (MedQA): +5% accuracy Then there's the comparison that matters most for developers choosing between models. Google benchmarked MedGemma 1.5 against Qwen3 VL 4B a state-of-the-art general-purpose multimodal model at the same parameter count. > On general text-based medical knowledge, Qwen3 VL 4B wins. > On every single vision task tested, MedGemma 1.5 wins. Not by a small margin on some tasks. On every one. CT classification, MRI analysis, chest X-ray conditions, diabetic retinopathy grading, pathology, dermatology, ophthalmology. The difference is domain-specific training. A generalist model that has read every medical textbook still struggles to interpret what it sees in a CT slice. A model trained on 282,000 CT studies, 167,000 MRI studies, and 335,000 whole-slide pathology image pairs learns something different. It develops what the Google researchers call a "medical prior" — the ability to extract meaningful signal from high-dimensional imaging even on datasets it has never seen before. MedGemma 1.5 also outperforms Gemini 3 Flash and Gemini 3 Pro on several specialized tasks. A 4B open-source model beating Google's own flagship on CT classification and MRI analysis. That is what domain-specific training does. There is one honest tradeoff worth flagging. General knowledge benchmarks like MMLU Pro dropped slightly compared to MedGemma 1 and Gemma 3 4B. Intensive medical specialization came at some cost to general reasoning. The Google team is explicit about this: they think the tradeoff is worth it for the medical imaging gains. For developers building medical AI tools, it almost certainly is. The practical value here is not just the benchmark numbers. It's that every developer building a medical AI application now has a starting point that already understands CT volumes, MRI sequences, pathology slides, and lab reports. Instead of building specialized pipelines from scratch for each modality, they can fine-tune this single foundation on their specific clinical use case. Radiology tools. Pathology assistants. EHR parsing systems. Longitudinal monitoring applications. All of them can start here. Open weights. Free to download. Fine-tune for your specific clinical context.
- 31
@burkov ·
This AAAI 2026 paper introduces UI-R1, a framework demonstrating how rule-based reinforcement learning with a novel action reward significantly enhances multimodal LLM' reasoning capabilities for accurate GUI action prediction, outperforming larger supervised models on challenging in-domain and out-of-domain tasks. Read with an AI tutor: https://t.co/1DxTqKmDyd PDF: https://t.co/1mM57KiIAI
- 32
@oliviscusAI ·
You can now serve text, image, video, and audio models from a single framework. vLLM just released vLLM-Omni, a massive upgrade to their original text-based serving engine. It eliminates the need to stitch together multiple frameworks for multimodal AI. → Serve any-to-any multimodal models natively. → Includes support for diffusion models. → Fast parallel generation across all modalities. 100% open-source.
- 33
@pauliusztin_ ·
Multimodal AI agents don't need special reasoning loops. All they need is better tools. A multimodal agent still follows the same ReAct cycle: Observe Reason Call tools Receive observations Repeat The only difference is what those observations contain. The agent simply adds these tool results back into short-term memory and continues reasoning. To sum up: The loop doesn't change… the context does.
- 34
- 35
@alexabelonix ·
Alibaba just unveiled Qwen3.7-Plus, a multimodal AI agent that can see, think, code, and take action across screens. This is the agent direction that feels really important. One model that understands text, images, visual interfaces, GUIs, command lines, code, and search-augmented workflows. Basically, AI is getting closer to operating software the same way humans do: looking at the screen, understanding context, making decisions, clicking around, writing code, fixing things. That’s where the real productivity jump comes from. The next big unlock is not smarter chat. It’s AI that can actually move through digital environments and get work done.
- 36
@IlirAliu_ ·
A humanoid robot autonomously executes a full long-horizon task, from ONE natural language command... for the first time. It goes downstairs to get a snack package, rides the elevator upstairs, opens the box, and puts the snacks into a drawer. The platform integrates reinforcement learning across all layers; from low-level whole-body control and motion skills to high-level VLM-based reasoning and replanning. On the mission-reasoning layer alone, RL fine-tuning lifts end-to-end completion from 38% to 90% on complex 16-step missions, while the system as a whole handles variations like box weights and mid-task interruptions. This approach emphasizes real-world robustness through autonomous failure recovery, precise interactions like button pressing and door handling, and a custom runtime, advancing scalable humanoid autonomy without human operators. Credit: @FlexionRobotics More here: https://t.co/xIoG5Dg8ky ——- Weekly robotics and AI insights. Subscribe free: https://t.co/9Nm01QUKlB
Watch video
- 37
@biogerontology ·
For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation
Watch video
- 38
@_vmlops ·
GEMMA 4 12B JUST CHANGED LOCAL AI DEVELOPMENT google dropped an encoder-free multimodal model no separate vision encoder. no audio encoder. just one decoder-only transformer handling everything ▫️ raw pixel patches projected directly to LLM hidden dim ▫️ raw 16kHz audio sliced into 40ms frames, fed straight in ▫️ LoRA/full fine-tuning updates all modalities in a single pass ▫️ runs locally on 16GB VRAM ▫️ OpenAI-compatible local API server via `litert-lm serve` the wild part...? it can process 5 minutes of video at 1FPS with audio, and build a working Gradio app using itself as the backend one model. vision + audio + text. zero encoder overhead. available on Hugging Face, Ollama, LM Studio & vLLM today
Watch video
- 39
@hasantoxr ·
ByteDance 🔥: China's giant AI player made a fully multimodal AI agent stack that controls your computer, browser, and terminal using natural language instructions. It's called UI-TARS Desktop + Agent TARS. It sees your screen, clicks buttons, fills forms, and completes multi-step tasks the same way a human operator would. Here's what's actually inside: → Agent TARS: A CLI + Web UI agent combining GUI vision, browser control, and MCP tool integration → UI-TARS Desktop: A native desktop app running a local or remote computer operator powered by the UI-TARS vision-language model Real tasks it can complete right now: > Book flights on Priceline ("earliest flight from San Jose to New York on September 1st") > Find and reserve hotels on Booking(dot)com within a specified budget > Pull live data and generate charts via MCP servers > Check GitHub issues and summarize them > Open VS Code settings and make precise configuration changes The browser agent runs in three modes: 1. Pure GUI vision, 2. DOM-based, or 3. A hybrid of both. Remote operators ship free with no configuration required. Click to remotely control any computer or browser and the agent handles the rest. Cross-platform: Windows, macOS, Browser. The whole stack runs locally. No data leaves your machine unless you configure a remote operator. https://t.co/HkBZO7pcx9
Watch video
- 40
- 41
@sabir_huss50540 ·
For a year the rule held: a lab's best model stays locked behind an API. Alibaba just broke its own rule. Qwen3.8-Max is out, and this time the weights are actually downloadable. Everyone will lead with the headline: 2.4 trillion parameters. Second-largest open model ever, behind Kimi K3's 2.8T. That's the wrong number to stare at. Here's the number that matters. 2.4 trillion is the total. Only 95 billion are active per token. It's a sparse mixture-of-experts, which means it serves far cheaper than its headline size suggests. Total parameter count was never the same as usable compute, and the people quoting the big number know it. The real story isn't the size at all. It's the word "open." Every Max-tier flagship Alibaba has shipped since Qwen3-Max was proprietary, handed to you through the cloud console, never Hugging Face. The company ran a two-track game: open-source the small Qwen3 and Coder lines under Apache 2.0, keep the crown jewel closed. Qwen3.8-Max is the first time the crown jewel itself got opened. That's the pattern break, and it lands the same month Moonshot and Thinking Machines opened models of their own. The capability claim is where you keep your guard up. Alibaba calls it "second only to Fable 5." That is the vendor's own internal evaluation. Not a benchmark table. Not a third-party number from Artificial Analysis or LMArena. Positioning, not a result. Keep the two in separate buckets until independent evals land. What is real: a multimodal model that reads text, images, video, and documents in one system, built on a hybrid of Gated Delta Networks and sparse MoE, speaking 201 languages, tuned for long-horizon coding and the kind of office cowork that used to eat a human afternoon. And now, the weights on disk. You still can't run it at home. A 2.4T model is a datacenter object no matter how hard you quantize it. But that was never the point of open weights at this scale. The point is that the frontier stopped being something only a handful of labs could rent you through a paywall. For a year the best models were rentals. This one, you own.
- 42
@sharbel ·
🚨SHOCKING: Researchers proved that AI models judge your intelligence based almost entirely on what you look like. And a handful of visual cues are doing almost all of the damage. Researchers built 500 photorealistic faces. Then they changed one thing at a time. Then they showed these faces to six of the most widely deployed multimodal AI models in the world and asked them to make social judgments. The models did not hesitate. They judged. Over and over. Across 25 different scenarios. One model, shown an older face, rated that person as significantly less competent than the identical face made younger. The same face. The same expression. The same background. Different age. Different verdict. The researchers wrote: "A few human visual cues drive most social biases in MLLMs." But this is not a story about faces. These models are being deployed right now in hiring pipelines, loan approvals, medical triage, and criminal risk assessments. They are being handed photographs of real people and asked to make consequential decisions. The researchers did not test hypothetical future systems. They tested the models companies are shipping today. What happens when the systems deciding your future learned to see your face the way the worst hiring manager in history would?
- 43
- 44
- 45
@OpenBMB ·
Visual comprehension requires high-level abstract semantics, while image generation demands fine-grained pixel details. How can we resolve this fundamental conflict within a single unified model? 🤔 Today, we present CHEERS—new research from @TsinghuaNLP (OpenBMB member), XJTU, and UCAS: a novel unified multimodal model that decouples patch-level details from semantic representations, achieving SOTA comprehension and generation simultaneously. 🤗 Paper: https://t.co/HdNQLL63Pa 📄 arXiv: https://t.co/9lAOoyGakl 💻 Code: https://t.co/7k05AvyRYv 🤖 Models: https://t.co/nHa1spLPsE Why it matters: 1️⃣ Decoupling Semantics & Details: Traditional unified models struggle to balance conflicting goals. CHEERS introduces a Unified Vision Tokenizer that rebuilds pixels before extracting semantics to prevent information loss, while achieving a 4x visual token compression for extreme LLM efficiency. 🧩 2️⃣ Unified LLM Transformer: Built on Qwen2.5-1.5B, CHEERS seamlessly integrates two decoding regimes. It uses autoregressive decoding for text (causal mask) and flow matching for images (bidirectional attention), processing heterogeneous tokens in a single shared feature space. 🔄 3️⃣ Cascaded Flow Matching: It mimics human painting! A two-stage cascaded head first generates a low-res semantic sketch, then a gated network dynamically injects high-frequency detail residuals from the vision tokenizer, yielding breathtaking fidelity. 🎨 4️⃣ Doing More with Less: CHEERS outperforms similar-sized models like Janus-Pro and Show-o2 on comprehensive benchmarks (MMBench, GenEval, etc.). Incredibly, it achieves this SOTA performance using only 83M training samples—just 20% of the data required by its peers! 🚀 CHEERS redefines unified multimodal modeling, proving that intelligently separating semantics and details is the key to mastering both understanding and creation. #AI #THUNLP #OpenBMB #LLM #Multimodal #ImageGeneration #ComputerVision
- 46
@burkov ·
Most systems that handle different kinds of data—text, images, video, audio—do so by training a separate encoder for each type and then forcing their outputs into a shared space, which works for matching an image to a caption but loses the connections that arise when those types appear together, as in a video with narration or a document mixing charts and prose. This recent work from Google takes a model already trained to understand all these types at once (Gemini) and adapts it into an embedding model, meaning one that turns any input into a single fixed-length vector of numbers whose geometric closeness reflects semantic similarity, so that related items end up near each other regardless of which modality they came from. The authors do this through contrastive training, where the model is shown matched pairs and learns to pull them close while pushing unrelated items apart, run in stages across many tasks and modalities, and they report that the single resulting model matches or beats specialized systems on text retrieval, image-text matching, video search, code search, and audio retrieval. Two specifics stand out: feeding raw audio directly into the model retrieves text better than first transcribing the audio into words and embedding that; and the model holds up across narrow fields it was not specifically trained on, from microscopy images to paintings to recipes, where competing models swing between strong and weak depending on the domain. Read with an AI tutor: https://t.co/RL1J9jkagC PDF: https://t.co/ho8YfyHQAe
- 47
@shawnchauhan1 ·
Google just made multimodal search infrastructure a commodity. Text, image, video, audio, PDF - one vector space, one API call. Startups have been raising on the premise that unified multimodal retrieval is hard to build. It was. Until yesterday. The question is not whether Google can do this. The question is what business models quietly just broke.
- 48
- 49
@did0f ·
What if the most important AI benchmark is not intelligence, but locality? Not, “Is this model smarter than the biggest cloud model?” But, “Can it run well on my machine?” That is why I find Gemma 4 12B interesting. Not because it magically solves multimodality. It does not. But because it makes local multimodal AI feel more realistic. Most multimodal models are stitched together. 🔤 A language model handles text. 👁️ A vision encoder translates images. 🗣️ A speech encoder translates audio. That approach works, but it is heavy. Gemma 4 12B takes a different route. People often call it “encoder-free,” but that phrase needs some precision. It does not mean images go straight into the model without any adaptation. There is still an adapter. The difference is that instead of relying on a heavy encoder that tries to interpret the image upfront, Gemma uses a much simpler step: a linear projection. In plain English, the image is split into patches. Those patches are converted into vectors with the right shape, then passed into the same internal space where the model already works with text. A thinner layer changes the format. The backbone does more of the reasoning. That matters. Because every piece you remove makes local AI less fragile. → Less memory. → Less latency. → Less cloud dependency. And that is the part I care about.
- 50
@LEAPTRADER_ ·
$NVDA has launched Nemotron 3 Nano Omni, an open-source multimodal model that natively combines video, audio, image, and text reasoning in a single efficient system. ✅ 30B MoE model (3B active parameters) with a hybrid Transformer-Mamba architecture. ✅ Handles long-context inputs (up to 256K tokens) and processes mixed modalities without chaining separate models. ✅ Designed specifically for agentic AI workloads such as computer-use agents, document intelligence, video/audio analysis, and GUI understanding. This gives $NVDA a stronger foothold in the software layer beyond GPUs by open-sourcing the model, enabling faster, lower-latency multimodal agents that run efficiently on NVIDIA hardware.
Best Tweets by Topic