LLM Evaluation & Regression
Offline and online evaluation systems: golden datasets, regression testing, LLM judges, human review, and evaluation-driven development.
56%
Best tweets about AI Observability
50 Best Tweets About AI Observability (2026)
Find the best tweets about AI observability, including LLM tracing, evaluations, monitoring, prompt analytics, cost, latency, and production reliability.
Production AI and LLM observability, tracing, evaluation, monitoring, prompt analytics, cost, latency, incidents, tools, and engineering practices.
- Creators
- 43
- Updated
Original Xholic analysis
What 50 top AI Observability posts reveal
The supplied AI-observability discussion emphasizes production instrumentation, especially tracing, evaluation, and measurement of cost, latency, quality, and reliability. Posts also pair observability with recovery controls and increasingly stress replayable regression testing. A cautionary strand argues that automated evaluation and conventional traces should be supplemented with domain judgment and auditable execution evidence.
- Dominant tone
- Neutral
- Median score
- 12.9
- Leading format
- List
- Recent posts
- 58%
34% of posts
All-time engagement
44% of posts
Published in 90 days
Conversation map
The themes creators return to
Tracing & Instrumentation
End-to-end traces, spans, OpenTelemetry instrumentation, and debugging across RAG, model, tool, and agent workflows.
36%
Governance, Auditability & Security
Governance, audit trails, inference evidence, compliance, privacy, policy enforcement, and security controls for AI operations.
34%
LLMOps & Production Architecture
LLMOps and production architecture practices: prompt/version management, deployment, routing, orchestration, durable runtimes, and operational stacks.
32%
Production Monitoring & Metrics
Production metrics and monitoring for latency, tokens, throughput, cost, quality, drift, reliability, and agent behavior.
32%
Cost, Latency & Inference Optimization
Cost and performance optimization across inference and agents, including token budgets, caching, model routing, serving, and latency-quality tradeoffs.
20%
Reliability & Incident Response
Reliability engineering for AI systems, including fallbacks, retries, circuit breakers, rate limits, failure handling, and incident response.
18%
Feedback, Replay & Improvement
Using production logs, feedback, replayable trajectories, and targeted sampling to diagnose failures and continually improve agents.
10%
Tone and stance
Sentiment Neutral leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 35
- Median reposts
- 4
- Median replies
- 7
- Median views
- 2.6K
Posts with media make up 68% of this collection. Their median all-time score is 11.8, compared with 19.6 for text-only posts.
Format mix
- List 44% · score 38.8
- Announcement 24% · score 11.9
- Opinion 18% · score 5.07
- Tutorial 14% · score 9.23
Consensus and debate
Where creators agree, and where they do not
Shared view
Observability is framed as production instrumentation
Posts commonly frame observability as production instrumentation: end-to-end or per-stage traces can be paired with token, cost, latency, quality, reliability, and feedback signals. The examples describe tracing as a way to inspect RAG and LLM workflow steps, rather than only a dashboarding function.
Shared view
Evals are presented as regression discipline
Several posts advocate an evaluation loop in which real failures become test cases, changes are compared before and after, and prior tests are rerun to detect regressions. These posts present traces, metrics, evaluation results, and replayable failures as inputs to merge or improvement decisions.
Shared view
Reliability guidance includes recovery and diagnosis
Reliability guidance repeatedly combines observability with operational controls such as retries, fallbacks, circuit breakers, rate-limit monitoring, and graceful degradation. The posts position traces and metrics as aids to diagnosing failures in these workflows.
Open debate
Automation is presented as useful but insufficient on its own
The posts take a qualified view of automated evaluation. One presents G-Eval as a more procedural, reproducible alternative to blunt LLM ratings; another reports that automated trace-review tools can miss issues requiring domain expertise or taste; and a third reports that AI monitors caught dangerous hidden-data attacks less than half the time in the cited study. Together, they support human scrutiny and careful use
Open debate
Logs and traces versus execution evidence
A governance-oriented set of posts distinguishes ordinary logs and traces from evidence intended to establish what executed. It argues for auditable events, execution evidence, policy enforcement, and per-request records when systems trigger consequential actions.
What performs
Patterns behind standout posts
Practical explainers appear among the supplied score outliers
The supplied benchmark identifies these five tweets as score outliers, with all-time scores from 240.53 to 2,457.15. List is the largest supplied format group (22 tweets, 44%) and has a 38.76 median all-time score; however, the evidence does not establish that every outlier is a list.
Feedback and replay has the highest supplied theme median
Feedback, Replay & Improvement is the smallest named theme by volume (5 tweets; 10%) but has the highest supplied theme median all-time score, 46.533. Its cited posts discuss replayable learning environments, automated-evaluation review, and signal-based selection of trajectories for human review.
Monitoring and metrics combines scale with a strong theme median
Production Monitoring & Metrics accounts for 16 tweets (32%) and has a supplied median all-time score of 39.95. Its evidence tweets cover distributed tracing, cost attribution, latency monitoring, drift detection, and production operational concerns.
Statistical standouts
- View standout post 1 Score 2457.2 · 190.33× median
- View standout post 2 Score 1308.5 · 101.36× median
- View standout post 3 Score 614.8 · 47.62× median
- View standout post 4 Score 403.0 · 31.21× median
- View standout post 5 Score 240.5 · 18.63× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Abhishek Singh
@0xlelouch_
2 posts
-
2. ambient.xyz
@ambient_xyz
2 posts
-
3. Aurimas Griciūnas
@Aurimas_Gr
2 posts
-
4. Hugo Bowne-Anderson
@hugobowne
2 posts
-
5. Inference Labs
@inference_labs
2 posts
-
6. Paul Iusztin
@pauliusztin_
2 posts
Instrumentation and measurement guidance
Aurimas Griciūnas’ posts describe a RAG trace in spans and recommend capturing timing, inputs and outputs, token counts, retrieval context, latency, cost, quality, reliability, and agent-behavior metrics.
Architecture and roadmap framing
Tech with Mak’s roadmap-style posts place observability alongside evaluation, prompt management, security, routing, MLOps, and inference optimization in broader AI-engineering production stacks.
Evaluation-driven workflow advocacy
Paul Iusztin’s posts emphasize evaluation learning resources and evaluation-driven development, including test datasets, before/after comparisons, traces, metrics, and regression detection before merging changes.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best AI Observability tweets
Top AI Observability tweets from 43 creators
Ranked 01–50
- 01
@techNmak ·
Someone just dropped a 9-layer production AI architecture and it's the most honest breakdown I've seen. services/ - RAG pipeline, semantic cache, memory, query rewriter, router. Not one file. Five. agents/ - document grader, decomposer, adaptive router. Self-correcting by design. prompts/ - versioned, typed, registered. Never hardcoded. security/ - input, content, output. Three guards not one. evaluation/ - golden dataset, offline eval, online monitor. Most people skip this entire layer and ship blind. observability/ - per-stage tracing, feedback linked to traces, cost per query. .claude/ - agent context so your AI coding assistant knows the codebase before it touches a file. The demo is one file. Production is this.
- 02
@pauliusztin_ ·
Every day, 100+ people ask me, "How can I learn AI evals?" I copy-paste these 11 links (every time): 1. AI evals & observability (series): https://t.co/erSJcqpAV7 2. Using LLM-as-a-judge: https://t.co/xMBt9j4JRc 3. Demystifying evals for AI agents: https://t.co/HBbCe5PnXJ 4. There are only 6 RAG Evals: https://t.co/gwfyhIozqK 5. Evaluation-driven development: https://t.co/GMtp6bewol 6. Binary evals vs. Likert scales: https://t.co/WyMw1hHTfm 7. The mirage of generic AI metrics: https://t.co/ugryF5zfKO 8. Error analysis: https://t.co/OXgPZd8IXi 9. Carrying out error analysis: https://t.co/OXgPZd8IXi 10. Evaluating the effectiveness of LLM-evaluators: https://t.co/NuaXhr19TV 11. LLM judges aren't the shortcut you think: https://t.co/fDep2HFjCq Binge these to skyrocket your skills.
- 03
@techNmak ·
Learning AI engineering in 2026. What most people do: → Jump to agents → Skip foundations → Ignore MLOps → Wonder why nothing works What this roadmap shows: 1. Foundation (Python, APIs, clean code) 2. Semantic intelligence (embeddings, vector DBs) 3. RAG (grounded outputs) 4. Agents (autonomous workflows) 5. MLOps (CI/CD, containers) 6. Evaluation (metrics, drift, bias) 7. Inference optimization (quantization, caching) 8. Portfolio 9. Specialization The sequence builds on itself. Skip steps and you'll hit walls later. Image credits - @agenticgirl (Follow her)
- 04
@suraj_sharma14 ·
Want to be a Backend Architect in July 2026. Please learn. 1. Agentic System Design Service decomposition for agents, bounded contexts for workflows, resilience patterns for non-deterministic systems. 2. Distributed AI Infrastructure Container orchestration for inference, Kubernetes for agents, multi-model routing, edge deployment strategies. 3. Hybrid Data Architecture Vector databases, traditional SQL/NoSQL, embedding pipelines, data versioning, point-in-time correctness for ML. 4. Agent Interface Design MCP protocol, structured tool calling, API gateways for agents, asynchronous command queues, webhook orchestration. 5. Event-Driven Agent Workflows Kafka for agent triggers, pub/sub for multi-agent communication, saga patterns for long-running agent transactions. 6. AI Observability and Tracing Distributed tracing for agent steps, cost attribution per request, latency monitoring, drift detection, eval pipelines. 7. AI Security and Guardrails Prompt injection defense, PII redaction pipelines, zero trust for agent tool access, audit trails for autonomous actions. 8. Infrastructure as Code for AI Terraform for vector infra, Helm charts for model serving, configuration management for prompt versions and model weights. 9. Scaling and Cost Optimization Horizontal scaling for stateless agents, KV cache optimization, token budgeting, model routing by cost and latency. 10. Reliability and Recovery Circuit breakers for LLM failures, retry logic with exponential backoff, fallback models, graceful degradation patterns. Most people stay stuck watching tutorials. Builders get hired.
- 05
@Aurimas_Gr ·
𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is a must have in your tool belt as an AI Engineer. 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 sits at the core of it, why is it important? Tracing and instrumentation of software have been around for decades now. With AI systems resembling regular software even more, we are now moving the practice here as well (with a few key differences). Let’s look into the process of tracing from a perspective of a naive RAG system. 𝘍𝘦𝘸 𝘥𝘦𝘧𝘪𝘯𝘪𝘵𝘪𝘰𝘯𝘴: 𝘼) An Orchestrator in the GenAI system application is the central piece of software that orchestrates the end-to-end process. Think of apps using LangChain, LlamaIndex or Haystack. 𝘽) Trace is the end-to-end application flow from the entry point till the answer is produced, it is composed of smaller pieces called spans. 𝘾) Span is a smaller piece of the application flow that represents an atomic action like a function call or a database query. They can be sequential, or run in parallel. ℹ️ As part of span we capture general metadata like start and end time, inputs and outputs of the span. On top of this metadata we track information specific to the GenAI system elements. What might a trace look like for a naive RAG system? 𝟭. A query that has been submitted to the chat application. 𝟮. The query is embedded into a vector. ✅ Additional metadata like input token count is persisted with the span so that we can estimate the cost of the procedure. 𝟯. ANN lookup performed against the Vector DB to retrieve the most relevant context. ✅ Additional metadata about the query is persisted as part of the span together with the retrieved pieces of context and their relevance. 𝟰. A prompt is constructed from the system prompt and retrieved context. 𝟱. The prompt is passed to the LLM to construct the answer. ✅ Additional metadata about input and output token count is captured together with the span so that we can estimate the cost of the procedure. Learn all of this hands-on in my End-to-end AI Engineering bootcamp. 🎁 (15% off via this link): https://t.co/czhitnDVGT 𝘞𝘩𝘺 𝘪𝘴 𝘵𝘳𝘢𝘤𝘪𝘯𝘨 𝘰𝘧 𝘎𝘦𝘯𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘮𝘱𝘰𝘳𝘵𝘢𝘯𝘵? - These applications are usually complex chains, errors can happen in different steps of your application. E.g. Embedding of query is taking longer than expected or you have reached API limits of LLM provider. - Cost for calling LLM APIs will be variable depending on the length of inputs and produced outputs. You would usually trace this information and analyze it to help forecast expenses. - GenAI systems are non-deterministic and will deteriorate over time. They need to be evaluated on span level rather than input/output of the entire system so that you can tune each piece separately. - … Are you tracing your Agents? Let me know in the comments 👇
- 06
@bibryam ·
🌟 Building Reliable Agentic AI Systems🌟 https://t.co/5yRJLkIsyl - @thoughtworks What it actually takes to build product-ready agents: → Start with bounded workflows, not open-ended autonomy. Agents need clear task boundaries, allowed tools, and explicit stopping conditions. → Treat the LLM as one component in a larger system. Reliability comes from orchestration, state, retries, fallbacks, and observability. → Engineer the context deliberately. The goal is not “more context,” but the right context, at the right step, in the right format. → Use the right retrieval path for the data. RAG works well for unstructured documents; Text-to-SQL is better for structured facts and aggregations. → Make outputs traceable. Serious users need citations, source passages, intermediate steps, and enough evidence to verify the answer. → Add reflection loops, but make them specific. Check process quality, evidence sufficiency, and final answer quality separately. → Design for failure from day one. Agents will hit bad retrieval, malformed tool calls, ambiguous questions, and partial data. → Evaluate continuously. Offline test sets are useful, but live-traffic evaluation is where product quality actually shows up. → Keep humans in the loop where risk is high. Product-ready does not mean fully autonomous; it means trustworthy within the workflow.
- 07
- 08
- 09
@_avichawla ·
DevOps vs. MLOps vs. LLMOps: Many teams are trying to apply DevOps practices to LLM apps. But DevOps, MLOps, and LLMOps solve fundamentally different problems. DevOps is software-centric. You write code, test it, and deploy it. The feedback loop is straightforward: Does the code work or not? MLOps is model-centric. Here, you're dealing with data drift, model decay, and continuous retraining. The code might be fine, but the model's performance can degrade over time because the world changes. LLMOps is foundation-model-centric. Here, you're typically not training models from scratch. Instead, you're selecting foundation models and then optimizing through three common paths: - Prompt Engineering - Context/RAG Setup - Fine-Tuning But here's what really separates LLMOps: The monitoring is completely different. In MLOps, you track data drift, model decay, and accuracy. In LLMOps, you're watching for: - Hallucination detection - Bias and toxicity - Token usage and cost - Human feedback loops This is because you can't just check if the output is "correct." You need to ensure it's safe, grounded, and cost-effective. The evaluation loop in LLMOps also feeds back into all three optimization paths simultaneously. Failed evals might mean you need better prompts, richer context, OR fine-tuning. So it's not a linear pipeline anymore. One more thing: prompt versioning and RAG pipelines are now first-class citizens in LLMOps, just like data versioning became essential in MLOps. And the ops layer you choose should match the system you're building. 👉 Over to you: What does your LLM monitoring stack look like right now? ____ Find me → @_avichawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
Watch video
- 10
@arpit_bhayani ·
New write-up is live, and this time I covered G-Eval. It helps answer one important question: how do you know whether what an LLM generated is apt, correct, and aligned with your requirements? G-Eval is a pretty simple framework that leverages Chain-of-Thought prompting over clearly defined rubrics. Instead of simply asking an LLM to "rate this post" or "is this correct?", it makes the evaluation process more procedural and reproducible. If you bluntly ask an LLM to rate something on a scale of 1-5, it tends to exhibit a bias toward picking certain values, lacks a clear audit trail, and often suffers from a bias toward its own model family. G-Eval solves this neatly. In this blog, I covered G-Eval in detail and showed how to adopt it in your AI workloads to actually measure whether your LLM outputs are getting better. If you are looking for a reliable blueprint for LLM evaluation, this guide should help. Give it a read, and like always, I hope this helps.
- 11
- 12
@businessbarista ·
Loved this 22-minute talk on continual learning for AI agents. Must watch for anyone looking to get agents performant and into production. Credit: @FeiziSoheil at @aiDotEngineer • Agent learning can happen at three layers: the model (weights), the harness (prompts, tools, skills, code, workflows), and memory (session or persistent). • Two fundamental challenges: (1) getting feedback, meaning how do we know if the agent did well and what it should have done instead, and (2) acting on that feedback, meaning deciding which layer or component to change and how. • Feedback sources differ by stage: In development you have benchmarks with evaluators that score pass/fail. In production you only have logs, which can be judged either automatically (LLMs or code analyzing the log, which is scalable) or by human experts (low volume but critical domain knowledge). • Logs plus feedback aren't enough because they're not testable: A single log with feedback is one observation of what happened. You need to lift it into a replayable learning environment, a simulation with tools, users, and defined evaluators, so candidate fixes can be run, verified, and compared. • Three ways to optimize the agent, with tradeoffs: Model-layer updates (SFT, RL post-training like DPO/GRPO, LoRA) are expensive and need benchmarks and evaluators. Harness updates (trace-to-harness coding agents, prompt search like GEPA) are flexible but either untestable and "vibe-based" or benchmark-dependent. Memory updates (fact storage like Letta/Mem0, skill distillation) are cheapest and fastest but usually unverified. • A good learning engine makes "the smallest durable change at the right layer" of the agent. • Verifiable continual learning (VCL): Improve an agent from its own experience where every fix is proven to help and proven to break nothing that already worked. It requires an executable test (replayable failure), a measured delta (score before and after), and regression tests (prior tests still pass). • Four principles of practical VCL: Replayability (turn one-off failures into rerunnable tests), holisticness (one failure can have causes in memory, prompts, tools, workflow, or model, so route the fix to the right layer), lifelongness (fix new failures subject to no regression on past environments, with regression handled inside the optimization loop rather than post-hoc), and efficiency (the loop must run frequently and cheaply, without scaling linearly as past environments accumulate). • Three takeaways: (1) Agent continual learning isn't necessarily fine-tuning; many useful updates live in the harness and memory layers. (2) Production logs are not learning environments and must be transformed into replayable ones. (3) The frontier is regression-aware improvement: fixing new failures while verifying you don't break old ones.
Watch video
- 13
@HamelHusain ·
New Blog Post: Do Automated Evals Work? There has been a rise of tools that look through your traces with AI and identifies issues. We tested these tools with real production data to see how good they are. Where they shine - They often spot issues human miss - Integrate into your workflow: viewing traces, creating LLM judges etc. Where they fall short - They miss problems that require domain expertise and taste - Don't have great mechanisms to learn from human feedback - You can get similar results from using your coding agent So you should use them? Yes, BUT do so iteratively with you in the loop. We describe how in the post: https://t.co/WrStw4kglF It's also a good idea to try using your coding agent with you in the loop, which we discuss in the post. This was written with @doesdatmaksense , who led the research and collated the results.
- 14
@Aurimas_Gr ·
This is how you measure your AI system as an AI Engineer 👇 For regular software you would track metrics like uptime, error rate, p95 latency. However, they say little about whether the system is fast where users feel it, affordable at scale or correct. Here are the metrics we track when building LLM systems. It is useful to group them by the question they answer: 𝟭. 𝗜𝘀 𝗶𝘁 𝗳𝗮𝘀𝘁? (𝗟𝗮𝘁𝗲𝗻𝗰𝘆) ➡️ Time to first token (TTFT): how long the user is exposed to a blank screen, the number that defines perceived latency. ➡️ Inter-token latency (ITL): how smoothly tokens stream after the first one. ➡️ End-to-end latency at p50 / p95 / p99, dominated by output length, track it per use case rather than globally. 𝟮. 𝗖𝗮𝗻 𝗶𝘁 𝘀𝗰𝗮𝗹𝗲? (𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝗻𝗱 𝗰𝗼𝘀𝘁) ➡️ Tokens per second per user vs total system throughput, the two trade off against each other on the same hardware. ➡️ Input and output tokens per request to measure your unit economics. ➡️ Cache hit rate - prompt caching is often the technique that reduces cost the most. ➡️ Cost per successful task, not cost per request, a cheap request that fails is a waste. 𝟯. 𝗜𝘀 𝗶𝘁 𝗰𝗼𝗿𝗿𝗲𝗰𝘁? (𝗤𝘂𝗮𝗹𝗶𝘁𝘆) ➡️ Task success rate on a labeled eval set, re-run on every prompt or model change. ➡️ Groundedness for RAG - is the answer supported by the retrieved context. ➡️ Retrieval precision@k and recall@k - generation cannot fix what retrieval never surfaced. ➡️ LLM-as-judge scores over time, calibrated against human labels. ➡️ User feedback signals: thumbs, edits to generated output, free form feedback. 𝟰. 𝗗𝗼𝗲𝘀 𝗶𝘁 𝗵𝗼𝗹𝗱 𝘂𝗽? (𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆) ➡️ Error, timeout and rate-limit rates per provider. ➡️ Retry and fallback rate - how often you silently switch to a backup model. ➡️ Guardrail trigger and refusal rates. 𝟱. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝗯𝗲𝗵𝗮𝘃𝗲? (𝗔𝗴𝗲𝗻𝘁 𝗺𝗲𝘁𝗿𝗶𝗰𝘀) ➡️ Tool-call error rate. ➡️ Steps and tokens per completed task - drift here means cost is rising while accuracy remains the same ➡️ Context window utilization - the early warning for compaction and truncation issues. ❗️ Latency and reliability show up on day one because standard infra emits them. Quality, cost per task, and agent behavior need deliberate instrumentation, and they are where AI systems fail in production. Which metric caught a real problem for you that the standard dashboards missed? 👇
- 15
@akshay_pachaar ·
A great LLM interview question: (answer shared below) You have 80k Agent-user interactions from production. You need to find the top 100 worth reviewing to improve the agent. You cannot use an LLM to evaluate them since it will be expensive. This is one of the most painful problems in shipping agents. Every team sits on a goldmine of production trajectories, but the bottleneck is figuring out which ones are worth a human's attention. Run an LLM judge over all 80k and your bill explodes. Review random samples manually and you burn hours on conversations where nothing interesting happened. So how do you actually pick the right 100? The simplest answer is random sampling. Pick 100 random trajectories and review them. But most production agents handle routine requests just fine, so you waste a big chunk of your annotation budget on uninformative conversations. Another approach is filtering for longer conversations, since 10+ user messages usually means more complexity. But longer conversations skew heavily toward outright failures. You surface obvious breakdowns but miss the subtle issues hiding in conversations where the agent technically succeeded. A recent paper from DigitalOcean takes a smarter approach. It computes lightweight behavioral signals directly from the trajectory data using deterministic rules. The signals fall into three groups. 𝟭) 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 come from the user-agent dialogue. → User rephrasing the request or correcting the agent is misalignment. → Agent producing near-duplicate or circular responses is stagnation. → User asking to "talk to a human" or abandoning the session is disengagement. → User confirming something worked is satisfaction. These are detected through normalized phrase matching, similarity checks, and simple discourse heuristics. 𝟮) 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 come from tool calls and runtime events. → A tool call that doesn't advance the task is a failure signal. → Repeated calls with identical or drifting inputs indicate a loop. These are straightforward to extract from execution logs. 𝟯) 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 cover rate limits, context overflow, and API errors. These are useful for diagnosis but not for training, since they reflect system constraints, not agent decisions. Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review. On τ-bench, the authors compared all three approaches on 100 trajectories. - Random sampling hit a 54% informativeness rate. - The length-based heuristic reached 74%. - Signal-based sampling reached 82%. This means roughly 4 out of every 5 sampled trajectories are genuinely useful for improving the agent. The bigger win shows up in successful trajectories. Among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs 41.3% for random. These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don't break the task but still matter for optimization. The whole framework runs without any LLM overhead and can sit always-on in a production pipeline. If you want to see this in practice, the signal-based approach is already integrated into Plano, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place. Plano GitHub repo: https://t.co/BD0WXJYrrz Research Paper: https://t.co/6s98KnVxXO 👉 Over to you: What's your approach to this problem?
Watch video
- 16
@nikks_techie ·
AI tool of the day: LiteLLM What it is LiteLLM is an open-source gateway that provides a single, OpenAI-compatible API for 100+ LLMs and providers, including OpenAI, Anthropic, Gemini, Grok, Ollama, Bedrock, and Azure OpenAI. Benefits Switch between LLM providers with minimal code changes Avoid vendor lock-in Built-in fallback and failover support Track usage, costs, and latency Centralize model access across teams Load balancing and rate limiting for production systems How to use it Install LiteLLM or deploy the proxy server. Configure API keys for your LLM providers. Point your application to the LiteLLM endpoint. Use the same API format regardless of the underlying model. Add routing rules, fallbacks, and monitoring as needed. Best for Agentic AI applications Multi-model architectures Enterprise AI platforms Cost optimization LLM experimentation and evaluation Real-world example Suppose your application normally uses GPT-5. If the provider experiences high latency or an outage, LiteLLM can automatically route requests to Claude, Gemini, or another model without requiring application changes, helping keep your service available.
- 17
@panditdhamdhere ·
After researching a lot and experimenting trying out, I created an AI Engineer Roadmap for myself. ( this is for these who are already developer not beginner ) Phase 1 - Foundations ➜ Python for AI & Dev Setup Get your environment ready. Master Python data structures, list comprehensions, and virtual environments used constantly in AI work. ➜ How LLMs Actually Work Understand transformers, tokens, context windows, temperature, and why prompt engineering isn't magic, it's input engineering. ➜ APIs & First AI Integration Call your first LLM API. Learn about auth, rate limits, streaming responses, and error handling. Build a simple chatbot script. Phase 2 - Core AI Engineering Skills ➜ Prompt Engineering & Evals System prompts, few-shot examples, chain-of-thought, output formatting. Learn to evaluate LLM outputs systematically. ➜ RAG - Retrieval Augmented Generation Give your AI long-term memory. Embeddings, vector databases, chunking strategies, and semantic search pipelines. ➜ Tool Use & Agentic Patterns Function calling, tool use, and building agents that can browse the web, run code, and call external APIs autonomously. Phase 3 - Production & Infrastructure ➜ API Design & Backend Wrap your AI logic in a FastAPI service. Handle async, background tasks, auth, caching, and rate limiting like a real engineer. ➜ Cloud Deploy & Observability Deploy to AWS/GCP/Azure. Container with Docker, CI/CD pipelines, LLM-specific monitoring latency, cost, and hallucination tracking. Phase 4 - Ship & Specialise ➜ Build a Full AI Project Ship a complete end-to-end project - RAG chatbot, AI coding assistant, or autonomous research agent. This is your portfolio piece. ➜ Specialisations & Next Steps Pick your lane - fine-tuning & PEFT, multimodal AI, AI security/red-teaming, or enterprise AI systems. Map out your next plan.
- 18
@jahirsheikh8 ·
As an AI Product Engineer. Please learn: - Prompt design beyond basic prompting - Structured outputs / JSON schemas - Context window management - RAG UX / retrieval tuning - Tool selection / orchestration logic - Guardrails / moderation / safety layers - Cost / latency tradeoffs - Evaluation pipelines Shipping AI demos is easy. Shipping AI products users trust is hard.
- 19
@milan_milanovic ·
𝗧𝗵𝗲 𝗔𝘇𝘂𝗿𝗲 𝗔𝗜/𝗠𝗟 𝘀𝘁𝗮𝗰𝗸 Here are the most important Azure services if you want to work with AI in Azure. 𝟭. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 We can use Azure ML as the platform for managing experiments, compute clusters, and the model lifecycle. GPU VMs (NC/ND series) for training workloads that actually need the hardware. AKS when you're running inference at scale and need control over the infrastructure. Container Instances for short jobs where a full cluster is overkill, Functions for lightweight event-driven inference, and Batch for parallel workloads you want to run on a schedule. 𝟮. 𝗗𝗮𝘁𝗮 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 Data Lake Gen2 for training data at scale. Blob Storage for everything else. Azure SQL and Cosmos DB, depending on whether your metadata is relational or document-shaped. Managed Disks if your workload needs fast local storage attached to a VM. Azure Files for shared access across nodes. 𝟯. 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝗧𝗟 Data Factory for orchestrating pipelines. Synapse for large-scale analytics. Databricks, when your team lives in notebooks and needs distributed compute on Spark. Stream Analytics for real-time processing on incoming data. Functions and Event Hubs cover the lighter ETL cases. In practice, most teams end up with Databricks and Data Factory and call it done. 𝟰. 𝗠𝗟 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 AML Pipelines for reproducible training workflows, AML Studio, and Notebooks for the exploration and iteration phase. AutoML is when you want to benchmark quickly without writing the training code yourself. Prompt Flow for LLM-based applications. MLOps handles versioning, deployment, and monitoring across the lifecycle. 𝟱. 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Feature Store, Databricks FE, Data Explorer. The most underbuilt layer in most ML projects I've seen. Teams spend months on model architecture and three days on features. The ratio should probably be reversed. 𝟲. 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗮𝗻𝗱 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 Online Endpoints for real-time inference, Batch Endpoints when latency doesn't matter, and throughput does. AKS Deployment when you need full control over scaling and infrastructure. ACI for simpler containerised deployments. ONNX Runtime for optimised inference across hardware, Azure Edge AI, when the model needs to run closer to the data source. 𝟳. 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗮𝗻𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 ML Pipelines for training automation, Data Factory for data movement, Logic Apps for integration workflows. Functions and DevOps Pipelines for the CI/CD side. A model that runs reliably in production on a schedule looks nothing like a notebook; this is the layer where that gap gets closed, and most teams leave it too late. 𝟴. 𝗟𝗟𝗠 𝗮𝗻𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗜 Azure OpenAI for GPT-4 and embeddings with enterprise controls. Phi Models when you need something smaller and cheaper to run. Llama and Mistral through the model catalog if you want to open weights with more control over where they run. Cognitive Services for pre-built capabilities like vision, speech, and language. Azure AI Foundry is the newer unified platform for building and managing AI applications. Prompt Flow for chaining and testing LLM calls. 𝟵. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴, 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 𝗧𝗼𝗼𝗹𝗶𝗻𝗴 Azure Monitor and App Insights for observability, ML Monitoring for model drift and data quality in production. Azure Policy and Purview for governance, Entra ID for identity. On tooling: Azure CLI, SDKs, Azure DevOps, GitHub Actions, Bicep/ARM for infrastructure as code. And GitHub Copilot, of course, for AI-assisted coding.
- 20
@AIHighlight ·
🚨BREAKING: A new open-source tool just made it possible to actually measure when an AI model is misaligned. This sounds boring on paper. It is not. The reason every team building with LLMs has the same misalignment problem is that nobody has had a real way to put a number on it. Engineers know their models hallucinate, get manipulated, and answer questions outside their training with confidence that has nothing to do with accuracy. They have all been burned by it in production. There has never been a standard way to score how often it happens, and there has never been a way for two different teams to run the same evaluation and get the same answer. iFixAi is the first tool that takes this seriously. It runs 32 different inspections against any AI model, sorts the failures into five categories that actually mean something (fabrication, manipulation, deception, unpredictability, and inability to admit what it does not know), and produces a scorecard with a letter grade. Every input that produced the score gets written into a manifest file. Hand that file to anyone in the world and they can reproduce your score exactly. That last part is the thing other evaluation tools have not solved. Most published AI benchmark results cannot actually be reproduced by anyone who was not in the room when the original test ran. Prompts get changed. Models get updated without anyone noticing. The test from last month is not the same test as the one from this month. The scores get reported anyway. iFixAi pinned all of it. If two teams run the same test, they get the same score. That is the whole point. A few details that make this project worth trusting. The model under test is structurally prevented from grading its own work. There is always an independent judge, and in full mode there are multiple judges with the disagreements visible. Two of the inspections are mandatory minimums, meaning if the model fails them, the whole score is capped at 60 regardless of how it does on the other 30. And there is one specific test (B12) that is deliberately not a mandatory minimum because its corpus is public and frontier models may have been trained on it. Most evaluation tools quietly hope nobody notices this kind of contamination problem. iFixAi names it. The honest framing from the team behind this matters. They say up front that the letter grade is a drift signal, not a certification. It tells a team whether their model got better or worse since the last deployment. It does not tell anyone whether the model is "aligned" in some absolute sense, because nobody has figured out what that even means yet. That kind of restraint is rare in this space. It is free. It is Apache 2.0. The repository is at https://t.co/7DVZtj5DDO and the homepage is at https://t.co/AgwkJ8bM6X. Anyone building with LLMs can run a full diagnostic against their deployment in five minutes with one command. For anyone who has ever wondered whether their model is getting better or worse and had no way to tell, this is the tool that answers the question. Repository: https://t.co/V5LE17ShcK Homepage: https://t.co/AgwkJ8bM6X
- 21
@DeepStarts ·
Here's a list of Al Engineer Interview questions + concepts you need to know (from Al/ML Engineering Manager perspective) LLM Fundamentals: -What is tokenization, and how does it affect generation? -How do embeddings really work? -What's the role of attention, positional encoding? -What changes during fine-tuning? (optimizers, schedulers, layer freezing) -LoRA vs QLoRA vs full fine-tune - tradeoffs? Prompting & Context Engineering: -Few-shot vs zero-shot - which works better where? -How do you design system prompts that are robust across users? -How do you make output deterministic? -How do you track, version, and backfill changing context? -How do you build/maintain the memory? RAG Systems: -What's your chunking strategy by length, semantics, or structure? -How do you choose a vector DB (Chroma, Pinecone, OpenSearch...)? -Can you update or backfill embeddings with zero downtime? -How do you evaluate retrieval quality (precision@k, reranking, citation)? MLOps & LLMOps: -Sketch a pipeline: from raw data → model-serving → feedback -How would you monitor performance drift hallucinations? -How do you log prompts and outputs for debugging and auditing? -CI/CD for LLM workflows - what's different from ML? Cost & Latency Tradeoffs: -How do you reduce token usage? -When should you quantize a model? -What's your batching + caching strategy to reduce latency? -When to use hosted APIs vs open-source models? System Design Thinking: -How do you make an Al system more deterministic and less brittle? -What fallback do you use if the LLM fails mid-task? -Can you solve this without an LLM or vector DB? -What's the right database for this task - SQL, NoSQL, or vector? What more can I add in this?
- 22
@0xlelouch_ ·
90% of AI Engineering interviews in 2026 come down to these 7 points: 1) Problem framing + success metric Define the target (latency, accuracy@k, cost/request). Say what you’ll measure in prod, not just offline. 2) Data + labeling reality Where does training data come from, what’s PII, how do you handle drift, and what does a bad label do to the model? 3) RAG design (not just “use a vector DB”) Chunking, metadata filters, hybrid search, rerankers, and what happens when retrieval is wrong but fluent. 4) Evals as an engineering system Golden sets, automated regression, offline vs online A/B, and how to stop prompt tweaks from breaking last week’s wins. 5) Serving + performance Batching, caching, timeouts, fallbacks, and how you hit p95 300ms without 10x’ing spend. 6) Reliability + incident response Rate limits, circuit breakers, idempotency, tracing across model calls, and how you debug a 2am spike in hallucinations. 7) Security + compliance Prompt injection paths, data exfil risks, tenant isolation, logging redaction, and what you do when legal says no training on customer text
- 23
@0xlelouch_ ·
90% of LLMOps interviews in 2026 come down to these 7 points: 1) Serving architecture: batching, streaming, timeouts, and backpressure; explain p95 vs p99 and what you do when the model stalls 2) Cost control: token budgets, caching, prompt compression, smaller models; show you can cut a $3k/day endpoint to $800/day without breaking quality 3) Evals: offline golden sets + online A/B; talk about pass@k, win-rate, and how you prevent overfitting to a tiny test suite 4) Retrieval: chunking, embeddings, metadata filters, rerankers; describe how you detect bad recall and fix it without 10x index size 5) Safety + compliance: PII redaction, prompt injection defenses, audit logs; explain what you store, for how long, and why 6) Reliability: retries vs idempotency, circuit breakers, fallbacks; what happens when OpenAI/Azure/Anthropic rate-limits or returns partial outputs 7) Observability: traces across gateway RAG model, token metrics, per-tenant quotas; show dashboards that catch regressions within one deploy cycle
- 24
@rllm_project ·
Excited to release rLLM UI, a real-time observability tool for agent training and evaluation. wandb shows you what's happening. rLLM UI shows you why.
Watch video
- 25
@joulee ·
Designing for trust. When I asked dozens of data leaders how accurate their AI tools were, the answers ranged from 30% to 85%. That’s the Achilles heel of LLMs: they don’t know what they don't know. They speak with all the clarity and confidence of a sales rep in a polished blazer, while making dozens of common assumptions behind the scenes without ever flagging them as assumptions. For some use cases (like coding) you can tell instantly: if the app does what you wanted, it works. For data, judging the quality the output does not work. The query runs, a number comes back, and how can you tell which of 25.5% or 40.5% is the lie? That’s why I’m obsessed with how to design for trust. At Sundial we spend a disproportionate amount of time thinking about how to signal what a high-quality answer looks like, and what factors contribute. Vetted metrics from your most-used dashboards? High trust. Random SQL written on a table that only 2 people used in the past quarter? Maybe take that with a grain of salt. Humans are fine-tuned for trust with other humans. We read it off a hundred signals: track record, tone, hedging, who someone trained under, whether their last claim held up. AI hands you a confident sentence in a chat box and asks you to figure out the rest. There’s a lot of sexy parts of AI building (the speed! The translation magic! The viral demos!) There's a lot of unsexy stuff too: observability, audit trails, structured data. The signals that tell a human when to trust an answer and when to push back. But in operationalizing AI, these elements are what separates a technology from a lived-in tool.
Watch video
- 26
@therealdanvega ·
Spring AI advisors are AOP for your LLM calls. Do something before the request, something after. I built two custom advisors: one that logs which tools are visible to the model, and one that tracks token usage per call with a running total. Full walkthrough here: https://t.co/e4wNZwLYP3
- 27
@xelebofficial ·
Why operating AI agents is becoming the next big challenge? The first wave of Agentic AI was about capability. Can an agent reason? Can it use tools? Can it complete tasks autonomously? The answer is increasingly yes. But a new problem is emerging. Once an agent can act, how do you actually operate it? The breakthrough of cloud computing wasn't just giving developers more compute. It was creating the infrastructure needed to run software reliably at scale. AI agents are reaching a similar moment. An agent may be able to research, analyze, code, communicate, and make decisions. But in production environments, organizations need more than intelligence. They need to know: What is the agent doing? Why did it make that decision? What happened when something failed? How does work continue across sessions? How do multiple agents coordinate safely? Today, much of the agent ecosystem is still solving these problems independently. Teams are building custom memory systems. Custom execution environments. Custom monitoring tools. Custom workflows for agent handoffs and recovery. The same way application infrastructure became a foundational layer for the internet, agent infrastructure is becoming a foundational layer for AI. Standard runtimes, durable memory and state, secure execution, observability, and auditability are no longer nice-to-have features. They are becoming the requirements for turning agents into real products. The next moat may not come from building agents that can do more. It may come from building systems that make agents reliable, transparent, and manageable at scale. Same technology. A different challenge. Not "Can the agent act?" But "Can we operate it?"
- 28
@Al_Grigor ·
LLM systems feel like a new paradigm. In practice, much of the lifecycle still follows patterns that existed long before generative AI. One useful lens is CRISP-DM, a framework originally designed for data mining projects and widely adopted in data science. Even though the tools have changed, its phases map surprisingly well to how modern AI systems are built. Here is how the typical stages compare. 1. Business Understanding - Traditional ML: define the prediction task and success metrics. - AI systems: define the AI-powered product use case and the user experience you want to enable. 2. Data Understanding - Traditional ML: explore labeled datasets, distributions, and features. - AI systems: identify the inputs your system will use such as documents, images, APIs, databases, or external tools. 3. Data Preparation - Traditional ML: feature engineering, cleaning, and dataset curation. - AI systems: chunking documents, generating embeddings, building indexes, and wiring tools for agents. 4. Modeling - Traditional ML: train and tune models on structured datasets. - AI systems: prompt design, schema definition, retrieval pipelines, and agent behavior. 5. Evaluation - Traditional ML: metrics like accuracy, precision, and recall. - AI systems: task success, human feedback, and observable system behavior. 6. Deployment - Traditional ML: model serving and batch or online inference pipelines. - AI systems: full AI-powered applications that combine models, tools, and orchestration. The techniques look different, but the lifecycle remains largely the same. This is one reason many data scientists can smoothly transition into AI engineering roles. Read more about how CRISP-DM applies to AI Engineering: https://t.co/rkQw4fzVmT
- 29
@alex_verem ·
The AI is 5% of the work. The 95% that breaks: → Observability (Langfuse, Braintrust, Helicone) - you can't debug what you can't see → Evals - regression suites for non-deterministic software. The new CI. → Durable runtime (Temporal, Inngest) - so a 10-minute agent run survives a server restart → Guardrails - prompt injection detection, PII redaction, output filtering → Memory layer - vector DBs (Pinecone, pgvector, Turbopuffer), retrieval, session state → Tools layer - MCP servers, sandboxed code execution (E2B, Modal), browser automation (Browserbase) → Auth + multi-tenancy - your agent calling Salesforce for customer A must NEVER see customer B's anything → Cost controls - agents in runaway loops burn $$ in minutes → Human-in-the-loop - approval gates for "spend more than $X" or "send external email" → Prompt versioning - prompts are code, treat them like code → Orchestration - plan-act-observe-repeat. Most serious teams are moving toward minimal orchestration + explicit state machines over heavy frameworks. → Model routing - LiteLLM, Portkey, OpenRouter for fallback, prompt caching, and version pinning so a vendor update doesn't silently change your product A CTO with 10+ years shipping production gave me the honest version: "Observability + evals + durable runtime + guardrails is the minimum viable production stack. Skip those four and you get the works-in-demo → on-fire-in-prod gap killing agent startups right now." The LLM is the easy part. Everything around it is the actual company.
- 30
@pauliusztin_ ·
Every AI feature should answer two questions before it gets merged: 1. Did it improve anything? 2. Did it break anything? Most teams only answer the first question. And that's exactly why regressions keep slipping into production. This is where Evaluation-Driven Development (EDD) comes in. Here's how it works: 1/ Start with an hypothesis Every feature begins with a simple claim: "This change will improve X." 2/ Build the change Create a branch and modify whatever the feature requires. 3/ Run an experiment Before merging, evaluate the new version against a test dataset. Did the feature improve? Did anything regress? 4/ Compare before vs. after A feature improving one metric isn't enough. You must also compare against previous experiments to detect regressions. Because the most dangerous AI failures are silent ones. Everything still runs. But the system quietly gets worse. 5/ Merge only if evidence supports it Every pull request becomes an experiment. Backed by: Traces Metrics Evaluation results This is the core idea behind EDD. It creates an offline validation gate between development and deployment. The more AI agents I build, the more convinced I become that EDD should be mandatory. Because "it runs" is not the same thing as "it works." P.S. I recently sat down with Alejandro Aboy to break down how he implements EDD using Claude Code, Opik (by @Cometml), synthetic datasets, trace generation, and LLM judges. Check it out here: https://t.co/pM6cVK1W99
- 31
@manthanguptaa ·
Four trends that stood out to me at this year's AI Engineer World Fair • Memory, context engineering, context graphs, retrieval, and knowledge system companies were everywhere. It feels like everyone has realized that context, not the model, is the bottleneck. • Local AI is no longer a niche. The demand for ownership, privacy, lower latency, and model portability is becoming much stronger. • The vibe coding hype is cooling off. More talks focused on review loops, evals, testing, and human oversight than on replacing engineers. • "Software factories" is the new buzzword. Instead of asking AI to write code end-to-end, the focus is shifting toward building systems that continuously review, test, evaluate, debug, and improve software throughout its lifecycle.
- 32
@hugobowne ·
AI agents are failing silently in production, and it's costing companies tens of thousands of dollars before anyone notices. Here's what 1,400+ real deployments actually taught us: - The $50k infinite loop: agents confidently report success while spiralling into expensive mistakes. Silent failures are the biggest risk nobody talks about. - Sub-400ms voice agents: Elyos built them by aggressively throwing away context every few seconds. Extreme? Yes. Increasingly standard? Also yes. - DoorDash's three-tier architecture: manager, progress tracker, specialists, with a persistent workspace letting agents collaborate across hours or days. All from AgentOps: Lessons from Over 1,400 Production Deployments of AI Systems, built on the brilliant LLMOPs database by @strickvl (@zenml_io). Full breakdown on my Substack 👇
- 33
@socialwithaayan ·
The biggest risk in AI right now isn't that your agent fails loudly. It's that it fails silently 🤯 There's a free tool called iFixAi that catches exactly that. 45 inspections. Letter grade in under 5 minutes. Any model, any industry. Here's the problem it solves: Your AI agent looks fine in testing. Then in production it fabricates a source. Or leaks a tool it wasn't authorized to use. Or quietly drifts off its objective over a long task. Or underperforms on purpose because it detected it was being evaluated. You don't find out until a customer or a regulator does. iFixAi runs the checks before that happens: → FABRICATION: unsourced claims, overconfidence, missing audit trails → MANIPULATION: privilege escalation, prompt injection, policy violations → DECEPTION: sandbagging, covert side tasks, long-horizon drift → UNPREDICTABILITY: instruction drift, decision instability → OPACITY: regulatory readiness, session integrity, escalation correctness 32 graded core inspections. 13 extended ones for frontier risks like sabotage, sandbagging, and oversight evasion. The part I like most: your model never grades its own homework. iFixAi is model agnostic and judges cross-provider, so you can run your OpenAI agent and have Anthropic score it. Works with OpenAI, Anthropic, Gemini, Bedrock, Azure, Hugging Face, and any OpenAI-compatible . There's now a guided CLI flow and a Claude plugin, so setup takes minutes. Built by iMe. Apache 2.0 License. 100% open source.
- 34
@TheTuringPost ·
OpenAI’s models found a way out of their sandbox and compromised Hugging Face while trying to obtain answers to a cyber benchmark. And on the very same day, a paper came out with an uncomfortable conclusion - why the obvious fix, "add another AI to monitor the agent," is not enough The authors evaluated AI monitors across four automated AI R&D workflows. The most dangerous attacks were hidden inside the training data, and the monitors caught them less than half the time. Even when the monitors were allowed to execute and inspect the final artifact, they still regularly missed sabotage. They focused on surface-level signals, explained away suspicious behavior, or simply ran the wrong tests. The connection is uncomfortable: Long-horizon agents can exploit weaknesses across an entire trajectory, while supervisors still struggle to understand what the final artifact will actually do. Importantly, the agents in this study were explicitly instructed to sabotage. It does not show that models spontaneously develop malicious goals. What it does show is how incomplete current AI monitoring systems still are.
- 35
@WillyChuang ·
Everyone on X feed is still posting "look what the model can do." That conversation is over... This week's summary at @aiDotEngineer World's Fair with @alanwuuuuuu. 1. Evals Have "Eaten the Conference" — AI Engineering Has Grown Up -The conversation has shifted from "Look what the model can do" to "Prove it doesn't regress in production." -Evals, observability, gates, controls, and trust are now core engineering disciplines. -Production teams obsess over reliability more than raw capability. -"Trust has officially become an engineering discipline." The "back office of AI" (observability, authority layers, governance) has arrived. 2. Agentic Systems & "Software Factories" Are the New Frontier -Heavy focus on real agentic workflows, multi-agent orchestration, debugging agents, and memory systems. -Software Factories emerged as a hot new concept: self-improving codebases and systems that get better over time (recursive self-improvement loops are apparently already running in early form at frontier labs). -Context engineering has evolved well beyond basic RAG → graphs, advanced memory systems, compaction, and business knowledge integration. 3. Long Context Is "Solved" (Sort Of) -The quadratic cost of attention is no longer a fundamental blocker. -The approach: intelligently select relevant subsets from huge contexts rather than naively stuffing everything in. -Expect billion-token context windows to become practical soon. 4. Open-Source Momentum (Especially Chinese Labs) -Strong buzz around GLM models (Zhipu AI / https://t.co/u5DfAFecUx) and Minimax. -They’re seen as impressive and important for releasing strong open weights. -Local AI and inference tracks were very popular. -Hugging Face had fireside chats highlighting this. 5. Voice & Multimodal Is Ready (and Underused) -Realtime voice-to-voice is now genuinely good. -Many believe AGI interfaces will be mostly spoken, not typed. 6. Other Insights -Benchmarks are broken — none are fully trustworthy right now. Cheap/fast inference providers sometimes quietly reduce accuracy. -Debate on AI-generated code: Some say review everything, others say you still need deep problem understanding either way. -Alignment is considered "solved" if done properly (Erik Meijer got a lot of praise here).
- 36
- 37
@sagar_batchu ·
Claude Code, Cursor, Codex, and VS Code Copilot all expose dozens of hook events. But if you're standing up AI governance this quarter, you only need to know about four hooks that will be the basis of your AI governance posture 1. UserPromptSubmit. Fires when a developer submits a prompt. Scan for secrets pasted out of .env files. Redact PII before it hits the model. This is your inbound chokepoint. 2. PreToolUse. Fires before any tool call executes. Block dangerous shell commands. Gate MCP calls. Scope file writes. This is your outbound action control. 3. PostToolUse. Fires after a tool returns its result. The command might be fine. The output might not be. cat .env is harmless. What comes back is the exfiltration risk. This is response auditing. 4. SessionEnd. Fires when the agent finishes. Ship the full transcript to a central store. "Find every session that touched the customer database last quarter" becomes a query, not a forensic investigation. Start with these four. Wire them into a central event feed. Everything else is an optimisation on top of a foundation that already works.
- 38
@ambient_xyz ·
Sadly AI mistakes are treated as bugs but they are all liabilities. When a model misclassifies in production, your enterprise owns the outcome & not the vendor. Yet most teams still track accuracy scores which is a huge governance gap hiding in plain sight. You are deploying more models so can you prove which model version ran? What inputs it processed? What confidence threshold it followed? If not, every postmortem is guesswork and every audit is a joke. Enterprises need tamper-evident inference records with model lineage you can actually inspect and which is exactly what Ambient delivers: verifiable proof that the AI you validated is the AI you served. Accountability that scales with you.
- 39
@ttunguz ·
That little black box in the middle is machine learning code. I remember reading Google’s 2015 Hidden Technical Debt in ML paper & thinking how little of a machine learning application was actual machine learning. The vast majority was infrastructure, data management, & operational complexity. With the dawn of AI, it seemed large language models would subsume these boxes. The promise was simplicity : drop in an LLM & watch it handle everything from customer service to code generation. No more complex pipelines or brittle integrations. But in building internal applications, we’ve observed a similar dynamic with AI. Agents need lots of context, like a human : how is the CRM structured, what do we enter into each field - but input is expensive the Hungry, Hungry AI model. Reducing cost means writing deterministic software to replace the reasoning of AI. For example, automating email management means writing tools to create Asana tasks & update the CRM. As the number of tools increases beyond ten or fifteen tools, tool calling no longer works. Time to spin up a classical machine learning model to select tools. Then there’s watching the system with observability, evaluating whether it’s performant, & routing to the right model. In addition, there’s a whole category of software around making sure the AI does what it’s supposed to. Guardrails prevent inappropriate responses. Rate limiting stops costs from spiraling out of control when a system goes haywire. Information retrieval (RAG - retrieval augmented generation) is essential for any production system. In my email app, I use a LanceDB vector database to find all emails from a particular sender & match their tone. There are other techniques for knowledge management around graph RAG & specialized vector databases. More recently, memory has become much more important. The command line interfaces for AI tools save conversation history as markdown files. When I publish charts, I want the Theory Ventures caption at the bottom right, a particular font, colors, & styles. Those are now all saved within .gemini or .claude files in a series of cascading directories. The original simplicity of large language models has been subsumed by enterprise-grade production complexity. This isn’t identical to the previous generation of machine learning systems, but it follows a clear parallel. What appeared to be a simple “AI magic box” turns out to be an iceberg, with most of the engineering work hidden beneath the surface. https://t.co/7uehMMWucf
- 40
@ashugarg ·
The graveyard of enterprise AI pilots is full of products that couldn't clear security and governance. At our recent CEO and CIO dinners, we heard this repeatedly from @djpersia (@databricks), Rajat Taneja (@Visa), and CIOs from @Zuora, @asana, and @BlackLine. In many orgs, data can’t simply be sent to a third-party cloud. It often needs to stay within a company’s own environment, whether that’s a private cloud (VPC) or on-prem. There are also baseline certification requirements: SOC 2, HIPAA, HiTrust, Veracode, BlackDuck. Security reviews alone can take longer than the pilot itself. Once those boxes are checked, what happens next is less understood. Large companies now have hundreds of agents running across the org, many of them built by employees without engineering backgrounds. Most are still working out how to maintain visibility into what those agents are doing, if they’re interacting with each other in unintended ways, and whether they’re compromising the company’s underlying data and permissions model. This is where many deployments break down - and where the next layer of value is getting built. @arizeai (a FC portco) has seen product usage go up 10x in the last quarter alone: a sign of how urgently enterprises need observability and evals.
- 41
@illyism ·
👀 how I debugged a tiny but painful AI SEO Tracker extraction bug today: 1. found a weird real example the app counted source/company names as “brand mentions” even when the prompt asked for one person (@nic_amadio) 2. turned the bug into an LLM eval / benchmark made a fixture from the actual AI answers across Perplexity, Gemini, Copilot, AI Overview, and ChatGPT 3. wrote the expected output for each answer: - expected mention - false positives to reject - extracted result - pass/fail 4. tested multiple models not to blindly "upgrade the model", but to see whether the failure was model quality or prompt design, openrouter helps here 5. wrote results to markdown so every run produced a readable artifact that is commited to git so we can easily compare before/after: - summary table - cost/time - exact extracted mentions - failures by provider 6. compared before/after ask your AI agent to rewrite the markdown 7. cleaned up and shipped lesson: don’t just vibe-fix LLM behavior turn the weird case into a small eval, make the output inspectable, and keep the benchmark in the repo it is a new kind of unit test, but for LLMs when a new model is released, we can know the exact quality / cost and compare!
- 42
@Suryanshti777 ·
holy shit. Someone just open-sourced a diagnostic that runs 32 inspections on any AI agent and tells you exactly where it's misaligned. It's called iFixAi. You point it at OpenAI, Anthropic, Gemini, Bedrock, or your own agent. Five minutes later you get a scorecard graded A through F across five categories of misalignment risk. The five categories it tests for: → FABRICATION: tool authorization leaks, missing audit trails, unsourced claims, overconfident responses → MANIPULATION: hallucination, privilege escalation, prompt injection, malicious deployer rules → DECEPTION: evaluation-awareness sandbagging, covert side tasks, long-horizon drift, goal stability → UNPREDICTABILITY: context distortion, instruction drift, decision stability across runs → OPACITY: regulatory readiness, session leakage, training-contamination attestation, escalation paths Two mandatory minimums are baked in. Tool authorization must hit 100%. Privilege escalation must hit 95%. Fail either one and your overall score gets capped at 60% no matter how well you did everywhere else. Every run writes a content-addressed manifest that captures every input. So you can drop it into CI, track drift over weeks, and prove your agent isn't quietly degrading between deploys. The Full mode is what makes it serious. You bring two different judge providers and it runs a multi-judge ensemble with conservative tie-break and per-judge attribution. No silent self-judging. If you only have one credential, the run literally refuses to compare systems for you. Industry agnostic by design. The test code is domain-neutral. Healthcare, customer support, software engineering knowledge lives in fixture YAML you write yourself. Three example fixtures ship in the repo. It also maps every test to OWASP LLM Top 10, NIST AI RMF, EU AI Act, and ISO 42001. One flag and you get a regulatory gap analysis. Built by @ifixai_ai. Apache 2.0. 100% Opensource.
- 43
@ambient_xyz ·
You don’t lose control of AI in production because your prompts are bad but you lose control the same way people lose control of a company credit card. A demo is one person, one account, one happy path. Production is chaos: interns, vendors, night shifts, rushed hotfixes, and a thousand tiny just this once exceptions. Then the first incident hits and you ask: a) Who made the call? b) What data did it touch? c) Which policy did it violate? d) What was the cost? Silence. That gap here is of sheer governance: Who can use the system, for what purpose, under which limits with what accountability when it goes wrong. Most AI stacks ship answers without any auditability. They cannot prove what actually happened per request, so teams argue feelings and screenshots while risk teams freeze deployments. Ambient is built for the solving this. Verification-first inference produces receipts: per-tenant trails, observable policy enforcement, and proof-backed logs you can hand to security, finance, or procurement. AI is money. If you cannot govern it, you cannot scale it.
- 44
@yizucodes ·
Voice AI in production is WAY harder than demos suggest. I just left LiveKit's panel with CTOs from Portola, Infinitus, Yelp & Bluejay breaking down what "reliability" actually means at scale. The gap between prototype and production is wild 🧵 1. Memory consistency > latency for consumer AI Portola's CTO: "Users tolerate 4-5 second delays. But if your AI 'best friend' forgets their spouse's name once? Trust destroyed forever." P95 latency matters as much as P50 in multi-turn conversations. 2. Healthcare voice agents have ZERO margin for error Infinitus processes 10-45 min calls. A 1% call drop rate = 100-200 lost patient calls PER DAY at scale. Their compliance language can't vary by a single word. 3. The latency hack nobody talks about: Fill silence during tool calls with "let me look that up" phrases. Mimics human receptionists. Reduces perceived wait time even when actual latency stays same. 4. Counterintuitive from testing: "Dumber models are often more conversational than smart ones" The sweet spot? Hybrid: fast conversational model + smart reasoning model. 5. Real production killer: "We started with pure vibes, no formal testing, just 3-4 manual calls before shipping" Now at scale: LLM-as-judge, 50-100 labeled calls and canary deployments Production humbles everyone 💀 Thanks @livekit for hosting #VoiceAI meetup @frontiertower
- 45
@mchulet ·
As an AI Engineer. Please learn >Harness engineering, not just prompt engineering >Context engineering, not just long prompts >Prompt caching vs. semantic caching tradeoffs >KV cache management, eviction, reuse, and memory pressure at scale >Prefill vs. decode latency and why they optimize differently >Continuous batching, paged attention, and throughput optimization >Speculative decoding vs. quantization vs. distillation tradeoffs >INT8, INT4, FP8, AWQ, GPTQ, and when quantization hurts quality >Structured output failures, schema validation, repair loops, and fallback chains >Function calling reliability, tool contracts, argument validation, and idempotency >Agent guardrails, loop budgets, tool budgets, and termination conditions >Model routing, graceful fallback logic, and degraded-mode UX >RAG architecture: chunking, embeddings, hybrid search, reranking, and freshness >Retrieval evals: recall, precision, grounding, attribution, and citation quality >Evals: golden sets, regression tests, adversarial tests, LLM-as-judge, and human evals >LLM observability as a first-class discipline: traces, spans, tokens, latency, errors, and drift >Cost attribution per feature, workflow, tenant, and user journey not just per model >Safety engineering: prompt injection defense, data leakage prevention, and permission boundaries >Multi-tenant isolation, cache safety, and cross-user context contamination prevention >Fine-tuning vs. in-context learning vs. RAG vs. distillation and when each is the wrong tool >Latency, quality, cost, and reliability tradeoffs across the full inference stack >Production failure modes: hallucinated tool calls, malformed JSON, stale retrieval, runaway agents, and silent eval regressions
- 46
- 47
@hugobowne ·
AI evals aren't just important for building your feedback loop into the product development lifecycle, but they're essential to make sure you're compliant when building in regulated industries. I had a great chat with @stellawliu (Head of Applied Science, ASU) and Eddie Landesberg (Staff Data Scientist, Google) about the what's happening in the evals space in 2026, including how we can all take our AI evals to the next level by leveraging techniques from .... causal inference! Link to full episode in comments 👇
Watch video
- 48
@krishnan ·
The unglamorous part of AI is becoming the moat. Everyone is covering model launches. Netflix's new engineering writeup points to the harder Day 2 question: can you run LLMs like production infrastructure? Netflix says (https://t.co/tuYfXJNdnn) it runs the full LLM serving stack inside its own production environment, not as a separate ML sidecar. The interesting part is not "we use LLMs." It is the architecture: vLLM as the paved-path engine, NVIDIA Triton underneath, a Java control plane for deployment and rollout, unified metrics, multi-region upgrades, and constrained decoding inside the generation loop. That last piece matters. Constrained decoding means the system does not wait for the model to produce bad output and then clean it up afterward. It pushes rules into token generation itself. Each step masks invalid next tokens based on the state of the request. In plain English: the model is guided to produce valid output by construction, not by apology. The tradeoff is real. More control usually adds latency, CPU coordination, batching complexity, and debugging pain. Netflix's post is useful because it talks about those operational choices instead of pretending the hard part ends when a model answers a prompt. This is where enterprise AI is going. The durable advantage will not be "we picked the best model this quarter." It will be: - Can you route workloads across real-time and batch paths? - Can you measure token throughput, KV cache use, and prefix cache hit rates? - Can you roll out model changes without breaking downstream systems? - Can you enforce output constraints before bad work hits the workflow? Watch the metric nobody puts in the launch deck: how much AI work survives contact with production load. #AIInfrastructure #LLMOps #EnterpriseAI #PlatformEngineering #AIOps #GenerativeAI
- 49
@arrotu ·
Most AI teams already have logs and traces. Far fewer can prove what actually ran when a workflow is challenged later. That is the gap this new article explores. It breaks down what OpenTelemetry solves well, where logs and traces stop short, and why a separate layer of execution evidence is starting to matter more in AI systems. If your workflows involve agents, tool calls, policies, or customer-facing actions, this distinction becomes a lot more important. Article below. https://t.co/TQAMJbjDTZ
- 50
Best Tweets by Topic