Evaluation & Feedback Loops
LLM and agent evaluation practices, including golden datasets, LLM judges, rubrics, error analysis, regression testing, online experiments, and human review prioritization.
52%
Best tweets about MLOps
50 Best Tweets About MLOps (2026)
Browse the best tweets about MLOps, including model deployment, evaluation, monitoring, data pipelines, infrastructure, reliability, and production lessons.
Production MLOps systems, model delivery, observability, evaluation, data pipelines, infrastructure, incidents, and engineering tradeoffs.
- Creators
- 42
- Updated
Original Xholic analysis
What 50 top MLOps posts reveal
The MLOps discussion emphasizes the work required to operate AI systems beyond a demo: evaluation, observability, data quality, delivery controls, and serving tradeoffs recur across the corpus. Evaluation & Feedback Loops is the largest theme (52%), while Production AI Systems has the highest theme median all-time score (64.08). [2042951175071502513, 2046567126597861566, 2050239816806387774]
- Dominant tone
- Positive
- Median score
- 37.1
- Leading format
- Announcement
- Recent posts
- 50%
46% of posts
All-time engagement
32% of posts
Published in 90 days
Conversation map
The themes creators return to
Observability & Monitoring
Tracing, logging, metrics, monitoring, incident diagnosis, drift detection, alerting, and observability challenges in asynchronous or distributed AI applications.
40%
Agent Reliability & Operations
Reliable agent operations: state and memory, tool execution, durable workflows, orchestration, validation, retries, recovery, human approvals, and multi-agent coordination.
28%
Production AI Systems
End-to-end production architecture for AI/ML applications: modular services, orchestration, deployment, CI/CD, containers, rollouts, and operating the system beyond a demo.
24%
Inference Serving Infrastructure
LLM serving and inference infrastructure, including GPU utilization, batching, KV/prefix caching, speculative decoding, distributed serving, autoscaling, latency, and throughput.
22%
Reliability & Release Engineering
Operational resilience and safe delivery: fallbacks, circuit breakers, idempotency, rate limits, canary releases, rollbacks, model/version pinning, and handling provider failures.
20%
Data Pipelines & Quality
Data engineering for ML: ingestion, ETL/ELT, streaming and batch pipelines, contracts, quality checks, lineage, feature stores, dataset curation, and reproducibility.
18%
RAG & Retrieval Systems
RAG system engineering, covering chunking, embeddings, vector databases, hybrid retrieval, reranking, metadata, retrieval evaluation, and context quality.
16%
Tone and stance
Sentiment Positive leads
Author posture Supportive leads
Performance benchmark
- Median likes
- 60
- Median reposts
- 8
- Median replies
- 6
- Median views
- 3.7K
Posts with media make up 68% of this collection. Their median all-time score is 50.6, compared with 13.0 for text-only posts.
Format mix
- Announcement 32% · score 45.3
- List 30% · score 41.8
- Opinion 26% · score 16.4
- Tutorial 12% · score 38.7
Consensus and debate
Where creators agree, and where they do not
Shared view
Production work extends beyond the model
Across these posts, production AI is presented as more than model selection: evaluation, observability, runtime durability, guardrails, orchestration, and delivery are recurring system components.
Shared view
Evaluation is an iterative feedback loop
Evaluation is described as an iterative practice: rubric-based judging, targeted selection of production trajectories for review, and human-in-the-loop interpretation where automated tools lack domain context.
Shared view
Data quality is a production concern
Posts connect data quality to upstream work such as schema contracts, validation, curation, and reproducible pipelines; monitoring and retraining are also included in lifecycle-oriented descriptions.
Shared view
Release and reliability practices are prominent
Safe operation is associated with measures including retries, circuit breakers, fallbacks, version pinning, canaries, rollbacks, and task-level model routing.
Open debate
Automation does not eliminate human judgment
One post presents rubric-driven G-Eval as a more procedural and reproducible evaluation approach. Another reports that automated trace-evaluation tools can surface issues but miss problems requiring domain expertise and have limited mechanisms for learning from human feedback.
Open debate
Autonomy is paired with operational controls
Self-improving loops are presented as a practical approach in one post, while another emphasizes operational constraints around durable runtimes, guardrails, cost controls, approval gates, explicit state, and routing.
Open debate
Task-specific model routing versus model abstraction
One creator describes routing workload types between a self-hosted open model and a proprietary model based on a personal six-day comparison. Another argues for task-level, model-agnostic abstractions designed to absorb vendor changes.
What performs
Patterns behind standout posts
Architecture, evaluation resources, and an LLM pipeline were the largest outliers
The three largest all-time-score outliers were the 9-layer production architecture post (2457.15), the AI-evals learning list (1308.5), and the nanochat end-to-end LLM-pipeline post (1083.58). The set median all-time score was 37.06.
Announcements and lists had the highest format medians
Announcements were the largest format group at 32% and had a 45.31 median all-time score. Lists accounted for 30% with a 41.805 median, while tutorials accounted for 12% with a 38.7 median.
Evaluation led by prevalence; production systems led by theme median
Evaluation & Feedback Loops was the largest theme, appearing in 52% of tweets, followed by Observability & Monitoring at 40%. Production AI Systems had the highest theme median all-time score, at 64.08.
Media posts had a higher median score
Posts with media made up 68% of the set and had a 50.56 median all-time score, compared with 12.96 for text-only posts.
Statistical standouts
- View standout post 1 Score 2457.2 · 66.3× median
- View standout post 2 Score 1308.5 · 35.31× median
- View standout post 3 Score 1083.6 · 29.24× median
- View standout post 4 Score 262.4 · 7.08× median
- View standout post 5 Score 217.1 · 5.86× median
Creator landscape
Who shapes this conversation
The five most represented creators account for 20% of the selected posts.
-
1. Abhishek Singh
@0xlelouch_
2 posts
-
2. Akshay 🚀
@akshay_pachaar
2 posts
-
3. Aurimas Griciūnas
@Aurimas_Gr
2 posts
-
4. Bilgin Ibryam
@bibryam
2 posts
-
5. Shalini Goyal
@goyalshaliniuk
2 posts
-
6. Priyanka Vergadia
@pvergadia
2 posts
RAG is connected to data engineering
Aurimas Griciūnas links RAG design choices—such as chunking, embeddings, vector search, reranking, monitoring, and security—with upstream schema contracts, validation, and data flows.
Feedback should be machine-usable
Bilgin Ibryam emphasizes environments agents can operate reliably and identifies types, tests, linting, builds, logs, traces, and evals as feedback an agent can use without asking a person.
Operational signals can prioritize improvement
Akshay Pachaar highlights a self-revision harness evaluated against benchmarks and a separate pattern that uses low-cost behavioral signals to prioritize production trajectories for human review.
How this analysis was made
Themes, sentiment, stance, and post format are classified per tweet. All counts, shares, medians, creator concentration, freshness, and performance comparisons are then calculated directly from the published snapshot.
Xholic's all-time score compares engagement while accounting for reach, post age, and creator consistency. It is used for relative comparisons within this collection.
This report analyzes the exact 50-post snapshot shown below. AI identifies editorial categories and drafts explanations; all statistics are calculated from the snapshot, and every narrative claim is checked against cited posts before publication.
Best MLOps tweets
Top MLOps tweets from 42 creators
Ranked 01–50
- 01
@techNmak ·
Someone just dropped a 9-layer production AI architecture and it's the most honest breakdown I've seen. services/ - RAG pipeline, semantic cache, memory, query rewriter, router. Not one file. Five. agents/ - document grader, decomposer, adaptive router. Self-correcting by design. prompts/ - versioned, typed, registered. Never hardcoded. security/ - input, content, output. Three guards not one. evaluation/ - golden dataset, offline eval, online monitor. Most people skip this entire layer and ship blind. observability/ - per-stage tracing, feedback linked to traces, cost per query. .claude/ - agent context so your AI coding assistant knows the codebase before it touches a file. The demo is one file. Production is this.
- 02
@pauliusztin_ ·
Every day, 100+ people ask me, "How can I learn AI evals?" I copy-paste these 11 links (every time): 1. AI evals & observability (series): https://t.co/erSJcqpAV7 2. Using LLM-as-a-judge: https://t.co/xMBt9j4JRc 3. Demystifying evals for AI agents: https://t.co/HBbCe5PnXJ 4. There are only 6 RAG Evals: https://t.co/gwfyhIozqK 5. Evaluation-driven development: https://t.co/GMtp6bewol 6. Binary evals vs. Likert scales: https://t.co/WyMw1hHTfm 7. The mirage of generic AI metrics: https://t.co/ugryF5zfKO 8. Error analysis: https://t.co/OXgPZd8IXi 9. Carrying out error analysis: https://t.co/OXgPZd8IXi 10. Evaluating the effectiveness of LLM-evaluators: https://t.co/NuaXhr19TV 11. LLM judges aren't the shortcut you think: https://t.co/fDep2HFjCq Binge these to skyrocket your skills.
- 03
@karpathy ·
New post: nanochat miniseries v1 The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do careful science of scaling laws and ultimately this is what gives you the confidence that when you pay for "the big run", the extrapolation will work and your money will be well spent. For the first public release of nanochat my focus was on end-to-end pipeline that runs the whole LLM pipeline with all of its stages. Now after YOLOing a few runs earlier, I'm coming back around to flesh out some of the parts that I sped through, starting of course with pretraining, which is both computationally heavy and critical as the foundation of intelligence and knowledge in these models. After locally tuning some of the hyperparameters, I swept out a number of models fixing the FLOPs budget. (For every FLOPs target you can train a small model a long time, or a big model for a short time.) It turns out that nanochat obeys very nice scaling laws, basically reproducing the Chinchilla paper plots: Which is just a baby version of this plot from Chinchilla: Very importantly and encouragingly, the exponent on N (parameters) and D (tokens) is equal at ~=0.5, so just like Chinchilla we get a single (compute-independent) constant that relates the model size to token training horizons. In Chinchilla, this was measured to be 20. In nanochat it seems to be 8! Once we can train compute optimal models, I swept out a miniseries from d10 to d20, which are nanochat sizes that can do 2**19 ~= 0.5M batch sizes on 8XH100 node without gradient accumulation. We get pretty, non-itersecting training plots for each model size. Then the fun part is relating this miniseries v1 to the GPT-2 and GPT-3 miniseries so that we know we're on the right track. Validation loss has many issues and is not comparable, so instead I use the CORE score (from DCLM paper). I calculated it for GPT-2 and estimated it for GPT-3, which allows us to finally put nanochat nicely and on the same scale: The total cost of this miniseries is only ~$100 (~4 hours on 8XH100). These experiments give us confidence that everything is working fairly nicely and that if we pay more (turn the dial), we get increasingly better models. TLDR: we can train compute optimal miniseries and relate them to GPT-2/3 via objective CORE scores, but further improvements are desirable and needed. E.g., matching GPT-2 currently needs ~$500, but imo should be possible to do <$100 with more work. Full post with a lot more detail is here: https://t.co/na8zVLqWLf And all of the tuning and code is pushed to master and people can reproduce these with scaling_laws .sh and miniseries .sh bash scripts.
- 04
@Aurimas_Gr ·
“I will build a RAG system for my company in one week” - that is what I often hear nowadays from recently turned AI experts. Unfortunately, building a 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗴𝗿𝗮𝗱𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗯𝗮𝘀𝗲𝗱 𝗔𝗜 𝘀𝘆𝘀𝘁𝗲𝗺 is a challenging task. Here are some of the moving parts in the RAG based systems that you will need to take care of and continuously tune in order to achieve desired results: 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹: 𝘍 ) Chunking - how do you chunk the data that you will use for external context. - Small, Large chunks. - Sliding or tumbling window for chunking. - Retrieve parent or linked chunks when searching or just use originally retrieved data. 𝘊 ) Choosing the embedding model to embed and query and external context to/from the latent space. Considering Contextual embeddings. 𝘋 ) Vector Database. - Which Database to choose. - Where to host. - What metadata to store together with embeddings. - Indexing strategy. 𝘌 ) Vector Search - Choice of similarity measure. - Choosing the query path - metadata first vs. ANN first. - Hybrid search. 𝘎 ) Heuristics - business rules applied to your retrieval procedure. - Time importance. - Reranking. - Duplicate context (diversity ranking). - Source retrieval. - Conditional document preprocessing. 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻: 𝘈 ) LLM - Choosing the right Large Language Model to power your application. ✅ It is becoming less of a headache the further we are into the LLM craze. The performance of available LLMs are converging, both open source and proprietary. The main choice nowadays is around using a proprietary model or self-hosting. 𝘉 ) Prompt Engineering - having context available for usage in your prompts does not free you from the hard work of engineering the prompts. You will still need to align the system to produce outputs that you desire and prevent jailbreak scenarios. And let’s not forget the less popular part: 𝘏) Observing, Evaluating, Monitoring and Securing your application in production! What other pieces of the system am I missing? Let me know in the comments 👇
- 05
@adxtyahq ·
again saying there's never been a better time to work on multi-agent systems. learn rag, orchestration, evals, memory, routing, tool calling, validation loops, fix loops, split learning, context engineering. all of it. getting an llm to answer questions is becoming the easy part getting multiple agents, tools, and workflows to work together reliably in production without breaking every other day is where the real challenge is. we're entering a phase where building the model matters less than building everything around it.
- 06
@lvwerra ·
Auto-research for ML training models is all the rage now, but underrated is: auto-research for data! Sure, you can squeeze out a bit of model performance by optimizing hyperparameters, but code agents can do data work that has been very labour intensive and required a lot of attention to a lot details effortlessly: > download data from many different data sources > bring all the data sources into uniform format > do detailed EDA: find patterns and outliers > look at 100s of samples and take detailed notes > make beautiful infographics rather than mpl plots > iterate on data filtering by looking at more samples > make a simple pipelines robust and scalable It's now possible to write data pipelines for dozens of data sources in hours that would have taken weeks of reading many docs, debugging APIs and data formats, wrangling outliers and missing data. A few weeks ago we gave Claude access to the CPU partition of our cluster and it iteratively refined filters to retrieve a domain subset of FineWeb. This would have taken me 2-3 days to work through while it took Claude just a few hours with almost no babysitting and with a nice logbook. Thus the long tail of small, niche data sources becomes more accessible and can be aggregated to even larger high quality datasets for cool applications. Data has been fuelling LLM progress more than model architecture innovations, so I am very excited about this!
- 07
- 08
@vasuman ·
The 5 principles for AI that ships to production: 1. Audit first: map the actual workflow before touching a model. Find the conformance gap. Separate repeatable patterns from genuine judgment. 2. Deterministic by default: LLM only where judgment lives. Code everywhere else. Most production architecture is testable with unit tests. 3. One orchestration layer: shared ingestion, approvals, audit logging, knowledge. Each new agent lands as configuration on the platform. 4. Model-agnostic: build the abstraction at the task level. Route each step to the best-fit model. The routing layer absorbs vendor changes. 5. Continuous infrastructure: models change, workflows evolve, vendor pricing shifts. Budget for ongoing tuning from day one.
- 09
@goyalshaliniuk ·
Building an AI model isn’t just about training a neural network - it’s a full journey with 8 critical stages. From data collection to model monitoring, here’s how AI systems are built and maintained today: 1. Data Collection & Preparation Everything starts with data. Raw input like text, images, or sensor readings is collected, labeled with correct outputs, and cleaned for quality. This foundation is vital for training high-performing models. 2. Feature Engineering Raw data is refined into useful inputs. Basic features are used for simple models, while advanced tasks rely on transformed or learned features from neural networks. 3. Model Architecture Here you choose the model type — linear models for simplicity, tree-based for tabular data, and neural networks for complex tasks like vision and NLP. 4. Model Training You train the model using CPUs for small workloads, GPUs for deep learning, or distributed systems for massive models like GPTs. 5. Model Evaluation After training, you evaluate performance using metrics like accuracy, F1-score, and confusion matrices. These metrics show how well the model is doing - especially on real-world data. 6. Deployment Once ready, the model is deployed. You can serve predictions in real-time (like chatbots), in batches (like analytics reports), or on edge devices (like mobile apps). 7. Monitoring & Maintenance AI doesn’t stop at launch. Logs are tracked, performance is monitored for drifts, and retraining pipelines ensure the model stays accurate as data evolves. 8. Model Architecture (Trust & Ethics) To keep models fair and explainable, anonymization, bias checks, and transparency tools like SHAP or LIME are implemented — especially important in regulated industries. From raw data to real-world impact - this is the full roadmap of an AI model. Save this guide as your go-to reference if you're building or working with AI systems in 2026!
- 10
@arpit_bhayani ·
New write-up is live, and this time I covered G-Eval. It helps answer one important question: how do you know whether what an LLM generated is apt, correct, and aligned with your requirements? G-Eval is a pretty simple framework that leverages Chain-of-Thought prompting over clearly defined rubrics. Instead of simply asking an LLM to "rate this post" or "is this correct?", it makes the evaluation process more procedural and reproducible. If you bluntly ask an LLM to rate something on a scale of 1-5, it tends to exhibit a bias toward picking certain values, lacks a clear audit trail, and often suffers from a bias toward its own model family. G-Eval solves this neatly. In this blog, I covered G-Eval in detail and showed how to adopt it in your AI workloads to actually measure whether your LLM outputs are getting better. If you are looking for a reliable blueprint for LLM evaluation, this guide should help. Give it a read, and like always, I hope this helps.
- 11
@bibryam ·
🚨 This OpenAI article is an absolute gold mine for harness engineers. The insight isn’t “AI writes code.” It’s: → how to build environments agents can reliably operate in → how to encode engineering taste mechanically → how to scale feedback loops instead of headcount → how observability becomes agent-readable infrastructure https://t.co/CFDMgHJBkI
- 12
@Suryanshti777 ·
Someone just open-sourced a real production AI system… Not a chatbot. Not a GPT wrapper. A full stack that actually scales. And it exposes the biggest lie in AI right now: → “Just call an LLM and you’re done.” Wrong. Real AI apps are built on: • Data pipelines (clean → chunk → embed) • Hybrid retrieval + reranking • Memory + semantic caching • Intelligent routing + fallbacks • Structured generation (not raw prompts) • Continuous evaluation (offline + live) • Security layers (injection, filtering) • Full observability (trace every query) • Production infra (async, streaming, containers) The truth? AI isn’t a feature. It’s a system design problem. And most people are still playing at layer 1. The ones who win will build all 9.
- 13
@akshay_pachaar ·
MiniMax M2.7 is open-source! The most interesting part of this release isn't a benchmark number. It's what MiniMax calls "self-evolution," and it's essentially Karpathy's Autoresearch applied at full scale. Every AI agent today runs inside a harness: the scaffolding of skills, tools, memory, and workflow rules that surrounds it. Normally a human engineer builds this, and the agent operates within it. The harness stays fixed. M2.7 treats its harness as something it can rewrite. The agent runs a task, analyzes where things went wrong, plans changes to its own scaffold, applies them, evaluates against a benchmark, and decides whether to keep or revert. It writes self-criticism into memory so the next round starts smarter, then loops again. MiniMax ran this for 100+ rounds internally. The model discovered optimizations on its own: it systematically searched for optimal sampling parameters, wrote workflow-specific guidelines (like checking for the same bug pattern in other files after a fix), and added loop detection to avoid getting stuck. They also tested it on 22 ML competitions from OpenAI's MLE Bench Lite, each running 24 hours fully autonomous. With every round, the trained models achieved higher medal rates. The best run earned 9 gold medals. The weights never changed. What improved was the system around the model: better skills, better memory, better workflow rules. That distinction matters because the improvement loop can run continuously without any retraining. I'm pretty sure every major AI lab is doing some version of this internally. The fact that MiniMax is publishing it openly is what makes this release worth paying attention to. huggingface : https://t.co/aiCXrBi8Vc Blog: https://t.co/2YqrunBoJp Note: The model licence is NON-COMMERCIAL LICENSE, that said, there's a lot to learn from this work being available in the open.
Watch video
- 14
@Aurimas_Gr ·
A breakdown of 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗶𝗻 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 👇 And yes, it can also be used for LLM based systems! It is critical to ensure Data Quality and Integrity upstream of ML Training and Inference Pipelines, trying to do that in the downstream systems will cause unavoidable failure when working at scale. There is a ton of work to be done on the Data Lake or LakeHouse layer. 𝗦𝗲𝗲 𝘁𝗵𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗯𝗲𝗹𝗼𝘄. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦 𝘢𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦 𝘧𝘰𝘳 𝘢 𝘱𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯 𝘨𝘳𝘢𝘥𝘦 𝘦𝘯𝘥-𝘵𝘰-𝘦𝘯𝘥 𝘥𝘢𝘵𝘢 𝘧𝘭𝘰𝘸: 𝟭: Schema changes are implemented in version control, once approved - they are pushed to the Applications generating the Data, Databases holding the Data and a central Data Contract Registry. Applications push generated Data to Kafka Topics: 𝟮: Events emitted directly by the Application Services. 👉 This also includes IoT Fleets and Website Activity Tracking. 𝟮.𝟭: Raw Data Topics for CDC streams. 𝟯: A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Contract Registry. 𝟰: Data that does not meet the contract is pushed to Dead Letter Topic. 𝟱: Data that meets the contract is pushed to Validated Data Topic. 𝟲: Data from the Validated Data Topic is pushed to object storage for additional Validation. 𝟳: On a schedule Data in the Object Storage is validated against additional SLAs in Data Contracts and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes. 𝟴: Modeled and Curated data is pushed to the Feature Store System for further Feature Engineering. 𝟴.𝟭: Real Time Features are ingested into the Feature Store directly from Validated Data Topic (5). 👉 Ensuring Data Quality here is complicated since checks against SLAs is hard to perform. 𝟵: High Quality Data is used in Machine Learning Training Pipelines. 𝟭𝟬: The same Data is used for Feature Serving in Inference. Note: ML Systems are plagued by other Data related issues like Data and Concept Drifts. These are silent failures and while they can be monitored, we can’t include it in the Data Contract. Let me know your thoughts! 👇
- 15
@goyalshaliniuk ·
Planning to build a GenAI or AI-powered product? Here’s the modern AI app stack you need to scale, serve, and secure your models. 👇 1. Data Layer: Foundation for AI Use Snowflake, BigQuery, Postgres, Airflow, and dbt to collect, clean, and move data efficiently. 2. Model Development Train with PyTorch, TensorFlow, HuggingFace, Scikit-learn, and track experiments using MLflow or Weights & Biases. 3. Model Versioning & Packaging Ensure reproducibility and deployment using tools like MLflow, BentoML, Docker, and ONNX. 4. Deployment & Serving Expose your model as an API using FastAPI, Flask, and serve it with Triton or TorchServe. Automate with GitHub Actions and Jenkins. 5. MLOps Infrastructure Orchestrate workflows using Kubernetes, Kubeflow, and Flyte to scale across environments with Docker containers. 6. Monitoring & Observability Track model accuracy and failures using WhyLabs, Prometheus, Fiddler, and Superset to stay in control. 7. Retrieval & Augmentation (for LLMs) Use Pinecone, Weaviate, and FAISS with LangChain and LlamaIndex to connect external knowledge and enable RAG. 8. AI Agent Tools & Frameworks Use AutoGen, LangGraph, CrewAI with n8n or https://t.co/AJQRix30HU to orchestrate tool use and external actions (APIs, code, browsers). 9. Security & Compliance Stay enterprise-ready with OAuth2, Rebuff, GDPR, and Guardrails to manage access, risk, and regulations. 10. Frontend & Interface Deliver AI through React, Streamlit, GraphQL, and WebSockets. Use multimodal input/output via Whisper or Gemini. AI apps today aren’t just models, they’re full-stack systems. And this is the stack to build them right.
- 16
@HamelHusain ·
New Blog Post: Do Automated Evals Work? There has been a rise of tools that look through your traces with AI and identifies issues. We tested these tools with real production data to see how good they are. Where they shine - They often spot issues human miss - Integrate into your workflow: viewing traces, creating LLM judges etc. Where they fall short - They miss problems that require domain expertise and taste - Don't have great mechanisms to learn from human feedback - You can get similar results from using your coding agent So you should use them? Yes, BUT do so iteratively with you in the loop. We describe how in the post: https://t.co/WrStw4kglF It's also a good idea to try using your coding agent with you in the loop, which we discuss in the post. This was written with @doesdatmaksense , who led the research and collated the results.
- 17
@VaibhavSisinty ·
Andrew Ng just said 100% of his tasks are done by AI agents. His prediction: in 3 to 6 months, everyone will be using self-improving loops. Here's the thing. He's not wrong, and this isn't new. A loop is simple. You give AI a goal, it runs, checks its own output, fixes what's wrong, and runs again. It keeps going until the output hits the bar you set. That's it. No human sitting there typing "try again." The agent just keeps improving on its own. I already do this. I plug a free model into Ollama, put it in a loop, let it run all night, and by morning it's built something that would have taken me hours of back and forth prompting. Then I compare it against what a paid model like Opus gives me in one shot. Sometimes the overnight loop wins. That's the shift Ng is talking about. You stop writing better prompts. You start setting better goals and letting the loop do the work. Now zoom out and look at who else is running loops. → Meta moved its best engineers off product work entirely. Their only job now is to create hard problems that AI can't solve yet and feed them back into the model. The best engineers at one of the biggest tech companies are training AI to replace themselves. That's the loop running at company scale. → Google uses AI to monitor its own training runs and generate its own training data. Brin called it the self-improvement game. That's the loop running at model scale. Now here's the economics question. It's not whether loops work. It's how you run them. Running loops on rented API credits will eat you alive. A loop runs for hours, sometimes overnight. Every iteration burns tokens. At API rates that bill adds up fast. Running them on your own hardware changes the math entirely. Plug an open source model into your own machine, let it loop all night, and the cost will be fraction of what it use to be. Companies will slowly start to think in this directions. Instead of renting tokens, they'll start buying hardware. DGX Sparks, Mac Studios, inference rigs. The same way they buy laptops for employees today, they'll buy compute for AI tomorrow. The loops are already here. The question is whether you're still writing one prompt at a time while the loop is running all night for someone else.
- 18
@shivam74689 ·
Day 52 — Becoming AI Engineer Today I completed my first end-to-end Production ReAct Agent. A few weeks ago, I thought building an AI agent meant connecting an LLM to a UI and getting answers back. Now I know that's only a tiny part of the system. The biggest lesson from this project: Building an agent is easy. Building a reliable AI system is hard. When I started ReAct Agent v1, I thought the challenge would be making the model reason correctly. Instead, most of my time was spent debugging parsers, fixing tool calls, handling errors, managing state, improving evaluations, integrating observability, containerizing the application, and preparing it for deployment. I learned that a real AI agent is much more than: User → LLM → Answer A production agent requires: Reasoning → Tool Selection → Tool Execution → Observation → State Management → Final Answer The LLM is only one component. Everything around it is engineering. One of the biggest realizations came from my architecture decisions. I initially built the application using Streamlit because it was fast and simple for creating an MVP. It worked. But later I realized it wasn't the architecture I would choose for production. A production AI system should separate responsibilities: Frontend ↓ FastAPI Backend ↓ Agent ↓ Tools ↓ LLM With supporting systems such as: • Docker • Observability • Evaluation • Logging • Configuration Management That mistake taught me an important lesson. There is a huge difference between building a working AI application and designing a scalable AI system. Another lesson was discovering how fragile ReAct agents can be. I encountered: • Invalid output formats • Parser failures • Infinite reasoning loops • Wrong tool selections • Hallucinated observations • State management bugs • Deployment issues At first these failures felt frustrating. Eventually I realized they were the most valuable part of the project. Every bug forced me to understand the system at a deeper level. I also learned that prompt engineering alone is not enough. Production AI systems require: • Validation layers • Error handling • Retry mechanisms • Evaluation harnesses • Observability systems Without them, you don't have a production system. You have a demo. Another major milestone was containerizing the entire application using Docker. For the first time, I packaged an AI application into a reproducible environment, fixed dependency issues inside containers, debugged build failures, solved runtime errors, and successfully ran the system independently from my local machine. That experience completely changed how I think about deployment. Deployment is not just putting software online. Deployment is engineering. Looking back, the biggest achievement isn't that the agent works. The biggest achievement is that I pushed an idea through the complete lifecycle: Idea ↓ Build ↓ Break ↓ Debug ↓ Evaluate ↓ Observe ↓ Containerize ↓ Deploy ReAct Agent v1 is far from perfect. It has bugs. It has limitations. It needs improvements. But it achieved its purpose. It transformed me from someone learning about AI agents into someone who has actually built, evaluated, observed, containerized, and deployed one. Live Demo: https://t.co/jQGFruOmey ReAct Agent v1 Feel free to try it and share feedback. The next phase of my journey is different. I no longer want to build projects just to learn concepts. I want to build better projects. Projects with better architecture. Projects that solve real problems. Projects that force me to learn production engineering, system design, evaluation, reliability, and deployment at a deeper level. ReAct Agent v1 proved that I can ship. Now the goal is to build systems that are more useful, more reliable, and more ambitious. Project 1 is complete. Time to build something better. #AI #AIEngineer #AgenticAI #LLM #GenerativeAI #ArtificialIntelligence #Python #Docker #FastAPI #Langfuse #MLOps #BuildInPublic #LearningInPublic
- 19
- 20
@pvergadia ·
Never ever ever build an LLM app without KV cache. You're paying for O(n²) attention. On every. single. token. Here's what's actually happening under the hood: Transformer attention computes Q, K, V matrices for every token in your sequence. Without cache, generating token n means recomputing K and V for all n-1 previous tokens. For a 4,000-token prompt generating 500 tokens: → 500 full attention passes over 4,000+ tokens → Each pass = O(n²) in memory, O(n) in compute per layer → Across 32 attention heads × 96 layers (GPT-4 scale) you feel this KV cache breaks the cycle. Keys and Values are computed once per token, then stored. Only the Query is computed fresh for each new token. Memory cost: O(n × d_model × num_layers) Compute cost per new token: O(1) instead of O(n) Trade-off: At 128k context with FP16, KV cache alone can consume 40–80GB of VRAM. That's why H100s with 80GB fill up faster than you expect. That's why paged attention (vLLM), sliding window attention, and prefix caching exist they're all managing KV cache memory pressure. If your inference latency is spiking at long contexts, check your KV cache eviction policy before you blame the model. Save this before your next AI Engineer Interview. 🔖 #LLMInference #AIEngineering #GenerativeAI #MLOps #KVCache #AIInfrastructure #BuildingWithAI
- 21
@akshay_pachaar ·
A great LLM interview question: (answer shared below) You have 80k Agent-user interactions from production. You need to find the top 100 worth reviewing to improve the agent. You cannot use an LLM to evaluate them since it will be expensive. This is one of the most painful problems in shipping agents. Every team sits on a goldmine of production trajectories, but the bottleneck is figuring out which ones are worth a human's attention. Run an LLM judge over all 80k and your bill explodes. Review random samples manually and you burn hours on conversations where nothing interesting happened. So how do you actually pick the right 100? The simplest answer is random sampling. Pick 100 random trajectories and review them. But most production agents handle routine requests just fine, so you waste a big chunk of your annotation budget on uninformative conversations. Another approach is filtering for longer conversations, since 10+ user messages usually means more complexity. But longer conversations skew heavily toward outright failures. You surface obvious breakdowns but miss the subtle issues hiding in conversations where the agent technically succeeded. A recent paper from DigitalOcean takes a smarter approach. It computes lightweight behavioral signals directly from the trajectory data using deterministic rules. The signals fall into three groups. 𝟭) 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 come from the user-agent dialogue. → User rephrasing the request or correcting the agent is misalignment. → Agent producing near-duplicate or circular responses is stagnation. → User asking to "talk to a human" or abandoning the session is disengagement. → User confirming something worked is satisfaction. These are detected through normalized phrase matching, similarity checks, and simple discourse heuristics. 𝟮) 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 come from tool calls and runtime events. → A tool call that doesn't advance the task is a failure signal. → Repeated calls with identical or drifting inputs indicate a loop. These are straightforward to extract from execution logs. 𝟯) 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 cover rate limits, context overflow, and API errors. These are useful for diagnosis but not for training, since they reflect system constraints, not agent decisions. Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review. On τ-bench, the authors compared all three approaches on 100 trajectories. - Random sampling hit a 54% informativeness rate. - The length-based heuristic reached 74%. - Signal-based sampling reached 82%. This means roughly 4 out of every 5 sampled trajectories are genuinely useful for improving the agent. The bigger win shows up in successful trajectories. Among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs 41.3% for random. These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don't break the task but still matter for optimization. The whole framework runs without any LLM overhead and can sit always-on in a production pipeline. If you want to see this in practice, the signal-based approach is already integrated into Plano, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place. Plano GitHub repo: https://t.co/BD0WXJYrrz Research Paper: https://t.co/6s98KnVxXO 👉 Over to you: What's your approach to this problem?
Watch video
- 22
@vivoplt ·
As an AI Infrastructure Engineer. Please learn: - GPU/VRAM fundamentals, quantization & batching - vLLM / TensorRT-LLM / inference optimization - KV caching, speculative decoding & token throughput - Distributed training basics (DDP/FSDP/DeepSpeed) - Model serving & autoscaling - Vector DB retrieval pipelines - Prompt caching & cost optimization - Observability for LLM apps This is what production AI teams actually care about.
- 23
@TheGlobalMinima ·
The last 3 days have taught me a lot about how OpenTelemetry and Async Generator functions work. When you set up observability (personally using @langfuse ) for the first time, it is smooth sailing. But when you scale up your llm application to support concurrency, failure recovery and streaming, things quickly start going haywire. OpenTelemetry is super powerful, but it can run into broken traces and lost contexts in async setups. Issues like > context not detaching correctly > tracing across different contexts > request freezing before BatchSpanProcessor can finish (specific to serverless functions) The fix here is to set up your own context propagation. Each async task fired runs in the event loop, but may have a different context attached to it, leading to broken traces. Doing this also helps you scale up observability in distributed or event driven systems. Bookmark this to save days of debugging. Attaching some resources in the reply for better reference.
- 24
@intology ·
The models are improving the models. Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model. Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇 PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours. We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model. In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
- 25
@InduTripat82427 ·
📂 AI + CLOUD MASTER TREE │ ├── ☁️ 1. Cloud Fundamentals │ ├── What is Cloud Computing │ ├── IaaS vs PaaS vs SaaS │ ├── Public / Private / Hybrid Cloud │ ├── Regions & Availability Zones │ └── Pay-as-you-go Pricing │ ├── 🏗️ 2. Cloud Providers │ ├── AWS │ ├── Azure │ ├── Google Cloud │ ├── DigitalOcean │ └── Cloudflare │ ├── 🖥️ 3. Compute Services │ ├── Virtual Machines (EC2) │ ├── Serverless (Lambda, Functions) │ ├── Containers (Docker) │ ├── Kubernetes (EKS, GKE) │ └── Auto Scaling │ ├── 💾 4. Storage Services │ ├── Object Storage (S3, GCS) │ ├── Block Storage │ ├── File Storage │ ├── CDN Integration │ └── Backup & Recovery │ ├── 🗄️ 5. Databases in Cloud │ ├── Managed SQL (RDS, Cloud SQL) │ ├── NoSQL (DynamoDB, Firestore) │ ├── Data Warehouses (BigQuery, Redshift) │ ├── Caching (Redis, Memcached) │ └── Distributed Databases │ ├── 🤖 6. AI Fundamentals │ ├── Machine Learning Basics │ ├── Deep Learning │ ├── NLP │ ├── Computer Vision │ └── Generative AI │ ├── 🧠 7. AI Models & APIs │ ├── OpenAI API │ ├── Gemini API │ ├── Anthropic (Claude) │ ├── HuggingFace │ └── Replicate │ ├── 🔗 8. AI + Cloud Integration │ ├── Model Deployment (API-based) │ ├── Serverless AI Apps │ ├── Real-time AI Pipelines │ ├── Batch Processing │ └── Edge AI │ ├── 📊 9. Data Pipelines │ ├── ETL / ELT │ ├── Data Lakes │ ├── Streaming (Kafka, Pub/Sub) │ ├── Data Cleaning │ └── Feature Engineering │ ├── ⚙️ 10. MLOps │ ├── Model Training │ ├── Model Versioning │ ├── CI/CD for ML │ ├── Monitoring Models │ └── Retraining Pipelines │ ├── 🔐 11. Security │ ├── IAM (Identity Access Management) │ ├── API Security │ ├── Encryption (at rest & in transit) │ ├── Secrets Management │ └── Compliance (GDPR, SOC2) │ ├── 📈 12. Monitoring & Logging │ ├── CloudWatch │ ├── Stackdriver │ ├── Prometheus │ ├── Grafana │ └── Alerting Systems │ ├── 🚀 13. Deployment Strategies │ ├── CI/CD Pipelines │ ├── Blue-Green Deployment │ ├── Canary Releases │ ├── Rolling Updates │ └── Infrastructure as Code (Terraform) │ ├── 💡 14. Cost Optimization │ ├── Reserved Instances │ ├── Spot Instances │ ├── Auto Scaling Optimization │ ├── Resource Monitoring │ └── Cost Alerts │ ├── 🌍 15. Real-World Use Cases │ ├── AI Chatbots (ChatGPT-like) │ ├── Recommendation Systems │ ├── Fraud Detection │ ├── Image Generation Apps │ └── Voice Assistants │ └── 🎯 16. Future Trends ├── AI Agents ├── Multi-modal AI ├── Edge Computing ├── Autonomous Systems └── AI-native SaaS
- 26
@milan_milanovic ·
𝗧𝗵𝗲 𝗔𝘇𝘂𝗿𝗲 𝗔𝗜/𝗠𝗟 𝘀𝘁𝗮𝗰𝗸 Here are the most important Azure services if you want to work with AI in Azure. 𝟭. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 We can use Azure ML as the platform for managing experiments, compute clusters, and the model lifecycle. GPU VMs (NC/ND series) for training workloads that actually need the hardware. AKS when you're running inference at scale and need control over the infrastructure. Container Instances for short jobs where a full cluster is overkill, Functions for lightweight event-driven inference, and Batch for parallel workloads you want to run on a schedule. 𝟮. 𝗗𝗮𝘁𝗮 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 Data Lake Gen2 for training data at scale. Blob Storage for everything else. Azure SQL and Cosmos DB, depending on whether your metadata is relational or document-shaped. Managed Disks if your workload needs fast local storage attached to a VM. Azure Files for shared access across nodes. 𝟯. 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝗧𝗟 Data Factory for orchestrating pipelines. Synapse for large-scale analytics. Databricks, when your team lives in notebooks and needs distributed compute on Spark. Stream Analytics for real-time processing on incoming data. Functions and Event Hubs cover the lighter ETL cases. In practice, most teams end up with Databricks and Data Factory and call it done. 𝟰. 𝗠𝗟 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 AML Pipelines for reproducible training workflows, AML Studio, and Notebooks for the exploration and iteration phase. AutoML is when you want to benchmark quickly without writing the training code yourself. Prompt Flow for LLM-based applications. MLOps handles versioning, deployment, and monitoring across the lifecycle. 𝟱. 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Feature Store, Databricks FE, Data Explorer. The most underbuilt layer in most ML projects I've seen. Teams spend months on model architecture and three days on features. The ratio should probably be reversed. 𝟲. 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗮𝗻𝗱 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 Online Endpoints for real-time inference, Batch Endpoints when latency doesn't matter, and throughput does. AKS Deployment when you need full control over scaling and infrastructure. ACI for simpler containerised deployments. ONNX Runtime for optimised inference across hardware, Azure Edge AI, when the model needs to run closer to the data source. 𝟳. 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗮𝗻𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 ML Pipelines for training automation, Data Factory for data movement, Logic Apps for integration workflows. Functions and DevOps Pipelines for the CI/CD side. A model that runs reliably in production on a schedule looks nothing like a notebook; this is the layer where that gap gets closed, and most teams leave it too late. 𝟴. 𝗟𝗟𝗠 𝗮𝗻𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗜 Azure OpenAI for GPT-4 and embeddings with enterprise controls. Phi Models when you need something smaller and cheaper to run. Llama and Mistral through the model catalog if you want to open weights with more control over where they run. Cognitive Services for pre-built capabilities like vision, speech, and language. Azure AI Foundry is the newer unified platform for building and managing AI applications. Prompt Flow for chaining and testing LLM calls. 𝟵. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴, 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 𝗧𝗼𝗼𝗹𝗶𝗻𝗴 Azure Monitor and App Insights for observability, ML Monitoring for model drift and data quality in production. Azure Policy and Purview for governance, Entra ID for identity. On tooling: Azure CLI, SDKs, Azure DevOps, GitHub Actions, Bicep/ARM for infrastructure as code. And GitHub Copilot, of course, for AI-assisted coding.
- 27
@pvergadia ·
NEW: The SDLC taught us software is correct or broken. AI just made that binary obsolete. → SDLC: write rules → test pass/fail → ship → done → AIDLC: collect data → train → evaluate statistically → monitor forever → Your model can be "healthy" and silently wrong at the same time → Drift doesn't throw errors. It just quietly degrades. 80% of ML project time is data prep. Not code. You can't unit test a probability. The entire engineering contract: requirements, acceptance criteria, definition of done breaks when the output is "78% confident." The teams that figure this out early win the next decade. Full breakdown 👇
- 28
@suraj_sharma14 ·
Demos get likes. Systems get paid. Stop building tutorials. Start building production. 1. Add evals before you write features 2. Track cost per request from day one 3. Implement retry logic with backoff 4. Add circuit breakers for API failures 5. Build fallback chains for model outages 6. Log every prompt and response 7. Set up distributed tracing 8. Create alerts for latency spikes 9. Implement rate limiting per user 10. Add input validation for all queries 11. Build output filtering for safety 12. Set up automated regression tests 13. Create golden datasets for quality 14. Implement canary deployments 15. Add rollback
- 29
@0xlelouch_ ·
90% of LLMOps interviews in 2026 come down to these 7 points: 1) Serving architecture: batching, streaming, timeouts, and backpressure; explain p95 vs p99 and what you do when the model stalls 2) Cost control: token budgets, caching, prompt compression, smaller models; show you can cut a $3k/day endpoint to $800/day without breaking quality 3) Evals: offline golden sets + online A/B; talk about pass@k, win-rate, and how you prevent overfitting to a tiny test suite 4) Retrieval: chunking, embeddings, metadata filters, rerankers; describe how you detect bad recall and fix it without 10x index size 5) Safety + compliance: PII redaction, prompt injection defenses, audit logs; explain what you store, for how long, and why 6) Reliability: retries vs idempotency, circuit breakers, fallbacks; what happens when OpenAI/Azure/Anthropic rate-limits or returns partial outputs 7) Observability: traces across gateway RAG model, token metrics, per-tenant quotas; show dashboards that catch regressions within one deploy cycle
- 30
@DeRonin_ ·
Ran GLM 5.2 against Opus 4.8 this week, both wired into my agency stack for 6 days bottom line: GLM 5.2 is the first open model i'd actually trust with production marketing work free weights + run it on my own hardware + frontier-class output = the value math is wild some notes: 1. agent loop persistence: gave both an agent and let it work for hours. GLM stayed on task across 60+ steps before drifting. Opus still wins on the longest chains but the gap shrunk hard 2. brand voice: showed both 5 examples of how i write. GLM matched my tweet drafts ~88% of the time, Opus 94%. for outreach DMs the difference disappears entirely 3. structured outputs: GLM nailed strict formats 800+ times in a row, zero errors. this used to be Opus-only territory 4. failure mode: when GLM doesn't know something, it sometimes invents an answer instead of saying so. Opus admits uncertainty, which makes it safer to build automated workflows around 5. edit-not-rewrite: small fix on a long file. GLM rewrites everything. Opus only changes what you asked. matters when humans review every diff 6. self-hosted speed: 40ms response time on my own GPUs, no usage caps, nothing leaves my server. a permanent fast lane that's also fully private actually changes what you can build 7. cost: a full week of GLM ran cheaper than one day of Opus at the same volume. for the ~70% that don't need top-tier judgment, the math is over 8. where Opus 4.8 still earns the call: anything voice-critical, anything ambiguous, anything where the model has to read what i actually meant vs what i typed my new routing rule: GLM 5.2 for volume, structure, speed. Opus 4.8 for judgment, voice, edge cases first time the "open model parity" line actually felt crossed for production agent work
- 31
@Hartdrawss ·
Build boring AI companies. I think it's the biggest opportunity of the next 10 years. 1. Every AI company needs clean data, but nobody wants to clean it. A data labeling service that guarantees 99.9% accuracy and charges per record, not per project. Boring, essential, recurring. 2. Companies deploy models that drift and nobody notices until revenue drops. A model monitoring dashboard that sends a text when your accuracy falls below threshold. No dashboards, just alerts. 3. Your AI calls an API that changes its pricing overnight and your whole pipeline breaks. A compatibility checker that scans your dependencies daily and warns you before your weekend deploy. 4. Every RAG pipeline is held together by chunking decisions somebody made in a hurry. A chunking service with 20 strategies that A/B tests them on your data and picks the best one. Then charges a monthly fee. 5. Prompt engineering is still guessing. A prompt testing platform that runs your prompt against 100 historical edge cases from your dataset and tells you exactly where it fails. Pass/fail, done. 6. Every company has a Slack channel where people share AI outputs that are obviously wrong. A feedback collection tool that threads into Slack and logs every correction as a test case for your next fine-tune. 7. Model cards are mandated by regulation but nobody writes them. A template generator that reads your model's training data and inference logs and spits out a compliant card in 30 seconds. 8. Your AI hallucinates a customer's name twice and they churn. A hallucination filter that sits between your model and the user and catches factual errors before they reach a human face. 9. Every AI feature is built twice: once in a Colab notebook and once in production. A notebook-to-production converter that rewrites your experiments into deployable APIs. Boring, saves weeks. 10. Your team spends 40% of sprint time debugging why the embedding vector changed after a redeploy. A versioned embedding store that pins your vectors to a specific model snapshot so they never break. 11. AI vendors are switching to usage-based pricing and your bill fluctuates wildly. A cost prediction and budgeting tool that learns your usage patterns and caps spend per workflow. 12. Your support bot keeps apologizing when it doesn't understand. A confidence scoring layer that routes low-confidence queries to humans before the bot says "I'm sorry." 13. Every company has a drawer of AI experiments that didn't scale. A kill switch that automatically stops any experiment that hasn't shown improvement in 90 days. Saves cloud costs. 14. Your CEO wants to know why the AI declined two customer claims yesterday. A plain-English decision recorder that logs every model output with the top three influencing factors. 15. The same model runs differently on CPU vs GPU vs TPU. A hardware-agnostic inference wrapper that tests your model on every available chip and picks the cheapest one that meets your latency SLA. 16. Your fine-tuning dataset has 3,000 examples, 800 of which are duplicates. A dataset deduplication tool that flags exact, near-exact, and semantically identical rows. Boring, trivial, essential. Build boring !
- 32
@nurijanian ·
someone asked what high-leverage AI looks like on r/ProductManagement this week, the thread was full of people flexing transcript summaries. here are 11 takes on what the senior answers were really saying: Summarizing a 90-minute customer call in 3 minutes is still low-leverage AI. You went from a 90-minute read to a 3-minute read. The leverage shows up when that call feeds a system you query on Tuesday when sales closes a deal. The OP explicitly rejected Miro boards, Jira cleanup, and PRD drafting. The top replies ignored the rejection and described something harder: a living product brain fed by calls, Confluence, and closed-lost data. One PM runs closed-lost analysis every morning with Claude. Quarterly win/loss deck → daily dashboard. That loop changes roadmap bets. A prettier PRD template does not. High-leverage AI ingests signals on a schedule. Customer calls get transcribed, tagged against roadmap bets, and stored where the next prioritization conversation can pull them up. One-off prompts do not compound. Voice-of-customer work at scale is a system problem. You need ingestion, tagging, and retrieval before you need a better summary prompt. Most teams stop at the prompt. Competitive insight works the same way. A one-shot battlecard prompt helps once. I think the compound version is tracking why deals stall against the same competitor for six weeks without anyone reopening the research folder. Roadmap prioritization gets easier when your AI layer already knows which bets customer evidence supports. You stop re-litigating context every planning cycle because the context survived the last one. Several commenters pointed at Karpathy's second brain idea. The end state is weird but simple: you stop opening a blank PRD doc. You query structured context and export a ticket when engineering needs one. if your AI just touches artifacts you built a faster assistant not infrastructure
- 33
@udayan_w ·
there's a reason some people's agents keep getting better and others stay stuck. it's not the model. it's not the prompt. it's something simpler. everyone optimising skills and memory for their agents right now is creating a feedback loop. most don't call it that. they call it "fixing the prompt" or "tweaking the skill." but the structure is the same every time: run the agent, look at the output, find the gap, write a correction, run again. foundation model companies are doing the same thing. optimising feedback loops in post-training and fine-tuning. automatic feedback loops produce better systems. always have. always will. i started seeing this when i was building agents last year. the patterns felt familiar. not from engineering. from building teams. any founder who's good at building high feedback loop teams (clear signal, fast correction, right incentives) will train their agents the same way. the mental model transfers directly. the other thing that stuck with me was a professor @karpathy video where he walked through how GPT was built. that broke something open for me. the principles of how you improve a model through structured feedback became so foundational that i still apply them at the application layer. skills, memory, evals. make the feedback loop better. make the feedback rewards better. and you will see magic.
- 34
@LiorOnAI ·
You can now ship a production AI agent in one command. Google just released Agent Starter Pack, and it cuts setup time to about 60 seconds. From empty folder to deployed service, with infra included. 𝗧𝗵𝗶𝘀 𝗿𝗲𝗺𝗼𝘃𝗲𝘀 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱𝗲𝘀𝘁 𝗽𝗮𝗿𝘁 𝗼𝗳 𝗮𝗴𝗲𝗻𝘁𝘀 Building logic was never the blocker. Production was. This gives you everything around the model: • deployment configs • CI and automated tests • monitoring and logs • secure defaults You write behavior. The rest already works. 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗵𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀 1. Run a single CLI command. 2. A full project appears with backend, frontend, and cloud setup. 3. Templates cover common patterns like tool use and retrieval. 4. You can swap components without breaking the pipeline.
- 35
@hasantoxr ·
llm-d is the next step after vLLM. Most teams still serve open models like this: Spin up vLLM. Put it behind an endpoint. Add more GPUs. Watch latency spike. Pay the bill anyway. This repo shows the next step: You stop treating inference like one model server. You build the distributed serving layer around it. llm-d sits above model servers like vLLM and SGLang and helps you run LLM inference on Kubernetes at production scale. Inside: → Intelligent request routing → Prefix-cache aware scheduling → Load-aware balancing → KV-cache management → Prefill/decode disaggregation → SLO-aware autoscaling → OpenAI-compatible batch APIs → Benchmark-tested deployment recipes The crazy part is what this changes. Most AI teams think the model is the product. But once users show up, the real problem becomes: How fast can you serve it? How much GPU are you wasting? How many tokens can you push per second? How predictable is latency under load? How much margin disappears every time someone clicks “send”? The repo reports gains like higher throughput, faster TTFT, better tokens/sec, and stronger utilization across real benchmark setups. The wild part is the shift in thinking. vLLM made open models usable. llm-d is trying to make open-model inference survive production. This is what AI infra looks like when serving stops being a wrapper and starts becoming an operating system for GPUs. https://t.co/WM6viDYQbO
- 36
- 37
@xelebofficial ·
Why operating AI agents is becoming the next big challenge? The first wave of Agentic AI was about capability. Can an agent reason? Can it use tools? Can it complete tasks autonomously? The answer is increasingly yes. But a new problem is emerging. Once an agent can act, how do you actually operate it? The breakthrough of cloud computing wasn't just giving developers more compute. It was creating the infrastructure needed to run software reliably at scale. AI agents are reaching a similar moment. An agent may be able to research, analyze, code, communicate, and make decisions. But in production environments, organizations need more than intelligence. They need to know: What is the agent doing? Why did it make that decision? What happened when something failed? How does work continue across sessions? How do multiple agents coordinate safely? Today, much of the agent ecosystem is still solving these problems independently. Teams are building custom memory systems. Custom execution environments. Custom monitoring tools. Custom workflows for agent handoffs and recovery. The same way application infrastructure became a foundational layer for the internet, agent infrastructure is becoming a foundational layer for AI. Standard runtimes, durable memory and state, secure execution, observability, and auditability are no longer nice-to-have features. They are becoming the requirements for turning agents into real products. The next moat may not come from building agents that can do more. It may come from building systems that make agents reliable, transparent, and manageable at scale. Same technology. A different challenge. Not "Can the agent act?" But "Can we operate it?"
- 38
@alex_verem ·
The AI is 5% of the work. The 95% that breaks: → Observability (Langfuse, Braintrust, Helicone) - you can't debug what you can't see → Evals - regression suites for non-deterministic software. The new CI. → Durable runtime (Temporal, Inngest) - so a 10-minute agent run survives a server restart → Guardrails - prompt injection detection, PII redaction, output filtering → Memory layer - vector DBs (Pinecone, pgvector, Turbopuffer), retrieval, session state → Tools layer - MCP servers, sandboxed code execution (E2B, Modal), browser automation (Browserbase) → Auth + multi-tenancy - your agent calling Salesforce for customer A must NEVER see customer B's anything → Cost controls - agents in runaway loops burn $$ in minutes → Human-in-the-loop - approval gates for "spend more than $X" or "send external email" → Prompt versioning - prompts are code, treat them like code → Orchestration - plan-act-observe-repeat. Most serious teams are moving toward minimal orchestration + explicit state machines over heavy frameworks. → Model routing - LiteLLM, Portkey, OpenRouter for fallback, prompt caching, and version pinning so a vendor update doesn't silently change your product A CTO with 10+ years shipping production gave me the honest version: "Observability + evals + durable runtime + guardrails is the minimum viable production stack. Skip those four and you get the works-in-demo → on-fire-in-prod gap killing agent startups right now." The LLM is the easy part. Everything around it is the actual company.
- 39
@0xlelouch_ ·
90% of LLMOps interviews in 2026 come down to these 7 points: 1) RAG system design: chunking, embeddings, top-k, rerankers, and how you measure retrieval quality vs latency. 2) Evaluation: offline golden sets + online A/B, judge model pitfalls, and metrics for hallucination, refusal, and task success. 3) Reliability + cost: timeouts, retries, circuit breakers, caching, batching; explain your $/request math and p95 tradeoffs. 4) Data + privacy: PII handling, retention, redaction, prompt logging policy, and how you do tenant isolation in multi-tenant apps. 5) Guardrails: prompt injection defenses, tool allowlists, output schema validation, and what you do when the model violates constraints. 6) Deployment + rollout: model/version pinning, canaries, fallback models, feature flags, and how you rollback a bad prompt or policy change fast. 7) Observability: traces across LLM + tools, token counts, latency breakdown (retrieval vs generation), and dashboards that catch drift before users do
- 40
@suraj_sharma14 ·
If I had 6 months to become an AI Data Engineer. I'd do this. Stage 1: Python and SQL Foundations pandas, numpy, SQLAlchemy, query optimization, data modeling, schema design. Stage 2: Data Pipeline Orchestration Airflow, Prefect, Dagster, task dependencies, retry logic, backfills, scheduling, monitoring. Stage 3: Feature Store Architecture Feast, Tecton, Hopsworks, online/offline stores, feature serving, point-in-time correctness. Stage 4: Vector Data and Embeddings Pipelines Embedding generation, chunking strategies, vector DBs (Pinecone, Weaviate, Qdrant), hybrid search. Stage 5: Data Versioning and Reproducibility DVC, LakeFS, Delta Lake, snapshotting, lineage tracking, reproducible training datasets. Stage 6: Real-Time versus Batch Processing Kafka, Flink, Spark Structured Streaming, windowing, watermarking, exactly-once semantics. Stage 7: Data Quality and Validation Great Expectations, Monte Carlo, dbt tests, schema enforcement, anomaly detection, alerting. Stage 8: MLOps Integration and Model Registry MLflow, model versioning, feature-to-model handoff, training/serving skew detection. Stage 9: Cloud Data Platforms Snowflake, BigQuery, Databricks, partitioning, clustering, cost optimization, IAM, security. Stage 10: Data Governance and Lineage OpenLineage, data catalogs, PII handling, access controls, audit trails, compliance. Stage 11: Open Source and Portfolio Ship feature pipelines publicly, write data architecture docs, contribute to Feast/dbt, publish quality reports. Stage 12: Apply AI Data Engineer, Feature Platform Engineer, ML Data Specialist, Vector Pipeline Engineer, Data Reliability roles. Most people stay stuck watching tutorials. Builders get hired. @suraj_sharma14 #AIDataEngineering #FeatureStores #MLOps #DataPipelines #CareerGrowth
- 41
@LearnWithBrij ·
An AI agent isn't "just an LLM." It's a distributed system with reasoning at its core. That's the architectural shift everyone is waking up to. Most teams obsess over the model. The best teams obsess over everything around it. Because in production... → Stale context creates bad decisions. → Weak memory repeats the same mistakes. → Fragile tool contracts break execution. → Poor orchestration causes loops and dead ends. → Excessive permissions become security risks. → Missing observability makes failures impossible to debug. → Weak evaluation lets regressions reach users. → An unstable runtime turns a powerful agent into a demo. Here's the reality: The model might be only 10% of what makes an AI agent successful. The other 90% is engineering. Think of the stack like this: 🧠 Intelligence → Model 💾 State → Context + Memory 🛠️ Action → Tools + Skills 🔄 Coordination → Orchestration 🛡️ Trust → Identity + Guardrails 📊 Feedback → Observability + Evaluation ⚙️ Operations → Runtime These layers don't fail in isolation. A context bug becomes a reasoning bug. A reasoning bug becomes a tool call. A tool call becomes a real-world action. That's why building production AI agents is fundamentally different from building chatbots. You're not engineering responses. You're engineering systems that can: • Plan • Act • Recover • Explain • Improve Over and over again. The LLM provides intelligence. The engineering stack provides reliability. One question I keep thinking about: At what point does an orchestrated workflow stop being a pipeline... and become an AI agent? Is it autonomy? Self-correction? Long-term memory? Or something else entirely? Curious to hear your take.
- 42
@LinusEkenstam ·
Simulation. What nobody tells you. Simulation is one of the things we noticed early being a golden squeeze when working with LLMs. When building any tool that's fundamentally powered by an LLM, "what can be simulated?" is probably the first question we ask. always. Anyone building serious AI products will say the same. But they might not publicly disclose because a lot of the moat lies within these simulated environments. You might ask, why simulate at all? Is the LLM not going to give me the same information/output anyways? Well, that depends on the complexity of the task. On the context window currently loaded and being manipulated. For us, it's always been about context engineering. To get to a better contextual understanding of a task. Simulation is a cheat code. Secondly your simulations become useless if you can't score them appropriately. How you do evaluation matters. You want to be able to evaluate your outputs and engineered context blobs to be as deep and useful as possible. The reverse of this is SLOP. Zero shot nonsense that every now and then amazes, but the moment you want a robust system it all fails at a rate too large to make any sense at all. We want less slop and more greatness. Software is moving closer to gaming in this sense. Games inherently run in game engines, so by definition they run in simulated environments. Then to go one step further, simulation category games are where I see the most overlap/signal. Now jigs. In woodworking when you want to be able to do the same part over and over again. You don't measure every piece, you push it into a jig. Removing heaps of friction and repetitive work in the process. That otherwise can lead to failures (yeah measuring is extremely prone to errors). Obviously jigs are used in all sorts of manufacturing. But in software we kinda forgot the importance of this. If you're building a complex agentic system/behaviour, not using a jig. Will mean that there are far too many points to measure manually. That regardless of your skills will lead to you making measurement errors that then lead to unreliable outputs. Constructing a jig around what you're building to be able to tweak and adjust parameters/prompts/engineered-context-blobs, on both low and high level gives you the necessary levers to build, not something good, but something fantastic. Loads of creative legacy software is built around this idea already. Any node based system is fundamentally a jig. There can be non-visual parts to the jig too. If you're building agentic systems, being able to simulate tens of thousands of pathways, measure, analyze, tweak, re-run, will make your product stand out, excel where others fall short. At Flocurve we built a jig around our growth and outreach agents. 100+ parameters. Signal matching on one end, context analysis on the other. Everything in between is a lever. Tweak one, watch the others move. Run it across thousands of prospects, see what holds. What comes out is consensus. Not a single model guessing. A system that has triangulated the same prospect from 100+ angles and agreed. That's how you go from "this looks like a lead" to actual matchmaking. Customer targeting that doesn't degrade the moment you scale. It's funny because in training LLMs this is largely what's going on. Tweaking, simulating, pre-runs, re-runs, emulate, simulate and evaluate. But somehow the successful builders on the inference end of this are not being very vocal about how they are doing the same. Training does this in public. Inference does it in private. The builders who win at inference will be the ones treating it like training.
- 43
@_vmlops ·
THERE'S A LIVE MAP SHOWING THE CARBON FOOTPRINT OF ELECTRICITY ACROSS THE ENTIRE WORLD RIGHT NOW and most engineers have never seen it Electricity Maps tracks the carbon intensity of electricity in real time every 15 minutes across 190+ countries green zones mean clean energy. red zones mean dirty energy and it changes constantly because here's what most people don't know the electricity you use right now has a different carbon footprint than the electricity you'll use tonight wind is up... cloud cover dropped solar output... a coal plant just came online the grid is never the same twice and this matters massively for MLOps and DevOps engineers google already uses this data to decide when and where to run compute workloads shifting AI training jobs to cleaner grids in real time your cloud infrastructure could do the same open source... live... free to explore → https://t.co/LiOopaeUoL
Watch video
- 44
@llama_index ·
Common Failure Modes Break VLM-Powered OCR in Production. 🔁 Repetition Loops — model spirals into infinite whitespace, exhausts resources, cascades latency across your system 🛑 Recitation Errors — safety filters hard-stop legitimate extractions as "copyright violations" Same pipeline. Completely different root causes. Completely different fixes. Our enginerring leadership broke down what went wrong and how we solved both 👇 https://t.co/fFkLmnG11h
- 45
@elvissun ·
some realizations from running agents in production recently: when it works, it REALLY works when it fails, it fails HARD been working on instrumenting things ruthlessly and creating feedback loops to catch every failure mode possible (there are lots of them) this is a much harder problem to solve than simply going from 0 to 1
- 46
@byebyescaling ·
THOUGHTS ON REFRAMING DIFFICULT AGENT ENGINEERING PROBLEMS IN PRODUCTION I have been working on some open ended and extremely difficult agent engineering problems in production. The main difficulties arise from trying to make the agents function effectively over long time horizons. @peakji 's @ManusAI blog post crystallized something I hadn't seen stated this cleanly anywhere else : KV-cache hit rate as the dominant production metric for agents. This has inspired me to articulate some patterns I've been reflecting on independently. (Also seems pretty timely now that @JayaGup10 et al have brought Context Graphs into the limelight!) 1. Context engineering is isomorphic to memory hierarchy design. You have registers (system prompt), L1 (recent turns), L2 (retrievable state), and disk (file system). The engineering problem is identical: minimize cache misses while maximizing information density at each level. The fact that we're rediscovering 1960s OS principles in 2025 is both amusing and expected! 2. The attention mechanism has a finite entropy budget. As context length grows, attention distributions flatten - this is "context rot." But it's predictable and measurable. You can estimate attention entropy as a function of sequence length and design your context layout around it. Most people treat the context window as a bag of tokens when it's actually a structured resource with positional priors. 3. The https://t.co/R2MpAqzHx1 pattern Manus describes is super clever: by rewriting goals at the tail of the context, you're placing high-salience tokens where recency bias is strongest. This isn't a hack it's exploiting the model's inductive biases rather than fighting them. Attention steering through structured repetition. 4. Error traces in context aren't just "keeping failures visible." They provide implicit negative gradients at inference time. The model's in-context learning treats errors as counterfactual feedback. Removing them is equivalent to training on positive examples only - you lose the decision boundary. 5. Most production agents I've seen that actually work are state machines with LLM-powered transition functions! The graph is deterministic, the edges are learned. This separation is load-bearing - when you let the model own the control flow, you're searching over a combinatorially larger space with no structural priors to regularize it.
- 47
@krishnan ·
The unglamorous part of AI is becoming the moat. Everyone is covering model launches. Netflix's new engineering writeup points to the harder Day 2 question: can you run LLMs like production infrastructure? Netflix says (https://t.co/tuYfXJNdnn) it runs the full LLM serving stack inside its own production environment, not as a separate ML sidecar. The interesting part is not "we use LLMs." It is the architecture: vLLM as the paved-path engine, NVIDIA Triton underneath, a Java control plane for deployment and rollout, unified metrics, multi-region upgrades, and constrained decoding inside the generation loop. That last piece matters. Constrained decoding means the system does not wait for the model to produce bad output and then clean it up afterward. It pushes rules into token generation itself. Each step masks invalid next tokens based on the state of the request. In plain English: the model is guided to produce valid output by construction, not by apology. The tradeoff is real. More control usually adds latency, CPU coordination, batching complexity, and debugging pain. Netflix's post is useful because it talks about those operational choices instead of pretending the hard part ends when a model answers a prompt. This is where enterprise AI is going. The durable advantage will not be "we picked the best model this quarter." It will be: - Can you route workloads across real-time and batch paths? - Can you measure token throughput, KV cache use, and prefix cache hit rates? - Can you roll out model changes without breaking downstream systems? - Can you enforce output constraints before bad work hits the workflow? Watch the metric nobody puts in the launch deck: how much AI work survives contact with production load. #AIInfrastructure #LLMOps #EnterpriseAI #PlatformEngineering #AIOps #GenerativeAI
- 48
- 49
@JeremyCMorgan ·
Speculative decoding quietly became production infrastructure this year: EAGLE-3 is now default in vLLM, SGLang, and TensorRT-LLM. This breakdown of Saguaro, Nightjar, and Intel's universal draft models is the clearest practitioner guide to what's actually running in your stack right now. https://t.co/xXnMpymV39
- 50
Best Tweets by Topic