Langfuse

In June I gave a talk at AI Engineer World's Fair SF on how the best teams we work with at Langfuse are now self-improving their agents. The loop is the same as before: trace production, build datasets, write evals, experiment, ship. What's new is who drives it. Agents now propose fixes and hill-climb against your evals. Over the last month we've also seen teams let agents suggest new dataset items and evaluators based on errors they find in production. My take: don't hand off everything. Fully automated loops burn a lot of tokens and produce a lot of slop. Stay involved at the top, deciding what goes into the dataset, what counts as failure, and which changes ship, and let agents handle the tedious parts. You spend less time, and quality goes up, because an agent can read far more traces than you ever will. This also changes the data layer. Langfuse used to be mostly write-heavy. Now agents read and query a lot more of it, and traces from a year ago are useful context. So you want to own that data and keep all of it, unsampled. Link to full recording in comments!

Observabilidade de agentes em escala: o caso do iFood com Langfuse. Como saber o que seus agentes de IA realmente fizeram, onde acertaram e onde falharam? No dia 8 de outubro, em São Paulo, Felipe Rodrigues Garé , Sr. AI Engineer no iFood, mostra como o time usa ClickHouse e Langfuse para rodar agentes de IA sobre dezenas de terabytes de logs. Investigações que levavam uma semana de trabalho de um analista hoje terminam em cerca de duas horas. Quinta-feira, 8 de outubro, das 18h às 21h, em São Paulo. Gratuito, vagas limitadas. Solicite participação: https://lnkd.in/eewCB442 No mesmo dia, das 9h às 17h, temos o treinamento gratuito Observabilidade com ClickStack: laboratórios práticos com ClickHouse, OpenTelemetry e HyperDX, com certificação no final. Reserve seu lugar: https://lnkd.in/eDWwKngz #ClickHouse #Langfuse #Observabilidade #AgentesDeIA #SaoPaulo

Two great engineering community events coming up at Deutsche Bank in Cary: 📅 Oct 7 - TriJUG with Java Champion Bruno Souza: building a great developer career in the AI era. 📅 Oct 15 - Triangle Tech Talks with Marc Klingen, Langfuse (now part of ClickHouse) co-founder, visiting from Berlin: building agents that work. Join us! Details and registration links in the comments. #TriJUG #TriangleTechTalks #AIEngineering

Evals are the most important part of building AI systems, but there’s not a lot out there about building them well. So I hosted a fireside chat on evals with Liam Bush (LangChain), Lotte Verheyden (Langfuse), Braden Holstege (Mercor), Emmanuel Turlay (CoreWeave), and Soumya Mohan (Galileo) to walk me through the tricks of the trade - how they actually work in production, where teams get them wrong, and where they're headed over the next 12 months. We get into: -online vs offline evals -why running evals can cost you more than running the actual agent -what nobody can see after the agent makes a tool call -why understanding accounting rules might be harder than AGI -and much, much, more… Some of my favorite parts: 1/ Running your evals can cost more than running your agent. "We deployed an agent where we had a 100% sample rate initially, and it 4x’d the budget, because we're spending 3x as much tokens running the live evals as we're actually running the agent." Mercor’s team caught this on one of their own deployments. If you're running live evals on every user input, you're paying for a second agent whose entire job is watching the first one - that gets expensive, really fast. Over time, they reduce the sampling rate to reduce the cost, especially as the agent improves. 2/ “Ship WITHOUT evals.” "I can't believe I have to say that, because I would say the opposite to people a few years ago. But for an agent that is not safety critical or not regulated, just launch and let it fail and learn from that, and build your evals from there." You can guess how users are going to use your product all you want, but you won’t actually know until you launch and observe them using it. Broad online evals catch the stuff you'd never have tested for, and you convert that into offline evals afterward. He did add one caveat - don't let it fail for too long or your users will churn. 3/ Never ask an LLM to rate anything 1 to 10. "It's tempting to ask the LLM judge to score on a scale of zero to ten. This is really bad. You should probably not do that at all." Just like its hard for humans to know the difference between a 5 and a 6, its hard for models too. Classifiers are cheaper and more accurate. Example: if you're grading politeness on call transcripts, your options should be insufficiently polite, sufficiently polite, and extremely polite: "I don't want my agent to be extremely polite. That's actually a bad case. I don't want overly wordy responses, or the user will be frustrated if something is not working and the LLM is like, 'you're absolutely right' - because we all hate that response." A 1-10 scale gives that response a 10. Categories force you to decide what you actually want, which is the harder and more useful exercise. Link to full episode in the comments. — Follow me Basil Chatha for everything AI Agents. And thanks to AngelList for making this possible!