LLM Zoomcamp

I just wrote an article about doing evaluations It complements what we studied in the course so I'm sure you'll find it useful https://alexeyondata.substack.com/p/how-to-do-evals-in-2026 Enjoy!

Your certificates are not ready yet. There will be a separate announcement about it after the second attempt is over. Please be patient =)

The results for attempt 1 are ready Good job everyone!

Great job working on your projects! Now it's time to learn from your peers. If you submitted your project for attempt 1, you will find your peer review assignments here: https://courses.datatalks.club/llm-zoomcamp-2026/project/project1/eval Have fun!

Stream about the FAQ assistant Join now or watch later in recording: https://www.youtube.com/watch?v=CyH61xiYSnk

Today is the deadline for Attempt 1 of the final project. Submit your project by 1:00 AM CET today. As you continue working, check Module 7 for a complete project example that you can use as a reference. It covers: • Generating data, setting up the project, and building the initial RAG flow • Evaluating retrieval with ground truth data, Hit Rate, MRR, and boosting • Evaluating RAG with LLM-as-a-Judge and model comparisons • Building a Flask API, ingestion pipeline, and project structure • Adding Docker Compose, PostgreSQL logging, and Grafana monitoring • Reviewing the final result, costs, and practical project tips • Applying different chunking strategies to articles, transcripts, and slides For submission guidelines and peer-review requirements, read the Project section. If you cannot submit today, you will have Attempt 2, with a deadline of Tuesday, August 11, at 1:00 AM CET.

In the course we use the FAQ dataset as the main running example In this article I describe how it's curated and how the FAQ assistant you see in Slack works https://alexeyondata.substack.com/p/rebuilding-a-faq-system-for-datatalksclub Also, if you'd rather watch me explain it than read the article, we'll have a live stream about it on YouTube https://luma.com/fb91wje9 See you soon! I hope you're having fun with your projects!

We're talking about using tracking to get insights user tracing using Snowplow and Vercel AI https://www.youtube.com/watch?v=A2nMgZWyza8 Watch now or later in recording

We are hosting a live workshop on tracking and personalizing AI agents. 📅 Tuesday, July 21 🕟 16:30 CEST Agent applications usually use the current conversation as context. But users also interact with the rest of the product. They open pages, change filters, compare options, and complete different steps before asking the agent a question. In this workshop, we will show how to capture that behavioral data and pass it back into an AI agent. We will use a travel chatbot as an example and instrument it across 3 layers: • Client-side user activity • Server-side application events • Agent actions and decisions You will learn how to: • Set up a pipeline • Validate events against schemas • Track activity across the application • Compute live session attributes • Add behavioral context to the agent's system prompt Register here: https://luma.com/vz9iv4f2

It's time to start working on your final project. Check out Module 7 to see an example of a complete project as a reference. It covers: • Intro: Generating data, setting up the project, initial RAG flow • Evaluating Retrieval: Ground truth data, Hit Rate, MRR, boosting • Evaluating RAG: LLM-as-a-Judge, comparing models • Interface and Ingestion: Flask API, ingestion pipeline, project structure • Monitoring and Containerization: Docker Compose, PostgreSQL logging, Grafana • Summary: Final result, cost analysis, tips for your project • Chunking for Longer Texts: Different chunking strategies for articles, transcripts, slides For more information about the project guidelines and peer review, read the Project section. You also asked us to postpone the deadline for dlt workshop homework. You'll have one more week to complete it.

We have prepared the homework for the dlt workshop as well as the monitoring module And they are both about monitoring and observability, and in both you will learn something new in addition to what we covered in the course: - for the monitoring homework we'll learn about OTel and instrument the RAG assistant with OTel collectors - for the dlt hub homework, we'll use Pydantic Logfire and see how we can use dlt to ingest the traces back from Logfire for our local analysis Have fun!

This week in LLM Zoomcamp: Module 5, Monitoring. We build: - A Streamlit chat app with RAG - Metric capture for LLM calls and cost - PostgreSQL storage for conversations - User feedback with thumbs up and thumbs down - Automatic relevance evaluation with a built-in judge - Streamlit and Grafana dashboards - A Docker Compose setup for running everything together You’ll see what your LLM application is doing after it is deployed. You need conversations, feedback, cost, relevance, and dashboards. Otherwise it is hard to know what to improve next. Materials: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/05-monitoring

On Tuesday, July 21, we have a workshop on tracking and personalizing AI agents with Snowplow and the Vercel AI SDK. Your agent can see the chat history but not what the user is doing in the product. If you want the agent to respond based on the user's current session, capture that context and make it available to the model. This workshop shows how to do that with a Next.js travel chatbot. The approach: 1. Track client behavior (page views, clicks, filters) 2. Track server-side events (purchases, bookings) 3. Track agent decisions (tool calls, outcomes) 4. Validate events against schemas in real time 5. Compute session attributes with Snowplow Signals 6. Inject those attributes into the system prompt You'll set up the pipeline, add tracking at each layer, and use live session data to shape the agent's responses. Tuesday, July 21 4:30 PM - 6:00 PM GMT+2 YouTube Register: https://luma.com/vz9iv4f2

We're starting in a few minites https://www.youtube.com/watch?v=rG2YW3YCq64 Join now or watch later in recording

We will have office hours today with Will from Kestra at 18:00 CET We will share the link here 5-10 minutes before the start If you want to get a reminder, you can use this link https://luma.com/d19r4ko1 You can ask questions in advance using this link https://app.sli.do/event/pH1w9PSq8dJZahTia24SKe

We're starting in 3 minutes https://www.youtube.com/watch?v=A0LmmZf-ggM Join now or watch later in recording

As part of this module, we’ll also look at agent traces during dltHub workshop: the final answers, tool calls, and intermediate steps your agents produce while solving tasks. You’ll build a pipeline that ingests agent traces and transforms them into structured, queryable data. Then, you’ll turn that data into reports that show what your agents are doing, where they fail, and how you can make them better. You’ll learn how to: • Capture and normalize agent traces into a consistent schema • Transform and model nested, variable-length trace data with dltHub Pro • Deploy your pipeline and build reports that surface real insights about agent performance By the end, you’ll have a working reporting layer for your AI agents, so you can debug faster, evaluate behavior more systematically, and improve your LLM applications with data. We start today, at 4:30 PM. Join here: https://luma.com/3sint80a

This week in LLM Zoomcamp: Module 4, Evaluation. Once you have retrieval and generation, you need to measure quality. In this module, we evaluate search, RAG, and agent systems systematically. We cover: - Generating ground truth data with an LLM - Evaluating retrieval with Hit Rate and MRR - Tuning search parameters using metrics - Generating RAG answers for test questions - Using LLM-as-a-judge for answer quality - Saving final answers and tool-call trajectories for agent evaluation This is one of the most important parts of working with LLM applications. Without evaluation, every change is a guess. With evaluation, you can compare retrieval settings, prompts, models, and agent behavior using repeatable checks. Materials: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/04-evaluation Course: https://github.com/DataTalksClub/llm-zoomcamp

We're talking about switching to AI Engineering: https://www.youtube.com/watch?v=3IFENyERGgQ Join now or watch later in recording

This week in LLM Zoomcamp: Module 3, AI Orchestration with Kestra. After RAG and vector search, we move to workflows. An LLM application often becomes a sequence of steps: - Prepare data - Run retrieval - Call models - Store outputs - Handle retries - Inspect failures - Track cost and observability When this is all hidden in one script, it is hard to run, debug, and maintain. In this module, Will Russell from Kestra shows how to orchestrate AI workflows with Kestra. The module covers context engineering, Kestra setup, AI Copilot, RAG workflows, agentic workflows, multi-agent systems, and production topics like cost, security, and observability. Materials: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/03-orchestration