LLM Evaluation with Opik
Expert Instructors
Taught by industry leaders
Self-Paced
Learn at your own speed
Built with Opik
Use open source tools and models
Course Description
- Level: Beginner
- Duration: 1 Hour
- Audience: Data Scientists/Software Engineers
- Prerequisites: Basic ML knowledge, Python experience
Who Is This For?
- AI Developers
- Anyone curious about LLMs
- Engineers
- Data Scientists
Why This Course?
This is the only course completely focused on applying state-of-the-art LLM evaluation techniques to real world applications. We will cover some theory, but this is first and foremost a course in applied AI, not mathematics.
Taught By An Expert
Elvis is the co-founder of DAIR.AI, where he leads all AI research, education, and engineering efforts. He focuses on training and building large language models (LLMs) and information retrieval systems. Previous to this, he was at Meta AI where he supported and advised world-class products and teams such as FAIR, PyTorch, and Papers with Code. He was also previously an education architect at Elastic where he developed technical curriculum and courses for the Elastic Stack.
What You’ll Learn
Syllabus
1 Course – 8 Lessons – 1 Project
1 – Introduction to LLM Evaluation
- Brief introduction to LLM evaluations
- Explore the challenges of evaluating LLM applications
- Overview some common use cases
2 – Tools & Frameworks for LLM Evaluation
- Explore the LLM evaluation ecosystem
- Familiarize yourself with Opik
- Stand up your first simple evaluation suite
3 – Project: Evaluating a Chatbot
- Kickoff the main section of the course with a real project
- Learn about common chatbot architectures
- Implement evaluations for a real LLM application
4 – Building an LLM Evaluation Pipeline
- Go from simple evaluations to robust evaluation pipelines
- Explore common workflows for production evaluation systems
- Get your feet wet with manual evaluations
5 – Heuristic Metrics for LLM Evaluation
- Familiarize yourself with some “classic” metrics
- Implement heuristic metrics from scratch
- Understand the benefits and challenges of heuristic evaluations
6 – LLM-Based Metrics for LLM Evaluation
- Learn about LLM-as-a-judge metrics
- Implement custom LLM-based metrics from scratch
- Learn to test for hallucinations, factuality, and more
7 – Testing & Monitoring LLM Applications
- Learn to monitor deployed LLM applications
- Implement LLM unit tests with PyTest and Opik
- Understand the role of observability in LLM applications
8 – The Future of LLM Evaluation
- Explore advanced techniques for LLM evaluation
- Understand safety evaluations and responsible AI
- Overview the next steps in your learning journey
Frequently Asked Questions
What are the prerequisites for this course?
This course assumes no advanced math background. We will not be diving deep into the theory behind LLMs. All you need to get started is some basic proficiency in Python and a general understanding of deep learning.
Will it cost me anything?
The course content is 100% free. Every lesson can be completed using completely free and open source models via LiteLLM and Opik. You can also use the OpenAI or Anthropic API, if you prefer.
How much time should I commit?
The course is self-paced, so you can spend as little or as much time as you want. That said, students who set aside a meaningful block of time each week—whatever “meaningful” means for your schedule—tend to see the best results.
How long will this course take?
Your time to completion will vary depending on how much time you have available. In general, we recommend one week per module as a realistic pace for most people, meaning the course would take six weeks total. Of course, you can take as long as you’d like.