A Hands-on Testing & CI Workshop

Testing LLM-based applications has become one of the most crucial challenges in modern software development. While traditional software testing gives us clear pass/fail criteria, how do you verify that your AI is consistently giving good responses? When is a response “correct enough”? And how do you automate this testing process in a way that scales?

In this hands-on workshop, we tackle these challenges head-on by building and testing three different types of AI applications. Rather than getting lost in theoretical discussions, we focus on practical solutions that you can implement today.

Watch the recap video above, and/or sign up to join the next one! Register here – we are running them at 9am PT / 5pm UK every Monday.

The traditional approach to testing AI applications often relies on manual review and subjective evaluation – also known as testing based on “vibes”! A team member might spend hours chatting with the AI, trying to catch edge cases and inconsistencies. While this has its place, it’s neither scalable nor reproducible.

Instead, we demonstrate a more systematic approach using Helix.ml’s testing framework. The key insight is using another AI model as an automated evaluator (judge), with clearly defined criteria for what makes a response acceptable. This, plus the tooling and configuration format to run these tests automatically, creates a reproducible testing process that can be integrated into your CI/CD pipeline.

Throughout the workshop, we create three distinct applications that showcase different testing challenges:

1. A Comedian Chatbot: Seems simple, but raises interesting questions about consistency and personality. How do you verify that every response is actually a joke? We show how precise prompt engineering and automated testing can ensure consistent behavior.

2. Document Q&A System: Using real HR documentation, we build a system that can accurately answer policy questions. This demonstrates how to test against ground truth while allowing for natural language variation.

3. Exchange Rate API Integration: We tackle the challenges of testing AI systems that interact with external APIs, ensuring they handle currency pairs correctly and present information clearly.

The most exciting part? We show how to automate all of this testing in your CI pipeline. By the end of the workshop, you’ll see how to:

• Write testable specifications for AI applications in YAML

• Create automated evaluations using LLM judges

• Integrate these tests into GitHub Actions or GitLab CI

• Deploy tested changes automatically

We’re running regular workshops to help teams implement these testing practices. Join the next workshop to learn these critical skills to build reliable GenAI applications that have access to knowledge and API integrations to business systems.

Register for the next workshop here: Register – workshops run at 9am PT / 5pm UK every Monday.

We also offer private workshops to help you implement these testing practices with your specific use cases. Email luke@helix.ml to schedule a session.

The code and examples from this workshop are available on GitHub: https://github.com/helixml/testing-genai

Watch the walkthrough video:

Building reliable AI applications doesn’t have to be a shot in the dark. With the right testing framework and practices, you can develop AI systems with the same confidence you bring to traditional software development. Join us in the next workshop to learn how.

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!