Tetrate steps up to handle traffic management for AI agents powered by multiple LLMs

Service mesh company Tetrate Inc. wants to help reduce the costs and improve the reliability of artificial intelligence agents with the latest addition to its AI gateway tool.

Today it announced the availability of the Tetrate Agent Router Service, a managed solution that makes it simpler for developers to direct AI queries and requests to AI agents to the most suitable model, based on their priorities, such as query and task complexity, inference costs and model performance or speciality.

According to Tetrate, this kind of flexibility is exactly what developers need. The Agent Router Service acts like a centralized tool for controlling AI traffic. It allows them to work around the limitations of various large language models, avoid vendor lock-in and mitigate cost overruns.

Tetrate made its name as one of the chief backers of the open-source Envoy service mesh, and its main product is the Tetrate Service Mesh, which is used by developers to manage, connect and secure cloud-native applications through application programming interfaces. It provides features for controlling traffic management, observability and security, and helps to enhance the performance of apps running in Kubernetes environments.

More recently, the company has pivoted to stay relevant in the rise of the AI industry, and its key offering there is the Tetrate AI Gateway, an open-source project that helps organizations integrate generative AI models and services into their applications. Through its unified API, developers can manage requests to and from multiple AI services and LLMs.

With the Tetrate Agent Router Service, developers are getting even more control. It allows them to access various AI models with their own API keys, or use keys provided by Tetrate. It also provides features such as an interactive prompt playground for testing and refining AI agents and generative AI applications, automatic fallback to more reliable and affordable models, plus A/B testing tools for evaluating model performance.

Tetrate said it supports multiple generative AI use cases. As the name suggests, it’s primarily focused on AI agents, which are autonomous AI systems that can perform tasks on behalf of their users without supervision. In this case, it will coordinate API calls across multiple LLMs, delegating the tasks assigned by the user to the most appropriate one.

In the case of AI chatbots, the Tetrate Agent Router Service will route the conversation to the most responsive and/or cost-effective model, based on the developer’s priorities. This can help to reduce latency and manage high traffic more efficiently.

It does a similar thing for AI coding bots too. With the router, applications can respond dynamically to user’s commands, based on the desired programming language, compliance policies and the contest of the request. In other words, it simply ensures the request goes to the best code generation model for each specific job.

Holger Mueller of Constellation Research Inc. said Tetrate’s router should come in handy for developers because the widespread adoption of AI tools is causing complications, resulting in heavy traffic that leaves some applications stuck with suboptimal AI models. “This means users face higher costs and struggle with diminished capabilities, but it can be rectified easily enough by Tetrate’s router, which automatically sends requests to the most appropriate model,” the analyst said. “It’s like an Uber for AI applications, helping them to select the best model each time.”

AI traffic routers are fast becoming essential tools for AI developers. In Gartner Inc.’s latest Hype Cycle for AI in Software Engineering, published in June, the analyst firm explains that they help route each prompt or query to the most effective model, and play a major role in enhancing the quality of AI applications that, increasingly, rely on multiple LLMs.

“GenAI model routers help achieve performance-cost trade-offs in harnessing model enhancements and innovations while limiting costs,” Gartner said in its report.

Tetrate Head of Product Management David Wang said developers have major headaches, because they’re under pressure to build high-quality AI applications that deliver to expectations, without overrunning tight budgets placed on them.

Image: SiliconANGLE/Meta AI

Enjoy the perfect blend of retro charm and modern convenience with the Udreamer Vinyl Record Player. With 9,041 ratings, a 4.3/5-star average, and 400+ units sold in the past month, this player is a fan favorite, available now for just $39.99.

The record player features built-in stereo speakers that deliver retro-style sound while also offering modern functionality. Pair it with your phone via Bluetooth to wirelessly listen to your favorite tracks. Udreamer also provides 24-hour one-on-one service for customer support, ensuring your satisfaction.

Don’t miss out—get yours today for only $39.99 at Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!