Aggregate Models

An Aggregate Model is a feature module that wraps one or more underlying large language models (LLMs) into a unified interface. External callers only need to connect to the aggregate model without worrying about backend model routing and load distribution.

Overview

In practice, you may need to consolidate multiple models into a single entry point, for example:

• Assigning different underlying models to different users or clients
• Implementing model load balancing and failover
• Hiding internal model details from external consumers

Aggregate Models address these scenarios. Once created, aggregate models can be exposed externally through API Clients.

The Aggregate Model management entry is located in the Aggregate Models tab under the Model API page in admin mode.

Creating an Aggregate Model

In the Aggregate Models tab, click the Create Model button.

Basic Information

• Model Name: The public-facing name of the aggregate model. Only letters, numbers, and underscores are allowed. Callers will reference the model by this name.
• Description: Purpose of the aggregate model.

Available Models

When creating an aggregate model, you need to select the specific underlying models to use. You can select one or more configured LLMs. When multiple models are selected, the system automatically routes requests based on quotas and load.

Testing a Model

After creating an aggregate model, you can click the Test button in the list to verify it. In the test window, enter a message, and the system will call the aggregate model and return results, making it easy to confirm the configuration is working correctly.

Managing Aggregate Models

In the aggregate model list, you can view all models' names, descriptions, and associated underlying models. Tenant administrators can edit or delete aggregate models. Deleting an aggregate model does not affect the underlying LLM configurations.

Viewing Usage

Aggregate model usage data can be viewed on the API Usage Data page. See API Usage Data.