Load balancing

Load balancing is a feature that allows you to balance the request load across different models or deployments. You could specify weights for each model/deployment based on their rate limit and your preference.

See all supported params here.

​

Load balancing between models

You could specify the load balancing weights for different models. This is useful when you want to balance the load between different models from different providers.

{
// you don't need to specify the model parameter, otherwise, the model parameter will overwrite the load balance group
    "messages": [
        {
            "role": "user",
            "content": "Hi, how are you?"
        }
    ],
    "load_balance_group": {
        "group_id":"THE_GROUP_ID" // from Load balancing page
    }
}

{
  "load_balance_group": {
      "group_id":"THE_GROUP_ID", // from Load balancing page
      "models": [
        {
          "model": "azure/gpt-35-turbo",
          "weight": 1
        },
        {
          "model": "azure/gpt-4",
          "credentials": { // add your own credentials if you want to use your own Azure credentials or custom model name
              "api_base": "Your own Azure api_base",
              "api_version": "Your own Azure api_version",
              "api_key": "Your own Azure api_key"
          },
          "weight": 1
        } 
      ]
  }
}

You could also set up fallback models to avoid errors. It will fall back to the list of models you specified in the fallback field once have any outages. Check out the Fallbacks section for more information.

​

Load balancing between deployments

You could go to the platform and add multiple deployments for the same provider. You could specify the load balancing weights for each deployment, which could be helpful when you want to enhance rate limits for a single provider.

​

Deprecated params

The loadbalance_models parameter is deprecated. You should use the load_balance_group parameter instead.

{
    // ...other parameters...
    "loadbalance_models": [
        {
            "model": "claude-3-5-sonnet-20240620",
            "weight": 34,
            "credentials": { // Your own Anthropic API key, optional for team plan and above
                "api_key": "Your own Anthropic API key"
            }
        },
        {
            "model": "azure/gpt-35-turbo",
            "weight": 34,
            "credentials": { // Your own Azure credentials, optional for team plan and above
                "api_base": "Your own Azure api_base",
                "api_version": "Your own Azure api_version",
                "api_key": "Your own Azure api_key"
            }
        }
    ]
}