Increase your LLM rate limits with our load balancing feature.
Go to the Load balancing page
Create new load balancer
Add models
Add model
to add models and specify the weight for each model and add your own credentials.Copy group ID to your codebase
model
parameter will overwrite the load_balance_group
! Add load balancing group in code (Optional)
models
field will overwrite the load_balance_group
you specified in the UI.Example code
fallback
field once have any outages. Check out the Fallbacks section for more information.
customer_credentials
field and specify the weight for each deployment.
Example:
gpt-3.5-turbo
in an OpenAI deployment, you could specify it in the available_models
field or do it in the platform.
Learn more about how to specify available models in the platform here.
Example code:
loadbalance_models
parameter is deprecated. You should use the load_balance_group
parameter instead.
Example code
Depracted params