Proxy
Fallbacks
Avoid downtime with failover models
Keywords AI will catch any errors occurring in a request to model and fall back to the list of models you specified in the fallback_models
field. This is useful to avoid downtime and ensure that your application is always available.
See all Keywords AI params here.
Set up fallbacks on Keywords AI platform
Go to Settings -> Fallback -> Click on Add fallback models
-> Select the models you want to add as fallbacks.
Set up fallbacks in code
Automatic fallbacks
We use Saftey net to automatically fallback to a system assigned model if fallback models are not responding or specified.
Turn on the Safety net here.
Exceptions for triggering fallbacks
- You turned off the safety net and did not specify your own fallback models
- You uploaded some invalid
customer_credentials
Fallback logic with retries
Below is the pseudo-code for the fallback logic with retries:
- To learn more about how
generate_response_with_load_balance
works, see the Load Balancing section. - To learn more about how
filter_by_hard_requirements
works, see the Model Filtering section.