LLM Proxy

POST

api

chat

completions

import requests
def demo_call(input, 
              model="gpt-4o-mini",
              token="YOUR_KEYWORDS_AI_API_KEY"
              ):
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {token}',
    }

    data = {
        'model': model,
        'messages': [{'role': 'user', 'content': input}],
    }

    response = requests.post('https://api.keywordsai.co/api/chat/completions', headers=headers, json=data)
    return response

messages = "Say 'Hello World'"
print(demo_call(messages).json())

OpenAI compatible parameters

To use Keywords AI parameters, you can pass them in the extra_body parameter if you’re using the OpenAI SDK.

import requests
def demo_call(input, 
              model="gpt-4o-mini",
              token="YOUR_KEYWORDS_AI_API_KEY"
              ):
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {token}',
    }

    data = {
        'model': model,
        'messages': [{'role': 'user', 'content': input}],
    }

    response = requests.post('https://api.keywordsai.co/api/chat/completions', headers=headers, json=data)
    return response

messages = "Say 'Hello World'"
print(demo_call(messages).json())

messages

array

required

List of messages to send to the endpoint in the OpenAI style, each of them following this format:

messages=[
  {"role": "system", // Available choices are user, system or assistant
   "content": "You are a helpful assistant."
  },
  {"role": "user", "content": "Hello!"}
]

Properties

Image processing: If you want to use the image processing feature, you need to use the following format to upload the image.

Example

{
  "role": "user",
  "content": [
    {
        "type": "text",
        "text": "What's in this image?"
    },
    {
        "type": "image_url",
        "image_url": {
        "url": "https://as1.ftcdn.net/v2/jpg/01/34/53/74/1000_F_134537443_VendrqyXIWyHrZgxdIsfyKUost734JDP.jpg"
        }
    }
  ]
}

model

string

required

Specify which model to use. See the list of model here

This parameter will be overridden by the loadbalance_models parameter.

stream

boolean

default:false

Whether to stream back partial progress token by token

tools

array[dict]

A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide an array of functions the model may generate JSON inputs for.

Example

{
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA"
          },
          "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"]
          }
        },
        "required": ["location"]
      }
    }
  }

tool_choice

dict

Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools.

none is the default when no tools are present. auto is the default if tools are present.

Specifying a particular tool via the code below forces the model to call that tool.

{
  "type": "function",
  "function": {"name": "name_of_the_function"},
}

frequency_penalty

number

Specify how much to penalize new tokens based on their existing frequency in the text so far. Decreases the model’s likelihood of repeating the same line verbatim

max_tokens

number

Maximum number of tokens to generate in the response

temperature

number

default:1

Controls randomness in the output in the range of 0-2, higher temperature will a more random response.

number

default:1

How many chat completion choices are generated for each input message.

Caveat! While this can help improve generation quality by picking the optimal choice, this could also lead to more token usage.

logprobs

boolean

default:false

Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.

echo

boolean

Echo back the prompt in addition to the completion

stop

array[string]

Stop sequence

presence_penalty

number

Specify how much to penalize new tokens based on whether they appear in the text so far. Increases the model’s likelihood of talking about new topics

logit_bias

dict

Used to modify the probability of tokens appearing in the response

response_format

object

An object specifying the format that the model must output. Compatible with GPT-4 Turbo and all GPT-3.5 Turbo models newer than gpt-3.5-turbo-1106.

Setting to { "type": "json_object" } enables JSON mode, which guarantees the message the model generates is valid JSON.

You must have a “json” as a keyword in the prompt to use this feature.

Properties

Vertex AI example

If you are using Vertex AI and want to use JSON mode, you should specify a response_schema in the response_format parameter. Check the details of response schema here.

Example.py

response_schema = {
    "type": "array", # or "string", "number", "object", "boolean"....
    "items": { # items only for array type
        "type": "object", 
        "properties": {  # properties only for object type
          "number": { "type": "number" },
          "street_name": { "type": "string" },
          "street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
        } 
    },
}

response_format={
        "type": "json_object",
        "response_schema": response_schema,
    },

parallel_tool_calls

boolean

Whether to enable parallel function calling during tool use.

Keywords AI parameters

See how to make a standard Keywords AI API call in the Quick Start guide.

Generation parameters

load_balance_group

object

Balance the load of your requests between different models. See the details of load balancing here.

The proxy will pick one model from the group and override the model parameter

Example

{
// you don't need to specify the model parameter, otherwise, the model parameter will overwrite the load balance group
    "messages": [
        {
            "role": "user",
            "content": "Hi, how are you?"
        }
    ],
    "load_balance_group": {
        "group_id":"THE_GROUP_ID" // from Load balancing page
    }
}

Example code with adding credentials

The models field will overwrite the load_balance_group you specified in the UI.

{
  "load_balance_group": {
      "group_id":"THE_GROUP_ID", // from Load balancing page
      "models": [
        {
          "model": "azure/gpt-35-turbo",
          "weight": 1
        },
        {
          "model": "azure/gpt-4",
          "credentials": { // add your own credentials if you want to use your own Azure credentials or custom model name
              "api_base": "Your own Azure api_base",
              "api_version": "Your own Azure api_version",
              "api_key": "Your own Azure api_key"
          },
          "weight": 1
        } 
      ]
  }
}

fallback_models

array

Specify the list of backup models (ranked by priority) to respond in case of a failure in the primary model. See the details of fallback models here.

Example

customer_credentials

object

You can pass in your customer’s credentials for supported providers and use their credits when our proxy is calling models from those providers.
See details here

Example

credential_override

object

One-off credential overrides. Instead of using what is uploaded for each provider, this targets credentials for individual models.

Go to provider page to see how to add your own credentials and override them for a specific model.

Example

"credential_override": {
    "azure/gpt-4o":{ // override for a specific model.
      "api_key": "your-api-key",
      "api_base": "your-api-base-url",
      "api_version": "your-api-version",
    }
  }

cache_enabled

boolean

Enable or disable caches. Check the details of caches here.

Example

cache_ttl

number

This parameter specifies the time-to-live (TTL) for the cache in seconds.

It’s optional and the default value is 30 days now.

Example

cache_options

boolean

This parameter specifies the cache options. Currently we support cache_by_customer option, you can set it to true or false. If cache_by_customer is set to true, the cache will be stored by the customer identifier.

It’s an optional parameter

{
    "cache_options": { // optional
        "cache_by_customer": true // or false
    }
}

prompt

object

The prompt template to use for the completion. You can build and deploy prompts in the Prompt.

Properties

prompt_id

string

required

The ID of the prompt to use. You can find this on the Prompts page.

variables

object

The variables to replace in the prompt template.

echo

boolean

default:true

With echo on, the response body will have an extra field. This is an optional parameter.

  "prompt_message" [an array of messages]

override

boolean

default:true

Turn on override to use params in override_params instead of the params in the prompt.

{
  "override": true,
}

override_params

boolean

default:true

You can put any OpenAI chat/completions parameters here to override the prompt’s parameters. This will only work if override is set to true.

{
  "override_params": {
    "temperature": 0.5,
    "max_tokens": 100
  }
}

override_config

object

This parameter allows you to control how you can override the parameters in the prompt.

Properties

Example

   request_body = {
       "prompt": {
           "prompt_id": "xxxxxx",
           "override_config": {"messages_override_mode": "append"}, # append or override
           "override_params": {"messages": [{"role": "user", "content": "5"}]},
       }
   }

Example

{
"prompt": {
      "prompt_id": "prompt_id", //paste this from the prompt management page
      "variables": {
        "variable_name": "variable_value"
      },
      // "echo": true //optional parameter
    }
}

retry_params

object

Enable or disable retries and set the number of retries and the time to wait before retrying. Check the details of retries here.

Properties

disable_log

boolean

When set to true, only the request and performance metrics will be recorded, input and output messages will be omitted from the log.

model_name_map

object

This parameter is for Azure deployment only!!

We understand that you may have a custom name for your Azure deployment. Keywords AI is using the model’s origin name which may not be able to match your deployment. You can use this parameter to map the default name to your custom name.

Example

models

array

Specify the list of models for the Keywords AI LLM router to choose between. If not specified, all models will be used. See the list of models here

If only one model is specified, it will be treated as if the model parameter is used and the router will not trigger.

When the model parameter is used, the router will not trigger, and this parameter behaves as fallback_models.

exclude_providers

array

default:[]

The list of providers to exclude from the LLM router’s selection. All models under the provider will be excluded. See the list of providers here

This only excludes providers in the LLM router, if model parameter takes precedence over this parameter, andfallback_models and safety net will still use the excluded models to catch failures.

exclude_models

array

default:[]

The list of models to exclude from the LLM router’s selection. See the list of models here

This only excludes models in the LLM router, if model parameter takes precedence over this parameter, andfallback_models and safety net will still use the excluded models to catch failures.

Observability parameters

metadata

dict

You can add any key-value pair to this metadata field for your reference. Check the details of metadata here.

Contact team@keywordsai.co if you need extra parameter support for your use case.

Example

custom_identifier

string

You can use this parameter to send an extra custom tag with your request. This will help you to identify LLM logs faster than metadata parameter, because it’s indexed. You can see it in Logs with name Custom ID field.

Example

customer_identifier

string

Use this as a tag to identify the user associated with the API call. See the details of customer identifier here.

Example

customer_params

object

Pass the customer’s parameters in the API call to monitor the user’s data in the Keywords AI platform. See how to get insights into your users’ data here

Properties

request_breakdown

boolean

default:false

Adding this returns the summarization of the response in the response body. If streaming is on, the metrics will be streamed as the last chunk.

Properties

{
"id": "chatcmpl-7476cf3f-fcc9-4902-a548-a12489856d8a",
//... main part of the response body ...
"request_breakdown": {
"prompt_tokens": 6,
"completion_tokens": 9,
"cost": 4.8e-5,
"prompt_messages": [
  {
    "role": "user",
    "content": "How are you doing today?"
  }
],
"completion_message": {
  "content": " I'm doing well, thanks for asking!",
  "role": "assistant"
},
"model": "claude-2",
"cached": false,
"timestamp": "2024-02-20T01:23:39.329729Z",
"status_code": 200,
"stream": false,
"latency": 1.8415491580963135,
"scores": {},
"category": "Questions",
"metadata": {},
"routing_time": 0.18612787732854486,
"full_request": {
  "messages": [
    {
      "role": "user",
      "content": "How are you doing today?"
    }
  ],
  "model": "claude-2",
  "logprobs": true
},
"sentiment_score": 0
}
}

Evals parameters

positive_feedback

boolean

Whether the user liked the output. True means the user liked the output.

Deprecated parameters

customer_api_keys

object

You can pass in a dictionary of your customer’s API keys for specific models. If the router selects a model that is in the dictionary, it will attempt to use the customer’s API key for calling the model before using your integration API key or Keywords AI’s default API key.

{
  "gpt-3.5-turbo": "your_customer_api_key",
  "gpt-4": "your_customer_api_key"
}

loadbalance_models

array

Balance the load of your requests between different models. See the details of load balancing here.

This parameter will override the model parameter.

Example

{
  // ...other parameters...
  "loadbalance_models": [
      {
          "model": "claude-3-5-sonnet-20240620",
          "weight": 34,
          "credentials": { // Your own Anthropic API key, optional for team plan and above
              "api_key": "Your own Anthropic API key"
          }
      },
      {
          "model": "azure/gpt-35-turbo",
          "weight": 34,
          "credentials": { // Your own Azure credentials, optional for team plan and above
              "api_base": "Your own Azure api_base",
              "api_version": "Your own Azure api_version",
              "api_key": "Your own Azure api_key"
          }
      }
  ]
}

Batch logging

import requests
def demo_call(input, 
              model="gpt-4o-mini",
              token="YOUR_KEYWORDS_AI_API_KEY"
              ):
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {token}',
    }

    data = {
        'model': model,
        'messages': [{'role': 'user', 'content': input}],
    }

    response = requests.post('https://api.keywordsai.co/api/chat/completions', headers=headers, json=data)
    return response

messages = "Say 'Hello World'"
print(demo_call(messages).json())

Integration methods

Logs

Threads

Prompts

Prompt versions

Multimodal integrations

User

Model

API keys management

OpenAI compatible parameters

Keywords AI parameters

Generation parameters

Observability parameters

Evals parameters

Deprecated parameters

Integration methods

Logs

Threads

Prompts

Prompt versions

Multimodal integrations

User

Model

API keys management

​OpenAI compatible parameters

​Keywords AI parameters

​Generation parameters

​Observability parameters

​Evals parameters

​Deprecated parameters

OpenAI compatible parameters

Keywords AI parameters

Generation parameters

Observability parameters

Evals parameters

Deprecated parameters