Async logging (recommended)

Model used for the LLM inference. Default is an empty string. See the list of model here

An Array of prompt messages. Default is an empty list.

"prompt_messages": [
  {
    "role": "user",
    "content": "Hi"
  },
  # optional function call
  {
    "role": "tool",
    "tool_call_id": "your tool call id",
    "content": "...." # tool call content
  }
],

Completion message in JSON format. Default is an empty dictionary.

"completion_message": {
    "role": "assistant",
    "content": "Hi, how can I assist you today?"
},

Cost of the inference in US dollars.

Number of tokens in the completion.

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"completion_unit_price": 0.0042 // $0.0042 per 1M tokens
}

Parameters related to the customer. Default is an empty dictionary.

{
      "customer_params": {
        "customer_identifier": "customer_123",
        "name": "Hendrix Liu", //optional
        "email": "hendrix@keywordsai.co" //optional
    }
}

Error message if the LLM inference failed. Default is an empty string.

The full request object. Default is an empty dictionary. This is optional and it is helpful for logging configurations such as temperature, precence_penalty etc.

{
"full_request": {
"temperature": 0.5,
"top_p": 0.5,
//... other parameters
},
}

Specify how much to penalize new tokens based on their existing frequency in the text so far. Decreases the model’s likelihood of repeating the same line verbatim

Total generation time. Generation time = TTFT (Time To First Token) + TPOT (Time Per Output Token) * #tokens. Do not confuse this with ttft.

Use this parameter to control the behavior of the Keywords AI API. Default is an empty dictionary.

{
  "keywordsai_api_controls": {
    "block": true // true or false.
  }
//...other params...
}

You can add any key-value pair to this metadata field for your reference.

{
  "my_value_key": "my_value"
}

Specify how much to penalize new tokens based on whether they appear in the text so far. Increases the model’s likelihood of talking about new topics

Number of tokens in the prompt.

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"prompt_unit_price": 0.0042 // $0.0042 per 1M tokens
}

The format of the response.

Whether the LLM inference was streamed. Default is false.

The status code of the LLM inference. Default is 200 (ok). See supported status codes here.

Stop sequence

Controls randomness in the output in the range of 0-2, higher temperature will a more random response.

A list of tools the model may call. Currently, only functions are supported as a tool.

"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
    },
  }
]

Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message.

  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_weather"
      }
  },

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

We generally recommend altering this or temperature but not both.

Time to first token. The time it takes for the model to generate the first token after receiving a request.

Any warnings that occurred during the LLM inference. You could pass a warning message here. Default is an empty string.