Async logging (recommended)

Model used for the LLM inference. Default is an empty string. See the list of model here

An Array of prompt messages. Default is an empty list.

"prompt_messages": [
  {
    "role": "user",
    "content": "Hi"
  },
  # optional function call
  {
    "role": "tool",
    "tool_call_id": "your tool call id",
    "content": "...." # tool call content
  }
],

Completion message in JSON format. Default is an empty dictionary.

"completion_message": {
    "role": "assistant",
    "content": "Hi, how can I assist you today?"
},

Cost of the inference in US dollars.

Number of tokens in the completion.

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"completion_unit_price": 0.0042 // $0.0042 per 1M tokens
}

An identifier for the customer that invoked this LLM inference, helps with visualizing user activities. Default is an empty string. See the details of customer identifier here.

Error message if the LLM inference failed. Default is an empty string.

The full request object. Default is an empty dictionary. This is optional and it is helpful for logging configurations such as temperature, precence_penalty etc.

"full_request": {
"temperature": 0.5,
"top_p": 0.5,
#... other parameters
},

Total generation time. Generation time = TTFT (Time To First Token) + TPOT (Time Per Output Token) * #tokens. Do not confuse this with ttft.

You can add any key-value pair to this metadata field for your reference.

{
  "my_value_key": "my_value"
}

Number of tokens in the prompt.

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"prompt_unit_price": 0.0042 // $0.0042 per 1M tokens
}

The format of the response.

Whether the LLM inference was streamed. Default is false.

The status code of the LLM inference. Default is 200 (ok). See supported status codes here.

A list of tools the model may call. Currently, only functions are supported as a tool.

"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
    },
  }
]

Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message.

  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_weather"
      }
  },

Time to first token. The time it takes for the model to generate the first token after receiving a request.

Any warnings that occurred during the LLM inference. You could pass a warning message here. Default is an empty string.