Async logging (recommended)

Model used for the LLM inference. Default is an empty string. See the list of model here

An Array of prompt messages. Default is an empty list.

"prompt_messages": [
  {
    "role": "user",
    "content": "Hi"
  },
  # optional function call
  {
    "role": "tool",
    "tool_call_id": "your tool call id",
    "content": "...." # tool call content
  }
],

Completion message in JSON format. Default is an empty dictionary.

"completion_message": {
    "role": "assistant",
    "content": "Hi, how can I assist you today?"
},

Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message.

  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_weather"
      }
  },

A list of tools the model may call. Currently, only functions are supported as a tool.

"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
    },
  }
]

Whether the LLM inference was streamed. Default is false.

An identifier for the customer that invoked this LLM inference, helps with visualizing user activities. Default is an empty string. See the details of customer identifier here.

You can add any key-value pair to this metadata field for your reference.

{
  "my_value_key": "my_value"
}

Number of tokens in the prompt.

Number of tokens in the completion.

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"prompt_unit_price": 0.0042 // $0.0042 per 1M tokens
}

Pass this parameter in if you want to log your self-host / fine-tuned model.

{
"completion_unit_price": 0.0042 // $0.0042 per 1M tokens
}

Cost of the inference in US dollars.

Total generation time. Generation time = TTFT (Time To First Token) + TPOT (Time Per Output Token) * #tokens. Do not confuse this with ttft.

Time to first token. The time it takes for the model to generate the first token after receiving a request.

The number of tokens generated per second.

Error message if the LLM inference failed. Default is an empty string.

The status code of the LLM inference. Default is 200 (ok). See supported status codes here.

Whether the LLM inference has any warnings. Default is false.

Any warnings that occurred during the LLM inference. You could pass a warning message here. Default is an empty string.

Whether the LLM inference is a test call. If set to true, you will only be able to see this log in Test Mode in the platform. Default is false.