LLM logging

​

Why you need LLM observability?

LLM observability is the comprehensive process of monitoring, evaluating, and gaining insights into the performance and activities of Large Language Models in real-time. LLM observability is crucial for several reasons:

1. Ensuring Accuracy and Relevance: LLMs can produce hallucinations, so observability helps detect and fix these issues.
2. Maintaining Performance: Tracking response times, throughput, and error rates ensures optimal LLM performance.
3. Enhancing Reliability: Monitoring helps prevent and quickly resolve downtime from provider outages, rate limits, or delayed alerts.
4. Optimizing Costs: Monitoring identifies cost-effective models and leverages caching to reduce expenses.

Keywords AI provides an Async Logging API that allows you to log your LLM requests and responses asynchronously, which offers complete observability of your LLM applications and won’t disrupt your application’s performance.

​

Benefits of async logging:

• Monitor your LLM performance with 0 latency impact.
• Operates outside the critical path of your application, ensuring no disruptions.
• Gain comprehensive observability of your LLM applications.

​

How to use Keywords AI Async Logging

​

1. Get your Keywords AI API key

After you create an account on Keywords AI, you can get your API key from the API keys page.

​

2. Integrate Async Logging into your codebase

import requests

url = "https://api.keywordsai.co/api/request-logs/create/"
payload = {
    "model": "claude-3-5-sonnet-20240620",
    "prompt_messages": [
        {
            "role": "user",
            "content": "Hi"
        },
    ],
    "completion_message": {
        "role": "assistant",
        "content": "Hi, how can I assist you today?"
    },
    "cost": 0.00042,
    "generation_time": 5.7,
    "ttft": 3.1,
    "customer_params": {
        "customer_identifier": "customer_123",
        "name": "Hendrix Liu",
        "email": "hendrix@keywordsai.co" 
    }
}
headers = {
    "Authorization": "Bearer YOUR_KEYWORDS_AI_API_KEY",
    "Content-Type": "application/json"
}

response = requests.request("POST", url, headers=headers, json=payload)

​

3. Check your logs on the platform

After you integrate the async logging into your codebase and send the request successfully, you can check your logs on the Logs page.

​

4. Parameters

Check out the Logging endpoint page to see all supported parameters.

Parameters like: cost, completion_tokens, and prompt_tokens will be automatically calculated if your model is supported. Check out our models page to see the list of supported models.

​

Next steps