​
Overview

Create a new evaluator to assess and score experiment results based on specific criteria.

​
Method Signature

# Synchronous
client.evaluators.create(
    name: str,
    description: Optional[str] = None,
    evaluator_type: str = "custom",
    config: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]

# Asynchronous
await client.evaluators.create(
    name: str,
    description: Optional[str] = None,
    evaluator_type: str = "custom",
    config: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]

​
Parameters

​
name
string
required
The name of the evaluator
​
description
string
Description of what the evaluator measures
​
evaluator_type
string
Type of evaluator (e.g., “accuracy”, “relevance”, “custom”)
​
config
Dict[str, Any]
Configuration parameters for the evaluator

​
Returns

Returns a dictionary containing the created evaluator information.

​
Example

from keywordsai import KeywordsAI

client = KeywordsAI(api_key="your-api-key")

# Create a custom evaluator
evaluator = client.evaluators.create(
    name="Response Accuracy",
    description="Measures accuracy of model responses",
    evaluator_type="accuracy",
    config={
        "threshold": 0.8,
        "metric": "exact_match"
    }
)

print(f"Created evaluator: {evaluator['id']}")

​
Error Handling

try:
    evaluator = client.evaluators.create(
        name="My Evaluator",
        evaluator_type="custom"
    )
except Exception as e:
    print(f"Error creating evaluator: {e}")