Overview
Create a new evaluator to assess and score experiment results based on specific criteria.
Method Signature
# Synchronous
client.evaluators.create(
name: str,
description: Optional[str] = None,
evaluator_type: str = "custom",
config: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]
# Asynchronous
await client.evaluators.create(
name: str,
description: Optional[str] = None,
evaluator_type: str = "custom",
config: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]
Parameters
The name of the evaluator
Description of what the evaluator measures
Type of evaluator (e.g., “accuracy”, “relevance”, “custom”)
Configuration parameters for the evaluator
Returns
Returns a dictionary containing the created evaluator information.
Example
from keywordsai import KeywordsAI
client = KeywordsAI(api_key="your-api-key")
# Create a custom evaluator
evaluator = client.evaluators.create(
name="Response Accuracy",
description="Measures accuracy of model responses",
evaluator_type="accuracy",
config={
"threshold": 0.8,
"metric": "exact_match"
}
)
print(f"Created evaluator: {evaluator['id']}")
Error Handling
try:
evaluator = client.evaluators.create(
name="My Evaluator",
evaluator_type="custom"
)
except Exception as e:
print(f"Error creating evaluator: {e}")