Skip to main content
GET
https://api.keywordsai.co
/
api
/
evaluators
/
{evaluator_id}
Retrieve evaluator
curl --request GET \
  --url https://api.keywordsai.co/api/evaluators/{evaluator_id}/ \
  --header 'Authorization: Bearer <token>'
Retrieves detailed information about a specific evaluator, including full configuration.

​
Authentication

All endpoints require API key authentication: 
Authorization: Bearer YOUR_API_KEY

​
Path Parameters

ParameterTypeDescription
evaluator_idstringThe unique ID of the evaluator

​
Examples

import requests

evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
url = f"https://api.keywordsai.co/api/evaluators/{evaluator_id}/"
headers = {
    "Authorization": "Bearer YOUR_API_KEY"
}

response = requests.get(url, headers=headers)
print(response.json())

​
Response

Status: 200 OK

​
LLM Evaluator Response (New Format)

{
  "id": "0f4325f9-55ef-4c20-8abe-376694419947",
  "name": "Response Quality Evaluator",
  "evaluator_slug": "response_quality_v2",
  "type": "llm",
  "score_value_type": "numerical",
  "eval_class": "",
  "description": "Evaluates response quality on a 1-5 scale",
  "score_config": {
    "min_score": 1,
    "max_score": 5,
    "choices": [
      {"name": "Poor", "value": 1},
      {"name": "Fair", "value": 2},
      {"name": "Good", "value": 3},
      {"name": "Great", "value": 4},
      {"name": "Excellent", "value": 5}
    ]
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>",
    "scoring_rubric": "1=Poor, 5=Excellent",
    "temperature": 0.1,
    "max_tokens": 200
  },
  "code_config": null,
  "configurations": {},
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "created_at": "2025-09-11T09:43:55.858321Z",
  "updated_at": "2025-09-11T09:43:55.858331Z",
  "custom_required_fields": [],
  "categorical_choices": null,
  "starred": false,
  "organization": 2,
  "tags": []
}

​
LLM Evaluator Response (Legacy Format)

{
  "id": "0f4325f9-legacy",
  "name": "Response Quality Evaluator (Legacy)",
  "evaluator_slug": "response_quality_v1",
  "type": "llm",
  "score_value_type": "numerical",
  "eval_class": "",
  "description": "Evaluates response quality on a 1-5 scale",
  "configurations": {
    "evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>",
    "scoring_rubric": "1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent",
    "llm_engine": "gpt-4o-mini",
    "model_options": {
      "temperature": 0.1,
      "max_tokens": 200
    },
    "min_score": 1.0,
    "max_score": 5.0,
    "passing_score": 3.0
  },
  "score_config": null,
  "passing_conditions": null,
  "llm_config": null,
  "code_config": null,
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "created_at": "2025-09-11T09:43:55.858321Z",
  "updated_at": "2025-09-11T09:43:55.858331Z",
  "custom_required_fields": [],
  "categorical_choices": null,
  "starred": false,
  "organization": 2,
  "tags": []
}

​
Human Evaluator with LLM Assistance (New Format)

This example shows how a human evaluator can have LLM automation configured, demonstrating the decoupling of annotation method from evaluator type.
{
  "id": "human-llm-123",
  "name": "Human Review with AI Assistance",
  "evaluator_slug": "human_ai_assist_v1",
  "type": "human",
  "score_value_type": "numerical",
  "eval_class": "",
  "description": "Human review with LLM-suggested scores",
  "score_config": {
    "min_score": 1,
    "max_score": 5
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Suggest a quality score for this response",
    "temperature": 0.1
  },
  "code_config": null,
  "configurations": {},
  "categorical_choices": null,
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "created_at": "2025-09-11T09:44:00.000000Z",
  "updated_at": "2025-09-11T09:44:00.000000Z",
  "custom_required_fields": [],
  "starred": false,
  "organization": 2,
  "tags": []
}

​
Human Single-Select Evaluator Response (Legacy Format)

{
  "id": "cat-eval-123",
  "name": "Content Quality Assessment",
  "evaluator_slug": "content_quality_categorical",
  "type": "human",
  "score_value_type": "single_select",
  "eval_class": "",
  "description": "Human assessment of content quality with predefined categories",
  "configurations": {},
  "score_config": null,
  "passing_conditions": null,
  "llm_config": null,
  "code_config": null,
  "categorical_choices": [
    { "name": "Excellent", "value": 5 },
    { "name": "Good", "value": 4 },
    { "name": "Average", "value": 3 },
    { "name": "Poor", "value": 2 },
    { "name": "Very Poor", "value": 1 }
  ],
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "created_at": "2025-09-11T09:44:00.000000Z",
  "updated_at": "2025-09-11T09:44:00.000000Z",
  "custom_required_fields": [],
  "starred": false,
  "organization": 2,
  "tags": []
}

​
Code Evaluator Response (New Format)

{
  "id": "bool-eval-456",
  "name": "Length Check",
  "evaluator_slug": "length_check_v1",
  "type": "code",
  "score_value_type": "boolean",
  "eval_class": "",
  "description": "Checks if response is longer than 10 characters",
  "score_config": {},
  "passing_conditions": null,
  "llm_config": null,
  "code_config": {
    "eval_code_snippet": "def main(eval_inputs):\n    output = eval_inputs.get('output', '')\n    return len(str(output)) > 10"
  },
  "configurations": {},
  "categorical_choices": [],
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[email protected]"
  },
  "created_at": "2025-09-11T09:45:00.000000Z",
  "updated_at": "2025-09-11T09:45:00.000000Z",
  "custom_required_fields": [],
  "starred": false,
  "organization": 2,
  "tags": ["automation", "validation"]
}

​
Response Fields

New Format: Evaluators now include score_config, passing_conditions, llm_config, and code_config fields. These allow any evaluator type to have both LLM and code automation configured, decoupling annotation method from evaluator type.
FieldTypeDescription
idstringUnique evaluator identifier
namestringDisplay name of the evaluator
evaluator_slugstringURL-friendly identifier
typestringEvaluator type: llm, human, or code
score_value_typestringScore format: numerical, boolean, percentage, single_select, multi_select, json, text
eval_classstringPre-built template class (if used)
descriptionstringDescription of the evaluator
score_configobjectNew: Score type configuration (min/max, choices, etc.)
passing_conditionsobjectNew: Passing conditions using universal filter format
llm_configobjectNew: LLM automation config (if configured)
code_configobjectNew: Code automation config (if configured)
configurationsobjectLegacy type-specific configuration settings
categorical_choicesarrayLegacy choices (use score_config.choices in new format)
created_byobjectUser who created the evaluator
updated_byobjectUser who last updated the evaluator
created_atstringISO timestamp of creation
updated_atstringISO timestamp of last update
custom_required_fieldsarrayAdditional required fields
starredbooleanWhether the evaluator is starred
organizationintegerOrganization ID
tagsarrayTags associated with the evaluator

​
Configuration Fields by Type (Legacy)

​
LLM Evaluators (type: "llm")

FieldTypeDescription
evaluator_definitionstringThe evaluation prompt/instruction with template variables
scoring_rubricstringDescription of the scoring criteria
llm_enginestringLLM model to use (e.g., “gpt-4o-mini”, “gpt-4o”)
model_optionsobjectLLM parameters like temperature, max_tokens
min_scorenumberMinimum possible score
max_scorenumberMaximum possible score
passing_scorenumberScore threshold for passing

​
Code Evaluators (type: "code")

FieldTypeDescription
eval_code_snippetstringPython code with evaluate() function

​
Human Evaluators (type: "human")

  • No specific configuration fields
  • Use categorical_choices field when score_value_type is "categorical"

​
Error Responses

​
404 Not Found

{
  "detail": "Not found."
}

​
401 Unauthorized

{
  "detail": "Your API key is invalid or expired, please check your API key at https://platform.keywordsai.co/platform/api/api-keys"
}

​
403 Forbidden

{
  "detail": "You do not have permission to access this evaluator."
}