Skip to main content

​
What is an online evals automation?

Online evals automatically runs evaluators on your production logs in real-time based on conditions and sampling rules.

​
How to set up an online evals automation?

​
1. Create a condition

A condition defines the filtering rules that determine which production logs will be evaluated—only logs that match the condition proceed to the sampling and evals stages.
Go to Conditions.
You can choose one of the following condition types:
  • Single log – triggers when a specific log matches the rule.
  • Aggregation – triggers based on aggregated metrics over time.
We will choose Single log for Online evals.
Following is a simple condition that triggers everytime a log with status code 500 is logged.

​
2. Create an automation

Once your condition and evaluator are ready, navigate to Automations and create a new automation.
In the creation form:
  1. Select Online evals as the type.
  2. Select the condition you want to track.
  3. Choose the evaluator you want to run through.
  4. Set the Sampling rate (range from 0 to 1).
After saving, your online evals will appear in the Automations list.
When the defined condition is met, sampling will be done and run through evaluators.