In this article
A polished demo may show that an n8n workflow can classify a few support tickets. For a small business deciding whether to approve or commission that workflow, the practical question is whether its behavior can be tested repeatedly across documented cases.
That requires distinguishing two approaches:
- Light evaluation: Run a small set of examples and inspect the outputs.
- Metric-based evaluation: Run a maintained dataset, score each case using declared rules, and compare results across runs.
Light evaluation is useful during exploration. Metric-based evaluation provides a repeatable basis for assessing changes. Neither establishes performance beyond the cases tested.
This explanation is based only on n8n documentation. No workflow was implemented, and no accuracy, latency, cost, or customer outcome was measured. All examples and scores below are hypothetical and untested.
Separate the reference, result, and score
An evaluation dataset keeps three concepts distinct:
- Input: The ticket submitted to the workflow.
- Expected output: The category, priority, or response the business considers correct.
- Actual output: What the workflow produces.
The expected output may be called a reference answer or ground truth. That label does not independently prove it is correct. Someone still needs to define the business rule, resolve ambiguous tickets, and revise references when those rules change.
The actual output is not a score. It is simply the recorded result.
A metric converts a defined comparison into a number. For example, an exact-match metric can return 1 when the actual category matches the expected category and 0 otherwise.
n8n describes an evaluation dataset as multiple test cases, each containing an input and often an expected output (n8n evaluation overview).
A practical ticket-evaluation worksheet
n8n’s documented light-evaluation dataset includes an optional workflow input, an expected or correct output, and an actual output. Actual-output columns begin blank and are populated during evaluation (n8n light-evaluation guide).
For a support-ticket classifier, a small business could use these columns:
- Ticket text
- Expected category and priority
- Actual category and priority
- Exact-match result
- Optional AI-based score
- Reviewer note
The following rows are hypothetical and untested:
| Ticket input | Expected category | Actual category | Exact match | AI correctness | Reviewer note |
|---|---|---|---|---|---|
| “I was charged twice for order 1842.” | Billing | Billing | 1 | 5 | Straightforward duplicate charge |
| “My card failed, but the bank shows a pending charge.” | Billing | Technical | 0 | 2 | Mixed payment and technical language |
| “Please cancel tomorrow’s delivery.” | Cancellation | Cancellation | 1 | 5 | Time-sensitive request |
| “I can’t log in after changing my account email.” | Account access | Technical | 0 | 3 | The business must decide which label takes precedence |
This structure keeps mismatches visible beside the expected labels and the reasoning behind them. It also makes clear that a reference answer, an actual answer, and a score are different artifacts.
For broader workflow context, see RodyTech’s AI services overview. RodyTech’s consulting information offers another place to consider how evaluation requirements fit the business decision.
What light evaluation establishes
In n8n’s light-evaluation process, each evaluation-trigger emission represents one dataset row. “Evaluate all” runs the workflow sequentially, once per row, and configured outputs can be written back to the dataset (n8n light-evaluation guide).
Expected and actual outputs can then be compared side by side. This is more structured than manually entering a few tickets because the test cases and results remain available for review.
It is still visual inspection rather than aggregate measurement. Without metrics, it does not directly summarize match counts, compare runs, or show whether a workflow change helped some cases while harming others. Those analyses could be performed separately, but they are not supplied by visual review alone.
That does not make light evaluation inadequate. n8n presents it as useful for early development and proof-of-concept work. Its evidence is limited to the examples reviewed.
What metric-based evaluation adds
Metric-based evaluation can assign one or more numeric scores to each test run and roll individual scores up across the dataset. It also supports comparing metrics between runs (n8n metric-based evaluation guide).
Two documented metrics illustrate why the scoring method matters.
Categorical exact match
The Categorization metric returns 1 when the answer exactly matches the reference and 0 otherwise (n8n metric definitions).
For controlled ticket labels:
- Expected
Billing, actualBilling:1 - Expected
Billing, actualTechnical:0
This rule is clear but unforgiving. The documentation does not specify partial credit or semantic equivalence, so formatting and label variations may need separate handling.
AI-based Correctness
The AI-based Correctness metric assesses whether an answer’s meaning is consistent with a supplied reference. It returns a score from 1 to 5, with 5 being best (n8n metric definitions).
This may suit generated summaries or replies better than fixed category labels. However, it remains a model-assigned judgment. The documentation provides no validated acceptance threshold or repeatability measurement. A numeric result is therefore not automatically objective.
Turn failures into regression rows
Suppose a ticket combines a failed login with a disputed payment and reaches the wrong queue. The relevant scenario can become another test row with an expected result and documented reasoning.
After changing the prompt or workflow, rerun the entire dataset. n8n recommends this regression-testing approach to check whether fixing one case made another worse (n8n evaluation overview).
Adding one edge case improves coverage but does not make the dataset representative. That depends on whether it includes the ticket categories, writing styles, languages, and ambiguous situations material to the business.
Questions to settle before approval
Document the answers to these questions:
- Who creates and approves reference answers?
- How are ambiguous or mixed-intent tickets resolved?
- Which outputs require exact matching?
- Which outputs, if any, justify AI-based scoring?
- Which important ticket types are absent?
- Will changes be tested against the same maintained dataset?
- What evidence will trigger approval, revision, or further review?
The documentation does not prescribe an acceptance threshold, so a displayed score should not become one by default.
Feature availability is configuration-sensitive. At retrieval, n8n documented light evaluations for all Cloud plans and registered self-hosted Community, Business, and Enterprise editions (light-evaluation availability). Metric-based evaluation was documented for Cloud Pro and Enterprise and self-hosted Enterprise, with single-workflow access for Registered Community and Starter users (metric-based availability). Verify current plan details before making a decision.
The useful shift is from “the demo looked convincing” to “these inputs produced these outputs under these scoring rules.” That evidence can be inspected and rerun, while remaining limited by the dataset, references, and metrics.
If useful, talk through your workflow.
No comments yet