Automation

How to Evaluate AI Email Drafting: Test Retrieval and Replies Separately

blog-image-83258ffc647ba12492baa05f3395b17f637acb64851bff78d190f80f6e2c6426

If your small business is considering an AI workflow that drafts customer emails from company documents, the decision is not simply whether the writing sounds professional. You need to determine whether the workflow can:

  1. Find the correct company information.
  2. Turn that information into an acceptable reply.
  3. Route unsupported or sensitive requests to the right person.
  4. Preserve clear human authority over sending.

The first two capabilities require separate tests. A polished answer can conceal weak retrieval, while correct source material can still produce an inaccurate or incomplete draft. This framework is based on documentation; no hands-on test, local deployment, or performance measurement was conducted.

Map the workflow and its control points

A document-grounded email workflow can be divided into distinct stages:

  1. Intake and preprocessing: Receive the email and convert it into usable text.
  2. Classification: Identify the request type and select a route.
  3. Retrieval: Search approved company documents for relevant passages.
  4. Drafting: Generate a proposed response from the email and retrieved material.
  5. Review and exception handling: Check the draft and route unsupported or sensitive cases.
  6. Human approval: A designated employee edits, approves, rejects, or escalates the response.
  7. Sending: Send the approved email and retain any required record.

One documented n8n template combines preprocessing, summarization, classification, vector retrieval, drafting, AI review, formatting, and sending (n8n workflow template). It illustrates possible components, but it does not establish reliability or the suitability of unattended sending. Its AI review step is not human approval.

Anthropic’s customer-support guidance recommends dividing an interaction into separate tasks so each can be prompted, evaluated, and represented in test cases (customer-support guide). Applied to email, that approach helps locate failures instead of treating the workflow as one opaque system.

Test one: Did retrieval find the necessary evidence?

Retrieval testing stops before the reply is written. For each representative customer question, identify the passages—or chunks—from approved company documents that the system should find.

Anthropic’s RAG guide separates retrieval evaluation from evaluation of the complete retrieval-and-generation path. Its example records include a question, the chunks expected to be relevant, and a reference answer (RAG evaluation guide).

Suppose a customer asks:

Can I return a custom-printed order after it has shipped?

The expected evidence might include:

  • The return policy for custom products
  • The shipping policy’s cancellation cutoff
  • An exception for manufacturing defects

Compare those expected passages with the passages actually retrieved. Do not give retrieval credit merely because the eventual email sounds plausible.

Some questions may require several passages. Anthropic’s example dataset includes questions that require synthesis across multiple chunks, so finding one related paragraph may not be sufficient (RAG evaluation guide).

Record whether retrieval was complete, incomplete, irrelevant, outdated, or missing. This narrows the investigation to areas such as source selection, document versions, chunking, indexing, metadata, or query construction.

Test two: Did the complete path produce an acceptable reply?

The second test compares the actual draft with an approved reference answer and the business’s requirements. A reference answer is a benchmark, not the only acceptable wording.

Review whether the draft:

  • Is supported by the retrieved passages
  • Includes necessary limits and exceptions
  • Answers the customer’s question directly
  • Avoids unsupported promises or commitments
  • Uses an appropriate tone
  • Escalates or requests follow-up when required

Anthropic defines end-to-end evaluation as assessing the entire path from retrieval through answer generation (RAG evaluation guide). Its example uses an AI judge, but an automated correctness judgment is not equivalent to human approval of a customer email. Results from that guide also depend on its dataset and model configuration, so they should not be transferred to your company’s workflow.

Separating the tests exposes four useful patterns:

Retrieved evidence Draft What to investigate
Wrong or irrelevant Wrong Retrieval failed before drafting
Correct and complete Wrong Drafting instructions or generation
Partially complete Incomplete Missing evidence and its downstream effect
Weak or missing Apparently acceptable Unsupported inference or outside knowledge

The final pattern is especially important: a correct-looking answer may still come from an unreliable process. Retrieved text does not guarantee a correct reply, and plausible prose does not prove that retrieval worked.

A commissioning worksheet

Use one row per test case:

Column What to record
Case ID Stable identifier
Email category Billing, scheduling, returns, product information, or another route
Business question The customer’s underlying request
Expected relevant chunks Approved passages the search should return
Expected answer Essential facts, limits, exceptions, and next steps
Retrieved chunks Passages the workflow returned
Actual draft Complete proposed response
Retrieval finding Complete, incomplete, irrelevant, outdated, or missing
Reply finding Acceptable, needs editing, unsupported, or incorrect
Reviewer decision Approve, edit, reject, or escalate
Exception route Responsible person or team
Notes Suspected cause, document issue, or follow-up action

This worksheet applies the documented evaluation structure; it is not a deployment-tested template.

Include routine questions, ambiguous requests, multiple questions in one email, outdated terminology, policy exceptions, and cases with no approved answer. When the documents do not support a response, escalation may be the correct outcome.

Choose review and sending controls deliberately

A Farmingdale SBDC event description listed on the SBA website presents a workflow moving from customer inquiry through AI-assisted drafting, human approval, and follow-up. It also identifies review placement and exception handling as design decisions (SBA event page). This describes a workflow pattern, not measured results.

Another n8n template saves a generated response as a Gmail draft for a person to review and send (draft-review workflow). That is a possible control point, not evidence of accuracy or time savings.

Start with one narrow email category, current authoritative documents, routine and exception cases, a named reviewer, explicit escalation rules, and clear send authority. If the business cannot identify authoritative sources and accountable reviewers, retaining the manual process may be the better choice.

For broader context, see RodyTech’s AI services overview and workflow consulting information.

The two commissioning questions are simple: Did the workflow find the necessary evidence? And did the complete process turn that evidence into a reply the business is prepared to approve?

If useful, talk through your workflow

Sources and further reading

Back to top ↑

RodyTech Editorial

Founder & CEO · RodyTech LLC

Founder of RodyTech LLC in Iowa. I write practical notes on automation, infrastructure, security, and software decisions for builders and business operators.

No comments yet

Leave a comment

Your email address will not be published. Required fields are marked *