Quality Control

Quality Control is where you check how well your agent answers — both on prepared test cases and on real conversations with your customers. You score answers against your own criteria, by hand or with an LLM judge, and compare agent configurations side by side before you switch the live one.

Where to find it

1. Log in to console.flutch.ai
2. Go to Agents and open your agent
3. Open the Quality Control tab

URL: https://console.flutch.ai/agents/{agentId}/acceptance-testing

The tab has four sections. Each one has its own link, so you can bookmark or share it:

How it works

Quality Control is built from four pieces:

• Criteria — the checklist for a good answer. They belong to the agent and are shared by test cases, real dialogs, people and judges.
• What gets assessed — two different things:
  • a test run in the Test Matrix: one customer message from a test case and the agent's answer to it. This is a single turn, not a dialog.
  • a real dialog from production, taken into a sample: the whole thread, every customer message and every agent reply.
• Assessments — a filled-in checklist for one test answer or one real dialog. There are two kinds: Human (shown as H) and LLM judge (shown as LLM). Both are kept, so you can see where a judge disagrees with a person.
• Score — every assessment turns into one number from 0 to 100%: each criterion is normalized to 0–1 and averaged by weight. Text criteria are comments only and don't affect the score.

A typical loop:

1. Write criteria.
2. Add test cases for the situations you care about.
3. Run them on your configurations, assess a few answers by hand.
4. Add an LLM judge and check that its scores agree with yours.
5. Let the judge score new runs and real-dialog samples, and look at the low ones yourself.

Evaluation criteria

Open Evaluation Criteria and click Add Criteria.

Criteria are ordered — use the arrows to move them up and down. The order is the order of the assessment form.

Tips:

• Describe when a criterion applies, e.g. "If the customer asks about price, the answer names the exact price". The LLM judge can mark a criterion as not applicable for a conversation, and then it doesn't count in the score.
• Leave Required Field off for criteria that only apply sometimes, so the person assessing can skip them.
• One criterion should check one thing. "Correct and polite" is two criteria.

Test Matrix

The matrix shows test cases as rows and agent configurations as columns. The Active badge marks the configuration that answers your customers right now. A configuration is a saved version of the agent's settings (model, prompt and so on), so the matrix tells you how a new version would do before you activate it.

The matrix checks single answers: each test case is one customer message, and what you assess is how the agent answered it. To judge how the agent leads a whole conversation, use Real Dialogs.

Buckets and test cases

Test cases are grouped into buckets — e.g. "Pricing questions", "Delivery", "Edge cases".

• New Bucket — create a bucket (name and optional description).
• + in a bucket row — add a test case to it.
• Click a bucket name to expand or collapse it; click a name or a prompt to edit it in place.

A test case is the full request a customer would send:

Use the pencil icon on a test case to edit all of it at once.

Running tests

Tests run in the background and the cells update by themselves. Each run creates a separate test conversation on the web channel with the chosen configuration — nothing is sent to your Telegram, Instagram or other channels, and test conversations don't show up in your analytics or quality reports.

Reading the results

Each cell shows the latest run: its number, status (queued, running, completed, error) and time. View all (N) shows earlier runs.

For a completed run the cell shows two scores — H (your assessment) and LLM (the judge's) — and an Assess button. Click the run itself to open its full trace: every step, tool call and model request, the same view as in Debugging: Message Audit & Logs.

The bottom row, Average score (latest runs), averages the latest run of every test case for each configuration. This is the number to compare configurations by.

Assessing an answer

Assess opens a window with two halves:

• Left — the conversation (Dialog) with the request, the expected result and the agent's answer. Switch to Trace to see the steps behind the answer.
• Right — Your assessment: one field per criterion (required ones are marked with *) plus a comment, with the running score in the header. Under each criterion you also see what other assessments gave — the LLM judge's value with its reasoning, or another person's.

The Save button stays disabled until all required criteria are filled in.

Real Dialogs

Test cases show how the agent answers single messages you thought of. Real dialogs show how it handles your actual customers from start to finish: here you assess whole conversations from production, not individual answers — did the agent understand the customer, keep the thread going, get to the sale. Instead of reading every conversation, you take a sample.

Creating a sample

Click New sample and set:

Preview shows how many conversations match and how many are flagged, and what the sample would contain. Create sample saves it.

Only conversations where the agent actually replied are included.

Attention threads

A conversation is flagged for attention when:

• human_takeover — a manager replied in the conversation
• unanswered — it ends with the customer's message
• empty_reply — it ends with an agent message that has no text, so the customer got nothing
• long_no_sale — the customer wrote 6 or more messages and never bought

These are where problems usually hide, so a part of every sample comes from them. The rest is random.

Reading a sample

The sample shows each conversation with its channel, number of messages, who answered, outcome, why it was picked (random or the attention reasons), and its H and LLM scores.

Averages are shown separately for the random part and the attention part. Only the random part is a fair estimate of the agent's overall quality — the attention part is skewed toward problems on purpose.

Open a conversation to assess the whole thread with the same criteria. If you find a case worth keeping, click Make test case: the last customer message, with its attachments and metadata, becomes a test case in a bucket of your choice. Now you can check that the next configuration handles it.

LLM judges

A judge is a model that fills in the criteria for you. Open Judges and click New judge.

What the judge sees

• Test run — the test case's message and the agent's answer to it.
• Real dialog — the whole thread: every customer message and every agent reply.

In both cases the judge also sees the tools the agent called and what they returned, and the confirmation cards it showed. So it can tell "the agent didn't do it" from "the agent offered it and the customer declined".

For each criterion the judge writes its reasoning first, then decides whether the criterion applies, then gives a value. Criteria it marks as not applicable don't count in the score. Open Assess on any judged answer to read the reasoning.

Running a judge

• Judge latest answers (Test Matrix) — judges the latest completed run in every cell.
• New runs: judge by … (Test Matrix) — pick a judge, and every new run is judged as soon as it completes.
• Run LLM judge (a sample in Real Dialogs) — judges every conversation in the sample.

The control shows progress while the judge works and the time of the last run.

Deleting a judge keeps the scores it already gave.

Cost

Judge calls are billed like regular agent usage, from your balance: with Prompt here — to the agent being tested, with Another agent — to that agent. If you've connected your own provider key, the judge uses it the same way the agent does.

Can you trust the judge?

Check before you rely on it:

1. Assess 15–20 answers yourself.
2. Run the judge on the same answers.
3. Compare H and LLM answer by answer.

If they often disagree, read the judge's reasoning: usually a criterion description is vague or the judge prompt is missing context. Fix the description, rerun, compare again. Try a couple of models and keep the one that agrees with you most — not the most expensive one.

Best practices

• Start with 5–10 test cases: the most common questions, sales-critical moments, and cases that went wrong before.
• Write expected results as behavior, not wording: "names the price and offers a payment link", not a full sample reply.
• Turn real failures into test cases with Make test case, so the same mistake is caught next time.
• Run the whole matrix before activating a new configuration and compare the average score with the active one.
• Take a sample of real dialogs regularly, e.g. once a week, with Skip threads from earlier samples on.
• Look at low scores yourself. The judge tells you where to look; you decide what to fix.

Troubleshooting

A cell stays in queued or running. Runs go through a queue; wait a minute. If it doesn't move, run the case again.

A run ended with error. Click it to open the trace and see which step failed.

The LLM score is "—". The run hasn't been judged yet, or every criterion was text or not applicable.

Make test case says "Create a test bucket first". Create a bucket in the Test Matrix, then try again.

I don't see the Quality Control tab. It's hidden for system-managed agents and for users with the agent manager role.