Analyst guide
The factory builds small models that make a closed decision for you: sorting a message, choosing a team, qualifying a request. Whether they can be trusted is up to you.
Your role
You are the basis of the guarantee
A model is guaranteed only on cases reviewed by a person. Without your review, its figures are only landmarks, not guarantees.
You bring reality
Your already settled cases (tickets, requests, decisions) are the best examples there are. A model trained on invented texts gets real ones wrong.
You do not build the models
Training, promoting, rolling back: that is the space administrator's job. Your key does not allow it, on purpose: the one who judges is not the one who builds.
Before you start
- Your invitation
Your client's administrator gives you a link, which works once and expires. Open it, enter your name: you receive your personal key, shown only once. Keep it in a password manager. - One key, in your name, for all your spaces
It signs your reviews: do not share it, and do not use a colleague's. If you lose it, the administrator regenerates it (the old one stops at once). - Sign in
Console, with your key. If you work in several spaces, choose one at the top of the page. You land on “My tasks”: the work waiting for you, counted for you. - Practise first
On a demo space (sorting support messages), before touching real data: the path is the same.
The rules
Review blind
Do not look at the labels before reviewing the test: the console hides them. A label seen in advance steers the answer, and the review is then worthless.
Never a model-written text in the test
The test contains only real cases. A generated text, even a good one, grades itself.
The test is not the training
A text cannot be in both: the tool refuses copies, and even near copies. Do not try to get around this refusal.
Do not correct to raise a score
Your answer is the one you would give without knowing what the model says. If the model is right, good; if it is wrong, that is valuable.
Report an ambiguous definition
If two options overlap for a case, do not settle it alone: tell the administrator. A vague definition does more harm than an error.
“I don't know” beats a guess
In the random check, a call you cannot settle is marked “I don't know”. In the test review, skip it.
The path, step by step
Each question has a “Path” tab that repeats these steps, with what is left to do.
- Understand the question
Its options and their definitions, the safe option (returned when the model doubts), the dangerous errors (the costly ones). Everything else follows from them. - Review the test blind
Frozen test tab, “Start the review”: one case at a time, without its label. A few seconds per case. Cases where other reviewers hesitated come first. - Contribute examples
Import tab: a CSV or JSONL file of already settled cases. Check first (nothing is written), map your labels to the options, then import. - Fill the guarantee part
Frozen test tab, “Import test cases”: real cases, labelled by you. Each one goes at random into the design part (visible) or the guarantee part (never shown case by case). Only the reviewed guarantee allows promising an error rate. - Validate the generated batches
Build tab, “Generated batches”: a random sample, reviewed blind. If you find too many errors, the batch stays to be reviewed in full. - Check usage at random
Feedback tab: real calls drawn at random, labelled without seeing the answer given. It is the only honest measure of the error in production. - Review the feedback
Examples tab, “to review” filter: what usage brought back. Validate, correct or reject, one by one or in batches.
A good test case
Real
As it arrived, with its mistakes and its length. Do not rephrase it, do not summarise it.
Representative
In the proportions of usage: if 60% of messages are about billing, so is the test. A test made only of hard cases gives a misleading error rate.
Varied
Several authors, several periods, several sources. A single writer flatters any model.
Numerous enough
With zero errors on n cases exposed to a dangerous error, you can only guarantee a rate below 3/n: 60 cases for less than 5%, 300 for less than 1%.
File format
CSV (separator guessed: semicolon, comma, tab) with one column for the text and one for the answer, or JSONL. Common column names are recognised (message, text, category, answer...). Each import screen offers a template to download. Case and accents in answers do not matter; your own labels can be mapped to the options.
Reading the figures
The interval
“83% (between 70% and 91%)”: on 48 cases, the true value lies somewhere in this interval, at 95%. Two versions whose intervals overlap cannot be told apart.
“Not significant”
The gap between two versions may come from the luck of the test. It does not mean “equal”: it means “we do not know”.
The guaranteed margin
The only figure that can be promised: it comes from the guarantee part, reviewed by humans, never looked at case by case.
Questions
A colleague disagrees with me on a case.
That is normal, and it is measured. The case goes “to arbitrate” and leaves the measures until the administrator settles it. If you think you were wrong on a case you already reviewed, tell the administrator: they arbitrate.
The import refuses a row “too close to the test”.
This text is almost word for word a test case: learning it would mean knowing the test answer in advance. That is on purpose.
I cannot see the text of some test cases.
They are the guarantee part cases: they are never shown case by case, so nobody tunes the model on them. You still review them, blind, in the review.
The model looks wrong, what should I do?
Record it: feedback in the Try tab, or a review. You never change the model directly: your decisions become examples, and the administrator decides to train and promote a new version, which only passes if it does no worse.
Does my data leave?
No. Each space is isolated, and nothing is sent anywhere without the administrator's explicit consent.