Skip to main content
A test is only useful if its result is about your agent. A test that gives a tool no answer, or a simulated caller that wanders off its script, can fail a good agent or pass a broken one. Auto-improve catches those runs and fixes the test, so a failure you read is a failure of the agent. It is on for every agent unless an admin turns it off, and the Tests page says Auto-improve is on. It works from the runs you already make: your own runs and scheduled ones. Accepting a test starts nothing. A run that needs no fix costs nothing extra.

​
The two questions

When a run of a test finishes, SignalMD asks one question about that run: The answer is Yes, No or Unclear, with the reason. SignalMD answers Yes only when it can point at something in the run that the test decided rather than your agent. For example:
  • The agent called a tool the test gave no answer for.
  • The simulated caller left its script, or did not do what the test asked of it.
  • The test did not hand in a value the agent needed when the call started.
  • The run was cut off before the part the test checks.
  • A pass never reached the step or tool a criterion is about.
A failure where every tool had an answer, the caller kept to its script and every check was decided is answered No: the agent failed, not the test.

​
What it changes, and what it never does

When the answer is Yes, SignalMD fixes only the part of the test the reason points at: It never changes what the test checks: the criteria and their wording, whether the test should pass or fail, its persona, or its source call. It never removes a line or behaviour from the original caller. If the criteria look wrong, SignalMD suggests a change and waits for you to approve or reject it. Each fix is saved as a new version of the test. SignalMD then checks the fixed test in text conversations until it settles. If it cannot settle the test, it stops and the test reads Needs you, with the reason. Nothing more is changed until you edit the test.

​
Credits

A run that needs no fix spends nothing on auto-improve. Each run auto-improve plays to check a fix spends credits as a test run does, and working out a fix spends credits as chat does. When your workspace has none left, nothing more is run and the test reads Paused. See Pricing.

​
Tests found at fault together improve together

When one run of several tests finds more than one of them at fault, those tests improve as one batch:
  • SignalMD checks every test of the batch that is still improving together.
  • When several tests need the same fix, such as an answer for the same tool, SignalMD works it out once and applies it to each of them. Each test still gets its own new version, and the version says it came from the batch fix.
  • Each test keeps its own rules: its own question, its own fixes and its own reason when it needs you.
  • Nothing a batch plays is charged to your Retell account or your test budget. Each run spends credits like a test run. If a batch is ever paused, select Resume on it.
The Tests page links to the newest batch above the list. A batch shows, at the top, how many tests are Ready, Improving, Possible bot problem and Needs you, and what it cost. Below, each test is one line with its state and its version. Select a test to open its versions.

​
Where a test stands

The Tests list shows each test’s state in the Auto-improve column, and the test’s own page shows it beside the title:

​
Versions

Open a test and select Versions. At the top, two versions are compared, the newest against the one before it. Pick any two in Compare. The comparison covers the whole test: the caller’s goal, lines and behaviours, the persona, how the call starts and its values, each tool’s answer, the criteria and the expected result. Removed lines are marked red and added lines green. The parts that did not change are folded into one line that names them; select it to read them. When the newer version was made by Auto-improve, its note sits above the comparison:
  • the run and the question, such as “Run failed. Should it have passed? Yes”;
  • Why: what SignalMD found in the run, such as “The tool lookup_patient was called and had no mock”;
  • Fix: what it did about it;
  • Batch fix, when the same fix was made for several tests of a batch.
A version you or a teammate made shows A person, and is compared the same way. Below the comparison, every version is one line, newest first, with who made it and the parts it changed. Select one to compare it with the version before it. Below the versions, Runs answered lists every run with its question, answer and reason. The Runs tab shows the same answer on each run’s row. The Auto-improve section at the top shows that auto-improve is on, where the test stands and what it cost.

​
Turn it off

An admin changes it in Runs under Test, then Settings, in the Testing panel: Auto-improve tests is on by default. Off, every run is still answered so you can read it, and no test is changed. Select Save after changing it. The runs auto-improve plays to check a fix are text conversations. They cost nothing on your Retell account and do not count against your monthly test budget. A run you start yourself with Run, on Retell or as a phone call, works as before and is answered the same way.
Runs that auto-improve plays to check a fix are kept apart from your other runs. They never count toward a test’s pass rate or its calibration.