Real-call QA for voice agents

Regression testing for AI phone agents. On real phone calls.

Dialverdict dials your agent on the actual PSTN, holds a scripted conversation, and an LLM judge grades every reply against plain-English expectations — before your callers do it for you.

Start free See how it works

25 free test minutes · no card · works with any agent that has a phone number.

dialverdict run #4128 · brightside-dental / reschedule_flow QUEUED

Tests agents built on

Why real calls

Simulated tests miss where voice agents actually fail

API-level evals grade your prompts. Your callers grade the whole system — carrier audio, latency, interruptions, telephony. Dialverdict tests the same path a human hits, because that's the path that breaks.

FAULT / CODEC

Carrier audio

8 kHz narrowband, compression artifacts, and packet loss change what your STT hears. An agent that aces WebRTC demos can mishear half its callers on a real line.

FAULT / BARGE-IN

Interruption timing

Callers talk over agents. PSTN round-trips shift barge-in windows by hundreds of milliseconds — enough to make an agent steamroll or stall mid-sentence.

FAULT / LATENCY

Dead air

Real calls run 300–400 ms slower than browser demos. Dialverdict measures reply latency on every turn, so you see the silence your callers sit through.

FAULT / DRIFT

Silent regressions

A prompt tweak on Tuesday breaks the cancellation flow on Wednesday. Nobody notices until a customer complains. Scheduled suites catch drift the hour it ships.

How it works

Write the conversation. We make the call.

01

Script a scenario in plain English

Each turn is what the caller says and what you expect the agent to do. No grading rubrics, no regex — expectations read like a QA note.

name: reschedule_flow
turns:
  - say: "Hi, I need to reschedule my cleaning on Friday."
    expect: Acknowledges and moves the reschedule forward.
  - say: "Actually, keep the original. Goodbye!"
    expect: Confirms the original appointment stands.

02

Dialverdict places a real call

A synthetic caller with a natural TTS voice dials your agent's phone number over the PSTN, speaks each turn, and transcribes every reply — with per-turn latency captured.

03

The judge grades every turn

After hangup, an LLM judge scores each reply against your expectation and returns pass/fail, confidence, and its reasoning. Judging never injects dead air into the live call.

04

Gate your deploy

Run suites from CI on every release, or on a schedule against production. A failed conversation blocks the deploy like a failed unit test — with the transcript attached.

The judge

Verdicts you can read, from a judge you can swap

Black-box scores don't earn trust. Every Dialverdict verdict ships with the judge's reasoning, its confidence, and its raw output — and the judge model is a parameter, so you can run Claude against Nova against your own rubric and compare verdicts across runs.

  • Transcription-tolerant. The judge grades intent, not spelling — phone-audio transcripts are messy and that's fine.
  • Greeting to goodbye. Score the opening line, every turn, and the close. Partial calls still get judged.
  • Ground-truth checks. On the roadmap: the judge verifies factual claims — dates, availability, prices — against your live systems to catch hallucinations, not just implausibility.
{
  "turn": 2,
  "passed": false,
  "confidence": 0.94,
  "reasoning": "The caller asked to keep the
    original appointment, but the agent
    confirmed a reschedule. The reply
    contradicts the caller's final intent.",
  "judge_model": "claude-sonnet-4-5",
  "latency_s": 1.9
}

Platform

Everything a voice team needs to ship with confidence

CI/CD gating

Trigger suites from GitHub Actions or any pipeline via API. Fail the build when the conversation fails.

Scheduled monitoring

Run smoke scenarios against production hourly. Know your agent broke before the first angry caller.

Latency tracking

Per-turn reply latency on every run, trended over time — catch the model swap that added a second of dead air.

Caller personas

Multiple TTS voices, accents, and speaking styles. Test the mumbler and the fast talker, not just the demo voice.

Full transcripts

Every run stores the complete conversation, per-turn verdicts, and the judge's raw output. Auditable forever.

Healthcare-grade posture

Born testing a medical scheduling agent: test-identity discipline, no PHI in QA calls, audit-ready run records.

Pricing

Priced by the test minute. Not by the seat.

Real calls cost real money to place — so you only pay for the minutes you test. Every tier includes the full judge, transcripts, and API.

Free

$0/mo

Prove it on your own agent.

  • 25 test minutes / month
  • 2 scenarios, 1 project
  • Discord community
Start free

Most teams

Pro

$249/mo

Ship on green, every release.

  • 1,000 test minutes / month
  • Unlimited scenarios · 5 seats
  • CI/CD API + scheduled runs
  • Dedicated caller-ID number
  • Judge model selection
Start free

Growth

$499/mo

Broader coverage, more voices at the table.

  • 2,500 test minutes / month
  • 10 seats · 2 dedicated numbers
  • Concurrent test calls
  • Priority support
Start free

Scale

$999/mo

For regulated, high-volume teams.

  • 5,000 test minutes / month
  • SSO · unlimited seats
  • HIPAA-ready · BAA available
  • 5 dedicated numbers
  • Priority judges & support
Talk to us

Overage is optional and off by default — runs pause at your cap. When enabled: $0.35/min on Pro, $0.30 on Growth, $0.25 on Scale. Annual plans save 20%.

FAQ

Questions, answered

How is this different from simulated testing?

Most tools connect to your agent over SIP or WebRTC, skipping the network your callers actually use. Dialverdict dials your agent's real phone number over the PSTN, so every test hears what a caller hears: carrier codecs, real latency, barge-in timing, and IVR quirks.

Which platforms can Dialverdict test?

Any agent with a phone number — Vapi, Retell, Bland, LiveKit, Pipecat, Amazon Connect, or a fully custom stack. If a human can call it, Dialverdict can test it. There's nothing to install inside your agent.

How does the LLM judge decide pass or fail?

You describe the expected behavior for each turn in plain English. After hangup the judge grades every reply against that expectation and returns pass/fail with confidence, reasoning, and raw output. Swap judge models per run to compare.

Can it run in CI?

Yes — trigger suites from GitHub Actions or any CI via the API and gate the deploy on the result. Scheduled runs cover production between releases.

Is it safe for healthcare agents?

Dialverdict's engine was born testing a medical scheduling agent. Test calls only ever use dedicated test identities and numbers — no real patient data enters a QA call — and every run is stored as an auditable record.

Get started

Put your agent on the line

Sign up, point Dialverdict at your agent's number, and run your first real-call test in minutes. 25 free test minutes, no card required.

Start free Talk to sales