Reference · frozen contract

Earshot documentation

Every channel in a contact center has a test suite except the one the customers actually call. This is the reference for the one that does.

The DSL is a frozen contract. Anything not documented here is a load error — in the builder and on disk alike. That is deliberate: a silently skipped test definition is worse than a loud failure.

DSL reference

One file, one scenario, in scenarios/<name>.yaml. Names are a-z 0-9 _ -. Two files declaring the same name is an error.

Top-level keys

name: billing_balance_happy
description: Auth then balance readback
dial: "15135660137"
ani: "15135550100"
max_duration_sec: 240
tags: [billing, happy]
steps:
  - wait: 6

Verb reference

Ten verbs. Every one has a short form and a long form; the long form is the same verb with its fields spelled out.

wait

Passive listen. Audio keeps recording throughout.

- wait: 6
- wait: 2.5

press

Send DTMF. Allowed characters 0-9 * # and ,, where , inserts a 500 ms gap.

- press: "1"
- press: "5135551234#"
- press: "5133982774,,,,,,,,5135652210"   # account, eight pauses, then a PIN

wait_for

A live gate. Blocks until the audio recorded so far contains one of texts, polling as the call runs. It exists because IVR prompt timing drifts with back-end latency: a press keyed to the prompt survives that, a press keyed to a clock does not.

- wait_for:
    texts: ["for technical support press 1", "order new service"]
    stop_on: ["office is closed", "no one is available"]
    timeout_sec: 130
    min_score: 0.8
    lookback_sec: 2

Short form, when the defaults are right:

- wait_for: "enter your 5-digit billing zip code"

expect_prompt

A deferred assertion against the transcript. Transcribes the audio window ending at this step's evaluation point — window_sec backwards, padded ±1.5 s — and passes if any of texts matches at or above min_score.

- expect_prompt: "for billing press 1"

- expect_prompt:
    texts:
      - "enter your 5 digit billing zip"
      - "billing zip code"
    window_sec: 10
    min_score: 0.7
    fatal: false

expect_value

A typed value assertion. The value is expanded into every plausible spoken rendering and any of them satisfies the check, so you assert the number rather than the wording.

- expect_value:
    type: currency
    value: 142.17
    context: "your total balance"

What it accepts for 142.17: "one hundred forty two dollars and seventeen cents", "a hundred forty two seventeen", "142 dollars and 17 cents", and the rest. Dates expand to "august eleventh", "august eleven", "8 11". Digit strings expand digit by digit with "oh" for zero. The builder previews the full candidate list before you place the call, so you can see whether the assertion is going to match how your IVR actually says it.

expect_silence

Voice-activity check: the window must be at least 90 % non-speech.

- expect_silence: 5

A window the recording does not cover is an ERROR, never a PASS — absent audio must not read as silence.

expect_hangup

Passes when the far end releases the call within within seconds.

- expect_hangup: { within: 20 }

say

Speaks synthesized audio into the call. It exists for RECORD steps that demand a voice and reject DTMF — verbal authorization, name capture. Never use it to stand in for a caller's menu choices; those stay press.

Letters, digits, space and , . ' - only. Maximum 120 characters.

- wait_for: "speak your full name to verbally authorize"
- wait: 1
- say: "Chris Testing"
- wait: 1
- press: "#"

hangup

Local release. Used for abandonment tests, and to get off the line before a real agent answers.

- hangup: {}

note

An INFO row in the report. Annotation only, never graded.

- note: "readback: 'you entered 8 5 9 6 3 5 1 4 1 4. if this is correct press 1'"

Failure semantics

Step statuses: PASS · FAIL · INFO · ERROR · SKIPPED.
Scenario verdict precedence: ERROR > FAIL > PASS.

A failed expect_* marks that step FAIL and the call keeps going, so one run collects all the evidence rather than stopping at the first problem.

SKIPPED never fails a verdict on its own, so the runner records a call error — which makes the verdict ERROR — for every reason a step could not run and the result would otherwise read as a clean pass:

• Call dropped mid-script. Later steps are SKIPPED with call ended. A skipped press also sets a call error: the digits the scenario exists to send never went out.
• Operator stop. Every remaining step is SKIPPED with stopped by user, including when the stop lands on the last step. An interrupted expect_hangup or expect_silence is SKIPPED, not graded — the far end never got its full window, so a verdict drawn from it would be about the abort, not about the IVR.
• Missing audio. A window the recording does not cover is ERROR.

Unhappy-path patterns

API reference

The console is a client of this API and nothing more. Everything it does, you can drive from curl.

Bound to 127.0.0.1:8474, loopback only. A non-loopback Host, or a cross-site Origin on a write, is refused with 403. The SIP password is never returned by any endpoint.

POST /api/run

Start a run.

curl -s -X POST http://127.0.0.1:8474/api/run \
  -H 'Content-Type: application/json' \
  -d '{"names":["bsc_billing_queue","bsc_repair_queue"],"confirm":true}'

{ "run_id": "20260917-011804-7c3a", "engine": "pjsua", "mode": "LIVE", "scenarios": 2 }

409 need_confirm in LIVE mode until confirm: true is sent. The body carries what a human would need to approve: the destination, the ANI and the max duration.

{
  "error": "need_confirm",
  "dial": "15133064718",
  "ani": "15135550100",
  "max_duration_sec": 600,
  "message": "LIVE mode places a real PSTN call and spends money"
}

GET /api/status

Live state, for polling while a call is in flight.

{
  "running": true,
  "run_id": "20260917-011804-7c3a",
  "mode": "LIVE",
  "engine": "pjsua",
  "scenario": "bsc_billing_queue",
  "step_index": 11,
  "step_verb": "wait_for",
  "elapsed_ms": 128400,
  "last_event": { "t_ms": 2310, "kind": "answered", "detail": "200 OK" }
}

{"running": false} when idle. POST /api/stop hard-stops the current call; the engine releases it.

GET /api/runs/<run_id>

The complete stored run. This is the RunResult structure, the same object the JSON export writes.

{
  "run_id": "20260917-011804-7c3a",
  "created_at": "2026-09-17T01:18:04",
  "engine": "pjsua",
  "scenario_results": [
    {
      "verdict": "PASS",
      "scenario": { "name": "bsc_billing_queue", "dial": "15133064718", "tags": ["live","queue"] },
      "call": {
        "scenario_name": "bsc_billing_queue",
        "dial": "15133064718",
        "ani": "15135550100",
        "engine": "pjsua",
        "started_at": "2026-09-17T01:18:04",
        "ended_at": "2026-09-17T01:24:57",
        "connected": true,
        "hangup_by": "local",
        "recording_path": ".../recordings/20260917-011804-7c3a/bsc_billing_queue/call.wav",
        "recording_started_ms": 2310,
        "events": [
          { "t_ms": 2310,   "kind": "answered",   "detail": "200 OK" },
          { "t_ms": 3000,   "kind": "dtmf_sent",  "detail": "3216352882" },
          { "t_ms": 412800, "kind": "local_hangup", "detail": "script complete" }
        ]
      },
      "step_results": [
        {
          "index": 11,
          "verb": "wait_for",
          "status": "PASS",
          "started_ms": 58200,
          "ended_ms": 121900,
          "expected": "monitored and recorded | survey",
          "observed": "this call may be monitored and recorded for quality assurance",
          "score": 0.94,
          "audio_segment": "seg_011.wav",
          "detail": "matched at t+117.200"
        }
      ]
    }
  ],
  "notes": ""
}

Field names are the frozen contract. started_ms and ended_ms are call-clock milliseconds; score is present on expect_* and wait_for steps only.

Related: GET /api/runs lists stored runs. GET /api/runs/<id>/export/html|csv|json produces the exports — add ?embed=1 to inline audio in the HTML. GET /audio/<run_id>/<file> serves one recorded slice.

POST /api/adapt

Adapt a scenario from the evidence of a completed run.

curl -s -X POST http://127.0.0.1:8474/api/adapt \
  -H 'Content-Type: application/json' \
  -d '{"run_id":"20260916-233901-1f02","scenario":"bsc_billing_queue","mode":"suggest"}'

{
  "scenario": "bsc_billing_queue",
  "mode": "suggest",
  "stable": false,
  "changes": [
    {
      "kind": "wait_converted",
      "step_index": 4,
      "before": "wait: 10",
      "after": "wait_for: {texts: [\"new customer\",\"pending order\"], timeout_sec: 45, min_score: 0.5}",
      "evidence": {
        "reason": "DTMF sent at t+20.000 while utterance t+10.460-t+24.800 was still playing",
        "utterance": "thank you for calling. if you are a new customer or have a pending order press 1",
        "audio_segment": "seg_005.wav"
      }
    },
    {
      "kind": "phrase_replaced",
      "step_index": 9,
      "before": "expect_prompt: \"your call may be monitored or recorded for quality\"",
      "after": "wait_for: {texts: [\"monitored and recorded\"], timeout_sec: 60, min_score: 0.5}",
      "evidence": {
        "reason": "best score 0.62 against threshold 0.75; verbatim text found at t+117.200",
        "utterance": "this call may be monitored and recorded for quality assurance",
        "audio_segment": "seg_011.wav"
      }
    }
  ],
  "yaml": "name: bsc_billing_queue\ndial: \"15133064718\"\n..."
}

kind is one of press_moved, phrase_replaced, phrase_dropped, timeout_set, stop_on_added, wait_converted. Every change carries the evidence that justified it, and every evidence.audio_segment is playable from GET /audio/<run_id>/<file>.

Also available on the command line as earshot adapt <run_id>.

Scenario CRUD rounds out the API: GET /api/scenarios, GET /api/scenarios/<name>, POST /api/scenarios/validate (dry run through the real loader, writes nothing), POST /api/scenarios, PUT /api/scenarios/<name> (guarded by base_sha, 409 if the file changed underneath you), DELETE /api/scenarios/<name> (soft delete into _trash/), POST /api/preview_value, GET /api/preflight, GET /api/library.

GET /api/testlog

The test log. Every run files itself here automatically: one row per scenario per run, with the auto verdict the runner produced, the reviewed verdict an engineer set (if any), and the notes. Per-step auto verdicts are in the run's RunResult; the log is the flat, filterable view.

curl -s 'http://127.0.0.1:8474/api/testlog?days=2&verdict=FAIL'

{
  "rows": [
    {
      "run_id": "20260917-011804-7c3a",
      "scenario": "bsc_repair_queue",
      "started_at": "2026-09-17T01:25:11",
      "dial": "15133064718",
      "ani": "15135550100",
      "auto_verdict": "FAIL",
      "reviewed_verdict": "FAIL",
      "effective_verdict": "FAIL",
      "steps_passed": 9,
      "steps_total": 11,
      "notes": "Callback offer never played on the repair skill. IVR bug, not a scenario bug. Filed as 118.",
      "reviewed_by": "chris",
      "reviewed_at": "2026-09-17T08:41:02",
      "report": "/api/runs/20260917-011804-7c3a/export/html"
    }
  ],
  "total": 1,
  "generated_at": "2026-09-17T13:30:00"
}

reviewed_verdict is null until an engineer sets one. effective_verdict is what the report banner, the Evidence Library and the CI exit code use: the reviewed verdict when there is one, the auto verdict otherwise. The auto verdict is never rewritten.

GET /api/testlog.csv

The same rows, same filters, as a flat CSV. This is the file you attach to the UAT sheet. Both verdict columns are always present, so a pivot across a month of runs can tell which verdicts a human touched.

run_id,scenario,started_at,dial,auto_verdict,reviewed_verdict,effective_verdict,steps_passed,steps_total,notes,reviewed_by,reviewed_at
20260917-011804-7c3a,bsc_billing_queue,2026-09-17T01:18:04,15133064718,PASS,,PASS,11,11,"90.1 s hold and 120.0 s cadence proven from the recording.",,
20260917-011804-7c3a,bsc_repair_queue,2026-09-17T01:25:11,15133064718,FAIL,FAIL,FAIL,9,11,"Callback offer never played on the repair skill. Filed as 118.",chris,2026-09-17T08:41:02
20260916-233901-1f02,bsc_billing_queue,2026-09-16T23:39:01,15133064718,FAIL,PASS,PASS,3,7,"Scenario error, not IVR: press landed early. Adapted; rerun passed.",chris,2026-09-17T08:44:19

Add ?steps=1 for one row per step instead of per scenario — the per-step auto status (PASS / FAIL / INFO / ERROR / SKIPPED) with the scenario's two verdicts repeated on every row.

PATCH /api/runs/<run_id>/review

Set or clear the reviewed verdict and edit the notes for one scenario in a run. This is the only write the test log accepts; it never touches the auto verdict, the step ladder or the audio.

curl -s -X PATCH http://127.0.0.1:8474/api/runs/20260916-233901-1f02/review \
  -H 'Content-Type: application/json' \
  -d '{"scenario":"bsc_billing_queue","verdict":"PASS","notes":"Scenario error, not IVR. Adapted; rerun passed."}'

{
  "run_id": "20260916-233901-1f02",
  "scenario": "bsc_billing_queue",
  "auto_verdict": "FAIL",
  "reviewed_verdict": "PASS",
  "effective_verdict": "PASS",
  "notes": "Scenario error, not IVR. Adapted; rerun passed.",
  "reviewed_by": "chris",
  "reviewed_at": "2026-09-17T08:44:19"
}

Every review is appended to the run's audit trail — previous verdict, new verdict, who, when — and the exported HTML report shows the auto verdict beside the reviewed one with the note underneath. 404 if the run or scenario does not exist; 422 for a verdict outside the three values.

CLI reference

earshot run [scenarios...] [options]

earshot adapt <run_id> [--scenario <name>] [--mode suggest|auto] [--max-iterations N]
earshot validate [path]        # run every scenario through the loader, write nothing
earshot preflight              # engine, transcriber, model, credentials, scenarios
earshot export <run_id> --html <path>
earshot import <sheet.csv|sheet.xlsx> [--dry-run]   # CSV / XLSX test-case sheet to scenarios/
earshot testlog [--days N] [--csv <path>]            # the test log, table or CSV
earshot schedule add "0 1 * * *" --tag queue --junit out/queue.xml --alert slack

Exit codes: 0 all passed · 1 at least one FAIL · 2 call or configuration error.

earshot --help ends with the same pointer the installer prints: docs: https://earshot.channel/docs.

AI assistant (bring your own key)

Earshot works with the latest AI models through a key you supply. There is no Earshot-side model, proxy or account: the desktop app calls the provider you configure, directly, with your key, only when you use an AI feature. Nothing about the assistant is required to run, grade or report a test.

Configuration: ai.env

One file, on your machine, outside the application folder. On macOS and Windows alike it is ~/.config/ivr-tester/ai.env (Windows resolves ~ to %USERPROFILE%). Make it readable by you only; the console warns if it is group- or other-readable.

# provider: anthropic | openai
AI_PROVIDER=anthropic

# exactly one key, for the provider above
ANTHROPIC_API_KEY=sk-ant-…
# OPENAI_API_KEY=sk-…

# model id at that provider
# Claude:  claude-sonnet-5 (default) · claude-opus-5 · claude-fable-5-1
# OpenAI:  any GPT model id your account can use
AI_MODEL=claude-sonnet-5

The assistant never dials and never changes a verdict. Every draft goes through the same strict loader as a hand-written file, and every proposal is shown as a scenario or a diff for you to save, edit or discard.

GET /api/ai/status

Whether the assistant is configured, and with what. Key presence only; the key value is never returned.

{ "configured": true, "provider": "anthropic", "model": "claude-sonnet-5", "key_source": "~/.config/ivr-tester/ai.env", "file_mode": "600" }

{"configured": false, "reason": "no ai.env"} when the file is absent; the AI buttons in the console are hidden in that state.

POST /api/ai/scenario

Draft one scenario from a plain-English description.

curl -s -X POST http://127.0.0.1:8474/api/ai/scenario \
  -H 'Content-Type: application/json' \
  -d '{"description":"Call the billing line, authenticate with account 5135551234, expect the balance to read $142.17","dial":"15135660137"}'

{
  "name": "billing_balance_happy",
  "yaml": "name: billing_balance_happy\ndial: \"15135660137\"\nsteps:\n  - wait: 6\n  - expect_prompt: \"for billing press 1\"\n  - press: \"1\"\n  ...",
  "valid": true,
  "warnings": ["step 7 expect_value: context phrase is a guess; confirm from a recording"],
  "model": "claude-sonnet-5",
  "elapsed_ms": 1840
}

Optional fields: dial, tags, template (happy_path, no_input, invalid_entry, out_of_retries, caller_abandons, barge_in), from_run (a run id whose transcript the model may quote from — the best way to get real prompt fragments). The draft is validated before it is returned; valid: false carries the loader's error so nothing broken is silently saved.

POST /api/ai/plan

Plan a test set for a goal: the happy path and the unhappy paths that goal implies. You review, save and run; nothing is written by this call.

curl -s -X POST http://127.0.0.1:8474/api/ai/plan \
  -H 'Content-Type: application/json' \
  -d '{"goal":"Test the billing balance path on 15135660137: authenticate with account 5135551234, and cover no input and a wrong account number"}'

{
  "scenarios": [
    { "kind": "happy_path",    "name": "billing_balance_happy",   "yaml": "…", "valid": true },
    { "kind": "no_input",      "name": "billing_menu_noinput",    "yaml": "…", "valid": true },
    { "kind": "invalid_entry", "name": "billing_account_invalid", "yaml": "…", "valid": true }
  ],
  "rationale": "Three paths cover the goal; out_of_retries is implied by invalid_entry and folded in.",
  "model": "claude-sonnet-5",
  "elapsed_ms": 2810
}

Optional: paths to restrict to specific kinds, from_run as above, max (default 6). Save any subset with POST /api/scenarios — the console's Save button does exactly that — and run with POST /api/run.

POST /api/ai/explain

Explain a failed run and draft the test-log note. Input is the run's step ladder, transcripts and timings; output is a short explanation, the most likely cause classified as a scenario error or an IVR error, and a note you can drop into PATCH /api/runs/<id>/review.

{
  "summary": "Step 15 waited 130 s for the callback offer and never heard it. Every earlier step matched.",
  "likely_cause": "ivr",
  "confidence": 0.8,
  "notes_draft": "Callback offer never played on the repair skill; hold music ran 220 s straight to the first delay message. Likely queue config, not the scenario.",
  "suggested_verdict": "FAIL",
  "evidence": [ { "step_index": 15, "audio_segment": "seg_015.wav", "t_start_ms": 222180, "t_end_ms": 352180 } ]
}

suggested_verdict is a suggestion; the reviewed verdict is only ever set by the PATCH you send.

POST /api/ai/review-adapt

Review the changes an adaptive run proposes before you accept them. Send the changes array from POST /api/adapt; get back a recommendation per change, with the reason, so a change that would lower a threshold or drop an expectation that was actually a real regression gets flagged.

{
  "reviews": [
    { "index": 0, "recommend": "accept", "why": "The gate replaces a guessed wait with the prompt that actually precedes the press." },
    { "index": 4, "recommend": "reject", "why": "Dropping 'press one to continue holding' hides a missing prompt; the phrase was heard in run 9b31." }
  ],
  "model": "claude-sonnet-5"
}

recommend is accept, edit or reject. The console shows it beside each change in the diff view; applying is still your click.

Every AI endpoint returns 409 ai_not_configured when ai.env is absent, 502 provider_error with the provider's message when the upstream call fails, and never blocks a run — a failed AI call fails the AI call, not the test.

Scenario import (CSV / XLSX)

Most teams already have a test-case sheet. Import it. Two row shapes are accepted, detected from the header; both go through the same strict loader as a YAML file, and an error names the row and the column.

One row per scenario

Steps in one cell, separated by |, each as verb value. Quotes are optional unless the value contains |.

name,dial,tags,steps
billing_balance_happy,15135660137,"billing,happy","wait 6 | expect_prompt ""for billing press 1"" | press 1 | expect_prompt ""account number"" | press 5135551234# | expect_value currency 142.17 | expect_hangup 20"
billing_menu_noinput,15135660137,"billing,unhappy","wait 6 | expect_prompt ""for billing press 1"" | wait 12 | expect_prompt ""didn't get that"" | hangup"

One row per step

Easier to write in a spreadsheet and easier to diff. Rows with the same name are one scenario; step orders them; dial and tags are read from the first row of each scenario.

name,dial,tags,step,verb,value,option
billing_balance_happy,15135660137,"billing,happy",1,wait,6,
billing_balance_happy,,,2,expect_prompt,for billing press 1,min_score=0.75
billing_balance_happy,,,3,press,1,
billing_balance_happy,,,4,expect_prompt,enter your 10 digit account number,
billing_balance_happy,,,5,press,5135551234#,
billing_balance_happy,,,6,expect_value,142.17,type=currency;context=your total balance
billing_balance_happy,,,7,expect_hangup,20,

option takes key=value pairs separated by ; for any verb field in the verb reference. XLSX uses the first sheet, same columns. Column names are matched case-insensitively and a few aliases are understood (scenario for name, number for dial, expected for value).

earshot import uat_round_3.xlsx --dry-run     # validate, report per row, write nothing
earshot import uat_round_3.xlsx               # write scenarios/*.yaml, existing names go to _trash/

Also POST /api/scenarios/import (multipart, field file, add ?dry_run=1), returning {"created": [...], "replaced": [...], "errors": [{"row": 14, "column": "verb", "message": "unknown verb 'expect_prompt_text'"}]}. Any error and nothing is written; the sheet is fixed and re-imported. The imported files are ordinary scenarios — open them in the builder, edit them by hand, or send them to the assistant.

SIP trunk setup

Earshot registers as a plain SIP user agent. Any SIP provider works — voip.ms, Twilio, Telnyx, Bandwidth. The steps below are for voip.ms because that is what the reference setup uses; the shape is the same everywhere.

1. Create a dedicated sub-account. Do not use your main SIP credentials. Sub-accounts are free, revocable, and can be rate- and route-limited.
2. Device type: SIP. Not IAX. The adapter registers as a plain SIP UA.
3. Username is the full <mainaccount>_<suffix> string, e.g. 123456_test.
4. Note the POP server assigned to that sub-account, e.g. seattle.voip.ms. Pick the POP closest to you — a distant POP costs you audio quality, and audio quality is transcription accuracy.
5. Caller ID. If your flows branch on the calling number — known customer against unknown, in-footprint against out-of-footprint, employee line — set the sub-account to allow a user-supplied CallerID value. A scenario's ani: field is then presented on the call. Otherwise omit ani: and every test presents the same number.
6. Check the balance. Outbound calls are prepaid; a zero balance fails the call at the provider with no useful error.
7. Confirm outbound calling is permitted for the destinations you plan to dial, including toll-free, which some routes treat separately.

Credentials file, chmod 600, outside the application folder:

SIP_USER=123456_test
SIP_PASS=your-subaccount-password
SIP_SERVER=seattle.voip.ms
# Optional default caller ID (the sub-account must allow user-supplied CLID)
#SIP_CLID=15135550100

The mode is chosen by whether this file exists, so the app cannot surprise you by dialing: delete the file and you are back in SIM mode. The mode and the engine in use are shown in the header, and every stored result records which engine produced it.

Preflight checks credentials, the SIP binary, the transcriber binary, the model file, ffmpeg and the scenario directory, and shows one chip per check with the resolved path or the reason it is red. It never places a call and never costs anything. Run it before a session rather than discovering a missing binary mid-suite.

SIM mode

With no credentials configured, Earshot runs the identical pipeline — record → VAD → transcribe → match → report — against a local fake IVR. Full pass/fail report, audio, transcripts and exports, without dialing anything or spending a cent.

In SIM mode only scenarios named sim_demo* can run — they are the ones the fake IVR has a script for. Any other scenario is refused with a clear error rather than being graded against a line that was never really called.

SIM is not a toy. It runs the same runner, recorder, VAD, transcriber and matcher. If a scenario passes in SIM, the plumbing is proven and only the telephony is untested.

Platforms

Earshot needs a phone number, not an API key, so any platform that answers a call is testable at call level. NICE CXone is the platform Earshot was built and proven on and carries the deepest integration: contact correlation, Data Streaming capture, Studio trace alignment and the simulator: block. See integrations.

• NICE CXoneSupported
• Avaya Aura · AAEP · Communication Manager
• Genesys Cloud
• Amazon Connect
• Cisco
• Five9
• Twilio
• Asterisk / FreePBX
• Anything that answers a call