How to Measure Answering Machine Detection Accuracy

A dialer report may show that 70% of answered calls were classified as machines, but that does not tell you whether the detector was right. Without independently reviewing what actually answered each call, the report only measures what the detector said about itself.

That mistake can hide lost contacts, agent delays, wasted voicemail transfers, and systematic failures on call screening services. AMDY provides detection results for ViciDial and predictive dialers, but those results still need a separate ground-truth process if you want a defensible accuracy benchmark.

Quick answer: how to test answering machine detection accuracy

Use this workflow to produce a benchmark you can repeat and compare:

1. Define human, machine, ambiguous, and invalid ground-truth labels before reviewing calls.
2. Select a representative sample across campaigns, carriers, hours, regions, and call outcomes.
3. Save the detector's original classification, confidence or reason code, and decision time.
4. Have trained reviewers label each call without seeing the detector's answer.
5. Send disagreements and ambiguous calls to a second reviewer for adjudication.
6. Build a confusion matrix and calculate false-positive, false-negative, precision, recall, and latency results.
7. Add confidence intervals for every rate and report subgroup results separately.
8. Repeat the test after configuration, traffic, carrier, model, or dialer changes.

What answering machine detection accuracy actually measures

Answering machine detection is a classification task performed during the first seconds of an answered call. The detector listens for speech, silence, greeting structure, tones, and other audio or timing patterns, then returns a result such as human, machine, or uncertain.

The exact categories vary by product. Asterisk, for example, exposes an AMD dialplan application with configurable thresholds and status causes. Its official AMD application documentation shows why a single accuracy percentage cannot describe every deployment: greeting duration, word count, silence, and analysis time all affect the result.

Accuracy is the proportion of evaluated calls classified correctly. If 927 out of 1,000 independently labelled calls match the detector's output, accuracy is 92.7%.

That number is incomplete on its own. A detector could obtain high overall accuracy while still sending too many live people to a machine branch. Call centers usually care about several outcomes:

• False positive: a live human is classified as a machine.
• False negative: a machine is classified as a human.
• True positive: a machine is correctly classified as a machine.
• True negative: a human is correctly classified as a human.
• Uncertain result: the detector declines to make a confident binary decision.

Some teams reverse the meaning of positive and negative. State your convention in every report. In this article, machine is the positive class.

A false positive often costs a contact opportunity because the live person may hear silence, a message, or a disconnect. A false negative consumes agent time or puts voicemail audio into the live-call queue. Both matter, but they do not have equal operational cost.

Read the AMD terms glossary if your dialer, vendor, and reporting team use different names for these outcomes.

How to define ground truth for answering machine detection testing

Ground truth is the reviewed answer to a simple question: what actually answered the call? It must come from evidence independent of the detector being tested.

Create the labelling guide before selecting reviewers. At minimum, use four labels:

• Human: a live person speaks, including a short greeting such as “hello” or silence followed by a live response.
• Machine: voicemail, an answering machine greeting, an automated attendant, or another recorded endpoint answers.
• Ambiguous: the available audio does not support a reliable human-or-machine decision.
• Invalid: the call lacks usable audio, has incorrect recording boundaries, or was not truly answered.

Do not force every call into a binary label. A recording that starts late may contain only “leave your message after the tone.” That is probably a machine. A recording containing half a syllable followed by a disconnect may be impossible to label responsibly.

The policy for automated attendants and call screening must also be explicit. A consumer call screening service may play a synthetic prompt, wait for the caller's name, and then connect a live person. You could label that endpoint as machine because automation answered first, or create a separate screening category because the correct dialer action differs from ordinary voicemail.

Either choice can work. Consistency is essential.

The call screening detection guide explains why screening should not be buried inside a general machine bucket when it requires different routing or retry behavior.

Use blind review and adjudication

Reviewers should not see the detector's classification, confidence, or reason code before labelling. Otherwise, uncertain recordings tend to be labelled in agreement with the machine result.

Give two reviewers an overlapping subset. When they disagree, a senior reviewer adjudicates the call using the written policy. Track disagreement rates by category. Frequent disputes around screening prompts or long human greetings indicate that the labelling guide needs another example.

Preserve the original recording and label history. If the definition changes later, you need to know which calls were judged under the old policy.

Define the evaluation unit

Use one answered call attempt as the evaluation unit. Do not count individual audio frames, detector events, or repeated API messages as separate observations.

If the same phone number receives five attempts, those attempts are still five detector decisions. They are not fully independent, however. Repeated voicemail greetings and identical network paths can make confidence intervals look tighter than they should. Report the number of unique destinations alongside the number of calls, and consider sampling only one attempt per destination for a sensitivity check.

How to select calls for answering machine detection accuracy testing

Export the last 90 days first. Include enough fields to reconstruct what happened: campaign, list, destination region, call time, carrier or trunk when available, dialer disposition, detector output, decision timestamp, call identifier, and recording reference.

A simple random sample is suitable when production traffic is stable and you want one overall estimate. It becomes weak when important conditions are rare. Weekend traffic, Spanish-language greetings, business numbers, call screening, and a newly added SIP trunk may disappear inside the average.

Stratified sampling solves that problem. Divide calls into operational groups and draw a planned number from each. Useful strata include campaign, trunk, hour block, destination country or region, language, detector result, and answer type.

Consider an illustrative ViciDial campaign named TEST_AMD_Q2. The team exports 10,000 answered attempts and initially reviews the first 1,000 recordings returned by the database. Most came from one daytime campaign because the query was ordered by call time. The resulting benchmark says little about evening consumer traffic or the new trunk that prompted the test.

The corrected sample draws calls randomly within each campaign, time block, and trunk. It also oversamples uncertain classifications and suspected screening calls for diagnostic analysis.

Oversampling creates one important reporting issue. If uncertain calls are 5% of production but 20% of the test set, the raw test-set accuracy is not an estimate of production accuracy. Apply sampling weights for the overall rate, or report the representative benchmark separately from the diagnostic sample.

How many calls do you need?

Sample size depends on the precision required, not just total call volume. For a proportion near 50%, approximately 384 independent observations produce a 95% confidence interval with a margin of error close to five percentage points. About 1,067 observations reduce the margin to roughly three points.

Those are worst-case planning values for one overall proportion. They do not guarantee precise subgroup results. If only 40 reviewed calls used a particular carrier, that carrier's result remains uncertain even when the total test contains 5,000 calls.

Rare but expensive failures need targeted samples. If false positives on live humans are the main concern, the relevant denominator is the number of ground-truth human calls, not the total sample. Plan for enough human calls to estimate that rate.

How to calculate answering machine detection accuracy

Start with a confusion matrix. The following table uses an illustrative random sample of 1,000 calls so the arithmetic is visible. It is not an AMDY benchmark or a claim about any vendor.

Accuracy is (570 + 357) / 1,000, or 92.7%.

The machine false-negative rate is 50 / 620, or 8.1%. These machines reached the path intended for humans.

The human false-positive rate is 23 / 380, or 6.1%. These live people were sent toward the machine path. For many outbound teams, this is the first metric to inspect because it represents contacts that were already answered.

Machine precision is 570 / 593, or 96.1%. It answers a different question: when the detector says machine, how often is that classification correct?

Machine recall is 570 / 620, or 91.9%. This measures how many actual machines the detector caught.

Specificity, which is also human recall under this convention, is 357 / 380, or 93.9%.

Do not combine uncertain results with whichever class improves the score. Report uncertain coverage separately, then publish at least two policy views. One can treat uncertain as an error. Another can apply the actual production routing rule, such as sending uncertain calls to agents.

The article on how often AMD is wrong gives more context for interpreting error types instead of relying on one headline percentage.

Add confidence intervals to every rate

A measured rate is an estimate drawn from a sample. A 6.1% false-positive rate from 380 human calls is less precise than the same rate from 3,800 human calls.

Use a binomial proportion interval such as Wilson's interval. Avoid presenting only the normal approximation when counts are small or the rate is close to zero or one. The NIST guidance on confidence limits for proportions describes the underlying treatment, while SciPy documents a practical proportion confidence interval implementation for reproducible analysis.

For the illustrative table, the 92.7% overall accuracy has an approximate 95% interval of 91.1% to 94.2% using a Wilson calculation. The human false-positive rate of 23 out of 380 has a wider interval, approximately 4.1% to 8.9%.

Write the denominator next to the interval. “6.1%, 95% CI 4.1% to 8.9%, n=380 ground-truth humans” is much more useful than “94% accurate.”

A confidence interval does not correct biased sampling, weak labels, duplicated calls, or missing recordings. It only quantifies sampling uncertainty under the assumptions of the calculation.

How to test real-time answering machine detection behavior

Offline replay is useful because every detector can receive the same recordings. It is not a complete production test.

Real-time detectors make decisions while audio is arriving. Packet timing, early media, clipping, codec changes, jitter, silence suppression, and answer supervision can alter the input. Replaying a clean WAV file through a script may remove the exact conditions causing production errors.

Run two complementary tests. Use fixed recordings for repeatable comparisons, then conduct a shadow or controlled live test through the production media path. In the live test, save the final classification and the time between answer and decision.

Latency belongs beside accuracy. A detector that waits 6 seconds may classify a long voicemail greeting correctly but leave a live person hearing dead air. A faster detector may connect agents sooner while making more errors on long human greetings.

Measure at least the median, 90th percentile, and 95th percentile decision time for each ground-truth class. Also record the share of decisions that exceed your operational threshold.

Configuration changes require a fresh benchmark. In Asterisk, modifying greeting, silence, word-length, or analysis thresholds changes detector behavior. FreeSWITCH media handling, ViciDial routing, trunk changes, and answer supervision can have the same practical effect even when the detector itself is unchanged. The ViciDial AMD guide covers where detection fits into that call flow.

Why production classifications are not an accuracy benchmark

A production dashboard contains predictions. Accuracy requires predictions paired with independent truth.

Suppose TEST_AMD_Q2 reports 6,200 machines, 3,500 humans, and 300 uncertain calls. Dividing those counts does not reveal whether any category was correct. It only describes the detector's output distribution.

Agent dispositions do not automatically solve the problem. Agents usually hear calls classified as human, so they cannot label the machine branch. A voicemail classified as human may receive a machine disposition, but a human classified as machine never reaches an agent and may have no corrective label. The data is selectively observed.

Dialer dispositions also mix detection with downstream events. An agent can mark a live call as no answer, a short recording can be marked as dead air, and an automated attendant may be coded as voicemail. These labels were designed for workflow and reporting, not blinded evaluation.

Production data is still valuable. Use it to find shifts in classification mix, latency, uncertain rates, and campaign-level anomalies. Treat those signals as reasons to sample and review calls, not proof that accuracy improved.

How AMDY supports answering machine detection accuracy testing

A repeatable test needs access to classifications, timing context, and stable routing behavior. AMDY removes part of that collection work while leaving the independent review where it belongs.

Start with the dialer path. Teams using ViciDial can review the AMDY ViciDial implementation and operating model to identify where the detector receives the call and where its result affects routing.

Next, capture the original response rather than translating every result immediately into a broad dialer disposition. The AMDY Detection API guide documents the WebSocket protocol and classification codes that the client receives. Store those values with the call identifier and timestamp used by your test export.

Then inspect what the returned detection data means. The AMDY features page explains the available detection information so the evaluation file can preserve useful detail instead of retaining only “human” or “machine.”

After collection, reviewers still listen to the sampled calls without seeing AMDY's answer. AMDY does not become its own ground truth. Once labels are complete, join them to the saved detection results by call identifier, construct the confusion matrix, and calculate rates by campaign, trunk, and call type.

Finally, decide how production should handle uncertain outcomes. The guide to ViciDial NOTSURE behavior explains the routing trade-offs. Sending uncertain calls to agents can protect live contacts but adds machine audio to the agent queue. Sending them to the machine path reduces agent waste but risks losing people.

This workflow gives AMDY users the material needed for testing without pretending that vendor output is objective truth. Teams evaluating operational fit can also review current AMDY pricing after establishing the call volume and test requirements.

Common answering machine detection testing mistakes

The most damaging mistake is labelling from dialer dispositions alone. This creates missing truth for false positives because live people routed away from agents rarely generate a human disposition.

Another mistake is reviewing only calls that operators complained about. Complaint samples are useful for diagnosis, but they exaggerate visible failure modes and miss silent losses.

Testing one campaign can also produce a misleading result. Debt collection, solar, appointment reminders, and business-to-business dialing encounter different greetings and answer patterns. Report each major traffic segment before combining them.

Avoid changing thresholds during the test. Freeze the detector version, dialplan, trunk routing, and classification policy for each benchmark run. If a change is necessary, start a new versioned cohort.

Do not discard short calls automatically. Short duration may be caused by a false positive, an agent delay, a consumer hang-up, or incorrect answer supervision. Review a sample before writing an exclusion rule.

Finally, do not compare rounded vendor claims with your own audited result. A published “up to” figure may use another definition, traffic mix, exclusion policy, or latency target. Run every detector against the same labelled calls and live-call conditions.

How to compare answering machine detection providers fairly

Create one frozen evaluation set and one scoring script. Give each provider identical audio where offline testing is supported, then repeat a live test through equivalent routing.

Before scoring, map provider-specific outputs into the same policy categories. MightyCall, VMHunter, Amdify, built-in Asterisk AMD, and specialist APIs may expose different uncertain states or reason codes. Preserve the original output as well as the normalized label.

Compare false positives, false negatives, uncertain coverage, and decision time. Include operational requirements such as ViciDial integration, API behavior, data availability, support for call screening, and the effort required to reproduce a result. The comparison of AMD options can help identify the questions to ask, but your independently labelled sample should decide accuracy.

Keep commercial evaluation separate from technical scoring. Check each vendor's own pricing page for current numbers, then model cost against answered calls, concurrent calls, or whichever billing unit applies. A cheaper detector can cost more operationally if false positives remove valuable live contacts.

Answering machine detection accuracy testing FAQ

What is a good answering machine detection accuracy rate?

There is no universal threshold. A useful target must include the traffic mix, false-positive rate on humans, false-negative rate on machines, uncertain coverage, decision latency, sample size, and confidence intervals.

For an outbound sales campaign, protecting live contacts may take priority. A voicemail-heavy notification campaign may put more weight on machine recall. Compare results against your current routing under the same conditions.

Can I use agent dispositions as ground truth?

Use them to locate likely errors, not as the only truth source. Agents observe mainly the calls routed to them, which excludes many potential false positives. Review recordings from every detector branch and label them independently.

Should ambiguous calls count as wrong?

Report them separately first. Then show results under the production policy. If uncertain calls go to agents, score the effect of that routing. If they go to voicemail treatment, calculate the resulting human-loss risk.

How often should AMD accuracy be retested?

Retest after detector updates, threshold changes, SIP trunk changes, dialer upgrades, new campaigns, major geographic shifts, or a rise in uncertain and short-call outcomes. Stable operations still benefit from scheduled samples because carrier behavior and consumer screening tools change.

Can prerecorded audio provide a valid AMD benchmark?

Yes, for repeatable classification comparison, provided every detector receives equivalent audio and the recordings reflect production. Add a real-time test because file replay does not reproduce answer supervision, packet timing, clipping, jitter, or live decision latency.

Does a larger sample fix bad labels?

No. More calls narrow sampling uncertainty, but they do not correct biased selection or inconsistent review. Improve the label guide, blind the reviewers, adjudicate disagreements, and preserve ambiguous cases before increasing sample size.

Which AMD metric should a call center monitor first?

Start with the false-positive rate among independently confirmed humans. It estimates the share of live answers incorrectly routed as machines. Monitor machine false negatives and decision latency beside it because reducing one error can increase another.

A qualified evaluation should end with a versioned dataset, a written label policy, a confusion matrix, subgroup rates, latency percentiles, and confidence intervals. If a provider cannot be tested with independent labels or cannot expose stable classifications for joining to call records, its headline accuracy figure should not drive the decision. Read next: FreeSWITCH Answering Machine Detection: Setup and Testing Guide.