How to evaluate AI earnings call analysis

Publication correction

An earlier version of this page reported comparative accuracy scores from a 50-call test. We do not have the underlying evaluation records needed to substantiate those figures. The scores and winner claims have been removed. This page is now an evaluation guide; it does not report a completed benchmark.

Define a sample before testing

Choose companies, reporting periods, and industries before inspecting model answers. Record why each company is included. A proposed 50-call sample can provide a manageable test set, but sample size alone does not make results representative.

• Record issuer, ticker, reporting period, and call date
• Preserve the original transcript and earnings release
• Include straightforward and difficult questions

Keep inputs comparable

Provide each tool with the same source materials and questions. Record the product, model, plan, date, and whether web access or retrieval was enabled. If you compare complete research workflows, report their different data access explicitly.

• Save the exact prompt and unedited response
• Record available tools and source access
• Repeat selected questions to assess consistency

Check facts against primary sources

Use the issuer's earnings release, investor-relations materials, and relevant SEC filings as the reference documents. SEC EDGAR provides searchable filings. Preserve the supporting passage for each assessed claim.

• Check reporting period, currency, and units
• Distinguish GAAP figures from adjusted measures
• Check whether guidance is a range or a point estimate
• Verify quotations in context

Use a declared scoring rubric

Define correct, partially correct, unsupported, incorrect, and unanswered before scoring. Report denominators and individual task results. Keep extraction, synthesis, citation quality, and uncertainty separate; one combined percentage can hide material failures.

• Count unsupported claims even when they sound plausible
• Credit a justified statement that data is missing
• Have a second reviewer check disputed assessments

Publish enough to reproduce the comparison

A credible result includes the source manifest, prompts, model settings, outputs, rubric, and reviewer decisions. Publish permitted excerpts and source links where full documents cannot be shared. Declare the publisher's commercial interest and limits of the sample.

Where Ecomerate fits

Ecomerate offers AI-assisted company and filing research. Use these same checks to assess its answers. We publish this guide as the product provider; it is not an independent certification or a claim that Ecomerate outperforms every other tool.