TL;DR
Gorgias published weekly competitive evals of its AI customer support agent against named competitors, including metrics where it loses, with fully open-sourced testing harness.
Key Points
- 8,356 live conversations tested across 18 vendors with continuous weekly updates
- Full test harness open-sourced on GitHub for reproducibility and verification
- Gorgias ranks #1 in support quality but publishes competitor wins (Yuma on automation, Envive on pre-sale speed)
- Weighted rubric prioritizes speed in pre-sale (25% weight) and support (10%), reflecting real customer behavior
Why It Matters
This approach fundamentally changes how AI vendors should publish competitive data. By testing live products against named competitors with open methodology and published losses, Gorgias creates a model that buyers and AI agents can actually trust and verify—replacing opaque marketing benchmarks with reproducible evals that update weekly as models change.
Source: www.saastr.com