APIEval-20 is a task benchmark for evaluating AI agents on real-world API testing. It includes 20 scenarios across 7 domains, measuring bug-finding capability from schema and payload alone, with a public dataset on Hugging Face.
Free
How to use APIEval-20?
APIEval-20 provides a structured benchmark to assess AI agents' ability to detect API bugs using only schema and payload information. Developers and researchers can use it to test their AI models' performance across 20 scenarios in 7 domains, identifying weaknesses and improving testing automation.
APIEval-20 's Core Features
20 diverse scenarios across 7 domains covering real-world API testing challenges, from authentication to data validation.
Measures bug-finding capability using only schema and payload, simulating real-world testing conditions.
Public dataset available on Hugging Face for easy access and reproducibility.
Designed to evaluate AI agents, not just humans, making it ideal for AI-native engineering teams.
Includes 98 planted bugs to ensure rigorous and standardized evaluation of bug detection.
APIEval-20 's Use Cases
AI researchers use it to benchmark their models' API testing capabilities.
Engineering teams evaluate AI tools for automated bug detection in APIs.
QA teams improve testing pipelines by identifying AI agent weaknesses.
DevOps engineers integrate it into CI/CD to assess AI testing performance.
Students and educators use it to teach AI-driven software testing concepts.
