Visa has launched an artificial intelligence shopping benchmark to assess how effectively AI models can handle purchases on behalf of consumers, with the first round showing that the models failed 42.8 per cent of their test runs.
The Visa Agentic Commerce Benchmark evaluated 22 AI models from 12 research laboratories across 15 shopping tasks, according to a statement released by the payments company on Thursday.
The tests involved everyday purchases, including televisions, laptops and personal care appliances. Rather than providing all their requirements upfront, consumers gave the AI models incomplete information, requiring them to ask questions and establish their preferences before recommending products.
The models were also expected to recognise when available products did not meet the customer’s requirements and decline to make a purchase.
To prevent successful outcomes based on guesswork, they had to identify sufficient information to justify their recommendations.
Visa said the benchmark was the first of its kind, designed to measure AI agents’ ability to make purchasing decisions accurately and responsibly.
To pass a test, an AI model must identify every requirement specified in the task and make the correct decision on whether to complete or decline the purchase.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, global head of growth, Visa.
The AI models achieved an overall pass rate of 57.2 per cent across the tests, with the best-performing model recording 82.2 per cent and the weakest scoring 28.9 per cent.
Although Visa did not identify the models in its statement.
The results also highlighted the importance of asking customers the right questions.
The five models that identified the highest number of hidden purchasing requirements achieved a combined pass rate of 70.2 per cent, compared with 44.4 per cent for the five that uncovered the fewest.
Cost also appeared to influence performance. The five most expensive models per test run recorded a 67.1 per cent pass rate, compared with 50.2 per cent for the five cheapest.
