Artificial Intelligence is becoming increasingly involved in the online shopping process, from product discovery and price comparison to selecting models that meet specific consumer needs. However, new research from Product.ai suggests that relying on these tools for purchases still faces significant limitations.
Key Research Findings
The study examined 8,794 responses generated by testing both free and paid versions of ChatGPT, Gemini, Claude, and Perplexity. Researchers submitted 220 questions regarding products such as laptops, televisions, mattresses, sunscreens, and robot vacuums, repeating each query five times to evaluate consistency.
- 86% of the queries resulted in conflicting factual information across repeated answers from the same tool.
- 97% of questions requesting a direct comparison between two products showed such inconsistencies.
- 75% of queries regarding specific product features or details yielded differing information.
The Price Accuracy Gap
Pricing remains a particular challenge. While 85% of verifiable price-related answers were accurate, the errors were substantial. When AI models provided incorrect pricing, the median deviation reached $300.
Performance varied notably between platforms and subscription tiers. For Gemini, significant pricing errors appeared in 56% of free responses and 54% of paid ones. In contrast, Claude saw its error rate drop from 44% to 21% when moving to the paid version. Perplexity's paid version had a 14% error rate, while ChatGPT's was 17%. Additionally, the free version of Gemini provided contradictory answers for 29% of the questions.
Consumer Guidance
Product.ai suggests that consumers should view AI more as a conversational search tool rather than a final authority for purchasing. Cross-referencing AI suggestions with different tools or, preferably, the retailer's own website remains essential to ensure accuracy before completing a transaction.
