Chatbot Arena, a benchmarking tool for AI models, has become very popular in the tech industry. As Kyle Wiggers reports on TechCrunch, companies like OpenAI and Google use the platform to test the performance of their chatbots. Millions of people visited the LMSYS website last year.
However, experts are critical of the validity of the benchmark. According to Yuchen Lin of the Allen Institute for AI, there is a lack of transparency regarding the skills tested. Also, the composition of the users rating the chatbots may not be representative. Mike Cook of Queen Mary University of London points out that Chatbot Arena provides relative ratings rather than empirical tests.
Despite these limitations, experts see the platform as a useful tool for gaining insight into the performance of AI models.