OpenAI is reportedly developing new strategies to deal with a slowdown in AI model improvements. According to The Information, OpenAI employees testing the company’s next flagship model, code-named Orion, found less improvement compared to the jump from GPT-3 to GPT-4, suggesting the rate of progress is diminishing. In response, OpenAI has formed a foundations team to investigate ways to continue enhancing models despite the dwindling supply of new training data.
Reuters reports that OpenAI and other AI companies are seeking to overcome unexpected delays and challenges in the pursuit of more advanced large language models. They are exploring training techniques that use more human-like ways for algorithms to “think,” such as the methods behind OpenAI’s recently released o1 model. These techniques could reshape the AI arms race and impact the types of resources AI companies demand, from energy to specific chips. Although the potential of this “chain of thought” technique has been shown to be limited.
Prominent AI scientists, including Ilya Sutskever, co-founder of OpenAI and Safe Superintelligence (SSI), are acknowledging the limitations of the “bigger is better” approach. Researchers are now focusing on “test-time compute,” a technique that enhances existing AI models during the “inference” phase, allowing models to dedicate more processing power to challenging tasks that require human-like reasoning. The implications of this shift could alter the competitive landscape for AI hardware, potentially moving from a world of massive pre-training clusters to inference clouds.