Meta AI researchers have developed a new method called Coconut (Chain of Continuous Thought) that allows large language models to reason in continuous latent space rather than only through words. The research presents an alternative to traditional Chain-of-Thought (CoT) reasoning methods.
The new approach enables AI models to process information in a more abstract way, similar to how human brains work during problem-solving tasks. While conventional language models must express their reasoning process through words, Coconut alternates between language mode and latent thought mode, allowing for more flexible problem-solving.
The researchers demonstrated that Coconut performs particularly well on complex planning tasks. In tests comparing it to traditional methods, Coconut showed significant improvements on the ProsQA dataset, which requires advanced planning capabilities. The method also proved competitive on mathematical reasoning (GSM8K) and logical reasoning (ProntoQA) tasks.
A key innovation of Coconut is its multi-stage training procedure, which gradually teaches the model to reason in continuous space. The process begins with standard language-based reasoning and progressively replaces verbal reasoning steps with latent thought tokens.
The study revealed that Coconut can develop reasoning patterns similar to breadth-first search (BFS), allowing it to explore multiple solution paths simultaneously before committing to an answer. This capability proved particularly valuable in complex reasoning scenarios where traditional methods might fail.
Meta’s researchers noted that the new method requires fewer tokens than conventional Chain-of-Thought approaches, potentially making it more efficient in practical applications. The team suggests that future research could explore combining latent thoughts with traditional reasoning methods or developing models pre-trained specifically for continuous thought processing.