Anthropic’s latest AI feature “Computer Use” can demonstrate remarkable capabilities in interacting with computers, according to a recent hands-on experience by Thariq Shihipar. While the system is still slow, unreliable, and prone to taking over the user’s computer, it represents an exciting step forward in AI-driven computer interaction, write Shihipar. Claude Computer Use excels at screen reading, navigation, function calls, and step-by-step task execution, but struggles with knowing when to read the screen, fetching additional data, remembering state, and navigating modals and popups.
To optimize performance, the author suggests providing the AI with as much system state information as possible, minimizing its reliance on vision, and developing a way for the AI to handle uncertainty. Despite its limitations, Claude Computer Use is seen as a significant milestone in the development of true AI agent behavior, with the potential for further advancements in the future.