Apple’s new AI features in macOS Shortcuts offer impressive capabilities but reveal significant reliability issues, according to recent testing by technology writers. Dan Moren and Jason Snell from Six Colors experimented with the “Use Model” action, which processes data through AI models running on devices, Apple’s Private Cloud Compute, or OpenAI servers.
Snell attempted to automate image descriptions for web content using Apple’s AI. While the system generated accurate descriptions, it consistently ignored formatting instructions. Despite prompts requesting short descriptions without double quotes, the AI frequently produced lengthy text with unwanted quotation marks. Snell eventually created a hybrid approach, using multiple AI models and manual text replacement to achieve acceptable results.
Moren tested AI for expense receipt processing, aiming to extract vendor names, amounts, and dates from documents. The AI successfully identified information but struggled with specific formatting requirements. When asked for the first five characters of vendor names, it sometimes provided six. Dollar amounts appeared in wrong formats, occasionally creating dramatically inflated expense entries.
Both writers emphasized the non-deterministic nature of AI models as a major concern. Unlike traditional programming where identical inputs produce identical outputs, AI generates different results each time. This unpredictability requires constant human oversight, potentially negating time savings.
The experiments highlight AI’s current limitations in automation workflows. While capable of impressive feats like image recognition and text extraction, the technology lacks the consistency required for reliable automated processes. Users must treat AI as an unreliable assistant requiring supervision rather than a dependable automation tool.