Copilot has become deeply integrated into the Microsoft ecosystem, with a dedicated keyboard key now available for the AI chatbot. It appears that much of its functionality may soon be accessible offline, with processing occurring on your laptop. However, this is not possible with the current generation of Intel Core Ultra chips.
During the recent Intel AI Summit in Taipei, executives from the chip manufacturer informed Tom’s Hardware that existing Intel Core Ultra chips fall short of the minimum requirements for running Copilot offline on your device. The key metric in question is Trillion Operations per Second (TOPS), which measures the speed at which the NPU can execute AI tasks. To run Copilot offline, an NPU with at least 40 TOPS is required, while current Intel Core Ultra chips only offer 10-15 TOPS.
One commonly used processor capable of meeting these requirements is the Qualcomm Snapdragon X Elite with 45 TOPS. Intel has indicated that their next generation of Core Ultra chips will meet the necessary specifications.
Reasons for Local Processing
Currently, Copilot’s AI capabilities rely on cloud-based processing, sending data to Microsoft servers for analysis and generating responses. The long-term goal is to bring some of this processing to the device itself to improve privacy, security, offline access, and cost efficiency.
Running AI tools like Copilot locally ensures that your device handles the processing, reducing the reliance on cloud services. However, achieving this without impacting user experience, battery life, or system performance is a challenge.
Intel’s focus on optimizing NPUs over GPUs aims to address battery consumption concerns while maintaining processing efficiency. Microsoft’s preference for running Copilot on NPUs reflects their desire to minimize battery drain during AI operations.
In contrast to personal AI models like Meta’s Llama 2, mainstream applications like Copilot and future versions of digital assistants need to operate smoothly without causing system freezes or interruptions.
Understanding TOPS
TOPS, or trillions of operations per second, measure a chip’s computational performance in AI and machine learning tasks. Higher TOPS values signify greater processing capabilities, with NPUs playing a crucial role in enhancing AI performance.
While a combined CPU and NPU TOPS figure is common, Intel emphasizes the importance of a high NPU TOPS count for optimal performance in language understanding and generation tasks.
Intel aims to reach 40 TOPS for NPUs in future chips to enable more local processing. While some Copilot features may still require cloud services, the trend is to shift AI processes to on-device endpoints for enhanced efficiency.
Future of AI on Laptops
Several AI applications already utilize local processing, particularly image and video editors utilizing GPUs. Intel’s collaboration with developers aims to leverage NPUs more effectively, though significant advancements may require next-generation chips.
In the interim, Qualcomm leads the Windows AI market with its Snapdragon X Elite chips offering 45 TOPS for onboard NPUs. Intel’s low TOPS count in current chips may limit AI capabilities until new hardware is introduced.
Qualcomm’s CFO highlighted the limitations of low TOPS counts and emphasized the importance of higher values for a superior AI user experience. Despite Apple’s competitive TOPS performance, optimization between hardware and software enhances processing efficiency.
While current laptops can run AI using GPUs, future advancements in hardware and software are expected to make local AI processing commonplace. The focus on NPUs and TOPS highlights the industry’s shift towards efficient AI computation without compromising battery life.
More from Tom’s Guide
