Powerful New Computer Use Feature in Microsoft Copilot Studio

Microsoft has unveiled a robust new capability within Copilot Studio, known as computer use. This exciting enhancement allows AI agents to independently engage with websites and desktop applications, replicating human actions such as clicking buttons, entering information, and navigating through menus. Currently available in research preview for a select group of users, this functionality enables organisations to develop intelligent agents capable of executing intricate tasks across browser and desktop environments, even when APIs are not accessible.

Natural Language Task Description

With the computer use feature, users can articulate the tasks they wish for the agent to perform simply by using natural language. The agent then emulates the action, facilitating testing and adjustment prior to launch. After setup, these AI agents are prepared to automate workflows in various web browsers, including Microsoft Edge, Google Chrome, and Mozilla Firefox, in addition to native desktop applications.

Seamless Interaction with Applications

Charles Lamanna, Corporate Vice President of Microsoft’s Business and Industry Copilot, stated that if an individual can operate the application, the agent can do so as well. Computer use empowers agents to engage with websites and desktop applications by executing actions such as clicking buttons, making selections from menus, and typing into fields visible on the screen.

Automation of Real-World Business Tasks

Developed to address genuine business needs, this new feature supports task automation across a variety of functions, such as extensive data entry, conducting market research, and processing invoices. Microsoft has shown how enterprises can utilise this tool to extract data from diverse sources into centralised systems, enhancing operational efficiency while reducing errors.

Autonomous Adaptation to Changes

In contrast to some existing agent solutions that need human intervention for interface alterations or CAPTCHA challenges, Microsoft’s computer use is equipped with integrated reasoning capabilities. These enhancements allow AI agents to autonomously adjust when screen elements vary, ensuring that tasks proceed without interruptions. Users are also provided with a comprehensive activity history, complete with screenshots and reasoning logs to ensure transparency and oversight.

Focus on Security and Privacy

Security and privacy are paramount concerns for Microsoft. The company has verified that enterprise data remains securely within the boundaries of Microsoft Cloud and will not be employed to train its Frontier models. Furthermore, as the feature operates entirely on Microsoft-hosted infrastructure, organisations can take advantage without the challenge of managing their own servers, accelerating deployment while decreasing maintenance and infrastructure expenses.

Early access to the computer use feature has been rolled out for Copilot Studio users, with a wider release anticipated in the near future.