Microsoft’s Computer Use Feature in Copilot Studio

Microsoft has unveiled a remarkable feature within Copilot Studio known as computer use, which empowers AI agents to independently engage with websites and desktop applications. This innovation allows these agents to replicate human behaviours—such as clicking buttons, entering information into fields, and navigating menus—effectively enhancing user interactions. Currently, it is accessible in research preview mode for select users, enabling organisations to develop intelligent agents capable of executing intricate tasks across both browser and desktop environments, even in situations devoid of API capabilities.

How Computer Use Works

With computer use, users can articulate the desired task for the agent in everyday language. The AI agent then emulates the action, facilitating testing and adjustments prior to actual deployment. Once set up, these AI agents can automate processes in web browsers including Microsoft Edge, Google Chrome, and Mozilla Firefox, in addition to native desktop applications.

Intelligent Interaction

“If a person can use the app, the agent can too,” remarked Charles Lamanna, Corporate Vice President of Microsoft’s Business and Industry Copilot. The computer use feature equips agents to interact with both websites and desktop applications through actions like button clicking, menu selections, and data entry.

Applications for Business Operations

This innovative tool is tailored for real-world business applications, enabling automation for tasks such as large-scale data entry, conducting market research, and processing invoices. Microsoft has showcased how enterprises can utilise this feature to funnel data from diverse sources into centralised systems, optimizing operations and reducing the potential for errors.

Advanced Autonomous Features

Unlike some other agent tools requiring human oversight for interface changes or CAPTCHA challenges, Microsoft’s computer use comes equipped with inherent reasoning capabilities. These allow AI agents to autonomously adjust to any alterations in screen elements, ensuring that tasks proceed seamlessly. Users can also access a comprehensive activity history featuring screenshots and reasoning logs for transparency and accountability.

Security and Privacy Considerations

The aspect of security and privacy is paramount. Microsoft has assured that enterprise data remains confined within Microsoft Cloud parameters and will not be utilised for training its Frontier models. Furthermore, since the computer use function operates entirely on Microsoft-hosted infrastructure, organisations can enjoy benefits without the burden of managing their own servers, facilitating quicker deployments while lowering maintenance and infrastructure costs.

Availability of Computer Use Feature

Early access to the computer use feature is currently available for users of Copilot Studio, with a more extensive rollout anticipated in the near future.