Microsoft Copilot Preview Tracks User Browsing

Your screen, or more specifically, the websites you are seeing, can now be read by Microsoft’s AI.

On Thursday, Microsoft unveiled a new Copilot artificial intelligence (AI) feature. Now known as Copilot Vision, it allows the AI chatbot to perceive and comprehend the context of a user’s online activity. According to the software behemoth based in Redmond, Copilot has access to the Microsoft Edge browser. Now, the vision-enabled chatbot can keep track of the websites and pages the user visits and, when instructed, can carry out certain actions. A limited number of Copilot Pro subscribers in the US are presently allowed to preview this feature.

To use Copilot Labs, you must have a $20/month membership to Microsoft’s Copilot Pro plan.

In addition to providing answers, Copilot Vision can perform activities like highlighting sales in a store catalog and translate and summarize content. It can also act as a game assistance, giving hints during Chess.com matches, for instance.

The Windows manufacturer has been working to elevate the PC experience with its AI-powered chatbot Copilot. According to the blog post, it previously introduced specially designed AI agents to perform activities on the PC. Now, it has given Copilot an additional feature that will enable it to respond to user inquiries based on their browser-related activity. Notably, the AI cannot act on behalf of the user because Copilot Vision is not an agentic function.

With the Edge browser, Copilot Vision is able to comprehend the entire context of what a user sees. It is able to read the text, view the images and objects, and even understand the navigation buttons. Users can either type or voice the prompt to the vision-enabled chatbot, which will be located at the bottom of the screen.

When asked to do so, the AI can highlight certain areas of the page, summarize and translate information, and respond to queries. According to Microsoft, Copilot Vision may also serve as a game assistant, providing guidance on how to play games like GeoGuessr and advice while playing online chess.

Microsoft stated in the blog post shared that “when you choose to enable Copilot Vision, it sees the page you’re on, it reads along with you, and you can talk through the problem you’re facing together.” Nestled neatly at the bottom of your Edge browser, it’s a new method to allow AI to accompany you on your web navigation and assist you anytime you need it.

Giving examples, the IT titan claimed that Copilot Vision may assist customers in organizing a trip to the museum by highlighting crucial details. By suggesting appropriate products based on the user’s requirements and preferences, it can also assist consumers with their purchasing.

According to Microsoft, Copilot Vision would be fully opt-in in terms of privacy and security, allowing users to choose when to activate and deactivate it. Anything that the user does not want to view will not be shown to the AI. Furthermore, at the conclusion of each session, all discussion data with the AI will be erased. Nonetheless, the Copilot’s answers are recorded and will be utilized to enhance the business’s security measures.

Microsoft went on to say that Copilot Vision doesn’t collect, save, or utilize any publisher data for AI model training. To increase the new feature’s safety, it is also collaborating with a team of outside testers known as “red teamers.” It’s emphasizing that Copilot Vision removes data after each session, presumably in an effort to prevent further negative publicity from AI privacy gaffes. The company claims that, at least in this preview release, processed text, audio, or images are not saved or utilized to train models.

Additionally, the kinds of websites that Copilot Vision can visit are restricted. Microsoft is now restricting Vision to a pre-approved list of “popular” websites and preventing the functionality from functioning on paywalled and “sensitive” content.

Copilot Labs provides access to Copilot Vision, which is presently being made available to a select group of Copilot Pro subscribers in the US. Notably, Vision will only function on a limited number of websites at this time, with plans to further expand its usefulness.

What precisely is “sensitive”? Porn? Horrible violence? Microsoft refused to say.

A Microsoft representative stated, “The list of permitted websites is decided by category and on a case-by-case basis.” “We will be adding more sites over time, starting with a small list of sites that we have thoroughly tested.”

A portion of Microsoft’s cautious attitude stems from legal fights with news organizations. The New York Times claims in one pending lawsuit that Microsoft allowed users to circumvent its paywall by delivering NY Times stories via the Bing Copilot chatbot.

In addition to being concerned that their data may be misused, many large publishers have chosen to stop AI technologies from crawling their websites in order to save their server expenses from skyrocketing. According to Microsoft, Copilot Vision would adhere to websites’ “machine-readable controls on AI,” such as those that forbid bots from using data scraping for AI training. The business hasn’t said which of the various controls in use Vision will adhere to, though.

Some of the most popular news websites on the internet might not support Copilot Vision if the current trend continues. Microsoft, however, stated that it is dedicated to “taking feedback” in order to ease publishers’ worries.

Some of the most popular news websites on the internet might not support Copilot Vision if the current trend continues. Microsoft, however, stated that it is dedicated to “taking feedback” in order to ease publishers’ worries.

According to the company’s blog post, “We have worked with third-party publishers, who help us understand how Vision could be used to help people better engage and make decisions on their pages.” “We will be able to design Vision to be more useful in its interactions with web pages thanks to these inputs.”