Google AI Mode Update Introduces Multimodal Features

Google is unveiling a significant update to its AI Mode chatbot, which now incorporates advanced multimodal features enabling it to analyse and respond to inquiries about images. This innovative capability merges the power of Gemini AI with Lens image recognition and is now accessible to millions more users across the US via the Labs programme.

Enhanced Image Interaction with AI Mode

With this latest update, users can easily take or upload a photo and inquire about the elements present in the image. Google states that AI Mode will deliver a “rich, comprehensive response with links” for further exploration, all fuelled by a tailored version of Gemini designed specifically for search functionalities.

Understanding Visual Context with AI

The update builds on Google’s longstanding expertise in visual search, taking it to an advanced level, according to Robby Stein, VP of product for Google Search. With the enhanced multimodal capabilities of Gemini, AI Mode can interpret entire scenes within images, including the contextual relationships between objects and their specific materials, colours, shapes, and arrangements.

Innovative Search Techniques

This feature employs what Google refers to as a “fan-out technique,” which generates multiple simultaneous queries regarding both the overall image and the specific objects within it. The company highlights that this method produces outcomes that are more refined and contextually relevant compared to traditional search. For example, the AI can recognise books depicted in a photo and suggest similar titles that have garnered positive reviews.

User Engagement and Practical Applications

Google has reported impressive engagement from the initial users of AI Mode, who are asking longer and more intricate questions than those typically encountered in standard Search. Users are increasingly utilising this tool for various tasks, such as comparing products, organising trips, and exploring how-to guides.

Broader Access to AI Mode

Originally available exclusively to Google One AI Premium subscribers through the Labs programme, this update marks an expansion in access to a broader audience in the US. However, Google has not yet specified when this feature will be accessible to users in other global regions.

The multimodal update is currently live and can be utilised through the Google app on both Android and iOS devices.