Google Unveils New AI Features in Gemini Live Platform

Google has started to implement groundbreaking artificial intelligence features within its Gemini Live platform, enabling it to visually interpret smartphone screens and camera feeds in real-time. These advancements, confirmed by Google spokesperson Alex Joseph, are part of the company’s extensive AI initiative referred to as “Project Astra.”

Innovative Capabilities of Gemini Live

The recently launched features include screen-reading and live video interpretation, which empower Gemini Live to respond to user inquiries regarding what is visible on their phone screen or through their camera lens. This rollout is exclusively available to Gemini Advanced Subscribers enrolled in the Google One AI Premium plan, with access progressively expanding throughout the month.

According to Joseph, the screen-reading functionality allows users to query Gemini about any content displayed on their smartphone screen, delivering contextual responses. In addition, the live video capability utilises a smartphone’s camera to conduct real-time analysis of whatever is being viewed. For example, users can request Gemini to identify objects, propose aesthetic options, or assist them with tasks such as selecting a paint colour for newly-glazed pottery.

Public Demonstration and Competitive Landscape

The initial public demonstration of the screen-reading feature was noted by a Reddit user utilizing a Xiaomi device, later validated by 9to5Google. In a video provided by the user, Gemini adeptly read the phone’s screen content and accurately addressed inquiries concerning it.

Google’s introduction of these features comes at a time when its competitors are racing to keep pace. Amazon is set to launch its Alexa Plus upgrade featuring comparable functionalities but is currently in early access. Meanwhile, Apple has postponed the launch of its revamped Siri, which is also anticipated to include advanced AI capabilities.

Samsung continues to depend on its Bixby assistant, yet Gemini’s smooth integration within its smartphones offers Google a unique edge.

First hinted at almost a year ago, Project Astra embodies Google’s ambition to reshape the possibilities of what digital assistants can achieve. By merging visual analysis with natural language processing, Google intends to deliver a more engaging and intuitive AI experience.