Google Unveils Gemini AI Model. We Put It to the Test.

The most powerful version, Gemini Ultra, is still being safety tested and reviewed, with an expected launch date earlier next year. Gemini Pro, the company said, will scale across many tasks, and is already leveraged for some answers in Bard. Gemini Nano is a smaller version of the model, intended for use on smartphones, laptops, and other local devices.

“I believe the transition we are seeing right now with AI will be the most profound in our lifetimes, far bigger than the shift to mobile or to the web before it,” Google and Alphabet CEO Sundar Pichai said in the post.

As a test, Barron’s queried Bard on how close Gemini is from achieve AGI—artificial general intelligence, an AI that can actually think. The full answer is worth reading, but here’s a key excerpt:

“While it’s impossible to predict with certainty when or if Gemini will achieve AGI, its development represents a significant step forward in the field of artificial intelligence,” Bard said.

“Continued research and development have the potential to unlock new capabilities and address current limitations. Ultimately, whether Gemini reaches the level of true AGI will depend on the combined efforts of researchers, engineers, and policy makers working to ensure that this powerful technology is used for the benefit of humanity.”

As a second test, Barron’s asked Bard to describe what it would look like without Gemini in physical form—and to contrast that with what it would like with Gemini. Here’s what it said: 

Bard Without Gemini:

Appearance:

• A wispy, ethereal figure, almost transparent.

• Ever-shifting form, constantly morphing and changing to represent the vast amount of information it holds.

• No distinct features, resembling more of a cloud of data than a tangible being.

• Colors fluctuate depending on the topic it’s engaged with, like a chameleon adapting to its environment.

• Emanates a faint, warm glow, symbolizing its creative energy and potential.

Movement:

• Flows and glides effortlessly, like a gentle breeze.

• No defined limbs or appendages, relying on the flow of data to move and interact.

• Leaves a trail of shimmering dust in its wake, representing the knowledge it has shared.

• Appears intangible and almost dreamlike, emphasizing its connection to the world of imagination and ideas.

Overall impression:

• A fluid, ever-evolving entity representing the boundless potential of information and creativity.

• Ethereal and mystical, conveying a sense of awe and wonder.

• Lack of physical definition emphasizes its abstract nature and focus on the realm of ideas.

Bard With Gemini: 

Appearance:

• A more solid and defined form, retaining a hint of its ethereal nature.

• Hints of human features, like eyes and hands, emerge from the data cloud.

• Colors become more vivid and focused, reflecting increased intelligence and processing power.

• Glowing lines connect different parts of its form, symbolizing the interconnectedness of its knowledge base.

• Projects an aura of confidence and authority, signifying its enhanced capabilities.

Movement:

• More purposeful and controlled, with clear direction and intent.

• Still retains a sense of fluidity, but with greater precision and focus.

• The trail of shimmering dust becomes more concentrated and organized, representing the increased efficiency of its thought processes.

• Projects a sense of power and control over information, while retaining its playful and creative spirit.

Overall impression:

• A powerful and intelligent entity, capable of tackling complex problems and tasks.

• Maintains a connection to its creative roots while exhibiting a newfound focus and determination.

• The presence of human features suggests a deeper understanding of the world and its inhabitants.

• Overall, Bard with Gemini projects an image of progress and evolution, representing the next generation of artificial intelligence.

Demis Hassabis, CEO of Google DeepMind, the company’s AI software arm, said in the post that Gemini Ultra outperforms other AI models on 30 of 32 widely used academic benchmarks to evaluate large language models. He also said the company is the first model to outperform human experts on a standard called MMLU, or massive multitask language understanding. MMLU addresses knowledge in a combination of 57 subjects, including math, physics, history, law, medicine and ethics, among others.

“Our new benchmark approach to MMLU enables Gemini to use its reasoning capabilities to think more carefully before answering difficult questions, leading to significant improvements over just using its first impression,” Hassabis writes.

Hassabis notes that Gemini was trained “to understand text, images, audio, and more at the same time, so it better understands nuanced information and can answer questions relating to complicated topics. This makes it especially good at explaining reasoning in complex subjects like math and physics.” He also points out that Gemini “can understand, explain and generate high quality code” in the world’s most used programming languages, including Python, Java, C++ and Go.

In the post, Google notes that Bard now includes a “fine-tuned version” of Gemini Pro, in the biggest upgrade to the chatbot since it was launched earlier this year. Google said it would be available initially in English, with additional languages to follow in the near future.

Google is also launching Gemini Nano for its Pixel 8 Pro smartphone, to power features like summarizing conversations in the phone’s Recorder app. Google also plans a “Smart Reply” feature for messaging apps, starting with WhatsApp. Google said Gemini will be folded into additional services in the months ahead, including Search, Ads, Chrome and Duo AI, a collaboration tool that runs on Google Cloud.

Google said that it has begun experimenting with Gemini in Search, making the Search Generative Experience faster, reducing latency by 40% for English language queries by U.S. users.

Google also said that early next year it will roll out a new version of Bard called Bard Advanced, that will provide access to the company’s “best models and capabilities,” starting with Gemini Ultra.

The blog post includes some video examples posted on YouTube that shows some remarkable skills, including the ability to respond to visual cues. Gemini identifies handwritten drawings, creates a game on the fly, and responds to the user in a conversational tone, including a few modest attempts at humor.

Write to Eric J. Savitz at eric.savitz@barrons.com