Can AI Comprehend Flowers Without Touching or Smelling Them?

The newest artificial intelligence models demonstrate a comprehension of the world akin to human understanding. Yet, their sensory limitations hinder their grasp of concepts like flowers and humor.

Qihui Xu from Ohio State and her team explored the understanding of nearly 4,500 words by both humans and large-scale language models (LLMs), covering terms such as “flowers,” “hooves,” “humorous,” and “swings.” Both human participants and AI models evaluated these words based on emotional arousal and physical interactions associated with various body parts.

The objective was to analyze how LLMs, such as OpenAI’s GPT-3.5 and GPT-4, along with Google’s Palm and Gemini, compared with human rankings. While both humans and AI exhibited similar concept maps for words unrelated to sensory interaction, substantial discrepancies arose when it involved physical sensations and actions.

For instance, AI models often suggested that flowers could be perceived through the torso, a notion that most people find peculiar, as they typically enjoy flowers visually or through scent.

The challenge lies in the fact that LLMs develop their understanding from a vast array of text sourced from the internet, which falls short in tackling sensual concepts. “They are fundamentally different from humans,” she explains.

Certain AI models have undergone training using visual data like images and videos alongside text. Researchers have noticed that these models yield results more closely aligned with human evaluations, enhancing the chances that future AI will bridge sensory understanding with human cognition.

“This illustrates that the advantages of multimodal training might surpass expectations. In reality, it seems that one plus one can yield two or more,” states Xu. “In terms of AI advancement, this underscores the significance of developing multimodal models and the necessity of embodying these models.”

Philip Feldman at the University of Maryland in Baltimore County suggests that simulating an AI with a robotic body, exposed to sensorimotor experiences, could greatly enhance its capabilities, but he cautions about the inherent risks of physical harm to others.

Preventing such dangers requires implementing safeguards in robotic actions or opting for softer robots to avoid causing injury during training, warns Feldman, although this approach has its downsides.

“This may distort their perception of the world,” Feldman remarks. “One lesson they might learn is that they can gently bounce objects. [In a real robot with mass] The humanoid robots might believe they can collide with one another at full speed. That could lead to serious issues.”