Chatbots Perform Best When Communicating in Formal Language.

The manner in which you converse with an AI chatbot, especially using informal language, can significantly impact the accuracy of its replies. This indicates that we might need to engage with chatbots more formally or train the AI to handle informal dialogue better.

Researchers Fulei Zhang and Zhou Yu from Amazon explored how users begin chats with human representatives versus chatbot assistants that utilize large language models (LLMs). They employed the Claude 3.5 Sonnet model to evaluate various aspects of these interactions, discovering that exchanges with chatbots were marked by less grammatical accuracy and politeness compared to human-to-human dialogues, as well as a somewhat limited vocabulary.

The findings showed that human-to-human interactions were 14.5% more polite and formal, 5.3% more fluent, and 1.4% more lexically diverse than their chatbot counterparts, according to Claude’s assessments.

The authors noted in their study, “Participants adjust their linguistic style in human-LLM interactions, favoring shorter, more direct, less formal, and grammatically simpler messages,” though they did not respond to interview requests. “This behavior may stem from users’ mental models of LLM chatbots, particularly if they lack social nuance or sensitivity.”

However, embracing this informal style comes with challenges. In another evaluation, the researchers trained an AI model named Mistral 7B using 13,000 actual human-to-human interactions, then assessed 1,357 real messages directed at the AI chatbot. They categorized each conversation with an “intent” derived from a restricted framework summarizing the user’s purpose. Unfortunately, Mistral struggled with accurately defining the intentions within the chatbot conversations.

Zhang and Yu explored various methods to enhance Mistral AI’s understanding. Initially, they used Claude AI to transform users’ succinct messages into more polished human-like text and used these rewrites to fine-tune Mistral, resulting in a 1.9% decline in intent label accuracy from the baseline.

Next, they attempted a “minimal” rewrite with Claude, creating shorter and more direct phrases (e.g., asking about travel and lodging options for an upcoming trip with “Paris next month. Where’s the flight hotel?”). This method caused a 2.6% drop in Mistral’s accuracy. On the other hand, utilizing a more formal and varied style in “enhanced” rewrites also led to a 1.8% decrease in accuracy. Ultimately, the performance showed an improvement of 2.9% only when training Mistral with both minimal and enhanced rewrites.

Noah Jansiracusa, a professor at Bentley University in Massachusetts, expressed that while it’s expected that users communicate differently with bots than with other humans, this disparity shouldn’t necessarily be seen as a negative.

“The observation that people interact with chatbots differently from humans is often depicted as a drawback, but I believe it’s beneficial for users to recognize they’re engaging with a bot and adjust their communication accordingly,” Giansiracusa stated. “This understanding is healthier than a continual effort to bridge the gap between humans and bots.”