In the fast-paced world of artificial intelligence, LLMs (Large Language Models) have become indispensable tools for automating complex tasks, such as content writing. But can these models write texts that truly captivate and connect with the audience? To answer this question, we put three LLMs – Claude Sonnet 3.5, ChatGPT-4o, and Gemini 1.5 Pro – to the test, challenging them to create a welcome email for new clients of "Tess."
The Mission: Create an Email that Wins Hearts
The objective was simple: generate a welcome email that was informative, inviting, and, above all, human. To evaluate the performance of the LLMs, we analyzed the generated emails based on four essential criteria for each model:
The Prompt:
We used Tess AI's Multi-LLM chat and provided the following briefing for the models:
The Results? A True Battle of Styles!
Each model left its own mark on the welcome email, revealing its strengths and distinct styles:
Claude Sonnet 3.5:
Quality and Relevance of Content: Claude Sonnet 3.5’s email contained relevant information about the platform and its benefits but fell short in detailing how the user could start using the tool immediately.
Personalization and Tone of Voice: The tone was positive and friendly, but personalization was limited to the use of the client’s name.
Structure, Organization, and Humanized Writing: The structure was organized, but the writing included "marketing-like" phrases that felt somewhat unnatural.
Grammatical, Spelling, and Effectiveness: The email was free of grammatical and spelling errors, but the call to action was somewhat vague.
ChatGPT-4o:
Quality and Relevance of Content: ChatGPT-4o, in turn, followed a similar path, highlighting the benefits of the platform but also neglected to provide a practical guide for the new user.
Personalization and Tone of Voice: ChatGPT-4o also showed good performance in tone of voice, using receptive and enthusiastic language, but personalization was limited to the use of the client’s name.
Structure, Organization, and Humanized Writing: The structure was organized, but some sentences sounded "stiff" and lacked naturalness.
Grammatical, Spelling, and Effectiveness: The email was free of grammatical and spelling errors, and the call to action was more specific but could still be more directed.
Gemini 2.0 Flash:
Quality and Relevance of Content: Gemini 2.0 Flash stood out in this regard. The generated email presented relevant and targeted information to the user, addressing the benefits of the platform and including a clear call to action to access resources and the community.
Personalization and Tone of Voice: Gemini 2.0 Flash opted for a more professional tone (despite the excessive emojis) and less personal, prioritizing the presentation of the brand and its features. This approach, while not as close to the user, fits better with the platform’s proposal. However, the model failed to provide any personalization beyond using the client’s name.
Structure, Organization, and Humanized Writing: The email generated by Gemini 2.0 Flash excelled with a clear structure, more natural language, and avoided "marketing-like" phrases. However, the excessive use of emojis may have harmed the formality of the email, making it seem less professional.
Grammatical, Spelling, and Effectiveness: The email was free of grammatical and spelling errors, and the call to action was clear and specific, including links to resources and the community.
The Battle of the Giants: Who Takes the Trophy?
Gemini 2.0 Flash stood out for presenting a more natural and user-friendly language, with an organized structure and a clear and targeted call to action. However, the excessive use of emojis and the lack of personalization detracted from the formality of the email, making it less professional. Claude Sonnet 3.5 and ChatGPT-4o, while performing well, still exhibited “robotic vices” such as stiff content or exaggerated adjectives.
The Evolution of LLM Writing
This analysis shows us that LLMs are evolving quickly, but there is still a long way to go to achieve the natural and personalized writing of a human. The choice of the ideal LLM for your needs will depend on the type of text you need to generate and the target audience you wish to reach.
Remember, the ultimate goal is to build strong relationships with your customers, and humanized and authentic writing still plays a fundamental role in this process.