AI Is Coming for the Marketing Department

When a 99-point wine appears at $40, you pay attention.” That’s how Anthropic’s Claude opened a promotional email for customers of online wine retailer Wine Access.

The company’s writers took a different approach with “Charles Smith’s Motor City Kitty Syrah is one of the must-have red wines of the West Coast.”

In a field experiment, these two emails went out to separate, randomly assigned groups of customers, and both boosted wine purchases. But the first approach cost a fraction of the second. “Once the annual overhead costs are considered, there is scope for WA [Wine Access] to down-size its writing teams and shift some of the creative content design to AI,” conclude Chicago Booth’s Jean-Pierre Dubé and Booth PhD student Ariel Xu.

Over the course of a year, Dubé and Xu conducted three randomized controlled trials comparing the effectiveness of AI- and human-written marketing emails. Each ran for two weeks and drew on the 27,500 customers subscribed to Wine Access’s twice-daily email newsletter; the newsletter spotlights a single bottle of wine and has been shown via A/B testing to roughly double the gross profits generated by that bottle relative to other bottles available through Wine Access.  

Each trial randomly assigned customers to one of four groups. A control group received no newsletter. A second group received newsletters written by Wine Access’s three salaried writers, who each spent roughly three hours composing a 200–600-word email that drew on information provided by Wine Access about standard wine attributes such as vintage, varietal, region, tasting notes, and sweetness. A third group of customers received newsletters generated by an AI model that had access to the same information that Wine Access gave its writers on the wine’s attributes. In two of the trials, a custom-built AI model had been trained on five years of the company’s best-performing emails. In the third trial, the researchers used Claude and only a simple prompt. Finally, the fourth group of customers received emails that had been drafted by AI and edited by a person.

Across all groups, the researchers tracked purchase probability, bottles sold, and revenue and gross profits from the sales. Profits for the three groups receiving the email far outpaced the control group and were generally indistinguishable from one another. “This result is striking,” the researchers write, “as it suggests the direct and hybrid implementations of the LLMs are generating email copy that is approximately as effective as the team of human writers.”

But when the researchers prompted Claude instead of using the model that they had pretrained, the AI significantly edged out staff writers on gross profits for those wines in the newsletters. Projected annual profits on the bottles mentioned by the writers were about $5 million versus roughly $5.5 million for bottles mentioned by Claude.

This outperformance was partially due to the quality of Claude’s copywriting, according to the study. But equally important were the cuts to overhead costs enabled by AI. Wine Access’s three writers earn a combined $375,000 in annual salary and benefits, according to the report. Fully automating their work with the trained model would cut those costs by about 90 percent, while using Claude to automate their work would cut costs by 83 percent. The company would save less by implementing a hybrid approach in which the AI drafts emails that a human edits, the study suggests—those cost savings would be 83 percent with the trained model and 75 percent with Claude, the researchers calculate.

Notably, neither customers nor uninvolved employees appeared to notice anything unusual during the trials—they did not submit complaints or inquiries—suggesting the AI successfully adopted the company’s voice. The findings indicate that Wine Access could radically shrink its writing team without sacrificing results.

While this work examined one narrow slice of marketing, the copy contained within a single recurring email campaign, the implications are broader. “We believe the results should be useful for other marketing practitioners and researchers seeking ways in which AI can automate daily marketing functions,” write Dubé and Xu. Looking ahead, they envision LLMs automating a much wider range of digital content, raising the possibility that AI could, eventually, handle the full arc of a company’s written marketing output.