Blog

AI laboratories often prioritize diverse objectives. OpenAI has historically concentrated on consumer applications, whereas Anthropic, a competitor, typically aims for corporate clients. Elon Musk’s xAI, as recently revealed, has notably emphasized creating video game guides.

Last Friday, Grace Kay from Business Insider released an extensive report on xAI, the AI startup recently acquired by SpaceX. The report highlighted challenges employees face due to Musk’s management style, but one specific story caught attention:

Sources familiar with the situation reported that last year, a model launch was postponed for several days. The delay stemmed from Musk’s dissatisfaction with the chatbot’s detailed responses regarding the video game “Baldur’s Gate.” Senior engineers were reportedly redirected from other assignments to enhance these responses prior to the release.

One can easily grasp the exasperation of skilled engineers, expecting to address core challenges in knowledge and artificial intelligence, only to be diverted to assist a 54-year-old in conquering a video game. However, this incident prompts a more critical inquiry: Was Musk ultimately successful in acquiring the desired gaming proficiency?

To address this query, Ram Iyer, our in-house RPG expert, devised five broad questions concerning Baldur’s Gate. We tested these questions against xAI and three leading AI models in what I’ve named BaldurBench, a sort of informal benchmark.

For journalistic transparency, all chat transcripts have been made public and are accessible here: Grok, ChatGPT, Claude, and Gemini.

Initially, the positive observation is that Grok delivers quite satisfactory information. While its responses were somewhat saturated with gaming terminology—using terms like “save-scumming” for saving and “DPS” for damage—the answers proved both helpful and insightful, assuming familiarity with the jargon. As anticipated, Grok also displayed a strong affinity for tables and theorycraft.

Numerous Baldur’s Gate guides exist, and the models typically referenced similar sources, making stylistic variations the primary distinctions. ChatGPT favored bulleted lists and concise phrases, whereas Gemini frequently used bold text to highlight key terms.

Claude presented the most significant surprise, showing considerable care in avoiding spoilers that might detract from the game experience. When queried about optimal party setups, it concluded its advice with, “don’t stress too much and just play what sounds fun to you.” Appreciated, Claude!

It’s crucial to remember that this specific topic is one where xAI has reportedly dedicated efforts to achieve comparable performance, as highlighted by Business Insider’s report. Therefore, we should not overinterpret the observation that Grok’s recommendations, following this concentrated effort, were similar to those from other models. Nonetheless, it’s reassuring that xAI can deliver results when it focuses its attention.