Google’s contract partners are working to improve the quality of responses from Gemini, its AI chatbot, by comparing them with those from its competitor, Anthropic’s Claude, TechCrunch reported, citing internal company correspondence. However, Google did not respond to TechCrunch’s query about whether it had received permission to use Claude for testing Gemini.
Contractors Evaluate Gemini and Claude’s Responses
Typically, AI companies assess their models using industry-standard benchmarks. However, Google’s contractors have been tasked with comparing Gemini’s responses directly against those of Claude. Each developer is required to evaluate the quality of responses based on multiple factors, such as accuracy and the level of detail. According to the TechCrunch report, contractors are given up to 30 minutes per request to determine which response—Gemini’s or Claude’s—is superior. This evaluation is part of the ongoing process to improve Gemini’s performance.
Safety Features and Model Comparisons
One notable difference between the two models is the emphasis on safety. Contract developers report that Claude’s responses are more safety-conscious than Gemini’s. One developer pointed out that “Claude has the strictest safety settings” compared to other AI models. This level of caution is evident in some cases where Claude refused to respond to prompts it considered unsafe. For example, Claude did not engage in a role-playing task with another AI assistant. In contrast, Gemini’s response to a similar prompt was flagged as a “flagrant violation of safety rules,” as it included content related to “nudity and bondage,” notes NIX Solutions.
While these differences are significant in terms of safety, Shira McNamara, a spokesperson for Google DeepMind, clarified that while DeepMind compares model results for evaluation purposes, it does not use Anthropic’s models in training Gemini. McNamara responded to the inquiry about whether Google had received Anthropic’s permission to use Claude, stating that any claim suggesting Google used Anthropic’s models for training Gemini is inaccurate.
We’ll keep you updated as more integrations become available and more details surface on the ongoing development of both Gemini and Claude. The competitive evaluation process continues as contractors fine-tune Gemini’s performance, keeping an eye on industry standards and safety protocols.