Composio Tests AI Coding Agents with DeepSeek V4 Flash Model

AI integration platform Composio conducted tests on August 6th, evaluating four AI programming agent frameworks to see which could best utilize the performance of the official version of the DeepSeek V4 Flash model. The evaluation involved 30 tasks across real-world tools such as Gmail, GitHub, Slack, and Notion.

The "Oh My Pi" framework achieved the highest success rate, completing 17 out of 30 tasks. However, it was also the slowest, with each task taking an average of 272 seconds. In contrast, "OpenCode" proved to be the most cost-effective, with a cost of only $0.073 per successful task.

The "Claude Code" framework was the fastest, completing tasks in an average of 122 seconds. It was also the most expensive, costing $0.195 per task. Despite using fewer tool calls and generating fewer output tokens, "Claude Code" demonstrated strong performance relative to the other tested frameworks.

Composio, founded in 2023 and based in San Francisco, provides an open-source integration platform for AI agents. The company aims to connect large language models with various external applications, enabling developers to rapidly build production-ready AI agents.