综合能力
GDPval-AA v2
真实职场任务:能否交付可用的文档、表格、幻灯片和图示
它测什么
GDPval-AA v2 是 Artificial Analysis 基于 OpenAI GDPval 数据集进行的独立评测。它使用公开 gold 集中的 220 个任务,覆盖 9 个行业、44 种职业,要求模型提交实际工作成果,而不只是回答问题。
比如,官方示例会让模型给一支巡演乐队制作一页舞台布置 PDF:把歌手、鼓手、音箱和话筒放对位置,再列清楚输入、输出线路。能说出“舞台图应该怎么画”还不够,模型需要真正做出文件,内容和排版都要满足需求。
所有模型通过 Stirrup Agent 框架工作,可以使用命令行、文件处理、网页搜索与浏览等工具,每个任务最多执行 250 轮。评估时隐藏模型身份,把同一任务的两份成果交给从三模型评审团中抽取的 LLM 裁判比较,再将这些对比结果汇总成 Elo 评分。
怎么看分数
Elo 越高,表示交付成果在这套任务的两两比较中更受评审偏好;人类专家成果的基准设为 1,000 分。比如 A 得 1,500 分、B 得 1,200 分,只能说明 A 在这一评测中的相对表现更强,不表示 A 的正确率是 150%,也不表示效率高 25%。Elo 可以低于 0,表中的 ± 数值是官方公布的 95% 置信区间,分数接近时不要把名次差当成确定差距。本页保留 2026 年 9 月 17 日核对到的全部 247 条成绩和官方模型配置;标有 Fallback 的条目包含备用模型机制,不能当成单一模型独立完成的结果。榜单会随新增比较而变化,不能直接与 v1、模型发布时的旧快照或 OpenAI 自己公布的 GDPval 胜率混用;它也不是 AA Intelligence Index 的综合分。
模型表现
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)AnthropicAgent · Stirrup | Stirrup | 1764± 19 |
|---|---|---|
Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback)AnthropicAgent · Stirrup | Stirrup | 1745± 19 |
Claude Opus 5 (Adaptive Reasoning, Max Effort)AnthropicAgent · Stirrup | Stirrup | 1735± 16 |
Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)AnthropicAgent · Stirrup | Stirrup | 1708± 15 |
Muse Spark 1.3 (max)MetaAgent · Stirrup | Stirrup | 1703± 19 |
Qwen3.8 Max (0902)AlibabaAgent · Stirrup | Stirrup | 1689± 21 |
Grok 4.6 (xhigh)SpaceXAIAgent · Stirrup | Stirrup | 1663± 15 |
Muse Spark 1.3 (xhigh)MetaAgent · Stirrup | Stirrup | 1662± 19 |
GLM 5.3 FlashZ AIAgent · Stirrup | Stirrup | 1655± 15 |
Claude Fable 5.1 (Adaptive Reasoning, High Effort, Default Fallback)AnthropicAgent · Stirrup | Stirrup | 1650± 18 |
Qwen3.8-Flash-NextAlibabaAgent · Stirrup | Stirrup | 1647± 17 |
Grok 4.6 (high)SpaceXAIAgent · Stirrup | Stirrup | 1643± 14 |
Grok 4.6 (medium)SpaceXAIAgent · Stirrup | Stirrup | 1643± 15 |
GLM-5.3 (max)Z AIAgent · Stirrup | Stirrup | 1642± 14 |
DeepSeek V4.1 Flash (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1632± 21 |
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)AnthropicAgent · Stirrup | Stirrup | 1631± 14 |
Qwen3.8 MaxAlibabaAgent · Stirrup | Stirrup | 1630± 14 |
Claude Opus 5 (Adaptive Reasoning, High Effort)AnthropicAgent · Stirrup | Stirrup | 1629± 15 |
Qwen3.8 2.4T A95BAlibabaAgent · Stirrup | Stirrup | 1628± 15 |
GPT-5.6 Sol (max)OpenAIAgent · Stirrup | Stirrup | 1624± 14 |
GPT-5.6 Sol (xhigh)OpenAIAgent · Stirrup | Stirrup | 1585± 14 |
GPT-6 Astra (max)OpenAIAgent · Stirrup | Stirrup | 1580± 20 |
Claude Fable 5.1 (Adaptive Reasoning, Medium Effort, Default Fallback)AnthropicAgent · Stirrup | Stirrup | 1579± 18 |
DeepSeek V4 Flash 0731 Vision (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1577± 15 |
Agnes 3.0 FlashSapiens AIAgent · Stirrup | Stirrup | 1573± 22 |
GPT-6 Astra (xhigh)OpenAIAgent · Stirrup | Stirrup | 1555± 20 |
Kimi K3 (max)KimiAgent · Stirrup | Stirrup | 1548± 14 |
GPT-6 Astra (high)OpenAIAgent · Stirrup | Stirrup | 1530± 20 |
Claude Opus 5 (Adaptive Reasoning, Medium Effort)AnthropicAgent · Stirrup | Stirrup | 1525± 15 |
GPT-5.6 Sol (high)OpenAIAgent · Stirrup | Stirrup | 1524± 14 |
Muse Spark 1.2 (xhigh)MetaAgent · Stirrup | Stirrup | 1523± 15 |
Claude Fable 5.1 (Adaptive Reasoning, Low Effort, Default Fallback)AnthropicAgent · Stirrup | Stirrup | 1504± 18 |
GPT-6 Astra (medium)OpenAIAgent · Stirrup | Stirrup | 1501± 20 |
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)AnthropicAgent · Stirrup | Stirrup | 1501± 14 |
DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1493± 15 |
Claude Opus 4.8 (Adaptive Reasoning, Max Effort)AnthropicAgent · Stirrup | Stirrup | 1489± 13 |
GPT-5.6 Terra (xhigh)OpenAIAgent · Stirrup | Stirrup | 1479± 14 |
GPT-5.6 Terra (max)OpenAIAgent · Stirrup | Stirrup | 1477± 14 |
Gemini 3.8 Flash (high)GoogleAgent · Stirrup | Stirrup | 1464± 19 |
Qwen3.8 27B (xhigh)AlibabaAgent · Stirrup | Stirrup | 1463± 16 |
Grok 4.6 (low)SpaceXAIAgent · Stirrup | Stirrup | 1461± 15 |
GPT-5.6 Luna (max)OpenAIAgent · Stirrup | Stirrup | 1460± 13 |
Gemini 3.8 Flash (medium)GoogleAgent · Stirrup | Stirrup | 1455± 19 |
DeepSeek V4 Flash 0731 (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1442± 14 |
Gemini 3.7 Flash (high)GoogleAgent · Stirrup | Stirrup | 1433± 15 |
Qwen3.8 27B (medium)AlibabaAgent · Stirrup | Stirrup | 1430± 16 |
Grok 4.5 (high)SpaceXAIAgent · Stirrup | Stirrup | 1430± 15 |
GPT-5.6 Luna (xhigh)OpenAIAgent · Stirrup | Stirrup | 1428± 14 |
GPT-5.6 Sol (medium)OpenAIAgent · Stirrup | Stirrup | 1427± 13 |
GPT-6 Astra (low)OpenAIAgent · Stirrup | Stirrup | 1419± 20 |
GPT-5.6 Terra (high)OpenAIAgent · Stirrup | Stirrup | 1415± 14 |
Gemini 3.7 Flash (medium)GoogleAgent · Stirrup | Stirrup | 1407± 16 |
GLM-5.2 (max)Z AIAgent · Stirrup | Stirrup | 1406± 13 |
Qwen3.8 27B (low)AlibabaAgent · Stirrup | Stirrup | 1403± 16 |
Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)AnthropicAgent · Stirrup | Stirrup | 1401± 14 |
K2 Horizon 375B A23BMBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | 1400± 19 |
GPT-5.5 (xhigh)OpenAIAgent · Stirrup | Stirrup | 1396± 13 |
Claude Opus 4.7 (Adaptive Reasoning, Max Effort)AnthropicAgent · Stirrup | Stirrup | 1396± 14 |
Agnes 2.5 Pro BetaSapiens AIAgent · Stirrup | Stirrup | 1376± 17 |
GPT-5.6 Luna (high)OpenAIAgent · Stirrup | Stirrup | 1375± 14 |
Claude Opus 5 (Adaptive Reasoning, Low Effort)AnthropicAgent · Stirrup | Stirrup | 1372± 15 |
GPT-5.5 (high)OpenAIAgent · Stirrup | Stirrup | 1372± 13 |
Gemini 3.8 Flash (low)GoogleAgent · Stirrup | Stirrup | 1359± 16 |
GPT-5.6 Sol (low)OpenAIAgent · Stirrup | Stirrup | 1355± 14 |
Gemini 3.7 Flash (low)GoogleAgent · Stirrup | Stirrup | 1354± 16 |
Gemini 3.6 Flash (high)GoogleAgent · Stirrup | Stirrup | 1331± 15 |
GPT-5.6 Terra (medium)OpenAIAgent · Stirrup | Stirrup | 1318± 14 |
Claude Sonnet 5 (Adaptive Reasoning, High Effort)AnthropicAgent · Stirrup | Stirrup | 1308± 14 |
GPT-5.4 (xhigh)OpenAIAgent · Stirrup | Stirrup | 1307± 14 |
GLM-5.2 (Non-reasoning)Z AIAgent · Stirrup | Stirrup | 1306± 16 |
MiniMax-M3MiniMaxAgent · Stirrup | Stirrup | 1304± 14 |
Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)AnthropicAgent · Stirrup | Stirrup | 1295± 14 |
Muse Spark 1.1 (xhigh)MetaAgent · Stirrup | Stirrup | 1294± 16 |
GPT-5.5 (medium)OpenAIAgent · Stirrup | Stirrup | 1288± 14 |
Claude Sonnet 5 (Non-reasoning, High Effort)AnthropicAgent · Stirrup | Stirrup | 1283± 15 |
GPT-5.6 Sol (Non-reasoning)OpenAIAgent · Stirrup | Stirrup | 1273± 14 |
Apodex 1.1ApodexAgent · Stirrup | Stirrup | 1273± 17 |
Gemini 3.5 Flash (high)GoogleAgent · Stirrup | Stirrup | 1259± 14 |
K2 Horizon MoVA 36B A4BMBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | 1243± 21 |
Quasar 438B (max, based on GLM-5.2)Multiverse ComputingAgent · Stirrup | Stirrup | 1239± 19 |
Ling-3.0-flash-VLInclusionAIAgent · Stirrup | Stirrup | 1225± 21 |
DeepSeek V4 Pro 0424 (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1223± 14 |
Claude Sonnet 5 (Adaptive Reasoning, Medium Effort)AnthropicAgent · Stirrup | Stirrup | 1223± 14 |
DeepSeek V4 Pro 0424 (Reasoning, High Effort)DeepSeekAgent · Stirrup | Stirrup | 1219± 17 |
Motif 3Motif TechnologiesAgent · Stirrup | Stirrup | 1199± 17 |
JT-4.1 Flash 236B A21BChina MobileAgent · Stirrup | Stirrup | 1193± 16 |
Inkling SmallThinking MachinesAgent · Stirrup | Stirrup | 1191± 17 |
Qwen3.7 MaxAlibabaAgent · Stirrup | Stirrup | 1190± 14 |
G9v3-39A5BAI9StarsAgent · Stirrup | Stirrup | 1189± 18 |
MiMo-V2.5-ProXiaomiAgent · Stirrup | Stirrup | 1186± 14 |
Kimi K3 (low)KimiAgent · Stirrup | Stirrup | 1185± 17 |
Motif 3 (Beta)Motif TechnologiesAgent · Stirrup | Stirrup | 1183± 17 |
GLM-5.1 (Reasoning)Z AIAgent · Stirrup | Stirrup | 1180± 14 |
GPT-5.6 Terra (low)OpenAIAgent · Stirrup | Stirrup | 1178± 15 |
Nex-N2-ProNex AGIAgent · Stirrup | Stirrup | 1175± 15 |
Solar Pro 4UpstageAgent · Stirrup | Stirrup | 1173± 16 |
Ling-3.0-flash-FinInclusionAIAgent · Stirrup | Stirrup | 1171± 21 |
GPT-5.6 Luna (medium)OpenAIAgent · Stirrup | Stirrup | 1169± 14 |
Inkling (xhigh)Thinking MachinesAgent · Stirrup | Stirrup | 1165± 17 |
GPT-5.6 Terra (Non-reasoning)OpenAIAgent · Stirrup | Stirrup | 1146± 14 |
Claude Sonnet 5 (Adaptive Reasoning, Low Effort)AnthropicAgent · Stirrup | Stirrup | 1145± 15 |
Grok Build 0.1 0616SpaceXAIAgent · Stirrup | Stirrup | 1138± 14 |
Hy3TencentAgent · Stirrup | Stirrup | 1136± 17 |
DeepSeek V4 Flash 0420 (Reasoning, Max Effort)DeepSeekAgent · Stirrup | Stirrup | 1116± 14 |
Kimi K2.7 CodeKimiAgent · Stirrup | Stirrup | 1114± 14 |
Qwen3.8 27B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | 1113± 17 |
GPT-5.5 (low)OpenAIAgent · Stirrup | Stirrup | 1113± 15 |
Kimi K2.6KimiAgent · Stirrup | Stirrup | 1096± 13 |
GLM-4.7 (Reasoning)Z AIAgent · Stirrup | Stirrup | 1096± 16 |
GPT-5.4 mini (xhigh)OpenAIAgent · Stirrup | Stirrup | 1095± 14 |
K2 Horizon 7BMBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | 1091± 21 |
Nemotron 3 Ultra 550B A55B (Reasoning)NVIDIAAgent · Stirrup | Stirrup | 1091± 14 |
Agnes 2.5 Pro AlphaSapiens AIAgent · Stirrup | Stirrup | 1088± 17 |
MiniMax-M2.7MiniMaxAgent · Stirrup | Stirrup | 1087± 14 |
DeepSeek V4 Flash 0420 (Reasoning, High Effort)DeepSeekAgent · Stirrup | Stirrup | 1083± 17 |
MiMo-V2.5XiaomiAgent · Stirrup | Stirrup | 1079± 17 |
Muse SparkMetaAgent · Stirrup | Stirrup | 1076± 14 |
GPT-5.6 Luna (low)OpenAIAgent · Stirrup | Stirrup | 1075± 15 |
Qwen3.6 27B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 1069± 14 |
Qwen3.6 PlusAlibabaAgent · Stirrup | Stirrup | 1066± 14 |
Gemini 3.5 Flash-LiteGoogleAgent · Stirrup | Stirrup | 1063± 17 |
GPT-5.5 (Non-reasoning)OpenAIAgent · Stirrup | Stirrup | 1054± 14 |
Solar Open2 250BUpstageAgent · Stirrup | Stirrup | 1053± 18 |
A.X-K2SK TelecomAgent · Stirrup | Stirrup | 1044± 18 |
Qwen3.6 27B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | 1043± 16 |
GPT-5.4 nano (xhigh)OpenAIAgent · Stirrup | Stirrup | 1035± 14 |
Grok 4.3 (Non-reasoning)SpaceXAIAgent · Stirrup | Stirrup | 1031± 14 |
Grok 4.3 (high)SpaceXAIAgent · Stirrup | Stirrup | 1018± 14 |
Ling 3.0 FlashInclusionAIAgent · Stirrup | Stirrup | 1017± 17 |
GPT-5 (high)OpenAIAgent · Stirrup | Stirrup | 1015± 16 |
Qwen3.6 35B A3B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 991± 14 |
Claude 4.5 Sonnet (Reasoning)AnthropicAgent · Stirrup | Stirrup | 989± 16 |
GPT-5.6 Luna (Non-reasoning)OpenAIAgent · Stirrup | Stirrup | 987± 15 |
LongCat 2.0LongCatAgent · Stirrup | Stirrup | 965± 16 |
K2 Horizon 3.7BMBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | 962± 21 |
Step 3.7 FlashStepFunAgent · Stirrup | Stirrup | 954± 14 |
Qwen3.6 35B A3B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | 953± 18 |
Kimi K2.5 (Reasoning)KimiAgent · Stirrup | Stirrup | 936± 17 |
GPT-5.1 (high)OpenAIAgent · Stirrup | Stirrup | 930± 17 |
Qwen3.5 122B A10B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 925± 15 |
K-EXAONE 2.0 0803LG AI ResearchAgent · Stirrup | Stirrup | 918± 19 |
Qwen3.5 397B A17B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 905± 15 |
Gemini 3.1 Pro PreviewGoogleAgent · Stirrup | Stirrup | 904± 15 |
Muse Glimmer (high)MetaAgent · Stirrup | Stirrup | 893± 19 |
Qwen3.7 PlusAlibabaAgent · Stirrup | Stirrup | 886± 15 |
GPT-5 mini (high)OpenAIAgent · Stirrup | Stirrup | 880± 17 |
Mistral Medium 3.5MistralAgent · Stirrup | Stirrup | 875± 15 |
GLM-4.6 (Reasoning)Z AIAgent · Stirrup | Stirrup | 874± 17 |
Ring-2.6-1TInclusionAIAgent · Stirrup | Stirrup | 864± 15 |
Claude 4.5 Haiku (Reasoning)AnthropicAgent · Stirrup | Stirrup | 854± 15 |
KAT-Coder-Pro V1KwaiKATAgent · Stirrup | Stirrup | 845± 16 |
KAT Coder Pro V2KwaiKATAgent · Stirrup | Stirrup | 843± 18 |
Qwen3.5 122B A10B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | 833± 18 |
MiniCPM5-2BOpenBMBAgent · Stirrup | Stirrup | 828± 22 |
DeepSeek V3.1 Terminus (Reasoning)DeepSeekAgent · Stirrup | Stirrup | 826± 18 |
Claude 4 Sonnet (Reasoning)AnthropicAgent · Stirrup | Stirrup | 819± 17 |
DeepSeek V3.2 (Reasoning)DeepSeekAgent · Stirrup | Stirrup | 810± 19 |
G9v3-3BAI9StarsAgent · Stirrup | Stirrup | 807± 26 |
MiMo-V2-Flash (Non-reasoning)XiaomiAgent · Stirrup | Stirrup | 788± 19 |
Nemotron 3.5 LightningNVIDIAAgent · Stirrup | Stirrup | 767± 19 |
Gemma 4 31B (Reasoning)GoogleAgent · Stirrup | Stirrup | 755± 16 |
gpt-oss-120b (high)OpenAIAgent · Stirrup | Stirrup | 745± 15 |
Qwen3.5 35B A3B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | 740± 20 |
GPT-5.4 mini (Non-Reasoning)OpenAIAgent · Stirrup | Stirrup | 734± 16 |
Granite 4.2 30BIBMAgent · Stirrup | Stirrup | 721± 20 |
Ling 3.0 TinyInclusionAIAgent · Stirrup | Stirrup | 716± 20 |
Gemma 4 26B A4B (Reasoning)GoogleAgent · Stirrup | Stirrup | 693± 15 |
Gemma 4 31B (Non-reasoning)GoogleAgent · Stirrup | Stirrup | 691± 18 |
Devstral 2MistralAgent · Stirrup | Stirrup | 689± 17 |
GPT-5.4 nano (Non-Reasoning)OpenAIAgent · Stirrup | Stirrup | 681± 21 |
Devstral Small 2MistralAgent · Stirrup | Stirrup | 680± 16 |
Qwen3 Coder NextAlibabaAgent · Stirrup | Stirrup | 664± 18 |
GPT-5.5 Instant (June 2026)OpenAIAgent · Stirrup | Stirrup | 662± 17 |
Command A+CohereAgent · Stirrup | Stirrup | 658± 18 |
Mercury 2InceptionAgent · Stirrup | Stirrup | 646± 19 |
Granite 4.2 8BIBMAgent · Stirrup | Stirrup | 645± 21 |
Nemotron 3 Super 120B A12B (Reasoning)NVIDIAAgent · Stirrup | Stirrup | 644± 16 |
Nova 2.0 Pro Preview (medium)AmazonAgent · Stirrup | Stirrup | 630± 16 |
EXAONE 4.5 33BLG AI ResearchAgent · Stirrup | Stirrup | 628± 20 |
Gemini 2.5 ProGoogleAgent · Stirrup | Stirrup | 616± 17 |
HyperNova 60B 2605 (high, based on gpt-oss-120b)Multiverse ComputingAgent · Stirrup | Stirrup | 604± 19 |
Nova 2.0 Pro Preview (low)AmazonAgent · Stirrup | Stirrup | 601± 16 |
Gemini 3.1 Flash-LiteGoogleAgent · Stirrup | Stirrup | 596± 16 |
Gemma 4 12B (Reasoning)GoogleAgent · Stirrup | Stirrup | 591± 20 |
Qwen3.5 9B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 588± 18 |
Mistral Large 3MistralAgent · Stirrup | Stirrup | 588± 16 |
Mistral Small 3.1MistralAgent · Stirrup | Stirrup | 550± 18 |
K-EXAONE (Reasoning)LG AI ResearchAgent · Stirrup | Stirrup | 540± 20 |
Mistral Small 4 (Reasoning)MistralAgent · Stirrup | Stirrup | 537± 19 |
Mistral Medium 3.1MistralAgent · Stirrup | Stirrup | 536± 17 |
Nova 2.0 Lite (high)AmazonAgent · Stirrup | Stirrup | 536± 17 |
gpt-oss-20b (high)OpenAIAgent · Stirrup | Stirrup | 514± 16 |
Trinity Large ThinkingArcee AIAgent · Stirrup | Stirrup | 512± 20 |
Nova 2.0 Pro Preview (Non-reasoning)AmazonAgent · Stirrup | Stirrup | 509± 16 |
DiffusionGemma 26B A4BGoogleAgent · Stirrup | Stirrup | 501± 17 |
Qwen3 235B A22B 2507 (Reasoning)AlibabaAgent · Stirrup | Stirrup | 495± 19 |
Ling 2.6 FlashInclusionAIAgent · Stirrup | Stirrup | 493± 19 |
Celeris-1CelerisAgent · Stirrup | Stirrup | 483± 21 |
DeepSeek R1 (Jan '25)DeepSeekAgent · Stirrup | Stirrup | 479± 18 |
North Mini CodeCohereAgent · Stirrup | Stirrup | 472± 18 |
GPT-4.1 miniOpenAIAgent · Stirrup | Stirrup | 453± 18 |
Nemotron Cascade 2 30B A3BNVIDIAAgent · Stirrup | Stirrup | 447± 21 |
Solar Pro 3UpstageAgent · Stirrup | Stirrup | 447± 16 |
NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)NVIDIAAgent · Stirrup | Stirrup | 438± 17 |
Ministral 3 14BMistralAgent · Stirrup | Stirrup | 432± 17 |
o3-mini (high)OpenAIAgent · Stirrup | Stirrup | 421± 19 |
Nemotron 3 Nano Omni 30B A3B ReasoningNVIDIAAgent · Stirrup | Stirrup | 416± 21 |
Claude 3.5 HaikuAnthropicAgent · Stirrup | Stirrup | 408± 18 |
Ministral 3 8BMistralAgent · Stirrup | Stirrup | 405± 17 |
Granite 4.2 3BIBMAgent · Stirrup | Stirrup | 390± 21 |
Granite 4.1 30BIBMAgent · Stirrup | Stirrup | 379± 16 |
gpt-oss-120b (low)OpenAIAgent · Stirrup | Stirrup | 364± 21 |
Magistral Medium 1.2MistralAgent · Stirrup | Stirrup | 362± 18 |
K2 Think V2MBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | 328± 18 |
Qwen3 Next 80B A3B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 325± 20 |
DeepSeek V3 0324DeepSeekAgent · Stirrup | Stirrup | 275± 19 |
Qwen3 30B A3B 2507 (Reasoning)AlibabaAgent · Stirrup | Stirrup | 274± 20 |
Qwen3 32B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 240± 18 |
Ministral 3 3BMistralAgent · Stirrup | Stirrup | 234± 18 |
Magistral Small 1.2MistralAgent · Stirrup | Stirrup | 213± 19 |
LFM2.5-2.6BLiquid AIAgent · Stirrup | Stirrup | 202± 22 |
GPT-4OpenAIAgent · Stirrup | Stirrup | 189± 18 |
GPT-4o miniOpenAIAgent · Stirrup | Stirrup | 188± 19 |
Qwen3 14B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 188± 19 |
DeepSeek V3 (Dec '24)DeepSeekAgent · Stirrup | Stirrup | 183± 19 |
Gemma 4 E4B (Reasoning)GoogleAgent · Stirrup | Stirrup | 177± 22 |
Qwen3.5 2B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 166± 21 |
Qwen3 8B (Reasoning)AlibabaAgent · Stirrup | Stirrup | 165± 19 |
NVIDIA Nemotron 3 Nano 4BNVIDIAAgent · Stirrup | Stirrup | 159± 21 |
Granite 4.1 3BIBMAgent · Stirrup | Stirrup | 74± 20 |
Llama 4 ScoutMetaAgent · Stirrup | Stirrup | 60± 17 |
Llama 3.3 Instruct 70BMetaAgent · Stirrup | Stirrup | 50± 19 |
Gemma 4 E2B (Reasoning)GoogleAgent · Stirrup | Stirrup | 36± 21 |
Mistral Small 3.2MistralAgent · Stirrup | Stirrup | 29± 19 |
GPT-4.1 nanoOpenAIAgent · Stirrup | Stirrup | 13± 19 |
Llama 4 MaverickMetaAgent · Stirrup | Stirrup | -45± 17 |
NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)NVIDIAAgent · Stirrup | Stirrup | -122± 17 |
Qwen3.5 2B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | -130± 18 |
Qwen3.5 0.8B (Non-reasoning)AlibabaAgent · Stirrup | Stirrup | -133± 18 |
MiniCPM-V 4.6 1.3BOpenBMBAgent · Stirrup | Stirrup | -134± 17 |
Llama 3.1 Instruct 8BMetaAgent · Stirrup | Stirrup | -151± 17 |
Qwen3.5 0.8B (Reasoning)AlibabaAgent · Stirrup | Stirrup | -152± 17 |
Nanbeige4.1-3BNanbeigeAgent · Stirrup | Stirrup | -166± 17 |
Gemma 3 12B InstructGoogleAgent · Stirrup | Stirrup | -170± 17 |
Gemma 3 27B InstructGoogleAgent · Stirrup | Stirrup | -171± 16 |
Phi-4 Mini InstructMicrosoftAgent · Stirrup | Stirrup | -171± 17 |
K2 Horizon 0.9BMBZUAI Institute of Foundation ModelsAgent · Stirrup | Stirrup | -174± 17 |