These findings add context without changing ranks or scores.
- Codex + GPT-6 Astra
Current High-effort implementation remains competitive, but the older Astra agentic/refactoring rationale mixes effort settings and dates. Keep it as a historical recommendation, not proof that every Codex Astra setting defeats newer contenders. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Fable 5.1
Keep Fable 5.1's historical careful-codebase rationale with its original dates. The new effort table uses Medium, so it cannot settle High/Max performance or refresh the old general configuration. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Opus 5
Opus 5 remains the historical baseline; this pass adds no current exact-version advantage over Opus 5.5 or Sol 6.1. Retention is not a new win. Agentic placement additionally requires sustained completion and intervention evidence.
- Cursor + Fable 5.1
Cursor + Fable 5.1 needs a Cursor-specific result; Claude Code findings cannot establish this configuration's quality or agent reliability. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-5.6 Sol
GPT-5.6 Sol is an older Codex baseline. New Sol 6.1 findings are not transferable evidence for this version. Agentic placement additionally requires sustained completion and intervention evidence.
- Replit Agent + Automatically Routed
Automatic routing offers an integrated workflow, but the routed stack and cost per completed task must be disclosed before model-specific quality attribution. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot + Automatically Routed
Copilot Auto's convenient workflow does not identify which model completed each task. Require current routing, intervention and task-completion records. Agentic placement additionally requires sustained completion and intervention evidence.
- Devin + SWE-2
Devin + SWE-2 has saved mixed repair/terminal results; the older vendor and selected-suite evidence cannot establish a current across-task lead. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Opus 5.5 High
Opus 5.5 High remains a reasonable implementation shortlist choice: recent short-task and practical-build findings support it, while intervention and sustained-task reliability remain limitations. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-6 Sol High
GPT-6 Sol High is a separate predecessor to Sol 6.1. Its saved evidence is retained without borrowing successor results. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Sonnet 5.5 High
Sonnet 5.5 High is represented in the current short-task comparison. A credible lower-cost candidate, but not a demonstrated all-workload replacement for the frontier choices. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-6.1 Sol High
Sol 6.1 High is a credible quality/value shortlist alternative. Current short-task evidence is close to the leaders, with narrow reconstruction support; exact independent replication and broad reliability remain open. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-6.1 Sol Medium
Sol 6.1 Medium is a sensible speed/value setting to consider for bounded changes. Its weaker-effort result is its own tradeoff, not proof of High-level quality. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot + GPT-6.1 Sol
Copilot's Sol 6.1 default setting needs Copilot-specific observation. Codex effort results cannot establish its behavior. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot HydraFusion
September HydraFusion cannot inherit results for the October adaptive or Max releases; record the routing and build used by any comparison. Agentic placement additionally requires sustained completion and intervention evidence.
- GPT-6.1 Sol + Replit Agent Max Mode (recommended effort)
Replit Max Mode changes harness and effort selection. Codex Sol High findings do not establish this usable configuration's quality or cost. Agentic placement additionally requires sustained completion and intervention evidence.
- Sonnet 5.5 + Replit Agent Power (recommended effort)
Replit Power with Sonnet requires its own completion and recovery evidence; Claude Code Sonnet High is a different configuration. Agentic placement additionally requires sustained completion and intervention evidence.
- Step 5 Preview
Step 5 Preview's coding eligibility does not establish a reproducible agent/harness configuration or comparative repository performance. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Claude Haiku 5.5 High
Haiku 5.5 High is a separate effort configuration requiring task-specific correctness and recovery evidence; speed or vendor availability alone is insufficient. Agentic placement additionally requires sustained completion and intervention evidence.
- Claude Code + Claude Haiku 5.5 Medium
Haiku 5.5 Medium needs its own observed tradeoff against High; neither effort inherits the other's result. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-6.1 Sol Low
Sol 6.1 Low is worth considering for quick, bounded changes. Initial effort findings suggest weaker quality than High; require review and tests rather than treating lower cost as equal capability. Agentic placement additionally requires sustained completion and intervention evidence.
- Cursor + Grok 4.7 High
Cursor + Grok 4.7 High is a usable challenger. Available narrow benchmark signals require artifact, funding and workload checks before a broad editorial order. Agentic placement additionally requires sustained completion and intervention evidence.
- Mellum2.1 Thinking · vLLM · documented thinking settings
Mellum2.1 Thinking in vLLM needs actual repository tasks with the documented thinking parameters, not model-level or completion-only evidence. Agentic placement additionally requires sustained completion and intervention evidence.
- Muse Wonder · Muse Code · provider defaults
Muse Wonder defaults are distinct from Muse Wonder Full High goal mode. The latter's practical failures are not evidence that this exact configuration fails. Agentic placement additionally requires sustained completion and intervention evidence.
- Muse Wonder Full · Muse Code · provider defaults
Muse Wonder Full defaults cannot inherit the High goal-mode result. Default execution, quota behavior and task recovery still require direct observation. Agentic placement additionally requires sustained completion and intervention evidence.
- Gemini 4 Argon · Antigravity September 2026 · Medium
Argon in Antigravity Medium needs harness-matched repository outcomes; Cursor, Copilot and bare-model results are not interchangeable. Agentic placement additionally requires sustained completion and intervention evidence.
- Gemini 4 Argon · GitHub Copilot CLI · provider default
Argon in Copilot's provider default needs recorded default/routing behavior and actual task outcomes. Agentic placement additionally requires sustained completion and intervention evidence.
- Muse Wonder Full · Muse Code · High goal mode
Pat tested High goal mode with Contributor off; VulcanBench High used Contributor on in 1.4.9. Preserve Pat’s interrupted quota and estimated cost. These are not a replication.
- Gemini 4 Argon · Cursor · High
Argon Cursor High requires its own exact-effort outcomes and cost/intervention record. Agentic placement additionally requires sustained completion and intervention evidence.
- Gemini 4 Argon · Cursor · Medium
Argon Cursor Medium must be assessed independently from High; no effort uplift is inferred. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot HydraFusion · October 14
October 14 HydraFusion adaptive routing is a materially different stack from September. Require disclosed routed models and repeatable task records. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot HydraFusion Max · October 14
HydraFusion Max needs evidence for its own maximum-quality routing and usage costs, not automatic inheritance from adaptive mode. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + Beam 1.0 · Max
Beam 1.0 OpenCode Max is in the usable field. Current short-task signals do not resolve independent exact-configuration reliability and completion quality. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + Nemotron 3.5 Super VL · High
Nemotron 3.5 Super VL OpenCode High needs corroborated repository outcomes with exact routing/build details. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + Mistral Large 4 · High
Mistral Large 4 OpenCode High needs task-matched independent replication before a broad placement. Agentic placement additionally requires sustained completion and intervention evidence.
- DeepSeek Harness + V4.1 Flash 1017 · Max
DeepSeek V4.1 Flash 1017 in the native harness is distinct from OpenCode. Assess each one's completion, failures and costs separately. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + DeepSeek V4.1 Flash 1017 · Max
DeepSeek Flash 1017 OpenCode Max is a separate comparison arm from native-harness Max; shared model identity is not shared agent performance. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + DeepSeek V4.1 Flash 1017 · High
DeepSeek Flash 1017 OpenCode High cannot inherit Max's measured quality or resource use. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + MiniMax M3.1 Flash · Max
MiniMax M3.1 Flash OpenCode Max needs disclosed route, exact effort and reproducible outcomes before comparative qualification. Agentic placement additionally requires sustained completion and intervention evidence.
- OpenCode + Recursive Flint · rec 4-bit · Default
Recursive Flint rec 4-bit is a precision-specific configuration. Results from a different quantization or provider route cannot establish its coding quality. Agentic placement additionally requires sustained completion and intervention evidence.
- Kimi K3.1 Pro + Kimi Code — default effort
Kimi K3.1 Pro Kimi Code default is not the same arm as OpenCode Max or explicit Kimi Code High. Resolve the actual default before comparing. Agentic placement additionally requires sustained completion and intervention evidence.
- Kimi K3.1 Pro + Atomic Agent 0.6.9 — Low
Atomic Agent 0.6.9 Kimi Low needs its own task record; Kimi Code and OpenCode results do not transfer. Agentic placement additionally requires sustained completion and intervention evidence.
- Kimi K3.1 Pro + Atomic Agent 0.6.9 — Max
Atomic Agent 0.6.9 Kimi Max must be compared separately from Low and from other Kimi harnesses. Agentic placement additionally requires sustained completion and intervention evidence.
- Codex + GPT-6.1 Terra — provider default
Terra's fresh benchmark page reports explicit effort arms; these cannot rank the registered provider-default Codex configuration. Determine its actual effort and match the usable configuration first. Agentic placement additionally requires sustained completion and intervention evidence.
- GitHub Copilot Fusion · Copilot CLI · Standard
This GA CLI effort is available, but the selected models depend on organization policy. No matched independent trial with that pool establishes quality.
- GitHub Copilot Fusion · Copilot CLI · High
This GA CLI effort is available, but the selected models depend on organization policy. No matched independent trial with that pool establishes quality.
- GitHub Copilot Fusion · Copilot CLI · Max
The Oct 28 original AA test verified Fusion Max in the Copilot CLI: led terminal tasks, trailed Opus Max overall and on repository questions with a lower estimated cost per task. Organization-specific routing, intervention cost and independent same-configuration work remain open.
- Particle-1 · Codex via Vercel AI Gateway · provider default
Gateway access is verified. No matched repository comparison establishes whether token savings survive retries or preserve correctness.
- Codex + GPT-6.1 Terra — XHigh
One partially reviewed October 27 BuilderBench run used Terra XHigh in Codex. The comparison used a different harness for Kimi; neither costs nor general performance are settled.
- Muse Wonder Full · Muse Code Contributor · High
VulcanBench's Oct 13–17 reconstruction suite tested High in Muse Code Contributor mode, a subscription with quota interruptions and estimated cost. One three-hour timeout; judging only finished work hides the failure. No equivalent evidence for Contributor-off goal mode or broad independent replication.
- Muse Wonder Full · Muse Code Contributor · Extra-high
VulcanBench's Oct 13–17 reconstruction suite tested Extra-high in Muse Code Contributor mode, a subscription with quota interruptions and estimated cost. One three-hour timeout; judging only finished work hides the failure. No equivalent evidence for Contributor-off goal mode or broad independent replication.
- Muse Wonder Full · Muse Code Contributor · Low
VulcanBench's Oct 13–17 reconstruction suite tested Low in Muse Code Contributor mode, a subscription with quota interruptions and estimated cost. 18 of 23 tasks fully solved. No equivalent evidence for Contributor-off goal mode or broad independent replication.
- Recursive Flint · mini-SWE-agent 2.4.6 · offline NVFP4
Resolved 40 of 50 offline SWE-bench Verified instances on one RTX PRO 6000. Same seeded subset has not been run on current podium rivals, so not a head-to-head win.
- GPT-6.1 Terra · Codex CLI 0.159.0 · Low
Exact Codex CLI0.159.0 Terra effort retained. October26–27 Bug Hunt three-repeat means: Low6, Medium7.3, High9.7, XHigh11.3, Max40.7 of105 bugs. October28 NerdsChalk small Python repair saturated Low–XHigh; Max untested there. Different tasks and original dates; no broad incumbent superiority inferred.
- GPT-6.1 Terra · Codex CLI 0.159.0 · Medium
Exact Codex CLI0.159.0 Terra effort retained. October26–27 Bug Hunt three-repeat means: Low6, Medium7.3, High9.7, XHigh11.3, Max40.7 of105 bugs. October28 NerdsChalk small Python repair saturated Low–XHigh; Max untested there. Different tasks and original dates; no broad incumbent superiority inferred.
- GPT-6.1 Terra · Codex CLI 0.159.0 · High
Exact Codex CLI0.159.0 Terra effort retained. October26–27 Bug Hunt three-repeat means: Low6, Medium7.3, High9.7, XHigh11.3, Max40.7 of105 bugs. October28 NerdsChalk small Python repair saturated Low–XHigh; Max untested there. Different tasks and original dates; no broad incumbent superiority inferred.
- GPT-6.1 Terra · Codex CLI 0.159.0 · XHigh
Exact Codex CLI0.159.0 Terra effort retained. October26–27 Bug Hunt three-repeat means: Low6, Medium7.3, High9.7, XHigh11.3, Max40.7 of105 bugs. October28 NerdsChalk small Python repair saturated Low–XHigh; Max untested there. Different tasks and original dates; no broad incumbent superiority inferred.
- GPT-6.1 Terra · Codex CLI 0.159.0 · Max
Exact Codex CLI0.159.0 Terra effort retained. October26–27 Bug Hunt three-repeat means: Low6, Medium7.3, High9.7, XHigh11.3, Max40.7 of105 bugs. October28 NerdsChalk small Python repair saturated Low–XHigh; Max untested there. Different tasks and original dates; no broad incumbent superiority inferred.
- OpenCode + Tencent Hy4 · OpenRouter default
OpenCode can route to production Tencent Hy4. No matched repository run verified at default effort.
- OpenCode + Inkling 2 Small Free · default effort
Newly registered production configuration; comparable category-specific trials and independent corroboration remain incomplete.
- Claude Code + Inkling 2 Small · default effort (Tinker 256K)
Bito directly tested the 256K PEFT checkpoint in Claude Code at default settings. Promising for code tracing but weaker planning; commercial single-publisher result, no independent exact-route replication.
- Claude Code + Fable 5.2 Medium
Fable 5.2 pinned Medium cannot inherit the new AA default-fallback Medium/High scores or CodeRouter’s API-default High bash-only trials. Seek like-for-like correctness, costs, defaults, restrictions, and interventions.
- OpenCode + Ripple-65B Preview BF16 · maximum effort
OpenCode BF16 maximum effort lacks an independent matching comparison; the thinking-off format trial and Plotly analytics agent used different configurations.
- Google Antigravity + Gemini 4 Carbon · Medium
Medium in the Antigravity single-agent app is distinct from CLI High. Business-workflow outcomes do not establish a coding lead.
- Google Antigravity CLI + Gemini 4 Carbon · High
AA v2.1 Carbon High CLI leads Astra Max Codex on composite/terminal/repo Q&A, not merge-ready changes. Lower estimated task cost on the revised method but slower; app High and Claude Code Default are different. No independent matching practitioner confirmation.
- Muse Code + Muse Ember · provider default
Muse Code with Ember at provider default is verified. Results for a different orchestrating model or delegated subagent do not rank this standalone configuration.
- mini-SWE-agent + Nanbeige4.3-33B · highest thinking
The creator documents mini-SWE-agent v2.4.6 at highest thinking with preserve_thinking=true. No matching independent comparative repository trial was established.
- Google Antigravity 2.25.0 + Gemini 4 Carbon · High
Holter exercised Antigravity 2.25.0 High, with installed skills, a truncated answer and a follow-up. Useful workflow evidence, but the baseline and comparator settings are not controlled.
- Claude Code + Gemini 4 Carbon Default (1M context)
Bito original Nov13 follow-up compares ClaudeCodeCarbonDefault1M and HydrogenHigh1M: Carbonmore accurate but slower/costlier; both repeat Redis misconception. Commercial evaluator, mixed efforts and unpublished build/judge details; no qualified same-configuration replication.
- Claude Code + Qwen4-Max-Preview Xhigh (Token Plan)
Verified fixed Qwen4-Max-Preview in Claude Code via Alibaba Token Plan, default xhigh and1M context. Time-limited preview and dedicated credentials affect access/workflow. No original independent exact-configuration performance, practical-success, cost-per-task or failure/restriction comparison; vendor claims cannot fill those gaps.
- Claude Code + LongCat-2.5 Thinking 512K
LongCat 2.5 Thinking with Claude Code2.1.280/512K is registered. Vendor mixed-harness benchmarks and TOKRACE's unspecified-build creative tasks do not independently corroborate this exact configuration.
- Antigravity Managed Agent + Gemini 4 Hydrogen High
Hydrogen High managed Antigravity API is independently registered from CLI/Cursor/mini-SWE-agent. Current retrieved benchmarks do not establish this configuration's relative correctness, reliability, cost and workflow quality.
- Antigravity CLI 1.5.8 + Gemini 4 Hydrogen · Xhigh
BugHunt Nov13 agy1.5.8 HydrogenXhigh mean32.7/105, runs43/27/28. Pro-to-Ultra plan switch, judge changes, uncertain context and list-cost estimates constrain inference; source funding/artifact qualification and second exact-config comparison missing.
- Claude Code + Gemini 4 Hydrogen High (1M context)
Claude Code Hydrogen High 1M: Bito found inexpensive short answers but repeated Redis reconnect mistakes and poorer sustained work than Opus. Commercial single publisher; client build and independent replication missing.
- Antigravity CLI + Gemini 4 Argon High
Antigravity CLI Argon High is an AA v2.1 arm, distinct from Argon app Medium or Cursor. One objective result, without an independent matching sustained task and total operating cost.
- Codex + GPT-6 Astra Max
Codex Astra Max in AA v2.1 beat Carbon High CLI on merge-ready code and finished faster but trailed on terminal and repo questions. Mixed efforts, distinct cost basis, no matched second qualified publisher.
- Codex + GPT-6.1 Sol Max
Codex Sol6.1 Max in AA v2.1 is economical against Astra Max but not the same as the established Sol High/Medium hands-on trials. Do not transfer reliability, restrictions or practical fit across efforts.
- Claude Code + Fable 5.2 Xhigh (default fallback)
Fable Xhigh and High tie on displayed index; Xhigh took longer and had higher estimated API cost.
- Claude Code + Fable 5.2 High (default fallback)
Fable High was close to Opus Max on composite, better on merge-ready changes but lower on terminal and repo Q&A; substantially less time and modeled cost.
- Claude Code + Fable 5.2 Medium (default fallback)
Fable Medium was cheaper/faster than High and Xhigh but weaker on merge-ready work. Its default fallback may complete tasks.
- Claude Code + Fable 5.2 Low (default fallback)
Fable Low saved time but had the weakest displayed composite among the five Fable 5.2 efforts.
- Claude Code + Fable 5.2 Max (default fallback)
Fable Max took the longest and cost the most among its efforts without leading the composite.
- Claude Code + Opus 5.5 Max (default fallback)
Opus Max tied the leading displayed Fable index; stronger on terminal and repo Q&A than Fable High, but slower at higher estimated cost.
- Claude Code + Sonnet 5.5 Max (default fallback)
Sonnet Max was competitive on terminal tasks, behind Fable High on merge-ready work and slower.
- Claude Code + Fable 5.1 Max (default fallback)
Fable 5.1 Max showed stronger repository Q&A than its merge-ready results; its max default-fallback score cannot refresh a generic historical placement.
- Gemini 4 Hydrogen Low in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 4 Hydrogen Medium in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 4 Hydrogen High in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 4 Hydrogen Extra in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 3.8 Flash Low in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 3.8 Flash Medium in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Gemini 3.8 Flash High in Antigravity CLI (AI Pro, Nov 13–15)
New exact AI Pro CLI setting: one Windows task with mixed sandbox protections, no independent matched replication or multi-repo reliability evidence.
- Qwen4-Max High + Atomic Agent
Qwen4-Max High + Atomic Agent: Unmeasured omitted-effort setup; do not attribute explicitly configured results.
- Atomic Agent + Qwen4-Max High
Atomic Agent + Qwen4-Max High: One repair repository with three repetitions per effort; ceiling saturated, concurrency and incomplete costs limit generalization.
- Atomic Agent + Qwen4-Max Xhigh
Atomic Agent + Qwen4-Max Xhigh: One repair repository with three repetitions per effort; ceiling saturated, concurrency and incomplete costs limit generalization.
- Codex + GPT-6.2 Astra high
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- Codex + GPT-6.2 Astra xhigh
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- Codex + GPT-6.2 Astra max
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- Codex + GPT-6.2 Sol (Xhigh)
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- Codex + GPT-6.2 Sol (Max)
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- Codex + GPT-6.2 Sol (High)
Exact Codex GPT6.2 effort registered before AA evidence. Benchmark task quality, runtime and modeled cost are available; agent build, independent real-project preference, intervention/failure reliability and complete workflow restrictions remain unverified. No cross-effort inference or placement.
- GitHub Copilot CLI + MAI Code 2 (Max)
Public Copilot CLI Max route registered. Microsoft vendor MSBench findings and50%promotional pricing do not provide independent task quality/preference or durable value. Need exact build, repeated repo completion, interventions, cost and restrictions.
- GitHub Copilot CLI + MAI Code 2 Flash (Max)
Public Copilot CLI Max route registered. Microsoft vendor MSBench findings and50%promotional pricing do not provide independent task quality/preference or durable value. Need exact build, repeated repo completion, interventions, cost and restrictions.