Methodology | AI Wars of 2026
How AI Wars of 2026 distinguishes evidence, hypotheses, and open questions.
Methodology
Start with the Big Idea
A Big Idea is the specific takeaway a chapter or visual is trying to establish. It must be strategically relevant, understandable, evidence-bound, and falsifiable: a skeptical reader should be able to see both what supports it and what evidence could change it. The convention is broadly used across data analysis, visualization, and editorial practice; this book applies it through the evidence-first approach developed in Fabio Correa's The Defensible Decision.
For this book, a Big Idea is not a verdict such as "Microsoft is winning enterprise AI." It is a qualified proposition that can be tested:
The strategic contest may move from generating intelligence to controlling the path from delegated intent to consequential action.
The book treats this as a working hypothesis, not a conclusion.
The analytical frame
AI Wars are plural: overlapping strategic contests for positions that may create leverage, economic value, or the ability to shape what other actors can do. An intelligence host is one potentially valuable position in that system, not the universal prize. Intelligence, compute, distribution, information, context, authority, execution, transactions, standards, and trust may each matter differently to different actors.
The book therefore separates an observed asset from a strategic conclusion:
Asset to mechanism to observable outcome to strategic leverage and/or economic capture
An actor can possess a model, user relationship, identity system, distribution channel, device, or cloud platform without proving that the asset produces a durable advantage. A defensible claim names the proposed mechanism, the observable outcome that would support it, the alternative explanation, and the evidence that would weaken it.
Competing hypotheses
The book tests three explanations for where durable value may accrue:
- Intelligence captures value. A sufficiently differentiated model remains the critical position, even when it is embedded in other systems. This strengthens if sustained capability leads produce durable adoption and economic value; it weakens if equivalent or substitutable models support comparable outcomes.
- The host captures value. A system that combines delegated intent, context, authority, execution, governance, and distribution captures the most durable advantage. This strengthens if those integrations produce repeat use, trust, retention, and economic value; it weakens if users or organizations can switch without rebuilding those capabilities.
- The stack fragments. Different actors retain distinct valuable positions across intelligence, infrastructure, interfaces, identity, execution, transactions, and standards. This strengthens if no actor captures a durable cross-layer advantage; it weakens if one position reliably absorbs the others.
These are competing hypotheses, not a sequence of predetermined stages. A company can be positionally strong in one war while losing momentum in another, and multiple actors can win different prizes without producing a single overall winner.
Make the decision defensible
A claim is defensible when a reader can inspect:
- The unit: What does a number or observation represent?
- The population and period: Who or what is included, and when?
- The comparison: What is the baseline, and are the measures genuinely comparable?
- The source: Who reported the evidence and how direct is it?
- The limitation: What does the evidence not establish, and what could change the conclusion?
The book does not turn company-reported product metrics into independent market facts, treat missing disclosure as zero, or use a polished chart to imply causation or certainty.
Evidence priority
The primary evidence of AI-driven transformation is operational: repeated use, task success, reliability and recovery, throughput, governed deployment, and product-defined workflow outcomes. These measures are stronger than financial scale because they show whether a system changes work or a real-world service.
Realized ROI is the financial evidence that can support a claim of sustainability. It must identify the product, costs, outcome, population, and period. Revenue, valuation, investment, and capital spending remain context: they can show resources or demand, but not a transformed workflow. Expected ROI belongs to a scenario, especially for long-horizon infrastructure, commerce, or physical-operation systems; it is not evidence of a realized outcome.
How the method works in practice
Primary sources establish what a company announced, reported, or documented. Independent research and reporting are needed when the argument concerns market behavior, productivity, adoption, retention, or other outcomes a company cannot establish alone.
Charts must earn their place. Each one should communicate a bounded finding, expose its source and material caveat, and make the argument easier to inspect. When the data is incompatible, incomplete, or not strong enough, the correct outcome is prose, an open question, or no visual.
What makes a comparison valid
Keep the test conditions with the result. The official ImageNet 2012 results distinguish SuperVision submissions using extra training data from submissions using only the supplied data. A result detached from that condition is not the same comparison. A paper's publication year also does not identify the test year: the abstract of the 2012 convolutional-network paper reports results on the LSVRC-2010 test set. Use the competition record for the competition result. Source 1 Source 2
Do not average away different work. The NBER customer-support study and METR's developer studies concern different tasks, experience levels, and tool periods. METR's early-2025 study also separates perceived speed from measured time, while its early-2026 continuation reports selection problems and uncertain estimates. These are reasons to state each setting, not to combine the findings into a universal productivity effect. Source 3 Source 4 Source 5
Test whether an improvement generalizes. The Amazon Science compression experiment asks whether a fresh research agent can reproduce an approach from a short prompt without access to the validation set. The reported deliberately induced overfitting did not survive that restriction. The method's claim depends on the restricted information path: pretraining memorization remains untested, and the authors did not use fresh post-cutoff datasets. It is a particular test of generalization, not proof that research agents never overfit. Source 6
Keep a disciplinary objection distinct from a measurement. The mathematics declaration argues that using mathematical problems as AI benchmarks harms the discipline. It is evidence of a stated professional position, not an experiment establishing that any particular score is invalid. Its argument can motivate scrutiny of a benchmark without substituting for testing that benchmark. Source 7
The propositions we are testing
The propositions below are not conclusions; each can be weakened by contrary evidence.
- Systems may matter more than isolated models. A model is necessary, but useful agent outcomes also depend on context, identity, permissions, authority, tools, orchestration, governance, and user experience. The proposition weakens if model quality alone consistently determines adoption and value across workflows.
- Agent products change the competitive unit. Agent products extend AI beyond individual answers toward longer-horizon, delegated action on a user's behalf. AgentBench frames that challenge as sequential action in interactive and partially observable environments, not simply producing a strong response. Source 8
- Distribution becomes an advantage only through use. Distribution matters only when it helps produce trusted, repeated use and measurable value. Paid seats are not active use, retention, workflow control, or return on investment. Source 9 Source 10
- Model choice can be a host feature. An intelligence host may benefit from better models only if it can change them without breaking prompts, tools, task quality, authority boundaries, or governance. Source 11 Source 12
- Context and authority may be harder to replace than a model endpoint. Standardized tool connections can help, but they do not make identity, authorization, data quality, workflow design, or organizational policy portable by default. Source 13
- Governance and reliability are product capabilities. In consequential work, the ability to constrain, inspect, approve, and recover from agent actions may influence which systems people are willing to use. Source 14 Source 15
- AI value may be captured in several places. Model providers, platforms, applications, clouds, system integrators, standards, transaction providers, and customers may capture different parts of the value created by delegated action.
- The contenders start from different positions. The contenders should be compared as different strategic routes, not as interchangeable model vendors or a single leaderboard.
What could change the argument
The three competing hypotheses above are not rhetorical caveats. Two additional conditions could change how any of them develops:
- Applications retain the objective. People may delegate discrete tasks while familiar applications remain the primary places where work is designed, reviewed, governed, and completed.
- Autonomy stalls at the boundary of trust. Agents improve, but high-value actions retain strict approval requirements; the most valuable systems remain assistive, auditable, and narrowly authorized.
Evidence that supports a competing hypothesis or either condition would require the book to change its conclusion.
The evidence-record boundary
The retained evidence record uses the same standard. A series becomes chart-eligible only when its unit, date, source, and comparison are defensible. In the current enterprise AI paid-metrics review, Microsoft 365 Copilot and Google Gemini Enterprise are the only chart-eligible paid-seat series. The Evidence Board is withheld until it can offer more useful reader-facing evidence; other contenders remain excluded from the chart-eligible set until compatible public data is available.
For the current research agenda and its evidence boundaries, see the Roadmap.
Sources
Eyebrow: Evidence workflow
Title: A defensible argument is built step by step
Description: Each step narrows the distance between an interesting claim and a conclusion a skeptical reader can inspect.
Note
Label: How to read this
Body: The sequence is deliberately iterative: new evidence or a failed caveat check can send the work back to an earlier step.
Layers
Name: Big Idea
Description: State the specific, strategically relevant takeaway the reader should be able to test.
Name: Analytical question
Description: Turn the idea into a precise question about a comparison, change, relationship, or decision.
Name: Evidence
Description: Find the strongest available primary and independent material for that question.
Name: Source verification
Description: Confirm the number, unit, population, period, definition, and source status.
Name: Claim boundary
Description: Name what the evidence establishes, what it does not establish, and what could change the conclusion.
Name: Simplest useful visual
Description: Choose a chart or diagram only when it clarifies the evidence better than prose.
Name: Caveat
Description: Make the material limitation visible rather than letting design imply false certainty.
Name: Strategic interpretation
Description: Explain why the bounded evidence matters while keeping fact, inference, and open questions separate.
Takeaway
Label: Defensible decision
Body: A visual is ready only when its claim, evidence, comparison, and limitation remain clear enough for a skeptical reader to challenge.
Source references
Source 1: ImageNet Large Scale Visual Recognition Competition 2012 - All results
ImageNet Large Scale Visual Recognition Challenge
Published: 2012-10-13
Official ILSVRC 2012 results list the University of Toronto SuperVision entry first in classification and describe a large deep convolutional network trained on two NVIDIA GPUs. This benchmark result does not establish general intelligence, consumer adoption, or a single causal explanation for later commercial AI development.
Source 2: ImageNet Classification with Deep Convolutional Neural Networks
Advances in Neural Information Processing Systems 25 (NIPS 2012)
Published: 2012
The conference paper describing the convolutional network behind the University of Toronto entry that the ILSVRC 2012 results page records. The abstract on this page reports top-1 and top-5 error rates of 39.7% and 18.9% on the LSVRC-2010 test set, and a network of 60 million parameters and 500,000 neurons across five convolutional layers. It does not state an ILSVRC 2012 error rate, so figures for that year must be read from the competition results page instead. Self-reported error rates do not establish independent replication, cross-year comparability, or general capability.
Source 3: Generative AI at Work
National Bureau of Economic Research
Published: 2023-04-24
Field study of a generative-AI assistant for customer-support workers. It reports an approximately 14% average productivity gain in that setting, with larger gains for less experienced and lower-skilled workers; it does not test autonomous agents or generalize to all knowledge work.
Source 4: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR
Published: 2025-07-10
Randomized study of 16 experienced open-source developers and 246 issues found AI-allowed work took 19% longer in its early-2025 setting. METR now labels this result out of date; see its 2026 continuation for current but still inconclusive evidence.
Source 5: Update on our early 2026 AI productivity study
METR
Published: 2026-02-24
METR's continuation of its developer-productivity research. Early-2026 results suggest developers may be faster with newer AI tools, but METR reports severe selection effects and a confidence interval too wide for a reliable point estimate.
Source 6: Why don't machine learning research agents overfit?
Amazon Science
Published: 2026-09-10
Martin Bertran Lopez and Aaron Roth summarise a paper testing why ML research agents hill-climbing a reusable validation set do not overfit it. An explorer agent iterates freely, a compressor distils the winning strategy into a very short prompt, and a fresh reproducer with no access to the validation set, code or transcript must rebuild the strategy from that prompt alone. Across eight datasets spanning tabular and image classification, language modelling, diffusion and reward modelling, 32-token prompts let the reproducer match the explorer on the large majority of problems, and one language-modelling strategy survived compression to 16 tokens. The work carries a falsification arm: agents deliberately pushed to overfit produced a validation-versus-holdout gap above 10 percent in 38 of 102 runs, and those gains did not survive the compression bottleneck, so compression also works as a detector. The authors state their limits: the framework assumes the only path from validation data to the final model is the prompt, pretraining memorisation would be an untested side channel, they have not run fresh post-cutoff datasets, and the extension from LLM agents to human research communities is suggestive rather than demonstrated. This is a method-level result about agent research behaviour with stated denominators; it is not an adoption, productivity or deployment measure.
Source 7: A Severe Misalignment of AI in Mathematics
Math and AI
Published: 2026-09-11
A signed declaration, published 11 September 2026 and carrying DOI 10.5281/zenodo.22737750, arguing that the use of mathematical problems as AI benchmarks is detrimental to the discipline. The signatory list carries 25 Fields medalists, counted by distinct name on 2026-09-14, and the separate endorsers page reported 6,517 endorsers on the same date. The endorsement figure is a live counter and rose by thirteen within that day, so any quoted total needs its retrieval date. The declaration establishes that its signatories hold this position; it does not establish that any benchmark is invalid, and its own text acknowledges that the underlying mathematical capability of the models has improved.
Source 8: AgentBench: Evaluating LLMs as Agents
arXiv
Published: 2023-08-07
Independent academic benchmark for LLM agents operating through sequential actions in interactive environments. It evaluates 2023-era models and does not establish current enterprise reliability, productivity, or economic value.
Source 9: FY26 Q4 — Microsoft Cloud and AI Strength Fuels Fourth Quarter Results
Microsoft Investor Relations
Published: 2026-07-29
Primary FY26 Q4 results. Microsoft reported that Microsoft 365 Copilot reached over 30 million paid seats.
Source 10: A message from our CEO: Alphabet Q4 2025 earnings
Published: 2026-02-04
Primary CEO earnings statement. Google reported selling more than 8 million paid Gemini Enterprise seats to more than 2,800 companies as of Q4 2025; this is a company-reported lower bound, not an active-user count.
Source 11: Select a primary AI model for your agent
Microsoft Learn
Published: 2026-05-28
Microsoft documentation on selecting models for agent orchestration, including workload tradeoffs and external-model administration. This rolling document was updated August 3, 2026 and its model lineup can change.
Source 12: Powering Frontier Transformation with Copilot and agents
Microsoft 365 Blog
Published: 2026-03-09
Microsoft's 2026 discussion of multi-model intelligence and model choice in Copilot.
Source 13: Model Context Protocol Specification
Model Context Protocol
Published: 2026-07-28
Current specification for a protocol connecting LLM applications with external data sources and tools through hosts, clients, and servers. The 2026 revision uses stateless, self-contained requests and adds opt-in extensions; standardized integration does not make prompts, memory, identity, authorization, or workflows fully portable.
Source 14: Artificial Intelligence Risk Management Framework (AI RMF 1.0)
National Institute of Standards and Technology
Published: 2023-01-26
Voluntary, use-case-agnostic framework organized around GOVERN, MAP, MEASURE, and MANAGE. It does not certify an AI system as safe or prescribe universal approval thresholds.
Source 15: Running Codex safely at OpenAI
OpenAI
Published: 2026-05-08
Primary source on controls, boundaries, approval, telemetry, and governance for coding agents.