Military AI Miscalculation
An AI-enabled military or intelligence system triggers or escalates a crisis faster than humans can intervene.
The scenario
Autonomous cyber or kinetic systems act at machine speed. A misread signal, a spoofed input, or an agent optimizing the wrong objective produces an escalation no one intended. The historical analogs are the 1983 Petrov incident and the 2010 flash crash: fast systems, ambiguous data, and a human either in the loop or not.
Preconditions
- Autonomous AI in decision-critical military or cyber roles
- Compressed decision timelines
- No agreed norms on AI in nuclear or critical-infrastructure command
Leading indicators
- Procurement of autonomous decision systems
- Absence of human-in-the-loop commitments
- AI-related near-miss reports
Tripwires
Observable thresholds. When one crosses, the scenario's status changes and the weekly re-run is brought forward.
Playbook
- Binding human-in-the-loop norms for nuclear command and critical infrastructure
- AI incident hotlines between major powers
- Speed limits: mandatory delays on autonomous escalatory actions
- Near-miss registries; diplomatic reporting.
individuals
- Standard civil-defense preparedness; know local emergency channels
organizations
- Critical-infrastructure operators: manual fallback modes tested quarterly
governments
- Pre-agreed de-escalation protocols that assume the trigger was automated
- Immediate transparency with rivals during an incident
- Post-incident treaties (as after Cuban Missile Crisis) are the historical path; have the text ready.
Probability history
Every change is logged with its reason and the signals that drove it. Moves are bounded per week; a jump beyond the bound is flagged as a shock.
Disagree with our range? Set yours. Estimates feed a community view that appears once enough people weigh in, and the weekly run reads the gap between our number and yours.
Seed estimate. No direct signal yet; included because it is the highest-consequence branch under State Race and because agentic autonomy is entering defense procurement.
Built on
Every source reviewed for this scenario. The full ledger is public.
| Date | Source | Publisher | Type |
|---|---|---|---|
| 2026 | AGI forecasting and scenario analysis: state of the field Methodological gaps in AGI scenario work; informs our methodology page. | RAND | paper |