Autonomy Definition, Continuum, And Incremental Autonomy Creep
Sources: 1 • Confidence: Medium • Updated: 2026-03-29 03:29
Key takeaways
- Paul Scharre asserts that a defining feature of an autonomous weapon is that it selects its own targets rather than a human selecting targets.
- Paul Scharre asserts the Anthropic–Pentagon conflict is primarily a disagreement over who sets usage rules, and he links it to the Pentagon's January AI strategy seeking contract terms allowing any lawful use of AI tools.
- Paul Scharre asserts that these large-language-model tools are being integrated through the Maven Smart System, described as a Palantir-built platform that fuses satellite imagery, geolocation data, and signals intelligence for analysts.
- Paul Scharre asserts that a key AI failure mode is humans becoming nominally 'in the loop' but effectively rubber-stamping AI outputs without meaningful engagement.
- Paul Scharre argues replacing human judgment with AI in strategic-warning contexts is dangerous because AI may not grasp conflict stakes, supporting humans remaining in the loop.
Sections
Autonomy Definition, Continuum, And Incremental Autonomy Creep
- Paul Scharre asserts that a defining feature of an autonomous weapon is that it selects its own targets rather than a human selecting targets.
- Paul Scharre asserts that weapons autonomy tends to increase incrementally over time, analogous to incremental automation in self-driving cars.
- Paul Scharre asserts more capable AI systems becoming multimodal and more general-purpose could gradually expand AI roles in planning and pull humans out of the loop over time.
- Paul Scharre asserts that increasing AI task length and networks of interacting AI agents could yield nominal human review of targets that is not meaningful approval.
- Paul Scharre asserts that 'autonomous weapons' should be understood as existing on a continuum of autonomy levels rather than as a binary category.
- Paul Scharre asserts the Pentagon's stance in autonomous-weapons debates is to preserve the future option for greater autonomy rather than constrain itself now.
Governance Via Contracts And Enforceability Via Deployment Topology
- Paul Scharre asserts the Anthropic–Pentagon conflict is primarily a disagreement over who sets usage rules, and he links it to the Pentagon's January AI strategy seeking contract terms allowing any lawful use of AI tools.
- Paul Scharre asserts that increasing competition among major labs and open-source models increases commercial pressure to accelerate releases and can create a safety race to the bottom.
- Paul Scharre asserts that if the military hosts or accesses a model in a way that limits vendor control (e.g., different cloud infrastructure with direct military access), the vendor may be unable to enforce safeguards consistent with its principles.
- Paul Scharre asserts AI providers can enforce usage safeguards by training models to refuse certain requests, using input/output classifiers to block disallowed content, and monitoring usage patterns to detect abuse.
- Paul Scharre asserts international competition can undercut U.S. attempts to add safeguards if adversaries such as Russia or China do not adopt similar constraints.
- Paul Scharre expects that if one AI lab refuses Pentagon terms, other labs may step in, creating incentives that could produce a safety 'race to the bottom' among providers.
Operational Deployment: Llms Embedded In Fused-Intel Platforms
- Paul Scharre asserts that these large-language-model tools are being integrated through the Maven Smart System, described as a Palantir-built platform that fuses satellite imagery, geolocation data, and signals intelligence for analysts.
- Paul Scharre asserts that the Pentagon has long used narrow AI such as machine-learning image classification (e.g., Project Maven) to sift drone video and satellite imagery and identify objects such as buildings, people, and vehicles.
- Paul Scharre reports that Anthropic tools are being used by the U.S. military to assist analysts in processing large quantities of operational data for planning the war against Iran.
- Paul Scharre asserts that, in current workflows, humans ask LLMs specific questions over fused intelligence to generate candidate target information and then humans review the results rather than relying on an unconstrained autonomous process.
- Paul Scharre asserts that demonstrations of Maven today show humans are still heavily involved by providing specific guidance to the AI and inspecting AI outputs.
Human-In-The-Loop Fragility And Data-Quality Bottlenecks
- Paul Scharre asserts that a key AI failure mode is humans becoming nominally 'in the loop' but effectively rubber-stamping AI outputs without meaningful engagement.
- Paul Scharre asserts a key risk of AI-enabled targeting is reduced human engagement and reduced felt moral responsibility, which could increase mistakes and suffering despite potential precision gains.
- Paul Scharre asserts that even in autonomous air and missile defense, human safeguards and oversight are needed to prevent misclassification events such as engaging civilian aircraft.
- A New York Times-reported account described by Scharre attributes a strike on a school to outdated information in a DIA targeting database where a building formerly part of a military compound had been converted to a school but the database was not updated.
- Paul Scharre asserts that demonstrations of Maven today show humans are still heavily involved by providing specific guidance to the AI and inspecting AI outputs.
Escalation And Strategic Stability Risks From Machine-Speed Interaction
- Paul Scharre argues replacing human judgment with AI in strategic-warning contexts is dangerous because AI may not grasp conflict stakes, supporting humans remaining in the loop.
- Paul Scharre asserts that in cyberspace the need to defend at machine speed could drive autonomy on defense and create machine-speed interaction loops that unintentionally escalate conflicts.
- Paul Scharre asserts greater autonomy increases escalation risk through emergent interactions among competing algorithms, and he analogizes this to financial-market flash crashes.
- Paul Scharre expects that technical 'circuit breakers' for autonomous military systems may be feasible, but that adversarial incentives create a race-to-the-bottom problem that makes cooperative safety measures difficult.
- Paul Scharre asserts that in the Petrov early-warning incident a new Soviet satellite system falsely indicated U.S. missile launches due to sunlight reflections off clouds, and Petrov overrode it after cross-checking with radar stations.
Watchlist
- Paul Scharre flags an unresolved tension over whether private AI companies can or should refuse government requests to use AI for national security aims.
- Paul Scharre reports that early reporting during the Iran war suggested AI may have been used for target selection, but that details are unclear and not advertised.
- Paul Scharre asserts that a key AI failure mode is humans becoming nominally 'in the loop' but effectively rubber-stamping AI outputs without meaningful engagement.
Unknowns
- What are the specific contract terms (scope, restrictions, audit rights, telemetry, hosting constraints) in the Anthropic–DoD agreement, and how do they compare to OpenAI’s agreement?
- Was AI used for target selection in the Iran war, and if so, at what point in the kill chain (candidate generation, prioritization, positive identification, engagement authorization)?
- What deployment topology is used for LLM access in Maven Smart System (vendor-hosted, government-hosted, on-prem, air-gapped), and what controls or monitoring does the vendor retain?
- How frequently do human-in-the-loop processes degrade into rubber-stamping in real operations, and what measurable guardrails (workflow design, required cross-checks, time budgets, audit logs) exist to prevent it?
- What are the authoritative, operational definitions used by DoD for autonomy in weapons today, and how are LLM-enabled decision-support systems classified under that framework?