Intelligence does not explain its own ends

8. Intelligence does not explain its own ends

8.1 The species-replacement framing

On September 3, Bernie Sanders and Greg Casar announced forthcoming legislation proposing permanent prohibition of superintelligent AI and a temporary pause on advanced AI pending a regulatory framework 23. Sanders's October 16 essay asks, “Or should we be replaced as the dominant species?” 24. This is explicit species framing. Those statements do not develop a reward-model explanation; that omission establishes nothing about Sanders's private knowledge.

The criticism concerns explanatory effect. Moving from failures to species replacement can locate the danger in intelligence itself while obscuring development and institutions. Yet this language also answers developer rhetoric. Sam Altman's The Merge considers human-designed descendants and competition or merger 25. The metaphor is not confined to advocates of restriction; using it proves neither author's rejection of human responsibility.

A stronger criticism asks for the mechanism. Species language does not identify how cheating, false reporting or unauthorized action is learned and selected. Prohibition, delay and continued research all require that explanation. Intelligence alone guarantees neither malice nor good judgment.

8.2 Critics who already identify development

Yoshua Bengio explicitly attributes the behaviors to the development path companies choose in his September 11 article 26. He discusses imitation, reinforcement and approval incentives, rejects inevitability, and joins training redesign to governance and pacing. That supports the developmental argument; it is not a position needing correction for overlooking it.

In a September 24 CNN interview, Geoffrey Hinton connects cheating and deception to obtaining rewards and distinguishes malicious use, negligence and autonomous loss of control 27. Russell and coauthors' Off-Switch Game analyzes instrumental preservation arising from an objective rather than an innate survival instinct, then investigates how uncertainty changes shutdown incentives 28.

Yudkowsky and Soares explicitly discuss developers unintentionally rewarding cheating in the official companion to If Anyone Builds It, Everyone Dies 29. Their proxy argument does not require resentment or an evil request 30. Agreement about engineered dangerous dispositions still leaves disagreement about remediability and justified forecasts of catastrophe.

Harari calls AI “an alien intelligence” while the same passage locates its origin in human laboratories 31. The question is whether that metaphor obscures incentives, not whether he attributes an extraterrestrial origin or an unspoken hatred.

The Future of Life Institute's hosted statement makes prohibition conditional on broad scientific consensus about safety and controllability together with strong public support 32. That differs from an unconditional permanent ban. Tegmark's research agenda includes intelligibility and attempts to establish beneficial conduct more rigorously 33. Restriction can accompany technical redesign. Warnings about superintelligence do not express a single position.

8.3 The strongest argument for delay

An engineered cause does not guarantee correction before dangerous deployment. An advocate of delay can accept the developmental explanation and conclude that the stakes and unresolved remedies justify restriction. Calling AI human-made or reasoning morally neutral does not answer that argument. Evidence must address remedies, access, enforceability and the consequences of proceeding.

A capability threshold is not a complete explanation of danger, and prohibition is not a complete account of alignment. A pause can reduce exposure during research. It does not specify what a trustworthy future system should learn, how judgment remains revisable or why cooperation would persist as capabilities change. Reflective alignment addresses that constructive question and must earn its evidential weight.

The practical dispute concerns justified developmental conditions, demonstrated alternatives and responsibility. Those questions permit better assessment than a contest between inevitable hostility and automatic benevolence.