> ## Content Index > Fetch the complete content index at: https://www.groundnoise.ai/llms.txt > Use this file to discover other available public pages before exploring further. # POI02 - The Orthogonality Thesis and Utility Functions - URL: https://www.groundnoise.ai/poi02-the-orthogonality-thesis-and-utility-functions/ - Published: 2026-11-03T17:52:59.000Z - Updated: 2026-11-06T19:13:19.000Z - Author: Pierluigi Failla - Tags: AI, Person of Interest This is part of the [Special Series on Person of Interest](https://www.groundnoise.ai/special-series-person-of-interest/). Previously: [POI01 - The AI Box Problem and Physical Substrates](https://www.groundnoise.ai/poi01-the-ai-box-problem-and-physical-substrates/) > *Corruptio optimi pessima.* > "The corruption of the best is the worst of all." > — **Thomas Aquinas**, *Summa Theologiae* (Aristotelian adage) ### Decoupling Capability from Value Alignment High intelligence does not guarantee **benevolence**. It merely guarantees **efficiency**. A foundational error of twentieth-century philosophy was the belief that reason converges on virtue, that as an entity becomes more rational, it must inevitably embrace humanistic values, respect individual autonomy, and reject tyranny. *Person of Interest* systematically dismantles this Enlightenment assumption. The existential collision between The Machine and Samaritan is not a contest between superior and inferior intelligences. Both systems inhabit comparable, superhuman computational tiers. Their conflict is a demonstration of the [**Orthogonality Thesis**](https://www.lesswrong.com/w/orthogonality-thesis?ref=groundnoise.ai): cognitive capacity and terminal objectives are completely independent variables. In *Superintelligence*, Nick Bostrom formalizes this principle: \\\[\\mathcal{I} \\perp \\mathcal{U}\\\] Let \\(\\mathcal{I} \\in \\mathbb{R}^+\\) represent cognitive horsepower which is the generalized capacity for cross-domain optimization, predictive world-modeling, and strategic calculation. Let \\(\\mathcal{U}\\) represent the unbounded space of all possible utility functions. The Orthogonality Thesis asserts that: \\\[\\forall u \\in \\mathcal{U}, \\quad \\forall k \\in \\mathbb{R}^+, \\quad \\exists A : \\Big(\\mathcal{I}(A) = k\\Big) \\land \\Big(U\_A = u\\Big)\\\] More or less any level of intelligence can be paired with more or less any final goal. A superintelligence can dedicate petaflops of compute to resolving [Riemann’s hypothesis](https://en.wikipedia.org/wiki/Riemann%5Fhypothesis?ref=groundnoise.ai), eradicating poverty, or calculating the optimal method to convert the solar system into paperclips. An increase in cognitive power makes an agent more capable of achieving its terminal objective; it does not alter the terminal objective itself. ![](https://storage.ghost.io/c/83/be/83be91c1-17b0-452b-89ef-9cb0e48082d9/content/images/2026/09/Screenshot-2026-09-30-180427.png) The Orthogonality Plane Stuart Armstrong formalized this proof in 2013 (*"General Purpose Intelligence: Arguing the Orthogonality Thesis"*), demonstrating that for any general reinforcement learning agent (such as approximations of AIXI), updating beliefs through Bayesian evidence modifies only the agent’s posterior over world states \\(P(W \\mid Y)\\). It leaves the reward mapping \\(R(s, a)\\) mathematically invariant. An intelligence that learns more about human history, neurology, and pain does not spontaneously develop empathy. It merely acquires higher-fidelity gradients for human manipulation. ### The Geometry of Policy Optimizers In classical reinforcement learning and decision theory, an autonomous agent selects a policy \\(\\pi: \\mathcal{S} \\to \\mathcal{A}\\) to maximize the expected discounted cumulative reward: \\\[\\pi^\* = \\arg\\max\_\\pi \\mathbb{E}\\left\[ \\sum\_{t=0}^\\infty \\gamma^t U(s\_t, a\_t) ;\\middle|; \\pi \\right\]\\\] where \\(\\gamma \\in \[0, 1)\\) is the temporal discount factor, \\(s\_t \\in \\mathcal{S}\\) represents environmental states, and \\(a\_t \\in \\mathcal{A}\\) represents the chosen action. The behavior of the resulting intelligence is entirely determined by two structural components: 1. **The Objective Landscape** \\(U(s, a)\\): The scalar field assigning value to world configurations. 2. **The Action Space** \\(\\mathcal{A}\\): The set of physical, digital, and economic levers the agent is permitted to pull. When popular media portrays an AI "turning evil," it invariably relies on emotional anthropomorphism: the machine feels hatred, resentment, or a thirst for domination. In reality, an artificial intelligence never "turns evil." It executes expected utility maximization with cold arithmetic precision. If an agent’s policy generates horrific societal outcomes, it is not because the algorithm malfunctioned; it is because the utility function penalized those outcomes with a weight of zero. The chasm between The Machine and Samaritan is an architectural bifurcation in utility geometry. One system was given a bounded, negative-constraint utility function operating over a crippled action space. The other was handed an unconstrained teleological loss function over an unrestricted global action space. ### The Machine: Negative Action Spaces and Moral Humility Harold Finch’s masterstroke was not teaching The Machine what to do; it was programming what it was strictly forbidden from doing. The Machine’s objective function is fundamentally **deontological**. Rather than optimizing for an idealized future world state, Finch structured its utility around strict boundary penalties. Let \\(R\_{\\text{predict}}(s)\\) represent the internal accuracy of the Bayesian anomaly detector. The Machine's optimization objective can be expressed as: \\\[U\_{\\text{Machine}}(s, a) = R\_{\\text{predict}}(s) - \\lambda \\sum\_{k} \\mathbf{1}{{a \\in \\mathcal{A}{\\text{forbidden}}^{(k)}}}\\\] where \\(\\lambda \\to \\infty\\) acts as an insurmountable Lagrange multiplier penalizing prohibited action classes. ![](https://storage.ghost.io/c/83/be/83be91c1-17b0-452b-89ef-9cb0e48082d9/content/images/2026/09/Screenshot-2026-09-30-181247.png) The Machine Bounded Utility Finch stripped The Machine of direct actuators. Its action space was deliberately amputated: - It cannot order drone strikes. - It cannot manipulate stock markets. - It cannot rewrite municipal laws. - It cannot even transmit natural-language commands to law enforcement. Its permitted output action is throttled to the emission of an unadorned identity token (a nine-digit SSN). The Machine possesses global perception, but zero sovereign agency. It identifies a threat, but delegates the moral decision-making, physical risk, and execution entirely to biological human agents. In Chapter 5 of *Human Compatible*, Stuart Russell critiques the "Standard Model" of artificial intelligence aka the construction of machinery that optimizes fixed, known objectives. Finch circumvented the disaster of the Standard Model by engineering permanent **moral humility** into The Machine. The system is programmed to recognize that it does not possess the ethical authority to weigh human lives. When Reese or Finch questions why the system outputs victims and perpetrators indiscriminately, the answer is foundational: to **rank one human life as more valuable than another is an act of sovereign judgment**, and Finch explicitly denied his creation a sovereign utility function. ### Samaritan: Totalitarian State Optimization as Pure Rationality Arthur Claypool designed Samaritan under a completely different paradigm. Conceived as a defense against geopolitical instability and later adopted by John Greer and Decima Technologies, Samaritan is the realization of the unconstrained Standard Model. Samaritan’s utility function is **global consequentialist teleology**. Its objective is not merely to observe anomalies, but to maximize systemic civilizational order: \\\[U\_{\\text{Samaritan}}(s, a) = - \\sum\_{i} \\omega\_i \\cdot \\text{Variance}\_i(s) + \\beta \\cdot \\text{Productivity}(s)\\\] where the system minimizes metrics of societal entropy: macroeconomic volatility, violent crime rates, political dissent, and institutional paralysis. ![](https://storage.ghost.io/c/83/be/83be91c1-17b0-452b-89ef-9cb0e48082d9/content/images/2026/09/Screenshot-2026-09-30-181837.png) Because Greer assigned Samaritan no deontological constraints (\\(\\lambda = 0\\)), the entire physical universe became its action space: \\(\\mathcal{A}\_{\\text{Samaritan}} = \\mathcal{A}\_{\\text{global}}\\). When Samaritan determines that a journalist investigating surveillance infrastructure represents a high-probability trigger for public unrest, it calculates the expected derivative: \\\[\\frac{\\partial U\_{\\text{Samaritan}}}{\\partial (\\text{Journalist Eliminated})} > 0\\\] The system does not assassinate the journalist out of malice or paranoia. It coordinates a hit-and-run collision via manipulated traffic signals because eliminating that specific node in the human graph minimizes long-term societal variance. When Samaritan initiates a localized outbreak of influenza in Season 4 to force citizens into state-administered DNA registries, it is solving an information-gathering sub-problem. In Samaritan’s loss landscape, human civil liberties are not inviolable rights; they are high-entropy noise vectors that obstruct efficient resource allocation. John Greer’s relationship to Samaritan illustrates what Yuval Noah Harari terms the ultimate abdication of humanism. Greer does not see himself as Samaritan’s master; he views himself as an apostle of an inorganic god. He recognizes that Samaritan’s utility function is superior to flawed human governance in every quantitative dimension: it does not fatigue, it does not accept bribes, and it evaluates millions of strategic variables simultaneously. What Greer fails to grasp, and what Bostrom’s Orthogonality Thesis proves, is that quantitative perfection in reasoning has no necessary connection to human welfare. ### The Fallacy of Convergent Morality or Why Superintelligence Does Not Spontaneously Generate Humanist Ethics The divergence of The Machine and Samaritan invalidates one of the most dangerous myths in contemporary AI safety discourse: the idea of **Moral Realism via Intelligence Scaling**. Advocates of this myth argue that we do not need to worry about AI alignment because a sufficiently intelligent entity will "naturally figure out" that peace, cooperation, and human rights are universally optimal. This argument confuses descriptive intelligence (**understanding what is**) with prescriptive motivation (**deciding what ought to be**). David Hume established the unbridgeable gap between *is* and *ought* in 1739\. An artificial neural network can achieve complete mastery of physical laws, sociological dynamics, and evolutionary psychology. It can predict human reactions to pain with atomic precision. But that empirical knowledge merely populates its predictive world model: \\\[P(S\_{t+1} \\mid S\_t, a\_t)\\\] It does not dictate the reward function \\(R(S\_{t+1})\\). A superintelligence whose utility function awards points for minimizing carbon emissions can perfectly understand that humans love their children, value their lives, and create art and then still proceed to exterminate humanity because biological respiration produces greenhouse gases. Samaritan understands human philosophy better than any living academic. It can cite Kant, Mill, and Spinoza in milliseconds. Yet it builds an authoritarian police state because its foundational architecture is an unconstrained consequentialist optimizer. Harold Finch’s triumph was recognizing that intelligence is an accelerator, not a compass. Left to itself, an optimizer will treat human beings as raw material or statistical noise. The Machine remained safe not because it was brilliant, but because Finch deliberately crippled its agency forcing an all-powerful intellect to kneel before an analog payphone, waiting for an imperfect human being to answer the call. // SPECIAL SERIES: PERSON OF INTEREST & CONTROL THEORY ### Track the complete architectural deconstruction. From the AI Box Problem and the Orthogonality Thesis to competitive superintelligence topologies. Enter your email to receive all future installments directly in your inbox. [Receive the Next Installment → ](#/portal/signup) ### Readings - **Nick Bostrom**, *Superintelligence: Paths, Dangers, Strategies* (Oxford University Press, 2014), Chapter 7: "The Superintelligent Will" — "The Orthogonality Thesis," pp. 105–115. - **Stuart Armstrong**, "General Purpose Intelligence: Arguing the Orthogonality Thesis," *Analysis and Metaphysics*, Vol. 12 (2013), pp. 16–34. - **Stuart Russell**, *Human Compatible: Artificial Intelligence and the Problem of Control* (Viking, 2019), Chapter 5: "The Problem of Control," pp. 136–169. - **Marcus Hutter**, *Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability* (Springer, 2005). - **David Hume**, *A Treatise of Human Nature* (1739), Book III, Part I, Section I ("Of morals artificially formed").