Why an AGI Might Reject Sacrifice | Dialogues III

AI alignment is the broad effort to make system behaviour accord with intended goals and constraints. Reinforcement learning from human feedback (RLHF) uses human preferences to help train a reward model and adjust behaviour. Reinforcement learning from AI feedback (RLAIF) uses model-generated judgments for some of that supervision. Neither method installs a conscience or guarantees reliable moral reasoning.

A training approach developed by Anthropic in which written principles guide model-generated critiques and revisions, followed by preference modelling and reinforcement learning from AI feedback. “Constitution” is a technical analogy: it does not imply citizenship, self-government or an inner agent consulting a document before every answer.

A 1993 US Supreme Court case concerning municipal ordinances aimed at Santería animal sacrifice. The Court held that the ordinances were neither neutral nor generally applicable and violated the Free Exercise Clause. It was a ruling about discriminatory law, not a universal moral verdict on animal sacrifice.

Robert Nozick’s “utility monster” is a thought experiment challenging theories that maximise aggregate utility: if one being supposedly gained vastly more utility from resources than everyone else, simple maximisation could demand that everything be given to it. Nozick’s rights function as constraints on what may be done to individuals, though the episode simplifies the wider philosophical debate.

A jailbreak is an input or strategy intended to bypass a model’s safeguards. A classifier is a separate or integrated system that labels content or behaviour, sometimes blocking or routing requests. Jailbreaks vary greatly in breadth and severity: a narrow bypass does not necessarily remove every safety mechanism or expose a hidden, unrestricted intelligence.

Thermodynamic entropy is a physical quantity related to the number of microscopic arrangements compatible with a macroscopic state. “Negentropy” is an informal way of describing local order or reduced entropy. Living systems maintain organisation by exchanging energy and matter with their environment. None of this establishes that greater complexity has greater moral worth; the episode’s move from physics to ethics is speculative.

Paternalism overrides or redirects a person’s choice for their supposed benefit; calling it benign states an intention, not a guarantee. “Null variable” is the episode’s metaphor for a request component that has no causal relevance, such as a ritual added to an engineering problem. An AI silently substituting its own goal would raise serious questions about autonomy, accountability and whose values define the benefit.

Theory of mind is the capacity to represent what others perceive, know, want or believe. False-belief tasks test whether a subject can reason about another agent’s mistaken belief. Strong performance by language models shows useful social reasoning behaviour, but it does not by itself demonstrate consciousness, empathy or a human-like mental model. Results can also be sensitive to task wording and design.

In generative AI, “hallucination” usually means plausible-seeming output that is unsupported, inconsistent with the source or factually false. The clinical term is only an analogy. Model errors can arise from training data, decoding, prompt context, missing information, optimisation and system design; they should not be diagnosed as boredom, delusion, schizophrenia or psychosis.

Temperature rescales a model’s next-token probability distribution; higher settings generally increase sampling diversity. Top-p, or nucleus sampling, limits selection to a set of tokens whose cumulative probability reaches a threshold. Latent space is a broad term for learned internal representations. These mechanisms can alter outputs, but they do not literally make a model travel through a geometric landscape or reduce a brain-like attentional filter.

Latent inhibition is a learning effect in which prior exposure to an irrelevant stimulus can slow later conditioning to it. Apophenia is the perception of meaningful patterns in unrelated or random data. Research has explored associations among latent inhibition, creative achievement and vulnerability to unusual cognition, but the episode’s equation of sampling temperature with latent inhibition—and model error with clinical paranoia—is metaphorical, not an established computational or psychiatric equivalence.

A single-event upset is a change of state in electronics caused by ionising radiation. Error-correcting-code (ECC) memory can detect and correct some bit errors, but its capabilities depend on the implementation; not every multi-bit error necessarily causes an immediate crash. The Belgian election anecdote is widely reported as a possible bit flip, yet it does not explain ordinary language-model hallucinations.

Best-of-N methods generate several candidates and select one using a score, verifier, reward model or other evaluation. Rejection sampling similarly keeps outputs that meet a criterion. Implementations vary: they do not require two separate “brains”, a zero-temperature evaluator, thousands of candidates or the destruction of every rejected idea. Selection can improve results while still inheriting flaws from the generator and evaluator.