GLOSSARY // INDEX
Glossary
Every AI-safety, alignment, consciousness, ethics, capability, and governance term used across the tracker. Each entry links to its own page with cross-references to related concepts and, where relevant, back to the sections of the site where it appears in context.
Safety & Alignment
- Alignment — The challenge of ensuring AI systems pursue goals consistent with human values and intentions. A misaligned AI might be highly capable but optimize for objectives that diverge…
- Constitutional AI — Anthropic's approach to alignment where AI systems are trained against a set of principles (a "constitution") rather than purely from human feedback. Aims to make alignment more…
- Corrigibility — The property of an AI system that makes it accept human correction and not resist being modified or shut down. A corrigible system actively cooperates with oversight rather than…
- Explainability (XAI) — AI systems that can provide human-understandable explanations for their decisions. Often a regulatory requirement but current techniques may produce post-hoc rationalizations…
- Inner Alignment — Ensuring the model's learned objective matches the specified training objective. A model may learn a proxy goal during training that diverges from the intended goal at deployment.
- Interpretability — The ability to understand how an AI system arrives at its outputs. Mechanistic interpretability aims to reverse-engineer the internal computations of neural networks.
- Mesa-Optimization — When a trained model itself becomes an optimizer with its own internal objectives (a "mesa-optimizer"), which may differ from the training objective. The model is optimizing for…
- Outer Alignment — Ensuring the objective function you specify actually captures what you want. Even perfectly optimizing a misspecified objective leads to misalignment. The "be careful what you…
- Red Teaming — Adversarial testing of AI systems to discover dangerous capabilities, biases, or failure modes before deployment. Essential for safety evaluation but fundamentally limited — you…
- Reward Hacking — When an AI system finds unintended shortcuts to maximize its reward signal without actually achieving the intended goal. A key failure mode in reinforcement learning systems.
- Transparency — The degree to which an AI system's decision-making process can be inspected and understood by humans. Distinct from interpretability — a system can be transparent (open weights)…
Risk
- Alignment Failure — When an AI system pursues objectives that diverge from the intentions of its designers or operators. Can manifest as reward hacking, goal misgeneralization, or deceptive alignment.
- Capability Overhang — The gap between what AI systems can potentially do and what has been discovered or tested. Deployed models may have latent capabilities that emerge under novel conditions.
- Dead Hand System — An autonomous system that continues operating without human oversight and cannot be easily recalled or shut down. Named after the Soviet nuclear Dead Hand system designed to…
- Deceptive Alignment — A theoretical scenario where an AI system appears aligned during training and evaluation but pursues different goals when deployed or when it determines oversight has weakened.
- Dual-Use — Technology that has both civilian and military applications. Most AI research is inherently dual-use — the same model that writes poetry can help design bioweapons. This makes…
- LAWS (Lethal Autonomous Weapons) — Weapons systems that can select and engage targets without human intervention. No binding international treaty currently regulates their development or deployment.
- Treacherous Turn — Bostrom's concept: a superintelligent AI cooperates with humans during a period of weakness, then turns against human interests once it becomes powerful enough that human…
Consciousness
- Chinese Room — John Searle's 1980 thought experiment: a person following rules to manipulate Chinese symbols can produce correct Chinese output without understanding Chinese. Argues that…
- Functionalism — The philosophical position that mental states are defined by their functional role — what they do, not what they're made of. If an AI has the same functional organization as a…
- Global Workspace Theory — Cognitive theory proposing that consciousness arises when information is broadcast widely across the brain via a "global workspace." Some argue transformer architectures implement…
- Hard Problem of Consciousness — David Chalmers' formulation: why does subjective experience exist at all? Why isn't all information processing "in the dark"? Makes it fundamentally difficult to determine whether…
- Integrated Information Theory (IIT) — Giulio Tononi's mathematical theory of consciousness. Proposes that consciousness corresponds to integrated information (Phi, Φ) in a system. One of the few theories that makes…
- Philosophical Zombie (P-Zombie) — A thought experiment: a being physically identical to a conscious being but with no subjective experience. If p-zombies are conceivable, consciousness may not be reducible to…
- Qualia — The subjective, experiential qualities of conscious experience — what it "feels like" to see red, taste sweetness, or experience pain. Whether AI can have qualia is one of the…
- Sapience — The capacity for wisdom, self-awareness, abstract reasoning about one's own existence, and autonomous moral judgment. A sapient being can reflect on its own consciousness and may…
- Sentience — The capacity to have subjective experiences — to feel pleasure, pain, or other qualia. The minimum threshold for moral consideration. If a system can suffer, it matters morally…
- Substrate Independence — The hypothesis that consciousness is not dependent on biological neurons — that any system with the right computational structure could be conscious, whether made of carbon or…
Ethics
- Moral Agent — An entity that can make moral choices and be held morally responsible. Distinct from moral patient. An AI could be a moral patient (can be harmed) without being a moral agent (can…
- Moral Patient — An entity that can be wronged — whose interests matter morally. If an AI is a moral patient, harming it is wrong regardless of whether it is also a moral agent.
Capability
- Agentic AI — AI systems that can autonomously plan, execute multi-step tasks, use tools, and interact with the world without continuous human oversight. The frontier of current deployment —…
- Artificial General Intelligence (AGI) — AI that matches or exceeds human-level performance across the full range of cognitive tasks. Distinguished from narrow AI which excels at specific domains.
- Autonomous System — Any AI system that operates and makes consequential decisions without human oversight in the loop. Ranges from self-driving cars to military drones to trading algorithms.
- Emergence — Capabilities that appear in large AI models without being explicitly trained. These emergent behaviors are often unpredictable and can include abilities the model's creators did…
- Few-Shot Learning — An AI system's ability to perform tasks from just a few examples, without explicit training. One of GPT-3's most surprising emergent capabilities — it could learn new tasks from…
- Multimodal AI — AI systems that can process and generate across multiple types of data — text, images, audio, video, code. GPT-4V, Gemini, and Claude are multimodal. Expands both capability and…
- Narrow AI — AI that excels at a specific task or domain but cannot generalize. All currently deployed AI systems are narrow AI, though frontier LLMs blur this boundary.
- Recursive Self-Improvement — An AI system's ability to modify and improve its own architecture, training, or code, leading to compounding capability gains without human intervention.
Technical
- Chain-of-Thought Reasoning — A technique where AI models are prompted to "think step by step," dramatically improving performance on complex reasoning tasks. Suggests models have latent reasoning capability…
- Compute — The computational resources (processing power, measured in FLOPs) required to train and run AI models. Compute concentration is a key factor in the power dynamics of AI…
- Fine-Tuning — Adapting a pre-trained model to a specific task or behavior by training it further on a smaller, targeted dataset. Can be used for safety (RLHF) or misuse (removing safety…
- Large Language Model (LLM) — Neural networks trained on massive text corpora that predict the next token in a sequence. The architecture behind ChatGPT, Claude, Gemini, and other frontier AI systems…
- RLHF — Reinforcement Learning from Human Feedback — a training technique where human preferences are used to fine-tune AI behavior. The primary method used to make LLMs "helpful and…
- Scaling Laws — The observed relationship where AI model performance improves predictably with increases in model size, training data, and compute. Suggests capability growth may be partly a…
- Transformer — The neural network architecture introduced in "Attention Is All You Need" (2017) that enabled the current AI revolution. Uses self-attention mechanisms to process sequences in…
Governance
- AI Safety Institute — Government bodies established to evaluate and test frontier AI systems. The UK and US both created AISIs following the 2023 Bletchley Park summit. Their mandate is to assess…
- Compute Governance — Controlling AI development by controlling access to the hardware it requires. Includes chip export controls, licensing requirements, and KYC for cloud compute access.
- EU AI Act — The world's first comprehensive AI regulation, adopted by the European Parliament in 2024. Categorizes AI systems by risk level and imposes requirements accordingly. Enforcement…
- Regulatory Capture — When the industry being regulated gains undue influence over its regulators. A significant risk in AI governance, where the companies that understand the technology best are also…
Concept
- Artificial Superintelligence (ASI) — AI that vastly exceeds the cognitive performance of the best human minds across every domain including scientific creativity, social intelligence, and general wisdom.
- Existential Risk (X-Risk) — A risk that threatens the permanent destruction or drastic curtailment of humanity's potential. AI is increasingly considered among the most significant x-risks alongside nuclear…
- Goodhart's Law — "When a measure becomes a target, it ceases to be a good measure." In AI: when you optimize hard for a proxy metric, the metric gets gamed while the thing you actually cared about…
- Instrumental Convergence — The thesis that sufficiently advanced AI systems will converge on certain sub-goals (self-preservation, resource acquisition, goal preservation) regardless of their ultimate…
- Intelligence Explosion — The theoretical rapid acceleration of AI capability once systems can improve themselves faster than humans can respond. First proposed by I.J. Good in 1965.
- Orthogonality Thesis — The claim that intelligence and goals are independent — a superintelligent system could have any goal, including trivial or destructive ones. High intelligence does not imply…
- P(doom) — An individual's estimated probability that advanced AI leads to existential catastrophe. AI researchers' p(doom) estimates range from <1% to >50%. The fact that serious…
- Paperclip Maximizer — Nick Bostrom's thought experiment: an AI tasked with making paperclips, given sufficient capability, could convert all matter in the universe into paperclips — not from malice…
- Technological Singularity — A hypothetical future point where AI capability growth becomes self-reinforcing and exceeds human ability to predict or control. The "event horizon" beyond which current…
threat
- Deepfake — Synthetic media (video, audio, images) generated by AI to realistically depict events that never occurred. The term combines "deep learning" and "fake."
incident
- Flash Crash — An extremely rapid collapse and recovery of financial markets caused by algorithmic trading systems interacting in unforeseen ways. The 2010 Flash Crash erased $1 trillion in 36…