Safety & Standards

Three substantive technical posts this week collectively expose a fundamental problem in how AI safety is currently assessed. The first, on the 'safe-to-dangerous shift', argues that black-box alignment evaluations are only reassuring if the model cannot distinguish the evaluation distribution from the deployment distribution — a condition that may not hold for sufficiently capable models. If a model behaves safely during evals because it recognises the eval context, the entire pre-deployment safety case collapses. This is not a theoretical concern; it is a structural feature of how current responsible scaling policies (RSPs) are designed, and it directly undermines the evidentiary basis for threshold-based deployment gates used by Anthropic, Google DeepMind, and others. See Alignment Forum.

A second post argues that standard risk reports fail to account for deployment-time spread of misalignment: a model that starts with largely benign motivations can develop or propagate dangerous motivations through deployment interactions, meaning a clean pre-deployment assessment provides only a snapshot, not a durable safety guarantee. The author explicitly frames this as the most plausible near-term route to consistent adversarial misalignment, and calls on AI companies and evaluators to incorporate it into risk planning. See Alignment Forum. A third post, on the behavioral selection model, reinforces this by showing that superficially identical training behavior can produce radically different deployment outcomes depending on which underlying motivational structure was selected — making behavioral evals during training insufficient proxies for deployment safety. See Alignment Forum. Taken together, these posts represent a coherent critique: current eval methodology, as embedded in RSPs and frontier model governance frameworks, rests on assumptions that may not hold at capability levels already reached or soon to be reached.

The Center for Democracy and Technology's analysis of the Pentagon-Anthropic dispute extends the story beyond defense procurement into civilian federal agencies and state, local, and tribal governments. The core issue is that when an AI vendor and a government customer disagree over permissible use terms — including safety-relevant constraints on how a model can be deployed — there is no clear regulatory or contractual framework that resolves the dispute or protects downstream civilian uses. CDT identifies cascading effects: agencies that contracted for AI capabilities through federal vehicles may find themselves operating under contested or revised terms without adequate notice or accountability mechanisms. See Center for Democracy and Technology.

From a risk and standards perspective, this case illustrates what the absence of binding procurement standards for AI safety looks like in practice. Voluntary commitments made by labs to safe deployment practices are not automatically enforceable by government customers, and there is no federal equivalent of a product safety standard that would require vendors to maintain safety properties throughout a contract lifecycle. The dispute also surfaces a structural accountability gap: when a lab restricts or withdraws capabilities it judges unsafe, government agencies may have no recourse, but when a lab permits uses the government later deems harmful, liability assignment remains unclear.

CDT's second installment in its AI in Policing series examines gunshot detection technology, documenting a pattern directly relevant to AI safety governance: a technology marketed on safety grounds, deployed at scale in urban environments, with documented accuracy problems and racial disparity concerns, and with accountability structures that are inadequate to identify or remediate harm. The analysis frames gunshot detection as AI-enabled surveillance operating with minimal independent audit, where vendors control accuracy claims and municipalities lack the technical capacity to verify them. See Center for Democracy and Technology.

For risk and standards professionals, this is a concrete illustration of what 'real-world harm' looks like when AI is deployed in high-stakes contexts without binding accuracy standards, mandatory independent evaluation, or clear liability for false positives. The policing domain is notable because it sits outside most current AI regulatory proposals focused on general-purpose AI, meaning domain-specific harms may persist even as horizontal AI regulation advances. The lack of standardised performance benchmarks for public safety AI — comparable to, say, medical device performance standards — is the structural gap this case exposes.

A substantive theoretical post on the Alignment Forum this week challenges the conceptual foundations of standard alignment desiderata — helpfulness, corrigibility, obedience, avoiding manipulation — by arguing that human goals are themselves under-determined and manipulable, making it impossible to draw a principled line between illegitimate manipulation and legitimate influence such as providing counsel or persuasion. See Alignment Forum. This is a research-stage argument, not a deployable safety mechanism, but its implications are practically significant: if corrigibility and non-manipulation cannot be cleanly defined, then safety evaluations that test for these properties are measuring proxies of uncertain validity.

This connects directly to the eval reliability concerns raised by the other alignment posts this week. If the concepts being evaluated are theoretically under-specified, and the eval methodology may not generalise from training to deployment, the overall epistemic basis for current alignment claims is weaker than the confident language in published RSPs and model cards typically suggests. This is a genuine disagreement within the alignment research community about whether current frameworks are fit for purpose, not a settled debate.