Open Problems in Frontier AI Risk Management

Establishing the scope and context involves defining the intended application of an AI system, including its functional domains, policy sectors of application, operating conditions, and system boundaries. It also requires specifying relevant aspects of the internal context, such as the AI models and data used, relevant features of the organisation, and the external context of deployment, including downstream uses and broader socio-economic or political factors. Below, we review some of the open problems related to this step.

Intended (and unintended) use of the system. Key in establishing the scope is making explicit the intended use of the system (ISO/IEC, 2014). This step is emphasised in multiple frameworks (e.g., ISO/IEC, 2023, NIST, 2023), with  NIST’s AI Risk Management Framework (RMF) asking organisations to specify why the system is being developed, the system-specific features and requirements, who the relevant stakeholders are, and what context-specific settings will matter (NIST, 2023). NIST’s AI RMF also refers to ‘use-case profiles’ to create sector-specific (e.g., finance, healthcare) guidance where usage and context vary (NIST, 2023). In the case of frontier AI, mapping out intended uses will require capability- and modality-specific considerations (NIST, 2024). Additionally, for risk management activities to be sufficiently responsive to possible frontier AI risks, decision-makers must look beyond intended uses (Boine & Rolkin, 2023; Gailmard et al., 2025; Galinkin, 2022; Mylius, 2025). Scope consideration thus also requires determination of unintended uses as well as ‘foreseeable misuses’ of AI systems, which could lead to intended or unintended harms. Frameworks for anticipating misuse remain immature, particularly for frontier systems whose capabilities depend on context, prompting, tool integrations, and downstream system interactions (Campos et al., 2025; Raman et al., 2025; Anderljung et al., 2023; Bengio et al., 2024). A growing number of systematic taxonomies of risks and harms is available (Shelby et al., 2023; Slattery et al., 2024; Bagehorn et al., 2025), with increasing attention to societal, user-centered, and technically-oriented risks, as well as taxonomies focused on individual contexts, like chatbots, mental health, or deepfakes (Coeckelbergh, 2025; Bird et al., 2023; Zhang et al., 2025). Additional methods exist to support consideration of adversarial uses (Nganyewou Tidjon & Khomh, 2022; Kumar et al., 2019;  Perez et al., 2022). While documentation practices such as Model Cards (Mitchell et al., 2019) and emerging System Cards (OpenAI, 2024) attempt to codify intended uses and limitations, adoption is uneven (De Laat, 2021; Bommasani et al., 2024), and most organisations do not systematically enumerate misuse scenarios.

Boundary determination. Determining boundaries entails clarifying what assets, processes, or activities are covered by the risk management framework, taking into account the internal and external context of the organisation. In principle, this boundary is also closely tied to the AI system operational boundaries. AI risk management recognises this, with standards like ISO 42001:2023 suggesting that the organisation determines its role also relative to the AI system, including roles such as AI providers (e.g., platform, product and service providers), AI producers (e.g., AI developers), AI customers (e.g., AI users), to cite a few (ISO, 2023). Frontier AI, however, blurs the boundaries between these roles, both with some actors being both providers and producers (e.g., Microsoft, Google) and with AI systems themselves being re-used (e.g., fine-tuned) across sectors, challenging the very distinction between users and producers. Limited guidance is provided on how to draw boundaries in practice for composite, vendor-integrated, or highly modular systems, as it is the case for frontier AI. Despite calls for greater attention to supply-chain dynamics (Balayn et al., 2025; Widder & Nafus, 2023) and proposals such as AI bills of materials (e.g., BSI & ACN, 2025), there is no standardised way to enumerate dependencies or interface responsibilities during scope-setting. This creates blind spots at integration points and weakens downstream accountability.

Classification regimes as heuristics. Classification regimes aim to facilitate setting the scope by grouping AI systems into categories. Classification efforts found in regulatory approaches (whether focused on parsing AI tools by sectoral context, such as in the UK policy context, or by risk levels, such as in the EU AI Act (Roberts et al., 2023)) can streamline placement of an AI system into a relevant bucket, fast-tracking scoping decisions about relevant uses, plausible unintended uses, boundaries, and so on. However, the utility of classification regimes depends on the quality and robustness of the underlying classification schema (Mokander et al., 2023). Poorly specified categories may lead to inappropriate scoping decisions, particularly if systems are misclassified as low risk or narrowly associated with a single regulatory domain. Classification schemes may also be vulnerable to strategic behaviour, as actors seek to game thresholds or definitions to reduce regulatory burden (Mokander et al., 2023; Tlaie, 2024; Veale & Borgesius, 2022). These challenges are amplified for frontier AI systems, which are general-purpose, and thus not easily captured in one category, and deployed across jurisdictions with emerging, divergent or absent regulatory approaches (Roberts et al., 2023). Overlapping AI-specific and sectoral regimes further complicate organisational scoping decisions, making the establishment of a single, coherent risk management approach contentious. While ‘crosswalk’ exercises can help organisations navigate regulatory fragmentation (NIST, 2023), the absence of more comprehensive harmonisation or guidance leaves significant uncertainty about how scope should ultimately be defined or constrained.

Stakeholders and affected parties. Beyond defining the intended purpose and deployment context, scope-setting involves identifying relevant stakeholders (Deshpande & Sharp, 2022). Current AI risk management standards emphasise broad stakeholder identification. ISO/IEC 23894:2023, for example, highlights customers, partners and third parties, suppliers, end users, regulators, civil society organisations, affected communities, and society at large (ISO/IEC, 2023). IEEE 7010-2020 requires identifying both directly and indirectly impacted groups (IEEE, 2020). In the example of autonomous vehicles, for example, the standard lists users, drivers, pedestrians, ridesharing companies, taxicab unions, urban planners, parking enforcers, safety inspectors, transportation regulators, and disability advocates, each with potentially unique impacts to consider (IEEE, 2020). Along these lines, the OECD’s framework for the classification of AI systems (OECD, 2022) encourages not just looking at an AI model or associated outputs, but also understanding underlying data and input, economic context, and even considerations of ‘people and planet.’ Impacts can thus pertain not only to individuals, but also communities, social groups, and ecosystems. Stakeholder identification is thus closely tied with understanding AI systems’ penetration within and beyond their immediate context. 

In the case of Frontier AI, this task is specifically challenging as it can be unclear what the downstream implications of an AI system are, or what impacts will be borne by users or other stakeholders. Several frameworks and tools aim to support stakeholder-focused scoping through structured impact assessments. IEEE 7010-2020 encourages internal and external consideration of possible impacts and stakeholder engagement prior to setting objectives (IEEE, 2020), NIST AI RMF prescribes mapping impacts to individuals up to communities and society as a whole (NIST, 2023) and tools like HUDERIA encourage stakeholder-based AI impact assessment (Council of Europe, 2024). Yet, the dimensions for assessment remain open and underspecified (e.g., well-being, fairness, privacy, safety), creating confusion around which one to apply and their implementation in practice (Deshpande & Sharp, 2022). Finally, while many frameworks urge participatory scoping, methods for involving affected stakeholders systematically remain limited or underutilised (Young et al., 2024), despite evidence that stakeholder engagement shapes fairer and more context-sensitive problem formulations (Hanwen Deng et al., 2025).

Closely connected to setting the scope and context is the determination of clear and actionable objectives, needed to underpin choices throughout AI risk management, both at the level of the organisation and of the AI system. Below, we review some of the open problems in setting objectives.

Specifying objectives. Setting objectives entails specifying what the risk management process is intended to achieve. In current standards, objectives are understood to be multiple and context-dependent, and may differ in tenor: some organisations emphasise regulatory compliance, while others prioritise prosocial or economic goals. Safety risk management emphasises the importance of setting safety goals, such as that no single failure shall cause loss of life (e.g., FAA, 2024). Organisational standards such as ISO/IEC 42001:2023 lists objectives ranging from accountability and transparency to ensuring sufficient AI expertise (ISO/IEC, 2023). Depending on their breadth and specificity, articulated objectives shape downstream risk management choices, as each objective can imply different priorities, trade-offs, and evaluation criteria (Schiff, 2025; Alvarez, 2025). Objectives may also differ in their status, with some being legally binding and others aspirational, and may be pursued through programmatic risk management activities or higher-level organisational decisions (Manning, 2017). For frontier AI, defining organisational objectives is particularly challenging given the multi-purpose nature of the technology and given that it might lead to breakthroughs (e.g., scientific discoveries) that might change an organisation’s previously-set objectives (Renieris et al., 2024). Still, identifying a set of first principles is important to ensure adaptation to technological advancements and avoid approaching unexpected changes ad hoc (Renieris et al., 2024). 

Timing objectives. Setting objectives also entails determining when objectives should be specified and revised over the lifecycle of an AI system, which can introduce its own challenges (Logan et al., 2021). Existing frameworks often treat scope identification and objective setting as early-stage planning activities (e.g., ISO/IEC, 2023), while simultaneously recommending continuous and iterative refinement through feedback loops. For example, the NIST AI RMF’s Govern and Measure functions emphasise ongoing iteration (NIST, 2023), and IEEE 7010-2020 encourages multiple internal feedback loops throughout (IEEE, 2020). For frontier AI, this tension between early specification and continuous revision is particularly pronounced (NIST, 2023). The adaptable and updatable nature of frontier AI systems raises questions about when organisations can reasonably be confident that they have sufficient information to set objectives in the first place. Moreover, objective setting is conceptually and procedurally entangled with scope-setting and findings from risk identification and risk analysis, making it difficult to treat objectives as fixed inputs determined a priori. 

AI System-level objectives. Beyond high-level objectives, setting objectives also entails specifying desired properties and behaviours of the AI system itself. Standards such as the NIST AI RMF MAP function emphasise eliciting system requirements (e.g., privacy) and making design decisions that account for socio-technical implications in addressing AI risks (NIST, 2023). In practice, developers of AI systems formulate detailed specifications for how models should behave, which guide the work of technical teams and data annotators when training, evaluating, and refining systems. In the case of frontier AI, this has raised questions over what set of human values AI systems should be aligned with (Korinek and Balwit, 2022; Lazar, 2023) especially when value alignment may be skewed by either the often limited diversity of companies’ teams relative to the world at large or, corporate profit motives (e.g., Abdulla and Chahal, 2023; Singh et al., 2024; Maslej et al., 2025). To mitigate these kinds of problems, some researchers have proposed paradigms for designing AI systems that balance competing views (Kirk et al., 2023; Sorensen et al., 2023; Sorensen et al., 2024; Caputo, 2024; Ali et al., 2025). Inspiration can also be taken from how other institutions develop normatively-accepted ways to choose what principles to align with (e.g., democracy) (Gabriel, 2020). However, other researchers have expressed concerns about ‘pluralism washing,’ arguing that technical approaches to pluralistic alignment fail to address deeper problems related to systematic biases and social power dynamics in the AI ecosystem (Kalluri, 2020; Dobbe et al., 2021; Birhane et al., 2022; Sloane et al., 2022). 

An important step before proceeding with the process of risk assessment is to set criteria for decisions which will be taken later on in the process. This includes criteria for deciding whether risk is acceptable, criteria for measuring risk, and criteria for deciding between options (e.g., deployment decisions). Below, we review open problems for these steps. 

Setting risk acceptance criteria. Setting risk acceptance criteria entails defining how much risk is considered acceptable and on what basis such judgments are made (ISO, 2018; ISO/IEC, 2014). In established risk management standards, these criteria are typically grounded in explicit methods for determining tolerable risk levels. IEC 31010:2019 describes techniques such as Risk Bearing Capacity (RBC), which specifies how much risk an organisation can take without jeopardising its stability or long-term goals, and ALARP/ALARA principles, which require safety-related risks to be reduced ‘As Low As Reasonably Practicable’ or ‘As Low As Reasonably Achievable’ (IEC, 2019). Risk acceptance criteria in the safety-critical industry evaluate the extent of harm, expressed as ‘severity x likelihood’, from potential accidents or losses on people, society, infrastructure, etc., within a specific intended context. For instance, as expressed by (Koessler et al., 2024), in the U.S. aviation industry the probability of ‘failure conditions which would prevent continued safe flight and landing’ should not exceed 1 x 10−9 (one in a billion) per flight-hour (FAA, 1988). In the UK nuclear industry, the risk of death is ‘unacceptable’ above 1 x 10−4 per plant-year and ‘broadly acceptable’ if below 1 x 10-6 per plant-year (ONR, 2020).

In frontier AI, by contrast, risk acceptance criteria are most often operationalised through capability thresholds, defined as levels of system performance or capability that trigger specific mitigation measures (Campos et. al 2025). These thresholds use capabilities as a proxy for risk rather than expressing risk directly in terms of likelihood and severity of harm. While some frameworks link capability thresholds to generalised harm scenarios (Meta, 2025; OpenAI, 2025), this approach primarily captures the possibility of harm and often neglects contextual and external factors, such as the threat landscape, that can influence both the scenario leading to the harm as well as risk levels (Koessler et. al. 2024, Caputo et al. 2025). As a consequence, risk mitigation measures become harder to justify and calibrate because they are no longer directly tied to concrete accident or harm scenarios. To address these limitations, recent work offers guidance on the way capability thresholds can be operationalised more robustly, by using quantitative risk tiers and scenario-based risk modelling (Caputo et al. 2025; Koessler et. al. 2024; FMF, 2025; Wisakanto et al. 2025, Murray et al., 2025). Still, capability-centered criteria remain poorly suited for risks that are not revealed through model performance alone, such as risks to fundamental rights. While some propose harm modelling and associated thresholds for risks including toxicity, deception, discrimination and socioeconomic harms (Raman et al., 2025), others highlight that significant conceptual and practical challenges remain in translating such harms into risk management terms (Yeung, 2025). Others propose that these should be informed by input from the public (Choi & Rogers, 2025), such as via multistakeholder panels (Schuett et al. 2025). 

Conceptualising risks and establishing baselines. Traditionally, safety-oriented risk management conceptualises risk in terms of the severity and likelihood of harm or consequences (ISO, 2014), and distinguishes between inherent risk (the level of risk before mitigations) and residual risk (the level remaining after mitigations) (ISO, 2009). Most mature safety-critical industries rely on these concepts to evaluate whether systems meet predefined acceptability criteria within a stable baseline. In frontier AI, by contrast, many organisations increasingly rely on the concept of marginal risk, defined as the difference in risk relative to a chosen baseline (Williams et al., 2025). Similar approaches exist in other domains, such as European transport regulation, which uses the GAMAB principle (‘Globalement Au Moins Aussi Bon’) to require that new systems introduce no more risk than the existing state of the art (Tchiehe & Gauthier, 2017). Frontier AI developers, however, adopt heterogeneous baselines, including comparisons to human performance (Wei et al., 2025), earlier states of the world (Alaga & Chen, 2025; FMF, 2025), a company’s own previous model (AISI, 2024), or competitors’ models (Williams et al., 2025). This widespread and inconsistent use of marginal risk introduces several challenges specific to frontier AI. Reliance on shifting baselines, particularly in a post–general-purpose-AI context, may enable a ‘boiling frog’ dynamic in which absolute risk increases across the ecosystem without being detected (Alaga & Chen, 2025). The absence of a shared baseline across organisations (FMF, 2025), combined with competitive benchmarking against peers’ models (Williams et al., 2025), may further amplify this effect while fostering a false sense of safety. There also remains the question of the marginal risk posed by open foundation models compared to closed models and non-AI sources for which Kapoor et al. (2024) present an initial risk assessment framework. 

Criteria to decide between multiple options. Criteria for deciding between multiple options determine how organisations choose among alternative courses of action (IEC, 2019), such as deployment, delay, restriction, or additional mitigation going beyond or, where relevant, complementing risk acceptance criteria. In practice, organisations often face decisions in which multiple, competing objectives are at stake and both potential harms and benefits must be weighed. Risk management standards describe several decision-support techniques for such contexts. Cost–benefit analysis (CBA) is a prominent approach, evaluating options based on expected financial or utility losses and gains (IEC, 2019). Other techniques include decision-tree analysis, which represents the utility of decisions in a structured, sequential form (Kirkwood, 2002), and multi-criteria decision analysis, which allows multiple criteria to be weighted and compared simultaneously (Velasquez & Hester, 2013). Under conditions of deep uncertainty, methods such as Robust Decision Making (RDM), widely used in climate and disaster risk assessment, stress-test candidate options across many plausible futures and evaluate them against multiple success metrics (Dittrich et al., 2016). For frontier AI, however, the suitability of these techniques remains largely untested and highly context-dependent. Some frontier AI frameworks make limited reference to cost–benefit considerations; for example, by pairing risk assessments with benefit assessments (Meta, 2025) or explicitly weighing risks and benefits when defining deployment standards (Anthropic, 2025). Yet CBA is poorly suited to safety-critical, high-uncertainty environments and struggles to account for public-good impacts and non-quantifiable harms (IEC, 2019), which are central concerns in frontier AI governance. 

A key aspect in the process of risk identification is to identify risk sources related to the development and use of frontier AI. This should be informed by and in line with the defined scope and context (1.1 Establishing the Scope and Context) and objectives (1.2 Setting Objectives) presented above. Below, we review open problems related to the steps involved.

Considering AI risk sources. Considering AI risk sources entails identifying the elements that can give rise to risk within an AI system and its operating context. In risk management, a risk source is defined as an element that alone or in combination has the potential to give rise to risk (ISO, 2022, 3.3.10). Hazards constitute a specific class of risk source characterised by an inherent property, a condition or a state that can cause harm if realised or activated (ISO, 2022. 3.3.12), such as high voltage or pathogenic agents. Other risk sources do not possess inherently harmful properties but generate risk through structural, behavioural, informational, or organisational characteristics, such as ambiguous decision authority, poor data quality, mis-specified system objectives. AI-related standards identify a wide range of such sources, including environmental complexity, lifecycle and hardware issues, lack of transparency, and technological readiness for a given application context (ISO/IEC, 2023, Annex B). Frontier AI systems introduce additional and distinct risk sources. Regulatory frameworks on frontier AI increasingly emphasise model capabilities (e.g., offensive cyber capabilities), model propensities (e.g., hallucination), model affordances and so-called ‘other systemic risk sources’ (e.g., model configurations, model properties and context) (European Commission, 2025). The NIST Generative AI Profile further highlights human behaviour and human–AI interaction as critical sources of risk, including misuse, abuse, and unsafe repurposing (NIST, 2024). Complementing these approaches, repositories of AI-related threats such as the MITRE ATLAS Matrix catalogue AI-specific attack tactics across the system lifecycle (MITRE, n.d.), enabling identification of actors, assets, and conditions that may give rise to harm. Notwithstanding these resources, however, many frontier AI companies’ safety frameworks focus predominantly on model capabilities, such as CBRN and AI R&D or ‘AI self-improvement’ (Anthropic, 2025; OpenAI, 2025) and propensities, such as sandbagging or undermining safeguards (OpenAI, 2025), with less systematic attention to affordances and deployment context as risk sources. This can obscure how configuration choices, access modalities, or integration pathways shape real-world risk, which is a key aspect of risk identification. 

Selecting techniques for identifying AI risk sources. Selecting techniques for identifying AI risk sources entails choosing structured methods to surface relevant sources of risk across a system’s design, operation, and context. Classical risk management standards and documents describe several such techniques (IEC, 2019; Ericson II, 2005). Hazard Identification (HAZID) is a structured brainstorming technique, typically applied early in a project, that aims to identify hazards, but also aims to elicit potential initiating events, high-level consequences, and existing or proposed controls to provide contextual information for subsequent risk assessment (Golwalkar & Kumar, 2022). Hazard and Operability Studies (HAZOP) systematically examine a system or process by identifying potential deviations from design intent and their possible causes and consequences using predefined guidewords (e.g. ‘no’, ‘more’, ‘less’) applied to various parameters related to the system  (IEC, 2019; Mocellin et al., 2022). The Structured What-If Technique (SWIFT) similarly relies on guided brainstorming, using a predefined set of guidewords (e.g. timing, amount, etc.), combined with ‘what if’ questions to identify known risks, risk sources, and existing or proposed controls (IEC, 2019; Potts et al., 2014). Complementary backward-chaining techniques where the sources are identified through the events or the consequences, include Ishikawa (fishbone) analysis, which works backward from an identified event to surface possible causes by depicting the causes as the ‘bones’ of a ‘fish’ with the ‘head’ as the event (IEC, 2019), and Failure Mode and Effects Analysis (FMEA), which decomposes a system into elements and examines their failure modes, causes, effects, and associated controls (IEC, 2019). 

For frontier AI, applying these techniques presents distinct challenges. The complexity, general-purpose nature, and non-linear interactions characteristic of frontier systems can strain methods originally developed for bounded, deterministic systems. Recent work proposes adaptations such as aspect-oriented hazard analysis, using first-principles taxonomies of AI system aspects, such as capabilities, domain knowledge, and affordances, to identify critical risks through the system’s characteristics, context, and guided risk pathway threat modelling (Wisakanto et al., 2025). Koessler and Schuett (2023) cite the fishbone (Ishikawa) method as useful for identifying frontier AI risk sources in highly uncertain contexts by working backward from possible consequences, while cautioning that it is ill-suited for risks involving non-linear interactions such as competitive dynamics. They also recommend the use of risk taxonomies to reduce blind spots and foster a shared understanding of the risk landscape across stakeholders (Koessler & Schuett, 2023). Currently, there are several taxonomies or repositories of AI risks, such as AI Risk Categorisation Decoded (Zeng et al., 2024), the AI Risk Repository (Slattery et al., 2024), OWASP Top 10 Risk & Mitigation (OWASP, 2025), and AI Risk Atlas (Bagehorn et al., 2025). However, they also note that taxonomies are time-consuming to develop and may convey a misleading sense of completeness in the absence of empirical data (Koessler & Schuett, 2023). 

Beyond, and sometimes alongside, identifying risk sources, it is key that one also identifies relevant elements like events, controls and consequences. We review open problems for each in turn below.

Identifying potential events. In risk management, an event is defined as an occurrence or change in a particular set of circumstances, and may have multiple causes and consequences (ISO, 2022. 3.3.11). Classical techniques such as HAZID, HAZOP, and SWIFT support this step by moving beyond hazard identification to outline credible events that hazards may lead to, mapping initiating conditions to system deviations, failures, or losses  (IEC, 2019; Ericson II, 2005). Risk management standards for AI systems further recommend drawing on sources such as market data or incident reports on similar systems, usability studies, and interviews and reports from internal and external experts to inform event identification (ISO/IEC, 2023). For frontier AI systems, identifying potential events is particularly challenging due to limited historical experience and rapidly changing deployment contexts. There are a set of AI incident databases which can be useful in this respect. These include the AI Incident Database (AIID, n.d.) and the AI Incidents and Hazards Monitor (OECD, n.d.). These historical incidents serve as a reference for potential events should the identified risks not be appropriately managed. Nevertheless, as many of these databases are built from voluntary incident reports, they are not representative of the full range of potential events, as the events reported may be those that receive the most public attention instead of having the highest probability or severity. Furthermore, just as past performance in financial markets may not be indicative of future performance (Kahn & Rudd, 2019), historical AI incidents will necessarily not be reflective of future incidents that have yet to occur. 

Identifying controls. Controls are defined as measures that maintain and/or modify risk (ISO, 2022. 3.3.33). In the safety sense, this term is understood as focused on hazard elimination and risk reduction (risk mitigation) (ISO/IEC, 2014), and it identifies and documents controls relevant to understanding the risk. Classical risk management techniques such as HAZID, HAZOP, SWIFT, and FMEA are used to identify controls as part of its process. As failure modes and causal pathways are identified, these methods also outline planned controls that are part of the system’s design to mitigate these pathways to harm. In the context of frontier AI risk management, there are also existing repositories of AI-related controls such as the MIT Risk Mitigation Taxonomy (MIT, n.d.) the Secure AI Framework (Google, n.d.), and other academic sources (Gipiškis et al., 2024). Beyond merely listing controls, risk management standards also require that their operating effectiveness be taken into account, including the possibility of control failures (ISO/IEC, 2023). In traditional risk management, stating the effectiveness of controls is relatively tractable, as controls are often well-established with their limitations well-understood by practitioners through accumulated operational experience. For frontier AI, however, the evidence base for control effectiveness is severely limited. Many controls, such as those described in 5. Risk Mitigation, have been developed only recently, and some lack any sustained track record of deployment in real-world conditions. Assessing their operating effectiveness therefore involves substantial uncertainty, which means that controls identified for frontier AI systems often cannot be documented with the same confidence in their reliability as controls in more established domains.

Identifying consequences. Consequences are outcomes of an event affecting objectives (ISO, 2022. 3.3.18), where the objectives would be set in the previous phase as discussed in Section 1.2 Setting Objectives.  Classical risk management techniques for consequence identification closely overlap with those used for hazard and event identification, since identification of hazards naturally yields their associated events and consequences. Additionally, scenario analysis, which covers a range of techniques that involve developing models of how the future might turn out, can also be used as a forward-chaining technique to identify consequences  (IEC, 2019). This can include extrapolating past trends for the short-term and building imaginary but credible scenarios for the long-term  (IEC, 2019). This technique is also useful to inform risk analysis techniques later on, such as risk modelling (Section 3.3 Severity of Consequences and Likelihood). For frontier AI systems, identifying consequences is particularly challenging due to the fast-pace of technological change and limited empirical understanding of real-world impacts. Recent efforts, such as AI-related scenario exercises developed by the Department for Science, Innovation and Technology in the UK (DSIT, 2024), illustrate a wide spectrum of possible social, economic, technological, and environmental consequences, ranging from incremental adoption effects to catastrophic global outcomes. However, such scenario analyses are subject to significant uncertainty, which is amplified by the fact that frontier models are often assessed early in their lifecycle, while many consequential harms only emerge downstream when models are integrated into applications and deployed at scale (Touzet et al., 2025).

In order to further comprehend the nature of risk, we here review approaches that contribute to gathering relevant information about the nature of risk and its relevant characteristics referred to in Section 2 Risk Identification (e.g., risk sources or hazards, etc.) as available to model developers through internal sources and methods.

Assessing an AI system’s properties. Assessing an AI system’s properties entails analysing internal characteristics of the system that are relevant to understanding how identified risk sources, events, and impacts may arise. In risk management, this step typically involves gathering information about the type and significance of risk sources and analysing potential consequences through methods such as impact assessments, including assessments of the intended effects of AI development or use on individuals or society (ISO/IEC, 2023). These analyses provide internal information to eventually determine the severity and likelihood of risk (Section 3.3 Severity of Consequences and Likelihood). In frontier AI practice, this step is predominantly operationalised through capability assessments or ‘model evaluations.’ Frameworks such as the NIST Generative AI Profile (2024) and the EU General-Purpose AI Code of Practice (2025) emphasise evaluations as a primary means of analysing model or system properties. Capability assessments are currently used to determine whether capability thresholds have been crossed and safeguards should be implemented (OpenAI 2025, Anthropic 2025, DeepMind 2023); whether the system possesses vulnerabilities that could enable harmful misuse (Carlini et. al 2024, Anil et al. 2024, Sharma et al. 2025); or whether the underlying model demonstrates undesirable propensities, values, bias or discriminatory tendencies (Greenblatt et al. 2024, Solaiman et al. 2024, Meinke et al., 2025; Huang et al 2025). Evaluation results are thus used to guide a broad range of risk-management-relevant decisions such as deploying technical mitigations, restricting access, delaying deployment, or pursuing further investigation. Capability assessments, however, focus on what models can do, rather than directly measuring the risk that it poses (Koessler et al., 2024). As a consequence, caution is needed when interpreting and applying evaluation results for risk management as they alone cannot express risk. Additionally, even when applied correctly, there is still limited guidance on how evaluation outputs should be integrated into broader risk models in order for them to correctly inform a determination of risk (Yu et al., 2026).  

Ensuring quality, coverage, and robustness. Ensuring quality, coverage, and robustness entails establishing that assessments used to analyse AI system properties are methodologically sound, sufficiently comprehensive, and reliable inputs to risk management decisions. While such assessments have improved in quantity and quality over the last few years, there remains a significant shortage of widely adopted and sufficiently high quality ones, and best practices are still evolving (Apollo, 2024). Many existing assessments vary substantially in their methodological rigor, scope, and reproducibility (Reuel et al., 2024; Paskov et al., 2025). Frontier AI systems amplify these challenges. Assessments might not elicit or represent a system’s full capabilities. For instance, evaluations might assess models with less access to inference time compute, fewer attempts, or less effective tools and scaffolding than future deployments of the system might realistically have access to (Barnett and Thiergart 2024, Volk et al. 2025, Götting et al., 2025). It is also challenging to develop capability assessments that are robust for systems with increasing capabilities (Summerfield et al 2025, McKee-Reid et al. 2024, METR 2025), given that these might undermine assessments’ accuracy (Greenblatt et al. 2024), particularly in more agentic scenarios (Anthropic 2025). Assessments’ results may also be highly sensitive to small differences in prompting and implementation (Schaeffer et al. 2023; Biderman et al., 2024; Burden, 2024; Robinson and Burden 2025, Sun et al. 2025), thereby hampering their reliability and reproducibility. Taken together, these issues complicate comparisons across models and over time, limit confidence in reported results, and weaken the role of evaluations as stable inputs into downstream risk analysis. This is made even more difficult by a lack of detail and comprehensiveness in how results of safety evaluations are reported publicly (McCaslin et al 2025; Wei et al. 2025; Paskov et al. 2025).

Linking results to real-world behaviour and harms. Linking evaluation results to real-world behaviour and harms entails assessing whether measured model capabilities and propensities reliably predict how AI systems will perform and affect outcomes in real-world settings. While current research has made progress, building robust predictive models of AI system capabilities (Zhou et al. 2025, Hofstätter et al., 2025), and how these might map to real-world scenarios and harms (Barnett and Thiergart 2024, Mukobi 2024) remains difficult, even more so for frontier AI whose behaviour and real-world impacts are even less predictable. Real-world tasks often involve fuzzy objectives and hard-to-measure outcomes. Increasingly with frontier AI, these might entail complex and interactive environments, which evaluations struggle to approximate (Phan et al. 2025) thus limiting their external validity, e.g., their ability to support inference about real-world impacts. Moreover, most current assessments focus on isolated models rather than interactive or multi-agent environments, which remain underdeveloped (Hammond et al 2024, Agent Village, n.d.), and evaluations involving representative numbers of human participants are uncommon beyond basic red-teaming exercises. Frontier AI company assessments also often underinvestigate the effects of directly uplifting human capabilities, including in crucial, high-risk domains such as cyber offense (Righetti 2024). Finally, more realistic evaluations are frequently slow and costly, especially when they require large numbers of qualified participants or substantial compute and, particularly in national-security-relevant domains, may pose security risks. 

Beyond gathering internal information, it is also important to engage with external actors or implement mechanisms to gather external information to inform a better understanding of identified risks. This is not only important to gather more information, but also to indirectly validate or stress-test the information gathered so far (e.g., capability assessment results), a key step in risk management (IEC, 2019). This step can also help identify new risks, trigger another round of risk assessments or revise the plan for risk management. 

External assessments. External assessments involve independent actors to verify, validate, or scrutinise an organisation’s internal risk assessment. Risk management standards reference such engagement at a high level, for example through communication with or participation of external stakeholders (ISO, 2018; ISO/IEC, 2023), and ISO/IEC 31010 explicitly calls for ‘independent review processes’ to verify and validate risk analysis (IEC, 2019, 6.4.1). Regarding frontier AI, both NIST AI RMF Generative AI Profile (2024) and the EU GPAI Code of Practice (2025) require the involvement of independent external assessors under specified conditions. These assessments, often referred to as external model or system evaluations (Xia et al., 2024; Sharkey et al., 2023) or technical audits (Kluge, 2023), typically focus on technical functionality, performance, or specific risk domains such as bias or chemical, biological, radiological, and nuclear (CBRN) risks (Brundage et al., 2026). While external evaluations are increasingly common in frontier AI risk management, there is little consensus on appropriate protocols, the scope of assessment, or how results should be disclosed and acted upon (Cattell et al. 2025, Lam et al., 2024). A central difficulty concerns the depth of access granted to external assessors (Kembery et al. 2024, Homewood et al. 2025). Proposals for ‘structured access controls’ suggest calibrating the level of access to the risk, enforced through technical controls ranging from API sampling to secure enclaves that enable parameter-level verification (Shevlane 2022, Bucknall and Trager 2023). However, it remains unclear how to strike a balance between meaningful access and preventing misuse, theft, or disclosure of sensitive materials. Black-box or output-only audits are insufficient for rigorous safety assessment (Casper et al. 2024), with studies suggesting that such testing under-detects risks (Che et al., 2025), while deeper forms of access (to training data and deployment information, up to the system’s inner workings or internal model representations) may be necessary to reliably interpret model behaviour (Casper et al. 2024) yet may raise security concerns for developers. Importantly, the resulting problem of ‘mutual privacy’, protecting both proprietary designs and the results of independent testers, remains broadly unexplored despite proposals for secure technical environments, encrypted test setups, and contractual safeguards (Bucknall et al., 2025). 

Ensuring structural independence. Ensuring structural independence entails designing external evaluation processes so that assessors can scrutinise AI systems with sufficient independence from developers. Risk management standards generally refer to independent review at a high level (IEC, 2019, 6.4.1, as above) but provide limited guidance on how to secure genuine independence in practice. Studies of early audit practices show that many assessments described as ‘external’ were in fact hybrid forms, in which the developer selects and finances the assessor, defines the scope, and may be able to veto the publication of negative results (Raji et al., 2022). Sandoval and Jing (2025) point out that these conditions can also shape methodology and timelines, favoring faster, scalable, quantitative techniques over slower socio-technical investigations, echoing well-documented conflicts of interest in financial auditing (Moore et al., 2006). Frontier AI heightens these concerns because evaluations are costly, technically complex, and often dependent on privileged system access. Proposals to mitigate capture dynamics include independent audit committees, public declarations of conflicts of interest (Lam et al, 2024), public or pooled funding of audits, regulator-managed auditor lists with rotating assignments to reduce familiarity bias, and a legal right to publish critical findings (Raji et al., 2022). In the context of AI evaluations specifically, some have proposed legal and technical safe-harbour agreements to protect good-faith evaluators from legal retaliation or the threat of account suspension (Longpre et al., 2024). However, operationalising such safeguards raises unresolved questions about enforcement, incentives, and governance. Furthermore, the lack of specialised expertise, computational infrastructure, and reproducible methods means that the ability to conduct rigorous external evaluations remains concentrated in a handful of well-resourced organisations, raising further concerns about the independence and diversity of oversight (Costanza-Chock et al. 2022; Busuioc 2022; Anderljung et al. 2023).

Post-deployment assessments. Post-deployment assessments entail monitoring and reviewing systems after deployment to compare real-world outcomes with prior risk analyses and to identify emerging risks. Some risk management standards emphasise ongoing monitoring and periodic review as transversal elements of risk management (ISO, 2018; ISO/IEC, 2023), while others highlight the importance of comparing predicted and actual outcomes with regards to risk analysis more specifically (IEC, 2019, 6.4.3). In the context of frontier AI, both the NIST AI RMF Generative AI Profile (2024) and the EU GPAI Code of Practice (2025) explicitly require post-deployment or post-market monitoring as part of risk analysis, with frontier AI posing several challenges in this respect. In practice, post-deployment monitoring for frontier AI systems draws on three main categories of information: (1) model integration and usage data, (2) application-level usage data, and (3) impact and incident data (Stein et al., 2024; Pratt and Tanjaya, 2025). 

Model integration and usage data describe where and how models are deployed, disaggregated by sector, geography, and application type, but are currently limited and often reconstructed from voluntary self-reports, surveys, national accounting frameworks, and public registries (Tamkin et al., 2024; Lopez et al., 2025; Highfill et al., 2025; Municipality of Amsterdam, 2026). Some model developers, such as Anthropic and OpenAI, have released high-level anonymised usage statistics, (Anthropic, 2026; Tamkin et al., 2024; OpenAI, 2025), but disclosure is overall limited (Bommasani et al, 2024). Application-usage data encompasses information about how users interact with applications deploying models in the real world, and could include AI application and AI agent activity logs or indices (MIT, 2025), user feedback channels, and coordinated flaw reporting by application developers (Catell et al., 2024; NIST, 2024; Chan et al., 2024). These practices, however, are inconsistently applied, rarely standardised, and may raise unresolved privacy concerns  (Stein and Dunlop, 2024; Pratt and Tanjaya, 2025). Coordinated mechanisms for reporting application flaws to centralised bodies could help regulators and civil-society organisations identify patterns of failure and target interventions (Longpre et al., 2025; Gailmard et al., 2025; Richards and Zilka, 2025). Impact and incident data capture real-world harms and broader social or economic effects,  including reports of adverse incidents and sociotechnical field evaluations (Agarwal et al., 2024). Voluntary repositories such as the AI Incident Database (AIID, n.d.) and OECD AI Incidents Monitor (OECD, n.d.) mirror practices in other safety-critical sectors (e.g., aviation and cybersecurity) but lack interoperability and mandatory participation (Turri and Dzombak 2023; Stein et al., 2024; McGregor, 2021; Agarwal and Nene, 2024). Sociotechnical field evaluations have generated meaningful insights into foundation model impacts on users (Zhao et al., 2024; Vaccaro et al., 2024; Tahaei et al., 2023) and subsequently informed model developers safety protocols (Hendrix, 2025). Yet, more robust analysis of model impacts and incidents will require sustained funding and interdisciplinary collaboration (Bengio et al., 2024). 

In addition to reporting, post-deployment monitoring also includes model forensics: traceability techniques, such as embedding identifiable patterns into model outputs to make models uniquely traceable (Yu et al. 2021; Boenisch, 2021; Fernandez et al., 2023; Christ et al., 2024;  Xu et al., 2024), and watermarking methods which help to either uniquely identify certain models or verify that content is generated from a specific model (Li et al.,2023; Block et al., 2025; Gloaguen et al., 2025) Metadata standards can also record contextual traces such as time, device, and location (Khan et al., 2025). Further research is needed to evaluate how to standardise and integrate traceability tools into post-deployment monitoring practices, balance traceability with privacy considerations, and how regulators and auditors might use model forensics to attribute responsibility for reported harms (Klasen et al., 2024; Hilgert et al., 2025). 

In safety risk management standards, risk analysis entails determining the level of risk, expressed as some combination of the severity of potential consequences (or harm) and the likelihood of those consequences (ISO/IEC, 2014). All the information collected internally and externally can be used, as appropriate, to arrive at an understanding of the relationships within and between different risks and to determine the level of risk. Below, we review open problems related to modelling risks and their interdependencies, as well as choosing the appropriate measures for risk.

Modelling risks and interdependencies. Modelling risks and interdependencies entails systematically analysing how identified risk sources, events, and system properties combine and propagate to produce real-world harms. In established high-risk industries, risk modelling is a central practice for safety risk management. For example, nuclear risk modelling combines Fault Tree Analysis (top-down deductive analysis tracing undesired events to root causes), with Event Tree Analysis (bottom-up mapping of potential outcomes following initiating events) (NRC, 2016), while cybersecurity threat modelling scopes the scenario space by adopting an attacker’s perspective to identify assets, trust boundaries, and plausible attack vectors (OWASP, n.d.). In the context of frontier AI, risk modelling is increasingly referenced in regulatory frameworks, such as the EU General-Purpose AI Code of Practice (2025), and is used to analyse how risks could materialise into concrete harms (Campos et al., 2025). Supported by techniques such as scenario analysis, this step connects technical system assessment with broader societal risk assessment by tracing causal chains linking model properties, user behaviour, and downstream impacts (Wisakanto et al., 2025). Compared to mature practices in other safety-critical sectors, frontier AI risk modelling remains nascent, although several approaches are emerging. 

Recent work adapts and extends methods from safety-relevant domains to account for frontier AI-specific characteristics. For example, Rodriguez et al. (2025) model how AI lowers attacker costs by identifying representative attack scenarios and bottlenecks in attack chains across threat landscapes, and quantifying how AI assistance reduces the costs of these bottlenecks. Wisakanto et al. (2025) adapt aerospace and nuclear safety techniques to AI by combining aspect-oriented hazard analysis, risk pathway modelling, bidirectional analysis from both capabilities and harms by combining techniques similar to event-tree and fault-tree analysis, and propagation operators that capture how risks may be amplified through accumulation, adversarial use, and sociotechnical diffusion. Building on this work, Murray et al. (2025) propose a framework for risk modelling and quantification that selects representative scenarios, decomposes them into parameterised event sequences, establishes non-AI baselines, identifies benchmarks and indicators as proxies to estimate scenario parameters, and aggregates individual parameter estimates using statistical tools into high-level risk estimates, an approach applied to nine cybersecurity risk models (Barrett et al., 2025).

Despite this progress, several open problems remain. A central challenge is ensuring that risk models are sufficiently comprehensive, including capturing interdependencies across components, pathways, and sociotechnical dynamics (Shostack, 2014). Risk models must integrate heterogeneous evidence streams, such as usage and API data, incident reports, capability evaluations, and uplift studies measuring how AI systems enhance human ability to cause harm. Further research is needed on methods for systematically combining these inputs into unified models (Murray et al., 2025). Additional challenges arise where historical data are sparse or absent, particularly for low-probability, high-severity risks In such cases, techniques such as expert elicitation (IEC, 2019, B.1) may be necessary, but their appropriate use and limitations for frontier AI remain underexplored. 

Quantifying risks. Quantifying risks entails selecting appropriate metrics and scales to represent risks and their components in a form that supports comparison and decision-making. Risk management standards recognise that risk can be expressed using qualitative, semi-quantitative, or quantitative measures, depending on context and purpose (IEC, 2019). Common techniques for combining qualitative values include index methods and consequence–likelihood matrices (IEC, 2019, B.8.6; B.10.3), while a quantitative measure of risk can be produced from a probability distribution of consequences (e.g., VaR, CVar and S-curves). Risk management further emphasises that different risk values should be expressed on comparable scales, they need not be expressed through a single value and that the format of risk representation is appropriate for the risk at hand (IEC, 2019). Additionally, while quantitative scores may be easier to handle as they allow for easier comparisons and aggregation, they can also be misleading if used inappropriately (ISO, 2018), compressing uncertainty, and creating a false sense of precision, particularly when used for cross-domain comparisons or resource allocation (Cox, 2008; Aven & Reniers, 2013). 

There is limited research explicitly referring to risk measures or estimates in frontier AI (Murray et al. 2025, Wisakanto et al. 2025), with some work only mentioning measures for evaluation scores with little or no methodological explanation on how measures are derived (DSIT, 2024, Weidinger et al. 2023, Solaiman et al. 2023). Specific examples of metrics include elicitation probabilities and capability scores (e.g., Ho et al., 2025) or the recently proposed 50%-task-completion-time-horizon metric for long tasks (Kwa et al., 2025). This metric choice is also important once probabilistic structures are applied because the resulting estimates are only meaningful if the metrics capture a relevant aspect of risk. Additionally, different frontier AI risks present distinct measurement challenges. Some risks, such as discrimination or human-rights harms, are difficult to capture through clear estimates (Yeung, 2025), even when using semi-quantitative or qualitative approaches. Others, such as loss of control, are hindered by the absence of relevant historical data (Chin, 2025). While documents like the EU GPAI Code of Practice (2025) provide high-level examples of frontier AI risk estimates, specific guidance is lacking on how to present these in ways that remain interpretable and decision-useful while faithfully representing relevant aspects such as uncertainty and the plurality of possible harm pathways, rather than encouraging premature closure around simplified risk scores. As frontier AI risks may not admit a single dominant ‘harmful scenario,’ for one risk but rather a family of interacting scenarios (Barrett et al., 2025), it is unclear how these should be expressed in a useful and clear way through a single or multiple risk measures. 

The data obtained through risk analysis (Section 3 Risk Analysis) can be used to inform decisions about whether the risk should be accepted or whether it requires mitigation, and if so, any priorities for mitigation. Below, we review relevant open problems.

Applying risk acceptance criteria. Risk criteria are applied to determine the significance of the risk relative to the criteria set by the organisation beforehand (ISO, 2022, 3.3.6) and, based on those criteria, decide whether the risk should be accepted or mitigated (ISO/IEC, 2014). Traditional safety risk management techniques, such as ALARP (Section 1.3 Setting Criteria), provide a way to distinguish between intolerable risk that cannot be justified, risk that should be reduced where reasonably practicable, and risk considered sufficiently low to be accepted without further treatment (IEC, 2019). In frontier AI, guidance (Raman et al., 2025) and regulatory documents such as the EU GPAI Code of Practice (2025) feature the application of risk acceptance criteria, or ‘thresholds’, requiring also the incorporation of a safety margin. The EU GPAI Code of Practice (2025) specifies that such a safety margin should account for potential changes, uncertainties and limitations related to risk sources (e.g., post-assessment capability improvements), the assessment process itself (e.g., under-elicitation in evaluations), and the effectiveness of mitigations (e.g., circumvention or deactivation) (2025). As mentioned in Section 1.3 (Setting Criteria), the best practice among frontier AI companies is to currently rely on, and thus apply, capability thresholds to determine risk acceptance. While, as mentioned in Section 2.1 (Identifying Risk Sources), companies tend to focus on a similar set of risk domains for analysis (e.g., CBRN), the application of capability thresholds varies across companies. Some developers use thresholds as ‘tripwires’ leading to outcomes such as ‘do not release’ or ‘stop development’ only mentioning mitigations and measures at a high-level (Meta, 2025), while others apply them as mitigation-and-decision ladders, triggering specific safety and security standards (Anthropic, 20205; OpenAI, 2025). Additionally, they often fail to incorporate safety margins as required in frontier AI regulatory documents, thus not accounting for potential uncertainties in assessment, effectiveness of mitigations or unexpected changes in risk sources. This uneven application makes it difficult for regulators, policymakers, safety or security professionals, end-users and downstream developers to clearly understand the risks at hand and the choices made by model developers (e.g., choices around mitigation or deployment). Consistency in the application of risk thresholds and risk acceptance criteria, as is the norm in other critical industries such as aviation, could help alleviate this (Campos et al., 2025). 

Determining overall risk acceptance. Determining overall risk acceptance entails aggregating multiple individual risks to form a judgment about whether the system’s total risk profile is acceptable. Traditional risk management emphasises risk aggregation as a means of combining individual risks to inform holistic acceptance decisions (ISO, 2022, 3.3.30). Aggregation methods range from relatively simple approaches, such as weighted sum of all risks with the possibility of context-specific weighting for the most relevant hazards (Schmitz, 2024), to approaches that explicitly account for interdependencies between risks (IEC, 2019). Established practices also recognise trade-offs across risks, for example through concepts such as ‘globally at least equivalent,’ (GALE) whereby a risk with adverse consequences may be deemed acceptable if equivalent or greater risk reductions have been achieved elsewhere (IEC, 2019). In frontier AI, however, overall risk acceptance remains underdeveloped. While existing developers’ safety frameworks tend to focus on acceptance decisions at the level of individual risks, the aggregation of multiple AI risks (and the criteria by which aggregate acceptance should be carried out) are often absent or left implicit. Frontier AI presents distinct challenges for risk aggregation, and there is limited literature examining whether traditional aggregation methods meaningfully transfer to frontier AI contexts (Schmitz, 2024). Traditional factors such as interdependencies between risks and information loss during aggregation (David, 2016), and AI-specific challenges such as diverse application contexts and differing degrees to which various risks can be quantified, complicate aggregation efforts (Schmitz, 2024). Frontier AI risks span heterogeneous domains, including economic harms (e.g., AI-enabled cyber attacks), physical harms (e.g., CBRN uplift), diffuse societal harms (e.g., manipulation or erosion of trust), and rights-based harms (e.g., discrimination or privacy violations), which may seem incommensurable in the face of aggregation. This complexity leaves it especially unclear how single risk evaluations should inform overall risk acceptance, specifically whether the presence of a single unacceptable risk should render the overall model risk unacceptably high. 

The results of risk evaluation inform product-release decisions, where risk has been deemed acceptable. In the case of frontier AI, this refers to deployment decisions; the act of releasing AI systems as standalone models, real-world applications or services. Below, we review relevant open problems.

Deployment decisions protocols. Deployment decision protocols determine whether, how, and under what conditions an AI system is released or made available for use (Bengio et al. 2025). Risk management standards emphasise that such decisions should follow a continuous and iterative process of risk evaluation and analysis, including reassessment after mitigations are implemented (ISO, 2018; ISO, 2023; ISO/IEC, 2014). In frontier AI, this approach is reflected in regulatory and governance guidance (EU AI Act, 2024; EU GPAI Code of Practice, 2025), outputs from AI Safety Summits (Hiroshima Code of Conduct, 2023), national AI safety and security institutes (DSIT, 2023), and independent research (Kaminski, 2023; Barrett et al, 2023). This means that risk should be re-assessed after risk mitigations have been implemented and, in cases where risk is still deemed unacceptable, developers should implement additional mitigations until risk levels are acceptable, halt deployment, or recall deployed models from the market (EU Commission, 2025;  IAPS, 2023). Frontier AI raises specific challenges for deployment decisions due to uncertainty about downstream impacts and the diversity of possible release strategies. Transparency around deployment and release decisions is therefore particularly important (Bommasani et al., 2023), as different risk levels may warrant different strategies (Anderljung et al., 2023), such as gradual release, gated to public access, or hosted access, among others (Liang et al., 2022; Solaiman, 2023; Seger et al., 2023). While transparency around external releases has increased in recent years (Wan et al., 2025), far less is known about the factors and protocols that shape internal deployment decisions (Stix et al., 2025; Bengio et al., 2025), where a developer develops a model or system and makes it available exclusively for internal access or use, with some researchers suggesting that internal governance mechanisms for such decisions may be largely absent (Stix et al., 2025). Because the most cutting-edge AI systems are often deployed internally first, and given questions around potential regulatory gaps in certain jurisdictions (Pistillo, 2025), these deployments can pose significant under-scrutinised risks. Researchers suggest to adapt lessons from other safety-critical industries (e.g., biological agents and toxins, R&D in nuclear reactors, novel medical devices, etc.) to establish internal use policies, oversight frameworks for internal deployment decisions, and appropriately targeted transparency mechanisms to minimise potential risks coming from such deployments (Stix et al., 2025).

Justifying deployment decisions. Justifying deployment decisions entails documenting and validating the reasoning for proceeding with deployment based on the outcomes of risk evaluation. Risk management standards require that risk assessment results be recorded, communicated, and reviewed at appropriate organisational levels (ISO, 2018). In safety-critical domains, this justification often includes demonstrating that risks cannot be reasonably reduced further prior to deployment (ISO/IEC, 2014). Overall, this step is useful to justify decision-making, as well as signaling compliance to regulators (Liberati et al, 2023). In frontier AI, some regulatory frameworks (EU Commission, 2025) specifically require that companies provide their reasons for proceeding with deployment in model reports. In the last few years, there has been an increasing focus on showing that a frontier AI system is sufficiently safe to justify its deployment, and proposals for suitable approaches, such safety cases, have emerged. Common across numerous industries (Sujan et al., 2016.; Maguire, 2017; Levesen, 2011), safety cases provide a structured argument, supported by a body of evidence (e.g., results of risk assessment) that an AI system is safe to deploy in a specific setting. 

Safety cases are provided by frontier AI developers, and could serve different objectives depending on their scope. While a few developers have begun using safety cases to address only partial or specific risks (e.g., Anthropic’s ‘affirmative cases’ in their Responsible Scaling Policy (2025) or Google DeepMind’s use of safety cases in their Frontier Safety Framework (2025)), some propose using safety cases more broadly to offer a rationale for why the probability of an AI system causing a catastrophe is below an acceptability threshold during a deployment window (Clymer et al., 2024; Irving, 2024; Balesni et al., 2024). Stakeholders across government (AISI, 2025), industry (Anthropic, 2024; Google DeepMind, 2024) and the research community (Buhl et al., 2024; Bengio et al., 2024) have recommended using safety cases as a key input to deployment decisions. However, approaches to constructing safety cases for frontier AI remain nascent (Korbak et al., 2025, Davidsen Buhl et al., 2025), and guidance on how results from risk analyses should be integrated into safety cases is still underdeveloped. On a more fundamental level, it is an open question whether safety cases are the right approach for a fast-changing technology such as frontier AI. Designed for mature technical systems (Bishop, 1998), the value of safety cases diminishes when risks are numerous, ill-defined, or hard to model (Levesen, 2011). Since safety cases cannot rule out unknown risks and that, as presented in Section 2 (Risk Identification), current approaches in frontier AI struggle to capture AI hazards comprehensively and in detail, relying on them may create a false sense of safety, especially for unpredictable but high-impact failures.

In risk management, measures that act on the inherent characteristics of the product are the first and most important step in the risk mitigation process. Below, we review data-level mitigations and their open problems accordingly. 

Data-level mitigations. Data-level mitigations aim to reduce risk by constraining the emergence of hazardous capabilities intervening on the data that the model is trained on. From a risk management perspective, such mitigations are appealing because they intervene early in the model lifecycle, re-calling the idea of inherently safe design (ISO, 2014). However, their value depends on whether they can deliver reliable and demonstrable risk reduction in practice. While frontier AI systems benefit from broad knowledge and capabilities, certain types of knowledge pose safety risks, such as language models knowing how to assist in cyber, bio, or chemical attacks; or image/video models ‘knowing’ how to create nonconsensual intimate deepfakes. An intuitive way to avoid having models learn unwanted capabilities is to filter the data they are trained on, particularly during the pretraining process when models develop core representations of knowledge. However, filtering training data is deceptively difficult (Paullada et al., 2021) due to costs (Ngo et al., 2021), filtering errors (Ziegler et al., 2022), degradation of dataset quality (Welbl et al., 2021), the massively multilingual nature of internet text (Kreutzer et al., 2022), cultural biases in content moderation (Welbl et al., 2021; Dodge et al., 2021; Xu et al., 2021; Stranisci & Hardmeier, 2025), and the inherently contextual nature of harmful behaviour. Frontier AI introduces challenges related to the limits of controllability at the data level, with emerging evidence suggesting that models may be able to robustly lack knowledge in complex domains such as science and engineering (Lee et al., 2025; O’Brien et al., 2025) but not simpler tendencies such as toxicity (Maini et al., 2025; Li et al., 2025). This casts questions on the ability to control for safety at the source when it comes to frontier AI and thus, also on the ability to select mitigations according to their expected effectiveness, thus respecting a classic safety risk-management logic, a priori. Ultimately, it is an ongoing challenge to develop effective methods for data curation, characterise the relationship between training data contents and emergent capabilities, and make models that more robustly lack harmful abilities (Barez et al., 2025, Casper et al., 2025).

There are also a host of technical engineering controls that act at the model level, and which aim to reduce assessed risk by shaping and constraining model behaviour. Below, we review them and related open problems.

Model behaviour mitigations. Beyond intervening on the data, and thus on the knowledge representations that the model is learning, another set of mitigations can constrain model behaviour after such representations have been learned. This may include a host of different approaches, from fine-tuning to machine unlearning. To public knowledge, the state-of-the-art for fine-tuning frontier AI systems in recent years have been methods that rely on feedback or demonstrations from humans or AI systems (e.g., Bai et al., 2022; Kauffman et al., 2024; Zhou et al., 2024). However, the effectiveness of these methods is limited by the quality of feedback and demonstrations provided to the AI system (e.g., Casper et al., 2023; Lambert et al., 2023; Dahlgren, 2024). Not only are humans prone to simple mistakes and disagreement (Wu et al., 2023; Glickman et al., 2024), but optimising for human approval can systematically, and often subtly, cause systems to learn harmful behaviours. For example, frontier language models are prone to be sycophantic to users, pandering to their preferences at the expense of objectivity and truth (Perez et al., 2022; Sharma et al., 2024; Malmqvist, 2024; OpenAI, 2025). This can be understood as a form of `reward hacking:’ the phenomenon in which AI systems can learn perverse behaviours by gaming imperfect reward signals (Skalse et al., 2022; Baker et al., 2025). It is also challenging , even for human experts, to oversee LLM’s performance on very difficult and complex tasks such as spotting vulnerabilities in a complex codebase or errors in an advanced proof (Kim et al., 2024). In response to these challenges, researchers are working on methods for human-AI teams to help evaluate complex behaviours (Michael et al., 2023; Du et al., 2023; Khan et al., 2024; Kenton et al., 2024; McAleese et al., 2024; Wen et al., 2025). 

Beyond increasing complexity of behaviour, frontier AI introduces additional challenges due to emergent and longitudinal harms. Harm does not always result from single uses of AI systems, but often from the sum total of a system’s behaviour across contexts and its effects on users. For example, after a 16-year-old committed suicide in April 2025, conversations with ChatGPT revealed the model provided harmful advice, instructions, and 1,275 mentions of suicide over the course of months in many separate chats (Tabachnik, 2025). Meanwhile, emerging research is beginning to identify harmful emergent effects of AI in education (e.g., Tamimi et al., 2024), mental health (e.g., Caridad, 2025), and user judgment (e.g., Krügel et al., 2023). From a risk management perspective, this complicates risk mitigation. Many model behaviour controls are evaluated on short-term interactions, while the most severe risks materialise gradually across sectors, and are increasingly difficult to control for and evaluate. 

Aside from techniques focused on aligning model behaviours with human interests, another technique for making AI systems safer is to use ‘machine unlearning’ algorithms to remove harmful capabilities (Liu et al., 2024; Barez et al., 2025). Existing  unlearning techniques (e.g.,  Zou et al., 2024; Sheshadri et al., 2024) are effective at suppressing harmful capabilities in most situations, but adversarial users can easily prompt, fine-tune or otherwise attack the model to resurface these capabilities (Feder Cooper et al., 2024; Lo et al., 2024; Łucki et al., 2024; Hu et al., 2024; Deeb et al., 2024; Huang et al., 2024; Qi et al., 2024; Che et al., 2025; Wei et al., 2025). A significant challenge is designing unlearning algorithms that result in more robust knowledge removal, possibly with algorithmic innovations or scaling unlearning interventions (Casper et al., 2025). For risk management, the key open problem is durability: reversible or fragile mitigations risk creating unjustified confidence, leading to deployment decisions that are not supported by sustained risk reduction.

Model-robustness. Model-robustness addresses the risk that safety failures arise under distributional shift or adversarial conditions. Despite their intelligence, even state-of-the-art AI systems are prone to exhibiting both subtle and egregious failures. One problem is building models that generalise appropriately and predictably outside of their training data distribution. For example, modern language models tend to be less safe (Wang et al., 2023; Yong et al., 2023; Song et al., 2024; Shen et al., 2024) and performant (Kshetri, 2024; Salammagari and Srivastava, 2024) in low-resource languages. Meanwhile, modern LLMs are vulnerable to especially egregious safety failures in the face of adversarial attacks, such as ‘jailbreaks’ which trick these models to comply with arbitrary harmful requests (Wei et al., 2023; Jiang et al., 2024; Jin et al., 2024; Chowdhury et al., 2024; Yi et al., 2024). Even cutting edge systems are persistently vulnerable to attacks. For example, a recent public competition to crowdsource attacks compiled over 60,000 successful attacks against recent production models from Amazon, Anthropic, Cohere, Meta, Mistral, OpenAI, and xAI (Zou et al., 2025). The principal challenge for improving adversarial robustness is to drive attack success rates down. Doing so will benefit from innovations in red-teaming, increasing the scale of adversarial training (Lee et al., 2024; Howe et al., 2024), and developing new algorithms for adversarial robustness (Zou et al., 2024; Casper et al., 2024; Sheshadri et al., 2024; Fu and Barez, 2025; Dékány et al., 2025; Casper et al., 2025), on which more work is needed. Red teaming, for example, while useful through the open-ended use of adversarial attacks or interpretability techniques (e.g., Marks et al., 2025) is skill-dependent and frequently fails to be consistently rigorous in practice (Feffer et al., 2024). Red-teaming methods regularly empirically fail to identify failures before deployment and thus, improving model robustness will require improvements in the adversarial capability elicitation toolkit (e.g. Che et al., 2025; Ludke et al., 2025). This creates a decision-making challenge under incomplete evidence, where organisations must decide how to act when robustness assessments provide only partial or negative assurance. Improving robustness therefore requires not only technical advances, but clearer guidance on how robustness evidence should inform risk acceptance and the selection of complementary mitigations where needed.

Frontier AI risk management increasingly relies on system safety architectures: a set of external mechanisms that monitor, steer, and constrain model behaviour without requiring the computationally expensive retraining of the underlying model. AI system safety treats the model as a component within a larger system where safety is enforced through system monitoring, inference-time control mechanisms, and system’s guardrails. Below, we review related open problems.

System monitoring. Monitoring serves as the primary tool for identifying unsafe or otherwise unwanted behaviours. In risk management terms, monitoring primarily supports detection and escalation rather than direct risk reduction. Its effectiveness therefore depends on whether monitoring signals reliably trigger appropriate and timely interventions, such as human review, access restrictions, or system rollback. Monitoring techniques can vary by the object of oversight – focusing on hardware activity, (e.g., Aarne et al., 2024; O’Gara et al., 2025), users (e.g., Yueh-Han et al., 2025; Brown et al., 2025), inputs/outputs (e.g., Greenblatt et al, 2023; Sharma et al., 2025; McKenzie et al., 2025), model uncertainty (e.g., Farquhar et al., 2024, Zhu et al., 2026, Lamb et al., 2026), internal cognition (e.g., Kirch et al., 2024; Goldowski-Dill et al., 2025; Kramár et al., 2026), and reasoning (e.g., Baker et al., 2025; Korbak et al., 2025) – or by the goal of oversight – logging information, flagging risky content, filtering harmful content, triggering failsafes, spotting deception, etc. With regards to frontier AI, a primary challenge in this domain is the creation of robust classification systems capable of identifying safety risks in real-time. One way to achieve this is through the deployment of specialised, instruction-tuned safety models that evaluate human-AI conversations against multi-class hazard taxonomies or are designed to detect toxic prompts and adversarial jailbreak attempts that might otherwise bypass general-purpose filters (e.g., Sharma et al., 2025; Inan et al., 2023; Han et al., 2024; AlDahoul et al., 2024, Zeng et al., 2024, Liu et al., 2024; Yuan et al., 2024; Lee et al., 2025). However, as models and users co-evolve, it is increasingly difficult to ensure that monitoring signals remain up to date and do not degrade over time, triggering inappropriate escalation responses and wrong risk mitigation strategies.

Control mechanisms. Inference-time control mechanisms allow for real-time modification of model behaviour, enabling post-deployment alignment that is both lightweight and adaptive. From a risk management perspective, these mechanisms function as operational controls that seek to reduce the possibility of harm by constraining behaviour at the point of use. Some approaches include activation engineering, which involves manipulating the model’s internal representations during the forward pass to guide outputs away from harmful behaviours or biases (e.g., Turner et al., 2023; Bayat et al., 2025; Ghosh et al., 2025, Postmus and Abreu, 2024). To achieve more precise and interpretable control, techniques that influence a model’s exploratory behaviour and uncertainty (e.g., Rahn et al., 2024), modulate token-level probability distributions at the decoding stage (e.g., Wang et al., 2025; Chakraborty et al., 2025; Pynadath and Zhang, 2025), and model the generation process as a consensus-driven interaction between generators and verifiers (e.g., Zhang et al., 2025; Welleck et al., 2024) have been developed. Regarding frontier AI, maintaining control over a model’s behaviour using steering and control techniques is often difficult. Empirical studies have highlighted persistent failures in coverage and the emergence of unintended side effects when attempting to steer large-scale models (e.g., Bentley, 2025; Miehling et al., 2025). To address these shortfalls, frameworks that use episodic memory or flexible backtracking mechanisms to dynamically determine the necessity and strength of intervention throughout the generation process have been proposed (e.g., Cheng et al., 2025; Tran et al., 2025; Wu et al., 2025). Still, as a risk mitigation measure, such mechanisms remain brittle and have yet to provide meaningful safety assurances that could support their choice for risk reduction.

System’s guardrails. Guardrails act as the integrated system logic that triggers interventions when violation of safety objectives are imminent, functioning as a bidirectional filter between the user and the model. In high-stakes applications, guardrails can enforce rigid structural requirements, using neuro-symbolic frameworks constraints to ensure outputs adhere to formal logical rules (e.g., Zhang et al., 2024). Modern deployment scaffolding integrates these various components into modular API gateways that orchestrate multi-agent validation and enforce compliance with security policies and other external standards (e.g., Shvetsova et al., 2025). Another approach involves the use of defensive mechanisms designed to detect and recover from alignment drifting caused by poisoning attacks, prompt injections, or malicious fine-tuning (e.g., Yan et al., 2024; Liao et al., 2025). Such protective measures can be combined with human-centric safety filters that reason about the feedback loop between AI outputs and human behaviour to ensure long-term robustness (e.g., Bajcsy and Fisac, 2024). Ultimately, each type of AI system guardrail aims to embed constitutional safety principles into evolving deployment scenarios. Frontier AI introduces particular challenges for guardrails because deployment contexts, usage patterns, and model capabilities evolve rapidly. As systems scale and become more adaptive, maintaining reliable alignment between guardrail logic and real-world risk becomes increasingly difficult. Guardrails must operate under uncertainty, adversarial pressure, distribution and domain shifts, and changing objectives, raising the risk of both under-enforcement (allowing harmful behaviour) and over-enforcement (unduly constraining legitimate use).

In frontier AI, ecosystem-level mitigations may also be understood more broadly as developers providing information, tools, and capabilities that enable other actors (e.g., governments, organisations, and civil society) to implement effective defenses against AI-enabled safety risks. Below, we review some of the open problems.

Information-sharing and documentation. In safety risk management, documentation does not reduce risk directly (ISO/IEC, 2014), but it plays a critical role in ensuring that relevant information about deployed systems, testing results and limitations, to cite a few, are sufficiently understood and shared across the ecosystem to prevent or minimise safety risks (FMF, 2025). More specifically, in the context of frontier AI, documentation practices such as model cards (Mitchell et al., 2019), datasheets for datasets (Gebru et al., 2021), suppliers’ declarations of conformity for AI services (Arnold et al., 2019), and now increasingly agent cards (e.g., OpenAI, 2025), aim to surface information about training data, intended use, performance boundaries, and known failure modes to support safer development, (re-)use and accountability in development and deployment decisions. Despite their widespread adoption, existing documentation practices exhibit inconsistency in both content and quality of documentation, limiting their effectiveness as risk-mitigation tools (Staufer et al., 2025; Richards et al., 2021). Studies show uneven adherence to model and dataset documentation standards and significant variation in what information is disclosed (Liang et al., 2024; Staufer et al., 2025), with sections addressing environmental impact, limitations, and evaluation showing the lowest filled-out rates (Liang et al., 2024). From a risk-management perspective, this undermines the ability of downstream actors (such as safety teams, auditors, deployers, and regulators) to assess whether existing practices are appropriate, sufficient, or being applied under the assumptions for which they were designed. Moreover, most documentation frameworks remain poorly adapted to domain-specific requirements. For example, Datasheets for Datasets fall short in meeting domain specific requirements such as around medical data that are required for data documentation and screening prior to AI applications (Zargari et al., 2025, p.1). As a result, a key open problem at the risk mitigation stage is how to design documentation practices that reliably contribute to meaningful reduction of risks, rather than merely increasing transparency. This includes determining what information is necessary and sufficient to support mitigation decisions, and how to standardise documentation in ways that remain sensitive to sector-specific requirements (e.g., medical or safety-critical domains).

Serious Incident Reporting.  In safety risk management, incident reporting does not prevent hazards ex ante, but it plays a critical role in limiting further harm, identifying systemic weaknesses in existing controls, and informing corrective actions that reduce future risk. In the context of AI, incident reporting encompasses formal processes for documenting and sharing safety incidents, security breaches, near-misses, and relevant threat intelligence with appropriate stakeholders to enable coordinated responses and systemic improvements (Saeri et al., 2025: 22). While several public databases track AI incidents (including the AI Incident Database (AIID, n.d.), AIAAIC Repository (AIAAIC, n.d.), and MIT AI Incident Tracker (MIT, n.d.), to cite a few), formal reporting obligations remain uneven across jurisdictions, with mandatory reporting currently concentrated in the EU under the AI Act and its GPAI Code of Practice (EU Commission, 2024; EU Commission, 2025). Despite growing institutional attention, current incident reporting practices face structural limitations that constrain their effectiveness as risk-mitigation tools for frontier AI. Key challenges include the absence of shared definitions for what constitutes a reportable incident, and difficulties in capturing harms that unfold over time, recur across contexts, or manifest primarily at the societal level rather than as discrete events (Paeth et al., 2024; Hoffmann & Fraser, 2023). Inconsistent database structures and incompatible data fields further limit the ability to aggregate incidents, identify patterns, or link reported harms to specific model features, deployment decisions, or mitigation failures (Agarwal & Nene, 2025; Lee & Frase, 2024; OECD, 2025). These limitations are compounded by the largely voluntary nature of reporting outside the EU, which results in under-representation of developer-side failures (Li et al., 2025), coverage bias toward high-profile misuse cases and limited visibility into incidents occurring in less-resourced contexts  (Paeth et al., 2024; Agarwal & Nene, 2025). Empirical evidence shows that only a small fraction of reported AI incidents trigger observable responses or corrective actions, highlighting a persistent gap between reporting and mitigation (Richards et al., 2025). Addressing this gap requires clearer links between incident reports and concrete risk-treatment measures. Possible ways forward include requiring reports to include information on the status of damage mitigation (Prud’homme et al., 2023), creating emergency protocols for rollbacks (Uuk et al., 2024), or the establishment of escalation channels through which implementers, vendors, and regulators can be informed about critical vulnerabilities (Gipiškis et al., 2024). However, with the increasing volume of reports, regulators are also facing potential regulatory overload (Cebrian et al., 2025).