Concordia AI’s feedback on the UN AI Advisory Body Interim Report
In December 2023, the UN Secretary-General’s AI Advisory Body issued their Interim Report: Governing AI for Humanity, which serves as the initial version of a report that is scheduled for publication in summer 2024. The report discusses opportunities from AI, risks and challenges posed, and recommendations on institutional functions needed for international AI governance.
As part of the report process, organizations may provide feedback on the interim report. Concordia AI CEO Brian Tse is also part of the AI Advisory Body’s “Consultative Network.” Accordingly, we have provided the below feedback and suggestions on how we think the report can be further strengthened.
Key Takeaways
The report could provide definitions for general-purpose foundation models and/or frontier AI and discuss how the risks and benefits of such systems might differ from narrow AI systems.
The report could discuss additional projects for supporting the equitable and meaningful participation of communities from the Global South/Global Majority in particular, such as improving AI training data in under-resourced languages, talent development, and improving the safety and alignment of models across different languages and cultures.
Additional “red lines” for AI merit inclusion in the report, such as autonomous replication, cyberattacks, deception, and use in nuclear weapons systems.
The report could discuss how competition over AI development between a small number of leading actors could lead to a race to the bottom on AI safety.
The discussion around subfunctions for international governance could benefit from greater discussion of areas of overlap with existing multilateral efforts. It could also provide recommendations on prioritization and whether particular subfunctions can be included in existing institutions versus requiring new institutions.
The report could add a principle on “agile” or “adaptive” governance.
Defining types of AI
The report could provide definitions for general-purpose foundation models or frontier AI and discuss how the risks and benefits of such systems might differ from narrow AI systems. The report currently adopts the OECD definition for AI systems, but does not distinguish among different types of AI, particularly general-purpose foundation models or frontier AI. Many experts believe that frontier AI could pose risks “at even greater severity and scale than is possible with current computational systems,” and presents several unique risks, including the possibility of developing unexpected capabilities. Both the US and EU have introduced regulations for foundation models based on computational power, recognizing the significant expert concern regarding their risks. Chinese experts are also considering similar ideas around regulations targeting foundation models. Therefore, we suggest that the report clearly define foundation models or frontier AI and highlight the governance functions and risks unique to these models, which could merit additional specific analysis on their impact in relevant sections of the report.
Opportunities and projects for the Global South
We suggest raising examples of projects that would assist Global South countries in accessing the benefits of AI development. One idea is to support the collection of high quality datasets in underrepresented languages while ensuring local participation in the process, as lack of high quality training data for such languages negatively impacts foundation model performance. References include Project SEALD for Southeast Asian Languages, Chinese Peng Cheng Lab’s dataset on languages in the Belt and Road Initiative, and the Maori Data Sovereignty Network. The report could also suggest strategies to foster AI R&D, safety, and governance talent, especially from underrepresented groups. Examples include educational initiatives such as the Smart Africa Digital Academy, incentivizing top AI companies to establish offices in the Global South, and financing AI-focused entrepreneurship. Not least, there is an urgent need for the UN to support efforts to enhance basic digital infrastructure and electricity access in the Global South, without which AI development is infeasible.
The report could also include a paragraph on improving the safety and robustness of models across different cultural contexts. Models developed in one context (e.g. the US) are prone to additional risks if used in languages or cultures underrepresented in their training sets, such as jailbreak attacks in low resource languages.1 Relatedly, human values are complex and difficult to generalize into one set of principles that can be consistently applied to AI alignment. One solution would be drafting a set of collective values for alignment through public inputs globally, drawing from the Collective Constitutional AI project. Alternatively, companies could fine-tune models on the values of a given society in the vein of Singapore’s Sandbox project, which partners with Anthropic for red-teaming of models in the local cultural context.
Elaborating on red lines and competitive dynamics
The existing list of red lines should include a more complete accounting of the risks that potentially constitute severe global threats and thus especially merit the UN’s attention. In a recent dialogue, 24 top Chinese and Western AI experts agreed on five red lines: autonomous replication or improvement, power seeking, assisting weapon development, cyberattacks, and deception. These red lines highlight the shared concern among international experts about the potential for AI to cause significant harm if left unchecked and should be added to the report. In addition, the report should recognise the importance of keeping humans in-the-loop on decisions regarding the usage of nuclear weapons, given the catastrophic consequences of their use. As frontier foundation models and narrow AI systems in sensitive domains (military, biology, etc.) are the types of models most at risk of crossing these thresholds, the report should recognize these types of models and the special attention they deserve.
Separately, the report could consider acknowledging competitive factors that can exacerbate AI risks. In particular, the capital-intensive scaling paradigm driving current frontier AI development creates strong incentives for developers to capture the largest possible market share, leading to intense competition between AI developers to rapidly build and deploy systems. This competition raises concerns about potential “race to the bottom” scenarios, where actors compete to develop AI systems as quickly as possible while under-investing in safety measures.
In such scenarios, AI developers may seek to avoid committing unilaterally to stringent safety standards, as doing so might put them at a competitive disadvantage. Safe AI development may be further hindered by market failures among AI developers and collective action problems among countries because many of the harms are incurred by society as a whole. Moreover, the high concentration of market power among frontier AI developers could lead to unilateral decision-making by a small number of actors, which may significantly escalate AI risks and create single points of failure. The report could address these critical issues and their potential impact on the safe and responsible development of AI.
Prioritizing and institutionalizing governance functions
The “Institutional Functions” section could map out other ongoing multilateral efforts for the subfunctions that were identified in Table 1 in order to inform prioritization and phasing of efforts. For instance, the UK government-supported International Scientific Report on Advanced AI Safety is expected to cover topics similar to Subfunctions 1, 2, and 3. Therefore, the Interim Report could suggest whether the UN should defer to that effort, attempt to subsume that project under the UN, or establish a separate, UN-led process on scientifically assessing AI safety and risks. The third option likely would create unnecessary duplication, so the report should carefully consider the pros and cons of the UN taking a larger role in establishing scientific consensus. In addition, for “Subfunction 11. Standard setting,” it is unclear how UN agencies such as the ITU and other international technical bodies such as the ISO and IEC should coordinate responsibilities for standards setting. Such analysis would help the report provide recommendations on which governance subfunctions should receive greater priority.
The report could also more clearly outline which subfunctions should be allocated to existing institutions versus which would require a new institution to be established. It could also outline a proposed deliberative process for the UN to develop member state consensus around creating a new potential international institution(s) for AI. The present report does not account for these questions. China has already announced that it supports establishing a new UN-led international institution for AI governance, but a broader international dialogue is necessary for UN member states to agree on whether to establish a new international institution, how it would be structured, and what functions it might perform.
The report should consider mechanisms for ensuring that the Global South has a strong voice in discussions around AI development and governance. Discussions on AI governance could be easily monopolized by the countries with the strongest AI capabilities, but the entire globe would be affected by any catastrophic disasters resulting from misuse or loss of control of advanced AI systems. Currently, communities in the Global South have limited influence over the development of AI by leading companies, despite the fact that the potential negative consequences and risks associated with AI are not confined to these companies alone, but can have far-reaching impacts across national boundaries. The report could study relevant international models and provide concrete suggestions on mechanisms to ensure voice for the Global South.
Agile governance as a guiding principle
While Guiding Principle 4 notes that governance should be “rooted in adaptive multi stakeholder collaboration,” the report could elevate agile governance to a principle of its own. Agility in AI governance is crucial due to the technology's rapid evolution, which experts believe often outpaces regulations. Leading Chinese AI governance expert XUE Lan (薛澜) has been one of the most prominent proponents of “agile governance.” The definition developed by the Chinese Ministry of Science and Technology’s National New Generation AI Governance Expert Committee, chaired by Xue, is a useful reference for this subject.
The preprint, by researchers from the University of Science and Technology of China and Nanyang Technological University, analyzed risks from multilingual jailbreak attacks and designed a corresponding mitigation strategy.