Concordia AI holds the Frontier AI Safety and Governance Forum at the World AI Conference

Concordia AI participated in the 2024 edition of the World AI Conference (WAIC) in Shanghai on July 4-7. Concordia AI representatives attended the WAIC opening ceremony, where Premier LI Qiang (李强) gave opening remarks (En, Ch) and Shanghai Party Secretary CHEN Jining (陈吉宁) announced the “Shanghai Declaration on Global AI Governance” (En, Ch). Concordia AI will provide a full summary and analysis of these developments in the next issue of our AI Safety in China newsletter. 

On July 5, Concordia AI hosted the WAIC Frontier AI Safety and Governance Forum, bringing together a group of distinguished experts from around the world including former Baidu President ZHANG Ya-Qin (张亚勤), Turing Award Laureate Yoshua Bengio, Peng Cheng Lab Director GAO Wen (高文), Shanghai AI Lab Director ZHOU Bowen (周伯文), and UN High-Level Advisory Body members ZHANG Linghan (张凌寒), ZENG Yi (曾毅), and Ruimin He. The forum focused on four key topics: AI Safety Research, AI Safety Testing, AI Safety Guidance, and International Cooperation. Key discussions included the limitations of current AI safety methods, the importance of developing and harmonizing safety testing standards internationally, similarities and differences between national AI governance approaches, and the need for global collaboration in addressing potential AI risks. 

The video recordings of all of the speeches and remarks at the forum can be found as a playlist on Concordia AI’s YouTube channel.

Opening Remarks

Shanghai Municipal Government Deputy Secretary-General ZHUANG Mudi (庄木弟) gave opening remarks at the forum. In discussing AI development and safety, Mr. Zhuang referenced China’s Global AI Governance Initiative, the Bletchley Declaration, and Shanghai’s local policy on promoting the AI industry. He stated that Shanghai is committed to playing a leading role in AI safety and governance, including exploring large model testing and regulatory testing sandboxes as well as supporting AI safety and governance researchers.

Theme 1: AI Safety Research

Turing Award winner Yoshua Bengio gave an overview of key findings from the International Scientific Report on the Safety of Advanced AI: Interim Report, highlighting the shortcomings of current technical methods in reducing AI risks. In response to these limitations, Bengio called for international cooperation, the implementation of risk thresholds, the creation of early warning mechanisms, and investing in safe-by-design AI approaches. 

Academician and Director of Peng Cheng Laboratory GAO Wen (高文) discussed a broad range of AGI safety risks. He categorized these risks into intrinsic technical challenges, dangers arising from technological advancement, and ethical concerns stemming from both proper use and misuse. To combat these risks, Gao proposed a comprehensive, multi-tiered prevention strategy. This approach emphasizes enhancing model interpretability, controlling value orientations, creating technical standards, and fostering international collaboration.

Academician and Tsinghua Institute for AI Industry Research (AIR) Dean ZHANG Ya-Qin (张亚勤) identified key concerns including loss of control, potential misuse, challenges with embodied AI, and the existential risks from AGI. To tackle these challenges, he suggested five safety measures: a tiered governance approach, developing methods to identify AI-generated content, allocating 10% of AI R&D efforts to safety/security issues, establishing clear red lines for AI development and use, and strengthening international communication channels. 

UC Berkeley Professor Dawn Song emphasized the fragility of existing AI safety and alignment methods to adversarial attacks. She introduced recent progress in representation control and activation steering as important tools for controlling model behaviors. Song also emphasized the importance of a safe-by-design approach to building secure systems with provable guarantees.  

During the roundtable, Professor Song was joined by Shanghai AI Lab (SHLAB) Large Model Safety Team Head SHAO Jing (邵婧), Peking University Center for AI Safety and Governance Deputy Director YANG Yaodong (杨耀东), and Shanghai Jiao Tong University Professor ZHANG Zhuosheng (张倬胜), moderated by Concordia AI Technical Program Manager DUAN Yawen (段雅文). The experts discussed the vulnerabilities of current safety alignment methods and the unique challenges of safety for multimodal models and AI agents. The panelists agreed that existing safety alignment techniques are presently unable to ensure the safety of the models and highlighted various key potential risks from future AGI or superintelligent systems: 

• Professor Yang emphasized worries about autonomous self-replication and self-improvement.

• Dawn Song called for early detection of agents that derive dangerous subgoals, such as seeking power, from their overall goals. 

• Building on this, Professor Zhang stressed the need for active detection of harmful model behavior from within the model itself for AI agents acting in an open environment. 

• Looking ahead, Dr. Shao stated that safety risks of AI will intensify in specific domains or verticals, such as when used in scientific disciplines.

Theme 2: AI Safety Testing

Frontier Model Forum Executive Director Chris Meserole gave an overview (video forthcoming) of four different types of evaluations and five best practices for frontier AI safety. The five best practices he offered were to account for prompt sensitivity to ensure consistent and reliable AI responses; carefully consider transparency of evaluations; assess systems rather than focusing solely on individual models; evaluate both normal and adversarial use of systems to guard against potential misuse; and assess marginal risk to understand risks of AI systems compared to counterfactual applications (e.g. web search).

China Academy of Information and Communications Technology (CAICT) AI Research Institute Director WEI Kai (魏凯) discussed gaps in understanding of model performance given rapidly evolving AI capabilities. Wei introduced the AI Safety Benchmark developed by CAICT and its areas of focus in Q2 2024. Specifically, Wei emphasized the benchmark's role in assessing red lines or bottom lines in AI development and use, as well as its focus on societal ethics.

SHLAB Leading Scientist and Assistant Director QIAO Yu (乔宇) introduced the organization’s work on large model safety evaluations. He highlighted their research on multi-agent alignment and development of evaluation datasets to assess the unique challenges around the alignment of vision-language multimodal AI systems. Furthermore, Qiao outlined SHLAB's participation in a group under the Cyber Security Association of China to create a standard for “process and rules for generative AI safety evaluations.”¹

The three experts joined a roundtable discussion with Tianjin University Professor XIONG Deyi (熊德意) and Singapore Chief AI Officer Dr. Ruimin He, moderated by Concordia AI CEO Brian Tse. Participants discussed risk assessment for frontier AI models, the science of AI safety evaluations, the role of third-party auditing organizations, and international interoperability of AI safety testing:

• There was a clear consensus among experts that there is a range of existing harms and potential risks from AI that need to be tested and evaluated. Professor Xiong emphasized the risks of misalignment, catastrophic misuse, and further dangers associated with frontier or more autonomous AI systems. 

• Attendees discussed the role of third-party evaluators. Ruimin He explained how Singapore has shared the AI Verify open-source toolkit with industry. Director Wei also argued that third-party testers play a bridging role and provide a public good that can reduce safety costs for industry.

• There was a shared recognition that existing safety evaluation approaches are insufficient and ad-hoc. For example, Professor Qiao believed that our understanding of large model risks remains limited, calling for more rigorous methods and tools for assessment. 

• Chris Meserole and Professor Qiao both called for international cooperation, with Chris Meserole calling for the development of a “shared vocabulary” so that everyone has a similar definition of ideas such as “red-teaming.” 

Theme 3: AI Safety Guidance

Gaël Varoquaux from France’s Inria and the France AI Commission presented key findings from the Commission's report Our Ambition for France. He underscored distinctive features of the French AI ecosystem, emphasizing its multi-party nature and strong commitment to open-source development. Varoquaux also detailed France’s efforts to prevent AI-related harms, focusing on four areas: misinformation, privacy leaks, bias, and overconcentration of power in the AI industry.

Singapore Chief AI Officer Dr. Ruimin He listed four recommendations for AI policymakers. These included approaching AI governance with a spirit of humility and engaging with expert communities, avoiding false dichotomies such as between innovation and regulation, keeping up to date on AI technology to effectively govern AI, and cooperating internationally to create interoperable safety testing standards. 

China University of Political Science and Law Professor ZHANG Linghan (张凌寒) pointed out limitations of a risk-based AI governance approach. She noted that China instead is pursuing a value-based strategy. Zhang explained that this approach involves distinguishing immediate AI risks from uncertain future impacts and assessing both short and long-term governance needs, including safe development, service applications, and infrastructure construction. She emphasized China's commitment to pursuing a “community of shared future for humankind” in its governance approach. 

UC Berkeley Center for Human-Compatible AI (CHAI) Executive Director Mark Nitzberg outlined the US approach to AI governance. He explained how the US aims to promote AI industry growth while protecting public and economic interests. Nitzberg argued against misperceptions about the relationships between regulation and innovation, capability and safety, as well as red lines versus pausing AI development. 

Shanghai Jiao Tong University Professor JI Weidong (季卫东) and SHLAB Governance Research Center Deputy Director WANG Yingchun (王迎春) joined the roundtable chaired by Concordia AI Senior Governance Lead FANG Liang (方亮):

• Experts discussed balancing AI development and safety. Professor Ji supported investing one-third of AI R&D in safety and assurance and praised Singapore's AI Verify project. Deputy Director Wang explained Shanghai AI Lab's “45-degree law” approach to AI development. This strategy envisions AI capabilities and safety measures advancing hand-in-hand at approximately a 1:1 ratio to ensure that safety is not left behind in the race for innovation.

• Professor Zhang and Gaël Varoquaux shared their countries’ perspectives on domestic AI governance. Professor Zhang argued that China’s position as a “leading follower” means that Chinese regulation needs to foster development of AI technology while simultaneously ensuring that there is a reasonable legal responsibility framework. She also stated that China’s forthcoming national AI law should be a “calling card” for explaining China’s AI governance model to the rest of the world. Gaël Varoquaux described France's focus on open-source digital infrastructure and bridging the digital divide.

• Mark Nitzberg discussed CHAI’s cooperation with the US AI Safety Institute and the institute’s role as a centralized body to track and measure adverse events.  

Theme 4: International AI Governance

Carnegie Endowment for International Peace (CEIP) President Mariano-Florentino (Tino) Cuéllar made four key points. He argued that the changes we have seen in AI over recent years are not transitory given massive increases in compute, opportunities and risks are substantial, international cooperation will be affected by changing technology and geopolitical challenges, and international cooperation should not involve only nation-states.

Tsinghua Institute for AI International Governance Director XUE Lan (薛澜) discussed contradictions between AI development and governance. These contradictions are further exacerbated by geopolitical competition and the speed of technological progress. Dean Xue proposed increasing investment in AI safety research, pursuing agile governance, facilitating self-regulation within industry, and increasing international cooperation between technical experts. 

During the fireside chat moderated by Concordia AI Senior Research Manager Jason Zhou, both Tino Cuéllar and Dean Xue acknowledged that the US and China may have had differences in expectations for the intergovernmental AI dialogue. Tino Cuéllar argued that both countries have a mutual interest in sharing best practices on AI safety and evaluation. Dean Xue stressed China's openness to collaborate with all countries on AI safety, expressing a desire for inclusion in global discussions and a commitment to include others in China-led initiatives. Both speakers highlighted the value of expert-level dialogues outside of official government channels.

Chinese Academy of Sciences Center for AI Ethics and Governance Director ZENG Yi (曾毅) explained the importance of AI safety redlines and the challenges posed by the self-evolution of AI systems. He called for creating a sustainable, symbiotic society between humans and AI under a concept he dubbed “harmonious symbiosis,” which requires adaptation together by humans and AI. He also introduced the Chinese AI Safety Network.

Hugging Face Head of Global Policy Irene Solaiman explored the relationship between openness and AI safety. She highlighted how transparency contributes to AI safety through the development of improved datasets and facilitation of scientific peer review. Solaiman advocated for interdisciplinary approaches, emphasizing that AI safety and innovation can be advanced simultaneously.

Oxford Martin AI Governance Initiative Co-Director Robert Trager examined three priorities for international AI governance. First, he proposed agreeing on international standards for AI safety, while noting that not all areas of AI governance need to be addressed at the international level. Second, he outlined how an international reporting regime can incentivize adoption of such international standards. Finally, Trager argued that the world needs an ecosystem of international institutions to address different AI governance goals and functions.

Centre for International Governance Innovation (CIGI) Global AI Risks Initiative Executive Director Duncan Cass-Beggs presented CIGI’s recently published “Framework Convention for Global AI Challenges.” The Framework Convention has three main objectives: realizing and sharing global benefits of AI, mitigating global AI risks, and ensuring inclusive global coordination and decision-making on global AI issues. He explained how a framework convention offers a highly adaptable and flexible approach, encouraging global stakeholders to seek agreement on high-level principles while setting out a process to develop concrete protocols to address specific issues.    

CEIP Fellow Matt Sheehan joined the panel discussion, moderated by Concordia AI Senior Program Manager Kwan Yee Ng. The experts discussed various approaches to building international cooperation on frontier AI:

• Matt Sheehan argued for prioritizing “safety in parallel” rather than “safety by agreement.” He said that it may be more difficult for national leaders to come to an agreement on AI safety immediately. Instead, Matt Sheehan called first for a bottom-up dialogue on best practices between policy and technical experts within China and the US, which can set the stage for a potential high-level agreement. 

• Professor Zeng and Robert Trager both emphasized the importance of the UN. Professor Zeng called for the UN to create a centralized AI office similar to the EU AI office, and Robert Trager noted the role of the UN in promoting development, standard-setting, and responsible digitization. Meanwhile, Irene Solaiman highlighted how the UN can build trust across borders by bringing different experts together and called for greater attention to the history and culture of different areas. 

• Duncan Cass-Beggs warned that we are likely underestimating both the pace of AI development and the scale of its implications. Therefore, current governance efforts must anticipate future AI advancements, not just address today's technology. 

Closing Remarks

Shanghai AI Lab Director and Chief Scientist ZHOU Bowen (周伯文) delivered the forum's closing remarks, underscoring key points on balancing AI development and safety. Zhou warned of increasing AI risks as technology approaches “SuperAI,” including societal systemic risks, malicious use, and potential loss of control. He introduced the “45-degree law” concept for AI development, advocating for simultaneous advancement of AI capabilities and safety measures. Zhou proposed developing AI systems capable of reflection, value-based training, causal interpretation, and counterfactual reasoning. He stressed the importance of global collaboration on AI safety, technology sharing, and balancing investments between AI capabilities and safety measures. His remarks underscored the need for a holistic, internationally-coordinated approach to responsible AI development.

1. For more information on the Cyber Security Association of China, see slides 74 and 76 of Concordia AI’s The State of AI Safety in China: Spring 2024 Report.