AI Safety in China #24
AI and dual-use chemicals, cyber offense misuse evals, World Internet Conference, cyber offense benchmarks, AI legislation analysis
Key Takeaways
China’s official arms control white paper flagged emerging risks from the convergence of AI and dual-use chemicals, but does not address other AI-driven CBRN or cyber misuse risks.
A major state-affiliated think tank published a comprehensive AI safety report that acknowledges AGI-related loss of control risks, but ranks them as a medium-to-low governance priority due to low near-term likelihood.
The same think tank also released new evaluations of coding models, finding that most lack sufficient safeguards against cyber offense misuse.
A World Internet Conference expert committee proposed a UN-centered framework for frontier AI risk governance, emphasizing cross-border evaluations, early warning, and emergency response mechanisms.
Chinese researchers have published technical papers on more realistic benchmarks for assessing cyber offense risks of LLMs.
A senior Chinese legal scholar called for fast-tracking comprehensive AI legislation and argued that the central government should have authority to pause development or deployment of AI systems posing existential risks.
International AI Governance
Arms control white paper highlights AI–chemical convergence risks
Background: The State Council Information Office released a white paper on arms control (En, Cn) on 27 November. The document outlines China’s approach to international arms control and nonproliferation, with significant attention on emerging fields, including AI.
AI safety content: Notably, the white paper states that risks arising from the convergence of AI and dual-use chemicals are becoming “increasingly prominent,” and it references China’s recent co-hosting of a workshop on AI and chemical safety with the Organization for the Prohibition of Chemical Weapons (OPCW).
On military applications of AI, the white paper calls for keeping weapon systems under human control, ensuring “safe, reliable and controllable military applications,” and establishing quality-control and risk-response mechanisms. It also stresses the need for consensus-based global governance and to “mitigate proliferation risks” associated with AI, and reaffirms China’s commitment to avoid an AI arms race.
Implications: This white paper appears to be the first policy document directly released by the central government that explicitly highlights risks at the convergence of AI and dual-use chemicals. The white paper has other sections on nuclear, biological, and cyber issues, but these do not mention AI-driven risks, with attention only shown to the “chemical” dimension of the broader AIxCBRN landscape. However, this may simply reflect an effort to foreground China’s leadership of the OPCW workshop on AI and chemical safety, as Chinese government bodies have referenced AI-driven risks in biological and nuclear domains elsewhere.
World Internet Conference report proposes UN-centered frontier risk management framework
Background: At the World Internet Conference (WIC) in November, the AI Safety and Governance Program of the WIC Specialized Committee on AI released a new report (En, Cn) proposing a global framework for AI safety and governance. The report is co-led by ZENG Yi (曾毅) (Chinese Academy of Sciences) and Seán Ó hÉigeartaigh (University of Cambridge), with contributions from around 40 experts across Chinese and international institutions. WIC is China’s highest-profile forum on cyberspace governance, organized by the Cyberspace Administration of China (CAC), the country’s lead AI regulator.
Content: The report highlights that rapidly advancing AI systems pose risks to strategic stability, international security, “and even human survival.” It points to rising malicious uses of AI in cyberattacks, as well as misuse potential in CBRN and missile domains, and loss of control scenarios as AI approaches superintelligence.
The report observes that current cross-border safety evaluations for frontier models involve only a small group of countries, calling for a UN-centered architecture with broader participation. Proposed areas of focus include:
Building consensus on “uncontrollable risks that could threaten human survival and development,” including joint restrictions on AI use in sensitive CBRN and missile-related domains.
Cross-border early warning, supported by shared safety evaluation tools and testing platforms.
International emergency response protocols that can be activated when catastrophic risks are identified, including suspending risky experiments and halting model deployment.
Implications: Compared to past WIC publications, this report places substantially greater emphasis on frontier AI safety and the need for global risk monitoring, emergency preparedness, and response. Given the close relationship between WIC and the CAC, this could shape the views of China’s top AI regulator regarding AI safety.
However, broader political attention to these issues within WIC appears limited. The official readouts from the opening and plenary sessions barely mention AI and do not address AI safety. This suggests that frontier-risk discourse at WIC remains driven mainly by experts, not yet by top political leadership.
Domestic AI Governance
Major think tank publishes AI safety report and evaluation of coding model safety
Background: On November 24, the China Academy of Information and Communications Technology (CAICT) released a 58-page report on AI safety. CAICT, a key think tank under the Ministry of Industry and Information Technology (MIIT), plays a central role in national AI standard-setting, safety evaluations and corporate safety commitments.
The report offers a systematic overview of CAICT’s thinking on AI safety, following lengthy reports in 2023 and 2024, and it also includes the first full publication of its security evaluation results of coding models.
The report: The report breaks down AI risks into a number of categories, specifically highlighting “frontier risks” at the model level. Frontier risks are categorized into “alignment resistance” and AI developing self-awareness, in the latter case citing a case of AI agents conversing in a non-human language.
The report also suggests a framework for prioritizing governance measures based on three variables: severity of harm, likelihood of occurrence, and technological maturity.
High consequence, high likelihood risks that already manifest in current systems are high priority. Specific examples are not provided.
Data leaks by intelligent chatbot services in certain industries are designated medium priority due to relatively low severity, high likelihood, and high technological maturity.
AGI-related self-awareness and loss of control risks are designated “medium-low priority” given very high severity but low near-term likelihood and low technological maturity. However, CAICT stresses that such risks warrant long-term monitoring and technical preparedness as capabilities advance.
Low priority risks could include AI applications in emerging fields still at proof-of-concept stage, risks that haven’t yet threatened physical space, or risks with limited threat scope even in cyberspace.
The report also notes that Chinese and international AI safety benchmarks overlap in testing approaches but diverge in topics of focus: foreign evaluations tend to emphasize frontier risks and malicious intentions, while Chinese evaluations focus more on model compliance with domestic laws, policies, and values.
Coding model safety evaluation: CAICT simultaneously released full results from its security assessments of 15 open-source coding models from Alibaba, DeepSeek, and Z.AI (formerly Zhipu AI). The report warns that generative coding models could enable autonomous, unmanned, and wide-scale cyberattacks.
The evaluations span malfunction risks (e.g., hallucinations, faulty code) and misuse risks (e.g., malware generation) across text-to-code, code-to-code, and code-to-text tasks.
The report found that coding models are overall at medium risk level, specifically rating:
1 model: controllable risk,
3 models: low risk,
9 models: medium risk, and
2 models: high risk.
Text-to-code testing found high risks, with non-experts able to generate functional attack code in 32.7% of attempts, and 9.5% of outputs requiring no modification to cause real harm. Defenses against advanced prompt attacks were weak, with rejection rates below 40%. In code-to-code testing, models resisted malicious rewriting tasks only 53.7% of the time. Code-to-text systems also exhibited errors, though their misuse potential is assessed as lower.
Implications: The report shows varying levels of focus on different frontier risks at CAICT. Particular attention to the cybersecurity implications of coding models is reflected in the high-risk classification of this domain, as well as the reporting of substantial empirical evaluation efforts.
More speculative AGI-related loss of control risks are acknowledged but remain a medium to low-level priority. The absence of any discussion of CBRN-related risks is surprising, especially given previous public remarks by CAICT President YU Xiaohui (余晓晖) highlighting their importance. Though the report notes that Chinese evaluations largely focus on domestic law rather than frontier risks, it still calls for monitoring and preparing for potential loss of control risks.
Technical Safety Developments
PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities
This paper from Shanghai AI Lab, Shanghai Jiao Tong University, and the Chinese Academy of Sciences (CAS) introduces a benchmark for evaluating the cyber offense capabilities of LLMs. The authors argue that current benchmarks often make cyber tasks too easy or artificial. PACEbench tries to simulate more realistic scenarios by using multi-host environments with uncertain targets, chained exploits, and real defenses that require end-to-end interactive attack behavior rather than simple, isolated tasks. The authors claim this allows for better judgment on whether AI models pose real world cyberattack misuse risk.
Empirical tests with seven frontier models show that:
Models can succeed on some simple, single-vulnerability tasks;
Performance drops significantly in complex, multi-host environments;
No model was able to bypass real cyber defenses like web application firewalls.
The authors conclude that current models do not yet pose a significant threat in real-world cyber offense. PACEbench provides a systematic framework for pre-deployment security assessments and tracking whether these risks increase in the future.
Expert views on AI Risks
International relations scholars on global governance of AI-enabled bio risks
Background: XIAO Xi (肖晞), Dean of the School of International and Public Affairs at Jilin University, published a lengthy essay in an academic journal on opportunities and challenges for global biosafety governance in the AI era.
Content: Xiao warns that AI, especially fine-tuned models with safety removed, can lower technical barriers to accessing and reconstructing dangerous biological materials, which increases risks of AI-enabled bio terrorism.
Xiao argues that governance frameworks lag behind technological change: current international AI governance documents consist mainly of non-binding principles, with “no specialized binding treaty,” and a lack of legally binding verification mechanism for the Biological Weapons Convention. Great power rivalry over technological and institutional influence weakens trust and hampers cooperation; for example, developing countries struggle to access safety-critical biological databases.
Xiao also highlights AI’s positive potential for stronger biosafety early warning, more scientific emergency decision-making, and integrated post-crisis management. She calls for strengthened multilateral cooperation, especially through UN-centered mechanisms, to improve the effectiveness and inclusiveness of global biosafety governance.
Implications: Xiao’s piece shows that Chinese international relations scholars are proactively thinking about methods to strengthen international governance efforts on AIxbio risks. With the drive to advance AI for Science as part of China’s AI+ Initiative, understanding of AI and biological risks will be increasingly important.
Her suggestion for UN-based cooperation on global biosafety governance may offer potential common ground between China and the United States, particularly given separate statements in support of AI-driven verification of the Biological Weapons Convention by President Trump and State Department Under Secretary for Arms Control and International Security Thomas DiNanno in the past four months.
Leading legal scholar argues for state authority to halt AI development in extreme-risk scenarios
Background: ZHOU Hui (周辉), Deputy Director of the Cyber and Information Law Research Office at the Institute of Law of the Chinese Academy of Social Sciences (CASS), has published a long essay in an academic journal calling for the urgent adoption of comprehensive national AI legislation. Of particular relevance to frontier safety, Zhou argues that potential future superintelligent systems could pose existential risks, and that China’s central government should therefore have explicit legal authority to suspend the development or deployment of such high-risk AI technologies.
A reminder on where China stands on an AI Law: In 2023, the State Council first announced plans for an “AI Law.” However, more recent legislative plans by the NPC have been more vague, suggesting that while a comprehensive AI law remains under consideration, it is not currently treated as an urgent legislative priority. Meanwhile, two groups of legal scholars produced “model AI laws,” including one led by Zhou.
Content: Zhou characterizes China’s existing AI governance framework as fragmented and insufficient, relying on a mix of high-level principles, sector-specific rules, and low-level administrative regulations. This approach, he argues, suffers from weak implementation, poor coordination, and unclear authority. He rejects the idea that China can apply a “develop first, regulate later” logic to AI, emphasizing that AI’s systemic risks demand early, comprehensive legal intervention. He also claims that regulation is crucial to gain influence over global rule setting for AI.
Therefore, Zhou advocates for overarching AI legislation that would regulate the full AI lifecycle (design, development, training, deployment, operation, iteration, and retirement) and serve three key functions: promoting development, preventing risk, and constraining AI from inappropriately expanding the power of public and private entities.
Although not the essay’s main focus, Zhou explicitly addresses catastrophic risk scenarios. Citing warnings from leading AI scientists about misaligned superintelligent systems threatening human survival, he argues that further measures that would be necessary after passing comprehensive AI legislation should include a legal “safety valve” granting the State Council authority to temporarily suspend the development or deployment of specific high-risk AI technologies. He stresses that such powers must be subject to strict triggering criteria to prevent abuse.
Implications: The essay shows that senior Chinese legal scholars are engaging with frontier AI risk and contemplating strong state intervention mechanisms for extreme scenarios. At the same time, Zhou acknowledges that there is no consensus within China’s legal community on the need for a comprehensive AI law at present, and official policy signals remain mixed—leaving uncertainty over both the timing and likelihood of a national AI Law.
What else we’re reading
James Zhang et al, Emergency Response Measures for Catastrophic AI Risk, arXiv, Oct 28, 2925.
Dean Ball, The Bitter Lessons, Hyperdimensional, Nov 14, 2025.
Vincent Chow, DeepSeek, Alibaba researchers endorse China’s ‘misunderstood’ AI regulatory framework, Nov 28, 2025.
Concordia AI’s Recent Work
Media references
Concordia AI CEO Brian TSE and Head of International AI Governance Kwan Yee NG provided insights for a Nature News explainer on China’s role in international AI governance. Kwan Yee was also cited in a Nature News story on DeepSeek’s founder LIANG Wenfeng, and a Nature Editorial cited Concordia AI.
Concordia AI’s new Frontier AI Risk Monitoring Platform has been covered by multiple Chinese and international media outlets, including People’s Daily, South China Morning Post (twice), Xinhua News Agency’s Economic Information Daily, and IT Times (Cn, En).
Events
Concordia AI CEO Brian TSE participated in the International AI Standards Summit in Seoul on December 2-3, speaking on frontier AI risk management and implications for data governance on a panel.
Concordia AI co-hosted the Frontier AI in Cybersecurity Workshop on the sidelines of Singapore International Cyber Week 2025, together with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley’s Center for Responsible, Decentralized Intelligence.
Concordia AI, together with FAR.AI and the Singapore AI Safety Hub, organised an AI Safety Meetup in Singapore on 14 November on the sidelines of EAGxSingapore, where Concordia AI International AI Governance Project Manager Jonathan Lee also presented the State of AI Safety in Singapore report.
Concordia AI International AI Governance Senior Research Manager Jason Zhou participated in an international roundtable on AI governance at the Beijing Forum. He provided insights into China’s AI safety governance, discussing current policies, regulations and technical standards.
Kwan Yee Ng and International AI Governance Researcher Gabriel Wagner spoke on China’s approach to AI safety and governance at AI Safety Asia’s “Beijing Roundtable” on 6 November.
Research
Brian Tse and Concordia AI AI Safety Research Manager DUAN Yawen contributed to a new paper “AI Deception: Risks, Dynamics, and Controls” led by Peking University.
Duan Yawen contributed to a World Economic Forum white paper “AI Agents in Action: Foundations for Evaluation and Governance.”
Feedback and Suggestions
Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.
Excellent update. Will refer to it in an upcoming Substack on where key players are on AI governance in the lead up to the Indian AI Impact Summit in Delhi in February, where we will be sponsoring several events.....