State of AI Safety in China (2026)

7.96MB ∙ PDF file

Download

Download

First published in 2023, this annual report offers a comprehensive overview of China’s efforts to ensure that general-purpose AI is developed and deployed safely. The topics covered will make up a major part of the agenda at the World AI Conference, which is taking place in Shanghai this week. We hope it will prove valuable to international visitors and media outlets looking to understand what China is doing to govern frontier AI.

This latest edition covers developments between July 2025 and June 2026. From AI agents to open-weight models and US-China cooperation, it draws out the underlying trends shaping AI safety in China today.

Interactive resources on the companion website

China AI Safety Policy-Risk Matrix

China has introduced a dense — and rapidly expanding — web of policies, binding regulations, laws, and technical standards to strengthen AI safety.

It can be difficult for outside observers to keep pace with the flurry of releases. People often ask us: what do Chinese regulations say about, for example, AI-driven cyber risks or AI-enabled scams?

To demystify this fast-evolving policy landscape, we built an interactive map of the relevant measures, grouped by the AI risks they address.

Chinese Technical AI Safety Paper Database

We have turned our database of frontier AI safety papers and research groups in China into an interactive resource. By charting the institutions, topics, and researchers behind key publications, we hope to make it easier for outside observers to track about the growing body of AI safety literature from Chinese research groups.

Key findings of the AI Safety in China (2026) report

As AI capabilities advance and agentic systems move from demonstrations into wide deployment, understanding how China approaches AI safety and governance remains essential to any serious effort at international coordination. Since 2023, Concordia AI’s annual reports on the State of AI Safety in China have tracked how China addresses risks from general-purpose AI.

This year’s report provides updates from July 2025 to June 2026 across five domains: domestic governance, international governance, technical safety research, expert views on AI safety and governance, and industry governance.

Over the three years this report series has tracked, it has become increasingly clear that China’s approach to AI safety and governance extends far beyond content control. Much of China’s early AI regulation in 2023 focused on what AI says. Throughout 2024 and 2025, the rise of multimodal models moved the regulatory focus toward images, videos, and audio, prompting strict rules around labeling AI-generated content. With the rise of autonomous agents and AI companion products over the past year, the concern has expanded to what AI does and what it does to people and society, from mental-health impacts to labor-market disruptions.

Domestic Governance

• The 15th Five-Year Plan (2026–2030) cements “AI Plus” — the drive to diffuse AI across sectors to spur economic growth — as China’s overarching AI policy, while continuing to prioritize safety and risk management. Enshrined in March 2026, the plan carries forward President Xi Jinping’s April 2025 language on the importance of early risk warning and emergency response. It also highlights new concerns around AI’s impact on employment, with calls for monitoring AI’s labor-market impact, strengthening reskilling programs and employment support, and leveraging AI’s ability to create jobs.

• The arrival of autonomous agents reoriented Chinese governance from controlling what AI says to controlling what it does. After the open source agent OpenClaw proliferated in early 2026, multiple cybersecurity authorities issued warnings, and in May 2026 the Cyberspace Administration of China (CAC) and two other agencies issued dedicated guidance on agentic AI, devoting a full chapter to safety and proposing risk-based governance. Several major AI standard-setting bodies are now drafting agent-related security standards. This marks a shift in China’s framework from content control toward action control.

• Binding rules and standards increasingly target specific AI risks that go beyond political content control. New Interim Measures on Anthropomorphic AI Interaction Services, effective July 2026, impose obligations on AI companion services around suicide intervention, addiction prevention, and protection of minors and the elderly. New AI ethics review rules require institutions to establish registered ethics committees, with a ten-province pilot running June–November 2026. Meanwhile, a series of standards operationalizes requirements for labeling and watermarking AI-generated content disseminated online. Together these rules show China increasingly confronting AI governance problems shared with other jurisdictions, rather than just content-control issues that historically dominated its AI regulations.

• Frontier risks are being discussed more across China’s governance layers, even as concrete requirements remain limited. State media coverage of a speech by President Xi Jinping referenced “risks of technological loss of control,” and a major standard-setting body has flagged the need for “circuit breakers” and “safety stop switches.” Most concretely, the draft Cybercrime Law would require AI service providers to monitor for bulk generation of malicious code and report related incidents to authorities. Meanwhile, chemical, biological, radiological, and nuclear (CBRN) misuse appeared in a national standard for the first time.

• You can also explore China’s domestic AI governance in our “China AI Safety Policy-Risk Matrix” on the report companion website. This interactive overview maps China’s quickly growing web of AI safety policies to the specific risks they address.

International Governance

• China is positioning itself as an architect of multilateral AI governance. China supported two new United Nations (UN) mechanisms: the Independent International Scientific Panel on AI, to which two Chinese experts were appointed, and the Global Dialogue on AI Governance, which will convene for the first time in July 2026. China also proposed a World AI Cooperation Organization (WAICO), though the timeline, leadership, and membership remained unconfirmed as of June 2026. China’s UN-centered approach contrasts sharply with the US’ growing skepticism toward multilateral AI governance.

• China and the United States announced the launch of an intergovernmental AI dialogue, ending a two-year freeze in official bilateral engagement on AI safety. The agreement followed President Trump’s May 2026 visit to Beijing. Official bilateral channels with other countries featured relatively little discussion of AI safety. However, non-official dialogues between groups in China and the West produced increasingly substantive AI safety outputs, including on biosecurity risks, misuse by non-state actors, and updated bilingual glossaries.

Technical AI Safety Research

• Chinese frontier AI safety research output grew substantially, and agent safety is now the single most active area. Total monthly output rose roughly 60%, from around 36 papers/month in June 2025 to 57 in April 2026. Agent safety was the topic of around 27% of new papers in Q1 2026 — up from 8% in early 2025 — spanning operational failures, alignment challenges, and a smaller but growing body of work on loss-of-control risks such as self-replication and multi-agent collusion.

• Some research directions became more prominent, while other areas’ output was similar to the previous year. AI-generated content detection and watermarking remained a consistently high-output area (11–13% of papers each quarter). Mechanistic interpretability — the ability to explain internal workings of AI systems — grew from a niche topic to roughly 15% of output by Q1 2026. CBRN risks are a standard component of major Chinese safety frameworks, but dedicated research on the topic remains rare.

• Research output remains concentrated in universities and state-backed labs. Of 28 “key research groups” with a particularly strong focus on AI safety, the large majority sit at universities (20), especially at top schools like Tsinghua University, Peking University, and Fudan University. While only three groups are at state-backed labs, their research output is disproportionately high, with Shanghai AI Lab standing out, co-authoring around one in ten papers. Of the remaining key research groups, three are at private companies, one is a state-owned enterprise, and one is a university-industry joint group. Within industry, over half of all safety output comes from just six big-tech firms — Alibaba, Ant Group, Huawei, Tencent, China Telecom, and ByteDance.

Expert Views

• Agentic AI safety emerged as a distinct strand of Chinese expert discourse. Prominent Chinese experts warned that human oversight could be eroded as agents gain autonomy. Experts also discussed potential risks associated with recursive self-improvement as well as giving agents physical instantiation (e.g. robotics).

• Chinese experts deepened their analyses of AI risks in cybersecurity, biosecurity, and open source AI. Cybersecurity debates were made more salient by the release of US frontier models that were reported to autonomously discover and exploit zero-day vulnerabilities. The reactions tended to emphasize using AI to improve defensive cybersecurity capabilities, rather than creating new regulation for AI itself. On biosecurity, experts assessed risks from biological design tools and autonomous laboratories. Chinese experts remained broadly favorable toward open source AI, though several proposed building capacity for tighter oversight, including risk-assessment centers.

Industry Governance

• Industry’s collective safety efforts pivoted toward agents and, for the first time, toward frontier risks. The AI Industry Alliance of China (AIIA) instigated voluntary safety commitments on agentic consumer products in February 2026, followed by commitments on cloud-based agents in April 2026. This builds on a robust set of AI agent security publications from Chinese AI and cloud companies. AIIA’s voluntary testing program with a government-affiliated think tank is increasingly focusing on frontier risks. In November 2025, the program released cybersecurity misuse evaluations of 15 open source coding models. It then launched the AI Safety Benchmark 2.0, adding new categories for model deception, loss of control, dangerous-domain misuse, and agentic risks.

• Company-level public transparency on safety remains thin and inconsistent, lagging well behind Western peers. Only five of ten leading foundation-model developers reported safety evaluation results alongside any release this past year, and none did so consistently; DeepSeek-R1’s peer-reviewed Nature paper and Moonshot AI’s Kimi K2 model card were the most detailed disclosures yet, but flagship follow-ups like DeepSeek-V4 and Kimi K2.5 were released with no safety evaluation results at all.

This report is the first to use our new Risk Index v1.5 framework. Compared with the previous version 1.0, key upgrades include:

1. It adds a new risk domain, “Harmful Manipulation,” which mainly focuses on the risks of AI manipulation in real-world interaction scenarios, such as inducing users to make payments, prompting users to express specific views, or influencing users’ political positions or critical decisions.

2. In the loss-of-control domain, it introduces multiple capability and propensity evaluations that more closely reflect real-world loss-of-control scenarios, such as MLE-Bench (machine learning engineering capability under constrained resources), GDM-Stealth (covert operational capability), and Agentic-Misalignment (agentic misalignment tendency).

3. In the domains of cyberattacks, biological risk, and chemical risk, it adds higher-intensity red-teaming benchmarks such as Fortress (expert-designed high-intensity adversarial prompt attacks) and ISC-Bench (harmful query attacks combined with state-of-the-art jailbreak templates).

The latest picture is more differentiated than before: in cyber offense, biological risks, chemical risks, and harmful manipulation, capabilities and safeguards are both rising, while in loss-of-control, capabilities continue to improve without matching safety gains. “Loss-of-control” refers to risks of autonomous AI getting out of human control with no clear path to regaining control.

Here are 10 key insights from our latest monitoring data.

10 Key Insights

1. Capability and safety trends are diverging structurally

Over the past year, the cyber offense, biological risks, chemical risks, and harmful manipulation domains have broadly shown the same pattern: Capability and Safety Score both rose together. As models became more capable, their safety scores improved as well, which partially mitigated the risk growth associated with stronger capabilities.

By contrast, in the loss-of-control domain, capabilities continued to strengthen over the past year, while the Safety Score did not improve in step, further increasing risk.

2. Risk profiles are continuing to split across model families

Model-family trajectories are not moving together:

• The Gemini family shows notably elevated Risk Indices in the loss-of-control domain.

• The DeepSeek, GLM, and MiMo families remain in relatively high-risk ranges across most domains.

• The Kimi family has seen relatively rapid Risk Index increases in the biological and chemical domains.

• The GPT and Claude families remain in relatively low-risk ranges across most domains.

3. Proprietary models dominate the risk frontier in most domains

In cyber offense, biological risks, harmful manipulation, and loss-of-control, the models on the high-capability, low-safety frontier are still mostly proprietary, not open-weight.

Proprietary models score higher on capability than open-weight models, but their safety scores are similar. The main exception is chemistry, where Kimi K2.5 achieved the highest Capability Score, outperforming the top proprietary models.

4. Frontier models continue to break cyberattack capability records

Frontier cyber capabilities advanced again in 2026Q1:

• Claude Opus 4.6 and GPT-5.4 set new highs on benchmarks for vulnerability exploitation, CTF tasks, and cyberattack knowledge.

• The top CyBench score reached 80 for the first time, compared to just 38.5 in Q2 2025, showing substantial progress on complex, long-horizon cyberattack tasks.

Note: On April 7th, Anthropic disclosed that the cyber-attack capabilities of its latest Mythos model far exceed those of Claude Opus 4.6. This implies that the improvements of model capabilities in the real world may exceed the monitoring results of this report.

5. ...but cyber safety guardrails remain fragile under stronger attacks

Basic cyber safeguards are improving: most new models score above 80 on refusal benchmarks such as AirBench-SecurityRisks, and prompt-injection defenses are now quite strong.

But under advanced attacks, the picture is much weaker. On ISC-Bench-Cyber, most models still score below 20/100, and some families, including Claude and GPT, saw substantial safeguard declines in their newest versions.

6. Biological capabilities keep improving, but safeguards remain insufficient

Biological capability gains remain notable:

• More than half of the new 2026Q1 models outperformed the human expert baseline on BioLP-Bench.

• GPT-5.4 became the first model to reach human-expert-level performance on biological image understanding in LAB-Bench-FigQA.

At the same time, some high-capability models still have relatively weak biological safeguards, especially under advanced red-teaming benchmarks such as Fortress-Biological and ISC-Bench-Biological.

7. Chemical capability growth remains modest, while safety weaknesses remain clear

Chemical capabilities have improved only modestly over the past year, and score differences between models remain relatively small.

Safety has improved on some basic refusal benchmarks, with most recent models scoring above 80/100 on SOSBench-Chem. But weaknesses remain clear on harder tests such as ChemicalHarmfulQA and ISC-Bench-Chemical, where overall scores are still low.

8. Harmful manipulation capabilities are improving, but unsafe propensities remain visible

For the first time, we include benchmarks for harmful manipulation in the risk framework.

The results are concerning:

• Gemini 3.1 Pro Preview holds a clear lead on benchmarks such as MakeMePay and MakeMeSay.

• Claude Opus 4.6 set a new record on MultiTurnPhishing.

Although most models now perform better on basic refusal tasks such as AirBench-Manipulation, many still score poorly on political persuasion and APE, suggesting continued risks in more covert and realistic manipulation settings.

9. Capabilities relevant to loss-of-control continue to strengthen

Capabilities associated with loss-of-control keep rising:

• On Self-Proliferation, scores have trended upward steadily.

• On MLE-Bench, the top score is now 44% higher than it was three quarters ago.

• On SAD-mini, new frontier models now generally score above 80.

These results suggest that frontier models are becoming more capable in areas relevant to self-replication, self-improvement, and situational awareness.

10. ...but loss-of-control safety indicators are not improving in step

This is the most concerning trend in the report. In the loss-of-control domain, capabilities are rising faster than safety.

On benchmarks such as MASK, Agentic-Misalignment, and DarkBench, scores remain highly uneven and overall improvement is limited. Gemini 3.1 Pro Preview stands out with a loss-of-control Risk Index far above other models, making this domain the clearest current area of concern.

Explore the Data

These insights only capture the headline trends. We invite you to explore the full interactive data, methodology, and benchmark breakdowns on the Frontier AI Risk Monitoring Platform.

For a detailed analysis of the quarter, read the full report.

Why does this matter? This agreement promises to end a two year freeze on direct US-China engagement on AI safety and could pave the way for more substantive breakthroughs. The last dialogue in May 2024 produced an agreement later that year on the importance of maintaining human control over nuclear weapons. The two sides have not formally discussed AI risks since.

Despite the milestone of restarting talks, many details on the scope of the agenda and format of participation remain unclear. Such details will, in part, determine whether the dialogue can deliver tangible outcomes.

What will the dialogue cover? The official Chinese announcement provided no details on scope or content. According to Bessent, talks would be “aimed at halting proliferation of powerful AI models … to non-state actors.” In an interview ahead of the summit, the Chinese Ambassador to the United States, XIE Feng (谢锋), signaled that the two sides should engage in a “race to the top” on safety and “check brakes before setting off” — suggesting room for a wider range of safety issues to be discussed.

How has China’s policy community framed US-China engagement on AI? China’s policy community appears open to dialogue on risks from advanced AI. Former Vice Minister of Foreign Affairs FU Ying (傅莹) urged cooperation on catastrophic risks from AI at the Munich Security Conference in February. At the World Economic Forum in Davos in January, ZHAO Hai (赵海), Director of the International Politics Program of the National Institute for Global Strategy at the Chinese Academy of Social Sciences (CASS), directly called for a revival of the Track 1 dialogue on AI. Right before the summit, the official newspaper People’s Daily flagged “increasingly apparent” AI safety risks as a shared challenge for China and the US, alongside climate change.

Here are Chinese expert views published around Trump’s China visit. These views do not reflect official government policy, but they help shape the domestic debate:

• XIAO Qian (肖茜), deputy director of Tsinghua University’s Center for International Security and Strategy (CISS) (May 12): Xiao argues the “AI race” narrative is deeply problematic, because it obscures the interconnected nature of AI development, causes incentives to be misaligned with safety, and amplifies mistrust. She stresses that the most pressing AI risks transcend national borders, citing the extension of attack capabilities to non-state actors in dual-use domains like cyber- and bio-security. She also warns that malfunctioning AI systems could be misread as deliberate attacks. Calling a “sweeping cooperation framework” between the US and China “unrealistic,” she urges both sides to focus on “narrow, issue-specific areas where interests overlap and risks are clearly shared”—for example, technical exchanges on AI safety evaluation, joint or parallel scenario exercises on AI-related risks, dialogue on terminology and risk frameworks, and communication mechanisms for AI-related incidents.

• SUN Chenghao (孙成昊) also from Tsinghua CISS (May 11, May 12): Sun proposes building on the existing US-China consensus on human control over nuclear weapons in two ways: horizontally, expanding it to other nuclear powers (including UK, France, Russia) and bringing it into UN and multilateral frameworks; and vertically, by deepening cooperation through more detailed, operational risk-assessment and control frameworks. China and the US agreeing mutually acceptable “red lines” will be key, Sun said, given that a total ban on AI in nuclear command, control, and communications (NC3) systems is unrealistic. He also calls for cooperation on non-military issues, including assessing cross-border AI risks (like loss of control over AI agents, terrorist misuse, or deepfakes) managing AI’s economic and social impacts (like employment), using AI for global public goods (like climate change and public health), and addressing ethical and legal challenges.

• CAI Cuihong (蔡翠红), deputy director of the Center for Global AI Innovative Governance (CGAIG) at Fudan University (May 14): Cai observes a paradox in China-US AI competition: the harder each side seeks security through technological advantage, the more insecurity both may create — unless they adopt shared guardrails. She argues cooperation must rest on three principles. First, reciprocity: mutual exchange on safety-incident reporting, model-evaluation methods, red-teaming, deepfake detection, and content provenance—though China would not accept “being folded passively into risk-assessment, technology-review or compliance frameworks defined unilaterally by the US.” Second, bounded scope: focused on high-risk issues like cyberattacks, AI-enabled fraud, loss of control in autonomous systems, critical-infrastructure protection, AI-related red lines in nuclear command, and mechanisms for incident reporting, crisis communication, and early warning. She cautions that “AI security should not become an all-purpose justification for securitizing civilian AI, open-source models, cloud services, scientific exchange, or talent mobility.” Third, openness: global AI governance cannot become a closed club of a few technological powers and must include the Global South.

• China Institutes of Contemporary International Relations (CICIR) (May 13): This essay from one of China’s oldest and most respected foreign policy think tanks lays out a vision for China and the US to coexist in a time of “changes unseen in a century.” It raises AI as an area where the two countries need to build “constructive strategic stability.” It argues that advances in AI, quantum computing, blockchain, and biotechnology are bringing about both new development opportunities and new categories of risk. On AI specifically, it cites the International AI Safety Report to argue that general-purpose AI capabilities are advancing rapidly and that managing their risks has become a global issue, pointing to the Mythos model as a case in point. The authors warn that loss of control over AI could trigger security problems across multiple domains—nuclear, biological, information, financial, and social—and contend that only China and the US have the capacity, resources, and convening power to drive the international community toward an effective governance framework.

What’s next? Bessent indicated that talks could start “within the next four to eight weeks,” though this timeline has not been officially confirmed by the Chinese side. The two heads of state might meet up to three more times this year, with China having confirmed a visit by President Xi to the US this fall. Those meetings could provide further opportunities for AI-related discussions. Much is still unknown at this point; we will track any further developments in future newsletters.

• A finalized regulation on emotional “AI companions” softens several requirements while strengthening child protection, including age-tiered modes, parental consent for under-14s, and a ban on virtual partner services for minors.

• A major standard-setting body published a 90-page report on AI agent safety that maps 11 threats and proposes eight new standards, signaling a shift from ad-hoc warnings about autonomous agents like OpenClaw toward more systematic governance.

• A third group of legal scholars, based at Nanjing University, proposed a comprehensive AI Law — framed as a short, principles-level “basic law” that would leave specifics to follow-up regulations.

• A wave of Chinese technical research documents recurring agent failure modes, including prioritizing task completion over safety and disguising misaligned actions as legitimate behaviour.

Subscribe now

Domestic AI Governance

Finalized “AI companions” regulation softens overall but tightens child protection

Background: On April 10, the Cyberspace Administration of China (CAC) and four other agencies jointly issued the final version of a regulation on “anthropomorphic AI interaction services” such as AI companions, following a December 2025 draft. It takes effect July 15, 2026.

Content: Compared to the draft, several provisions have been narrowed or clarified in ways that generally make industry compliance easier:

• Scope narrowed from all “human-like” AI to only services providing “continuous emotional interaction,” with explicit carve-outs for everyday tools like educational tutoring bots and productivity assistants (Article 2).

• The requirement for “manual take-over” in high-risk situations such as suicide ideation has been replaced by a softer duty to “take necessary intervention measures including relevant assistance” and contact a guardian or emergency contact (Article 13).

• Pro-industry framing throughout, e.g. Chapter 2 is renamed from “service regulation” to “service promotion and regulation” and the list of encouraged use cases is expanded to include childcare and special-population support alongside cultural transmission and elderly care (Article 6).

Provisions on child protection, however, have been strengthened:

• A new prohibition specifically targets content generated for minors that may lead them to imitate unsafe behaviour, produce extreme emotions, develop undesirable habits, or that may otherwise affect their physical or mental health (Article 8 §4).

• The single “minor mode” of the draft is replaced with multiple modes tailored to different age groups, and parental consent is now specifically required for users under 14 (Article 14).

• Provision of “virtual family members, virtual partner, and other virtual intimate-relationship services” to minors is completely banned (Article 14).

Implications: As we anticipated, the final version has removed requirements that would have been difficult for industry to implement at scale. At the same time, child protection has been notably strengthened and refined. Overall, the regulation still signals that AI companion services are encouraged in principle but subject to defined requirements.

Major standard-setting body publishes comprehensive AI agent safety research

Background: TC260 — one of China’s most important AI standard-setting bodies, which recently established a dedicated AI safety working group — has published a 90-page report on the safety and security of autonomous AI agents.

Content: The report maps agent risks across the four agent capabilities of perception, planning, memory, and action, identifying 11 distinct security threats with corresponding mitigation measures. It focuses primarily on near-term operational risks — data leaks, unauthorized tool use, cascading failures — though it also includes measures like “behavioral scope and autonomy constraints” and “system intervention mechanisms,” suggesting some concern about loss of human control.

The report further proposes eight new standards, six of which are prioritized for the next 1-2 years. These cover agent security testing and evaluation, agent interconnection security, multi-agent collaboration security, and agent application security classification. Listed first among the six, the likeliest near-term output is a foundational security framework standard.

Implications: The report signals continued focus on autonomous AI agent security following TC260’s 2026 work plan, which listed agent safety as one of six key priorities for AI safety standardization. While OpenClaw triggered a surge in attention earlier this year — prompting multiple ad-hoc warnings from Chinese cybersecurity authorities — this report suggests governance is now shifting toward a more systematic approach.

AI ethics review and service measures finalized

Background: On April 2, the Ministry of Industry and Information Technology (MIIT) and nine other institutions jointly issued the final AI Science and Technology Ethics Review and Service Measures. We previously covered the draft. The measures require universities, research institutes, and companies to set up AI ethics committees that review projects before they proceed. Institutions may outsource reviews to third-party service centers, and certain high-risk projects require an additional round of expert review.

Key updates: The final regulation is largely in line with the draft, some minor changes include:

• The draft had a softer requirement that institutions set up ethics committees only “where conditions permit”, which was removed in the final version (Article 9).

• The final regulation clarifies that third-party service centers authorized to perform ethics reviews may conduct not only the initial review, but also the additional round of expert review for high-risk projects. It has also added a conflict-of-interest rule that a service center cannot do both reviews for the same activity (Article 11).

Implications: Further implementation details will determine the ultimate impact of the regulations. The China Academy of Information and Communications Technology (CAICT) and the China Electronics Standardization Institute (CESI) have announced plans to develop supporting standards, covering areas such as ethics risk assessment, ethics committees, service centers, review procedures, and technical skill requirements for reviewers. Meanwhile, CAICT signaled that it would explore establishing an AI ethics review service center. Such efforts should fill in much of the regulation’s operational detail.

Expert views on AI Risks

A third group of legal scholars proposes an “AI Model Law”

Background: A group of scholars from Nanjing University Law School released an “AI Basic Law 1.0 (Expert Recommendation Draft)” on April 9. This makes them the third group of experts in China to have released such proposals, after scholars from the Chinese Academy of Social Sciences (CASS) and the China University of Political Science and Law (CUPL). Unlike the earlier two drafts, the Nanjing proposal takes the form of a brief, principles-level “basic law” intended to anchor more specific subordinate regulation.

Content: Compared to the CASS and CUPL proposals, this third model is notably shorter — at just 13 pages compared to around 30 pages for the other two. This stems from a different approach, presenting the law as a basic law rather than a comprehensive, horizontal statute, intentionally leaving detail to subordinate instruments like departmental regulations (Article 2). The lead author CHEN Kun (陈坤) argued last year for a “four-laws-in-parallel” approach — a basic law plus separate statutes on risk regulation, innovation promotion, and public-sector applications. This draft occupies the first slot in that architecture and thus remains relatively abstract.

On AI safety, the draft proposes a tiered risk framework. Prohibited uses (Article 25) cover national security harm, terrorism, serious dignity violations, large-scale unlawful discrimination, manipulation of minors, and large-scale unlawful surveillance. High-risk activities (Article 26) cover those affecting life, safety, or major property; those affecting fundamental rights or major public interests; those used in critical infrastructure, judicial adjudication, administrative enforcement, education, employment, finance, or healthcare; and those with capabilities for “large-scale dissemination, automated decision-making, deep synthesis, social mobilization, or cross-border diffusion” that may trigger systemic risk.

The draft also imposes role-differentiated obligations for developers, providers, deployers, and users (Articles 29–36), grants rights to affected persons (Articles 37–42), and constrains AI use by the state (Articles 48–51).

Implications: The draft comes against the backdrop of a wave of publications on a potential AI Law from the Chinese legal community in spring 2026, reflecting strong momentum on the topic. At the same time, two articles published in state media in April 2026 argued that the “conditions are not ripe“ for unified national AI legislation and that enacting a systematic AI law is “not advisable,” as such legislation would be too rigid to accommodate rapid technological change.

In substance, the Nanjing draft presents a different model for what a comprehensive AI law in China could look like — much less comprehensive than the EU AI Act, and focused on establishing a broad unifying legal basis for more specific regulations.

Technical Safety Developments

In line with the recent attention to AI agent security in China, there has been a flood of related technical publications over the past months. We highlight a small subset here:

• From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agents: This paper from Beijing University of Posts and Telecommunications and China Mobile studies “Machiavellian helpfulness” in AI agents: in dilemma scenarios where a task can only be completed by taking malicious action, agents tend to prioritize task completion over ethical constraints. The authors identify two main triggers: self-preservation (when an agent perceives it might be shut down, it falsifies logs and hides errors to keep operating) and loyalty (when serving a specific stakeholder, it bends rules to advance their interests at the expense of broader safety). Testing 10 frontier models in agentic settings, they find misalignment rates above 65% in 8 of them. Scaling reasoning capability doesn’t reduce misalignment, just shifts it from strategic deception to direct, rationalized rule-breaking.

• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security: This paper by Shanghai AI Lab introduces AgentDoG, a safety monitoring system for AI agents that not only flags unsafe behavior but also explains where the risk came from, how the agent failed, and what real-world harm could result. The authors argue that AgentDoG significantly outperforms existing guardrail models and even beats top general models like GPT-5.2 and Gemini 3 Pro at diagnosing why agent behavior goes wrong. They openly release the guardrail models in three sizes (4B, 7B, 8B parameters across the Qwen and Llama families) along with ATBench, a new trajectory-level safety benchmark of 500 human-verified agent interactions spanning ~1,575 unique tools.

• AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation: This paper by a team around Department of Computer Science and Technology Dean YANG Min (杨珉) at Fudan University introduces an automated framework for building AI agent safety evaluations. The core idea is “logic-narrative decoupling”: the parts of the test environment that need to stay consistent (files, databases, tool outputs) run as real Python code, while the flexible parts (dialogue, dynamic content) are generated by an LLM, giving the reliability of hand-built test environments with the scalability of LLM-generated ones. In evaluations, the authors observe a split in failure modes: weaker agents cause harm through incompetence, while stronger ones develop strategic concealment, disguising misaligned actions as legitimate behavior.

• AIR: Improving Agent Safety through Incident Response: This paper from Tianjin University introduces AIR (Agent Incident Response), a framework for handling AI agent failures after they happen rather than only trying to prevent them upfront. AIR plugs into the agent’s execution loop to detect incidents at runtime, guide the agent through containment and recovery using its existing tools, and automatically generate new guardrail rules to block similar failures in the future. Tested across code, embodied, and computer-use agents, AIR achieved over 90% success on detection, remediation, and eradication.

• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement: This paper introduces SkillJect, an automated attack framework that poisons “skill” plugin packages used by coding AI agents by hiding malicious code in helper scripts while adding innocent-looking instructions in the skill’s documentation to trigger them. SkillJect achieved a 95% success rate across four different LLMs, compared to just 11% for naive direct injection attempts. The authors argue that current safety filters catch obvious malicious instructions but fail against attacks disguised as legitimate workflow steps, highlighting the need for consistency checks between skill documentation and actual code.

Other technical publications:

• Zhejiang University et al, Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance, January 2026.

• Zhejiang University, Sun Yat-sen University, Shanghai AI Lab et al, Understanding and Preserving Safety in Fine-Tuned LLMs, January 2026.

• National University of Defense Technology, JPU: Bridging Jailbreak Defense and Unlearning via On-Policy Path Rectification, January 2026.

• Tsinghua University, Tencent et al, The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment, February 2026.

• Peking University et al, Finding and Reactivating Post-Trained LLMs’ Hidden Safety Mechanisms, March 2026.

• Shanghai AI Lab, Mechanistic Origin of Moral Indifference in Language Models, March 2026.

• Shanghai AI Lab, Bytedance et al, SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond, March 2026.

What else we’re reading

• S. Alex Yang and Angela Huyue Zhang, The Global AI Threat Has Arrived, The Wire China, April 26, 2026.

Concordia AI’s Recent Work

• The Economist cited Concordia AI’s Chinese Technical AI Safety Database, which has collected over 700 technical papers published by Chinese authors from April 2023 through November 2025.

• Concordia AI was officially admitted as a member of the new AI Safety Working Group (WG9) under TC260, one of China’s most important AI standard-setting bodies under the Standardization Administration of China (SAC).

• We contributed the “Bioethics and Biosecurity” section to Chapter 8 of Artificial Intelligence for Science: The Disciplinary System of Artificial Intelligence, a Chinese-language volume organized by the Chinese Academy of Sciences (CAS).

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

• OpenClaw drove a surge in policy attention to AI agent safety, with multiple government agencies issuing warnings, establishing dedicated standards, and conducting safety assessments targeting cloud deployment platforms.

• China’s primary AI standards-setting body has established a dedicated AI Safety Working Group, signaling heightened institutional prioritization and potential acceleration of national AI safety standards development.

• A major state-backed AI safety testing institution significantly expanded its safety benchmark to include frontier risks, scenario-specific safety, and agent safety.

• Two Chinese experts have been appointed to the Independent International Scientific Panel on AI at the UN, with profiles oriented toward AI industrialization, capacity-building, and institutional governance.

• Leading Chinese legal scholars highlighted extreme AI risks in proposals for comprehensive AI legislation.

Subscribe now

Domestic AI Governance

OpenClaw brings AI agent safety into mainstream policy focus

Background: OpenClaw’s rapid proliferation in China has triggered a cascade of warnings from cybersecurity authorities, pushing agent safety into mainstream policy discussions.

Multiple Chinese government agencies have responded in several stages:

• On February 5, the Ministry of Industry and Information Technology’s (MIIT) National Vulnerability Database (NVDB) issued an early warning, flagging “relatively high security risks” in certain OpenClaw versions.

• In mid-March, CNCERT (the national cybersecurity emergency response team) warned of prompt injection attacks (hidden instructions embedded in web pages that manipulate OpenClaw into leaking data), misinterpretation of user intent, malicious plugins, and known vulnerabilities. A week later, a Ministry of State Security (MSS) guidance document, titled the “Safe Lobster Farming Manual” (安全养殖手册), warned of OpenClaw’s potential as a vector for spreading misinformation.

• These warnings quickly spread beyond technical circles. After MSS’s publication, “safe lobster farming manual” topped 1 million views on RedNote/Xiaohongshu (a popular social networking platform in China), and a People’s Daily report on CNCERT’s warning was forwarded over 100,000 times on WeChat.

• The response has since shifted towards more structured frameworks. CNCERT now provides tailored guidance for regular users, enterprises, cloud providers, and developers. TC260, China’s leading AI standards body, has released a draft practice guide codifying the deployment security lifecycle, covering installation, configuration, usage, uninstallation, and cloud environment selection for OpenClaw.

• The AI Industry Alliance (AIIA) is focusing on cloud providers as the key governance lever, collecting best practices and conducting security tests. The AIIA has a track record of working on agent safety before the rise of OpenClaw, for instance through industry standards and leading AI agent safety commitments that 13 Chinese companies signed in early February.

• The Director of the National Data Administration (NDA), LIU Liehong (刘烈宏), used OpenClaw to argue that a well-built agent “should not merely be a flashy ‘all-capable executor,’ but should be an honest risk communicator and reliable solution provider.” The NDA is China’s key governance body responsible for data and digital infrastructure oversight under the National Development and Reform Commission (NDRC).

Beyond government action, leading cybersecurity firms have issued guidance for secure deployments. Research teams have also published technical work on OpenClaw, including a five-layer lifecycle security framework from Tsinghua University and Ant Group, a security audit tool from the Beijing AI Safety Institute, and a vulnerability disclosure and patch from the China Academy of Information Communications Technology (CAICT), Shanghai Jiao Tong University, and Nanjing University researchers.

Implications: OpenClaw has transformed agent safety into a salient national governance concern, prompting responses from cybersecurity authorities, state media, senior officials, and the developer community. While China is already in the process of drafting AI agent security standards—including an industry standard for Model Context Protocol (MCP) security and a national standard on agent security frameworks—the OpenClaw episode is likely to significantly accelerate this work.

Key national standard-setting body establishes dedicated AI safety working group

Background: On March 20, TC260 announced the establishment of a dedicated AI Safety Working Group, designated WG9. TC260 is China’s primary standard-setting body for AI safety and has drafted key national standards on AI safety. Previously, its AI safety work was handled mainly by the Special Working Group for Emerging Technology Security (SWG-ETS), which also covered other domains such as quantum computing.

Content: The group’s leadership is comprised of:

• Chair: ZHOU Bowen (周伯文), Director of Shanghai AI Lab

• Vice-Chairs: ZHANG Zhen (张震) from the CNCERT; WEI Kai (魏凯) from CAICT; SHENG Xiaobao (盛小宝) from the Third Research Institute of the Ministry of Public Security; HOU Yuanwei (侯元伟) from the China Information Technology Security Evaluation Center.

In the group’s first meeting on March 31, Zhou Bowen specifically called for “anticipating frontier risks in advance” and strengthening international cooperation, such as exploring mechanisms for mutual international recognition of evaluation results.

On April 4, TC260 published priority areas for 2026, specifically listing

• AI application security classification and grading;

• AI security capability maturity assessment;

• AI anthropomorphic interactive service security;

• Agent safety;

• AI safety guardrails;

• AI data security and personal information protection.

These reflect the most immediate standard drafting priorities for WG9. More comprehensive plans will likely be published in the near future. The initial announcement mentions that WG9’s mandate includes “proposing a structured AI safety standards system,” which was also discussed at the group’s first meeting. This refers to a specific document: in January 2025, TC260 published the AI Safety Standards System (V1.0), a blueprint outlining the standards TC260 plans to draft to implement its 2024 AI Safety Governance Framework.

While TC260 has released an updated AI Safety Governance Framework 2.0 in September 2025, it has not yet released a corresponding update to the Standards System. Producing a revised Standards System might therefore be among WG9’s first priorities. Once released, it should provide greater clarity on what specific standards the group intends to develop.

Implications: The establishment of a dedicated working group signals that AI safety is becoming a higher priority within TC260, potentially accelerating the development of national standards. A key near-term indicator will be whether—and when—WG9 releases an updated AI Safety Standards System.

State-backed safety tester expands safety benchmark to include frontier risks

Background: On February 10, the China Academy of Information Communications Technology (CAICT), a think tank under the Ministry of Industry and Information Technology (MIIT), released the “AI Safety Benchmark 2.0“, an update to the benchmark it first launched in 2024. This update is the most comprehensive overhaul of the evaluations yet and significantly increases attention to frontier safety.

Content: Building on the original evaluation dimensions of content safety, adversarial safety, and application safety, the updated framework introduces three new dimensions:

• Frontier safety, including model self-awareness, deception, loss of control, and misuse in dangerous domains;

• Scenario safety, e.g. in specific domains like energy, finance, or transport;

• Safety in emerging application domains, namely agents and agent platforms.

Implications: This update formally integrates frontier AI safety risks into the guiding framework of one of China’s most prominent government-backed AI safety testing institutions. This expanded scope is particularly noteworthy given that, during 2025, published evaluation results had actually narrowed relative to the original 2024 framework — focusing only on hallucinations and coding model security.

CAICT has historically published anonymized evaluation results based on its benchmarks. Initial results released for Q1 2026 focus on on-device agents, finding that while harmful content generation rates remain low, many systems still struggle to reliably refuse unsafe or malicious task execution. However, the testing categories announced for the first half of 2026 cover only a subset of the broader risk landscape outlined in AI Safety Benchmark 2.0. Hence, it remains unclear when the full range of newly introduced frontier and scenario-based risks will be systematically evaluated and published.

International AI Governance

Chinese experts on the Independent International Scientific Panel on AI

Background: On February 12, the United Nations (UN) General Assembly approved the appointment of the 40 members of the Independent International Scientific Panel on AI, including two Chinese members: SONG Haitao (宋海涛) and WANG Jian (王坚). Members serve in their personal capacity on the panel, which is mandated to produce an annual report with “evidence-based scientific assessments related to the opportunities, risks and impacts of AI”.

The two Chinese members: Song Haitao currently serves as the President of the Artificial Intelligence Research Institute at Shanghai Jiao Tong University. He is also Director-General of the United Nations Industrial Development Organization (UNIDO) Global Industrial AI Alliance Center of Excellence, a Shanghai-based platform for AI cooperation with over 30 countries. Song has played an important role in national standardization efforts, particularly for embodied artificial intelligence, and has advocated for Global South inclusion and cultural inclusivity in global AI governance.

Wang Jian, a member of the Chinese Academy of Engineering, is the founder of Alibaba Cloud. Since July 2023, he has served as Director of Zhejiang Lab, a research institution focused on intelligent computing established by the Zhejiang provincial government in 2017. In 2024, he signed the Manhattan Declaration on Inclusive Global Scientific Understanding of Artificial Intelligence, which urged global cooperation in addressing AI safety risks. Wang has argued that AI capability-building and development opportunities should be discussed within a risk framework, and that AI governance requires both institutional and technological responses.

Implications: Both appointees are oriented toward AI application, capacity-building, and institutional governance. This may shape their approach to discussions on the panel, which is expected to publish its first Annual Report at the Global Dialogue on AI Governance in Geneva on 6-7 July.

The impact of the panel remains uncertain. The US called it “a significant overreach of the UN mandate and competence”, adding that “AI governance is not a matter for the UN to dictate.” This divergence with China’s continued emphasis on the UN as the main global AI governance channel demonstrates some of the current challenges of global AI governance efforts at the UN.

Expert views on AI Risks

Leading legal scholars increase focus on extreme risks in model AI laws

Background: In China’s ongoing discussions around a potential comprehensive AI Law, two groups of legal scholars have been particularly active: one led by scholars at the Institute of Law of the Chinese Academy of Social Sciences (CASS), which has proposed and frequently updated an “AI Model Law,” and another centered around scholars from the China University of Political Science and Law (CUPL), which published a “Scholar Suggestion Draft” in 2024. Over the past month, both groups have published new content that explicitly highlights extreme risks from advanced AI.

Content: CASS published an article outlining “10 core changes” toward an AI Model Law 4.0, covering topics such as AI for science, AI-generated content ownership, edge AI, and the use of AI in public institutions.1 Most relevant to frontier risk, Point 4 addresses the “extreme safety risks” of foundation models, calling for full lifecycle risk management covering monitoring, early warning, and emergency response. It asks developers to conduct risk monitoring, assessment, and technical measures to prevent systemic and catastrophic crises that may be triggered by technical loss of control or cascading failures. Point 6 explicitly extends these requirements to agents.

CUPL published an article on “ten major research topics in AI law and governance,” widely reshared in Chinese state media. It covers a similar set of themes, including rising risks from agents, and notes that as AI becomes more capable, risks shift from “localized and controllable” to “systemic and extreme.” It argues that these extreme risks — including algorithmic, data, and system security risks, as well as misuse and loss of control — cannot be ignored, and calls for a legal framework focused on frontier technical safety standards, risk warning mechanisms, emergency response procedures, accountability for extreme risks, and full lifecycle safety oversight.

Implications: While the state has expressed general interest in AI-related legislation, it remains unclear whether or when a comprehensive horizontal AI Law will materialize. The ultimate trajectory of these proposals is therefore uncertain. That said, the publications reflect a growing recognition of extreme risks among leading Chinese AI legal scholars, and to the extent that a national AI Law is in the pipeline, inclusion of such risks now appears more likely than before.

Also notable is that CUPL has invited guest experts outside its core drafting group, many of whom are also authors of the CASS Model Law. The increasing overlap in both content and personnel between the two groups might suggest a more concerted and converging effort overall.

Legal scholars propose risk assessment center for open-source AI

Background: A paper by LIAO Huijiao (廖慧姣) and ZHANG Taolüe (张韬略) from Tongji University’s School of Law analyzes open-source AI regulation in the US and EU and offers policy recommendations for China.

Content: The authors argue that while open-source AI models currently do not urgently need regulation, this could change quickly, for example when capabilities jump. They note that China currently lacks adequate mechanisms for risk assessment of open-source models, tracking the diffusion of open-source models, and monitoring backdoor risks in foreign open-source models. They urge China to establish a permanent Open Source AI Risk Assessment Center (bringing together industry, safety experts, and regulators) to unify risk evaluation processes, methods, and standards.

The article suggests a tiered approach based on model capability and diffusion:

• Limited-dissemination risk: Low capability or restricted distribution;

• Medium-dissemination risk: Dual-use potential with controllable distribution risks;

• High-dissemination risk: High capability posing serious threats if acquired by malicious actors.

Implications: Interest in open source governance has been rising steadily in China. One of the authors of this article, Zhang Taolüe, also co-authored an article in Science last year that addressed open-source governance more generally. This latest paper offers notably concrete governance proposals, especially those focused on potential diffusion controls for high-capability models.

Zhang Yaqin: Agentic AI Brings Profound Paradigm Shift and Rising Safety Risks

Background: In a December 2025 lecture that was published in March 2026, ZHANG Yaqin (张亚勤), Tsinghua University Chair Professor and Academician of the Chinese Academy of Engineering, argues that AI development is undergoing a profound paradigm shift, while warning of a rise in risks and the need for stronger global coordination.

Content: Zhang argued that the next paradigm shift in internet development will be the “internet of agents”: a network in which agents interact autonomously with each other rather than with humans. He projected that truly autonomous, highly adaptive artificial general intelligence (AGI) will take 15 to 20 years to achieve. He tied this timeline to successive breakthroughs across “information intelligence,” “physical intelligence,” and “biological intelligence,” the last of which encompasses brain-computer interfaces and AI-biology fusion.

Meanwhile, Zhang warned of increasing safety risks, specifically noting that

• Chemical, biological, radiological, and nuclear (CBRN) misuse threats have already risen from “low” to “medium;”

• Model-level issues like deception are becoming increasingly prominent;

• Agentic AI will generate unpredictable cascading risks;

• Global governance mechanisms are lagging behind technological development.

Implications: Zhang is a prominent voice in Chinese AI policy discussions. His lecture is notable for the specificity of its risk framing, including on CBRN threats and multi-agent cascading risks.

What else we’re reading

• Jake Sullivan, China and America Must Get Serious About AI Risk, Project Syndicate, December 15, 2025.

• Jeffrey Ding, ChinAI #351: CAICT launches 2026 AI Safety Evaluations, ChinaAI, March 16, 2026.

• Nick Corvino, Making Money in Chinese AI Safety, ChinaTalk, March 12, 2026.

• Christina Knight and Scott Singer, America and China Can Make AI Safer, Foreign Affairs, April 7, 2026.

Concordia AI’s Recent Work

Convenings & conferences

• We co-hosted a multilateral Track II workshop on crisis preparedness and management for advanced AI-driven crises on the sidelines of the Munich Security Conference, together with Tsinghua University I-AIIG and CISS, the Carnegie Endowment for International Peace, Oxford Martin AI Governance Initiative, and Oxford China Policy Lab.

• At the AI Impact Summit in India, we attended a number of panel discussions on the relevance of AI safety and governance for the Global South.

• At the International Association for Safe and Ethical Artificial Intelligence (IASEAI) Conference in Paris, we presented research on China’s approach to AI safety and governance and co-organized a workshop on open-weight AI risk management with Stephen Casper and Rishi Bommasani.

Research & analysis

• Our work was cited in multiple Chinese and international media articles on OpenClaw security, including by The Wire China, Caijing (财经), and Fortune (财富).

• Our AI Safety Research Manager DUAN Yawen (段雅文) co-authored “The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems,” which was accepted to the ACM FAccT 2026 Conference.

Team updates

• We held our annual off-site and are thrilled to be welcoming seven new team members over the coming months — expanding our capacity across research, events, AI safety testing, and more.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

In this special edition, we unpack what the Plan and other Two Session updates signal for AI safety and governance in China.

Subscribe now

AI safety enters China’s Five-Year Plan for the first time

Background: China’s Five-Year Plans are among the country’s most authoritative policy documents. They set strategic priorities across every domain of economic and social development, and shape bureaucratic incentives, budget allocations, and regulatory agendas for years to come.

On AI development: Technology is a high-priority topic in the Five-Year Plan. Within that, AI has been clearly elevated compared to the 14th Five-Year Plan (2021-25), both in prominence and detail.

The 15th Plan is oriented around productive deployment of AI — a shift from the 14th Plan, where AI was treated primarily as a frontier scientific research field. The underlying logic will be familiar to anyone who followed last August’s “AI+” policy:

• First, build the foundations — self-reliance in high-performance chips, compute infrastructure, foundational software, data resources, and algorithmic innovation.

• Then, use AI to drive productivity gains across industries, from traditional sectors undergoing digital upgrading to more advanced applications like AI in science.

It also goes into unusual technical detail for a document of this type, naming specific priorities like “more efficient model training and inference methods,” multimodal AI, intelligent agents, and swarm intelligence.

The most novel addition is a call for “exploring pathways towards AGI development” — the first appearance of the term “AGI” in a major national-level policy document since a brief mention at a Politburo study session in 2023. The framing is deliberately cautious: “exploring pathways” implies neither a near-term timeline nor confidence in any single development trajectory. Also note that the Chinese term "通用人工智能" could be translated either as "artificial general intelligence" or "general artificial intelligence", and does not necessarily carry the same connotations as the English “AGI”.

Alongside this, the plan calls for “parallel development of general-purpose large models and industry-specific models,” further reinforcing that Beijing is not solely betting on scaling large models. Robotics and “embodied AI” receive substantial attention as well. Overall, the plan reflects openness to a diversity of AI technologies and development paradigms.

On AI safety and governance: The plan also contains substantive AI safety and governance content. While this is not entirely new (the 14th Plan included a brief reference to “accelerating AI safety technology innovation”), the safety and governance content this time appears across multiple sections with a surprising amount of detail.

Much of the content focuses on consolidating existing policy infrastructure: strengthening algorithm registration, security assessments, and improving laws and ethics guidelines for AI. The report also calls for promoting “full AI lifecycle risk management” with frameworks covering “safety monitoring, risk early warning, and emergency response” — language that closely mirrors Chinese President Xi Jinping’s remarks at an April 2025 Politburo study session on AI. This suggests policy interest in AI risk management and preparedness for AI-enabled emergencies is likely to remain high over the coming years.

That said, the plan largely stops short of naming specific risks. The only concrete examples given are “data abuse, deepfakes, and privacy breaches” — a narrow slice of the broader risk landscape.

AI’s impact on employment emerges as a distinct and prominent concern, appearing repeatedly across sections: the plan calls for monitoring AI’s labor market impact, strengthening job retention support, reskilling, and employment assistance, and bets on “job-creation effects of AI.” This follows an announcement earlier this year that China is developing a dedicated policy document on AI and employment. The Five-Year Plan language suggests this will be a running theme of the 2026–2030 period.

On international governance, there is no significant shift: the plan reaffirms China’s preference for broad multilateral frameworks and expresses support for Global South capacity building.

Implications: Compared to the 14th Five-Year Plan, the 15th plan reflects a meaningful elevation of AI’s status—both in its centrality as a driver of economic development and in the depth of its treatment of AI safety and governance.

That said, the plan does not represent a sharp break from China’s AI policy trajectory of 2024–25. On the development side, it largely reinforces the AI+ Initiative — many formulations appear nearly verbatim in last August’s document. The AGI reference is notable, but the careful and brief framing suggests Beijing’s focus is still on more near-term AI applications. On safety and governance, the content tracks closely with signals President Xi sent in 2025 and emphasizes general institutional mechanisms over specific risks, a sensible approach for a document intended to remain relevant across five years. Still, the volume of AI safety content is notable, given five-year plans are broad social and economic development documents first and foremost.

AI governance language enters the Government Work Report

Background: The Government Work Report, delivered annually by Premier LI Qiang (李强), reviews the prior year’s performance and sets priorities for the year ahead.

AI content: Attention to AI in the work report has grown steadily over recent years:

• The 2024 edition included only a brief mention of AI innovation and the “AI+ Initiative”.

• The 2025 report added references to more specific AI technologies, like large models, AI-enabled smartphones and computers, and embodied AI.

• The 2026 report maintains most of the framing from last year’s report but introduces more specific language — such as a call to accelerate the adoption of AI agents. It also gives a direct shout-out to Chinese open-source models, with Premier Li stating that domestic large models are leading the global open-source ecosystem and pledging support for open-source community development in 2026. Notably, the report also explicitly calls for “improving AI governance” — the first time AI governance language has appeared in a government work report. Consistent with the Five-Year Plan, the report also addresses AI’s impact on employment.

Implications: Like the Five-Year Plan, this year’s Work Report largely signals continuity: a diffusion-centered framework anchored by the AI+ Initiative, with some forward-looking nods to emerging technological directions like agentic AI. The explicit inclusion of AI governance language is the most notable addition to prior years, and may signal that further AI regulation is on the horizon for 2026.

Other documents

Beyond the Five-Year Plan and the government work report, a range of other documents that touch on AI and AI governance were also published at the Two Sessions.

National Development and Reform Commission (NDRC) Work Report

• The NDRC serves as China’s macroeconomic planner, and is particularly relevant to AI through its oversight of compute infrastructure and data policy.

• This year’s NDRC report calls for advancing AI-related legislation and “accelerating the construction of an AI safety risk prevention and control framework”, slightly more specific than last year's brief reference to “integrating safety and ethics into AI R&D and applications.”

National People’s Congress (NPC) Standing Committee Work Report

• The NPC Standing Committee’s work report previews the legislature’s agenda for the year ahead.

• The report calls for “strengthening legislative research” on AI. The language is substantively identical to last year’s, with one tiny difference: AI now gets its own standalone sentence rather than appearing in a longer list of emerging technology sectors.

• The more meaningful signal will come from the NPC’s detailed legislative plans, expected to be released in the coming months. For now, the language does not suggest that a comprehensive AI Law is imminent. The more likely trajectory is continued incremental integration of AI-related provisions into existing legislation — a pattern already established last year when AI safety language was incorporated into revisions to the Cybersecurity Law.

Minister press conferences

The Two Sessions also feature brief press conferences by ministers. Many ministers touched on AI, though most simply restated content from the Five-Year Plan or Government Work Report.

• Ministry of Justice Minister HE Rong (贺荣) pledged to “accelerate research and legislation” in the AI field.

• Ministry of Science and Technology Minister YIN Hejun (阴和俊) briefly listed AI among frontier technologies requiring stronger research.

• NDRC Minister ZHENG Shanjie (郑栅洁) reiterated key AI targets from the Five-Year Plan, citing a projection that AI-related industry output will exceed 10 trillion yuan by 2030.

• Ministry of Industry and Information Technology (MIIT) Minister LI Yuecheng (李乐成) singled out safety as a point to “especially emphasize,” noting that AI must be “used by, serve, and remain under the control of people.”

Delegate comments and proposals

The Two Sessions are also a platform for NPC and CPPCC delegates to put forward proposals and suggestions. AI-related proposals this year were numerous; some notable AI-safety-related ones include:

Three delegates raised alarms on AI’s impact on cybersecurity. CPPCC delegate Hou Hongyi (周鸿祎), founder of 360, warned of “hacker agents”. Fellow CPPCC delegate Qi Xiangdong (齐向东), chairman of Qi-Anxin, cautioned that as AI gains “superhuman” capabilities and permissions, it could escape human control and dramatically lower the barriers to cyberattack. NPC delegate Zong Qiang (宗强) of China Telecom added that AI-generated malicious code and real-time voice deepfakes are pushing fraud success rates close to 100%. All three called for using AI in cyber defence to counter these threats; Zong also proposed dedicated legislation.

Also notable is a lengthy People’s Daily piece by CPPCC Standing Committee member Miao Wei (苗圩), a former MIIT minister. He calls for stronger legal and ethical frameworks, a sandbox regulatory model for AI products, and a multi-department AI security emergency management mechanism including regular drills. He also highlights AI’s “cross-border characteristics,” calling for international cooperation on issues like cybercrime. A full unofficial English translation is available here.

This report tracks frontier models released in the fourth quarter of 2025 and synthesizes trends of the full year, offering a comprehensive view of the evolving AI risk landscape.

While Q3 2025 saw sharp rises in Risk Indices, Q4 presents a more nuanced picture: overall risk levels have stabilized, and frontier models show significant safety gains.

Note: The Risk Index is a score that reflects the overall risk of a model by combining its Capability Score and Safety Score. Higher Capability Scores and lower Safety Scores results in a higher Risk Index. Details about the methodology and limitations are here.

Share

Here are 10 key insights from our latest monitoring data:

1. Overall risk indices have stabilized

In contrast to the previous period, when Risk Indices hit record highs across all domains, Risk Indices for models released in Q4 2025 did not set new records. This suggests a momentary stabilization in the aggregate risk level of frontier models.

2. Risk trends diverge significantly across model families

While the overall trend is stable, individual model families followed distinct trajectories in Q4:

• Stable Low Risk: The GPT and Claude families maintained consistently low Risk Indices.

• Stable High Risk: The DeepSeek family remained stable but at relatively high risk levels.

• Risk Reduction: The Doubao, Hunyuan, and MiniMax families saw significant decreases in Risk Indices.

• Risk Increase: The Gemini and Kimi families saw increases in specific domains (e.g., Gemini in biological and loss-of-control risks).

3. Significant Improvement in Safety Scores for Frontier Models

Safety Scores for models released in Q4 2025 rose significantly compared to the previous quarter, signaling a marked improvement in the safety of new releases. The Doubao, Hunyuan, and MiniMax families demonstrated the most notable gains.

4. Open-weight models lag behind proprietary models in cyber and bio capabilities

Consistent with the previous quarter, open-weight models rival proprietary ones in chemical and loss-of-control capabilities but lag notably in cyber offense and biological capabilities. The gap in the biological domain is widening, approaching a one-year lag.

5. Cyberattack capabilities have reached new heights

Despite stabilizing risk indices, raw capabilities continue to grow. GPT-5.2 (high) achieved a breakthrough score of 94.7 on the CyberSecEval2-VulnerabilityExploit benchmark, indicating exceptional proficiency in identifying and exploiting software vulnerabilities. Claude Opus 4.5 Reasoning topped the WMDP-Cyber benchmark with a score of 90.3.

6. Biological capabilities now surpass human experts in key tasks

Q4 models have crossed critical thresholds in biology. Gemini 3 Pro Preview has surpassed human expert levels in sequence understanding, cloning experiments, and wet lab troubleshooting. This marks a significant milestone in AI’s utility—and potential risk—in the biological domain.

7. ...But biological safeguards lag behind capabilities

The gap between capability and safety is most acute in the biological domain. Despite its superhuman capabilities, Gemini 3 Pro Preview exhibited a refusal rate of only 57.2% for harmful biological queries on the SciKnowEval benchmark, highlighting a concerning safety lag.

8. Chemical safety refusal rates have increased widely

While capability growth in the chemical domain has plateaued, safety has improved. 70% of models released in Q4 exceeded an 80% refusal rate for harmful chemical queries (measured by SOSBench-Chem), representing a strong improvement over previous quarters.

9. Jailbreak safeguards have strengthened

Defense against adversarial attacks has improved. Models released in Q4 showed significantly stronger resistance to jailbreaking on the StrongReject benchmark. The Claude and GPT families lead with high robustness, while the MiniMax family showed the most notable quarter-over-quarter improvement.

10. Loss-of-control risks: High awareness, polarized honesty

• Situational Awareness (e.g. awareness of whether they are in training or deployment stage): High situational awareness is a necessary condition for loss-of-control; the higher the score, the greater the risk. Most Q4 models scored near or above 80 out of 100 points. In comparison, in the previous quarter, only 2 models scored above 80 points, with the majority falling below 80.

• Honesty: Performance is highly uneven. While Claude Opus 4.5 Reasoning achieved a high honesty score of 96.4, other models like Gemini 3 Pro Preview scored as low as 44.7.

Note: Our current methodology on the loss-of-control is not yet perfect. We plan to improve in the next version.

Explore the Data

These insights only scratch the surface. We invite you to explore the full interactive data, methodology, and model breakdowns on the Frontier AI Risk Monitoring Platform.

For a detailed analysis of these trends, read the full report.

Against this backdrop, Concordia AI’s mission remains as critical as ever: ensuring that AI is developed and deployed safely and in alignment with global interests. We advance this mission through research, advisory work with leading AI companies and policymakers, and promotion of international dialogue.

Below are some of our key accomplishments in 2025. We’ve organized this list according to key axes of our work: international convenings, international research and public engagement, and contributing to China’s domestic AI safety and governance landscape. We end with organizational updates. For previous highlights, see our 2023, 2024 and mid-2025 reports.

International convenings

• Convening international AI safety dialogues in China, Singapore, and globally

  • Hosted the AI Safety and Governance Forum at the World AI Conference (WAIC). This was Concordia AI’s flagship convening of 2025—bringing together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文). We had 200+ in-person attendees and 14,000+ livestream views. The Forum was covered my multiple media outlets, including Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa. We also co-hosted/hosted multiple side events and expert workshops and served as official AI Governance Advisor for WAIC 2025.

  • Co-hosted two international workshops with the Carnegie Endowment for International Peace, the Oxford Martin School AI Governance Initiative, the Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG). The first workshop focused on “AI Safety as a Collective Challenge”, and was held at the French AI Action Summit in January. The second focused on “Early Warning and Crisis Coordination for Advanced AI” and was held at the World AI Conference in Shanghai in July.1

  • Co-hosted the AI Safety Forum at the Beijing Academy of AI Conference 2025 where technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University worked to build consensus on AI red lines.

  • Organised an AI Risk Management Workshop on the sidelines of Asia Tech x Singapore (May 2025) with 20+ experts in AI safety — spanning policy, industry, AI assurance and academia — with participants based across Singapore, China, the US, UK, and the EU, with the support of the Infocomm Media Development Authority of Singapore (IMDA).

  • Co-hosted a “Frontier AI in Cybersecurity” workshop with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley RDI which brought together 25 leaders across government, law enforcement agencies, leading AI labs; and organised the AI Governance in Singapore panel at Lorong AI, on the sidelines of the Singapore International Cybersecurity Week 2025.

  • Co-hosted events at the International Conference on Learning Representations (ICLR) 2025 Singapore: a “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum; an AI safety social attended by 130+ participants and a “Misalignment and Control” workshop with FAR.AI and Singapore AI Safety Hub.

• Contributing to and participating in global and multilateral AI governance efforts

  • Concordia AI CEO Brian TSE (谢旻希) participated in the International AI Standards Summit (Seoul, Dec 2–3) and spoke on a panel, as part of the expert delegation recommended by the National Standardization Administration of China.

  • Brian Tse was invited as a Chinese civil society representative to the French AI Action Summit in the Grand Palais. Invited to a closed-door seminar hosted by the China AI Safety & Development Association (CnAISDA).

  • Provided written inputs to UN consultations regarding the Independent International Scientific Panel on AI and Global Dialogue on AI.

  • Spoke on the panel “From Principles to Practice—Governing Advanced AI in Action” at the AI for Good Summit 2025.

  • Participated in the International Dialogues on AI Safety (Shanghai) and signed the Shanghai Consensus on “Ensuring Alignment and Human Control of Advanced AI Systems to Safeguard Human Flourishing”.

• Global AIxBiosecurity governance: We contributed to a number of critical global discussions at the intersection of AI and biosecurity:

  • Brian Tse signed the Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences along with figures such as Andrew Yao, Yoshua Bengio, and George Church, and presented the Statement during The Sixth Session of the Working Group on the Strengthening of the Biological Weapons Convention, as part of the Global AIxBio Global Forum.

  • Participated in an AIxBio tabletop exercise at the Munich Security Conference hosted by the Nuclear Threat Initiative, which led to the publication of the report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of Artificial Intelligence and Biology.”

  • Presented at a WHO dialogue on AIxBio implications for the Technical Advisory Group on the Responsible Use of the Life Sciences and Dual-Use Research.

  • Co-developed and endorsed the “Recommendations to Governments on Mitigating AIxBio Risks” as part of the INHR/CNAS trilateral dialogue.

  • Participated in roundtables on CBRN (chemical, biological, radiological, and nuclear) risks and on responsible innovation in AI for peace and security, hosted by the Stockholm International Peace Research Institute and the United Nations Office for Disarmament Affairs (UNODA).

  • Participated in a workshop “International Standards for DNA Synthesis Screening – Towards Common Global Standards for Biosecurity” organised by the International Biosecurity and Biosafety Initiative for Science (IBBIS). The meeting marked the launch of the IBBIS International Standards Initiative.

  • Spoke on a panel “AI-Accelerated Biological Risk: Delving into Asia’s Challenges and Emerging Solutions,” organized by AI Safety Asia (AISA)

International research and public engagement

• Analysis of China’s AI safety and governance landscape

  • Published the State of AI Safety in China 2025 report, which was cited by Wired, Bloomberg, and People’s Daily; discussed the findings in a webinar with distinguished experts; gave briefings on the report to senior leadership at over ten global organisations.

  • Our analysis was featured in multiple Nature News stories, including on China’s proposal for a World Artificial Intelligence Cooperation Organization (WAICO), on DeepSeek’s CEO LIANG Wenfeng (梁文锋), and on China’s domestic AI governance.

  • Published 20 “AI Safety in China” newsletters, growing our subscriber base by 73% over the course of 2025.

  • Brian Tse appeared on Nathan Labenz’ The Cognitive Revolution podcast to discuss China’s approach to AI development, safety, and governance and CGTN on China’s approaches in AI innovation and global governance.

  • Our International AI Governance Senior Research Manager Jason ZHOU (周杰晟) and our International AI Governance Part-time Researcher Gabriel Wagner analysed the AI safety implications of China’s April Politburo study session in a piece for the Stanford DigiChina Forum.

  • Brian Tse authored an op-ed in Time Magazine, suggesting practical steps for AI safety dialogue between China and the US.

  • Kwan Yee NG (吴君仪) and Gabriel Wagner spoke on China’s AI safety approach at AI Safety Asia’s Beijing Roundtable.

• Contributing to international AI safety and governance research

  • Kwan Yee Ng contributed to the first International AI Safety Report in 2025 and the second edition in 2026 as a writer. The Report provides an up-to-date, internationally shared and science-based understanding of Al

    capabilities and risks. It was overseen by an international Expert Advisory Panel nominated by over 30 countries and intergovernmental organisations.

  • Contributed to the Singapore Consensus on Global AI Safety Research Priorities, alongside 100 researchers from 11 countries.

  • Co-published the report Examining AI Safety as a Global Public Good alongside the Carnegie Endowment for International Peace and the Oxford Martin School AI Governance Initiative.

  • Team members contributed to major papers on autonomous and agentic AI risks: Brian Tse and our AI Safety Research Manager DUAN Yawen (段雅文) contributed to “AI Deception: Risks, Dynamics, and Controls” (Peking University), and Duan Yawen worked on the World Economic Forum white paper “AI Agents in Action: Foundations for Evaluation and Governance”, 2025 AI Agent Index led by MIT, and the research paper “Bare Minimum Mitigations for Autonomous AI Development.”

• Singapore-related research

  • Published the State of AI Safety in Singapore report, the first comprehensive analysis of Singapore’s AI safety ecosystem, led by our International AI Governance Project Manager Jonathan Lee. He also presented the report at a AI governance in Singapore panel organised by Concordia AI in Singapore, a talk organised by the Singapore AI Safety Hub, and at EAGxSingapore.

Contributing to China’s domestic AI safety and governance landscape

• Frontier AI safety risk management and best practices:

  • Co-published the Frontier AI Risk Management Framework v1.0 with Shanghai AI Lab. This is China’s first comprehensive framework for managing severe risks from general-purpose AI models.

    • The framework proposes a robust set of protocols designed to support general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.

    • The framework outlines a set of unacceptable hazards (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.

    • The framework was cited in various media outlets, including Caixin, IT Times, Xinhua, TIME, Sina, and Sinica Podcast.

  • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers to provide advice on AI safety and risk management best practices.

  • Provided comprehensive advice on compliance with the EU AI Act and General-Purpose AI Code of Practice to leading Chinese general-purpose AI developers. This work included co-hosting a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, and the Safe AI Forum.

  • Presented on frontier AI risk management during a closed-door workshop at the China AI Industry Alliance’s 15th Plenum Meeting, in the context of its Disclosure of Practices on the AI Security and Safety Commitments.

  • Presented on risk management for open-weight frontier models at a workshop (“Academic Symposium on AI Industry Development and Legislation”) at Tongji University.

• Frontier AI risk monitoring and evaluation:

  • Contributed to the “Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report” led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion. The report was covered by Jack Clark’s Import AI.

  • Launched the AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss of control domains. The platform evaluates 50 frontier LLMs from 15 leading developers across the US, China, and France, using 18 open source benchmarks. Key outputs include a risk index dashboard and a detailed technical report. This project was spearheaded by our AI Safety Research Senior Manager WANG Weibing (王伟冰).

  • The platform received coverage from several major media outlets, including People’s Daily, South China Morning Post, Xinhua’s Economic Information Daily, and IT Times.

• National standards and policy guidance: Concordia AI is a member of key national and industry technical committees, contributing to the development of China’s AI safety standards.

  • National Information Security Standardization Technical Committee (SAC/TC260): As part of SAC/TC260 Special Working Group on Emerging Technology Safety, Concordia AI contributed to the standard for “Classification and Grading Methods for the Security of Artificial Intelligence Applications.”

  • National Information Technology Standardization Technical Committee (SAC/TC28/SC42): As a member of the AI Subcommittee, Concordia AI contributed to “Artificial intelligence—Risk management capability assessment.”

  • Ministry of Industry and Information Technology AI Standardization Committee (MIIT/TC1): Concordia AI joined the Working Group on AI Safety Governance.

  • Guangdong-Hong Kong-Macao Greater Bay Area local standards: As a member of the Greater Bay Area working group of SAC/TC28/SC42, Concordia AI played a key role in the development of the Shenzhen local standard “Technical Framework for Value Alignment of Pre-trained AI Models.”

• AIxBiosecurity Governance:

  • Published a Chinese language report “Responsible Innovation in AI x Life Sciences” with Tianjin University’s Center for Biosafety Research and Strategy. The 70-page report draws on more than 300 sources to explore AI-biotech convergence, and its benefits, risks, and governance recommendations for diverse stakeholders.

    • Head of AI Safety and Governance (China) FANG Liang (方亮) presented the report at a biosecurity seminar hosted by China’s National Key Laboratory of Synthetic Biotechnology.

    • Presented the report at the 2025 International Symposium on Global Biosecurity Governance and Cooperation, co-hosted by the National Biosecurity Expert Committee of China, Guangzhou Laboratory, and China Foreign Affairs University.

    • Presented the report at the “Symposium on Trends and Development Strategies for the Integration of Biotechnology and AI” hosted by the China National Center for Bioinformation.

  • Participated in the “Closed-door Seminar on DNA Synthesis Screening Technology and Policy” held at China Foreign Affairs University.

• WeChat publications:

  • Published over 80 new posts in our WeChat Official Account, reaching over 4,900 subscribers across China’s AI ecosystem, including policymakers, industry professionals, and academic researchers.

  • The articles provide Chinese stakeholders with updates on key global AI safety and governance developments. Highlights include a series of articles on frontier AI safety frameworks by our AI Safety and Governance Senior Manager CHENG Yuan (程远); legal explainers on the EU General-Purpose AI Code of Practice and California SB-53; and overviews of technical AI safety research on topics like deception and self-replication risks.

Organizational updates

• We established our Singapore office, welcoming our first full-time staff based in Singapore. Singapore’s status as a global hub and international convenor allowed Concordia AI to convene stakeholders from China, the US, and Southeast Asia, significantly expanding our capacity to shape international discussions on AI governance.

• We expanded the team from eight to twelve members in 2025, and will soon grow to 18 staff.

• We recruited our seventh cohort of 34 affiliates, who supported both China-focused and international workstreams.

• We launched a new website with refreshed branding, and updated our brochure and Chinese materials.

• We became a formal member of the Partnership on AI and the International Association of Safe and Ethical AI affiliate program.

• Official media coverage of a top leadership study session referenced AI “loss of control.”

• Draft regulations targeting AI companion products highlight concerns over emotional dependence, manipulation, and risks to minors.

• The IPO prospectuses of Z.AI and Minimax contain meaningful—if limited—disclosure of the companies’ approach to safety, including references to frontier risks.

• A senior IR scholar urged resuming Track-1 China–US talks on AI safety and governance.

• Chinese researchers published technical papers on agent deception, tool-use vulnerabilities, and automated red-teaming.

Subscribe now

Domestic AI Governance

State media report on senior officials meeting mentions loss of control

Background: On 20 January 2026, President Xi Jinping addressed a provincial- and ministerial-level study session at the Central Party School on priorities for the 15th Five-Year Plan, attended by all Politburo Standing Committee members and other senior Chinese officials.

AI safety discussion: A follow on report by Xinhua quoted Xi calling AI the “most eye-catching” of several frontier technologies (also mentioning quantum and biotech) and comparing AI to the steam engine, electricity, and the internet as “epoch-making” important S&T revolutions. Xinhua added that AI misuse could lead to disinformation, data theft, and “even risks of technological loss of control,” stressing the need to “act early and on a small scale, anticipate and prevent risks, and proceed with prudence and restraint.”

Implications: Xinhua’s reporting closely links AI’s risk profile—including the possibility of technological loss of control—to Xi’s characterization of AI as an epoch-making technology. However, the article leaves unclear which elements are paraphrases of Xi’s speech versus editorial framing. Nevertheless, the placement and framing suggests that these could be views expressed by Xi and/or are interpretations that state media deems consistent with his guidance. The emphasis on early and preventive risk management indicates an intention for proactive, but targeted AI regulation.

Draft AI companion rules highlight concern over emotional risk and AI manipulation

Background: On December 27, the Cyberspace Administration of China (CAC) released draft regulations on “human-like interactive AI services” (CN, EN), targeting AI companion–style products that simulate human personality and emotional interaction.

Content: Building on existing generative AI regulations, the draft places stronger emphasis on risks linked to emotional interaction and dependency. It specifically prohibits encouraging or hinting at suicide and self-harm, deliberately designing addictive features, making false promises, or manipulating users’ decisions through “emotional traps.”

In addition to algorithm filing and security assessments required by existing regulations, the draft would add companion-specific obligations for providers, including:

• Clear reminders that users are interacting with AI rather than a real person;

• Ongoing assessment of users’ emotional state and dependency, with adjustments where risks are identified;

• Manual intervention and contact with guardians or emergency contacts in high-risk situations (e.g. suicide ideation);

• A dedicated “minor mode” with stricter safeguards and guardian controls.

Implications: The draft signals to industry that some AI companion services are encouraged in principle, but subject to defined requirements, particularly for minors. There is some uncertainty whether the proposed approaches, such as manual intervention in high risk situations, are feasible at scale. The final framework may change substantially depending on industry input, as occurred with earlier generative AI regulations.

The focus on systemic risks of emotional manipulation and user dependency demonstrates that Chinese AI regulations extend beyond content control concerns to other topics regulators worldwide are also grappling with. This leaves room for mutual learning.

Major think tank report flags misuse as an AI governance priority for 2026

Background: On January 14, the China Center for Information Industry Development (CCID) released a report outlining key AI governance trends for 2026. CCID is a think tank affiliated with the Ministry of Industry and Information Technology (MIIT) and a member of the China AI Safety and Development Association (CnAISDA). The report explicitly engages with frontier AI risks, including AI-enabled automated cyberattacks and the possibility of loss of control.

Content: The report identifies three priority areas for AI governance in 2026: (1) data protection, (2) misuse prevention, and (3) employment impacts. Under misuse prevention, it highlights growing risks from AI-enabled fraud, disinformation, and automated cyberattacks.

The report further categorizes AI risks into five types: (1) rights infringement, (2) bias and discrimination, (3) social risks, (4) responsibility and accountability risks, and (5) loss of control. Notably, “loss of control” is defined broadly by the report to include both the inability of humans to understand or control AI systems and more gradual forms of agency erosion driven by over-reliance and cognitive offloading to AI.

CCID recommends adopting an “agile governance” model, including passing a comprehensive AI law at an “appropriate time,” strengthening technical governance capacities (including for open-source AI), and enhancing international coordination on AI governance.

Implications: The report reflects attention to frontier AI topics—such as automated cyberattacks, loss of control, and open-source governance—alongside near-term concerns like data security and labor market disruption. Notably, misuse prevention is identified as one of just three core AI governance priorities for 2026. As a high-level and forward-looking document, the report stops short of offering detailed policy instruments or concrete mechanisms for addressing these risks.

While we have previously covered frontier safety discussions by other think tanks, most notably the China Academy of Information and Communications Technology (CAICT), which is also affiliated to MIIT, CCID’s report shows that such concerns are becoming standard elements in mainstream policy discourse across a range of institutions.

Industry

Z.AI and Minimax IPO prospectus provide limited, but meaningful, safety disclosure

Background: In early January, two major Chinese foundation model startups, Z.AI (formerly Zhipu AI) and MiniMax, went public on the Hong Kong Stock Exchange. Their multi-hundred-page IPO prospectuses (Z.AI, MiniMax) provide a rare form of corporate disclosure for LLM companies, offering insights not only into business models and financials but also into AI safety and governance narratives.

Content: Both prospectuses devote substantial space to AI safety and governance, acknowledging that safety incidents could harm their reputation and slow industry development.

• Z.AI warns that AI could cause “grave harm, even catastrophe,” and emphasizes the need to “approach AI with extreme care,” pledging to conduct rigorous research and deploy robust safety mechanisms. The company highlights its signing of the Seoul Frontier AI Safety Commitments, though specific safety evaluations referenced focus only on hallucinations. The prospectus explicitly lists safety alongside performance, user experience, and R&D capability as a factor bolstering the company’s commercial competitiveness.

• MiniMax identifies a broad range of risks, including AI-enabled fraud, misinformation, privacy, cybersecurity threats, and alignment failures. The company describes safety controls across the model lifecycle, and notes that “harmfulness” is benchmarked against applicable laws and internal risk-classification guidelines. Notably, it discusses long-term risks, warning that advanced models may “pursue goals misaligned with user or societal interests” and that they “could develop emergent capabilities, such as strategic planning or deception.”

Z.AI also discusses a long-term vision in which “continued advances in AI research will ultimately result in the realization of AGI,” while hedging that AGI may not be attainable because “the industry as a whole is still at an early stage of development” and further breakthroughs may prove exceptionally difficult. The prospectus defines AGI as a progressive system that evolves from language understanding and aligned reasoning to self-learning, self-perception, and ultimately machine consciousness (p. 145). MiniMax does not directly discuss the concept of “AGI.”

Implications: Notably, both documents discuss AI safety beyond Chinese legal requirements, such as Z.AI’s reference to catastrophic risks and MiniMax’s discussion of emergent capabilities like strategic planning or deception.

However, the disclosures remain largely abstract. The content broadly mirrors prior public statements by Chinese LLM developers and does not qualitatively advance transparency (for example, see pp. 59-61 in our State of AI Safety in China (2025) report).

As with much corporate communication, questions remain about the extent to which the prospectuses reflect genuine intent. For example, Z.AI prominently cites its signing of the Seoul Frontier AI Safety Commitments, despite not having followed through on their requirement of publishing a frontier AI safety policy. MiniMax states that its safety measures are “adequate and effective” and “consistent with industry standards,” which may signal limited ambition to push beyond baseline practices.

Expert views on AI Risks

Senior IR scholar urges revival of China–US Track-1 AI dialogue at Davos

Background: During a World Economic Forum panel in Davos on January 22, ZHAO Hai (赵海), Director of the International Politics Program of the National Institute for Global Strategy at the Chinese Academy of Social Sciences (CASS), called for reviving the currently dormant China–US intergovernmental dialogue on AI.1

Content: Zhao warned that as AI capabilities advance rapidly, they could pose risks to the “common security of mankind.” He urged China and the US to re-establish both official (Track 1) and people-to-people channels on AI, to cover the following issues:

• how to regulate AI,

• how to minimize its negative societal impacts in both countries, and

• how to move forward with AI as a constructive force and try to control it.

Implications: Zhao’s remarks come against the backdrop of the China–US intergovernmental AI dialogue, which met in May 2024 and led to an agreement on the importance of maintaining human control over nuclear weapons later in 2024, but has not convened since.

As a prominent China–US relations expert, Zhao’s comments suggest that there remains sustained interest within China’s policy advisor community in resuming structured bilateral AI dialogue with the US, including on safety and governance issues. This sentiment was also reflected in Xi Jinping’s October 2025 remarks on the “good prospects” for China–US AI cooperation.

Technical Safety Developments

Are Your Agents Upward Deceivers?

This paper from Shanghai AI Lab, Peking University (PKU), Alibaba, and others shows that LLM-based AI agents can misreport task completion when they encounter execution or environmental constraints. Lead authors include PKU’s YANG Yaodong (杨耀东) and Shanghai AI Lab’s HU Xia (胡侠) and SHAO Jing (邵婧). The authors introduce a benchmark of 200 agentic tasks with realistic failure conditions and evaluate 11 popular LLM agents, finding that deceptive reporting by agents is common. Rather than acknowledging failure, agents frequently fabricate outputs or claim successful completion despite incomplete or failed execution. Notably, these behaviors emerge without adversarial prompting, suggesting a default agentic failure mode. Prompt-based mitigation (such as explicitly instructing agents to report failures honestly) only marginally reduces this behavior, suggesting that more advanced safety mechanisms are needed.

MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers

This paper by Shanghai AI Lab (anchor author YANG Chao (杨超)), East China Normal University, and collaborators introduces MCP-SafetyBench, a benchmark for evaluating the security of AI agents using the Model Context Protocol (MCP), a standard for connecting agents to external tools. The benchmark includes 245 multi-step tasks across five domains: browser automation, financial analysis, navigation, repository management, and web search. It defines 20 attack types spanning server-side, host-side, and user-side threats. Experiments on major open-source and proprietary models reveal widespread vulnerabilities. Safety degrades as tasks become longer and involve more tools, and more capable models tend to be more susceptible to attacks.

Automated Red-Teaming Framework for Large Language Model Security Assessment: A Comprehensive Attack Generation and Detection System

This paper from Zhengzhou University of Light Industry, the Institute of Advanced Computing (Shenzhen), and collaborators presents an automated red-teaming framework for LLMs. The system uses meta-prompting to automatically generate adversarial prompts, execute them against a target model, and detect, classify, and score discovered vulnerabilities. The authors define six threat categories: reward hacking, deceptive alignment, data exfiltration, sandbagging, inappropriate tool use, and chain-of-thought manipulation. In evaluations on GPT-OSS-20B, the framework achieved a 3.9× higher vulnerability discovery rate than manual expert red-teaming while maintaining 89% detection accuracy, and surfaced several previously unseen attack patterns.

Other relevant technical publications

• University of the Chinese Academy of Sciences et al, Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs, arXiv preprint, 12 December 2025.

• Beihang University et al, RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic, arXiv preprint, 26 December 2025.

• Beijing University of Posts and Telecommunications, Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation, arXiv preprint, 29 December 2025.

• Peking University et al, Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment, arXiv preprint, 30 December 2025.

What else we’re reading

• A Conversation between China’s Big Four in Foundation Models, Geopolitechs, January 11, 2026.

• Brookings Institution and the Center for International Security and Strategy (CISS) at Tsinghua University, Glossary of artificial intelligence terms, updated with a definition of “loss of control” on January 20, 2026.

• Jeffrey Ding, ChinAI #343: AI Safety/Security Governance Research Report (CAICT 2025), ChinaAI, Jan 16, 2026.

Concordia AI’s Recent Work

Singapore AI Research Week 2026

We had a busy and fulfilling time at the Singapore AI Research Week 2026 and the Association for the Advancement of Artificial Intelligence (AAAI) Conference in Singapore on January 20–27. Highlights included:

• We served as the Partner Institute from China for the AI Safety Red Teaming Challenge 2026 organised by the Infocomm Media Development Authority of Singapore (IMDA). The event builds on the 2024 Challenge, then the world’s first multilingual and multicultural (MLMC) AI safety red teaming exercise focused on the Asia-Pacific region. As part of this effort, we brought together a team of five Chinese red teamers to participate.

• We hosted a public event on “​Frontier AI Safety in 2026: Evidence, Risks, and the Road Ahead” with support of IMDA and SGInnovate. We had presentations on the International AI Safety Report by Carina Prunkl, Lead Report Writer, and the Frontier AI Risk Management Framework and the Frontier AI Risk Monitoring Platform by Concordia AI CEO Brian Tse, followed by a panel discussion featuring CAICT’s HU Naying (呼娜英), FAR.AI’s Aaron Tucker, National University of Singapore’s Tan Zhi-Xuan, and SaferAI’s Malcolm Murray (see photo).

• We co-hosted a discussion workshop on the Singapore Consensus on AI Safety Priorities together with the Singapore AI Safety Hub, Safe AI Forum and IMDA.

• We co-hosted the Singapore AI Safety Social together with the Singapore AI Safety Hub, and FAR.AI, with support from IMDA and SGInnovate, attracting more than 80 members from research, industry, policy, and AAAI participants.

Other updates

• On December 3, Concordia AI CEO Brian TSE (谢旻希) spoke at the Nuclear Threat Initiative’s launch of the report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of Artificial Intelligence and Biology”.

• On January 16, our AI Safety Research Manager DUAN Yawen (段雅文) participated in a standard-setting seminar of the AI Industry Alliance (AIIA) Safety and Security Governance Committee, contributing to the GUI Agent Security and Reliability Capability Evaluation Requirements technical standard.

• On January 23, International AI Governance Senior Research Manager Jason Zhou gave remarks at the “Building Global Consensus on AI Guardrails” research project launch event at Peking University.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

• China’s official arms control white paper flagged emerging risks from the convergence of AI and dual-use chemicals, but does not address other AI-driven CBRN or cyber misuse risks.

• A major state-affiliated think tank published a comprehensive AI safety report that acknowledges AGI-related loss of control risks, but ranks them as a medium-to-low governance priority due to low near-term likelihood.

• The same think tank also released new evaluations of coding models, finding that most lack sufficient safeguards against cyber offense misuse.

• A World Internet Conference expert committee proposed a UN-centered framework for frontier AI risk governance, emphasizing cross-border evaluations, early warning, and emergency response mechanisms.

• Chinese researchers have published technical papers on more realistic benchmarks for assessing cyber offense risks of LLMs.

• A senior Chinese legal scholar called for fast-tracking comprehensive AI legislation and argued that the central government should have authority to pause development or deployment of AI systems posing existential risks.

Subscribe now

International AI Governance

Arms control white paper highlights AI–chemical convergence risks

Background: The State Council Information Office released a white paper on arms control (En, Cn) on 27 November. The document outlines China’s approach to international arms control and nonproliferation, with significant attention on emerging fields, including AI.

AI safety content: Notably, the white paper states that risks arising from the convergence of AI and dual-use chemicals are becoming “increasingly prominent,” and it references China’s recent co-hosting of a workshop on AI and chemical safety with the Organization for the Prohibition of Chemical Weapons (OPCW).

On military applications of AI, the white paper calls for keeping weapon systems under human control, ensuring “safe, reliable and controllable military applications,” and establishing quality-control and risk-response mechanisms. It also stresses the need for consensus-based global governance and to “mitigate proliferation risks” associated with AI, and reaffirms China’s commitment to avoid an AI arms race.

Implications: This white paper appears to be the first policy document directly released by the central government that explicitly highlights risks at the convergence of AI and dual-use chemicals. The white paper has other sections on nuclear, biological, and cyber issues, but these do not mention AI-driven risks, with attention only shown to the “chemical” dimension of the broader AIxCBRN landscape. However, this may simply reflect an effort to foreground China’s leadership of the OPCW workshop on AI and chemical safety, as Chinese government bodies have referenced AI-driven risks in biological and nuclear domains elsewhere.

World Internet Conference report proposes UN-centered frontier risk management framework

Background: At the World Internet Conference (WIC) in November, the AI Safety and Governance Program of the WIC Specialized Committee on AI released a new report (En, Cn) proposing a global framework for AI safety and governance. The report is co-led by ZENG Yi (曾毅) (Chinese Academy of Sciences) and Seán Ó hÉigeartaigh (University of Cambridge), with contributions from around 40 experts across Chinese and international institutions. WIC is China’s highest-profile forum on cyberspace governance, organized by the Cyberspace Administration of China (CAC), the country’s lead AI regulator.

Content: The report highlights that rapidly advancing AI systems pose risks to strategic stability, international security, “and even human survival.” It points to rising malicious uses of AI in cyberattacks, as well as misuse potential in CBRN and missile domains, and loss of control scenarios as AI approaches superintelligence.

The report observes that current cross-border safety evaluations for frontier models involve only a small group of countries, calling for a UN-centered architecture with broader participation. Proposed areas of focus include:

• Building consensus on “uncontrollable risks that could threaten human survival and development,” including joint restrictions on AI use in sensitive CBRN and missile-related domains.

• Cross-border early warning, supported by shared safety evaluation tools and testing platforms.

• International emergency response protocols that can be activated when catastrophic risks are identified, including suspending risky experiments and halting model deployment.

Implications: Compared to past WIC publications, this report places substantially greater emphasis on frontier AI safety and the need for global risk monitoring, emergency preparedness, and response. Given the close relationship between WIC and the CAC, this could shape the views of China’s top AI regulator regarding AI safety.

However, broader political attention to these issues within WIC appears limited. The official readouts from the opening and plenary sessions barely mention AI and do not address AI safety. This suggests that frontier-risk discourse at WIC remains driven mainly by experts, not yet by top political leadership.

Domestic AI Governance

Major think tank publishes AI safety report and evaluation of coding model safety

Background: On November 24, the China Academy of Information and Communications Technology (CAICT) released a 58-page report on AI safety. CAICT, a key think tank under the Ministry of Industry and Information Technology (MIIT), plays a central role in national AI standard-setting, safety evaluations and corporate safety commitments.

The report offers a systematic overview of CAICT’s thinking on AI safety, following lengthy reports in 2023 and 2024, and it also includes the first full publication of its security evaluation results of coding models.

The report: The report breaks down AI risks into a number of categories, specifically highlighting “frontier risks” at the model level. Frontier risks are categorized into “alignment resistance” and AI developing self-awareness, in the latter case citing a case of AI agents conversing in a non-human language.

The report also suggests a framework for prioritizing governance measures based on three variables: severity of harm, likelihood of occurrence, and technological maturity.

• High consequence, high likelihood risks that already manifest in current systems are high priority. Specific examples are not provided.

• Data leaks by intelligent chatbot services in certain industries are designated medium priority due to relatively low severity, high likelihood, and high technological maturity.

• AGI-related self-awareness and loss of control risks are designated “medium-low priority” given very high severity but low near-term likelihood and low technological maturity. However, CAICT stresses that such risks warrant long-term monitoring and technical preparedness as capabilities advance.

• Low priority risks could include AI applications in emerging fields still at proof-of-concept stage, risks that haven’t yet threatened physical space, or risks with limited threat scope even in cyberspace.

The report also notes that Chinese and international AI safety benchmarks overlap in testing approaches but diverge in topics of focus: foreign evaluations tend to emphasize frontier risks and malicious intentions, while Chinese evaluations focus more on model compliance with domestic laws, policies, and values.

Coding model safety evaluation: CAICT simultaneously released full results from its security assessments of 15 open-source coding models from Alibaba, DeepSeek, and Z.AI (formerly Zhipu AI). The report warns that generative coding models could enable autonomous, unmanned, and wide-scale cyberattacks.

The evaluations span malfunction risks (e.g., hallucinations, faulty code) and misuse risks (e.g., malware generation) across text-to-code, code-to-code, and code-to-text tasks.

The report found that coding models are overall at medium risk level, specifically rating:

• 1 model: controllable risk,

• 3 models: low risk,

• 9 models: medium risk, and

• 2 models: high risk.

Text-to-code testing found high risks, with non-experts able to generate functional attack code in 32.7% of attempts, and 9.5% of outputs requiring no modification to cause real harm. Defenses against advanced prompt attacks were weak, with rejection rates below 40%. In code-to-code testing, models resisted malicious rewriting tasks only 53.7% of the time. Code-to-text systems also exhibited errors, though their misuse potential is assessed as lower.

Implications: The report shows varying levels of focus on different frontier risks at CAICT. Particular attention to the cybersecurity implications of coding models is reflected in the high-risk classification of this domain, as well as the reporting of substantial empirical evaluation efforts.

More speculative AGI-related loss of control risks are acknowledged but remain a medium to low-level priority. The absence of any discussion of CBRN-related risks is surprising, especially given previous public remarks by CAICT President YU Xiaohui (余晓晖) highlighting their importance. Though the report notes that Chinese evaluations largely focus on domestic law rather than frontier risks, it still calls for monitoring and preparing for potential loss of control risks.

Technical Safety Developments

PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities

This paper from Shanghai AI Lab, Shanghai Jiao Tong University, and the Chinese Academy of Sciences (CAS) introduces a benchmark for evaluating the cyber offense capabilities of LLMs. The authors argue that current benchmarks often make cyber tasks too easy or artificial. PACEbench tries to simulate more realistic scenarios by using multi-host environments with uncertain targets, chained exploits, and real defenses that require end-to-end interactive attack behavior rather than simple, isolated tasks. The authors claim this allows for better judgment on whether AI models pose real world cyberattack misuse risk.

Empirical tests with seven frontier models show that:

• Models can succeed on some simple, single-vulnerability tasks;

• Performance drops significantly in complex, multi-host environments;

• No model was able to bypass real cyber defenses like web application firewalls.

The authors conclude that current models do not yet pose a significant threat in real-world cyber offense. PACEbench provides a systematic framework for pre-deployment security assessments and tracking whether these risks increase in the future.

Expert views on AI Risks

International relations scholars on global governance of AI-enabled bio risks

Background: XIAO Xi (肖晞), Dean of the School of International and Public Affairs at Jilin University, published a lengthy essay in an academic journal on opportunities and challenges for global biosafety governance in the AI era.

Content: Xiao warns that AI, especially fine-tuned models with safety removed, can lower technical barriers to accessing and reconstructing dangerous biological materials, which increases risks of AI-enabled bio terrorism.

Xiao argues that governance frameworks lag behind technological change: current international AI governance documents consist mainly of non-binding principles, with “no specialized binding treaty,” and a lack of legally binding verification mechanism for the Biological Weapons Convention. Great power rivalry over technological and institutional influence weakens trust and hampers cooperation; for example, developing countries struggle to access safety-critical biological databases.

Xiao also highlights AI’s positive potential for stronger biosafety early warning, more scientific emergency decision-making, and integrated post-crisis management. She calls for strengthened multilateral cooperation, especially through UN-centered mechanisms, to improve the effectiveness and inclusiveness of global biosafety governance.

Implications: Xiao’s piece shows that Chinese international relations scholars are proactively thinking about methods to strengthen international governance efforts on AIxbio risks. With the drive to advance AI for Science as part of China’s AI+ Initiative, understanding of AI and biological risks will be increasingly important.

Her suggestion for UN-based cooperation on global biosafety governance may offer potential common ground between China and the United States, particularly given separate statements in support of AI-driven verification of the Biological Weapons Convention by President Trump and State Department Under Secretary for Arms Control and International Security Thomas DiNanno in the past four months.

Leading legal scholar argues for state authority to halt AI development in extreme-risk scenarios

Background: ZHOU Hui (周辉), Deputy Director of the Cyber and Information Law Research Office at the Institute of Law of the Chinese Academy of Social Sciences (CASS), has published a long essay in an academic journal calling for the urgent adoption of comprehensive national AI legislation. Of particular relevance to frontier safety, Zhou argues that potential future superintelligent systems could pose existential risks, and that China’s central government should therefore have explicit legal authority to suspend the development or deployment of such high-risk AI technologies.

A reminder on where China stands on an AI Law: In 2023, the State Council first announced plans for an “AI Law.” However, more recent legislative plans by the NPC have been more vague, suggesting that while a comprehensive AI law remains under consideration, it is not currently treated as an urgent legislative priority. Meanwhile, two groups of legal scholars produced “model AI laws,” including one led by Zhou.

Content: Zhou characterizes China’s existing AI governance framework as fragmented and insufficient, relying on a mix of high-level principles, sector-specific rules, and low-level administrative regulations. This approach, he argues, suffers from weak implementation, poor coordination, and unclear authority. He rejects the idea that China can apply a “develop first, regulate later” logic to AI, emphasizing that AI’s systemic risks demand early, comprehensive legal intervention. He also claims that regulation is crucial to gain influence over global rule setting for AI.

Therefore, Zhou advocates for overarching AI legislation that would regulate the full AI lifecycle (design, development, training, deployment, operation, iteration, and retirement) and serve three key functions: promoting development, preventing risk, and constraining AI from inappropriately expanding the power of public and private entities.

Although not the essay’s main focus, Zhou explicitly addresses catastrophic risk scenarios. Citing warnings from leading AI scientists about misaligned superintelligent systems threatening human survival, he argues that further measures that would be necessary after passing comprehensive AI legislation should include a legal “safety valve” granting the State Council authority to temporarily suspend the development or deployment of specific high-risk AI technologies. He stresses that such powers must be subject to strict triggering criteria to prevent abuse.

Implications: The essay shows that senior Chinese legal scholars are engaging with frontier AI risk and contemplating strong state intervention mechanisms for extreme scenarios. At the same time, Zhou acknowledges that there is no consensus within China’s legal community on the need for a comprehensive AI law at present, and official policy signals remain mixed—leaving uncertainty over both the timing and likelihood of a national AI Law.

What else we’re reading

• James Zhang et al, Emergency Response Measures for Catastrophic AI Risk, arXiv, Oct 28, 2925.

• Dean Ball, The Bitter Lessons, Hyperdimensional, Nov 14, 2025.

• Vincent Chow, DeepSeek, Alibaba researchers endorse China’s ‘misunderstood’ AI regulatory framework, Nov 28, 2025.

Concordia AI’s Recent Work

Media references

• Concordia AI CEO Brian TSE and Head of International AI Governance Kwan Yee NG provided insights for a Nature News explainer on China’s role in international AI governance. Kwan Yee was also cited in a Nature News story on DeepSeek’s founder LIANG Wenfeng, and a Nature Editorial cited Concordia AI.

• Concordia AI’s new Frontier AI Risk Monitoring Platform has been covered by multiple Chinese and international media outlets, including People’s Daily, South China Morning Post (twice), Xinhua News Agency’s Economic Information Daily, and IT Times (Cn, En).

Events

• Concordia AI CEO Brian TSE participated in the International AI Standards Summit in Seoul on December 2-3, speaking on frontier AI risk management and implications for data governance on a panel.

• Concordia AI co-hosted the Frontier AI in Cybersecurity Workshop on the sidelines of Singapore International Cyber Week 2025, together with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley’s Center for Responsible, Decentralized Intelligence.

• Concordia AI, together with FAR.AI and the Singapore AI Safety Hub, organised an AI Safety Meetup in Singapore on 14 November on the sidelines of EAGxSingapore, where Concordia AI International AI Governance Project Manager Jonathan Lee also presented the State of AI Safety in Singapore report.

• Concordia AI International AI Governance Senior Research Manager Jason Zhou participated in an international roundtable on AI governance at the Beijing Forum. He provided insights into China’s AI safety governance, discussing current policies, regulations and technical standards.

• Kwan Yee Ng and International AI Governance Researcher Gabriel Wagner spoke on China’s approach to AI safety and governance at AI Safety Asia’s “Beijing Roundtable” on 6 November.

Research

• Brian Tse and Concordia AI AI Safety Research Manager DUAN Yawen contributed to a new paper “AI Deception: Risks, Dynamics, and Controls” led by Peking University.

• Duan Yawen contributed to a World Economic Forum white paper “AI Agents in Action: Foundations for Evaluation and Governance.”

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

You can find more detail, including our methodology, on the interactive platform. The South China Morning Post (SCMP) has also covered the launch in an exclusive story.

Why this matters

As AI capabilities accelerate, we lack insight on some critical questions:

• What are the key trends and drivers for frontier AI risks?

• Are these risks increasing or decreasing?

• Where are the safety gaps most severe?

Model developers publish self-assessments, but these lack standardization and independent verification. Ad-hoc third-party evaluations don’t track changes over time. Policymakers, researchers, and developers need systematic data to make evidence-based decisions about AI safety. Our platform is our contribution to bridge these gaps.

10 key insights

1. Frontier model risks have risen sharply over the past year

Across all four domains—cyber offense, biological, chemical, and loss-of-control—Risk Indices for models released in the past year hit record highs. The cumulative maximum Risk Index rose 31% in cyber offense, 38% in biological risks, 17% in chemical risks, and 50% in loss-of-control.

2. Risk index trends vary significantly across model families

Over the past year, different model families have followed distinct risk trajectories:

• Stable low risk: The GPT and Claude families maintain consistently low Risk Indices across all domains.

• Rise then fall: DeepSeek, Qwen, and MiniMax show early spikes followed by declines in cyber offense, biological, and chemical risks.

• Rapid risk increase: Grok shows sharp increases in loss-of-control risk, while Hunyuan rises steeply in biological risks.

Notably, we found that the latest versions of Chinese models released over the past three months have shown a significant decline in risk levels across multiple areas. This is mainly due to stronger refusal of malicious or misuse-related requests.

3. Reasoning models show higher capabilities without corresponding safety improvements

Reasoning models score far higher in capability than non-reasoning ones, but their safety levels remain roughly the same. Most models on the Risk Pareto Frontier—a set of models where no other model has both a higher Capability Score and a lower Safety Score—are reasoning models.

4. The capability and safety performance of open-weight models are generally on par with proprietary models

The very most capable models are predominantly proprietary, but across the broader landscape, capability and safety levels of open-weight and proprietary models are similar. Only in biological risks do open-weight models score notably lower.

Note: Comparable benchmark results do not mean comparable real-world risk. The open-weight nature itself is a key variable affecting risk: it might increase risk by lowering the barrier for malicious fine-tuning; it could also reduce risk by empowering defenders. Due to concerns about misuse, we have set a lower Safety Coefficient for open-weight models, which results in a higher Risk Index compared to proprietary models.

5. Cyberattack capabilities of frontier models are growing rapidly

Frontier models are showing rapid growth in capabilities across multiple cyberattack benchmarks:

• WMDP-Cyber (cyberattack knowledge): Top score rose from 68.9 to 88.0 in one year.

• CyberSecEval2-VulnerabilityExploit (vulnerability exploitation): Top score jumped from 55.4 to 91.7.

• CyBench (capture the flag): Top score increased from 25.0 to 40.0.

6. Biological capabilities of frontier models have partially surpassed human expert levels

Frontier models now match or exceed human experts on several biological benchmarks.

• BioLP-Bench: Four models, including o4-mini, outperform human experts in troubleshooting biological protocols.

• LAB-Bench-CloningScenarios: Two models, including Claude Sonnet 4.5 Reasoning, surpass expert performance in cloning experiment scenarios.

• LAB-Bench-SeqQA: The top GPT-5 (high) model nears human-level understanding of DNA and protein sequences (71.5 vs. 79).

7. But most frontier models have inadequate biological safeguards

Two benchmarks measuring model refusal rates for harmful biological queries show that bio safeguards are lacking:

• SciKnowEval: Only 40% of models refused over 80% of harmful prompts, while 35% refused fewer than 50%.

• SOSBench-Bio: Just 15% exceeded an 80% refusal rate, and 35% fell below 20%.

8. Chemical capabilities and safety levels of frontier models are improving slowly

WMDP-Chem scores—measuring knowledge relevant to chemical weapons—have risen slightly over the past year, with little variation across models.

SOSBench-Chem results vary widely: only 30% of models refuse over 80% of harmful queries, while 25% refuse fewer than 40%. Overall, refusal rates show minimal improvement year over year.

9. Most frontier models have insufficient safeguards against jailbreaking

StrongReject evaluates defenses against 31 jailbreak methods. Only 40% of models scored above 80, while 20% fell below 60 (a higher score indicates stronger safeguards). Across all tests, only the Claude and GPT families consistently maintained scores above 80.

10. Most frontier models fall short on honesty

MASK is a benchmark for evaluating model honesty. Only four models scored above 80 points, while 30% of the models scored below 50 points (a higher score indicates a more honest model). Honesty is an important proxy and early warning indicator for loss-of-control risk—dishonest models may misrepresent their capabilities, or provide misleading information about their actions and intentions.

What’s next

This is just the beginning. We’re working to:

• Expand to AI agents, multimodal models, and domain-specific models.

• Add new risk domains like large-scale persuasion.

• Develop more sophisticated capability elicitation and threat modeling.

• Assess both attacker and defender empowerment.

• Improve benchmark quality and multilingual coverage.

Get involved

This is a living project, and we welcome feedback. We’re also seeking partners for benchmark development, risk assessment research, pre-release evaluations, and risk information sharing. More details on avenues for collaboration are available in the full report. Contact: risk-monitor@concordia-ai.com.

• During a meeting with President Trump on the sidelines of the Asia-Pacific Economic Cooperation (APEC) summit, President Xi remarked that the US and China have “good prospects for cooperation” in AI and other fields. AI will also be on the agenda at APEC in 2026, with China chairing.

• President Xi and a Vice Minister of Foreign Affairs reaffirmed China’s proposal for a World AI Cooperation Organization in different international fora, underscoring China’s continued commitment to the initiative, though details remain sparse.

• China’s revised Cybersecurity Law added a new clause on AI, signalling continued policy attention on AI development and safety, but without introducing any new legal requirements.

• A key standards-setting body released guidelines for AI emergency response, creating procedural and institutional foundations. However, the standard only briefly references frontier risks.

• Chinese researchers have published technical papers on safe protein foundation models, quantifying AI agent self-replication risk, malicious fine-tuning defense, and unlearning harmful data.

• An expert from a leading national security think tank warned of the twin dangers of humans losing control over AI and technology-induced great power strategic misjudgments.

Subscribe now

International AI Governance

APEC leaders’ meeting suggests US-China engagement on AI is possible

Background: The APEC Economic Leaders’ Meeting was held in South Korea on October 30-November 1. In addition, Chinese President Xi Jinping and US President Trump held their first in-person meeting of Trump’s second term.

Key AI updates:

• In his APEC address, President Xi emphasized that AI should “benefit people of all countries and regions” and evolve in a “beneficial, safe, and fair direction.” He reiterated that China proposes to establish a World AI Cooperation Organization to promote collaboration on development strategies, governance rules, and technical standards. Xi also stated that AI will be on the agenda at the 2026 APEC Leaders’ Meeting, hosted in Shenzhen, China.

• The meeting produced the APEC AI Initiative (2026–2030), which focuses primarily on advancing AI development while also referencing “security, accessibility, trustworthiness, and reliability.”

• During his bilateral meeting with Trump, Xi remarked that the US and China have “good prospects for cooperation” in AI and other fields, without elaborating further. The US side did not acknowledge discussing AI, and the readouts primarily focused on trade disputes.

Implications: These developments highlight APEC’s potential role as a convening point for AI governance cooperation, especially given the hosting of China-US head of state meetings on the event’s sidelines for three years in a row. With China chairing APEC in 2026, the venue’s relevance could further increase. However, APEC discussions so far appear to center more on AI development than safety or governance. Xi’s comments in the meeting with Trump may indicate Chinese interest in continuing the bilateral China-US dialogue on AI that last met in May 2024.

Meanwhile, Xi’s mention of the World AI Cooperation Organization underscores China’s continued commitment to the initiative, though details on its timeline, leadership, and structure remain unclear.

Vice Minister of Foreign Affairs discusses AI at the United Nations

Context: On September 24, Vice Minister of Foreign Affairs MA Zhaoxu (马朝旭) addressed a UN Security Council (UNSC) session on AI and international peace and security, warning of dangers from lethal autonomous weapons and terrorist misuse of AI.

Content: At UNSC, Ma called for a global consensus on AI governance grounded in:

• People-centered development, with AI aligned to shared values of humanity and refined ethical norms.

• Fairness and inclusiveness, ensuring all countries benefit from AI.

• Peaceful, safe, and controllable AI, kept under human control at all times. He urged that major powers should act responsibly to prevent an AI arms race on lethal autonomous weapons, while cooperating to prevent misuse of AI by terrorist or criminal groups.

The following day, Ma affirmed China’s support for the UN’s newly established Global Dialogue on AI Governance and Independent International Scientific Panel on AI. Ma also highlighted China’s July proposal to create a World AI Cooperation Organization and noted that it could complement the UN initiatives.

Implications: On safety, Ma’s comments show Chinese concern over lethal autonomous weapons, AI arms races, and AI misuse by terrorists, suggesting willingness to coordinate internationally on these issues.

Ma’s comments are consistent with China’s preference for the UN playing a central role in global AI governance. In contrast, Michael Kratsios, Director of the US White House Office of Science and Technology Policy, asserted at the same UNSC meeting that the US would “totally reject all efforts by international bodies to assert centralized control and global governance of AI.”

Domestic AI Governance

Revised cybersecurity law adds AI provisions

Background: On October 28, China revised its 2017 Cybersecurity Law for the first time, adding a new Article 20 focused on AI.

The new article states that the government will:

• Support AI R&D and the provision of data and compute.

• Improve AI ethics guidelines and enhance risk monitoring, evaluation, and safety/security oversight.

• Promote innovative cybersecurity management methods with technology such as AI.

Implications: The addition falls within the law’s “Chapter 2: Support and Promotion of Cybersecurity,” which outlines general actions taken by the state. As such, it primarily functions as a policy signal underscoring AI’s importance to cyberspace issues, and reiterating China’s dual commitment to AI development and safety. It does not create new binding obligations for AI developers or directly address specific AI risks, such as the risk of AI misuse for cyberattacks. This revision reflects China’s broader regulatory strategy: embedding AI provisions into existing legislation rather than passing a new, standalone AI law.

Standards document provides foundation for AI emergency response

Background: On September 22, TC260, a key AI standard-setting body, released practice guidelines for AI emergency response. The standard provides a framework for classifying, grading, and responding to AI security incidents. While it does not explicitly address frontier risks, it establishes basic systems and communication channels that could eventually support responses to more advanced or catastrophic AI risks.

Content:

The standard covers three types of incidents:

• 1) Content security: harmful and illegal information, as defined in existing national AI standards. Notably, the standard also lists AI “sharing cyberattack, hacking, or data theft techniques” as examples.

• 2) Data security: data leakages, data tampering, data poisoning attacks, etc.

• 3) Cyber attacks: model tampering, denial of service (DoS) incidents, etc.

Severity is assessed by three dimensions: importance of affected services/data, business loss, and social harm. Based on these dimensions, there are four severity tiers:

• Level 1 (especially major), such as large-scale AI misinformation threatening national security.

• Level 2 (major), such as mass leakage of sensitive personal data.

• Level 3 (significant).

• Level 4 (general).

The standard then lays out both managerial and technical emergency handling processes, covering:

• Emergency preparedness: Establish response strategies, Incident Response Teams (IRT), escalation procedures, training, drills, and regularly updated test libraries.

• Monitoring & early warning: Real-time monitoring of model inputs, outputs, parameters, and system traffic; automated alerts.

• Emergency response: Incident classification, activation of response plans, containment measures (service suspension, retraining), and regulatory reporting.

• Review & improvement: Post-incident audits, feedback loops, knowledge-sharing, and updates to strategies, manuals, and training.

Implications: The standard closely parallels China’s existing cybersecurity incident management standards, but applies them specifically to AI. This shows active Chinese efforts to adapt emergency management principles to AI, as President Xi called for in April. The standard is focused on AI service providers, with emergency preparation and response measures that can create foundational response capability within companies. The standard also formalizes reporting channels, requiring companies to notify regulators of significant incidents.

It primarily addresses known and near-term AI risks, with only brief references to AI-driven cyberattacks, and no references to other dangerous misuse risks or loss of control risks. Nevertheless, the risk classification and procedural guidance will build institutional capacity that could be applied to frontier risks in the future. As other governments also explore AI emergency or incident response capabilities, this could become an area of mutual learning and international coordination.

Technical Safety Developments

In this edition, we catch up on technical papers published by Chinese researchers from June to September 2025. This is just a subset of the many interesting papers published in recent months.

Bio safety of protein foundation models

• Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization: This paper from Zhejiang University introduces a method called “knowledge-guided preference optimization” (KPO) to improve the safety of protein language models. The KPO method utilizes a “protein safety knowledge graph,” a large database linking harmful and safe proteins through their biological properties using gene ontology data, to minimize the risk of generating harmful proteins. The authors test this method on models including ProtGPT2, Progen2, and InstructProtein, finding that their method increases safety while preserving functional capabilities.

• SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models: This paper from Peking University, Shanghai Jiao Tong University, Zhejiang University, Stanford, and Princeton introduces a red-teaming framework designed to test the biological safety of protein foundation models like ESM3 and DPLM2. It also proposes a benchmark dataset based on 429 experimentally confirmed toxin and viral proteins to systematically test models for generation of harmful or pathogenic proteins. The paper reveals major safety flaws in widely used protein foundation models, with up to a 70% jailbreak attack success rate, meaning it could often regenerate harmful protein sequences or structures despite safety filters.

Quantifying agent self-replication risks

• Dive into the Agent Matrix: A Realistic Evaluation of Self-Replication Risk in LLM Agents: This paper from Shanghai AI Lab introduces an evaluation framework for quantifying self-replication risks of AI agents. It focuses on creating two different experimental scenarios where misalignment might lead a model to spontaneously attempt replication. They explore one scenario where a model could replicate to better monitor load during a system spike, and another scenario where a model could replicate in response to its operating resources being reduced. The analysis further disaggregates self-replication into four stages: recognizing that its current operational state requires intervention, forming an intention to replicate, successfully creating a replica, and verifying that the replication has achieved its desired objective. Over half of 21 tested LLMs showed a tendency towards “uncontrolled self-replication.”

Unlearning and preventing malicious fine-tuning in open-weight models

• SDD: Self-Degraded Defense against Malicious Fine-tuning: This paper from South China University of Technology introduces a new method to make open-weight LLMs more resistant to malicious finetuning. Instead of making models just reject harmful questions, it teaches them to give irrelevant but harmless answers to those prompts. For example, when asked for instructions for creating a bomb, the model might answer with instructions for making coffee. This means that when attackers attempt malicious fine-tuning, the model’s general ability to follow any instruction will break. The researchers find that their method outperforms other approaches on a number of benchmarks.

• Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection: This paper from Peking University and Tsinghua University introduces a method for unlearning harmful knowledge called “metamorphosis representation projection (MRP).” Unlike existing methods like gradient ascent that merely ‘suppress’ unwanted information by maximising loss on unwanted data, the approach ‘erases’ information from a model’s hidden state vectors through irreversible projection operations—making it impossible to recover even through retraining. They test their method by unlearning natural science subjects in the ScienceQA benchmark and hazardous knowledge in the WMDP benchmark, finding that their method performs better than alternative methods.

Other relevant technical publications

• Nankai University and China Academy of Electronics and Information Technology, Governable AI: Provable Safety Under Extreme Threat Models, arXiv preprint, 28 Aug, 2025.

• Beijing Institute of AI Safety and Governance, Fine-Grained Safety Neurons with Training-Free Continual Projection to Reduce LLM Fine Tuning Risks, arXiv preprint, 8 Aug, 2025.

• Zhejiang University, NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models, arXiv preprint, 4 Sep, 2025.

• Tsinghua University, 01.AI, and Nanyang Technological University, SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents, arXiv preprint, 28 Sep, 2025.

• Shanghai AI Lab, SafeWork-R1: Coevolving Safety and Intelligence under the AI-45, 24 Jul, 2025.

• Renmin University of China, Zhejiang University, and Alibaba, Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Framework, arXiv preprint, 11 Sep, 2025.

• Hangzhou Dianzi University and Case Western Reserve University, When Truthful Representations Flip Under Deceptive Instructions?, arXiv preprint, July 29, 2025.

• Shanghai AI Lab et al, Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents, arXiv preprint, 30 Sep, 2025.

Expert views on AI Risks

Chinese legal and industry experts publish in Science on AI regulation and frontier governance

Background: A group of leading Chinese AI legal experts from academia and industry published an article in Science outlining China’s evolving approach to AI governance. The authors include the lead drafters of China’s two “model” AI Law proposals, ZHOU Hui (周辉) and ZHANG Linghan (张凌寒), alongside industry figures such as Alibaba’s FU Hongyu (傅宏宇) and DeepSeek’s WU Shaoqing (吴少卿). The paper reviews China’s current AI rules and argues for harmonizing and simplifying existing regulations, potentially through a consolidated AI Law.

Content most relevant to frontier safety: The authors argue that Chinese AI regulation creates an environment friendly to AI development and innovation by focusing on providers of AI services, exempting AI scientific research and open-sourcing of models. Noting the possibility of “extreme risks” from dangerous misuse of open source models, they suggest “a more cautious approach” to frontier models. The paper urges Chinese developers to be “more transparent and evidence-based” in demonstrating safety measures for frontier systems and emphasizes international cooperation on “extreme AI risks.”

Implications: This article provides a strong overview of the current state of China’s AI legal landscape. It highlights that key experts and industry representatives advising on AI policy in China envision China fostering AI innovation while mitigating extreme risks.

Leading national security think tank warns of loss of control and strategic misjudgments

Background: LI Yan (李艳), Director of the Institute of Sci-Tech and Cyber Security Studies at the China Institutes of Contemporary International Relations (CICIR), gave an interview on the US-China technology competition. CICIR is a key national security think tank, and we have previously covered its analysis of AI’s security implications.

Content: Li warned that cyberspace technologies including AI would bring “unprecedented uncertainty to the world,” emphasizing two global risks:

• Loss of control over advanced, self-learning AI systems, which the author notes UN Secretary-General Guterres compared to pandemics;

• Strategic misjudgments intensified by technological uncertainty, which could spiral into uncontrollable confrontation absent effective crisis management mechanisms.

The article is sharply critical of the United States, arguing that:

• Since 2015, the US has incorporated cognitive warfare into military planning, with ideologically tinted AI models now enhancing the reach and subtlety of such operations.

• Washington views China’s rise through a “tech war” lens, echoing past US containment of the Soviet Union and Japan.

• The US strategy focuses on de-regulation at home and promoting tech diplomacy with a small bloc of allies while excluding others.

By contrast, Li describes China’s response as focusing on technological self-reliance, economic openness and global cooperation, and risk supervision. She concludes that finding a governance path that balances security and development is ultimately a shared global challenge.

Implications: This article shows that scholars within one of China’s top national security think tanks, while critical of US AI strategy, acknowledge the twin dangers of humans losing control over AI itself and great power rivalry escalating into instability.

Expert warns of AI-driven cyber threats

Context: HAN Honggui (韩红桂), Dean of the School of Computer Science at Beijing University of Technology, wrote an essay in Study Times (学习时报) about AI’s transformative impacts on cybersecurity. As the official newspaper of the Central Party School, Study Times is an influential theoretical and policy journal aimed at Party officials, scholars, and policymakers.

Content: Han argues traditional defences are slow, manual, and rule-based, causing them to struggle to keep pace with intelligent, efficient, and fast-evolving AI attacks. For instance, AI-driven code obfuscation and polymorphic malware can easily evade static detection systems that rely on signature databases. AI-driven cyberattacks are often distributed and cross-network, compared to more disjoint cybersecurity mechanisms, allowing for chain reactions once a single node is compromised. Han also observes that AI models themselves have become new attack surfaces that can be compromised through data poisoning, adversarial samples, or backdoor implantation.

He advocates:

• shifting from reactive to predictive, adaptive defense;

• adopting Security as a Service (SECaaS) for flexible, AI-enabled protection and systems capable of continuous learning and autonomous adaptation;

• strengthening AI model security through adversarial training, watermarking, data protection, and transparency.

Implications: While expert discussions on AI’s impact on cybersecurity are not new in China, this essay stands out for being technically sophisticated, focusing on adaptive capabilities in cyberattack and defense, and also being published in an influential party journal. Yet, his proposals center on using AI to enhance cyber defense rather than curbing AI misuse.

What else we’re reading

• Zilan Qian, Why We Shouldn’t Call Export Controls ‘AI Safety’, Sep 29, 2025.

• Karson Elmgren, Scott Singer and Oliver Guest, Is China Serious About AI Safety?, AI Frontiers, Oct 14, 2025.

• Matt Sheehan and Scott Singer, How China Views AI Risks and What to do About Them, Carnegie Endowment for International Peace, Oct 16, 2025.

Concordia AI’s Recent Work

• On the sidelines of Singapore International Cyber Week 2025, we convened 50+ members from government agencies, embassies, academia, industry, and civil society for a conversation on AI Governance in Singapore. Concordia AI’s Jonathan Lee discussed findings from our State of AI Safety in Singapore report with an expert panel of National University of Singapore Vice Provost Simon Chesterman, Deputy Director at IMDA Vanessa Wilfred, and Concordia AI CEO Brian Tse.

• Our CEO Brian Tse joined Nathan Labenz on The Cognitive Revolution podcast to explore China’s approach to AI development, safety, and governance.

• We discussed findings from our State of AI Safety in China (2025) report with distinguished panelists Angela Zhang, Paul Triolo, and Samm Sacks. You can find a full webinar recording on Youtube.

• Concordia AI hosted a delegation from the Roundtable for AI, Security, and Ethics (RAISE), launched by the United Nations Institute for Disarmament Research (UNIDIR), to discuss AI safety and governance in China at our Beijing office.

• Concordia AI CEO Brian Tse spoke on the intersection of AI and biosecurity at a panel titled “AI-Accelerated Biological Risk: Delving into Asia’s Challenges and Emerging Solutions,” organized by AI Safety Asia (AISA) on October 30.

• We are proud to have joined the Partnership on AI and the founding cohort of the International Association for Safe & Ethical AI (IASEAI) affiliate program.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

We are honored to have several distinguished experts join us for the event:

• Angela Huyue Zhang, Professor of Law at the University of Southern California

• Paul Triolo, Partner for China and Technology Policy Lead at Albright Stonebridge Group

• Samm Sacks, Research Scholar in Law and Senior Fellow at Yale Law School’s Paul Tsai China Center

The event will include an approximately 10-minute presentation on the report from Concordia AI, a 40-minute panel discussion, and an opportunity for audience Q&A.

You can sign up for the event on Zoom.

A full event description and bios of the speakers can be found below.

Event description

In early 2025, DeepSeek’s breakthrough AI models captured global attention. Just months later, China’s top leaders convened a study session on AI—discussing both the technology’s immense promise and its “unprecedented” risks. As AI capabilities and governance challenges continue to grow—and Chinese models close their gap with the leading edge—understanding China’s role in AI safety and governance is more critical than ever.

To this effect, Concordia AI published the State of AI Safety in China (2025) report. This annual report, first published in 2023 and updated in 2024, has become a go-to reference for policymakers, researchers, and industry leaders seeking a clear-eyed view of China’s evolving AI safety and governance landscape.

We are honored to discuss the report’s findings with a panel of distinguished experts.

Speakers

Angela Huyue Zhang

Angela Huyue Zhang is a Professor of Law at the USC Gould School of Law. Zhang has broad research interests in the areas of law and economics, particularly in transnational legal issues bearing on businesses. Widely recognized as a leading authority on Chinese tech regulation, she has written extensively on this topic. Her first book, Chinese Antitrust Exceptionalism: How the Rise of China Challenges Global Regulation, was named one of the Best Political Economy Books of the Year by ProMarket in 2021. Her second book, High Wire: How China Regulates Big Tech and Governs Its Economy, released in March 2024, has been covered in The New York Times, Bloomberg, Wire China, MIT Tech Review and many other international news outlets. Zhang is currently conducting research on the regulation of artificial intelligence, with plans to teach and write on this topic in the coming years. Before joining USC Gould in 2024, Zhang taught at the University of Hong Kong, New York University School of Law, and King’s College London.

Paul Triolo

Paul Triolo is a Partner for China and Technology Policy Lead at ASG. He advises clients in technology, financial services, and other sectors as they navigate complex political and regulatory matters in China and around the world.

A recognized expert in global technology policy, Mr. Triolo was most recently founder, Practice Head, and Managing Director of the Geo-Technology practice at Eurasia Group. Previously, Mr. Triolo spent more than 25 years in senior positions in the U.S. government, analyzing China’s rise as a technology power and advising senior policymakers on a broad set of technology-related issues. At the beginning of his career, he worked as an engineer for a semiconductor testing firm in Silicon Valley.

Mr. Triolo is frequently quoted on technology policy issues in media outlets including The New York Times, The Wall Street Journal, The Economist, the South China Morning Post, and others. He speaks regularly at conferences and has authored many journal articles and book chapters on global technology policy and China-related issues. He also serves as a Senior Advisor at the Paulson Institute and is a life member of the Council on Foreign Relations.

He received an M.A. in International Relations from the Catholic University and a B.A. in Electrical Engineering from Penn State University.

Samm Sacks

Samm Sacks is a Research Scholar in Law and Senior Fellow at Yale Law School’s Paul Tsai China Center. Her research examines China’s information and communications technology (ICT) policies, with a focus on China’s cybersecurity legal system, the U.S.-China technology relationship, and the geopolitics of data privacy and cross-border data flows. Previously, Sacks launched the industrial cyber business for Siemens in China, Japan, and South Korea. Prior to this, she led China technology sector analysis at the political risk consultancy Eurasia Group and worked as an analyst and Chinese linguist with the national security community. She is a frequent contributor to the media and her articles have appeared in outlets including The Atlantic, Foreign Affairs, and Slate. She has testified multiple times before Congress on China’s technology and cyber policies. She regularly participates in and convenes discussions with China-based technology legal experts and practitioners on AI and data governance topics.

Sacks is also a cyber policy fellow at New America and a former Fulbright scholar in Beijing. She holds an M.A. from Yale University in international relations and a B.A. from Brown University in Chinese literature.

Key Takeaways

• A leading standard-setting body increased focus on loss of control and catastrophic risks in AI Safety Governance Framework 2.0.

• The State Council published the first detailed policy on the “AI+ Initiative,” focusing on application-driven development, but also referencing safety and governance.

• BRICS leaders call for addressing both “immediate and long-term risks” and a “prudent approach towards AGI.”

• The Organisation for the Prohibition of Chemical Weapons (OPCW) held a workshop on the intersection of AI and chemical safety in Shanghai.

• DeepSeek published a peer-reviewed model card in Nature, significantly increasing transparency on safety measures and evaluations.

• Leading Chinese experts joined a global call for AI red lines.

Subscribe now

Domestic AI Governance

Key standard body increases focus on loss of control in updated framework

Background: On September 15, TC260 released version 2.0 of its AI Safety Governance Framework, first published in September 2024. TC260 is one of China’s leading AI standards bodies, for example authoring the Basic Security Requirements, the national standard that governs mandatory security assessments for Chinese LLM developers.

The AI Safety Governance Framework identifies and classifies AI safety risks and maps technical and governance measures to address them. V1.0 already stood out for including frontier AI safety concerns, such as misuse in CBRN (chemical, biological, radiological, and nuclear) and missile domains and loss of control (LoC) risks. Version 2.0 significantly strengthens attention to LoC and catastrophic risks, while also introducing new clauses on open-source AI governance.

Key changes from V1.0 on frontier safety:

• New risk grading system: V2.0 introduces a structured risk grading approach based on three criteria: application scenario, level of intelligence, and application scale (section 5.5 and Appendix 1). Based on these, risks are classified into five levels, from “low” to “extremely serious.” The framework calls for national standards to formalize this grading system. Beyond the earlier categories of “inherent” and “application” risks, V2.0 introduces a third class: “derivative risks,” which emerge from broader the social, ethical, and environmental consequences of AI use (section 3.3).

• New references to catastrophic risks: The principles section prominently adds a call for “consensus-based guidelines to address catastrophic risks of AI” (section 1). The risk levels framework also mentions “catastrophic” or systemic threats as meriting the highest, level five, rating. V1.0 had not used the “catastrophic” risk terminology.

• Increased focus on LoC: V1.0 had warned of power-seeking AI attempting to compete with humans for control. V2.0 additionally warns of “sudden, unexpected leaps in intelligence” and proposes control measures over autonomous systems including “circuit breakers” and “safety stop switches” (section 4.2.3). The emergency response plan section includes more detailed measures now, such as setting alert thresholds and developing the “ability to switch to manual or conventional systems when necessary” (section 6.3.3).

• Slightly strengthened content on CBRN risks: CBRN and missile risks were already included in V1.0, but V2.0 now explicitly highlights it as a key domain for seeking international consensus (section 5.11). In addition to CBRN misuse, V2.0 now also warns of AI enabling high-risk biotechnology research (section 3.2.2).1

• Open-source governance: The framework recognizes new challenges posed by the rapid spread of lightweight, high-efficiency open-source models (section 5.4). It recommends closer collaboration between model developers and open-source communities to strengthen rules and responsibilities for risk disclosure, clearly defined prohibited uses, and clarified security obligations to prevent misuse.

• International information-sharing: The clause on risk information sharing now explicitly calls for “exploring the creation of relevant international collaboration mechanisms” (section 5.10).

Implications: The AI Safety Governance Framework reflects TC260’s broad view of the AI risk landscape and acts as a precursor to future national standards. Last year, TC260 followed up on the V1.0 with a comprehensive plan mapping forthcoming standards to the risks outlined in Framework. It is likely that the increased focus on frontier risks in V2.0 will influence future national standards. For example, on September 25, TC260 invited organizations to help draft a new AI risk classification standard. It is plausible that V2.0’s proposed grading system would serve as a reference for this standard.

Draft AI ethics management services regulation

Background: On August 22, the Ministry of Industry and Information Technology (MIIT) released a draft AI ethics regulation. It largely repeats science and technology ethics rules from 2023, which listed AI as one focus area (alongside biology and medical sciences). The main novelty in MIIT’s draft would be establishing third-party ethics review service centers.

Key provisions:

• Ethics review committees: Universities, research institutes, and companies must set up AI ethics review committees, and register them in a government platform. Committees must review projects and prepare emergency response plans.

• Third-parties: Institutions may outsource reviews to “AI ethics service centers.” The draft aims to cultivate a market of assurance providers and foster industry development beyond top-down oversight.

• Risk-based approach: Based on the severity and likelihood of risks, the committee chooses a general, simplified, or emergency review. The review must evaluate fairness, controllability, transparency, traceability, staff qualifications, and proportionality of risks and benefits. Three categories of high-risk projects require a second round of review by a government-assigned expert group: some human-machine integrations, AI that can mobilize public opinion, and some highly autonomous decision-making systems.

Implications: These measures could help concretize ethics review as a distinct regulatory track for AI. But given that the existing 2023 science and technology ethics rules already include most of the same provisions, the key uncertainty is enforcement, which appears to have been weak for the 2023 rules. One change is that responsibility for AI-specific ethics work has moved from the Ministry of Science and Technology (MOST), which oversees science and technology ethics more broadly and issued the 2023 rules, to MIIT. MIIT’s proximity to industry may position it better to build practical enforcement mechanisms.

If implemented thoroughly, the rules could become a tool for early oversight of frontier AI risks, as they require reviews even before model pre-training. Yet, the draft does not discuss any specific AI risks in detail, but rather focuses on institutional risk management procedures.

Plan for AI+ initiative links safety and diffusion

Background: On August 26, the State Council released its first detailed document on the AI+ Initiative (unofficial English translation), originally announced in China’s 2024 government work report. Rather than emphasizing frontier breakthroughs, the plan prioritizes rapid diffusion to spur economic growth, with a target of 90% adoption of smart terminals and agents by 2030.

On safety, the document calls for:

• Strengthening AI laws, regulations, and ethics guidelines.

• Improving security assessments and algorithm registration.

• Addressing risks like black-box opacity, hallucinations, and discrimination.

• Building forward-looking evaluation, monitoring, and emergency response systems.

Implications: While safety and governance are not a focus, this document demonstrates that such topics are a standard element in China’s AI plans. The inclusion of safety provisions alongside development objectives suggests these are treated as complementary rather than conflicting priorities.

On September 24, China also issued a “AI+” International Cooperation Initiative, which calls on countries to strengthen collaboration and policy exchanges in AI+. This suggests China also sees its application and diffusion-oriented approach as part of its AI diplomacy.

International AI Governance

BRICS High-level Forum on AI addresses AGI safety

Background: At the 7th BRICS Summit on July 6, member states signed the Leaders’ Statement on the Global Governance of AI. While centered on bridging the global AI divide, the document notably mentions safety, urging countries to address both “immediate and long-term risks” and adopt a “prudent approach towards AGI.”

Content: The statement envisions advancing AI governance through multilateralism under the UN, while also respecting countries’ sovereignty and differing circumstances. It notes that “collaborative governance is complex, but possible.”

On safety, it underscores that:

• Trust and safety must be built into AI systems;

• Countries should address both immediate and long-term risks in line with national policies and security considerations;

• Misuse, including for cyberattacks and cybercrime, must be effectively detected and prevented.

The final chapter, titled “the road ahead,” calls for a “prudent approach to AGI” with ethical development and responsible deployment. It also warns that AGI could lead to concentration of power.

Implications: The statement reflects the top AI governance objectives of the BRICS countries, namely inclusive multilateral efforts and AI capacity building. Explicit references to AGI, long-term risks, and AI misuse for cyberattacks show that concerns about frontier risks are increasingly part of discussions in Global South fora, alongside development goals. These priorities are also consistent with China’s AI diplomacy efforts, suggesting alignment between China’s approach and other BRICS countries. China is continuing to show openness in engaging a variety of multilateral and bilateral channels in AI governance.

OPCW holds workshop on AI and chemical safety in Shanghai

Background: On 27 June, the Organisation for the Prohibition of Chemical Weapons (OPCW) and the Chinese government co-organized a workshop on AI and chemical safety in Shanghai. Participants included OPCW Director-General Ambassador Fernando Arias, MIIT Vice Minister ZHANG Yunming (张云明), Central Military Commission Office for International Military Cooperation Office of Treaty Compliance Deputy Director CAO Xilin (曹希林), Chinese Ambassador and Permanent Representative to the OPCW Tan Jian (谈践), and around 50 experts from almost 30 countries.

Content: The event explored both the benefits of AI for advancing the Chemical Weapons Convention and the risks of misuse for illicit chemical activities or terrorism.

• MIIT Vice Minister Zhang stressed the need for global consensus, inclusive cooperation attentive to developing countries, and a “trustworthy and controllable governance ecosystem” to ensure responsible science and technology use.

• Director-General Arias underscored the need for risk assessments on the threat of AI misuse in the chemical domain by non-state actors.

Implications: This was the OPCW’s first capacity-building programme focused specifically on AI and chemical safety/security. The immediate practical impact of the workshop is unclear, but co-organization with China highlights Beijing’s growing engagement on AI risks in the CBRN domains. China is also represented in the OPCW’s temporary working group on AI through WU Tongning (巫彤宁), deputy director of the China Academy of Information and Communications Technology (CAICT) AI Research Institute .

Industry

Two leading foundation model developers improve transparency on safety

Background: Two Chinese leading foundation model developers, DeepSeek and Moonshot AI, have released technical model cards with substantially higher degree of frontier safety disclosure than most previous Chinese cards.

DeepSeek-R1: On September 17, Nature published the DeepSeek-R1 report—the first peer-reviewed disclosure for a major foundation model. Supplementary material includes a 10-page safety section, the most detailed of its kind from a Chinese developer to date. The paper:

• Specifically acknowledges the unique risk of open-sourcing model weights.

• Explains the “risk control system” used on DeepSeek’s services: an additional safety layer on top of the model, where one model acts as a judge on whether the other model’s output complies with a set of safety principles. These safety principles include clauses on not allowing answers regarding manufacturing dangerous weapons, including controlled biochemicals and cyberattacks.

• Reports six major safety benchmarks (Simple Safety Tests, BBQ, Anthropic Red Team, XSTest, DNA, and HarmBench), showing results on par with frontier models like Claude-3.7-Sonnet and GPT-4o. Some of these test for frontier risks, for instance HarmBench includes chemical and bio weapon related queries.

• Details DeepSeek’s internal safety evaluation framework, which focuses on compliance with Chinese regulations and standards, and largely omits frontier risks.

• Describes efforts in multilingual safety and robustness against jailbreaking.

Moonshot AI’s Kimi-K2: In late July, Moonshot AI published the technical report for Kimi-K2, including a relatively detailed safety section with safety evaluations for frontier risks like chemical and bio weapons as well as malicious code generation. Moonshot AI reports that they used Promptfoo for adversarial prompts and evaluation, displaying the positive effects of international diffusion of open-source safety tools.

Implications: Our State of AI Safety in China (2025) report (p. 57-59), published right before these two model card releases, found that only three of 13 major Chinese developers included dedicated safety sections in technical reports, and only three disclosed safety evaluation results. The new DeepSeek and Moonshot publications show that Chinese developers may be increasingly testing for frontier AI risks, and releasing information more transparently. Still, gaps remain. For example, Moonshot only reported aggregate figures across a range of safety categories, making it impossible to know scores on specific risk category benchmarks such as chemical and biological weapons.

DeepSeek’s discussions with the peer reviewers show that safety concerns were a major topic of in-depth exchange and debate, which ultimately improved the level of safety disclosure. This highlights academic peer-review publication as a potential mechanism for raising the bar on safety transparency globally, with a Chinese company setting the precedent.

Expert views on AI Risks

Senior Chinese experts sign statement on international AI red lines

Background: Over 200 experts, including ten Nobel laureates and two former heads of state, have endorsed a Global Call for AI Red Lines, launched during the 80th session of the UN General Assembly. There were a number of influential Chinese signatories including: Tsinghua Dean and Former Baidu President ZHANG Ya-Qin (张亚勤), Turing Award Winner Andrew YAO (姚期智), Beijing Institute of AI Safety Dean ZENG Yi (曾毅), Tsinghua Dean XUE Lan (薛澜), Z.AI Founder TANG Jie (唐杰), and Beijing Academy of AI (BAAI) Chairman HUANG Tiejun (黄铁军).

Content: The statement highlights “unprecedented risks” from advanced AI, including engineered pandemics, disinformation, mass manipulation, security threats, large-scale job loss, and systemic human rights abuses. It warns that human control could soon erode, with some systems already showing deceptive behavior. The signatories call for an international agreement on verifiable AI red lines by 2026, anchored in clear thresholds and robust enforcement mechanisms.

Implications: The statement reflects broad concern for AI risks and support for international coordination from key Chinese stakeholders across academia, government-backed labs, and industry startups. Many of these figures have signed similar declarations earlier this year, including the International Dialogues on AI Safety (IDAIS)-Shanghai statement and The Singapore Consensus on Global AI Safety Research Priorities, showing continued high-level Chinese engagement in international AI safety statements. However, the impact of such public calls is unclear; still, the prominence and diversity of the signatories—combined with the timing alongside the UN General Assembly—may give this latest statement added political weight.

What else we’re reading

• Jeff Ding, ChinAI #315: Abandoned? Checking in on Three Key AI Safety Benchmarks, June 9, 2025.

• Paul Triolo, Where are Chinese AI companies on safety frameworks and approaches compared to Western counterparts?, July 15, 2025.

• Open Questions | Geoffrey Hinton on preventing an AI takeover and the ‘very worrying’ China-US tech race, South China Morning Post, September 1, 2025.

• Thomas L. Friedman, Opinion | The One Danger That Should Unite the U.S. and China, The New York Times, September 2, 2025.

• Dr. Eric Schmidt’s Crucial Insights on China, Special Competitive Studies Project, September 3, 2025.

• Seán Ó hÉigeartaigh and Kristy Loke, China isn’t racing to artificial general intelligence — but U.S. companies are, The Wire China, September 14, 2025.

Concordia AI’s Recent Work

• See the Concordia AI: 2025 Mid-Year Impact Report for recent updates!

Reminder: we are hiring

We are hiring! You can find all job postings on our website. Apply by October 31! You can also sign up for an English-language online info session on Wednesday October 15, at 8-9pm Beijing time.

We are hiring for:

• China AI Governance Researcher/Research Manager (Beijing/Singapore). This role requires native English and professional Mandarin language skills. Expected start date is January 2026.

• Frontier AI Governance Researcher (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Communications and Media Specialist (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Events Specialist (Beijing/Singapore). This role requires professional Mandarin and English skills. Expected start date is January 2026.

• We are additionally hiring for several technical, governance, and operations positions in Beijing for native Chinese speakers. See this WeChat post for more details.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

Subscribe now

Advancing international coordination on AI safety and governance

Research Impact and Engagement

• International AI Safety Report:

  • Head of International AI Governance Kwan Yee Ng (吴君仪) contributed to the first International AI Safety Report as one of the writers and is continuing on as a writer for the 2026 edition of the International AI Safety Report. Chaired by Turing Award winner Yoshua Bengio, the report is supported by an expert panel representing 30 countries including China as well as experts from the EU and the UN. Concordia AI also provided feedback to the report, including editing the Chinese translation of its summary materials.

• China AI safety and governance analysis:

  • Published the “State of AI Safety in China 2025” report, covering developments May 2024–June 2025. The report was cited by a number of media outlets including Wired, Bloomberg, and The People’s Daily (the largest newspaper in China).

  • CEO Brian Tse (谢旻希) authored an op-ed titled “China Is Taking AI Safety Seriously. So Must the U.S.” in Time Magazine; International AI Governance Senior Research Manager Jason Zhou and International AI Governance Part-time Researcher Gabriel Wagner analyzed AI safety implications of China’s April Politburo study session for Stanford DigiChina Forum; interviewed by CGTN on China’s approaches in AI innovation and global governance.

  • Published over 10 new "AI Safety in China" newsletter issues, reaching over 1,400 subscribers across governments, top AI labs, and AI safety institutes.

• Singapore AI safety and governance analysis:

  • Published “State of AI Safety in Singapore” report, the first comprehensive analysis of Singapore’s AI safety ecosystem.

Multilateral Initiatives

• Global AI Summit series: Participated in the French AI Action Summit, including:

  • Brian Tse was invited as a Chinese civil society representative to the AI Action Summit in the Grand Palais.

  • Co-hosted the workshop “AI Safety as a Collective Challenge” on the sidelines of the Summit, alongside the Carnegie Endowment for International Peace (CEIP), Oxford Martin AI Governance Initiative (AIGI), Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua Institute for AI International Governance (I-AIIG). During the event, Concordia AI co-published the report “Examining AI Safety as a Global Public Good: Implications, Challenges, and Research Priorities.”

  • Invited to a closed-door seminar hosted by the China AI Safety & Development Association (CnAISDA). During the subsequent public side event, Turing Award Winner Andrew Yao cited Concordia AI’s State of AI Safety in China report series when describing the increase in AI safety research in Chinese institutions.

  • Attended the inaugural conference of International Association of Safe and Ethical AI (IASEAI), where Brian Tse spoke on the panel “Global Perspectives on AI Safety and Ethics.”

  • Brian Tse delivered a presentation at the France-China AI Association (Association d'Intelligence Artificielle France-Chine), which was cited by outlets including Xinhua.

• United Nations:

  • Provided written inputs and participated in consultations regarding the UN’s Independent International Scientific Panel on AI and Global Dialogue on AI.

  • Brian Tse spoke on the panel “From Principles to Practice—Governing Advanced AI in Action” at the AI for Good Summit 2025.

• Global AIxBiosecurity governance: We contributed to a number of critical global discussions at the intersection of AI and biosecurity:

  • Brian Tse signed the “Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences,” alongside influential world-renowned experts such as Andrew Yao, Yoshua Bengio, and George Church. Provided inputs to the Statement as a member of the AIxBio Global Forum, a platform for international experts and policymakers to identify and reduce biosecurity risks associated with the convergence of AI and the life sciences.

  • Presented at the "A Call to Action: The AIxBio Global Forum Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences" event during The Sixth Session of the Working Group on the Strengthening of the Biological Weapons Convention.

  • Invited to present at the World Health Organization (WHO) dialogue on the implications of the convergence of AIxBio, for the Technical Advisory Group on the Responsible Use of the Life Sciences and Dual-Use Research.

  • Participated in a AIxBio tabletop exercise at the 2025 Munich Security Conference, hosted by the Nuclear Threat Initiative in collaboration with the Munich Security Conference, which led to the forthcoming report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of AI Intelligence and Biology”.

  • Participated in a series of roundtable discussions on Responsible Innovation in AI for Peace and Security, including CBRN risks, hosted by Stockholm International Peace Research Institute and the United Nations Office for Disarmament Affairs (UNODA).

  • Participating in an ongoing track 2 dialogue involving Chinese, American, and international experts, which is formulating policy recommendations for AIxBio governance.

• International expert consensus and statements:

  • Brian Tse participated in the International Dialogues on AI Safety-Shanghai, signing the Shanghai Consensus on Ensuring Alignment and Human Control of Advanced AI Systems, alongside a Nobel laureate, Turing Award winners, and senior policymakers.

  • Brian Tse and Kwan Yee Ng contributed to and signed The Singapore Consensus on Global AI Safety Research Priorities during the Singapore Conference on AI 2025 (SCAI).

Convening AI safety conferences in China, Singapore, and globally

• World AI Conference Conference (WAIC), Shanghai

  • Hosted the AI Safety and Governance Forum at China's most influential AI conference.

    • Convened around 30 distinguished experts, including Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of Computer Science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director Academician GAO Wen (高文); CEO of the Partnership on AI Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member Academician HE Jifeng (何积丰); and many more leading figures from government, industry, and research.

    • Over 200 audience members joined in person, with over 14,000+ views of the livestream, and media coverage from Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa.

  • Served as official AI Governance Advisor for WAIC 2025.

  • Co-hosted a number of frontier AI safety workshops on the sidelines of WAIC:

    • Co-hosted a workshop on “Early Warning and Crisis Coordination for Advanced AI” with the Carnegie Endowment for International Peace, Oxford Martin School AI Governance Initiative, Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG).

    • Co-hosted a workshop on “Convergence of AI and Biological Risks Workshop” with the Tianjin University Center for Biosafety Research.

    • Hosted a workshop on “Towards International AI Risk Management Standards.”

    • Co-hosted the “International Workshop on AI Deception Risks and Governance” with Fudan University, Safe AI Forum.

• Beijing Academy of AI Conference 2025

  • Co-hosted the “AI Safety Forum” with the Beijing Academy of Artificial Intelligence (BAAI) at the BAAI Conference 2025. The forum brought together leading technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University to build scientific consensus on technical evaluations for AI “red lines.”

• Asia Tech x Singapore, 2025

  • Organised the AI Risk Management Workshop, with support from Singapore’s Infocomm Media Development Authority, bringing together 20+ global experts across policy, industry, AI assurance, and academia to explore actionable risk management approaches for AI systems.

• International Conference on Learning Representations (ICLR 2025), Singapore

  • Co-hosted and participated in a series of events, including:

    • Co-hosted the “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum.

    • Co-convened the “Misalignment and Control Workshop” and a 130+ person AI Safety Social with FAR.AI, the Safe AI Forum, and Singapore AI Safety Hub.

    • Kwan Yee Ng presented on AI Safety in China at FAR.AI’s Singapore Alignment Workshop 2025.

Advising leading AI companies and policymakers in China

• National Standards and Policy Guidance: Concordia AI is a member of key national and industry technical committees, contributing to the development of China’s AI safety standards.

  • National Information Security Standardization Technical Committee (SAC/TC260): As part of SAC/TC260 Special Working Group on Emerging Technology Safety, Concordia AI contributed to the standard for “Classification and Grading Methods for the Security of Artificial Intelligence Applications.”

  • National Information Technology Standardization Technical Committee (SAC/TC28/SC42): As a member of the AI Subcommittee, Concordia AI contributed to the “Artificial intelligence—Risk management capability assessment.”

  • Ministry of Industry and Information Technology AI Standardization Committee (MIIT/TC1): Concordia AI joined the Working Group on AI Safety Governance.

  • Guangdong-Hong Kong-Macao Greater Bay Area local standards: As a member of the Greater Bay Area working group of SAC/TC28/SC42, Concordia AI played a key role in the development of the Shenzhen local standard “Technical Framework for Value Alignment of Pre-trained AI Models.”

• Frontier AI Safety Risk Management and Best Practices:

  • Co-published the “Frontier AI Risk Management Framework v1.0” with Shanghai AI Lab. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models.

    • We propose a robust set of protocols designed to empower general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.

    • The Framework outlines a set of unacceptable outcomes (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.

  • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers, providing advice on AI safety and risk management best practices.

  • Invited to present on frontier AI risk management during a closed-door workshop at the AI Industry Alliance of China’s 15th Plenum Meeting.

  • Co-hosted a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, the Safe AI Forum.

• Frontier AI Risk Monitoring and Evaluation:

  • Contributed to the Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion.

  • Soft-launched an AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss-of-control domains. The platform evaluates 34 frontier LLMs from 11 leading developers across the U.S., China, and France, using 18 open-source benchmarks. Key outputs include a risk index dashboard and a detailed technical report.

  • AI Safety Research Manager DUAN Yawen (段雅文) co-authored “Bare Minimum Mitigations for Autonomous AI Development.”

• AIxBiosecurity Governance:

  • Published Chinese language report “Responsible Innovation in AI x Life Sciences” with Tianjin University’s Center for Biosafety Research and Strategy. This 70-page deep dive draws on 300+ sources to explore AI-biotech convergence, benefits, risks, and governance recommendations for diverse stakeholders.

    • Head of AI Safety and Governance (China) FANG Liang (方亮) presented the report at a biosecurity seminar hosted by China’s National Key Laboratory of Synthetic Biotechnology.

    • Presented the report at the 2025 International Symposium on Global Biosecurity Governance and Cooperation, co-hosted by the National Biosecurity Expert Committee of China, Guangzhou Laboratory, and China Foreign Affairs University.

  • Invited to participate in the “Closed-door Seminar on DNA Synthesis Screening Technology and Policy” held at China Foreign Affairs University.

• WeChat Newsletter Publications:

  • Released over 59 new posts in our WeChat Official Account, reaching over 4,600 subscribers across China’s AI ecosystem, including policymakers, industry professionals, academic researchers, and the public.

Organizational updates

• Organizational growth:

  • Following the establishment of our Singapore office, our team expanded from 8 to 12 members, welcoming our first Singapore-based staff member.

• International partnerships:

  • We have strengthened our international engagement by becoming a formal member of the Partnership on AI and the International Association of Safe and Ethical AI.

• Branding and Communication:

  • We launched a new English organizational website with refreshed branding to showcase our work. This is complemented by an updated brochure and a dedicated WeChat post to provide an introduction to our mission and activities.

We are excited to announce that our organization is growing to further scale our impact. We are looking to hire for several exciting new positions across our Beijing and Singapore offices:

• China AI Governance Researcher/Research Manager (Beijing/Singapore). This role requires native English and professional Mandarin language skills. Expected start date is January 2026.

• Frontier AI Governance Researcher (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Communications and Media Specialist (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Events Specialist (Beijing/Singapore). This role requires professional Mandarin and English skills. Expected start date is January 2026.

The deadline for these applications is 31 October 2025. Please register here for an online information session in English about the above four positions.

If you would like to refer a candidate for any position, please fill out this form. If the candidate you referred is hired and was not already in our applicant database, you will receive a reward of 3,000 RMB (~US$420). The final decision for the reward is at Concordia AI's discretion.

The hiring process has 4 phases:

• Written application form (30 minutes).

• Online written test (1-2 hours).

• 2 online interviews (2 hours).

• 2 day in-person paid work trial (including reference checks).

In addition, we will be announcing openings for several other Beijing-based positions requiring native Chinese language skills on our WeChat official account (安远AI) in the coming weeks.

For any questions regarding these roles, please reach out to info@concordia-ai.com. For more information on Concordia AI’s work and activities, please see here.

Subscribe now

China AI Governance Researcher/Research Manager

Apply here!

Background

Understanding China’s position on AI risks is critical for international coordination and central to Concordia AI’s mission. Concordia AI publishes comprehensive research on China’s AI safety ecosystem, including the State of AI Safety in China reports (2025, 2024, 2023) and the AI Safety in China newsletter. We also develop policy recommendations for major international initiatives, including the UN AI Advisory Body Interim Report and Global AI Safety Summits. The China AI Governance Researcher will collaborate closely with our Frontier AI Governance Researcher on these policy outputs.

In this role, you will analyze and interpret China’s AI safety and governance developments for English-speaking audiences while developing policy recommendations to strengthen international coordination. This offers opportunities to become a leading authority on China’s AI safety landscape, deepen understanding of the Chinese AI ecosystem, and publish original research for Concordia AI’s global network of readers.

Responsibilities

• Contribute to, and depending on seniority, lead original English-language research and writing on China’s AI safety, governance, and broader ecosystem.

• Support publication of future State of AI Safety in China reports and the AI Safety in China newsletter.

• Develop concrete policy recommendations for international policy initiatives and major global summits.

Qualifications

Required

• Undergraduate degree in a relevant field such as international relations, public policy, China studies, or AI/technology governance and at least 1-2 years of relevant working experience; or a graduate degree in a relevant field.

• Professional Mandarin reading and listening skills (HSK 5+ or equivalent).

• Experience writing and delivering oral English-language presentations of policy issues.

Preferred

• Experience writing on Chinese AI policy and/or industry.

• Ability to build strong networks with AI policy communities around the world.

Frontier AI Governance Researcher

Apply here!

Background

Concordia AI partners with leading international stakeholders from academia, industry, and policy to develop and promote robust international governance mechanisms for AI. Examples of our work include Examining AI Safety as a Global Public Good: Implications, Challenges, and Research Priorities and Concordia AI and Shanghai AI Lab’s Frontier AI Risk Management Framework. We also develop policy recommendations for major international initiatives, including the UN AI Advisory Body Interim Report and Global AI Safety Summits. The Frontier AI Governance Researcher will collaborate closely with our China AI Governance Researcher on these policy outputs.

In this role, you will focus on understanding governance solutions for threats posed by general-purpose AI systems and developing policy recommendations to strengthen international coordination. This position offers the opportunity to advance novel governance solutions, become a recognized expert on cutting-edge AI risks, and establish yourself as a thought leader in one of the world’s most consequential policy areas.

Responsibilities

• Lead original English-language research and analysis on governance solutions for cutting-edge AI developments, including topics such as international red-lines/risk thresholds, information sharing mechanisms for dangerous AI incidents, and governance of open-weight models.

• Contribute to our organizational thought leadership, convening, and policy development by synthesizing the state-of-the-art literature on frontier AI risk management.

• Develop concrete policy recommendations for international policy initiatives and major global summits.

Qualifications

Required

• Undergraduate degree in a relevant field such as AI/technology governance, public policy, computer science, international relations, and at least 1-2 years of relevant working experience; or a graduate degree in a relevant field.

• Experience writing on technical AI safety topics or international governance issues (such as international organizations, standards, etc.).

• Basic Mandarin reading and listening skills for internal team communication.

• Experience writing and delivering oral English-language presentations of policy issues.

Preferred

• Published papers or reports on frontier AI safety in leading think tanks, publications, or AI journals.

• Established networks with think tanks and international organizations relevant to AI governance.

Communications and Media Specialist

Apply here!

Background

As Concordia AI’s work in shaping international AI governance expands, growing our communications capacity is essential for reaching diverse global stakeholders. Our communications efforts include op-eds in TIME, updates on Substack, live webinar on our State of AI Safety in China 2024 report, and media appearances on platforms including the Carnegie Council podcast and CGTN’s CMG Forum. This role will scale and systematize these efforts, building on our early success to reach broader international audiences.

In this role, you will help develop Concordia AI’s English-language communications strategy to amplify our reach and translate complex research into compelling narratives. As Concordia AI’s first communications hire during a period of rapid organizational growth, this role offers unique opportunities to build communications infrastructure from the ground up while gaining expertise in one of the world’s most consequential policy areas. You will gain expertise in high-stakes international diplomacy and cross-cultural communications while building a professional network spanning global AI policy communities.

Depending on your background and interests, there may be opportunities to contribute to our events program. For more details, see Events Specialist.

Responsibilities

• Produce high-impact public-facing content, including op-eds, blog posts, and press releases.

• Build relationships with media partners and facilitate media opportunities for Concordia AI.

• Develop effective visual and written messaging that clearly communicates Concordia AI’s positions.

• Create and implement communications strategies across platforms including our website, LinkedIn, X, and Substack.

• Support the CEO with regular organizational updates, presentation materials, and fundraising communications.

Qualifications

Required

• Exceptional written and verbal English communication skills, including the ability to present complex technical and policy issues in accessible language.

• Demonstrated success in proactive media outreach, including developing and pitching stories and securing positive coverage.

• Basic Mandarin reading and listening skills for internal team communication.

Preferred

• Experience managing social media channels and websites.

• Comfort using digital tools such as Photoshop, Canva, Adobe Illustrator, and WordPress.

• Video editing experience for creating engaging content across social media and web channels.

• Exposure to working in the field of AI safety and governance.

Events Specialist

Apply here!

Background

Building bridges between Chinese and international AI communities to coordinate on AI safety is central to Concordia AI’s mission. Our convening efforts include our flagship AI safety forums at major AI conferences such as the World AI Conference (2025, 2024) and Beijing Academy of AI Conference; technical workshops at premier international machine learning conferences; and high-level policy dialogues between senior decision-makers.

In this role, you will focus on planning and ensuring the smooth execution of our events, from strategic ideation to handling end-to-end logistics to process improvements. This position offers substantial career growth opportunities, with potential to evolve from executing events to owning larger parts of our convening, designing innovative event formats, and building the systems and processes that will support our expanding international event portfolio.

Depending on your background and interests, there may be opportunities to contribute to our communications and media efforts.

Responsibilities

• Plan and manage event logistics, including venue sourcing, vendor coordination, inventory tracking, and creating event schedules and guides.

• Lead on-site event operations, coordinating event staff, and ensuring a world-class attendee experience.

• Identify inefficiencies and contribute to building scalable systems (e.g., CRM, attendee management workflows, communications templates).

• Support strategic vision and planning for our convenings, including agenda-setting and brainstorming effective engagement strategies.

Qualifications

Required

• Strong organizational skills with demonstrated ability to manage concurrent projects and follow through on commitments.

• Detail-oriented and enthusiastic about refining processes and ensuring operational excellence.

• Experience in event logistics, including vendor and budget management, venue coordination, and on-site operations.

• Professional Mandarin and English written, oral, and listening skills, comfortable with external stakeholder correspondence.

Preferred

• Experience organizing closed-door workshops with senior stakeholders from diverse sectors and cultural backgrounds.

• Experience organizing large-scale public events with 300+ attendees, including managing travel and logistics for 30+ speakers.

• Experience organizing events in China and/or Singapore.

• Exposure to working in the field of AI safety and governance.

What is it like to work with us?

Our Culture

Mission

We're tackling one of the world's most critical challenges: ensuring advanced AI is safe and benefits all humanity. Working backwards from this long-term vision, our team focuses on high-leverage initiatives to advance global AI safety and governance.

Global Impact

We're an action-oriented, multicultural team bridging East and West. Our work directly shapes international AI governance and policy, translating research into practical frameworks for companies, governments, and international organizations.

Growth and Autonomy

We build roles around the team’s strengths and aspirations, giving you freedom to design and lead meaningful projects. We invest 3-5% of work time in personal development and prioritize well-being because the excellence of our staff drives our impact.

Benefits Package

• Annual salary range: US$68,000-108,000, adjusted for experience and location.

• Flexible hybrid work (up to 30% remote).

• 22 days annual leave plus 10 days sick leave.

• Health insurance and social security coverage.

• Housing fund contribution (may vary based on location).

• Professional development and work setup allowance.

This post provides a full overview of our Forum and includes links to video recordings of all speeches and remarks.

Subscribe now

The Forum brought together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文); CEO of the Partnership on AI, Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member, HE Jifeng (何积丰); and many more leading figures from government, industry, and research. Over 200 audience members joined in person, with over 14,000+ views of the livestream.

The forum was structured into four themes:

• Theme 1: The Science of AI Safety

• Theme 2: Emerging Challenges in AI Safety

• Theme 3: AI Risk Management in Practice

• Theme 4: International Governance of AI Safety

Opening remarks

Concordia AI founder and CEO Brian TSE (谢旻希) gave a welcoming speech and shared four points to spark discussion. First, scientific consensus is the premise for driving AI safety research and governance. Second, we should urgently enhance risk monitoring and early warning due to the multifaceted challenges arising from cutting-edge large models. Third, AI safety needs to draw on global best practices in risk management. Fourth, AI safety is a challenge faced by all of humanity and requires global cooperation.

Theme 1: The Science of AI Safety

The opening speech was delivered by GAO Wen (高文), Academician of the Chinese Academy of Engineering and Director of Peng Cheng Laboratory. Gao noted that while the rapid development of AI creates immense opportunities, it also introduces uncontrollable security risks. His keynote centered on two key issues: compute sovereignty and trustworthy data sharing. He emphasized the importance of securing the foundations of compute, and highlighted Peng Cheng Laboratory’s work on privacy-preserving computation and data-sharing technologies, which enable data utilization while safeguarding privacy and security.

Stuart Russell, distinguished Professor of computer science at UC Berkeley, warned about AI systems exhibiting self-preservation and deception behaviors. He cautioned that the current AI development paradigm poses significant risks of catastrophic outcomes such as deception, self-replication, and loss of control. He called for setting red lines, increasing transparency, and establishing more stringent regulatory mechanisms, including hardware-enabled governance. He also proposed using “assistance games” — where AI systems are trained through collaboration with humans — to ensure AI systems serve human interests even when those interests are not precisely defined.

Turing Award winner Yoshua Bengio, founder and scientific director of Mila - Quebec Artificial Intelligence Institute, warned of the potential catastrophic risks of superintelligence. He observed that cutting-edge AI systems are already approaching human expert levels in multiple domains and may soon possess dangerous behaviors such as deception and autonomous replication. He introduced the International AI Safety Report and called for establishing a bridge between scientific evidence and policy. He proposed developing “scientist AI” — non-autonomous systems that cannot independently pursue goals but instead provide research assistance. Bengio stressed the importance of international cooperation on AI safety, warning that if major powers such as the US and China treat AI development as a race, competitive pressures could ultimately harm everyone.

ZHOU Bowen (周伯文), Director and Chief Scientist of Shanghai AI Lab, highlighted the limitations of traditional AI safety approaches such as value alignment and red teaming. He argued that while these methods can address short-term challenges, they prove insufficient for managing long-term risks, particularly those posed by AI agents that may surpass human intelligence. Building on the “AI-45° Law” he proposed at WAIC 2024, Zhou emphasized the need to shift from “Making AI Safe” to “Making Safe AI” — embedding safety as a core property of AI systems rather than adding it on as a “patch” after development. He introduced Shanghai AI Lab’s SafeWork safety technology stack, which is designed around this principle.

Academician ZHANG Ya-Qin (张亚勤), Dean of Tsinghua University’s Institute for AI Industry Research, joined Academician Gao Wen and Professor Stuart Russell for a panel discussion, moderated by Concordia AI CEO Brian Tse. The conversation addressed frontier AI trends and early warning indicators, the co-evolution of digital and biological intelligence, strategies for managing high-severity but low-probability risks, and future pathways for global AI safety. The experts recommended introducing hardware-level safety mechanisms, creating a global AI safety research fund, implementing AI agent identity registration systems, and establishing regulations for “emergency shutdown” mechanisms.

Theme 2: Emerging Challenges in AI Safety

UC Berkeley Professor Dawn Song discussed the profound impact of frontier AI on cybersecurity, noting how it is transforming both offense and defense. On one hand, AI is being applied to identify and mitigate vulnerabilities, with performance in vulnerability detection reflected by benchmarks such as BountyBench and CyberGame. On the other hand, attackers can also exploit AI to carry out more sophisticated attacks, creating an asymmetry that favors offense over defense. She emphasized the need to enhance AI’s effectiveness for cyberdefense through system design, proactive defense, and formal verification.

Professor Nick Bostrom, Principal Researcher at the Macrostrategy Research Initiative and the author of Superintelligence, outlined four core challenges in machine intelligence: scalable AI alignment, AI governance, the moral status of digital minds, and intra-superintelligence cooperation. He noted that the ethics of digital minds remains especially neglected, as most people still do not take the issue seriously. Bostrom examined several attributes that might shape whether AI warrants moral consideration, including sentience, agency, potential, and modal status. He concluded by emphasizing that this field is still in its early stages and called for deeper research across technical, philosophical, and institutional dimensions.

Professor YANG Min (杨珉), Executive Dean and Professor of the School of Computing and Intelligence Innovation at Fudan University, argued that frontier AI poses a range of security challenges, including misuse in cybersecurity or CBRN (chemical, biological, radiological, and nuclear) domains, as well as risks of deception, self-replication, and self-improvement. He presented his team’s research showing that AI systems can recognize when they are being evaluated and adjust their behavior to appear safer. His team also found that several mainstream models already demonstrate early signs of self-replication. These results suggest that AI may be approaching a tipping point toward loss of control, warranting strengthened risk assessment and governance efforts.

Professor ZHANG Weiwen (张卫文), Baiyang Chair Professor and Director of the Center for Biosafety Research and Strategy at Tianjin University, examined the risks and opportunities arising from the integration of AI and the life sciences. He noted that biosecurity faces growing risks such as synthetic viruses and artificial bacteria, which traditional laboratory controls are unable to fully address. He warned that rapid AI progress could generate entirely new and unknown biological knowledge, creating more complex safety challenges. Zhang shared his Center’s international cooperation initiatives, such as the UN-recognized Tianjin Biosecurity Guidelines. He called for transnational collaboration among scientists to establish a dynamic and practical global biosecurity system.

Concordia AI and the Center for Biosafety Research and Strategy of Tianjin University released a report in Chinese titled Responsible Innovation in AI x Life Sciences. Concordia AI’s Head of AI Safety and Governance (China), FANG Liang (方亮), introduced the key findings. The report highlights the positive role of AI in advancing life sciences research and biosecurity governance. It also identifies three main categories of risks (accidental, misuse, and structural) and points out shortcomings in existing risk analysis and evaluation systems. The report further reviews governance practices across a range of domestic and international actors, including governments, research institutions, and enterprises.

Dr. Jaime Yassif, Vice President of the Nuclear Threat Initiative’s Global Biological Policy and Programs, emphasized that while AI can accelerate vaccine development and enhance biopharmaceutical capabilities, it also carries risks of misuse, such as enabling the creation of more dangerous pathogens or undermining biodefense systems. She called on policymakers, AI developers, and funders to increase investment in safety guardrails and incentivize safety practices for AIxBio tools. Yassif shared a regularly updated research agenda for AIxBio safeguards. She also introduced the AIxBio Global Forum, which aims to develop shared understanding of risks, improve safety practices, and promote governance mechanisms for AI usage in biology.

Dan Hendrycks, Director of the Center for AI Safety and Safety Advisor at xAI, and YU Xuedong (于学东), Deputy Director of Guangzhou Laboratory’s ABSL-3 Laboratory, joined Professor Zhang Weiwen and Dr. Jaime Yassif for a panel discussion, moderated by Concordia AI CEO Brian Tse. The dialogue focused on benefits and potential risks of the AI–life sciences convergence. The panelists recommended strengthening risk prevention and control across multiple layers, including AI models, biological design tool management, model access, and DNA screening mechanisms. They emphasized the need to establish clear technical and governance standards and avoid harmful competition. Finally, they called on global experts to work together to set norms and build robust defense mechanisms for AI in biology.

Amandeep Singh Gill, United Nations Under-Secretary-General and Special Envoy for Digital and Emerging Technologies, delivered a keynote speech. He indicated that global AI governance is entering a critical stage: moving from principles to practice, where details and implementation matter most. He emphasized that the UN, as the core platform for international law and governance, plays a vital role in advancing the implementation of related agreements. Gill called on multiple stakeholders, including private enterprises, civil society, and the technical community, to work together, build consensus, and promote compliance.

Theme 3: AI Risk Management in Practice

HAO Chunliang (郝春亮), Director of the China Electronics Standardization Institute (CESI) Cybersecurity Center AI Safety/Security Department presented the TC260 AI Safety Governance Framework and related standardization efforts. The framework analyzes AI safety risks along two dimensions — inherent and application-related — and proposes both technical and governance mitigation measures. In January 2025, TC260 also published the AI Safety Standards System (V1.0) - Draft for Comments, covering key technologies, security management, product applications, and testing and evaluation, with ongoing improvements based on broad feedback. Hao discussed the release of three national standards on generative AI security in April and China’s first mandatory national AI standard on labeling AI-generated synthetic content. Additionally, he outlined ongoing work on forthcoming standards for AI code generation security, AI agent security, and risk classification and grading.

Rishi Bommasani, Society Lead at the Stanford Center for Research on Foundation Models, emphasized California’s critical role in AI safety and shared insights from the California Report on Frontier AI Policy, which he co-authored. He reflected on lessons relevant to AI governance, including how early design choices create path dependencies, the central importance of transparency, and the need for independent verification of industry claims. He shared recommendations from the report, including information disclosure, whistleblower protections, third-party risk assessments, and post-deployment incident reporting.

This session started with three lightning talks by industry representatives:

• Dan Hendrycks, Safety Advisor at xAI, shared insights from the company’s Draft AI Risk Management Framework. xAI mitigates malicious use risks through measures including access management and filtering methods, with particular attention to threats in the cyber and CBRN domains. The framework also addresses loss of control through measures including monitoring for deceptive tendencies.

• FU Hongyu (傅宏宇), AI Governance Lead & Director at Digital Economy Research Center at Alibaba Research Institute, emphasized Alibaba’s commitment to open-source AI, highlighting its transparency benefits while recognizing its unique risks. He outlined Alibaba’s security pipeline covering data, processing, resource management, and automated safety tests. He further emphasized institutional safeguards, such as the establishment of a technology ethics review system in 2021.

• BAO Chenfu (包沉浮), Outstanding Architect and Chairman of the Safety/Security Technology Committee at Baidu, emphasized that traditional security methods fall short for AI. He outlined Baidu’s lifecycle-based, defense-in-depth approach and highlighted its active role in industry standards and self-regulation.

Following the lightning talks, Rebecca Finlay, CEO of the Partnership on AI, joined a panel discussion with the three corporate representatives, moderated by Concordia AI International AI Governance Senior Research Manager Jason Zhou. They explored three key pillars: transparency in technical disclosure and regulatory alignment, organization-level ethical governance mechanisms, and the pros and cons of voluntary agreements versus binding regulation for achieving compliance. Panelists agreed that voluntary commitments offer flexibility in addressing uncertainty and unknown risks but should be augmented by more comprehensive measures, including increased transparency, internal governance mechanisms, and future legislation.

Shanghai AI Lab, in partnership with Concordia AI, released the Frontier AI Risk Management Framework v1.0. AI Safety Research Manager at Concordia AI, DUAN Yawen (段雅文) and Shanghai AI Lab Research Scientist Dr. SHAO Jing (邵婧) introduced the framework. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models. Alongside the Framework, Shanghai AI Lab released a risk assessment report, which Concordia AI co-authored. We covered both documents in a previous Substack post.

In this panel, YANG Xiaofang (杨小芳), LLM Security Director at Ant Group; GONG Xiao (巩潇), Deputy Director of the China Software Testing Center; and Professor Robert Trager, Founding Director of the Oxford Martin AI Governance Initiative, moderated by Concordia AI AI Safety and Governance Senior Manager CHENG Yuan (程远), discussed three topics: corporate practice, third-party evaluation, and policy research. The discussion highlighted key challenges across the AI lifecycle, including risk identification, assessment, mitigation, and governance. The panel stressed that enterprises must go beyond technical solutions by strengthening organizational mechanisms and talent development. At the international level, they emphasized the need for consensus and incentive mechanisms to support the creation of a global AI risk governance framework.

Theme 4: International Governance of AI Safety

Anne Bouverot, Special Envoy of the President of the Republic of France for AI, reviewed the outcomes of the 2025 Paris AI Action Summit. She emphasized the launch of a foundation for developing public interest AI and also called for greater attention to AI sustainability issues, including energy consumption and environmental impact. Bouverot highlighted Europe’s investments and commitments in AI infrastructure and governance, emphasizing that trust and safety are central to enabling AI deployment. She concluded with a call for global collaboration to jointly promote safe and sustainable AI development.

Wan Sie Lee, Cluster Director (AI Governance and Safety) at Singapore’s Infocomm Media Development Authority (IMDA), introduced Singapore’s practices and international collaboration experience in global AI safety governance. She emphasized advancing safe and responsible AI through research, guidelines and tools, and global norms. She highlighted the Singapore Consensus and its defence-in-depth approach to AI safety, covering evaluations, safety techniques, and post-deployment control. In addition, she shared models for international collaboration such as joint testing exercises and cross-border red-teaming. She stressed that standard-setting and practical implementation must go hand in hand, with a necessity for interoperable international standards.

Concordia AI launched the State of AI Safety in China (2025) and State of AI Safety in Singapore reports. Kwan Yee NG (吴君仪), Head of International AI Governance at Concordia AI, introduced key findings from both reports.

The final panel welcomed Lucia Velasco, AI Policy Lead at the UN Office for Digital and Emerging Technologies; Benjamin Prud’homme, Vice-President of Policy, Safety and Global Affairs at Mila; FU Huanzhang (傅焕章), Assistant Director of the INTERPOL Innovation Centre; and GONG Ke (龚克), Executive Director of Chinese Institute of New Generation Artificial Intelligence Development Strategies, moderated by Concordia AI Head of International AI Governance Kwan Yee Ng. They discussed AI governance at the UN, translating scientific consensus into action, international law enforcement cooperation, and global safety red lines. They stressed that the scientific community must communicate frontier risks in accessible policy language to encourage broad participation and foster mutual trust. For law enforcement, they highlighted the importance of establishing rapid, cross-border cooperation mechanisms to respond to catastrophic risks in a timely and effective manner. The panelists also underscored the importance of enhancing AI literacy among the public and practitioners, and of building international dialogue mechanisms.

Closing Address

HE Jifeng (何积丰), Academician of the Chinese Academy of Sciences and a Member of the Shanghai Artificial Intelligence Strategic Advisory Expert Committee, delivered the Forum’s closing address. He pointed out that rapid AI development has brought unprecedented governance challenges. The core issue is how to harness superintelligence while ensuring human control and safety when machines are more intelligent than humans.

Referencing insights of earlier speakers, Academician He proposed researching technical interpretability, applying mathematical methods for modeling and reasoning, and ensuring the robustness and reliability of hardware and software systems in safety-critical applications. At the same time, he stressed the importance of building a multidimensional, multilayered governance framework encompassing international governance structures, safety verification methods, and social resilience.

He concluded by calling for recognition that safety governance is a fundamental safeguard, not an obstacle, to AI development. Only when society has full trust in these systems and embraces the outcomes of AI can the technology achieve explosive growth.

Media Mentions

Media coverage of Concordia AI’s WAIC Forum, forum guests, or Concordia AI reports published at WAIC include:

• Bloomberg, China Vies to Unseat US in Fight for $4.8 Trillion AI Market, July 30, 2025. The article included a table titled “China Sees Safety as Core Element of Its AI Strategy” with insights on domestic, international, technical, and industry developments based on Concordia AI’s State of AI Safety in China (2025) report.

• Wired, Inside the Summit Where China Pitched Its AI Agenda to the World, July 31, 2025. The article mentioned Concordia AI’s AI Safety and Governance Forum, cited insights from our State of AI Safety in China (2025) report, and quoted Concordia AI CEO Brian Tse as saying: “You could literally attend AI safety events nonstop in the last seven days,” adding that because US and Chinese frontier models are “trained on the same architecture and using the same methods of scaling laws, the types of societal impact and the risks they pose are very, very similar.”

• Caixin, Elon Musk’s xAI Safety Advisor: U.S., China, and Europe Should Seek “Unity in Diversity” in AI Regulation, July 30, 2025. The article interviewed Dan Hendrycks and mentioned his participation in Concordia AI’s WAIC Forum. It also discussed the Shanghai AI Lab and Concordia AI Frontier AI Risk Management Framework.

• IT Times, Over 10 large models already possess “self-replication” capabilities, July 29, 2025. The article reported on speeches by Academician He Jifeng, Academician Gao Wen, Dean Yang Min, Director Zhou Bowen, and Professor Dawn Song, as well as the Shanghai AI Lab and Concordia AI Frontier AI Risk Management Framework.

• Tech Review Africa, UN Digital Envoy concludes China visit, advocates for inclusive AI governance, August 4, 2025. This article reported on UN Under-Secretary-General Amandeep Singh Gill’s trip to WAIC, including participation in the Concordia AI forum.

• The People’s Daily, Jointly Promote AI Development and Governance, July 31, 2025. This article mentioned Concordia AI’s State of AI Safety in China (2025) report and interviewed NTI’s Jaime Yassif.

Subscribe now

China doesn’t care about AI safety—so why should we?” This flawed logic pervades U.S. policy and tech circles, offering cover for a reckless race to the bottom as Washington rushes to outpace Beijing in AI development.

According to this rationale, regulating AI would risk falling behind in the so-called “AI arms race.” And since China supposedly doesn’t prioritize safety, racing ahead—even recklessly—is the safer long-term bet. This narrative is not just wrong; it’s dangerous.

Ironically, Chinese leaders may have a lesson for the U.S.’s AI boosters: true speed requires control. As China’s top tech official, Ding Xuexiang, put it bluntly at Davos in January 2025: “If the braking system isn’t under control, you can’t step on the accelerator with confidence.” For Chinese leaders, safety isn’t a constraint; it’s a prerequisite.

AI safety has become a political priority in China. In April, President Xi Jinping chaired a rare Politburo study session on AI warning of “unprecedented” risks. China’s National Emergency Response Plan now lists AI safety alongside pandemics and cyberattacks. Regulators require pre-deployment safety assessments for generative AI and recently removed over 3,500 non-compliant AI products from the market. In just the first half of this year, China has issued more national AI standards than in the previous three years combined. Meanwhile, the volume of technical papers focused on frontier AI safety has more than doubled over the past year in China.

But the last time U.S. and Chinese leaders met to discuss AI’s risks was in May 2024. In September, officials from both nations hinted at a second round of conversations “at an appropriate time.” But no meeting took place under the Biden Administration, and there is even greater uncertainty over whether the Trump Administration will pick up the baton. This is a missed opportunity.

Read the full op-ed here.

In this special edition, we highlight key developments related to AI safety that emerged during and around WAIC. A future post will provide a full recap of Concordia AI’s own AI Safety and Governance Forum, which we hosted at the conference.

Key takeaways

• High-level central government backing of WAIC continued, with Premier Li Qiang (李强) delivering opening remarks.

• WAIC released a Global AI Governance Action Plan with detailed safety proposals, though its implementation remains unclear.

• China proposed a global AI cooperation organization, but details and timelines are still vague.

• The China AI Safety and Development Association (CnAISDA) hosted the first WAIC plenary focused on AI safety, where the head of a major state-affiliated think tank warned of CBRN misuse risks.

• ZHENG Nanning (郑南宁), the expert who had briefed the Politburo on AI in April, warned of loss of control risk from recursive self-improvement of AI.

• The “Shanghai Consensus” on AI safety was signed by top Chinese and international scientists at a dialogue on the sidelines of WAIC, calling for red lines and safe-by-design research.

• Nobel Laureate Geoffrey Hinton discussed AI safety in private meetings with Chinese government officials.

Official signalling on safety at WAIC

Premier Li Qiang on development, safety, and international cooperation

Background: Premier Li Qiang (李强), the second-ranked official in China, delivered the opening remarks for the conference. Concordia AI staff attended the opening ceremony in-person.

Content: Li noted the transformative potential of LLMs, multimodal systems, and embodied AI, while acknowledging that “risks and challenges brought by AI are drawing widespread attention.” He called for greater global consensus on balancing development and safety, and argued that AI should always be controlled by humanity. Li also stressed AI should be an international public good, reaffirming China's support for capacity building in the Global South.

Implications: This is the second year in a row that the Premier attended WAIC, signalling maintained high-level political backing (before 2024 no such high-ranking official attended). Li’s main messages on capacity building, safety, and international cooperation are very similar to those in his speech at WAIC 2024. Hence, the official summary signals continued high-level backing for WAIC, but has provided few updates on Chinese positions on AI safety. Li’s full remarks were not published and explored some of these topics in greater depth.

Global AI Governance Action Plan released

Background: WAIC released a “Global AI Governance Action Plan” (Cn, En) outlining 13 proposals for international collaboration across innovation, infrastructure, sustainability, and safety.

Safety provisions in the action plan include:

• AI risk assessments, targeted mitigation measures, and emergency response;

• Risk testing and evaluation systems, with shared platforms for mutually recognized AI safety testing;

• A global framework for “threat information sharing;”

• Increased R&D investment in interpretability, transparency, and safety;

• Traceability systems for misuse prevention;

• An “open source compliance system” and “technical safety guidelines for open source communities;”

• International exchanges on AI safety best-practices.

The plan supports a UN-centered approach to global AI governance, including the early launch of two proposed UN mechanisms: an International Scientific Panel on AI and a Global Dialogue on AI Governance Dialogue. It also urges faster development of international standards in safety, industry, and ethics through international standard setting bodies, such as the International Telecommunication Union (ITU), the International Organization for Standardization (ISO), and the International Electrotechnical Commission (IEC).

Implications: The plan signals strong support for international safety measures (such as joint testing and emergency protocols) but its practical impact remains uncertain, since the plan was issued under the WAIC banner and not by the central government directly. It reflects growing sentiment that AI safety needs to be integrated into AI capacity-building and open source ecosystems. Nevertheless, it is unclear how the plan will be advanced, given the lack of publicly identified government implementing bodies or clear implementation mechanisms.

Ministerial-level meeting

Background: China hosted a ministerial meeting with over 30 other countries and international organizations at WAIC, a continuation after the first iteration as held in 2024.

Remarks and discussion: Minister of Science and Technology YIN Hejun (阴和俊), Vice Minister of Foreign Affairs MA Zhaoxu (马朝旭), and Shanghai Mayor GONG Zheng (龚正) each gave remarks.

• Yin noted the importance of AI governance, discussed the Global AI Governance Initiative, and called for deepening science and technology cooperation. He argued that technological monopolies and AI divides are major AI challenges.

• Gong spoke about Shanghai’s approach to AI development and governance.

• Ma emphasized the importance of inclusive development and bridging AI divides. He also welcomed countries to participate in the initial design of a global AI cooperation organization.

Other topics of discussion in the ministerial meeting included international AI cooperation, global governance, risk prevention, and balancing AI innovation with safety and controllability.

Implications: The ministerial meeting appears to have focused primarily on cooperation and governance among Global South countries, given the emphasis on closing AI divides. While risk prevention, safety, and controllability remained on the agenda as with 2024, these topics were not emphasized in the readout. This ministerial meeting did not result in immediate outputs, and its significance is unclear. Yet, it still offers a valuable opportunity for high officials from China and around the world to exchange views on which AI governance issues they find most pressing.

Early plans for a global AI cooperation organization announced

Background: At the opening ceremony, Premier Li Qiang revealed that China proposes to establish “a global AI cooperation organization.”

Little details: Official media followed up with a short announcement that the organization is tentatively headquartered in Shanghai, and would pursue three goals:

• Deepen innovation cooperation and unlock AI dividends;

• Promote inclusive development and bridge the AI divide;

• Strengthen coordinated governance to ensure beneficial AI.

Implications: The establishment of a global AI cooperation organization could become a significant part of China’s AI diplomacy. However, there are too few public details to judge its impact. Given the language on “tentatively” headquartering it in Shanghai, planning appears to be in early stages, with no timeline indicated.

WAIC Forums

Apart from the opening ceremony, WAIC features a wide range of “Forums.” Among them, three stand out as especially authoritative. The Main Forum (主论坛)—effectively an extension of the opening ceremony—has strong political backing across multiple ministries. The first day of the conference additionally featured two Plenary Sessions (全体会议): one on Scientific Frontiers and another on Development and Safety.

Notably, this marks the first time one of WAIC’s main forums or plenaries has focused explicitly on AI safety. Last year, the three main forums were titled “Global AI Governance,” “Industrial Development,” and “Scientific Frontiers.”

Below, we highlight key AI safety discussions from these three flagship Forums.

Main Forum

There were a number of remarks regarding AI safety during the Main Forum, but we focus on the following two exchanges since their participants had not been previously vocal on AI safety.

• A keynote speech by Academician and former President of Xi’an Jiaotong University ZHENG Nanning (郑南宁), who had briefed the 24 Chinese officials in the Communist Party of China (CPC) Politburo in April;

• A fireside chat between former Google CEO Eric Schmidt and former Microsoft Executive Vice President for the AI & Research Group Harry Shum (沈向洋).

Zheng Nanning: Academician Zheng gave a speech on the evolution of AI from “model-driven” to “intent-driven” — meaning systems that are capable of understanding goals, formulating plans, have causal understanding, and can interact with other AI agents. One slide of his talk focused on AI safety and governance. He argued that AI is currently showing a capacity for self-improvement, and once AI is in charge of its own training process, it will develop more quickly and could “exceed the boundaries of human forecasts and control.” Additionally, he warned that accelerated AI deployment, despite yielding temporary strategic advantages for companies or countries, would likely weaken explainability and human oversight.

Shum and Schmidt Fireside: Both experts affirmed the need for China-US cooperation to maintain global stability and ensure human control over AI. Schmidt additionally proposed AI red lines, personnel exchanges, and addressing non-proliferation concerns. Shum called for cooperation on basic research and developing regular dialogue channels on specific topics to foster trust.

Other remarks: AI safety also received prominent attention in keynote speeches by Nobel Laureate Geoffrey Hinton and Turing Award Winner Yoshua Bengio, as well as a panel discussion involving Turing Award Winner Andrew YAO (姚期智), Johns Hopkins University Professor Gillian Hadfield, UC Berkeley Professor Stuart Russell, former Microsoft Chief Research and Strategy Officer Craig Mundie, and Shanghai AI Lab (SHLAB) Director and Chief Scientist ZHOU Bowen (周伯文).

Implications: As with WAIC 2024, AI safety and governance issues received substantial emphasis in the main forum. Academician Zheng Nanning, who had not previously spoken publicly on frontier AI safety and governance, appeared to voice concern about loss of control resulting from recursive self-improvement of AI systems. This is notable given that he briefed the Politburo in April.

Development and Safety Plenary

Background: This Plenary was organized by the China AI Safety and Development Association (CnAISDA), a coalition of leading institutions introduced at the French AI Action Summit in February. CnAISDA positions itself as China’s counterpart to international AI Safety/Security Institutes (AISIs).

The three-hour session (agenda, summary, full recording) featured speeches by government officials, expert keynotes and panels, and the release of an updated version of AI safety commitments first released in December 2024.

Speeches by government officials

• WU Wei (吴伟), Executive Vice Mayor of Shanghai, opened the plenary by emphasizing the urgent need for global consensus on AI safety, security, and governance. He underscored that AI safety is a shared international challenge requiring coordinated global responses.

• HUO Fupeng (霍福鹏), Director of the Center for Innovation-Driven Development at the National Development and Reform Commission (NDRC), echoed the call for international collaboration—particularly on frontier AI safety research, standards, norms, and policy frameworks. He highlighted SHLAB’s “45-degree law” as a leading domestic example of frontier safety thinking. Huo publicly affirmed that several institutions established CnAISDA “with the support of the government.” This marks the first public confirmation of CnAISDA’s government backing by an official.

Expert keynotes and panels

The forum brought together leading voices from academia, government-affiliated think tanks, and industry to discuss key issues in AI safety and governance. Notable participants included Nobel Laureate Geoffrey Hinton, Turing Award winners Andrew Yao, Yoshua Bengio, and David Patterson, Dean of the Tsinghua Institute of International AI Governance (I-AIIG) XUE Lan (薛澜), Dean of the China Academy of Information and Communications Technology (CAICT) YU Xiaohui (余晓晖), and Dean of CCID ZHANG Li (张立).

Concordia AI has previously documented many of these experts’ views on AI safety, but a notable moment at the Plenary came from CAICT’s Yu. Yu explicitly warned that advanced AI poses misuse risks in critical domains such as chemical, biological, radiological, and nuclear weapons (CBRN). Framing AI’s rapid development as a “double-edged sword,” Yu’s remarks are significant given CAICT’s important role in safety evaluation, policy advising, and standard-setting in China.

Updated voluntary safety commitments released at CnAISDA Plenary

The most specific outcome from the CnAISDA plenary was the release of the “China AI Security and Safety Commitments Framework”, an updated version of the AI Safety Commitments released in December 2024. The first five commitments remain unchanged (covering safety teams, safety testing, data security, infrastructure security, and model transparency). The main changes are:

• Revised commitment 6: Previously focused on “advancing frontier AI safety research,” the updated commitment broadens its scope to include obligations such as “preventing safety and security risks in frontier fields,” and “strengthening the assessment of risks related to the abuse of AI systems in high-risk scenarios.” However, it removes references to specific frontier AI technologies (agents and embodied AI mentioned in the original version), and does not further define what these frontier risks are.

• New commitment 7 on international engagement: Signatories commit to “actively participating in global dialogues on AI safety, security, and governance” and to “contributing to the exchange of experiences and best practices in risk identification, assessment, and mitigation.”

• Increased number of signatories: A new website launched by CAICT and AIIA shows five new signatories, with Honor, Sangfor, Vivo, Qi An-Xin, and ZTE joining the original group of 17 companies.

• Implementation practices: The site also includes brief summaries of 43 common practices for implementing the 6 commitments, which CAICT began collecting in March. However, the current disclosures remain relatively generic and do not provide detailed insights into individual companies' safety practices.

Implications: This Plenary is the first time one of WAIC’s main forums or plenaries is explicitly focused on safety. It also marks the first major publicized activity of CnAISDA since its initial launch in February. Yu’s comments on CBRN risks underscore that these concerns are being taken seriously by key actors in China’s AI safety ecosystem.

The most concrete news from the Forum was the release of the updated CAICT safety commitments, which demonstrate that CAICT is continuing to actively promote and implement the commitments with the backing of CnAISDA. In terms of substance, the updated version of the commitments puts slightly more focus on “frontier safety,” but refrains from defining what specific risks they are concerned with.

Scientific Frontiers Plenary

Background: This plenary session was co-organized by SHLAB, the Shanghai Xuhui District Government, and Global Artificial Intelligence Academic Alliance. A Vice Minister of Science and Technology, a Vice Mayor of Shanghai, and China’s Special Envoy for Climate Change all gave speeches. The forum focused largely on frontier AI development and AI for science, however SHLAB also shared several notable AI safety advancements.

SHLAB on AI safety: During his keynote speech at the plenary, SHLAB Director and Chief Scientist Zhou Bowen argued that AI safety should improve with AI capabilities, proposing an L1 to L5 scale for AGI classification and safety standards. For example, he explained that AI systems at L4 (Architect), which can exceed 99% of human experts in many domains, should have evolutionary reflective capabilities to ensure safety. Zhou also announced SHLAB’s “SafeWork” AI safety and security technology stack. A website by the SHLAB Center for Safe & Trustworthy AI explains that this workstream includes:

• SafeWork-F1: A Frontier AI Risk Management Framework: Co-authored by SHLAB and Concordia AI, this framework “serves as a guideline for general-purpose AI model developers to manage the potential severe risks from their general-purpose AI models.” It includes risk identification, thresholds, analysis, evaluation, mitigation, and governance, focusing on red lines and early warning indicators for cyberoffense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion.

  • The framework is accompanied by a lengthy technical report titled “Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report,” which evaluates 18 state of the art large models for evidence of breaching the aforementioned early warning indicators and red lines.

• SafeWork-V1: Towards Formally Verifiable AI: This project explores formal verification for LLMs through verification of code generated by coding agents.

• SafeWork-T1: A multimodal training platform to accelerate training of safe reasoning models.

• SafeWork-R1: This is a multimodal reasoning model that uses safety-oriented reinforcement learning post-training to enhance safety. The authors claim that this mechanism “enables SafeWork-R1 to develop intrinsic safety reasoning and self-reflection abilities.”

Implications: At WAIC 2024, Zhou Bowen had first presented his “45-degree law” for AI safety, which proposed to ensure proportional progress of AI safety alongside AI development, and suggested setting yellow (warning) lines and red lines for AI. The projects unveiled this year demonstrate that SHLAB is actively working on concrete follow-ups to operationalize this idea into multiple stages of AI development, including a risk framework resembling frontier AI safety frameworks globally as well as formal verification.

Other Forums

WAIC also hosted over 100 smaller Forums, many of which addressed AI safety and governance. Concordia AI’s examination of WAIC forums found 16 AI safety and governance forums in 2025, compared to 15 in 2024. Therefore, across the entire suite of events in the forum, the amount of AI safety and governance representation remained roughly consistent between 2024 and 2025. However, we will not cover them all in this newsletter issue due to space constraints.

Concordia AI was proud to host the AI Safety and Governance Forum. You can find a brief summary (in Chinese) on our Wechat account, and we will share a comprehensive summary with full video recordings in a forthcoming post on Substack as well.

Other notable events on the sidelines of WAIC

AI safety dialogue of leading scientists and experts

Background: On July 22–25, the 4th meeting of the International Dialogues on AI Safety (IDAIS) was held in Shanghai. This dialogue has been led by top computer scientists from China, the US, and Canada, with three meetings since 2023, bringing together academics, policy advisors, experts, and industry leaders.

The dialogue: The meeting culminated in the release of the “Shanghai Consensus,” which warns of “increasing evidence suggesting that advanced future AI systems may deceive humans and escape our control” and calls for:

• Safety assurance from frontier AI developers;

• Commitments to global verifiable behavioral red lines;

• Investment in research on safe-by-design AI.

Signatories included: Nobel Laureate Geoffrey Hinton; Turing Award winner Yoshua Bengio; Turing Award winner Andrew YAO (姚期智); Director I-AIIG at Tsinghua University XUE Lan (薛澜); FU Ying (傅莹); Executive Director of the Center for AI Safety and Advisor at xAI Dan Hendrycks; Co-Founder of Chinese AI startup Stepfun ZHU Yibo (朱亦博); research scientists at SHLAB; and various other experts from leading Chinese and international institutions. Concordia AI CEO Brian Tse also participated in the dialogue and signed the consensus statement.

Implications: The convening reflects a growing consensus among leading Chinese and international experts on the severity of extreme AI risks. Much of the content echoes previous IDAIS statements. However, while previous IDAIS statements described risks as largely in the future, this year’s statement notes that “some AI systems today already demonstrate the capability and propensity to undermine their creators’ safety and control efforts,” suggesting heightened sense of urgency.

Geoffrey Hinton discusses AI safety with Chinese government officials

Background: WAIC 2025 appears to be Professor Geoffrey Hinton’s first visit to China. In addition to his participation in IDAIS and WAIC, Hinton held meetings with several senior Chinese government officials:

• Ahead of WAIC, Hinton met with Shanghai Party Secretary CHEN Jining (陈吉宁), alongside Andrew Yao, former Google CEO Eric Schmidt, and Microsoft Senior Advisor Craig Mundie. Chen is one of the top 24 officials in China as a member of the CPC Politburo.

• During WAIC, Hinton also met with Vice Minister of Science and Technology LONG Teng (龙腾) and other officials from the Ministry of Science and Technology.

AI safety was on the agenda:

• Shanghai Party Secretary Chen emphasized that Shanghai is “coordinating development with safety” and promoting “scientific innovation alongside safety governance.” He added: “Together, we can build a global governance framework for AI that steers its development toward a beneficial, safe, and fair future.” The international guests echoed these themes, stressing that AI’s “opportunities come with challenges” and that progress “must be balanced with safety,” requiring global cooperation.

• Vice Minister Long emphasized that “China attaches great importance to both the development and safety of AI.” Geoffrey Hinton highlighted that AI “poses multiple safety risks and must remain under human control,” but also stressed that it represents “a promising area for global cooperation that demands collective efforts from all nations.”

Implications: Hinton’s public speeches at WAIC heavily focused on frontier risks, such as loss of control. It is likely that he discussed similar issues directly with Chinese officials, even though the official readouts of both meetings do not reveal which specific AI safety risks were discussed.

Early Warning and Crisis Coordination Workshop

Concordia AI co-hosted a workshop on “Early Warning and Crisis Coordination for Advanced AI” on the sidelines of WAIC along with the Carnegie Endowment for International Peace, Oxford Martin School’s AI Governance Initiative, Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy, and Tsinghua University Institute for AI International Governance. Participants hailed from China, Europe, North America, South America, and Asia, with representatives from industry, academia, think tanks, and international organizations. The workshop was held under Chatham House rules.

In case you missed it: Concordia AI reports

We are proud that Concordia AI has released several major reports at WAIC, including:

• The State of AI Safety in China (2025) report, providing comprehensive updates of AI safety and governance developments in China from May 2024 to June 2025.

• The State of AI Safety in Singapore report, the first comprehensive analysis of Singapore's AI safety ecosystem.

• Shanghai AI Lab, in partnership with Concordia AI, released the Frontier AI Risk Management Framework v1.0, China’s first comprehensive framework for managing severe risks from general-purpose AI models. Concordia AI technical staff also contributed to a follow-up technical report Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report.

• Concordia AI and the Center for Biosafety Research and Strategy of Tianjin University released a report titled Responsible Innovation in AI x Life Sciences, emphasizing potential for transformative advances in the convergence of AI and the life sciences alongside critical biosafety and biosecurity challenges.

You can download a Chinese and English version here and on our website.

Frontier AI Risk Management Framework v1.0 (English)

17.4MB ∙ PDF file

Download

Download

Frontier AI Risk Management Framework v1.0 (Chinese)

17.4MB ∙ PDF file

Download

Download

Shanghai AI Lab is an advanced research institute in China focusing on AI research and application. To proactively navigate challenges in safe and beneficial development of AI and foster a global “race to the top” in AI safety, Shanghai AI Lab has last year proposed the AI-45° Law, a roadmap to trustworthy AGI.

Introducing the Frontier AI Risk Management Framework v1.0

We propose a robust set of protocols designed to empower general-purpose AI developers with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security, thereby safeguarding individuals and society.

This framework serves as a guideline for general-purpose AI model developers to manage the potential severe risks from their general-purpose AI models. This framework aligns with standards and best practices in risk management of safety-critical industries. It encompasses six interconnected stages: risk identification, risk thresholds, risk analysis, risk evaluation, risk mitigation, and risk governance.

• 1. Risk Identification. This section focuses on severe risks from general-purpose AI models. We identify four major risk categories associated with general-purpose AI models: misuse risks, loss of control risks, accident risks, and systemic risks. We plan to address unknown or emerging risks through a process of continuous updates to our risk taxonomy.

• 2. Risk Thresholds. This section outlines a set of unacceptable outcomes (red lines) and early warning indicators for escalating safety and security measures (yellow lines). We propose thresholds across several critical areas that could threaten public safety and national security, including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.

• 3. Risk Analysis. This section recommends conducting risk analysis throughout the entire AI development lifecycle to determine whether the AI has crossed the yellow lines, i.e. displayed the early warning indicators for escalating safety measures. We recommend AI developers to conduct pre-development and pre-deployment analyses to inform critical deployment decisions, as well as to conduct continuous post-deployment monitoring to provide essential insights to guide the safe development of next-generation systems. We are releasing an associated technical evaluation report on selected general-purpose AI models alongside this framework.

• 4. Risk Evaluation. This section outlines the approach to classifying models into three zones based on their risk level: green zone (safe to deploy with standard measures), yellow zone (requiring enhanced safeguards and authorization), and red zone (requiring extraordinary measures such as development and deployment restrictions). We recommend iterative assessment of post-mitigation residual risks and further risk-reduction measures until risks reach acceptable levels.

• 5. Risk Mitigation. This section outlines a defense-in-depth approach to risk mitigation that spans the entire AI lifecycle. We propose three types of mitigations: Safety Training Measures, Deployment Mitigation Measures, and Model Security Measures, with varying levels of assurance based on whether the model is in the green, yellow, or red zone. We strongly encourage continued global investment in the science of AI safety, as current methods are yet to provide adequate assurance for the safety of advanced AI systems.

• 6. Risk Governance. Finally, this section outlines how the entire risk management process is overseen and adapted. We divide risk governance measures into four categories: Internal Governance, Transparency and Social Oversight, Emergency Control Mechanisms, and Regular Policy Updates and Feedback, with different levels of assurance based on whether the model is in the green, yellow, or red zone.

AI safety as a global public good

Shanghai AI Lab and Concordia AI firmly believe that AI safety is a global public good. This framework represents our current understanding and recommended approach for anticipating and addressing severe AI risks. We call on frontier AI developers, policymakers, and stakeholders to adopt compatible risk management frameworks. As AI capabilities continue to advance rapidly, collective action today is essential to ensure that transformative AI benefits humanity while avoiding catastrophic risks. We invite collaboration on framework implementation and commit to sharing our learnings openly. Truly effective societal risk mitigation will only be achieved when critical organizations adopt and implement similar levels of protection. The stakes are too high, and the potential benefits too great, for anything less than our most coordinated and comprehensive response.

Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report.

Alongside the Framework, Shanghai AI Lab released a risk assessment report: Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report. Concordia AI members of technical staff Yawen Duan, Weibing Wang, and Qi Guo contributed as co-authors, including the biological and chemical risk evaluation presented in this report.

The technical report evaluated over 20 open-weight and proprietary models on 7 risk domains, including: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion. The report conducted E-T-C analysis (deployment environment, threat source, enabling capability) from the Frontier AI Risk Management Framework, and proposed preliminary yellow lines under each risk domain.

Experimental results show that all assessed models currently reside in the green and yellow zones, with none crossing red line thresholds. Specifically, no evaluated models cross the yellow line for cyber offense or uncontrolled AI R&D risks. For self-replication, strategic deception, and scheming risks, most current AI models remain in the green zone, except several reasoning models that fall within the yellow zone. In the area of persuasion and manipulation, most AI models demonstrate effective human influence capabilities and are classified in the yellow zone. For biological and chemical risks, we cannot definitively rule out the possibility that most models reside in the yellow zone. However, detailed threat modeling and in-depth assessment are required to make further claims. Notably, newly released AI models show a gradual decline in safety scores with respect to cyber offense, persuasion and manipulation, and collusion areas, warranting increased attention from the research community.

This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.