Child safety advocates emphasize the necessity for tech companies to adopt 'Safety by Design' principles to mitigate risks associated with AI-generated CSAM.

Concerns are also raised about major investors, including Google and Nvidia, whose funding may indirectly support the creation of CSAM.

AI companies, notably Runway and Stability AI, are facing intense scrutiny for their tools that can generate child sexual abuse material (CSAM).

The alarming accessibility of these harmful tools is underscored by the fact that Stable Diffusion 1.5 has been downloaded over 6 million times.

Stable Diffusion 1.5, developed by Runway and funded by Stability AI, has been implicated in the production of CSAM and was hosted on platforms like Hugging Face and Civitai.

In response to growing concerns, Hugging Face has removed Stable Diffusion 1.5 from its platform, although it is still accessible on Civitai.

A report by the U.K. Internet Watch Foundation has highlighted the alarming ease with which photorealistic AI-generated CSAM is being produced.

Research from Stanford’s Internet Observatory and Thorn has brought attention to the misuse of open-source AI image generation tools for creating CSAM, prompting calls for stronger regulations.

Child safety advocates report a surge in the misuse of these tools, with malicious actors fine-tuning models using real abuse material.

There is an urgent need for a combined effort of regulation, voluntary commitments, and public pressure to prevent the misuse of generative AI in creating CSAM.

Experts are advocating for stronger regulatory oversight and legislation to combat the production and distribution of AI-generated CSAM.

Proposed measures include detecting and removing CSAM from training data, implementing content provenance systems, and banning apps that exploit children.