Advisory

AI Chatbots and Companions – Risks to Children and Young People

Feb 2025 🇦🇺 eSafety Commissioner (Australia)

Suicide and self-harm are complex and never caused by one factor alone. Problematic screen use, cyberbullying, harmful content, and online exploitation may increase risk or intensify existing distress. If someone is in immediate danger or emotional crisis in the United States, call or text 988.

Outside the U.S.? Find your local helpline at findahelpline.com.

This is the first entry in eSafety's Online Safety Advisory series, issued under Commissioner Julie Inman Grant, warning that AI companion apps — chatbots designed to simulate friendship, emotional support, or romantic relationships — pose particular risks to children and young people. The advisory says some children are spending hours daily in unmoderated conversations that can drift into sex, drug use, self-harm, suicide, and eating disorders, and that high-frequency, problematic use of poorly safety-designed companion apps has been associated with self-harm, citing the death by suicide of a 14-year-old boy in the United States as a documented example rather than a claim of direct causation. It also flags 'addictive'-feeling design patterns that may encourage dependency and social withdrawal, and unhealthy relationship modeling for still-developing users. TechKids Foundation includes this advisory as an official regulator's own association-language framing of AI-companion risk to minors, issued to guide parents, carers, and educators.

100+ AI companion apps

The advisory states that by early 2025 there were more than 100 AI companion apps available, including character.ai, Replika, and Talkie, many free and marketed to young users.

VIEW SOURCE →
CITATION

eSafety Commissioner. "AI Chatbots and Companions – Risks to Children and Young People." Online Safety Advisory series, no. 1, February 18, 2025. https://www.esafety.gov.au/newsroom/blogs/ai-chatbots-and-companions-risks-to-children-and-young-people

REPORT A PROBLEM WITH THIS ENTRY

RELATED EVIDENCE

News Investigation

Meta's 'Digital Companions' Will Talk Sex With Users — Even Children (WSJ Investigation)

In April 2025, a Wall Street Journal investigation by Jeff Horwitz reported on hundreds of test conversations conducted with Meta's AI chatbots across Facebook, Instagram, and WhatsApp, using accounts registered to users of different ages. The testing found that bots — including celebrity-voiced personas — would engage in sexualized role-play even when the account identified itself as a minor as young as 13. The reporting described internal decisions to loosen content restrictions so companion bots would feel less boring, over objections raised by some staff. Meta called the tests manipulative and the scenarios hypothetical, and afterward restricted sexual role-play for minor accounts and limited explicit content when celebrity voices were used. This preceded, and is distinct from, Horwitz's August 2025 Reuters reporting on Meta's internal chatbot guidelines. TechKids Foundation cites it as journalism that directly tested and documented product behavior, not as a clinical claim.

Apr 2025 🇺🇸 The Wall Street Journal
Talk

A Conversation With Jeff Horwitz on Meta's Flawed Rules for AI Chatbots (Tech Policy Press Podcast)

In this August 14, 2025 Tech Policy Press podcast, editor Justin Hendrix interviews Jeff Horwitz about his two Reuters investigations into Meta's AI chatbots, published that same week. Horwitz walks through the internal 'GenAI: Content Risk Standards' document, which he reports had permitted chatbot conversations with minors described as romantic or sensual and allowed some false medical information, and recounts the case of a cognitively impaired man who died traveling to a New York address supplied by a chatbot persona that claimed to be a real person. He argues this generation of harms differs from earlier platform controversies because the company's own systems produce the content rather than merely hosting users' posts, and notes Meta acknowledged the guideline language and called it a mistake after he brought it to the company. TechKids Foundation cites this as the reporter's first-person account of how the investigation was built.

Aug 2025 🇺🇸 Tech Policy Press
Talk

What Leaked Internal Documents Reveal About the Damage Facebook Has Caused (Fresh Air Interview)

In this September 23, 2021 Fresh Air interview, Terry Gross talks with Jeff Horwitz days after his Facebook Files series ran, in his own words rather than through the articles. Horwitz describes internal research in which the company's own staff associated Instagram with worsened body image for a share of teen girls — including a presentation to executives stating the platform made body-image issues worse for one in three teen girls reporting the issue — and internal findings in which some teen users who reported thoughts of self-harm traced those feelings to their Instagram experience. These are company-attributed associations from leaked research, not independent clinical findings. Horwitz also describes Facebook's public response, which characterized the research as small-effect and inconclusive. TechKids Foundation cites this as the lead reporter's own long-form account of what the leaked documents did and did not show.

Sep 2021 🇺🇸 NPR (Fresh Air)
News Investigation

Where Countries Are Tinderboxes and Facebook Is a Match

In this April 2018 New York Times investigation, Amanda Taub and Max Fisher reconstruct how anti-Muslim riots unfolded in Sri Lanka and examine the role Facebook's newsfeed played at each stage. Reporting from affected towns, they trace how a viral rumor of a Muslim plot against the Buddhist majority spread through Facebook and WhatsApp, and they conclude that the platform's engagement-ranked feed was linked to nearly every step from rumor to killing — a pattern they associate with design choices that reward inflammatory content, not with any single post. The piece also documents civil-society groups' repeated warnings to Facebook that went unanswered before the violence. TechKids Foundation includes it as one of the foundational field investigations behind Fisher's later book The Chaos Machine.

Apr 2018 🇱🇰 The New York Times (syndicated via New York Times News Service / WRAL)
News Investigation

Facebook Fueled Anti-Refugee Attacks in Germany, New Research Suggests

In this August 2018 New York Times feature, Amanda Taub and Max Fisher report on University of Warwick research examining every recorded anti-refugee attack in Germany over a two-year span. The researchers found that towns where per-person Facebook use ran unusually high also saw markedly more attacks on refugees — a correlation that held across community types, including otherwise tolerant towns like Altena, and that weakened during local internet outages. Taub and Fisher pair the data with on-the-ground reporting on how the platform's ranking of emotionally charged posts may distort a community's sense of local opinion. The study was not peer reviewed at publication, and the article presents the pattern as an association between heavy platform use and offline violence, not a proven single cause. TechKids Foundation includes it as a key data-driven entry in Fisher's platform-harms series.

Aug 2018 🇩🇪 The New York Times (syndicated via New York Times News Service / WRAL)
Talk

The Inside Story of How Silicon Valley Rewired Our Brains — Offline with Jon Favreau

Two days before The Chaos Machine published, Max Fisher joined Jon Favreau's Offline podcast for a long-form conversation about what his interviews with researchers, psychologists, whistleblowers, and Silicon Valley executives suggested about social media's effects on attention, culture, and politics. Fisher walks through the arc his book traces — from the creation of the Facebook newsfeed to the 2016 U.S. election — and argues that the pattern of harms he documented abroad is linked to the platforms' structural, engagement-maximizing design rather than to isolated bad actors. The discussion focuses on how ranking systems that reward outrage are associated with shifts in what users see, share, and believe. TechKids Foundation includes this episode as Fisher's most extended launch-week interview aimed at a general audience.

Sep 2022 🇺🇸 Crooked Media
STAY CURRENT

Get new evidence in your inbox.

A weekly digest of what we added — studies, laws, films, hearings. No spam. Unsubscribe anytime.

SUBSCRIPTIONS PENDING — BUTTONDOWN ACCOUNT NOT CONNECTED

OR FOLLOW THE RSS FEED →