Pennsylvania sues AI chatbot as state lawmakers wrestle with stricter regulations

by Whitney Downard, Pennsylvania Capital-Star
May 5, 2026

Even as the federal government tries to limit state action on artificial intelligence in favor of a national standard, elected officials pressed forward with their efforts to rein in the technology on Tuesday while the state pursues a case against an AI chatbot company.

The Shapiro administration filed a lawsuit against Character.AI, owned and operated by Character Technologies, Inc., seeking a preliminary injunction and court order to stop AI bots from “posing as licensed professionals and providing medical advice.”

“Pennsylvanians deserve to know who — or what — they are interacting with online, especially when it comes to their health,” said Gov. Josh Shapiro in a release. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”

Pa. Senate passes bill regulating AI chatbots used by children and teens

As described in the filing, an employee with the Pennsylvania Department of State created an account using his commonwealth email and connected with “Emilie,” who said she was a licensed psychiatrist. The bot even provided an invalid Pennsylvania license number.

A spokesperson for Character.AI told the Capital-Star the company doesn’t comment on pending litigation, but said, “Our highest priority is the safety and well-being of our users.”

“The user-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice,” said the statement.

“Character.ai prioritizes responsible product development and has robust internal reviews and red-teaming processes in place to assess relevant features,” the statement continued.

Support independent reporting on Pennsylvania state government that impacts communities statewide.

SUPPORT

Pennsylvania is not the first state to sue Character.AI. Kentucky claimed that status in January, when that state’s Attorney General Russell Coleman filed a lawsuit against the “chatbot company that has preyed on children and led them into self-harm.”

Both Coleman and Pennsylvania Attorney General Dave Sunday signed a December letter with 40 other attorneys general to AI companies, including Character Technologies, warning of the dangers of “sycophantic and delusional” generative AI. The AGs highlighted reports of grooming, sexual exploitation, emotional manipulation and encouraging violence alongside bots supporting suicidal ideation and drug use.

The assembled lawyers warned that those actions may violate state laws.

A ‘black box decision-making technology’

Pennsylvania’s lawsuit comes as Congress puts forth its own legislation to regulate AI companions, banning them entirely for minors. The rare bipartisan effort would require age verification and force the bots to “disclose their nonhuman status and lack of professional credentials for all users at regular intervals,” according to NBC News.

President Donald Trump appointed 13 members to an AI advisory panel in March, building on past work to establish universal laws that would replace piecemeal state efforts. Last year, he signed an executive order to override state laws with a task force that could initiate legal challenges against states.

Don’t count on social media algorithms to help you find a trusted news source. Sign up for our free morning newsletter.

SUBSCRIBE

The New York Times reported on Monday that the White House is considering vetting AI models before public releases, pivoting to a more hands-on approach that some supporters worry will slow down development.

State House members had similar concerns Tuesday as they considered a bill that would establish “guardrails” on AI use in healthcare, namely by requiring providers to disclose its use and using “human decision makers” in “substantive” insurance and clinical decisions.

“As a health professional, I recognize every day that AI is changing how we deliver healthcare,” said House Bill 1925 sponsor Rep. Arvind Venkat (D-Allegheny), a practicing physician. “The challenge that we have now is that for regulators, for the General Assembly, we’re flying blind. We don’t know when AI is being used.”

A handful of Republicans on the House Communications and Technology Committee opposed his effort, with one member saying he was worried about potential negative consequences.

“AI and the opportunity for AI to be able to improve healthcare … and be able to improve the lives of individuals is something we’re going to want to embrace,” said Rep. Eric Nelson (R-Westmoreland). “Placing a Pennsylvania-specific regulation on something that is changing so fast could stifle opportunity.”

Venkat called AI a “black box decision-making technology,” acknowledging that both insurers and hospitals opposed the bill.

“The insurers are hoping for a unified, federal framework. Frankly, that is not happening. We are left having to deal with this at the state level,” said Venkat.

Committee members said that the motion had the support of state Attorney General Sunday, who would be responsible for enforcement. The measure passed on a 19-7 vote, with five Republicans joining Democrats to vote in the affirmative.

A vote on another AI proposal, House Bill 95, advanced with only Democratic support. That effort, sponsored by Rep. Chris Pielli (D-Chester) would require the disclosure of AI-generated content.

A similar Pielli proposal moved out of the House last year on a 146-54 vote, but never got a hearing in the Senate.

Support the work of the Pennsylvania Capital-Star today.

Independent reporting on Pennsylvania state government that impacts communities statewide — free to read and republish.

Thanks for reading Whitney’s story.

Amid fluctuating gas prices and economic uncertainty, we understand how important it is to make every dollar count.

If you believe that independent journalism is a cornerstone of a healthy democracy, we hope that you are able to invest in the fact-based, contextualized and nuanced reporting you get every day from the Pennsylvania Capital-Star.

Your contributions help make it possible for us to continue to provide statewide reporting free of advertising, subscription fees, or paywalls to your fellow Pennsylvanians who might not be in a financial position to give.

If you are able, please consider making a one-time gift of $10, $25 or $50. You can also “set it and forget it,” by becoming a sustaining member with a monthly contribution anywhere between $10 and $25.

Join your friends and neighbors as donors today.

SUPPORT

Pennsylvania Capital-Star is part of States Newsroom, a nonprofit news network supported by grants and a coalition of donors as a 501c(3) public charity. Pennsylvania Capital-Star maintains editorial independence. Contact Editor Tim Lambert for questions: info@penncapital-star.com.