California lawmakers have approved Senate Bill 574, a first-of-its-kind state law that sets binding rules for how attorneys may use generative artificial intelligence in their practice — and the measure now sits on Governor Gavin Newsom's desk.
The bill, known as the Court A.I. Protection Act, cleared the Legislature this week after the Senate concurred in Assembly amendments on August 31 by a vote of 39-0. It was introduced by Senator Thomas Umberg, a Democrat from Santa Ana who chairs the Senate Judiciary Committee, whose office says the goal is to protect people receiving legal services as AI tools spread through the profession. Reuters characterized the measure as the first state law of its kind governing lawyers' use of generative AI. Until Newsom signs it, however, its requirements are not law. For more context on this story, see our ongoing AI trends.
What SB 574 Requires
At its core, the bill draws a bright line: lawyers cannot delegate the practice of law to a generative AI system. An attorney who uses AI remains fully responsible for the legal work produced with it.
The verification duties are the most concrete. Lawyers using generative AI must take reasonable steps to verify the accuracy of the output — including case and statutory citations — and correct erroneous or fabricated material before relying on it. Going further, any attorney responsible for a brief, motion, pleading, or other paper filed with a California court would have to personally verify every citation in the filing, whatever its source. Reliance on an AI tool would not excuse a false citation.
Disclosure is the second pillar. Attorneys would need to disclose the use of generative AI to a court for documents submitted to it, and those using AI to create material distributed to the public would have to consider whether to disclose its use.
Confidentiality Rules for the ChatGPT Era
The bill's third pillar addresses where lawyers' data goes. SB 574 would prohibit attorneys from entering confidential, personally identifying, or other nonpublic information into a generative AI system unless access to that information is restricted to the attorney and to people the attorney authorizes who are themselves obligated to protect its confidentiality.
The provision enumerates the sensitive categories in scope: Social Security numbers, dates of birth, medical and psychiatric records, financial information, account numbers, and addresses or telephone numbers belonging to parties, witnesses, victims, and court personnel. Notably, the bill does not ban the use of AI systems with confidential material outright — the test is whether the system adequately restricts access to what is entered.
Why Lawmakers Acted
The legislation responds to the most visible failure mode of AI in courtrooms: hallucinated authority. Large language models can produce answers that look authoritative while inventing cases, quotations, and statutes, and courts around the country have sanctioned attorneys who filed briefs citing fictitious legal authorities generated by AI.
Privacy concerns have grown alongside. Earlier this year, U.S. District Judge Jed Rakoff in New York ruled in United States v. Heppner that a criminal defendant's self-directed conversations with Anthropic's Claude were not protected by attorney-client privilege or the work product doctrine — in part because the defendant had disclosed information to a third-party AI platform. Although that case involved a client rather than a lawyer, it illustrates the exposure that sensitive legal information faces when entered into third-party AI systems.
California has already engaged with these questions through professional guidance. The State Bar of California updated its Practical Guidance for the Use of Generative Artificial Intelligence in the Practice of Law in May 2026, covering competence, confidentiality, supervision, client communications, and candor to courts. The California Supreme Court directed the State Bar in 2025 to consider incorporating AI-related principles into the Rules of Professional Conduct, with proposed changes under consideration that would clarify attorneys' responsibility for both what goes into AI systems and what comes out. SB 574 moves several of those concepts from guidance into statute.
Beyond Attorneys: Arbitrators and Judges
The bill reaches further than lawyers. Arbitrators would be prohibited from delegating any portion of their decision-making responsibilities to generative AI, and could not rely on AI-generated information outside the evidentiary record without first disclosing its use to the parties and, when practical, giving them an opportunity to respond. The measure also requires the California Judicial Council to revisit its standards governing judicial use of generative AI as the technology develops.
What Happens Next
Newsom has not yet acted on the bill, and his decision will be watched well beyond California. The state's market power has made it the de facto regulator of the AI industry — its SB 53 transparency law for frontier models was signed in 2025 — and a governor who has balanced AI promotion and AI caution in the past now decides whether professional AI use becomes a matter of statutory duty for the country's largest attorney population.
If signed, SB 574 would test a middle path that other states are likely to study: rather than banning AI in legal practice, California would bind its use to the profession's existing obligations — verification, candor, and confidentiality — with the attorney, not the tool, left holding the responsibility.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
