Table of contents
ChatGPT is good enough that regulated teams have already adopted it. In fact, research shows that 78% of employees are now “bringing their own AI” to work, often using personal accounts to maintain productivity without formal company approval.[1] A clinician drafts a patient letter. A paralegal summarizes a deposition. A financial advisor runs numbers on a client scenario. The problem is not that AI is incapable of these things — it clearly is. The problem is that using a general-purpose AI tool in regulated work creates compliance exposures that don’t resolve themselves just because the tool is useful.
The gap is not primarily about the model. It’s about everything that has to exist around the model before regulated data can flow through it.
What a general-purpose AI tool doesn’t provide
When a healthcare organization evaluates whether they can use ChatGPT with patient data, the first question is usually “will they sign a BAA?” OpenAI does offer a Business Associate Agreement for certain paid plans. That’s a meaningful step. But a signed BAA is not the same as a compliant deployment, and most teams find out the hard way that the BAA they have doesn’t cover the way the tool is actually being used.
A Business Associate Agreement covers specific services, defined in the agreement itself.[2] What it covers is as important as whether it exists. If the BAA covers “ChatGPT Enterprise” but a team member uses the personal plan on the same device, the work done on the personal plan is outside the BAA. If the BAA covers the API but not the consumer interface, the same applies. Compliance requires not just a signature but a deployment where the BAA scope and the actual data flows match. Which ChatGPT products can be covered at all — and what coverage still leaves undone — is its own question, answered in is ChatGPT HIPAA-compliant?
Then there are the structural gaps that a BAA doesn’t address at all:
No organizational access controls. General-purpose AI tools don’t know who in your organization should or shouldn’t have access to a given conversation. There’s no concept of role-based access, no way to restrict what a front desk employee can ask versus what a treating physician can ask, and no enforcement mechanism at the query level. Regulated organizations have access policies. ChatGPT doesn’t enforce them.
No policy controls on what gets sent. In a regulated environment, some data should flow to the AI under a BAA with full context; some should be handled differently. The tool has no way to know which category a given prompt falls into, and the user pressing send has no systematic safeguard. The organization’s policies can’t be operationalized in a way the tool respects.
No audit trail that meets the regulatory bar. HIPAA requires covered entities and their business associates to implement audit controls that create records sufficient to reconstruct what happened, when, and who was responsible.[2] Conversation history in a consumer or even enterprise AI tool is not a compliance audit trail — it isn’t tamper-evident, it isn’t tied to verified user identities at the level a regulator expects, and it isn’t retained under any policy the organization actually controls.
No organizational retention and deletion. Your organization likely has a records retention policy — some PHI retained for seven years, other records for different periods depending on state law and document type. A general-purpose AI tool’s data retention is governed by the vendor’s policy, not yours. If a conversation containing PHI needs to be produced in discovery, or needs to be deleted under a patient’s request, the organization typically has no mechanism to do either.
Why legal and financial services face the same problem
Healthcare organizations face HIPAA, but the structural problem isn’t HIPAA-specific. It’s the mismatch between what a general-purpose tool is designed to do — help individuals — and what regulated organizations need — policy enforcement at the organizational level.
Legal teams are subject to confidentiality obligations that govern how client information is used, stored, and shared.[5] When a lawyer pastes client matter details into a general-purpose AI tool, the relevant question isn’t just “is this conversation private?” It’s whether the tool’s data handling is consistent with the lawyer’s professional obligations to the client, and whether the firm can demonstrate that if asked. The American Bar Association has addressed AI use in practice specifically in terms of competence, confidentiality, and supervision obligations — not just privacy in the consumer sense.
Financial services add their own layer. FINRA requires that certain communications and records be preserved in a form that can’t be altered and retrievable on demand.[3] An AI-generated draft that isn’t captured in the firm’s records system may constitute a recordkeeping violation regardless of whether the draft ever sent or whether the data was otherwise secure.
The pattern is the same across all three verticals: the AI tool works, but the organizational controls don’t exist around it.
What “compliant AI” actually requires
The compliance gap for regulated teams is not primarily about encryption in transit or access control to the vendor’s data centers. Those are baseline properties that most serious vendors offer. The gap is at the application layer, where the tool interacts with real clinical, legal, or financial work.
A deployment that meets regulated requirements typically needs to satisfy four things simultaneously:
A BAA that covers the actual data path. Not just the product name, but the inference chain — including whoever is processing the prompt at inference time and every sub-processor in between. If the AI vendor routes requests through an inference provider, that provider needs to be covered under a BAA that your arrangement relies on. The chain can’t have gaps.
Access controls at the organizational level. The ability to define who in the organization can do what with AI — not just at the account login level, but at the policy level. Who can submit PHI? Who can see what kinds of AI-generated content? Those controls need to be enforceable, not advisory.
Policy enforcement before data leaves your environment. If your organization’s policy is that certain information should be handled differently based on content type, that policy needs to be enforced technically, not just stated in a training document. Policies that depend on users making the right choice every time aren’t policies — they’re hopes.
An audit trail that survives scrutiny. Not application logs, but tamper-evident records tied to verified identities, retained under a policy the organization controls, exportable for regulatory review, and verifiable without trusting the vendor’s word.[4]
None of this is exotic. These are the same properties any other information system in a regulated environment needs. AI doesn’t get an exemption because the productivity gains are real.
The practical consequence
The teams that end up with the most difficult compliance situations are usually not the ones who tried to deploy AI recklessly. They’re the ones who used a reasonable tool, assumed the compliance piece would sort itself out, and discovered months later that the BAA didn’t cover what they thought it did, that they had no way to produce records in response to a request, or that their access logs couldn’t answer a basic question about who saw what.
Fixing that situation retroactively is much harder than getting the deployment right the first time. A general-purpose tool that was used outside its BAA scope, with PHI, is not a problem you can solve by stopping.
The question for any regulated team isn’t whether AI is useful. It is. The question is whether the deployment produces a state of affairs you can defend — to a patient whose records were discussed, to opposing counsel in discovery, to a regulator conducting an investigation. That bar requires more than a useful tool.
Sources
Microsoft and LinkedIn. 2024 Work Trend Index Annual Report (2024). Survey of 31,000 workers across 31 countries.
microsoft.com
U.S. Department of Health & Human Services. “Business Associates.” HHS.gov.
hhs.gov
FINRA. “Books and Records.” FINRA.org.
finra.org
U.S. Department of Health & Human Services. “HIPAA Enforcement Highlights.” HHS.gov.
hhs.gov
American Bar Association. “Formal Opinion 512: Generative Artificial Intelligence Tools” (2024).
americanbar.org