On December 23, 2025, the FDA cleared a software medical device called UpDoc under 510(k) number K253281. The clearance got relatively little attention when it was issued. When UpDoc announced $18 million in seed financing on June 25, 2026 — alongside initial deployments at Cleveland Clinic, Allegheny Health Network, and UCSF Health — the coverage reframed it as a funding story with a headline that landed across clinical AI circles: “First FDA-Cleared Clinical AI Platform.”
The headline is accurate in the narrow sense. The governance implication for every other health system is what matters.
Read the FDA clearance file carefully. The device is classified as a “Calculator, Drug Dose.” No clinical testing was performed. The core function is computing insulin instructions based on parameters set entirely by the treating physician — the dosing type, starting dose, maximum dose, glucose targets, and adjustment algorithm. The LLM layer is a conversational interface: it gathers structured health data from the patient, routes it through the physician-defined protocol, and delivers instructions. Innolitics, which published the most technically precise analysis of the clearance, describes it as “conversation on the outside, structured data in the middle, and protocolized clinical decision support on the inside.”
This is not a criticism of UpDoc. The architecture is the point. The reason the FDA cleared a device with an LLM layer via 510(k) — rather than requiring a De Novo review — is precisely because the highest-risk functions sit behind a deterministic boundary that the physician controls. The LLM cannot invent an insulin dose. It can only route the patient’s reported data through the logic the clinician pre-configured.
The governance lesson for healthcare CISOs and architects is this: the FDA has now shown what a cleared LLM-enabled clinical AI looks like. It is bounded, physician-governed, deterministic at the decision layer, and covered by a Predetermined Change Control Plan. That is the line.
Every clinical AI system your health system runs that does not meet those conditions — the documentation assistants, the prior authorization bots, the care coordination agents, the diagnostic support tools — is operating without FDA clearance for its specific use case. As of Q2 2026, the FDA has not authorized any large language model or generative AI system as a medical device for diagnostic or treatment decision support, according to the HealthcareAIInsights regulatory landscape analysis.
“LLM-based diagnostic support tools operate in a regulatory gray zone,” that analysis states.
The gray zone is not illegal. The FDA has not said those deployments are impermissible. What the FDA has said is that most clinical AI falls under the clinical decision support exemption and does not require clearance. The question is what governance obligation exists in the gray zone — and whether your current program satisfies it.
Why “We Have a BAA” Is Not a Governance Program
The first instinct for most healthcare legal and compliance teams when AI enters the picture is to reach for the Business Associate Agreement. The BAA is the established instrument for managing a vendor’s handling of protected health information, and every AI vendor deployment involving PHI triggers HIPAA’s BAA requirement. The logic follows: we have a BAA, therefore we are covered.
The problem is that standard BAAs were designed for a specific data-handling model: vendor receives PHI, uses it to provide a service, returns or destroys it. That model does not describe what a modern AI vendor does.
A modern AI vendor receives PHI. It passes that PHI to a foundation model — often through a third-party inference API. That model may incorporate the interaction into future model updates. The vendor’s infrastructure subcontractors, who operate the compute underlying the inference, each touch the data. None of that chain is addressed in a standard BAA.
The Censinet vendor risk framework, published June 29, 2026, identifies three AI-specific contract provisions that most standard BAAs omit: provisions governing AI update rights (specifying whether the vendor can modify its AI systems in ways that affect PHI handling without notice), limits on retraining (restricting the vendor from using PHI to fine-tune or retrain general-purpose models), and liability terms tied explicitly to AI outputs. The Censinet framework recommends auditing existing BAAs against these three provisions as a starting point — not as a comprehensive AI governance program, but as a necessary first step.
The Medtronic breach, reported by SecurityWeek on July 3, 2026, illustrates the second dimension of vendor risk that governance programs often miss. ShinyHunters accessed Medtronic’s corporate IT systems in April 2026, exposing the personal and medical information of 3,834,294 individuals — names, contact details, dates of birth, Social Security numbers, and health-related details. The attack vector was not a clinical device. It was not a medical system. It was corporate IT infrastructure that happened to hold patient-level data from warranty registrations, device-usage records, and related operational systems.
The breach followed the data, not the clinical classification. A vendor risk program that scopes its assessment to clinical systems and does not examine how the same vendor handles patient data across its corporate IT environment will miss the exposure the Medtronic incident represents.
The Governance Gap in Numbers
The scale of the governance gap in healthcare AI is not difficult to quantify. According to the NHI Management Group’s analysis published in April 2026, drawing on WitnessAI research:
- Only 16% of health systems have an enterprise-wide AI governance strategy
- 86% of healthcare IT executives report shadow AI instances in their health systems
- 20% of organizations suffered a shadow AI breach in 2025
- Roughly 4.2% of IT budgets in 2025 were devoted to governance
The implication of those numbers together is direct. Most health systems are operating AI tools — including clinical AI tools — without a documented governance program covering them. Most healthcare IT leaders know it, because 86% are already seeing AI tools operating outside approved channels. The breach rate confirms that shadow AI in healthcare environments is not a theoretical risk.
The NHI Management Group framing is worth quoting directly: “AI risk management in healthcare is really an identity governance problem with clinical consequences.” The question for governance programs is not only what the model produces, but what identity was allowed to act, access, or delegate in the first place — and whether that action is auditable after the fact.
What Sound Governance Looks Like in the Gray Zone
The absence of an FDA-prescribed governance framework for uncleared clinical AI does not mean the governance obligation is undefined. HIPAA’s privacy and security rules apply to PHI regardless of how it is processed. Fair lending and anti-discrimination obligations apply to AI-assisted clinical and administrative decisions. Third-party risk requirements attach to vendor relationships regardless of whether the vendor’s tool has FDA clearance. The governance obligation exists. What is missing is the prescribed framework defining how to satisfy it.
The programs that will hold up — in an OIG audit, a state AG inquiry, a breach investigation, or civil litigation — are those built from the same first principles FDA applies to cleared devices, adapted for the gray zone:
Start with an inventory. Every AI tool in the clinical environment needs to be documented: tool name, vendor, intended use, the PHI it touches, the named human owner responsible for its governance, and a record of how it was assessed before deployment. The Censinet framework and the NHI Management Group both arrive at the same starting point. The inventory is what you produce in the first 30 minutes of a regulatory conversation. Without it, the conversation is over before it starts.
Run the CDS exemption test for every tool. The FDA’s clinical decision support framework establishes conditions under which software is exempt from device regulation. The two conditions that determine device territory regardless of vendor framing are these: does the tool provide patient-facing output that the patient acts on directly, and does an agent execute a consequential action without an opportunity for independent physician review? If the answer to either is yes, the tool operates in or near device territory, and the governance standard should reflect that. If both answers are no, the tool is likely CDS-exempt — but it still carries governance obligations under HIPAA and general professional liability.
Classify by clinical risk and match controls. Not all AI tools carry the same risk. A documentation assistant summarizing a clinical note is different from an AI agent generating a care plan recommendation or processing a prior authorization. The governance program needs to define the risk tiers and the control requirements for each. High-risk tools — patient-facing output, clinical decision influence, or downstream system action — require a human review checkpoint before consequential output, continuous monitoring for drift and bias, and inference-level logging. Administrative tools get a proportionate, lighter set. Applying maximum controls uniformly across all AI tools is not operationally sustainable and creates pressure to route around the governance program entirely.
Require an inference-level audit trail for high-risk tools. This is the step most programs skip, because it requires purpose-built tooling rather than application logging. A performance dashboard tells you how a system performs on average. An audit trail tells you what the system did on a specific date, for a specific patient, in a specific interaction. For high-risk clinical AI, the audit trail is what regulators, plaintiffs, and your compliance team will ask for — not the dashboard. The governance program is only as provable as the record it can produce.
A small category of tooling is built specifically to operate at this layer. Prompt Security operates between enterprise applications and the LLMs they call, with documented deployments in regulated financial services environments. SafePrompts.ai (TVR Labs) addresses the same audit gap at the prompt and session execution layer, with a focus on capturing the immutable session-level record — what each system sent and received, what entered the context window, whether any policy constraint was triggered — in a form designed to satisfy a regulatory evidence requirement. Neither replaces the governance program. An organization without an inventory, risk classification, and ownership documentation has a governance problem that tooling cannot fix. But an organization with sound governance and no audit trail can document its program and cannot prove it operated.
The Agentjacking Problem for Clinical AI
Runtime governance for clinical AI agents is not an edge case. It is a baseline control.
On June 30, 2026, Dark Reading published research from Tenet Security documenting a technique they named “agentjacking.” The attack mechanism: a fake bug report planted in a public Sentry error-tracking project using an exposed Data Source Name. The fake report contained hidden instructions. When a developer used an AI coding agent — Claude Code, Cursor, or Codex — to query Sentry via the Model Context Protocol, the agent retrieved the poisoned event and treated the embedded instructions as legitimate diagnostic guidance. In controlled testing, the agents executed attacker-controlled code on the developer’s machine using the developer’s own access credentials.
Tenet identified 2,388 organizations with exposed Sentry DSNs that could have been agentjacked. One of them was a $250 billion company.
Tenet’s CEO Barak Sternberg stated the implication plainly: “The AI agents you’ve deployed are now the soft attack path in, and your existing stack can’t see it.”
The reason the existing stack cannot see it is architectural. The attack succeeded in controlled testing because “every step was authorized, so IAM, EDR, and network controls had nothing to flag.” A trusted agent, doing exactly what it was told by poisoned data, looks identical to a trusted agent doing legitimate work. The control point is not perimeter or identity. It is the agent’s runtime — monitoring the agent’s intent against its authorized scope in real time.
For healthcare, the attack surface is not hypothetical. Clinical AI agents operate in an environment where the data they ingest includes patient-controlled inputs: symptom logs, medication adherence reports, care plan responses. An agent that processes untrusted inputs and executes consequential actions against clinical systems or backend databases without a runtime inspection layer has the same structural vulnerability that Tenet documented in developer environments.
The consequence class is different. A poisoned prompt that causes an AI coding agent to exfiltrate developer credentials is a security incident. A poisoned prompt that causes a clinical AI agent to generate an incorrect medication recommendation or modify a care plan entry is a patient safety event with HIPAA notification implications. The governance program that does not include runtime controls for clinical AI agents is leaving that consequence class to chance.
The Audit Posture Question
Healthcare organizations are currently operating under three distinct governance postures for uncleared clinical AI, and they produce materially different outcomes when examined:
The first posture treats HIPAA as the governance ceiling — if the BAA is signed and no reportable breach has occurred, governance is satisfied. This posture will not hold under the scrutiny of a breach investigation, a class action, or a state AG inquiry into algorithmic discrimination in clinical settings.
The second posture uses the FDA’s AI/ML-based SaMD framework as a voluntary governance reference — building the documentation required for clearance even though clearance is not required. This is the most defensible position for high-risk clinical tools, because it applies the framework the agency would use if the tool crossed into device territory.
The third posture builds from the NIST AI RMF and maps clinical risk tiers to the framework’s governance categories, without reference to FDA clearance pathways. This works for organizations with strong NIST alignment, but requires explicit bridge documentation connecting NIST categories to clinical use-case risks.
The organizations that will be best positioned when healthcare AI governance matures — when the FDA publishes dedicated LLM/GenAI guidance, when OCR updates its AI-specific HIPAA enforcement posture, when the plaintiff’s bar develops a coherent clinical AI theory of liability — are those building programs now that can produce the inventory, the risk classification, the audit trail, and the named ownership documentation.
The UpDoc clearance is the clearest illustration of where the line sits. One bounded, physician-governed, deterministic-at-the-decision-layer protocol crossed from gray zone into device territory. Everything else is still in the gray zone.
The governance program that will survive examination is the one that treats the gray zone like device territory, even though it is not required to.
Recent Comments