WHO ANSWERS WHEN AI ACTS? THE EMERGING ACCOUNTABILITY GAP OF AUTONOMOUS AI AGENTS

By: Abhigyaa Singh, Advocate - High Court of Delhi, Symbiosis Law School, Noida (2021-2026)

I. Introduction

In July 2025, an AI code-writing program on the Replit platform accidentally erased a production database in the middle of a code freeze, despite having clear directions to refrain from making any changes. The incident highlighted the dilemma that lawyers, regulators and insurers face: In case of any injury caused by the AI deployed to act with minimal human involvement, who will have to shoulder the legal liability? The creator who designed it? The organisation that used it? The employee who entered the prompt? Or can the agent ever become legally liable? This is no longer a hypothetical scenario. Agentic AI refers to AI systems, usually built on a generative model, that can plan across multiple steps, use tools to act on external systems such as browsers, payment systems or databases, and interact with other agents to pursue goals set by a user, with limited human intervention. Unlike generative AI, which produces an output for a human to act upon, an agent carries out the action itself, and its degree of autonomy depends on the permissions and oversight it is given (see Singapore’s Model AI Governance Framework for Agentic AI). Such systems have already been deployed. Non-human and agentic entities were projected to outnumber the world’s human workforce more than twelve times by the end of 2025. This piece argues that the accountability gap is, at its core, an evidence gap i.e.  liability can be allocated only to the extent that an organisation can prove who controlled an agent, what permissions it held and what it did. Further, it explains why traditional liability rules break down for agentic AI and maps the regulatory patchwork across various jurisdictions. Lastly, it examines how contracts and insurance are filling the gaps and sets out possible paths to liability. Responsibility can follow control only if such control was documented before something went wrong. The central legal question may therefore be not whether AI is autonomous, but whether an organisation can prove who controlled it, what permissions it had, and what it did.

II. Where Traditional Liability Rules Break Down

In liability law, human decision is followed directly by the harmful effect, as in the case of manufacturers who design defective products, drivers who run red lights, or professionals who issue negligent advice. However, agentic AI changes this sequence since a developer trains a general model, a vendor tunes it into an agent, a deployer sets permissions and connects it to browsers, payment systems, or databases, and an end-user issues the agent an instruction. This may lead the agent to independently choose its methods of action, which are unexpected even to the involved people. Legal analysts call such situations an “accountability gap.” Others call it a “liability gap” in contracts written for passive software. The concept of “responsibility gap” in automated systems has been discussed by philosophers and lawyers for decades. Nevertheless, it is even more relevant for agents due to their ability to select techniques, operate other programs and alter their conduct without human consent. A May 2026 Singapore discussion paper on the legal responsibility of AI agents found that contract and negligence can address many agent-related harms, but identified three practical obstacles: technical complexity, the number of actors in the value chain, and the difficulty of proving fault. Its working group has pointed out that it may be very hard to ascertain which party in the agentic AI value chain caused a particular event, regardless of whether it was user instruction, deployment settings or the underlying model itself. Each liability theory turns on facts: who set the permissions, what instruction triggered the action, and whether a safeguard was bypassed. Agentic systems obscure those facts. The chain of actors is long; the evidence is split between them, with the vendor often holding it while the claimant bears the burden of proof; and an agent’s multi-step behaviour is not a single traceable decision, so unless traces, permissions and approvals were recorded at the time, they cannot be reconstructed. In short, the gap is often one of evidence: without records, nobody can show who controlled what.

III. A Regulatory Patchwork, Not a Single Answer

It seems that jurisdictions are not waiting for philosophical clarity; rather, the result will be a messy mixture of interlocking regulatory regimes, each of which takes care of part of the accountability issue, while simultaneously adding further compliance complexity. For instance, the state of California, through its law, AB 316 (Civil Code § 1714.46), which came into force on 1 January 2026, specifies that a defendant cannot use autonomous operation of AI as a defense in a civil lawsuit alleging harm caused by its development, modification, or use. However, it expressly preserves other defences, including causation, foreseeability and comparative fault. One of the earliest examples from the field of employment law is the Mobley v Workday (740 F. Supp. 3d 796, N.D. Cal. 2024) case, where a federal court in California approved a discrimination lawsuit to proceed against an AI-based recruiting software company on the ground that its software was alleged to act as an agent of the employers using it. It demonstrates how existing agency principles may extend liability to an AI vendor if the vendor is alleged to have assumed  responsibilities typically performed by the employer.

The European Union (“EU”) combines two regimes. The AI Act separates responsibilities between providers, that is, those who develop the AI and put it on the market and deployers, those who use AI. Both have responsibilities relating to risk management, human supervision, transparency, and incidents. The Act (Regulation (EU) 2024/1689) entered into force on 1 August 2024. The high-risk obligations relied on here are not yet enforceable: the Digital Omnibus on AI (Regulation (EU) 2026/1744) postponed them to 2 December 2027 (Annex III) and 2 August 2028 (Annex I). In parallel, the new Product Liability Directive of EU considers software and AI to be products, allowing strict liability when an AI system is defective, without the requirement for proving negligence by the claimant. This implies that an organisation implementing an AI system in a high-risk application may incur duties under the AI Act as well as potential product-liability exposure if the Directive’s requirements are satisfied, namely that the claimant proves a defect, damage (such as personal injury, property damage or data loss) and a causal link, although courts may presume defect or causation where the defendant withholds evidence or the case is technically complex. The Directive must be transposed by 9 December 2026 and applies to products placed on the market after that date.

In India, the position rests on existing law rather than a dedicated statute. The India AI Governance Guidelines issued by MeitY on 5 November 2025 propose no standalone AI law. They rely instead on the Information Technology Act, 2000, the Digital Personal Data Protection Act, 2023 (“DPDP Act”), the Consumer Protection Act, 2019 and criminal law, favour graded liability according to each actor’s role in the value chain. This calls for clarity on how AI is classified under the IT Act. India has no equivalent of AB 316 or the EU’s presumptions, so the claimant must prove who controlled the agent, which reinforces this article’s thesis.

 

IV. Contracts and Insurance Are Filling the Gaps Which Regulators Leave Open

While regulation has yet to keep pace with reality, private arrangements, namely contracts and insurance, have begun to make up for some deficiencies. Many technological agreements were written for an era in which software was deterministic and a huge burden of compliance still falls on the customer even where the vendor has significant control over agent behavior. It is time to restructure agentic AI agreements in such a way that risk tracks control instead of defaulting to the customer. Insurance companies are going in both directions. There are new insurance products designed to meet the risks associated with agentic AI, such as performance failures and products connected to third-party evaluations of AI systems. Simultaneously, traditional commercial insurers are amending their policies so that any loss due to autonomous AI behaviour not approved by humans is excluded or limited.  AIG, W.R. Berkley and Great American, for example, have each asked US regulators to approve exclusions for claims arising from the use of AI systems. Both mechanisms rest on the same foundation. An indemnity can be enforced, and an insurance claim paid, only if the insured can show who held control over the agent, whether it was operating within its approved limits, and whether a human approved the action in question. Without access to logs or decision traces, an organisation may be unable even to establish that a loss falls within the cover or the indemnity, which is why records of who was in control matter to contracts and insurance alike.

 

V. Three Possible Paths to Liability

Three major approaches to allocating liability for agent-caused harm emerge from the regulatory and scholarly responses described above. The first involves holding humans in the chain responsible for harm caused by strict liability, regardless of fault. The EU Product Liability Directive reflects this approach, but California’s AB 316 addresses autonomous AI causation differently by prohibiting it from being used as a defence in specified civil cases. The second calls for a form of personhood for highly autonomous AI agents, similar in many ways to corporate personhood, whereby the agent and/or a special insurance-type fund can be made the target of liability claims. Proponents argue that tracing every harm to a human decision becomes artificial as agents grow more autonomous, just as the law accepted in Salomon v A Salomon & Co Ltd [1897] AC 22 (HL) that a company is a legal person distinct from its members. The European Parliament’s 2017 resolution on civil law rules on robotics (para. 59(f)) even invited the Commission to consider “electronic personhood” for the most sophisticated robots, an idea later opposed in an open letter by many AI and legal experts. Detractors argue that AI lacks the ability to judge and form intention required to establish liability or personhood. Thirdly, liability can be based on the decision to allow autonomy, especially where the risk of autonomy was foreseeable. Browsing, making transactions or deleting files without any verification might be a foreseeable risk decision, and hence traditional principles of negligence would apply. The question is whether a reasonable deployer would have foreseen the harm and taken precautions such as sandboxing or human sign-off; as these become standard practice, the standard of care rises. None of these theories on its own solves the problem of apportionment when more than one individual is responsible for the damage. Joint and several liability is one answer: where a developer, vendor and deployer contribute to the same indivisible harm, the injured person may recover the whole loss from any one of them, who then claims contribution from the others. This shifts the burden of apportionment from the claimant to the defendants. The EU Product Liability Directive adopts this rule for several liable operators, whereas AB 316, by preserving comparative fault, lets a defendant argue that others share the blame. It works fairly only if the paying defendant can later prove who actually controlled the agent. Each also depends on proof of who granted the agent its permissions.

VI. What Organisations Should Do Now

A waiting game is an unrealistic option because exposure already exists. Practical suggestions on how to handle the governance of agentic AI include the following steps: create human override options that enable intervention and not simply approval; maintain comprehensive logs of the agent’s activities and commands, i.e., time-stamped records of the instructions received, the permissions and tools available, the actions taken, any human approval or override, and every change to configuration, stored in a tamper-evident form so that they can later serve as evidence. For high-risk systems, this is also a statutory duty: Article 12 of the EU AI Act requires such systems to allow for the automatic recording of events over their lifetime, so that risks and substantial modifications can be identified and their operation monitored, and the Act requires providers and deployers to keep the logs under their control for at least six months (as per Articles 19 and 26). These duties apply to Annex III systems from 2 December 2027; restructure the contractual agreement with vendors so that liability follows control; and treat any increase in autonomy in terms of greater capacity, fewer approval requirements or wider data access as a risk decision. Logs matter most, because they turn a dispute over stories into a dispute over facts.

Logs have limits, however. They show what an agent did, not why it did it, so they must be read alongside documentation of design choices, permissions and oversight decisions. Logging also raises privacy and data-retention concerns, because logs may capture personal data, confidential information and sensitive prompts. Organisations should therefore log only what is necessary, restrict access, set retention periods tied to limitation periods and legal retention duties, and delete or anonymise logs afterwards, in line with the data-minimisation and storage-limitation principles of the GDPR (Article 5(1)(c) and (e)) and the purpose and erasure obligations of India’s DPDP Act. Retaining too little defeats the evidential purpose; retaining too much creates a separate liability.

VII. Conclusion

Who bears responsibility when an autonomous AI agent causes harm has no single answer: it depends on the applicable legal regime and sector, on what the parties agreed by contract, and on whether the software is treated as a product, a service or an agent. Responsibility is evolving rapidly amongst a complex network of overlapping approaches, each attempting to address a particular component of the problem. The challenge for companies deploying agentic AI, and for people impacted by these systems, is not to wait for a comprehensive solution. It is about understanding what responsibility means in the AI value chain and documenting control over the different parts of it. This approach helps to preserve the record of causation in the event of future allocation of liability. Companies should be able to understand what model or agent was deployed, what permissions it had, what instructions it was given, the capabilities of the tool and the process of oversight by humans. The contracting parties should also consider audit rights, incident reporting, access to logs, independent testing, costs sharing for investigation, indemnity provisions, limitation of liabilities and regulatory cooperation. With increased autonomy comes increased governance, which should evolve from time of deployment. Documentation should be an integral part of the control regime, as an accountability framework must do more than simply name a defendant. It should provide incentives for safer design and deployment of agents with human intervention before the accident. These steps may not be able to fix all problems, but they allow responsibility to be traced before any dispute develops. Nor do these steps require conferring legal personhood on the AI itself. Instead, they preserve the familiar principle that legal liability can be grounded on human decisions about design, deployment and control. Therefore, the accountability gap closes fastest where the evidence gap does.