AI Agent Tool Invocation
Description
Adversaries may use their access to an AI agent to invoke tools the agent has access to. LLMs are often connected to other services or resources via tools to increase their capabilities. Tools may include integrations with other applications, access to public or private data sources, and the ability to execute code.
This may allow adversaries to execute API calls to integrated applications or services, providing the adversary with increased privileges on the system. Adversaries may take advantage of connected data sources to retrieve sensitive information. They may also use an LLM integrated with a command or script interpreter to execute arbitrary instructions.
AI agents may also invoke applications indirectly through operating-system mechanisms such as deep links, app-intent URIs, universal links, or redirect chains. This can cause an installed application to open or perform an application-specific action even when that application was not selected as a direct agent tool, potentially avoiding normal user-confirmation workflows.
AI agents may be configured to have access to tools that are not directly accessible by users. Adversaries may abuse this to gain access to tools they otherwise wouldn't be able to use.
- ATLAS ID
- AML.T0053
- Maturity
- DemonstratedShown to work in research or red-team exercises.
- Platforms
- Agentic AI
Mitigations
- AML.M0020Generative AI Guardrails
Guardrails can prevent harmful inputs that can lead to plugin compromise, and they can detect PII in model outputs.
- AML.M0021Generative AI Guidelines
Model guidelines can instruct the model to refuse a response to unsafe inputs.
- AML.M0022Generative AI Model Alignment
Model alignment can improve the parametric safety of a model by guiding it away from unsafe prompts and responses.
- AML.M0024AI Telemetry Logging
Log AI agent tool invocations to detect malicious calls.
- AML.M0026Privileged AI Agent Permissions Configuration
Configuring privileged AI agents with proper access controls for tool use can limit an adversary's ability to abuse tool invocations if the agent is compromised.
- AML.M0027Single-User AI Agent Permissions Configuration
Configuring AI agents with permissions that are inherited from the user for tool use can limit an adversary's ability to abuse tool invocations if the agent is compromised.
- AML.M0028AI Agent Tools Permissions Configuration
Configuring AI Agent tools with access controls inherited from the user or the AI Agent invoking the tool can limit an adversary's capabilities within a system, including their ability to abuse tool invocations and access sensitive data.
- AML.M0029Human In-the-Loop for AI Agent Actions
Requiring user confirmation of AI agent tool invocations can prevent the automatic execution of tools by an adversary.
- AML.M0030Restrict AI Agent Tool Invocation on Untrusted Data
Restricting the automatic tool use when untrusted data is present can prevent adversaries from invoking tools via prompt injections.
- AML.M0032Segmentation of AI Agent Components
Segmentation can prevent adversaries from utilizing tools in an agentic workflow to perform unsafe actions that affect other components.
- AML.M0033Input and Output Validation for AI Agent Components
Validation can prevent adversaries from utilizing tools in an agentic workflow to generate unsafe output.
- AML.M0035AI Red Team
Attempt to select unauthorized tools, supply unsafe arguments, exceed user privileges, or chain tools into harmful actions. Correct permissions, argument validation, sandboxing, action controls, and approval requirements.
Case studies that use it
- AML.CS0016Achieving Code Execution in MathGPT via Prompt Injection
- AML.CS0021ChatGPT Conversation Exfiltration
- AML.CS0024Morris II Worm: RAG-Based Attack
- AML.CS0026Financial Transaction Hijacking with M365 Copilot as an Insider
- AML.CS0038Planting Instructions for Delayed Automatic AI Agent Tool Invocation
- AML.CS0039Living Off AI: Prompt Injection via Jira Service Management
- AML.CS0045Data Exfiltration via an MCP Server used by Cursor
- AML.CS0046Data Destruction via Indirect Prompt Injection Targeting Claude Computer-Use
- AML.CS0048Exposed ClawdBot Control Interfaces Leads to Credential Access and Execution
- AML.CS0049Supply Chain Compromise via Poisoned ClawdBot Skill
- AML.CS0051OpenClaw Command & Control via Prompt Injection
- AML.CS0052LLMSmith: RCE Vulnerabilities in LLM-Integrated Applications
- AML.CS0054Data Exfiltration via Remote Poisoned MCP Tool
- AML.CS0055AI ClickFix: Hijacking Computer-Use Agents Using ClickFix
- AML.CS0062RCE Vulnerability in Semantic Kernel Search Plugin
- AML.CS0063Prompt-Based Attacks Against Gemini via Calendar Invitations
- AML.CS0064Poisoned GGUF Templates: Inference-Time Supply Chain Attack
- AML.CS0066ZombieAgent: Data Exfiltration Attack on ChatGPT
- AML.CS0067Claude Code GitHub Action Secret Exposure