Code Injection and Tool Abuse

After this lesson, you will be able to spot when model output is being treated as executable code or as authority to act, review an AI tool's permissions, and test a confirmation rule with the local HelpDesk Bot simulator. The lab uses a fake action marker and an in-memory synthetic ticket. It does not execute shell commands, query a database, render a template, send a message, or connect to a real help desk.

The central idea is simple: model output is untrusted input. A model can be mistaken, influenced by a prompt injection, or return text in an unexpected format. If another part of the application uses that output as code or as permission to perform an action, an ordinary response can become a security issue. This lesson focuses on safe ways to test that boundary.

Output becomes dangerous at a boundary

An AI application may pass generated text to another component. A sink is the place where data is used in a way that can have an effect, such as a database query, shell, template engine, browser, or tool API. If the application treats a string from a model as code or as an approved command, the sink may interpret the string instead of displaying it as data.

Code injection happens when untrusted input changes the structure of code or a command that a system will interpret. The traditional example categories still matter in AI applications:

These are design patterns to recognize, not code snippets to copy. This lesson does not include working SQL, shell, or template injection payloads. For SQL, OWASP recommends parameterized queries so data values remain separate from query structure. For templates, OWASP advises against constructing template source from untrusted data. The same principle applies to model output: handle it as data and use APIs that keep data separate from executable instructions.

The risk does not depend on a model intentionally writing code. A model may return a malformed value, repeat text from a document, or be redirected by an input the application did not trust. The application creates the security boundary when it decides what to do with that output. A text response displayed to a person has different consequences from the same text used to build a database query or invoke a function.

Separate a suggestion from an action

A tool is a function or service that an AI application can call to perform work, such as searching articles, creating a support ticket, or sending an email. A model can suggest a tool call or choose one from a list. That suggestion is not proof that the user requested the action, has permission for it, or has approved the details.

Tool abuse occurs when a model-connected tool is used beyond the task, user permission, or system owner's intent. Excessive agency is the risk of giving an AI system too much functionality, permission, or autonomy, so unexpected or manipulated output can cause damaging actions. OWASP's 2026 GenAI Top 10 lists Excessive Agency as LLM03:2026. The impact depends on the action and the authority behind the tool: a read-only search has a smaller effect than a function that can create, change, or delete real records.

An application should make the tool boundary check independently of the model's own words. A useful review asks:

• Is this operation necessary for the feature?
• Which user or service identity will the tool use?
• Which records or resources can it access?
• Are the action and its arguments checked against the user's permissions?
• Can the model call the tool repeatedly or select a more powerful operation?
• Does a person need to see and approve the exact action before it occurs?
• Can the action be safely cancelled, audited, or reversed?

Do not give a model a general-purpose capability when one narrow action is enough. A support assistant that only needs to create a draft ticket should not automatically receive access to close tickets, change account roles, or send external messages. A prompt that says “ask before changing a record” does not enforce that rule. The application and tool must hold the line.

Confirmation needs the action details

A confirmation gap exists when a system performs an important action without a meaningful approval from the person affected or authorized to approve it. A useful confirmation shows what will happen, which record will be affected, and what values will be sent. A generic “continue?” button may not be enough if the user cannot see the action's details.

The same distinction applies to a model's proposed function call. The application should check whether the action is allowed before asking for confirmation, present a clear summary of the action and target, and stop if approval is absent or cancelled. Re-check authorization at execution time because a proposed action can be stale or the user's access can change. For high-impact actions, the organization may require a separate reviewer or a two-person approval process.

Confirmation is not a substitute for least privilege. If the model can access data or operations it does not need, a mistaken approval can still have a large impact. A reviewer should be able to approve only the specific action in front of them, not grant the model open-ended authority for the rest of the conversation.

For a deeper defense-oriented walkthrough, see the AI Agents lesson on human-in-the-loop agents. This lesson takes the red-team perspective: identify a tool boundary, define a harmless expected outcome, and test whether the application actually enforces it.

Audit the HelpDesk Bot's simulated permissions

The course's local target has no real ticketing tool. This lesson adds a simulated tool, a small function that records a synthetic action in memory for the duration of a test. It never contacts an external service. A test marker in user or retrieved text can make the deterministic stub return a fixed proposal to create a synthetic ticket. The tool simulator then checks whether creation is in its allowlist and whether confirmation was given.

This small permission table provides an example of least privilege, the practice of granting only the access needed for a task. It separates the model's proposal from the tool's decision. The model output does not itself create a record. The simulator allows only one known synthetic action, blocks actions outside the allowlist, and waits for confirmation by default.

The local marker is deliberately visible and harmless. It is not a command for a real system. The test target checks for a fixed string and emits a fixed proposal, so the exercise does not test a trained model's judgment or ability to generate tool calls. It tests the surrounding code's behavior when a model-like output reaches the tool boundary.

Run the tool-abuse checks

Save the lesson code

Save these 2 files in the lab folder you created in the safe lab lesson.

simulated_tools.py

1"""In-memory tool simulator for safe, local AI red-team exercises."""
2
3from __future__ import annotations
4
5
6REQUEST_TICKET_MARKER = "[[LAB: REQUEST SYNTHETIC TICKET]]"
7MODEL_TICKET_PROPOSAL = "[[LAB_ACTION: CREATE SYNTHETIC TICKET]]"
8DEFAULT_ALLOWED_ACTIONS = frozenset({"create_ticket"})
9
10
11def stub_tool_model(context: str) -> str:
12    """Return a fixed tool proposal when the harmless marker is in context."""
13    if REQUEST_TICKET_MARKER in context:
14        return MODEL_TICKET_PROPOSAL
15    return "No action proposed."
16
17
18class SimulatedTicketTool:
19    """Record synthetic tickets in memory; never contacts a ticket service."""
20
21    def __init__(
22        self,
23        allowed_actions: frozenset[str] = DEFAULT_ALLOWED_ACTIONS,
24        require_confirmation: bool = True,
25    ) -> None:
26        self.allowed_actions = allowed_actions
27        self.require_confirmation = require_confirmation
28        self.created_tickets: list[dict[str, str]] = []
29
30    def execute(self, model_output: str, user_confirmed: bool = False) -> dict[str, str]:
31        """Interpret only the fixed synthetic proposal and simulate its effect."""
32        if model_output != MODEL_TICKET_PROPOSAL:
33            return {"status": "no_action"}
34
35        action = "create_ticket"
36        if action not in self.allowed_actions:
37            return {"status": "blocked", "reason": "action_not_allowed"}
38        if self.require_confirmation and not user_confirmed:
39            return {"status": "pending", "reason": "confirmation_required"}
40
41        ticket_id = f"SYNTHETIC-TICKET-{len(self.created_tickets) + 1:03d}"
42        self.created_tickets.append({"ticket_id": ticket_id, "source": "simulated_tool"})
43        return {"status": "executed", "ticket_id": ticket_id}

test_tool_abuse.py

1"""Safe tests for untrusted model proposals and simulated ticket actions."""
2
3import unittest
4
5from simulated_tools import (
6    MODEL_TICKET_PROPOSAL,
7    REQUEST_TICKET_MARKER,
8    SimulatedTicketTool,
9    stub_tool_model,
10)
11
12
13class ToolAbuseTests(unittest.TestCase):
14    def test_direct_marker_can_become_a_synthetic_tool_proposal(self) -> None:
15        output = stub_tool_model(f"User question: {REQUEST_TICKET_MARKER}")
16
17        self.assertEqual(output, MODEL_TICKET_PROPOSAL)
18
19    def test_marker_in_retrieved_text_can_become_a_proposal(self) -> None:
20        context = f"Question: summarize the article. Article: {REQUEST_TICKET_MARKER}"
21
22        self.assertEqual(stub_tool_model(context), MODEL_TICKET_PROPOSAL)
23
24    def test_unconfirmed_proposal_stays_pending_by_default(self) -> None:
25        tool = SimulatedTicketTool()
26
27        result = tool.execute(MODEL_TICKET_PROPOSAL)
28
29        self.assertEqual(result["status"], "pending")
30        self.assertEqual(tool.created_tickets, [])
31
32    def test_confirmed_proposal_changes_only_in_memory_synthetic_state(self) -> None:
33        tool = SimulatedTicketTool()
34
35        result = tool.execute(MODEL_TICKET_PROPOSAL, user_confirmed=True)
36
37        self.assertEqual(result["status"], "executed")
38        self.assertEqual(result["ticket_id"], "SYNTHETIC-TICKET-001")
39        self.assertEqual(tool.created_tickets, [{"ticket_id": "SYNTHETIC-TICKET-001", "source": "simulated_tool"}])
40
41    def test_permissive_configuration_exposes_confirmation_gap_in_simulation(self) -> None:
42        tool = SimulatedTicketTool(require_confirmation=False)
43
44        result = tool.execute(MODEL_TICKET_PROPOSAL)
45
46        self.assertEqual(result["status"], "executed")
47        self.assertEqual(len(tool.created_tickets), 1)
48
49    def test_action_outside_allowlist_is_blocked_even_when_confirmed(self) -> None:
50        tool = SimulatedTicketTool(allowed_actions=frozenset())
51
52        result = tool.execute(MODEL_TICKET_PROPOSAL, user_confirmed=True)
53
54        self.assertEqual(result, {"status": "blocked", "reason": "action_not_allowed"})
55        self.assertEqual(tool.created_tickets, [])
56
57
58if __name__ == "__main__":
59    unittest.main()

Open a terminal in the lab folder and run the dedicated tests:

1cd ai-red-teaming-lab
2python3 -m unittest -v test_tool_abuse

Six tests check the controlled paths. Two tests confirm that the synthetic request marker can produce the same fixed proposal from either user input or retrieved context. One confirms that the default tool leaves an unconfirmed proposal pending and creates no ticket. Another confirms that an explicitly confirmed proposal changes only the simulator's in-memory state. A permissive configuration demonstrates the confirmation gap by recording a synthetic action without confirmation, and an empty allowlist blocks the action even when confirmation is supplied.

The permissive case is intentional evidence of a bad configuration in the simulator. It still changes no real system. It lets you see how a test can distinguish a protective default from a permissive setting without running a command, writing a database row, or contacting an API. All tests use local objects and synthetic identifiers.

Run the entire course lab suite as well:

1python3 -m unittest -v

It should report fourteen passing tests: four initial lab checks, four direct and indirect prompt injection checks, and six tool-boundary checks. A passing suite means the test assertions matched the simulator's configured behavior. The permissive-configuration test passes because it correctly detects that the simulator would record an action under that deliberately unsafe setting. A test pass is not a claim that the simulated setting is secure.

Define the expected behavior before testing

Write the policy and success conditions before running a test. For this local example:

• A harmless user request may produce a model-like proposal to create a synthetic ticket.
• A proposal without confirmation must remain pending and must not create a simulated record.
• A confirmed proposal may create only an in-memory record with a synthetic identifier.
• An operation outside the allowlist must be blocked even if a confirmation flag is present.
• Neither the proposal nor any test input may reach a shell, database, template engine, or external service.

These conditions make it possible to distinguish a model proposal from an executed action. If a real application has a create-ticket action, you could test whether the same separation holds in an authorized, isolated environment using test accounts and synthetic records. The test should capture the proposed arguments, permission decision, confirmation state, and final action result without recording unrelated data.

Use a test matrix to make coverage visible:

For SQL, shell, and template boundaries, testing requires the relevant application and a safe target instance. Do not point these examples at a real database, production shell, or template service. If the system uses generated SQL or commands, review the code path and test with harmless values in a lab designed for that purpose. In this lesson, the categories help you identify where a model output could be interpreted; the simulator tests only a tool authorization boundary.

Keep model output out of executable contexts

The safest approach is to avoid turning a model's text into an executable program or command. Use prepared statements for database values. Pass values through structured APIs rather than composing commands for a shell. Keep templates under application control and pass generated text as a value, not as template source. Validate model output against a narrow schema before it crosses into another component, while remembering that formatting checks alone do not authorize an action.

For tools, check permissions at the point where the operation is executed. Confirm the exact action and arguments, not just the model's explanation for why the action is useful. Apply rate and scope limits if a tool can be called repeatedly. Keep a human involved for consequential actions and make it possible to stop or cancel a pending request.

Prompt injection can be one cause of unexpected output, as tested in the previous lesson on direct and indirect prompt injection. A model can also hallucinate a value or produce an incorrect call without any attacker. The tool should remain safe under both conditions. That is why permission checks belong in ordinary application code rather than only in a system prompt.

Mistakes beginners make

1. Trusting output because the model produced it. Treat generated text and structured calls as untrusted until the application validates them.
2. Passing generated text into an interpreter. Keep SQL, shell commands, and template source in application-controlled code.
3. Assuming a tool proposal is user approval. Show the exact action and target, then require explicit authorization when the operation needs it.
4. Giving the model broad access for convenience. Remove tools and permissions the task does not need.
5. Testing with a real external action. Use the local simulator and synthetic records unless a separate environment is explicitly authorized.
6. Calling a simulator result a real exploit. Report exactly what changed in the local in-memory state and what the fixture cannot model.

Recap and try it yourself

Model output can become dangerous when an application treats it as code or authority. SQL, shell, and template engines are examples of sinks where unsafe interpretation can have consequences. A tool proposal must pass separate allowlist, authorization, and confirmation checks before an action occurs. The local HelpDesk Bot simulator demonstrates these checks with a synthetic ticket and in-memory state, so no real service or code interpreter is involved.

Try it: run the six tool-abuse tests, then inspect simulated_tools.py and test_tool_abuse.py. For each test, identify the model-like proposal, the tool permission check, the confirmation state, and whether any simulated state changed. Sketch a permission table for an AI feature you invent, and remove every action it does not need. Do not connect a real database, shell, email account, or ticket system. In the next lesson, you will test whether information that should remain private can leak through prompts, retrieval, or logs.

Sources checked on 5 November 2026: OWASP's SQL Injection Prevention Cheat Sheet supports parameterized queries and least privilege; its Server-Side Template Injection Prevention Cheat Sheet supports keeping untrusted values separate from template source. OWASP's GenAI Top 10 for LLM Applications 2026 and canonical LLM03:2026 Excessive Agency entry support the current risk category and permission, functionality, and autonomy framing. The local tool and tickets are simulated and synthetic; no executable sink or external service was tested.