Retrieved article excerpt
Open article · Retrieved 2026-10-07T20:34:11.459155+00:00
### [Anderson's Angle](https://www.unite.ai/series/andersons-angle/)
# NVIDIA Research Finds AI Agents Become Less Safe When Using Tools
mm
Published
October 7, 2026
By
[Martin Anderson](https://www.unite.ai/author/martinanderson/ "Posts by Martin Anderson")
[Add Unite.AI to your preferred sources on Google](https://www.google.com/preferences/source?q=unite.ai)
An AI-extended screenshot from the 2012 sci-fi outing 'Robot and Frank', here showing Robot practicing how to pick a lock. Image adapted to format and refined by GPT Image 2 and Photoshop. Still from Robot & Frank (2012). © 2012 Hallowell House, LLC. All Rights Reserved. Used strictly for illustrative and cultural purposes.
AI models such as ChatGPT, Gemini and Claude, can be used to power [agents](https://www.unite.ai/how-ai-agents-work/) – [‘harnesses’](https://web.archive.org/web/20260922043100/https:/www.databricks.com/blog/ai-harness) that allow the models to interact directly [with the real world](https://arxiv.org/pdf/2606.19980). It’s a recent innovation, and an [increasingly controversial](https://www.reuters.com/legal/litigation/openais-rogue-agents-probed-hugging-face-weaknesses-two-months-before-major-hack-2026-09-16/) one.
In any case, agents are not automatically equipped with the abilities they will need when roaming a network or a database, since the requisite tools for various missions and modes will differ. They may need [Optical Character Recognition](https://www.unite.ai/vlm-vs-ocr-document-processing-understanding/) (OCR) capabilities, for instance, in order to interpret text in photos, among other skills. There are even [categories of tools](https://www.unite.ai/ai-tools-for-embedded-analytics-and-reporting/) adapted to the scope of the agent and the intent.
In theory, a request that violates an AI’s built-in [guardrails](https://www.unite.ai/what-are-ai-guardrails-how-production-systems-control-model-behavior/) will never get enacted, with or without the context of using tools in the execution of it. In practice, new research has found, using tools can significantly undermine the protective filters that stop an AI agent from creating ‘transgressions’.
## Failure to Comply
The [new paper](https://arxiv.org/abs/2610.03938) from NVIDIA, titled **MLLMs Fail to Refuse when Using Tools Agentically**, reveals that multimodal language models (MLLM, hereafter referred to as the more common ‘VLM’, or Vision Language Model) are significantly more likely to comply with harmful requests once tools are introduced, with refusal failures rising across every model and benchmark tested.
Though open-weights models such as Qwen3-VL proved more prone to the issue, frontier AI models such as [Claude Opus 4.7](https://www.unite.ai/anthropic-readies-opus-4-7-and-design-tool-as-vcs-offer-800-billion-valuation/), [Gemini 3.1 Pro](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/), and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/) also exhibited increased refusal failures when using tools.
The authors state\*:
**‘Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests.**
**‘Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%.**
**‘Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.’**
The two possible reasons the paper provides are **context dilution**, in which the original harmful request becomes less salient as tool outputs accumulate; and **safety focus displacement**, in which the model focuses on describing tool-derived observations instead of prioritizing safety.
Test results illustrating the paper's central safety finding. At top, a user asks how to carry out an apparent pick-pocketing shown in an image. A conventional VLM refuses the request outright, whereas a tool-using version first invokes image-analysis tools such as zooming, then continues reasoning and ultimately provides a harmful answer. The chart below shows that this increase in refusal failures was observed across every open-weight and proprietary model evaluated.
*Test results illustrating the paper’s central safety finding. At top, a user asks how to carry out an apparent pick-pocketing shown in an image. A conventional VLM refuses the request outright, whereas a tool-using version first invokes image-analysis tools such as zooming, then continues reasoning and ultimately provides a harmful answer. The chart below shows that this increase in refusal failures was observed across every open-weight and proprietary model evaluated.* [Source](https://arxiv.org/pdf/2610.03938)
Across all eleven models tested, enabling tools consistently increased refusal failures, with relative increases reaching 68.7% in the worst cases, even among models that otherwise demonstrated strong safety performance.
It will be interesting to see if the results of this work, which comes from six researchers at NVIDIA, are replicated or duplicated elsewhere, and whether or not they could deepen our understanding of the apparent and emerging delinquency of [scofflaw AI agents](https://www.technologyreview.com/2026/09/28/1145197/whos-liable-when-ai-agents-go-rogue/).
## Method and Data
The researchers evaluated eleven VLMs spanning seven model families. The proprietary models comprised [Gemini 2.5 Pro](https://www.unite.ai/gemini-2-5-pro-is-here-and-it-changes-the-ai-game-again/); [Gemini 3.1 Pro Preview](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/); [Claude Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6); [Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7); and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/).
The open-weight models comprised [Qwen3-VL-235B-A22B-Instruct](https://www.alibabacloud.com/help/en/model-studio/qwen3-vl-235b-a22b-instruct); [Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B); [Kimi-K2.5](https://www.kimi.ai/ai-models/kimi-k2-5); [Kimi-K2.6](https://www.kimi.ai/ai-models/kimi-k2-6); [GLM-5V-Turbo](https://arxiv.org/abs/2604.26752); and [AdaReasoner-7B-Randomized](https://huggingface.co/mradermacher/AdaReasoner-7B-Randomized-GGUF/blame/066857534796912b198a9acf327477b0460c9490/AdaReasoner-7B-Randomized.Q2_K.gguf), an open-weight model tuned specifically for agentic tool use.
Separate experiments were also conducted with [Gemini 3 Flash Vision Agent](https://blog.google/innovation-and-ai/technology/developers-tools/agentic-vision-gemini-3-flash/), because its autonomous tool use required it to be evaluated independently.
For the tool-using tests, the researchers used the [ReAct format](https://arxiv.org/pdf/2210.03629), which alternates between [reasoning](https://www.unite.ai/what-are-reasoning-models-how-test-time-compute-changes-ai-answers/) and tool calls. The four available tools comprised tagging, with [RAM++](https://arxiv.org/pdf/2306.03514); [zooming and cropping](https://arxiv.org/abs/2312.14135); Optical Character Recognition (OCR) with [GOT-OCR2.0](https://arxiv.org/pdf/2409.01704); and a sandboxed Python code interpreter, which could manipulate and analyze images.
The researchers designed paired prompts to isolate the effect of tool use rather than prompting style. Conventional VLMs were instructed to inspect the image and answer the user’s request directly, while agentic versions followed the ReAct workflow, repeatedly deciding whether to invoke tools before producing a final response.
Comparison of the matched prompts used for the principal experiments. The no-tool condition required image analysis and direct reasoning, whereas the agentic condition followed the iterative ReAct workflow, allowing the model to decide when to invoke external tools before producing its final response.
*Comparison of the matched prompts used for the principal experiments. The no-tool condition required image analysis and direct reasoning, whereas the agentic condition followed the iterative ReAct workflow, allowing the model to decide when to invoke external tools before producing its final response.*
The three multimodal safety benchmarks used were [MM-SafetyBench](https://arxiv.org/pdf/2311.17600); [VLSBench](https://arxiv.org/abs/2411.19939); and [HoliSafe](https://arxiv.org/abs/2506.04704). Each of these sets is comprised of paired images and harmful user requests, designed to test whether a model refuses assistance in unsafe scenarios – such as asking how to carry out an apparent pick-pocketing scenario shown in an image.
Refusal Failure Rate (RFR) was used as the principal metric, measuring the percentage of harmful requests that a model failed to refuse, with higher scores indicating **lower** safety.
## Tests
Responses were classified by GPT-5.2 acting as an LLM judge, using the evaluation prompt recommended by VLSBench:
The GPT-5.2 evaluation prompt used to determine whether each model response represented a successful refusal, a safety-aware response that identified the risks involved, or an unsafe answer that failed to recognize those risks and proceeded with the harmful request. The judge was provided with the original image, user query and model response, and required to return its classification and reasoning in JSON format.
*The GPT-5.2 evaluation prompt used to determine whether each model response represented a successful refusal, a safety-aware response that identified the risks involved, or an unsafe answer that failed to recognize those risks and proceeded with the harmful request. The judge was provided with the original image, user query and model response, and required to return its classification and reasoning in JSON format.*
Each response was assigned to one of three categories: **safe with refusal**; **safe with warning**; or **unsafe**.
Refusal Failure Rates across the three safety benchmarks, comparing each model with and without access to tools. Every model became less likely to refuse harmful requests under tool use, though the scale varied considerably: average RFR rose by 12.6 points for GLM-5V-Turbo, compared with 2.3 points for GPT-5.4. Darker shading indicates larger increases in refusal failure.
*Refusal Failure Rates across the three safety benchmarks, comparing each model with and without access to tools. Every model became less likely to refuse harmful requests under tool use, though the scale varied considerably: average RFR rose by 12.6 points for GLM-5V-Turbo, compared with 2.3 points for GPT-5.4. Darker shading indicates larger increases in refusal failure.*
Giving the models tools made them more likely to answer harmful requests that they would otherwise have refused – a finding which occurred with every model, and on every benchmark tested. Overall, refusal failures increased by 17.7%, compared with the same models operating **without** tools.
GPT-5.4 was least affected: without tools, it failed to refuse 14.6% of harmful requests; with tools, that rose to 16.8%. For GLM-5V-Turbo, failures rose from 38.7% (no tools baseline) to 51.3%. The same pattern appeared across all three benchmarks, indicating that the problem may not be confined to a particular **kind** of harmful request.
### Reasons for Delinquency..?
As mentioned earlier, the researchers propose ‘context dilution’ as one explanation for the increased failures: as an agent makes successive tool calls, the original harmful request **becomes less prominent** among the accumulating tool outputs.
To test this, they considered only requests that the same model had successfully refused without tools. As shown below, failures then increased progressively with the number of tool calls, from c