What Is AI Security? Understanding AI's Role in Cybersecurity

What Is AI Security?

AI security is the practice of protecting AI systems from attacks that manipulate, steal, or corrupt them, and of governing how those systems are used. It defends three things attackers now target directly: the models that make predictions or generate text, the data those models learn from, and the applications and autonomous agents built on top of them.

AI security and AI in cybersecurity are two distinct concepts, and it is important to distinguish between them. ‘AI in cybersecurity’ uses machine learning as an operational tool to automate threat detection and triage. ‘AI security’ refers to defending the AI itself. Even a model built to hunt network threats needs its own protection from prompt injection, data poisoning, and model theft.

Traditional application security assumes software behaves the same way every time it runs. Today, AI has broken that assumption. An LLM produces different outputs for similar inputs, accepts instructions written in plain language, and depends on training data an attacker may be able to influence. Those properties open an attack surface that firewalls and code scanners can’t cover.

Why AI Systems Need Security

AI systems need security because attackers now target their models, training data, and agents directly, and because most organizations deployed AI faster than they could deploy controls to protect it. Attackers already treat AI as core infrastructure. They weaponize off-the-shelf models and hunt for weak points across the machine learning stack. 

Group-IB’s Weaponized AI research found that the volume of dark web first posts featuring AI keywords rose 371% between 2019 and 2025, and the same research identified at least three active vendors selling “Dark LLMs,” large language models stripped of safety controls, on subscriptions ranging from $30 to $200 a month. Today, criminal demand for AI capability is measurable, not speculative.

The expanding AI attack surface

Every AI deployment adds new entry points. Attackers probe model endpoints, poison the data a model retrieves, and abuse the tools that AI agents are allowed to call. A model exposed through an API is a networked asset that attackers can query, map, and manipulate like any other. 

MITRE ATLAS, the adversarial threat knowledge base for AI systems, catalogs how attackers reconnoiter models, gain access to training pipelines, and extract data from them.

These assets rarely appear in a traditional inventory. Shadow AI tools, forgotten model endpoints, and third-party integrations expand what defenders have to watch. A formal Attack Surface Management program closes that gap by continuously discovering and mapping an organization’s internet-facing assets, so the AI endpoints and APIs that never made it into an inventory get found and monitored instead of sitting exposed.

Risks associated with AI adoption

AI adoption is outpacing control. Employees paste sensitive data into unapproved tools, developers ship models before security review, and business units connect AI to production systems without governance. The same generative models that help analysts also enable attackers to automate reconnaissance and impersonation, as documented in the ENISA Threat Landscape 2025.

Gartner expects more than half of enterprises to adopt dedicated AI security platforms by 2028 to manage risks such as prompt injection and data misuse. Most enterprises are only now adopting those controls, leaving AI in production without dedicated protection in the meantime.

Security challenges unique to AI

AI systems fail in ways traditional software does not. 

• Non-deterministic outputs. The same test can pass on one run and fail on the next.
• A natural-language attack surface. An instruction hidden in a document can hijack a model.
• Data-dependent behavior. Corruption in the training data propagates to everything the model does downstream.
• Growing autonomy. More systems now act on their own, without a human approving each step.

That autonomy raises the stakes. When an agent can send an email, query databases, or move money, a manipulated output is no longer just a wrong answer but becomes an unauthorized action. NIST’s 2025 adversarial machine learning taxonomy documents these failure modes, with dedicated sections on indirect prompt injection and AI agent security.

Key Components of AI Security

The key components of AI security are securing AI models, protecting training data, securing applications and APIs, and governing how models and agents are allowed to act. Attackers pivot between these four layers, so defending one while neglecting another leaves the whole system exposed.

Securing AI models

AI models are both intellectual property and targets of attack. Adversaries steal model weights, tamper with a model to plant a backdoor, or extract a functional copy through repeated queries. MITRE ATLAS documents these extraction and tampering techniques through real-world observations and controlled research demonstrations. 

In the case of Proofpoint Email Protection (CVE-2019-20634), researchers collected the scores that Proofpoint exposed in email headers, trained an offline copycat classifier, and used it to determine how to craft emails that received more favorable scores. The exercise shows how exposed model outputs can support evasion, though it stopped at a research demonstration rather than a live attack.

A separate PoisonGPT demonstration shows how attackers could tamper with and redistribute an AI model. Researchers at Mithril Security edited the open-source GPT-J-6B model to implant a specific false fact while preserving its broader behavior and normal benchmark performance. 

They then uploaded the modified model to Hugging Face under “EleuterAI,” a typosquatted lookalike of the legitimate “EleutherAI” organization, illustrating how a poisoned model could slip into an open model supply chain.

Protecting a model means guarding access to its weights, the integrity of its files, and the query volume that could let an attacker reconstruct it.

Protecting training data

A model is only as trustworthy as the data it learns from. Attackers who corrupt training data, fine-tuning sets, or the documents a retrieval system pulls from can bias its outputs or plant hidden triggers. OWASP classifies this as LLM04 Data and Model Poisoning in its 2025 Top 10. 

The defense starts with knowing your data. Know where every dataset came from, validate it before use, and control who can write to the sources a model reads at runtime.

Securing AI applications and APIs

The application layer is where most attacks land, because it accepts untrusted input. Prompt injection (LLM01) manipulates a model through crafted input. Improper output handling (LLM05) lets a model’s response trigger downstream code execution or data exposure. 

Treat every model input and output as untrusted. Validate what goes in, sanitize what comes out, and never pass a raw model response straight into a system that can act on it.

AI governance and access controls

Governance decides what a model and its agents are allowed to do. OWASP labels this failure as LLM06 Excessive Agency, where an AI system is given more functionality, permissions, or autonomy than its task requires. 

Apply least privilege to models and agents the same way you do for users. Agents hold their own credentials and act on behalf of people, so treating agent identity as part of identity security keeps their access scoped and auditable.

Common AI Security Threats

The most common AI security threats are prompt injection, data poisoning, model theft and extraction, adversarial attacks, and sensitive data leakage. Each maps to a category in the OWASP 2025 Top 10, and each calls for a specific defense.

Prompt injection attacks

Prompt injection manipulates a model through crafted input that overrides or redirects its instructions. Direct injection alters the user prompt itself, while indirect injection hides instructions within external content, such as a website, document, or email, that the model processes as trusted context.

The attack is both effective and hard to stamp out. In a 2025 evaluation by NIST’s Center for AI Standards and Innovation and the UK AI Security Institute, tailored indirect prompt injection attacks against a leading AI agent succeeded up to 81% of the time, compared with 11% for standard baseline attacks. The UK’s National Cyber Security Center warns that prompt injection may “never be properly mitigated in the same way” as older flaws like SQL injection, leaving defenders to reduce its likelihood and impact rather than remove it.

The National Vulnerability Database lists CVE-2025-32711, an AI command-injection vulnerability in Microsoft 365 Copilot, published in June 2025. Public records describe it as allowing an unauthorized attacker to disclose information over a network, and Microsoft assigned it a CVSS score of 9.3, while NIST lists it at 7.5.

Data poisoning

Data poisoning corrupts a model at its source. An attacker who tampers with training data, fine-tuning sets, or retrieval knowledge bases can skew outputs, degrade accuracy, or plant backdoors that activate on a chosen trigger. 

It takes remarkably little to poison a model’s data. Research published by the UK AI Security Institute in 2025 found that around 250 malicious documents were sufficient to backdoor large language models, regardless of size (600 million to 13 billion parameters), because the number of poisoned samples needed does not scale with model size. 

Poisoning the public data these models learn from is cheap too. Researchers showed they could plant malicious examples in 0.01% of a widely used web-scale image dataset for about $60.

The damage lives in the model’s learned behavior, so it can persist until security teams identify and remove the poisoned data or compromised pipeline. OWASP tracks this as LLM04 Data and Model Poisoning, and NIST’s adversarial machine learning taxonomy includes poisoning as a core attack class against model integrity.

Model theft and extraction

Model theft takes the asset itself. Attackers may steal weights directly when access controls are weak, or reconstruct a working copy by repeatedly querying a model and training a surrogate on the responses, a technique the NIST adversarial machine learning taxonomy classifies as model extraction. The impact is both competitive and financial, because an attacker can analyze a stolen model offline for weaknesses. 

Extraction is cheap. Researchers showed they could recover part of the projection layer behind OpenAI’s Ada and Babbage, through ordinary API access for under $20. They also estimated it would cost under $2,000 to do the same against gpt-3.5-turbo.

However, the outright theft of a full model’s weights is a different story. The International AI Safety Report found no confirmed, publicly documented case of model weight theft as of December 2025. Still, they caution that weights can still be exposed through theft or leakage, and that current defenses may not hold against the most capable attackers.

Adversarial attacks

Adversarial attacks feed a model inputs engineered to force a wrong answer. In many cases, small perturbations to an image, file, or signal can cause a classifier to misread it, which is especially dangerous when models influence security or safety decisions. NIST classifies this as an evasion attack and points to markings on a stop sign that make a self-driving car read it as a speed-limit sign.

Researchers have shown how well these attacks work in the physical world. In one study, black-and-white stickers on a real stop sign made a road-sign classifier read it as a speed-limit sign in 100% of lab images and 84.8% of video frames shot from a moving vehicle. In another study, an adversarial pattern printed on a T-shirt let someone evade an AI person detector 57% of the time while moving, the kind of model a security camera might rely on.

Sensitive data leakage

Attackers deliberately probe models to pull out data they were never meant to reveal. With crafted queries, they can make a model surface personal or proprietary information it memorized during training, or disclose the system prompt that steers its behavior. Both are active techniques, not lucky accidents. OWASP tracks these as LLM02 Sensitive Information Disclosure and LLM07 System Prompt Leakage.

In one study, researchers spent only about $200 querying ChatGPT and pulled out more than 10,000 unique training examples the model had memorized, including the personal contact details of real individuals and verbatim passages from published books. They estimated a determined attacker could extract far more with a larger budget.

How Organizations Can Secure AI Systems

Organizations secure AI systems by hardening the agentic supply chain, monitoring models against real adversary behavior, testing them with AI red teaming, and gating every high-impact action behind a policy that the model cannot override. These controls assume the model itself can be manipulated, so they contain what a compromised model can do rather than trusting it to behave.

Secure AI development practices

Secure AI development means finding a system’s weaknesses before attackers do, then hardening everything the model depends on. The first practice is threat modeling. You map how an attacker could reach and abuse each part of the system, and extend that map beyond the application itself to the agents, tools, and data sources the model can call, since each one is a way in.

The second is supply chain vetting. Treat every component, including pretrained models, datasets, plugins, and connectors, with the suspicion you would give an unverified binary. OWASP tracks this as LLM03 Supply Chain and, for agent-based systems, ASI04 Agentic Supply Chain Vulnerabilities in its Top 10 for Agentic Applications. Verify each component’s source, and test it before it reaches production.

Continuous AI monitoring

Log model inputs and outputs, and watch for anomalies that signal an attack in progress, such as sudden spikes in token volume, semantic drift, or the model generating code or calling tools outside its task. AI-enabled attacks leave little forensic trace, so build this visibility in rather than reconstruct it after the fact. 

Anomaly detection only works if you know what a live attack looks like. Group-IB Threat Intelligence provides that context by tracking the underground market for Dark LLMs and AI-enabled crimeware, so your monitoring can detect tool misuse and multi-turn exploitation before data is exfiltrated.

AI risk assessment and testing

Test AI systems the way an attacker would, before deployment and on a schedule after it. Static analysis and traditional penetration testing miss the probabilistic weaknesses of GenAI, so use AI red teaming to surface prompt injection, model inversion, membership inference, and poisoning that code review never sees. Route the findings into your existing vulnerability management workflow so that AI flaws are tracked and remediated like any other. 

Group-IB AI Red Teaming runs exactly these tests on your production models, surfacing prompt injection, data poisoning, and model extraction before attackers exploit them. Each engagement maps to OWASP and other emerging AI safety frameworks and ends with a tailored remediation roadmap, so you can fix weaknesses in priority order and demonstrate due diligence to partners and regulators.

Human oversight and governance

Keep humans in control where the stakes are highest, and make that control something the model cannot route around. Gate every state-changing action, such as code commits, database writes, payments, or data deletion, behind a policy engine or human approval that the agent cannot bypass. Grant each agent the least privilege and autonomy its task requires, which OWASP frames as ASI03 Identity and Privilege Abuse and LLM06 Excessive Agency. 

For shadow AI, replace written policy with active discovery. Use API gateways to find and block unsanctioned AI tools before they become unmonitored exits for corporate data.

The Future of AI Security

The future of AI security is a contest between autonomous attackers and autonomous defenders, and it favors whoever adapts faster. AI cuts both ways, speeding up detection and triage for defenders while also empowering attackers with automation, impersonation, and scale. As AI agents take on more of the work on each side, that dynamic sharpens.

This is where AI’s role in cybersecurity shifts from a single tool to the terrain itself. Agentic AI security moves to the center because an autonomous attacker and an autonomous defender can both act faster than a human can intervene. Group-IB’s own threat researchers see the same trajectory.

Over the next few years, the main AI use case in cybersecurity will be defending the AI systems that both sides now depend on. Gartner expects that by 2028, half of all enterprise cybersecurity incident response efforts will focus on custom-built AI-driven applications, and more than half of enterprises will run dedicated AI security platforms. 

Strengthen Your AI Security Posture With Group-IB

Protecting AI requires a dual approach of understanding adversaries’ real-world tactics and rigorously stress-testing your internal environments against those threats. 

Group-IB does both, drawing on its investigations into the underground market for Dark LLMs, deepfake services, and AI-enabled crimeware documented in its Weaponized AI research.

Here is how Group-IB helps secure your AI systems:

1. Group-IB Threat Intelligence tracks the tools and tactics behind AI-enabled attacks, from Dark LLMs to crimeware kits, so your detection is grounded in real adversary behavior instead of assumptions.
2. A Group-IB AI Red Teaming engagement stress-tests your models, applications, and agents against prompt injection, data poisoning, model extraction, and excessive agency before attackers find the gaps.
3. Findings route into your existing vulnerability management workflow with a clear, prioritized fix plan, so AI weaknesses get tracked and resolved like any other risk.

Each engagement is tailored to your stack and mapped to OWASP and other emerging AI safety frameworks, reducing the risk of breaches, leaks, and reputational damage, and providing demonstrable due diligence for users, partners, and regulators.

Group-IB pairs adversary intelligence with hands-on offensive testing, a combination few vendors offer, and its intelligence draws on cooperation with INTERPOL, Europol, and AFRIPOL. 

Talk to our experts about securing your AI systems end to end.