What is generative AI in cybersecurity?

Generative artificial intelligence creates new content, including text, code, images, audio, and video, by learning patterns from existing data. In cybersecurity, it works on both sides of the fight. Attackers use it to produce convincing phishing messages, synthetic voices and faces, and working code at a speed and quality that once required more skill and time. Defenders use it to reduce the analytical work that fills a security team's day, from summarizing an incident to translating a question into a query. Generative AI is often treated as if it covers all artificial intelligence in security. It doesn't. Security teams have used machine learning for years to classify and detect threats. The defining difference is that generative AI creates new output. This page explains that distinction, how attackers and defenders use generative AI, the security risks it introduces, and how to adopt it without creating new exposure.

Key takeaways

Generative AI creates, analytical AI classifies. That distinction matters because security tools used machine learning for detection long before generative models became widely available.
It lowers barriers for attackers. Fluent phishing in many languages, cloned voices, synthetic video, and code assistance are now available to actors who previously lacked the skills or resources.
Its defensive value is analytical. Summarizing incidents, answering questions in plain language, and drafting detection logic save analyst time, which is one of the scarcest resources in a security operations center.
It introduces new risks. Prompt injection, sensitive data leakage into external models, and confident but incorrect answers require controls that traditional security programs were not designed to provide.
Autonomy has limits. An assistant that drafts and accelerates work is useful. A security operation that acts without human judgment creates a different risk profile and needs much stronger safeguards.

‍

How generative AI differs from the AI already in security tools

Artificial intelligence has been part of security tooling for years. Anomaly detection, malware classification, and user behavior analytics all rely on machine learning models that make judgments about data. Is this activity normal? Does this file resemble known malware? Does this login fit the usual pattern? These are discriminative models. They sort, score, and flag.

Generative models produce new output. They can write an explanation, generate code, summarize evidence, or turn a natural-language question into a query. The distinction prevents confusion about what generative AI contributes to a security program.

	Analytical AI in security	Generative AI
What it does	Classifies, scores, and flags data.	Produces new text, code, images, audio, or video.
Typical security use	Anomaly detection, malware classification, and behavior analytics.	Incident summaries, plain-language queries, explanations, and draft detection content.
Output	A judgment, such as a score, label, or alert.	New content that must be checked before it is trusted.
Main failure mode	False positives and false negatives.	Plausible but incorrect output and susceptibility to manipulated input.

Both approaches are useful, and they increasingly work together. The practical difference is how errors appear. A classifier produces an alert that is right or wrong about a specific signal. A generative model can produce a fluent, confident paragraph that sounds correct while containing a serious error. That difference determines where human review is necessary.

Where generative AI came from

The field is older than the current attention suggests. Generative approaches gained momentum in 2014 with generative adversarial networks (GANs), a design in which two models compete. One produces candidate data while the other judges whether it looks real. GANs produced synthetic images convincing enough to matter and became part of the technical foundation behind deepfakes.

The larger shift came from language. The transformer architecture published in 2017 proved highly effective for text, and large language models (LLMs) built on it. The public release of a conversational model in late 2022 turned generative AI from a research topic into a tool that anyone could use. The security implications grew with that accessibility.

How attackers use generative AI

Generative AI does not give attackers every capability from scratch. Its effect is to remove constraints that once limited who could use existing techniques and how extensively they could use them.

Phishing and social engineering at scale: Attackers can produce fluent, context-specific messages in many languages and distribute them in volume. This removes the clumsy phrasing that users were trained to notice and reduces the disadvantage faced by actors working outside their native language.
Synthetic voice and video: Cloned voices can make a fraudulent phone request sound credible, while synthetic video can place a fabricated participant in a meeting. These techniques support payment fraud, executive impersonation, and other forms of social engineering.
Code and malware assistance: Generative tools can help write, adapt, and obfuscate malicious code. That lowers the skill needed to produce a functional tool and speeds up the changes that can defeat signature-based detection.
Reconnaissance and pretexting: Attackers can quickly turn public information about an organization and its people into the specific detail that makes a pretext believable.
Disinformation and fraudulent content: Fabricated articles, reviews, and personas can be produced cheaply enough to operate at a scale that manual effort could not reach.

Sekoia's threat research has tracked this progression into malware itself, documenting implants that use an LLM as part of their operation. The model then becomes a component of the attack running on the target, rather than only a tool the attacker uses during preparation.

How defenders use generative AI

Generative AI is strongest at the analytical and communication work surrounding security decisions. The most useful applications keep a human responsible for checking the output and making the decision.

Investigation support: Summarizing what happened in an incident, reconstructing a timeline from scattered evidence, and explaining an unfamiliar technique to the analyst handling it.
Plain-language querying: Turning a question into the query syntax a platform needs, which lowers the barrier for less experienced analysts and speeds up experienced ones.
Alert triage assistance: Adding context to an alert and proposing a likely assessment, so analysts can focus on judgment rather than assembling the initial picture.
Detection engineering support: Drafting and refining detection rules, and translating a described behavior into logic that can be tested.
Reporting and communication: Producing incident write-ups, executive summaries, and documentation that consume analyst hours and are often difficult to complete under pressure.
Synthetic data and adversary simulation: Generating realistic test data and attack variations to exercise detection coverage without exposing real information.

These uses can improve the speed and consistency of security work. They do not remove the need for investigation, validation, or accountability.

The security risks generative AI introduces

Adopting generative AI creates exposure that existing controls may not cover. The risks apply whether an organization deploys its own models or staff use public tools.

Prompt injection: Instructions hidden in content processed by a model can cause it to ignore its original task and follow an attacker's instructions instead. The risk is especially serious in systems where a model can take actions, because manipulated input can lead to unauthorized activity.
Sensitive data leaving the organization: Staff may paste source code, contracts, customer information, or security data into public tools. This practice is often called shadow AI, and it can move sensitive information beyond the organization's oversight.
Confident errors: Models produce fluent output whether or not it is correct. In a security context, a fabricated indicator or incorrect conclusion can send an investigation in the wrong direction.
Agents with permissions: An AI agent that can act, rather than only answer, has access to systems and data. It deserves the same scrutiny as any privileged application and is attractive to attackers because of what it can reach.
The AI supply chain: Frameworks, packages, datasets, and models are dependencies. A compromise in a widely used AI component can affect every organization that relies on it.
AI systems as targets: Models embedded in workflows and connected to sensitive data become valuable targets in their own right. Attackers can target their training data, prompts, connected systems, or access controls.

How to adopt generative AI safely

Organizations that gain value from generative AI without adding unnecessary exposure tend to handle a few basic controls well:

Decide what may go into a model. Create a clear policy covering which data can be shared with which tools. Vague guidance encourages shadow AI.
Keep a human accountable for security decisions. Use generative output as a draft, summary, or suggestion. Keep the decision and accountability with a person.
Treat AI components as part of the attack surface. Inventory them, control what they can access, apply least privilege to agents, and monitor their activity.
Test for prompt injection and misuse. Assess AI-integrated applications like any application that handles untrusted input, because that is what they do.
Update awareness training for synthetic media. Teach verification habits rather than relying on people to identify every convincing voice or face. An out-of-band check through a known channel remains effective.
Log and monitor AI usage. If an organization cannot see which tools are in use or what they are being asked, it cannot govern their use or investigate incidents afterward.

Sekoia's approach to generative AI

Sekoia is a European cybersecurity vendor, and generative AI features in its work from both directions. On the threat side, the in-house Threat Detection & Research (TDR) team tracks how threat actors adopt these tools, including research documenting malware that incorporates an LLM into its operation. Understanding how attackers use generative AI in practice keeps detection grounded in observed behavior.

On the platform side, Sekoia Elevate provides AI agents that work alongside SOC analysts on parts of the investigative and triage workload. The design principle is straightforward: give analysts time back without removing them from the decision loop. Detection remains grounded in behavioral rules mapped to the MITRE ATT&CK framework and in cyber threat intelligence (CTI) produced by TDR. Generative capability sits on top of that foundation to accelerate human work rather than replace the analytical basis for a detection.

Why the autonomous SOC deserves scrutiny

There is strong commercial pressure to promise a security operation that runs itself. That proposition is attractive to teams overwhelmed by alerts, but it asks security leaders to trust a system that can produce fluent and confident answers that are wrong with consequential decisions and little or no supervision. Giving an AI agent authority to isolate hosts or close investigations without defined safeguards transfers risk rather than reducing it.

A more defensible position focuses on the work generative AI handles well: gathering context, summarizing, explaining, drafting, and translating questions into queries. That work represents a large part of an analyst's day, and much of it benefits from acceleration without removing human judgment. Detection should continue to rest on behavior and intelligence because those foundations can be tested, investigated, and explained.

Attackers already use generative AI, so organizations need a clear position on where it belongs in their security program. The practical answer is to use it to accelerate analysts while governing it like any component that handles sensitive data or holds permissions.