RISKS OF USING GENERATIVE LANGUAGE MODELS IN PHISHING ATTACKS: ANALYSIS AND THREAT MODELING
RISKS OF USING GENERATIVE LANGUAGE MODELS IN PHISHING ATTACKS: ANALYSIS AND THREAT MODELING
Abstract
The article examines the risks associated with the use of large language models and generative artificial intelligence (hereinafter referred to as AI) in phishing attacks. The purpose of the study is to analyze how LLMs transform phishing, what new vulnerabilities emerge as a result, and how a threat model for such attacks can be constructed. It was found that the use of LLMs changes phishing in several ways: it accelerates OSINT preparation, increases the plausibility of texts, simplifies message personalization, expands the range of delivery channels, and enhances the reproducibility of attacker actions. The authors propose an original threat model linking threat actors, assets, delivery channels, vulnerabilities, and consequences. It is concluded that generative AI should be regarded as an independent factor increasing cyber risk, while countering AI-enabled phishing should rely on the combined implementation of organizational, technical, and analytical measures.
1. Introduction
The proliferation of large language models has transformed the nature of emerging digital threats. While early discussions of risks associated with generative AI focused primarily on adversarial attacks on models and issues of output reliability, attention has increasingly shifted toward the risk of generating malicious content using AI. From this perspective, LLMs function as a tool for preparing attack vectors, which is particularly relevant in the context of phishing, as this form of attack is inherently built around text, trust, the imitation of legitimate communication, and the exploitation of user behavioral responses. For this reason, generative AI should be considered as an independent source of new cyber threats . At the same time, phishing remains one of the most widespread forms of cyberattacks, relying on fraudulent communication to obtain credentials, financial information, or access to protected resources. The contemporary academic literature emphasizes that victim vulnerability is determined by a combination of factors: outcomes are influenced by cognitive, emotional, social, and situational conditions, including time pressure, trust in the perceived authority of the sender, patterns of digital behavior, and the overall level of cybersecurity awareness. In the context of the active use of generative AI, this dependency is further amplified, as attackers gain the ability to rapidly tailor message content to a specific recipient and situation .
The aim of this study is to analyze the risks associated with the use of generative language models in phishing attacks and to develop a threat model that links the technological capabilities of LLMs with typical attack scenarios, victim vulnerabilities, and organizational consequences.
The research materials consisted of academic publications addressing the cybersecurity risks of generative AI, the human factor in phishing, AI-enhanced social engineering, tools for generating phishing content, methods for its detection, and security issues of systems incorporating large language models. The study is based on methods of comparative analysis, problem-oriented thematic classification, structured description of AI-enabled phishing scenarios, and threat modeling.
2. Main results
The emergence of generative language models is transforming phishing primarily by increasing its accessibility. Previously, the preparation of a plausible attack depended on the attacker’s competence (at a minimum, language proficiency), the ability to imitate corporate communication styles, and the manual crafting of messages; however, with the widespread availability of LLMs, much of this work is delegated to the model. Existing research on offensive-side scenarios demonstrates that publicly available tools can be used to rapidly generate multiple variations of similar phishing messages, implement phishing schemes involving voice modulation, and even configure chatbots capable of conducting conversations on behalf of the attacker. Notably, generative models are involved already at the attack preparation stage, as they assist in collecting and structuring information about the target, constructing a credible pretext, generating message content, selecting an appropriate tone, and maintaining communication after the initial contact .
At the content level, the threat is associated with several key changes (Fig. 1).
The impact of generative models on the transformation of phishing and associated threats
In addition, contemporary threats are becoming increasingly multimodal; attackers may employ not only email, but also voice phishing (vishing), smishing, spoofed video calls, and chatbots that sustain the victim’s engagement as the attack unfolds. In the context of AI-enhanced social engineering, the academic literature identifies three key properties that provide attackers with the highest likelihood of successful phishing: plausibility, personalization, and automation. Taken together, these factors enable an unprecedented level of influence over the user during an attack .
In light of this, the following classification of AI-enabled phishing can be proposed (Fig. 2).
Classification of attacks associated with AI-enabled phishing
Comparative analysis of existing academic approaches to the study of AI in phishing
compiled by the author based on [5], [6], [7], [8]
Direction | Focus of the study | Purpose | Relevance for threat modeling |
Human factor | Recipient behavior and vulnerabilities | Explains the causes of attack success | Enables the identification of victim vulnerabilities |
AI-enhanced social engineering | AI as a tool of persuasive influence | Demonstrates increased personalization and plausibility | Links LLMs with influence techniques |
Phishing content generation | Creation of phishing messages using LLMs | Reveals offensive capabilities of AI | Describes the preparation and delivery stages |
Detection of AI-enabled phishing | Methods for identifying AI-generated attacks | Forms the basis for new defensive solutions | Establishes foundations for countermeasure |
Protection of LLM-based systems | Threats to LLM infrastructure and protection measures | Expands risk understanding beyond the message itself | Incorporates protection of the LLM environment into the threat model |
Multi-agent detection systems | Architectures for intelligent threat detection | Enhances prospects for threat detection | Refines future directions of defensive strategies |
Thus, LLMs can be used to bypass traditional filtering systems and to generate messages that account for anti-spam features already at the construction stage. In one of the most illustrative studies, the Phish-Master algorithm was proposed, combining Chain-of-Thought, MetaPrompt, and domain-specific prompts. The authors demonstrated that emails generated in this manner were successfully delivered in 99% of cases in a real network environment; accordingly, the detection model developed on their basis achieved very high performance on test data .
In addition, it is widely recognized that traditional defense approaches have long relied on URL-based features, email header analysis, domain anomalies, and other formal indicators. However, these methods are no longer sufficient for detecting LLM-generated phishing, as such messages can be grammatically correct, stylistically consistent, and contextually relevant while still containing malicious intent. As a result, the importance of semantic analysis and the detection of underlying persuasion patterns is increasing. In particular, the academic literature proposes a two-stage model in which the presence of persuasion principles is first assessed, followed by binary classification of the message based on this evaluation. In one study, the authors used a dataset of 2,995 AI-generated emails, achieving a detection accuracy of 94% with an AUC of 98%. Notably, this approach analyzes the message as a mechanism of psychological influence on the recipient .
Based on this, a threat model for the use of LLMs in phishing attacks can be proposed (Fig. 3), the construction of which is grounded in the vulnerabilities of LLM-based systems themselves when they are integrated into an organization’s business processes , .
Threat model for the use of LLMs in phishing attacks
3. Conclusion
Thus, the analysis demonstrates that the use of generative language models transforms both the nature of traditional phishing and the approaches to its execution, affecting the entire attack lifecycle. At a minimum, LLMs accelerate reconnaissance, increase the likelihood of successful message delivery, facilitate personalization, and support the scaling of attacks across multiple channels. As a result, phishing becomes less costly to prepare, more precise in targeting, and more difficult to detect. The core of the threat lies in the human factor, as well as in the limited capabilities of existing defenses and detection methods against AI-generated phishing. In this context, the most effective approach appears to be the integration of the identified threats within a unified model, followed by the development of appropriate defensive strategies and mechanisms.