Generative AI on the Rise
AI is no longer a niche topic. In fact, it has become a key driver of digital transformation. At the forefront is generative AI, which differs from conventional AI trained for specific classifications or associated tasks. Rather, it can independently generate new content, such as text, images, audio, or program code. This opens up a wide range of applications without requiring dedicated training for every single task. This functionality offers enormous potential for increasing workplace productivity by automating routine tasks, accelerating decision-making processes, and efficiently supporting creative activities. Analyses by leading consulting companies confirm this development: according to a recent cross-sector, cross-position survey by BCG, nearly half of survey participants believe generative AI allows them to devote more than an hour per day to other important tasks [1]. McKinsey even estimates that generative AI could boost annual labor productivity of up to 0.6% by 2040 [2]. Yet, despite these impressive numbers, generative AI comes with its downsides.
Reconnaissance and Next-Generation Malware
One downside of generative AI is that this technology not only increases workplace productivity but is actively exploited by cybercriminals during reconnaissance. During this initial attack phase, cybercriminals systematically collect information about target systems and networks of potential victims, uncover vulnerabilities, and develop customized tools and malicious code, laying the foundation for a successful operation. Generative AI plays an increasingly important role in this process, particularly in vulnerability analysis and malware generation. Although there are no statistical data on AI-generated malware currently in circulation, security researchers regularly publish new findings in the form of proof-of-concepts. In May 2025, for instance, a polymorphic AI-based malware was disclosed that, unlike traditional variants limited to predefined mutation patterns, leverages generative AI to produce an almost unique instance with each execution [3]. Similar results were published in August and September 2025 with the malware PromptLock [4] and MalTerminal [5], both developed with the same objective: to maximize impact while deliberately deceiving existing detection methods. Remarkably, all three examples are essentially built on legitimate large language models (LLMs). Beyond that, so-called dark LLMs, such as FraudGPT, DarkBERT, and PoisonGPT, have circulated for some time. Accessible via the dark web, Telegram, or online forums, these models are explicitly tailored for malicious use. This includes phishing.
Dangerous Eloquence: (Voice) Phishing 2.0
According to recent reports, phishing remains one of the top attack vectors [6, 7, 8]. It is a form of social engineering in which attackers send deceptive messages to trick victims into revealing sensitive data or executing malicious links or attachments. LLMs make the difference here: if there is one thing LLMs are known for, it is generating seemingly creative and eloquently written texts that are almost indistinguishable from those written by humans. Whereas phishing emails in the past could often be exposed by linguistic flaws such as spelling or grammar errors, that is no longer the case with the help of generative AI. As a result, such attacks can be scaled much more effectively and carried out across language boundaries. This is especially true when no legal or ethical constraints stand in the way, as is the case with certain Dark LLMs. IBM’s latest cybersecurity report states that, with the assistance of AI, most data breaches are caused by phishing and that generative AI can reduce the time required to create a convincing phishing message by almost 200 times [8]. The threat posed by voice phishing (vishing) is also prevalent and, according to a report by CrowdStrike, has increased by more than 440% since last year [9]. This upward trend is likely to persist as generative AI becomes more widely adopted. Traditionally, attackers conducted such scams by impersonating family members, bank employees, or other trusted parties over phone or VoIP channels to extract sensitive information or prompt transactions. With generative AI, however, vishing is reaching an entirely new level of sophistication. With suitable voice material, voice-cloning can convincingly imitate familiar voices. Combined with an appropriate LLM that generates questions or responses according to an attacker’s instructions, such an attack can even be fully automated and carried out at large scale. Recent research indicates the feasibility of such a system [10] – with alarming efficiency: in the study, AI-assisted vishing was able to expose sensitive data from every second participant.
When Good AI Becomes a Risk: Manipulation of Legitimate LLMs
Platforms like OpenAI’s ChatGPT and other major online services that have integrated LLM features have become increasingly appealing to cybercriminals. Their interest stems not only from the ability to exploit these models for malicious activity as stated earlier or from the potential to exfiltrate sensitive information through direct prompt injection attacks. The platform’s reach is equally crucial. This makes it obvious to attackers that they can use these popular applications to spread malicious content, comparable to classic malvertising tactics in search engines. Experts and authorities have long warned about indirect prompt injections [11, 12]. In such an operation, hidden instructions in insecure data sources cause an LLM to display unwanted content or perform unwanted actions. A striking example occurred in September 2025, when a malvertising campaign emerged on the social media platform X. Attackers exploited X’s proprietary chatbot, Grok, by tricking it into reading manipulated metadata from a video advertisement. As a result, Grok unknowingly posted the embedded malicious link under its own name, lending it an appearance of legitimacy to other users [13]. The danger of indirect prompt injection becomes especially serious when an external LLM service is deeply integrated into corporate infrastructures and governs routine operations. In June 2025, a team of researchers discovered an unknown vulnerability in Microsoft Copilot by sending a specially crafted email. Copilot read and executed the embedded instructions in the email. This allowed private data to be stolen without any user interaction [14]. A similar zero-click exploit was detected in Google’s Gemini in August 2025. In this case, a manipulated calendar invitation triggered unwanted smart home actions, including the leakage of emails and other sensitive content [15].
AI Governance for a Responsible use
By now, the dilemma of generative AI is clear: it is a symbol of digital transformation, but it is also used by threat actors to plan and execute classic attacks or to attack generative AI itself. Numerous incidents and new findings confirm these trends well into 2025. The latter is particularly appealing to attackers and will likely become a common modus operandi as generative AI becomes more integrated into the workplace. Overall, this development poses considerable challenges for decision-makers when introducing and dealing with this technology. To counter these challenges, a suitable framework of rules, processes, and responsibilities is needed that leverages potential while reliably mitigating risks. Important elements of such an AI governance include the conscious, measured, and purposeful use of generative AI. This use must remain under control by continuously considering cybersecurity from the outset and throughout the entire data and processing pipeline. This includes managing third-party risks, setting out emergency and shutdown procedures, and assigning clear accountability. These elements are consistent with the provisions of the EU AI Act, which explicitly or implicitly calls for such structures. Employees play an equally central role: AI governance can only be effective if it is understandable and provides concrete guidance. The permitted and prohibited uses of the technology must be explained transparently in order to promote acceptance and prevent shadow IT and circumvention strategies. At the same time, there is a need for education about the threat situation, which has reached a new dimension with generative AI. Employees must be prepared for this and learn how to use AI responsibly. This includes using only approved generative AI and not sharing confidential content with external services. A mindset of healthy, constructive skepticism remains vital, as even trusted systems can be flawed or manipulated. In this sense, the employee represents one of the most important – if not the decisive – pillars of effective AI governance.
We have outlined further measures and recommendations in our report ‘Data Leakage 2024+: Current Cyberthreat Tactics and Beyond.’
