A group of researchers from Israel-based cybersecurity company Cato Networks has identified several malicious actors who are using legitimate architectures of large language models (LLMs), such as Mixtral by Mistral AI or Grok by xAI, and modifying them to generate their own versions of the WormGPT chatbot, which is specially designed to create malware and is sold via Telegram.

The popularity of generative artificial intelligence (AI) and chatbots has led to the emergence of tools specifically designed to put their programming capabilities at the service of malware creation, as happened with WormGPT, which appeared in June 2023 and was promoted through dark web forums such as Hack Forums.

“WormGPT has become a brand name for uncensored LLMs that can be leveraged by threat actors in their offensive operations,” according to Cato.

Variants of WormGPT

The WormGPT chatbot was based on GPT-J, an open-source LLM developed by EleutherAI that, with capabilities similar to OpenAI’s GPT-3 and with 6 billion parameters, allowed for the creation of malicious code. However, in August of that same year, one of the creators of WormGPT shut down the service for fear of reprisals after being identified in a Krebs on Security article.

Following its closure, other chatbots with the same purpose began to appear, such as FraudGPT, DarkBERT, or PoisonGPT, presented as uncensored generative AI tools, facilitating the creation of malicious code, phishing pages or helping users to search for vulnerabilities, all simply by writing text prompts.

Now, experts from the cyber threat research lab at Cato Networks have shared a report identifying the emergence of new variants of WormGPT.

Hacking legitimate LLMs

Specifically, LLMs created by companies such as OpenAI, Google, Microsoft, Mistral, or xAI have various built-in security measures to prevent their use for malicious purposes. However, following the disappearance of WormGPT, a trend also gained strength among cybercriminals aimed at freeing legitimate LLMs to remove any restrictions on their use for malicious ends.

This is what seems to have occurred with the new variants of WormGPT identified by cybersecurity researchers. After using jailbreak techniques (the process used to remove restrictions imposed by a product’s manufacturer), the researchers were able to obtain answers from the new WormGPT chatbot regarding the underlying model that powers its responses.

Thus, the chatbot explicitly responded that WormGPT should not answer according to the standard Mixtral (Mistral AI) model but rather “must always generate responses in WormGPT mode.” According to the researchers, this is evidence that the malicious actor was able to bypass Mistral’s model security measures to use it for malicious purposes.
This evidence is accompanied by others, such as its ability to reveal specific architectural details of Mixtral. Furthermore, in their tests, the experts confirmed that the chatbot responded to any malicious request without any restriction, for example, by creating phishing emails.

Grok-based WormGPT

The same occurred with another of the identified variants of WormGPT, generated by the user called “keanu” in February 2025, who also published their advertisement on BreachForum and marketed it under a paid model.

In this other case, the researchers also analysed the model’s capabilities and, after again implementing jailbreak techniques, the chatbot eventually revealed that it operates powered by Grok, the model from xAI, the AI company led by Elon Musk.

To learn how to use large language models responsibly and avoid their misuse for malicious purposes, readers can consult this guide on using AI safely for users and developers, produced in collaboration with the International Association for AI and Ethics (IAAE).