What you need to know: Artificial Intelligence at the Heart of Cyber

As AI technologies become more embedded in cyber strategies, they enhance the capabilities of threat actors while also offering innovative defenses to organizations [1]. AI tools can amplify adversaries’ traditional Techniques, Tools, and Procedures (TTPs) by automating the generation of sophisticated threats such as polymorphic malware — which can dynamically alter its code to evade detection systems — or enhancing adversaries’ traditional capabilities from reconnaissance to defense evasion [2].

In social engineering, AI-powered phishing emails and deepfakes can deceive individuals at target organizations and ultimately serve as a point of compromise into networks, and as a leeway to collect sensitive information [3]. The proliferation of AI applications and capabilities along the threat landscape not only enhances the strategic capabilities of adversaries but also potentially lowers the barrier of entry for bad actors, by making attack vectors more accessible and potentially more stealthy [4]. The scale and speed that adversaries gain from utilizing AI tools means that any organization or sector could be a target, anywhere, and anytime. Since the ways in which AI models will modify offensive cyber-operations are still unclear, it is crucial for organizations to implement security measures, and keep track of the threat landscape to build operational readiness to effectively contain any damages from emerging attack vectors.

The applications of AI tools can bring about both malicious and beneficial outcomes, and it is becoming increasingly challenging to contain the repurposing of models and the outputs they might generate, due to a variety of reasons:

1. AI development is often a ‘black box,’ meaning that AI tools are built in proprietary settings where training methods, model architecture, and the utilized data are often not disclosed [5]. Most AI models with large computing power require a high volume of data for training purposes, which usually requires scraping data off the internet. Data scraping to train AI models could introduce a layer of uncertainty regarding the content fed to the model and the inferences it might make during training and post-deployment [6]. Given that these models are often fed with vast quantities of scraped data, it becomes difficult to identify the nature of data patterns the model has ingested and memorized. This black box raises concerns not only about the model’s potential outputs but also about its reliability and the types of predictions it may generate. For instance, if a model is jailbroken (manipulating the model to perform outside its intended operational parameters) or poisoned (injecting harmful data into the training set), there is a risk that private information might be exposed [7,8].

2. Malicious actors are able to build ‘custom,’ smaller AI models — by fine-tuning from larger models — that perform well for specific use cases. The wide availability of pre-trained models that can be downloaded and fine-tuned offline can allow adversaries to bypass detection and retrain these distilled models on malicious repositories of data, used for creating deep fakes, automating hacking processes, or generating phishing emails [9]. These customized models pose a significant challenge in terms of detection and mitigation because they often operate under the pretext of legitimate AI functionalities and do not exhibit overtly malicious behavior until deployed. Additionally, their outputs — especially AI-generated phishing emails— are closing the gap with traditional email writing quality, which complicates the efforts of security systems/teams to detect and flag them [10]

3. AI can be leveraged by adversaries to amplify plausible deniability in cyber-attacks [11]. Adversaries can utilize AI to obfuscate their ‘digital footprints’ (whether it is creating several copies of malware but leveraging AI to modify its source code to ‘make it seem like a different malware family’, but preserve its functionality), which makes attribution of cyber-attacks much more challenging [12]. AI models could potentially assist adversaries in generating adaptive malware that evades detection and morphs its behavior based on the environment it compromises (i.e., effectively mimicking benign software).

AI also presents opportunities for organizations seeking to reinforce their cybersecurity posture. By integrating AI tools into the operational and tactical folds of the organization, stakeholders can enhance and speed up their ability to detect anomalies, automate threat responses, and perform predictive analytics to pre-empt and prepare for potential breaches [13]. This proactive use of AI not only fortifies the security posture of organizations but also aligns them with cutting-edge technological advancements that can learn and keep pace with TTPs and cyber-attacks:

4. Since AI models can sift through large volumes of data, they can be trained on patterns of past exploited vulnerabilities and repositories of high-risk vulnerabilities. In this way, they can support host organizations in vulnerability mapping and provide patch prioritization recommendations [14].

5. AI models can help organizations be proactive rather than reactive regarding their cybersecurity posture by automating network intrusion detection and supporting organizations in understanding the multi-sectoral threat landscape [15]. By also enhancing incident response strategies, AI can automate a slew of tasks that would typically require human intervention and allow human resources to focus on more complex security challenges.

6. AI models can potentially support organizations in reviewing source codes to mitigate weaknesses that adversaries could leverage [16].

7. Defenders can take advantage of AI innovation’s growth trajectory to develop more sophisticated threat detection systems that learn and adapt to new threats faster. By implementing AI-driven security protocols, defenders can better predict and prepare for their risks, using predictive analytics to model potential attack vectors and identify anomalies [17]. This approach not only strengthens the security posture but also reduces the response time, making vulnerability exploitation more difficult.

Terms to Know

Generative AI (GenAI)

Refers to a category of AI technologies capable of creating new content using advanced machine learning models to analyze and learn from large datasets, and understand the underlying patterns and structures of the input data. Once trained, GenAI models can generate outputs that are novel yet plausible based on the learned information. While GenAI models have been a catalyst for innovation in many industries, they can also be misused to create harmful content [18].

Machine Learning (ML)

Machine Learning (ML) is a subset of artificial intelligence (AI) that focuses on utilizing datasets and algorithms to allow AI models to imitate human learning patterns and automatically improve output accuracy [19].

AI Model Hallucinations

A hallucination occurs when an AI model generates incorrect, fabricated, or incoherent outputs. This typically occurs in response to inputs or data patterns that the model has not encountered during training or when the model is forced to make predictions or generate content from inadequate or ambiguous data. In these cases, the model tends to “hallucinate” information that is not grounded in the input or data provided. This issue is commonly observed in language models, image generation models, and other generative AI systems [20].

Large Language Models (LLMs)

LLMs are central to AI progress, enabling tasks like code generation and text classification. Malicious applications like FraudGPT and WormGPT exploit LLMs to bypass safeguards [21].

Finetuned AI Models

Fine-tuned AI models refer to pre-trained artificial intelligence models that have been further trained or adjusted on a specific, often smaller dataset to specialize or improve their performance on particular tasks [22].

FraudGPT and WormGPT

These malicious chatbots facilitate various cybercrimes, including fraud and phishing, by providing resources and tutorials to less experienced threat actors [23].

Black Mamba

Black Mamba is an AI-enabled polymorphic malware proof of concept – malware that can morph or alter its source code to evade detection [24].

Deep Fake

A deepfake is a video, photo, or audio recording that seems real but has been manipulated with AI. [25]

AI-enabled Code Obfuscation

It involves applying various transformations to source code, making it harder to comprehend and analyze, with the help of AI [26].

Denial of Machine Learning (ML) Service

Denial of Machine Learning (ML) Service is a type of cyber-attack that targets ML systems with the intent to degrade their performance, by specifically exploiting vulnerabilities in ML models to generate operational failures [27].

Jailbreaking AI Model

Jailbreaking an AI model involves modifying an AI system’s ‘behavior’, by prompting it to bypass or evade software restrictions instilled by its developers [28].

Poisoning AI Data

Data poisoning involves an adversary inserting harmful or malicious data into the model’s training datasets with the goal of compromising the AI model’s accuracy or reducing its overall effectiveness [29].

Supervised Learning

Supervised learning is a category of machine learning that uses labeled datasets to train algorithms to predict outcomes and recognize patterns [30].

Unsupervised Learning

Unsupervised learning is a category of machine learning that learns from data without human supervision. Unlike supervised learning, unsupervised machine learning models are given unlabeled data and allowed to discover patterns and insights without any explicit guidance or instruction [31].

Reinforcement Learning

Reinforcement Learning is a machine learning (ML) technique that trains software to make decisions to achieve the most optimal results. It mimics the trial-and-error learning process that humans use to achieve their goals [32].