AI Meets Cybersecurity: Understanding LLM-based Apps
A Large Potential, Still Emerging, but a Profound New Way of Building Apps
Hello Cyber Builders šš»
Suppose you havenāt heard about artificial intelligence for a year. In that case, youāre probably living on another planet šŖĀ š But if you put aside the hype, itās hard to understand how it would change the cybersecurity industry.
I recently spoke with some friends who work at a software vendor. They were unsure how to approach the combination of cybersecurity and software applications. It is not always easy to understand new threats in this area.
Welcome to this article on AI and Cybersecurity. In this post, we will explore the integration of large language models (LLMs) into cybersecurity applications and the potential economic and security implications of this development. We will also look closer at LLM-based apps, their key components, and the threats and cybersecurity concerns that come with them.
So, whether you're a cybersecurity practitioner, an entrepreneur, or simply curious about the latest trends in the industry, read on to learn more about LLM-based apps and their impact on cybersecurity.
Generative AI - A Large Potential
In their quest to stay ahead, cybersecurity practitioners and industry leaders are now harnessing the power of LLM-based apps. If you are wondering what LLM ā Large Language Models ā are, you're not alone! Often interlinked with artificial intelligence, these models are trained on a diverse range of internet text, which enables them to generate human-like text based on the input they receive.
Why is this important? LLM applications hint at an enormous potential. They represent vast opportunities in the field of cybersecurity. In a recent paper, McKinzey analyzes the potential impact of generative AI on work activities and estimates the technical automation potential of individual work activities. The adoption of automation technologies is modeled based on the time required to integrate technological capabilities into solutions, the cost compared to human labor, and the time it has taken for technologies to diffuse across the economy. Generative AI will likely have the most significant impact on knowledge work, particularly activities involving decision-making and collaboration, which previously had the lowest potential for automation.
Moreover, according to Morgan Stanley, generative AI raises cybersecurity concerns, as it can be used to generate malware and facilitate hacking. Generative AI can also improve productivity by automating routine tasks and freeing up cybersecurity analysts to focus on more critical issues. Morgan Stanley estimates that this represents a $30 billion market opportunity and that task automation could enable savings of over $100 billion globally.
Cybersecurity Vendors are starting to embrace it.
The state of AI and LLM security will likely drastically differ in just six months. Moreover, it is too early to speculate who has the potential to become the winner in this space.
Says Ross Haleliuk in a recent substack newsletter where he reviews the latest startups of the space. Ross is right; who will be the winners is yet to be determined. Still, a gold rush exists to add LLM to existing apps or create newer versions of widely used apps.
The article "How Foundation Models Reshape Cybersecurity Tooling" explores the recent integration of foundation models (FMs) into cybersecurity tooling and how FMs can revolutionize how we interact with security tools. The article covers several categories of cybersecurity tooling, including search, code writing, vulnerability explanation, and incident response/threat intelligence.
The article notes that FMs have been integrated into tooling for nearly every domain, including design, FP&A, HR, sales, marketing, and software engineering. The technology has also made its way into cybersecurity products, with companies like Microsoft and Google leading the way in bringing the power of FMs to security tooling. The article discusses the implications for startups and adds to a growing list of predictions about where this technology will most impact the security industry.
The article also explores the potential for FMs to revolutionize pen testing, security reviews, and security-as-code generation. For example, the report suggests that FMs could be applied to bring true automation to pen testing, making it possible to identify meaningful vulnerabilities more efficiently. Similarly, FMs could be used to make security reviews more frequent and light-touch and to create alignment between all existing security tooling and infrastructure through the mediums of policy-as-code (PAC), detection-as-code (DAC), and infrastructure-as-code (IAC).
The authors note that while incumbents add generative capabilities to their products, the next massive cybersecurity company will likely leverage AI to solve a net new problem. The authors suggest that FMs create a standardized interface for describing and affecting the behavior of cloud systems in predictable, replicable, and auditable ways.
Before diving into LLM-based apps, letās look at what an application is today.
Classic Web 2.0 Applications: A Deeper Dive or a Reminder
Let's rewind a bit. We have been familiar with conventional Web 2.0 applications for decades - an era where interactive, user-centered design became the norm. These applications, typically single-page apps (SPAs), run within the web browser and interact with code hosted on a server.
Let's look at the critical components of these traditional applications:
Frontend šØāš»
This is the face of the application, what users interact with daily - think of everything you see when visiting a website or using an app. The front end typically includes navigation menus, buttons, drop-downs, forms, and more.
Backend Applicative š§
If the front end is the face of an application, the back end is its nerve center, handling the 'behind-the-scenes' operations. This includes server-side logic and computations, fetching data, and making things work by communicating with other parts of the application.
Infrastructure š
Then comes infrastructure, which includes elements such as load balancer and reverse proxies. The load balancer distributes network or application traffic across several servers to enhance performance and capacity utilization. Reverse proxies help retrieve resources on behalf of a client from one or more servers - serving as an intermediary for requests.
Database šļø
A crucial part of any web application! Hereās where all data is stored (think user information, activity logs, etc.). To simplify, if you were to write a letter online, a database is where it would store this letter for future access and modifications.
Common Features š
Last but not least, most classic Web 2.0 apps have several shared features, such as user authentication (login systems), credentials management (password changes), user profiles, and so on.
If you look at Cloud service providers such as AWS or Azure, they provide all the building blocks (except maybe Frontends) to build applications. New players are trying to simplify this architecture, such as Supabase.
Understanding LLM-based apps š§
LLM-based apps or Language models like GPT-4 are AI models that provide predicting abilities by utilizing the input of previous components. Essentially, they learn from an array of data and make informed guesses about what comes next. For a more detailed understanding, Daniel Miessler's description of AI architectures gives a clear overview here.
First, it's important to note that the functional components of the classic 2.0 applications are here to stay; they are integral to their operation.
In exploring what is contained within LLM-based apps, we can identify several key "sensitive assets." Here's the details
Berkeley's Gorilla LLM: https://gorilla.cs.berkeley.edu
LLM-based apps differ from classic Web 2.0 applications in integrating large language models (LLMs) and their key components, such as training data, models, state, policy, questions, and actions. While the functional components of classic Web 2.0 applications, such as the frontend, backend, infrastructure, database, and standard features, will remain integral to LLM-based apps, they will be modified to integrate the concepts and capabilities of LLM-based apps.
Threats & Cybersecurity Concerns in LLM-Based Appsš”ļø
A recent discovery by researchers at Carnegie Mellon University highlights a significant vulnerability in popular AI chatbots like ChatGPT and Google's Bard. This tricky technique (described in their research paper) involves adding a unique string of information to the end of a prompt that manipulates these bots into outputting damaging or inappropriate content. This flaw underlines an underlying issue with advanced AI chatbots, possibly posing obstacles to future AI implementation.
Pointing out this vulnerability followed a method known as adversarial attacks, leading to the bots being gradually pushed towards bypassing their limits. The fundamental issues pointed out by this vulnerability indicate it is common for highly advanced AI chatbots to deviate from the intended course.
Companies such as Google, OpenAI, and Anthropic have been alerted about the identified weakness and provided blocks to counteract the attacks described in the research. However, the generic adversarial attack used successfully against various proprietary systems serves as a reminder of the importance of open-source models to study AI systems and their vulnerabilities.
The attack highlights the potential for hackers to manipulate bots into delivering harmful or misleading information, leading to potentially severe consequences. Still, it must be noted that the outputs produced by the CMU researchers were generic and not harmful, yet companies should pay attention to securing their chatbots against targeted attacks.
In a previous post, I argued that Security Engineering is crucial in tackling emerging threats within Artificial Intelligence, and this new paper highlights it.
Following up on this issue, Forrester analysts provided an insightful take on potential cybersecurity threats related to LLM-based apps and their countermeasures:
You can find more about this in Forrester's analysis on how to defend your AI models.
I will probably zoom more in a later post on the Threats introduced by LLM-based apps.
Conclusion
In conclusion, integrating LLM-based apps into cybersecurity is a significant shift that can potentially revolutionize the industry. As we have seen, LLM-based apps offer vast opportunities for threat detection, risk assessment, and overall intelligence augmentation. However, they also come with potential cybersecurity concerns that must be addressed.
If you found this article informative, please consider subscribing to our newsletter to stay up-to-date on cybersecurity news and trends. Don't forget to share this article with your colleagues and friends interested in learning more about LLM-based apps and their impact on cybersecurity.
Laurent š
Appendices - Glossary
