As AI technology continues to evolve, it is reshaping cybersecurity, creating both new opportunities and challenges. According to Gartner and other analysts, generative AI (GenAI) intersects with cybersecurity in four critical ways: 

1. Defending against threats using AI
2. Preparing for AI-driven attacks
3. Securing AI application development
4. Managing the consumption of AI tools across the organization. 

Let’s explore each of these intersections in more detail.

1. Defending with Generative Cybersecurity AI

Using ML and AI defensively is not new. In fact, vendors have been talking about this for years, with a healthy mix of innovation and hype. (See our previous blog: AI-Washing at RSA). But the GenAI revolution has made the use of AI for security more tangible and directly interactive with users. In the past, we had to trust vendors about claims of using AI under the hood, not it’s easy to interact with security bots and copilots to design security strategies.

Organizations can use GenAI to improve threat detection, optimize resource allocation, and streamline operations, all while reducing costs. With its ability to process vast amounts of data quickly, GenAI enables:

• Advanced Threat Detection: AI models can identify suspicious patterns and anticipate potential attacks by analyzing anomalies in data, offering an early warning against emerging threats.
• Operational Efficiency: Automated workflows and prioritization of vulnerabilities allow security teams to focus on high-risk areas without being overwhelmed by routine tasks.

Using GenAI defensively is about enhancing resilience and staying ahead of attackers, but it’s essential to also understand the limitations and potential blind spots of AI-driven systems.

2. Preparing for “Attacks by” Generative AI

By their nature, AI-driven attacks are hard to detect, but make us all anxious. We’ve long suspected that large scale attacks were driven by ML/AI, now it seems clear that GenAI tools can be directly leveraged by attackers. From automated phishing campaigns to generating code for malware, GenAI introduces new threat vectors that organizations must guard against:

• AI-Powered Social Engineering: Attackers can use GenAI to create highly convincing phishing emails and social engineering scripts, making it harder to differentiate between legitimate and malicious messages.
• Advanced Malware Creation: Generative models enable the development of polymorphic malware that adapts to evade detection, posing a serious challenge to traditional security controls.

The risk of “attacks by” GenAI underscores the importance of proactive threat intelligence and constant vigilance to keep up with increasingly AI-driven adversaries.

3. Securing AI Development for Enterprise Initiatives

As enterprises accelerate their AI adoption, they often rush to build GenAI applications without fully understanding the expanded attack surfaces and unique security challenges these applications present. Traditional application security practices are a good starting point, but AI systems introduce unique vulnerabilities that need specialized defenses. Key areas of focus include:

• AI Discovery and Governance: As new teams of data scientists, often outside of traditional security channels, experiment, test, and develop AI uses, many organizations don’t have adequate visibility or security guardrails in place. Starting with the basics: Do you have policies around AI? Do you have approved platforms for developing AI capabilities? Do you know what models and datasets are in use and where they came from? Systems to discover, manage, and ensure basic compliance for AI projects are essential.
• Model and Data Integrity: AI models are highly sensitive to the data they are trained on. If an attacker can manipulate training data, they can potentially alter the model’s behavior or introduce biases. Data integrity measures, like controlled data pipelines and regular validation checks, are essential to prevent attackers from poisoning the model with misleading information.
• Intellectual Property Protection: AI models, particularly those designed for complex tasks, are valuable intellectual assets. If a model is accessed by unauthorized individuals, it could be stolen or reverse-engineered, exposing critical business strategies or proprietary processes. Ensuring secure access and encryption of model data helps protect these assets from cyber threats.
• Explainability and Compliance: AI models, especially in regulated industries, must be auditable to ensure their outputs can be explained and justified. Security frameworks for AI development should incorporate audit logs and model interpretability to track model decisions and confirm they align with regulatory and ethical standards. Without explainability, organizations risk regulatory non-compliance and ethical breaches, especially when AI models impact critical decisions.
• Lifecycle Management: Securing AI applications is not a one-time effort. Organizations need to monitor AI models for performance, adjust them based on new threats, and update security measures as AI applications evolve. By embedding security checks into each phase of the AI lifecycle, from training and validation to deployment and monitoring, organizations can better manage the risks associated with their AI assets.

Securing GenAI applications requires adapting security practices and frameworks to fit the unique needs of AI development. It’s critical to recognize that AI systems are dynamic, meaning that security measures must also be flexible and scalable as models evolve and new use cases emerge.

4. Managing and Monitoring GenAI Consumption

As organizations adopt generative AI tools more widely, particularly those embedded in existing applications, the management and monitoring of GenAI usage become crucial. Embedded AI features, like virtual assistants and intelligent data analytics, often operate outside the scope of traditional security controls, introducing new challenges in data privacy, access control, and governance. Addressing these challenges requires a comprehensive approach:

• Data Privacy and Compliance Monitoring: Generative AI tools, such as embedded chatbots or analytical assistants, often handle sensitive data, including customer information, proprietary insights, and operational data. It’s essential to monitor these data flows and ensure they meet regulatory standards like GDPR and HIPAA. Organizations must implement mechanisms to monitor AI data usage, protect against unintended data exposure, and ensure that only necessary data is fed into AI-driven applications.
• Granular Access Controls: Not every user within an organization requires access to GenAI tools. By implementing robust access management policies, companies can limit the exposure of sensitive information and prevent unauthorized interactions with AI systems. Role-based access controls (RBAC) and multi-factor authentication can help restrict access to AI features and limit the potential for accidental or malicious misuse.
• Ongoing Model Governance and Auditing: To maintain trust and security, embedded GenAI tools should be subject to regular audits. Organizations need a governance framework that assesses model accuracy, detects biases, and evaluates compliance with internal and regulatory guidelines. Regular audits help ensure that AI outputs are aligned with organizational standards and ethical expectations, which is especially critical as GenAI becomes more integrated into operational processes.
• Tracking User Interactions and Feedback: As employees and customers interact with AI-driven applications, it’s valuable to track these interactions to gather feedback and identify any areas where the AI may be underperforming or creating security risks. User feedback can reveal issues like data leakage, inaccurate predictions, or other unintended consequences of AI consumption. This continuous feedback loop allows organizations to make necessary adjustments and improve security measures for GenAI tools.

The management and monitoring of GenAI consumption go beyond traditional security. It requires a clear governance framework, well-defined policies, and an understanding of the unique security demands that embedded AI tools introduce. As the adoption of GenAI expands, a proactive approach to managing AI consumption will be crucial for both operational efficiency and security compliance.

As organizations look to harness GenAI, they must balance innovation with security. By focusing on these four intersections—defending with AI, preparing for AI-driven attacks, securing AI development, and managing AI consumption—security teams can secure their path to AI adoption while safeguarding their enterprises against emerging threats.