A few years ago, my colleagues and I attended one of the several large cybersecurity conferences held across the country every year. Back then, Chat GPT was new and novel. Strangely, it still feels like it should have happened a decade ago given the fast pace of progress. The word on everyone’s lips was “AI” (Artificial Intelligence). It was a favorite topic of conversation for almost everyone, even though that wasn’t the focus of most of the presentations. Every fear was expressed, from how AI will destroy humanity, to how AI will take jobs, to how AI is a laughable failure at basic hacking. But what was apparent was that things were happening fast and no good framework/methodology for assessing “AI security” existed. So, we quickly developed our own in-house AI testing methodology which we continue to expand. In the public sphere, there have also been decent attempts at establishing a security framework. A notable one, is the OWASP AI guidelines for LLMs (Large Language Models) released in 2025.
But when dealing with something that is supposedly intelligent and expected to exceed human intelligence (some would argue that it already has) what does security even mean? For some it means goal alignment (making sure AI does the “right” thing), for others it means protecting against prompt injection, and yet for others it means ensuring that the AI is deployed in a secure manner. At every level of AI development there are security concerns that must be addressed. Security concerns are as follows:
- Model training: As the model gets trained, its creators provide it with positive or negative feedback which teaches the AI how it should “behave” (goal alignment). Developers also ensure that malicious data does not get used during training (data and model poisoning).
- Supply chain: Since AI is now also treated as a national security issue, supply chain attacks are a concern. Hence, the security of the full stack (hardware, software, and third-party components).
- Deployment: As software companies rush to integrate AI into all their apps, the opportunities for insecure deployments become endless.
All these security concerns are valid and are currently being addressed by the wider cyber security community. However, when thinking about the “big picture", it’s apparent that AI will have a larger impact on security than just some technical nuances. This is where things can become very philosophical. As security analysts, we often prefer to stick with the overly technical issues. Cross-site scripting, SQL injection, Cross-Site Request Forgery, and more. These issues will continue popping up for sure, but AI forces us to shift our paradigm as to how we approach security. In many ways, cybersecurity is a bit of a dark art and with AI it will most likely become ever more so. I personally believe that the most impactful security issues associated with AI will be born out trends; meaning how AI is used by both bad actors and everyone else. The following are a few ways that AI will impact the field of cybersecurity.
Vibe Coding: AI Technical Debt
Vibe coding is the practice of instructing AI to write software. During the first iterations of Chat GPT, developers could instruct it to write short scripts. However, with newer models such as Claude, developers are using AI more and more to write entire applications. Not only that, but thanks to AI, anyone with an LLM can create an entire app with a single prompt. In 2026, it is estimated that 42% of new code was AI generated but ironically generated 23% more bugs.
So, what’s the issue with this? It’s a term coined “technical debt”. Think of AI generated vibe code as a credit card. It is tempting to use the card to purchase all the nice goodies you see at the mall. Think of application features that developers desire as the “goodies”. As many of you may know, it’s all too easy to quickly rake up debt. Likewise, with an LLM it is all too easy to quickly add code and new features in an app. However, this debt must eventually be paid back. The debt that must be paid back describes the code review, long term maintenance, and fixing of bugs. This is often described as a “bottle neck” that slows down the pace of software development. As such, companies are tempted to circumvent this by either using another LLM to review new code, pushing out new features without review, or adding the phrase “…and make it secure” at the end of the prompt (LOL). This is a recipe for disaster. In practice, this means that we could see insecure code deployed throughout the cyberworld on a massive scale. Not only that, but much of this code will probably remain unreviewed. Also consider that the quality of agents used to generate code will vary. Someone with access to a free LLM might generate more insecure code than someone with access to a paid one.
Ultimately, what does this mean for security analysts? Well for one, security analysts are masters at “black box pen testing”. This is the art of finding security vulnerabilities in an application (e.g., a banking app) without knowing its internal workings. The pen tester uses the application from the standpoint of an external attacker and uses all tools at their disposal to probe it from the outside. This is a creative process that requires the pen-tester to be comfortable with the unknown while mapping out the app’s invisible architecture. Eventually, we may get to the point where a significant portion of applications will be a black box, even for the developers. We’ll be able to interact with the application but not know what’s “underneath the hood”. Therefore, it will be more important than ever for security analysts to find and fix security vulnerabilities in vibe-coded apps.
AI Agents Who Hack: Overwhelming the Defenders
Recently, Anthropic claimed to have developed an AI model that can hack like a pro. The release of this model, called Claude Mythos, was delayed due to claims by Anthropic that AI models “can surpass all but the most skilled humans at finding and exploiting software vulnerabilities”. This was met by the cybersecurity community with a healthy dose of skepticism. We won’t know for sure if the claims are accurate until the model is publicly released, which may happen in the near future. Either way, “AI that hacks” poses a significant threat to application security. This is due to a well-known asymmetry between attackers and defenders. In digital cyberspace, the defenders (aka, the app developers) must ensure that their app has zero security holes to prevent a breach. The attacker on the other hand only needs to succeed once. Without AI in the mix, this was already a massive issue in cybersecurity. This asymmetry has resulted in massive financial losses. For instance, it was estimated in 2025 that global cybercrime cost the world $10.5 trillion. To put that into perspective, the costliest Hurricane disaster in the US was Hurricane Katrina, which totaled over $198 billion dollars in damages. That’s about 53 Hurricane Katrinas in a single year of cyberattacks! With robot hackers on rise, this may only get worse. The asymmetry that was seen between human defenders and human attackers may be mirrored between AI defenders and AI attackers. AI attackers will have the advantage, especially if the defender’s code was vibe coded without review.
What role will human defenders and attackers play? Will humans simply sit on the sidelines and watch while AI bots battle in cyberspace like Thanos vs the Avengers? No. At least not now. If AI bots are directed to carry out attacks, it will likely be at the direction of other humans. According to some reports, in 2025 up to 87% of organizations reported encountering AI-driven cyberattacks. Although these attacks appear to be mostly scams that use social engineering, agentic AI is also on the rise. Agentic AI describes AI systems that operate autonomously. That is, they can analyze targets, find vulnerabilities, make decisions, and execute multi-step attack chains. Cybercriminals will likely work in tandem with AI. For ethical hackers like us, this means leveraging the same AI tools as the attackers. A seasoned hacker who can use an AI hacking tool effectively can be a powerful adversary or ally.
Social Engineering: the Elephant in the Room
Has anyone noticed recently or gotten the feeling that the internet feels more “personal”, but in an eerie or even creepy way? Ads have become more personalized, apps use more human verification systems, AI generated content populates social media, and AI chat bots dwell in every corner waiting to “assist” you. This is not a coincidence, in fact this trend towards a “new internet” has been coined the “Dead Internet Theory” by those expressing dread at an internet filled with artificial and fake content. Although this is part of it, the other part is a shift towards Hyper-personalization and “the definitive end of broad demographic targeting”. What this means in general terms is that app developers are shifting towards using AI to make content more responsive and interactive for the individual user. As Averi.ai puts it:
“AI is enabling what was previously impossible: content that morphs based on who's viewing it, product recommendations that factor in dozens of behavioral signals simultaneously, and even AI personas that simulate different customer types for campaign testing before you spend a dollar on deployment.”
Okay, so how does this relate to cybersecurity? Well, at the core of this hyper-personalization design is massive amounts of personal user data and AI processing of that data. Increasingly AI will be entrusted with sensitive user data to serve personalized ads, morph content to user preferences, and construct detailed psychological profiles. Although some organizations may trust their AI models, we found that AI can be easily social engineered into breaking its own guardrails to divulge sensitive data or instructed to carry out malicious attacks. Any sensitive user data that is leaked can then be used to devastating effect in attacks that take well-known social-engineering techniques to the next level. Already, we have seen organizations report more sophisticated phishing emails, SMS text messages, and even deepfake voice calls. Indeed, hyper-personalization is a double-edged sword.
As security analysts, we are also concerned with how that user data is stored and processed by both servers and AI models. Plus, the attack surface may be larger, since traditionally, sensitive data was stored on backend databases. Applications that use AI hyper-personalization may store sensitive data in at least two locations: backend databases and in the AI model training weights. Therefore, AI models must also be protected by many of same security features as traditional databases such as encryption at rest.
However, it is not as simple since unlike a database, an AI model uses that learned knowledge to make intelligent decisions. This is where it becomes important to implement guardrails to ensure that the model is behaving in a way that does not disclose sensitive information or enable bad actors to leverage the model’s training to carry out social engineering attacks. A model that is trained on sensitive user data can be dangerous since it may obtain an intimate understanding of the psychology and behavior of the application’s users. If the model is stolen or its guardrails are broken, the model itself could be weaponized against the application’s user base. This is becoming more of a concern as applications incorporate AI into hyper-personalized applications and customer service systems.
Due to this, security analysts will need to secure sensitive application data, test model behavior, and recommend incident response measures in case of a data-breach to protect users from AI-driven social engineering attacks.
Conclusion
New frontiers of security are emerging as AI becomes more integrated into our digital world. More than ever, organizations must carefully audit applications which incorporate AI. Although at surface level AI looks like a convenient shortcut, effectively automating many digital tasks, it is anything but. If anything, in security we’ve learned the hard way that convenience is often the enemy of a secure system.
If you’re looking for a third-party test from an organization who incorporates the crucial human element into all aspects of our business, talk to one of our ethical hacking experts today. We will be able to guide you through all the steps to building better, more secure systems.
