When it comes to testing applications for security vulnerabilities, terms are used incorrectly all the time. If you don’t realize it’s happening, it can have dire consequences.
Pretend that your cat is sick. You call the veterinarian, explain the symptoms, and the vet gives you advice. However, your cat doesn’t get better. So you bring your cat to the vet’s office. When she asks to see the cat, she’s met with a surprise: it’s not a cat, it’s a horse.
That’s why the treatment wasn’t working: a cat and a horse are not the same things. They require different treatments.
Yes, this metaphor is ridiculous. But there’s truth in absurdity. Words matter. When talking about security, people are often as ridiculous as saying “cat” when they mean “horse.”
Different security terms mean different things, yet they’re often used interchangeably. They should not be. They’re different things!
You may not even realize this problem exists. It’s hard to tell the difference, but it’s important that you do. Otherwise, you might invest in security that doesn’t meet your goals.
They’re Not the Same Thing!
“Security assessment” is an umbrella term for all varieties of security testing. The most commonly referenced one is “penetration testing.” That has become a catch-all term. Unfortunately, it’s misleading.
True penetration testing is a tactical service suitable only for robust, hardened, thoroughly tested systems. However, when you ask for “penetration testing,” many providers tend to deliver something else entirely: lightweight vulnerability scanning. And if you weren’t already frustrated enough, there’s this: what you actually need is usually something completely different from either of those things. You probably need manual, white-box, vulnerability assessments.
Those are three exceedingly different things. Each entails different approaches, levels of effort, cost, and outcomes.
Your security struggles are hard enough as it is. The last thing you need is to get the wrong outcomes simply because terms got mixed up. The marketing nonsense has gotten so bad that it’s challenging to distinguish what’s what. The reason for this is simple: proper penetration testing is badass! It pits a skilled expert against your defenses in a win-or-die mission to see if you’re secure. Any marketer would want their security service to sound like that! Unfortunately, it’s also very hard to do (and isn’t even appropriate for most systems anyway). As you learned earlier, there’s a skills shortage, so there simply aren’t enough people with enough expertise to do this properly. So what naturally happens is that other approaches—those driven largely by tools instead of experts—simply adopt the terminology. The marketing logic is pretty straightforward: if it sounds like the service people want, then people will probably buy it.
This term confusion is so widespread that regulatory bodies and even your customers may explicitly require “penetration testing,” which drives you to seek that specific service by name. But, of course, they’re usually misusing the term, too. What they really want is to ensure you have solid security and continue to improve it (as you’d get with vulnerability assessments). They’re usually not looking for something that just scratches the surface (as you’d get when your “penetration testing” winds up being just vulnerability scanning instead). Yet, they’re unwittingly directing you with the wrong terms.
The deck is unfairly stacked against you. There are lots of attackers, attack surfaces, and unknown vulnerabilities, and the very services that address this chaos are themselves unclear.
That’s a lot to deal with.
If you want to have a chance at finding and fixing your vulnerabilities, defending against attackers, and proving that your app is secure, you absolutely must distinguish these terms. That’s how you make sure that you get the testing you need. Because it’s not the term that matters; it’s the outcome.
What Is Penetration Testing?
Penetration testing seeks a binary outcome: did you or did you not achieve a specific end result?
Yes or no. That’s it.
Often referred to as a “pen test” or sometimes “red teaming” (which is actually something different; red teaming tests your security team’s response capabilities), penetration testing is a time-constrained effort to measure a single outcome. For example, a penetration test might seek to determine “could an attacker escalate basic user privileges to admin rights?” The outcome is either yes or no. There is no other outcome.
Penetration testing is like a crash test. When auto manufacturers want to understand the safety implications of a specific situation (such as a frontal collision), they run a crash simulation. They literally crash the car into a wall to see what happens. As a result, they have a clear answer about how the system performs in that specific situation.
Penetration testing is about depth in a specific area. It usually focuses on known classes of vulnerabilities rather than on finding new zero-days (the type of catastrophic vulnerability that you have literally zero days to fix because it’s exploitable in the wild right now). However, when you say you want “penetration testing,” there’s a good chance you’re actually looking for something else: you need to learn what you don’t know, find vulnerabilities across many attack surfaces, understand how severe they are, and fix them. Penetration testing is not designed to do that.
Penetration testing is great when you have a mature, robust, well-tested defense, and you want to determine if that defense still stands up to a simulated attack. It’s appropriate after you’ve invested heavily in vulnerability assessments, code reviews, and other security testing. It’s a reality check, performed periodically, to measure how you’re progressing in your relentless pursuit of becoming secure.
What Is Vulnerability Scanning?
Vulnerability scanning is running an automated tool that looks for common vulnerabilities that are known to exist. The goal is to quickly and inexpensively find basic issues, including checking for unpatched vulnerabilities. Given that running a scanner is one of the first steps your attacker will take, it’s a good idea for you to do this, too. You want to see what they see.
Vulnerability scanning is great when you need to find common vulnerabilities or find unpatched vulnerabilities. It’s great for gathering information to use in a broader security assessment. It’s good when you need to keep timelines, effort, or cost to an absolute minimum (with the understanding that these tradeoffs also limit the value of what you get as a result).
It’s like the diagnostic tool that mechanics use when the “check engine” light comes on in your car. The tool scans for known issues, spitting back readable codes. It’s easy, inexpensive, and quick. However, it may report false positives, and the codes don’t always tell you what the root cause is. This isn’t a comprehensive way to evaluate vehicle safety.
Penetration Testing vs. Vulnerability Scanning
Here’s what’s crazy about the term confusion surrounding penetration testing: even if you didn’t realize it, you technically asked to crash-test your car, but then you were sold a diagnostic scan of what’s causing the “check engine” light. Those things are pretty different! Sometimes it’s easy to tell the difference in service descriptions; the method will literally describe an automated scan. Other times, it’s not obvious because the method will mention a manual component, but that only references the work done to remove false positives. Sometimes descriptions are even intentionally misleading. If you’re unsure how to tell the difference between what you need and what’s being sold, here are a couple of ways to sniff out the difference:
- Price. Vulnerability scanning usually costs around 10–20 percent of vulnerability assessments (which we’ll discuss in a moment). Not 20 percent less, 20 percent of (or said another way: 80 percent less). Two dollars instead of ten dollars.
- Timeline. Vulnerability scanning is usually done in a few days rather than the weeks that vulnerability assessments require.
In the home robbery metaphor, vulnerability scanning is like the thief walking up to your front door and trying the handle. He knows that many people don’t lock their doors, so he’s checking to see if you locked yours. And that’s about it. He wouldn’t try to pick your lock, guess your garage keypad combination, or determine how to get onto your roof and enter through a skylight. He certainly couldn’t tell you anything about the security measures inside the house.
Think of it this way: Tools are useful and should be part of your strategy. Tools should not be your entire strategy.
You must go past the basics to find your most important vulnerabilities. The zero-days, custom exploits, and unknown unknowns (flaws so unexpected you don’t even consider them) all require a higher level of effort and expertise. They require intuition, manual investigation, and an attacker mindset. You want real humans reviewing your code and hacking your system.
You can’t get these with vulnerability scanning. You get all of them with vulnerability assessments.
Why You (Probably) Need Vulnerability Assessments Instead
Vulnerability assessments are comprehensive, rigorous, manual efforts to discover security vulnerabilities, assign severity ratings to them, and determine how to fix them. The objective is to find as many as possible and remediate them. As a result, you understand and reduce risk.
Vulnerability assessments consider assets, attackers, attack surfaces, workflows, whole system configuration, and your future development roadmap. They adopt an attacker’s perspective to look for both common vulnerabilities and custom exploits. They consider your business itself because vulnerability severity is influenced by factors such as who uses the app and how you make money.
Vulnerability assessments go for breadth. Instead of a single crash test or a quick diagnostic scan of your car, it’s more like automotive safety engineering: a complete review where testing looks for all possible faults in all safety systems, from seat belts to airbags to driver-assisted braking—and how all those systems work together. Then a report is compiled with all findings quantified and ranked by criticality. A plan is outlined for how to fix each flaw in order to improve overall vehicle safety.
Vulnerability assessments are great for both well-hardened systems as well as those still figuring it out (and everyone in between). They are the most effective way to reduce the likelihood that a real-world attacker can exploit your system.
If you’re currently doing security testing, but you’re not sure whether it’s vulnerability scanning or vulnerability assessments, just look at the reports you’re getting. You can see a glaring difference.
The beauty of vulnerability assessments is that although they require time, effort, and resources, they produce valuable outcomes as a result. They help you find and fix issues. They help you get better.
What to Do When You’re “Not Ready”
It’s fairly common to hear people say, “We’re not ready. I know you’ll find vulnerabilities.” As a result, they delay security assessments, because they think it would be a waste of money to pay someone to tell them things they already know. I empathize with that concern; everyone wants to be smart about how to spend their limited resources, especially time and money.
The good news, though, is that you can get the help you need without wasting those resources. The solution is simple: just tell your security partner about the issues you’re aware of! They’ll include those in the context of their recommendations but focus their effort on other things. As a result, you’ll get the remediation plan you need without wasting time, money, or effort on the things you already know about. (Remember: you want to do white-box testing. This is a great example of why.)
You want to avoid the unpleasant side effect of the “we’re not ready” concern: when you delay security, you delay getting better. You allow issues to linger. You remain vulnerable longer than you should. Instead, just get started. That keeps you on the path to security excellence. You’ll always have some degree of vulnerabilities to address, and your security partner should be able to help you address them and get value out of your investment (assuming you chose the right partner). Don’t let the fact that you know about security issues be the reason you don’t fix them!
The whole point of the vulnerability assessment is to help you fix your security issues and improve development practices. That’s where we come in. Make the first step to securing your application by contacting us about a vulnerability assessment.
Content adapted from:
