Unfortunately for a medieval king, pretty much everybody wanted to kill him pretty much all of the time.
Imagine a castle. Let’s pretend that the king wants to know if he could be assassinated. He orders a loyal noble to send some knights to try to break into the castle. He gives no information to those knights about the castle defenses. After all, the king thinks that what he needs is for them to pretend to be his enemies, and his enemies don’t have any of that information.
The knights are given a limited amount of time to attack the castle. They find a few weaknesses in the castle defenses, some of which the king already knew about and some he didn’t. Ultimately, they determine they cannot assassinate the king.
A few weeks later, the king is murdered. His enemies found a secret tunnel that the knights didn’t know about and used it to get to the king. The king knew about this secret tunnel; it was his escape route in the event of a siege. But he never told the knights about it.
The king represents your assets.
The castle represents your defenses.
The knights represent your security evaluators.
This is what black-box testing is like. When the king withholds information that would help the knights do their jobs, he reduces their ability to help him. Security testing that’s done with a black-box methodology intentionally limits information. All that does is limit value. Instead, you want to share information in order to maximize value.
The less information you share with your external security partner, the less value you get. The more information you share with your external security partner, the more value you get.
After you’ve figured out who to work with, you need to figure out how to work with them. There are two approaches: black-box and white-box.
Black-box testing is a security-testing methodology that limits information in a (flawed) attempt to replicate real-world conditions.
White-box testing is a security-testing methodology that maximizes information sharing to amplify the value of the assessment.
Many people mistakenly seek out black-box. Let’s dive into both and learn their differences.
In black-box, you don’t tell your security evaluators anything about how the system works. You don’t share diagrams or documentation. You don’t allow them to glean insights from your engineers.
This all comes down to information. Attackers have all the time in the world to attack. Yet, your available resources to defend against them are limited. You need to squeeze every advantage you can just to keep up. Information is one such advantage. You have it, and your attackers don’t.
Security is about finding and fixing flaws. However, black-box methodology is about limiting information, which limits value.
Limiting the supply of information is counterproductive. But it happens all the time because people mistakenly believe this approach somehow delivers “real-world results.” It does not do that. Instead, it causes these three bad things to happen:
Black-box Testing Flaws:
Security is about being able to prioritize risk. Black-box testing delivers incomplete results that do not empower you to prioritize risk.
In black-box testing, your partners don’t know how the system works, so they can’t recommend how to fix any issues they find. This handcuffs you. You might be able to figure out the solution on your own, however, (a) it puts the onus back on you to do the problem-solving, (b) you lose the many years of experience that your security partner has with solving problems just like yours, which means that (c) you get less value out of what you’re paying for.
Security is about prioritizing, understanding, and minimizing risk, coupled with fixing issues in order to get better. A black-box methodology does a poor job of all of these things.
Imagine instead that the king walks the knights around the castle, pointing out the features of the walls, moats, and turrets. He has the knights speak directly with the guards themselves and review the blueprints with the castle architect. As a result, the knights intimately understand the castle defenses and can probe accordingly for weaknesses.
That’s what white-box testing is like. In white-box, you collaborate closely with your security partner. You provide information that makes them faster, more efficient, and more effective. They review design documents, work with your engineers, and analyze source code. You treat them as part of your team. You show them how the system works. You explain workflows, access provisioning, user onboarding, and the future development roadmap.
Perhaps most importantly, you explain your business. You teach about your customers, what matters, and why. By understanding the business, your security partner knows how to think about your technology. It helps them understand your security vulnerabilities in the proper context. By understanding your mission, they help you secure it. White-box is about collaboration and collaboration delivers value. Here’s why:
White-box Testing Advantages:
Security is an uphill battle. You’re outmatched by motivated adversaries who have more time, money, and resources than you do. You need to squeeze maximum efficiency out of your investment.
Security is about finding vulnerabilities, fixing them, and adapting. These ultimately make you better. A white-box methodology is the best way to do that.
When it comes to the mission of finding and fixing your security vulnerabilities, white-box delivers far superior outcomes than black-box does. However, there are some cases where black-box is usually unavoidable:
Here’s an example of this contrast playing out in real life.
A client of ours wanted to test their newest application for security vulnerabilities. For years, they’d only ever asked us for white-box testing, but this time they asked for black-box.
Surprised, we asked their director of product security why.
“First of all,” he said, “I definitely want white-box—I desperately need to know what security vulnerabilities exist so I can fix them.” He explained, “But my boss promised a black-box test to his boss who then promised it to her boss. They didn’t understand the difference, but nevertheless, I must deliver it. It’s political now.”
This was quite the conundrum and one that commonly plays out in many companies. (Does this sound like your company, too, where business decisions drive bad security decisions? If so, you’re not alone.)
Together, we developed a plan: do both.
On the same system, we performed two security assessments: one black-box and then one white-box. This satisfied both the technical need and the political need, but it also delivered something else: a side-by-side comparison of outcomes.
(If you’re wondering how the director of product security came up with funding for two assessments: he made a compelling case for why they needed the side-by-side comparison to inform future decisions. His finance department agreed.)
First, we performed the black-box assessment. We allocated 200 person-hours and found four security vulnerabilities as a result. Of those, our client already knew about two of them. All of the time and effort invested in discovering these delivered zero value to the customer. The third issue turned out not to be a vulnerability after all but rather a misunderstanding. Remember: in black-box testing, information is intentionally withheld, which led to a misinterpretation of why certain inputs produced certain outputs. Because it was not actually a security vulnerability at all, this too was of no value to the customer. The fourth vulnerability was indeed a previously unknown, critical security vulnerability. This was very valuable to our customer. However, we couldn’t recommend how to fix it because we didn’t know how the system worked.
Outcome: 200 person-hours = 1 vulnerability and 0 fixes
Next, we performed the white-box assessment. Like the black-box assessment, we allocated the same 200 person-hours. Unlike the black-box assessment, where we found only one security vulnerability, in the white-box assessment, we found 21.
Yes, 21 important issues instead of just one!
All of them were previously unknown, critical security vulnerabilities. This was enormously valuable to the customer! All of these issues existed in the system while we did the black-box assessment; we just didn’t find them within the allotted time. Such is the power of information. For every single issue, we could recommend at least one remediation and, in some cases, more than one. Because information is provided in white-box, you equip your partner to recommend remediations for any vulnerabilities found. This, too, was enormously valuable to the customer.
Outcome: 200 person-hours = 21 vulnerabilities and 21+ fixes
Same effort. Same system. Different methodologies. Dramatically different outcomes.
This data overwhelmingly makes the point: methodology matters.
When the director of product security received this comparison, his eyes grew as wide as dinner plates. He literally shouted, “We’re never going black-box again!” And he hasn’t.
The metaphorical king lost his life because he didn’t understand the difference between methodologies, but now you do. Implement these ideas, and you’ll be able to better protect your most valuable assets. Start by reaching out to us about your security challenges.
Content adapted from: