Unfortunately for a medieval king, pretty much everybody wanted to kill him pretty much all of the time.
Imagine a castle. Let’s pretend that the king wants to know if he could be assassinated. He orders a loyal noble to send some knights to try to break into the castle. He gives no information to those knights about the castle defenses. After all, the king thinks that what he needs is for them to pretend to be his enemies, and his enemies don’t have any of that information.
The knights are given a limited amount of time to attack the castle. They find a few weaknesses in the castle defenses, some of which the king already knew about and some he didn’t. Ultimately, they determine they cannot assassinate the king.
A few weeks later, the king is murdered. His enemies found a secret tunnel that the knights didn’t know about and used it to get to the king. The king knew about this secret tunnel; it was his escape route in the event of a siege. But he never told the knights about it.
The king represents your assets.
The castle represents your defenses.
The knights represent your security evaluators.
This is what black-box testing is like. When the king withholds information that would help the knights do their jobs, he reduces their ability to help him. Security testing that’s done with a black-box methodology intentionally limits information. All that does is limit value. Instead, you want to share information in order to maximize value.
The less information you share with your external security partner, the less value you get. The more information you share with your external security partner, the more value you get.
Black-box vs. White-box Testing
After you’ve figured out who to work with, you need to figure out how to work with them. There are two approaches: black-box and white-box.
Black-box testing is a security-testing methodology that limits information in a (flawed) attempt to replicate real-world conditions.
White-box testing is a security-testing methodology that maximizes information sharing to amplify the value of the assessment.
Many people mistakenly seek out black-box. Let’s dive into both and learn their differences.
Black-box Testing
In black-box, you don’t tell your security evaluators anything about how the system works. You don’t share diagrams or documentation. You don’t allow them to glean insights from your engineers.
This all comes down to information. Attackers have all the time in the world to attack. Yet, your available resources to defend against them are limited. You need to squeeze every advantage you can just to keep up. Information is one such advantage. You have it, and your attackers don’t.
Security is about finding and fixing flaws. However, black-box methodology is about limiting information, which limits value.
Limiting the supply of information is counterproductive. But it happens all the time because people mistakenly believe this approach somehow delivers “real-world results.” It does not do that. Instead, it causes these three bad things to happen:
Black-box Testing Flaws:
- You Waste Time and Money. A black-box assessment limits the information supplied to your assessor. The only thing this does is require them to invest time — your time and your money — in obtaining information you could supply in minutes. You literally pay them to figure out the things you already know. Worse yet, it rarely results in the same level of knowledge that would be delivered if you just told them.
- You Don’t Test Your System; You Test Your Partner. The sneakiest drawback to black-box testing is that you aren’t testing the system; you’re testing your partner. But this is not the purpose of your test. You hire them to test you, not to test them. In black-box, you literally determine whether this security expert can compromise this system within this amount of effort.
- You Get Low-Value Results. If vulnerabilities aren’t found, it does not mean they don’t exist. If some vulnerabilities are found, it doesn’t mean all of them were found. You also don’t get helpful remediations.
Security is about being able to prioritize risk. Black-box testing delivers incomplete results that do not empower you to prioritize risk.
In black-box testing, your partners don’t know how the system works, so they can’t recommend how to fix any issues they find. This handcuffs you. You might be able to figure out the solution on your own, however, (a) it puts the onus back on you to do the problem-solving, (b) you lose the many years of experience that your security partner has with solving problems just like yours, which means that (c) you get less value out of what you’re paying for.
Security is about prioritizing, understanding, and minimizing risk, coupled with fixing issues in order to get better. A black-box methodology does a poor job of all of these things.
White-box Testing
Imagine instead that the king walks the knights around the castle, pointing out the features of the walls, moats, and turrets. He has the knights speak directly with the guards themselves and review the blueprints with the castle architect. As a result, the knights intimately understand the castle defenses and can probe accordingly for weaknesses.
That’s what white-box testing is like. In white-box, you collaborate closely with your security partner. You provide information that makes them faster, more efficient, and more effective. They review design documents, work with your engineers, and analyze source code. You treat them as part of your team. You show them how the system works. You explain workflows, access provisioning, user onboarding, and the future development roadmap.
Perhaps most importantly, you explain your business. You teach about your customers, what matters, and why. By understanding the business, your security partner knows how to think about your technology. It helps them understand your security vulnerabilities in the proper context. By understanding your mission, they help you secure it. White-box is about collaboration and collaboration delivers value. Here’s why:
White-box Testing Advantages:
- You Make the Best Use of Your Time and Money. You want the testing effort focused on using the information, rather than on finding the information. When you supply information, you get a shortcut that your enemy doesn’t have. This helps skilled security professionals quickly zoom in on where the problems might be and then probe from there.
- You Can Make Decisions Confidently. White-box is about receiving information — the more you have, the more confidently you make decisions. You can confidently communicate risk to your board, customers, shareholders, and employees. A white-box methodology delivers all of these, whereas black-box delivers none.
- You’ll be Able to Solve Security Problems in Ways that Advance Your Business. A security flaw that affects core business functionality is much more severe than one that doesn’t. By accounting for technical issues in the context of how they relate to the business, you resolve vulnerabilities in ways that support your company’s mission.
- You Get Better. A white-box methodology helps you not only find security vulnerabilities but also understand them and learn how to fix them. These insights help you evolve so you can avoid similar vulnerabilities in the future. They help developers enhance their skills. They transfer knowledge from your security partner to your in-house teams. They help you get better.
Security is an uphill battle. You’re outmatched by motivated adversaries who have more time, money, and resources than you do. You need to squeeze maximum efficiency out of your investment.
Security is about finding vulnerabilities, fixing them, and adapting. These ultimately make you better. A white-box methodology is the best way to do that.
Why Would Anyone Choose Black-box?
When it comes to the mission of finding and fixing your security vulnerabilities, white-box delivers far superior outcomes than black-box does. However, there are some cases where black-box is usually unavoidable:
- When dependencies are involved. Applications commonly integrate with other technologies, such as third-party libraries, shared components, or other aspects of the supply chain. Whenever the scope includes another company’s technology, any testing on those components will almost certainly be done black-box.
- When it’s security research. This is almost always done black-box because the company being researched usually doesn’t know it’s happening. Even when they do know, they usually don’t want to participate or share information. In either case, black-box is usually the only way forward.
- When the goal is to test for reconnaissance rather than for vulnerabilities. If you want to evaluate how easy it is for attackers to gather information, that’s a valid thing to test via black-box. You’d identify processes that leak information and, as a result, improve them.
Our White-box vs. Black-box Case Study
Here’s an example of this contrast playing out in real life.
A client of ours wanted to test their newest application for security vulnerabilities. For years, they’d only ever asked us for white-box testing, but this time they asked for black-box.
Surprised, we asked their director of product security why.
“First of all,” he said, “I definitely want white-box—I desperately need to know what security vulnerabilities exist so I can fix them.” He explained, “But my boss promised a black-box test to his boss who then promised it to her boss. They didn’t understand the difference, but nevertheless, I must deliver it. It’s political now.”
This was quite the conundrum and one that commonly plays out in many companies. (Does this sound like your company, too, where business decisions drive bad security decisions? If so, you’re not alone.)
Together, we developed a plan: do both.
On the same system, we performed two security assessments: one black-box and then one white-box. This satisfied both the technical need and the political need, but it also delivered something else: a side-by-side comparison of outcomes.
(If you’re wondering how the director of product security came up with funding for two assessments: he made a compelling case for why they needed the side-by-side comparison to inform future decisions. His finance department agreed.)
Black-box Assessment
First, we performed the black-box assessment. We allocated 200 person-hours and found four security vulnerabilities as a result. Of those, our client already knew about two of them. All of the time and effort invested in discovering these delivered zero value to the customer. The third issue turned out not to be a vulnerability after all but rather a misunderstanding. Remember: in black-box testing, information is intentionally withheld, which led to a misinterpretation of why certain inputs produced certain outputs. Because it was not actually a security vulnerability at all, this too was of no value to the customer. The fourth vulnerability was indeed a previously unknown, critical security vulnerability. This was very valuable to our customer. However, we couldn’t recommend how to fix it because we didn’t know how the system worked.
Outcome: 200 person-hours = 1 vulnerability and 0 fixes
White-box Assessment
Next, we performed the white-box assessment. Like the black-box assessment, we allocated the same 200 person-hours. Unlike the black-box assessment, where we found only one security vulnerability, in the white-box assessment, we found 21.
Yes, 21 important issues instead of just one!
All of them were previously unknown, critical security vulnerabilities. This was enormously valuable to the customer! All of these issues existed in the system while we did the black-box assessment; we just didn’t find them within the allotted time. Such is the power of information. For every single issue, we could recommend at least one remediation and, in some cases, more than one. Because information is provided in white-box, you equip your partner to recommend remediations for any vulnerabilities found. This, too, was enormously valuable to the customer.
Outcome: 200 person-hours = 21 vulnerabilities and 21+ fixes
Same effort. Same system. Different methodologies. Dramatically different outcomes.
This data overwhelmingly makes the point: methodology matters.
When the director of product security received this comparison, his eyes grew as wide as dinner plates. He literally shouted, “We’re never going black-box again!” And he hasn’t.
The metaphorical king lost his life because he didn’t understand the difference between methodologies, but now you do. Implement these ideas, and you’ll be able to better protect your most valuable assets. Start by reaching out to us about your security challenges.
Content adapted from:
