During my time at ISE, I have scoped out a lot of pen tests. I’ve scoped so many, the details start to blur. But while the fine details may fade over time, I see a few problems pop up over and over like clockwork.
“I’m talking to you because our previous pen test wasn’t accepted by our auditor. It wasn’t enough.”
“After our previous pen test, many critical vulnerabilities were discovered after the fact.”
“When we saw our last pen test report, we realized that several critical components were not tested at all.”
“I’m not sure I trust the results of our previous pen test. They just didn’t find very much.”
This feedback loop is a little frustrating for someone like me who does this kind of work, so imagine how someone stuck with these less than ideal results might feel. If you didn’t really achieve the goal of a pen test, what was the point? Even if the test was “cheap”, it probably had enough of an investment that it required budgeting and approval. Needless to say, these are the consequences that are less painful. Pen testing costs are cheap compared to a major attack or breach.
Let’s assume the people you are working with are competent testers with proper certifications and knowledge. They aren’t selling you snake oil. The reason this happens is that your testers aren’t going far enough, or looking in the right places, or operating with all the knowledge they could to get to the important results as effectively as possible. In other words, the project was not scoped properly.
This type of mistake isn’t unique to penetration testing or technology projects as a whole. I’m sure as you are reading this you can think of a few projects that had similar problems in a totally different context. Maybe you signed a new contract to get new gutters for your home and the installers failed to put in a proper downspout that led to flooding. Now they want to charge you a lot more to do the thing they should have done in the first place. Maybe you are so irritated about that gutter project to the point you are referencing it in a blog years later.
Anyway, we shouldn’t be surprised this happens, because humans make mistakes in all types of businesses. It’s a learning opportunity! We’ll focus specifically in this blog on some common issues in scoping and what you can do to avoid these kinds of mishaps when it comes to planning for your next pen test.
Unclear or Unfocused Goals/Objectives
Imagine walking into your doctor’s office and asking them to remove one of your kidneys. Maybe Dr. Nick Riviera (shout outs to The Simpsons) would just ask for your insurance information and begin the operation, but a good doctor would ask you why you want to do that (and hopefully ask you to reconsider).
Projects need goals. There has to be a reason driving a project such as a pen test. But sometimes I’ll be chatting with someone and they don’t really have a clear reason for doing the test. They just want to know what it costs or the logistics, like they are shopping for pants instead of a service that could find a critical issue.
Pen tests are so important that they must be done correctly. Not having a serious discussion with your testers about the key goals will inevitably lead to disappointment.
There are many ways to think about goals. Is this driven by compliance or the need to please a specific stakeholder? That’s fine as an objective, but even so, there’s more that can be done if you want to hit the mark, and you definitely want to make sure that an auditor doesn’t reject your results.
Here’s a short list of potential goals you might consider. This is by no means exhaustive.
- Protecting Sensitive and Valuable Assets - This should always be a goal. Step back from the test and think about what the organization cares about. Not just money, but data, reputation, business partnerships and other things that you care about protecting. Your testers should know. Vulnerabilities really only matter in the context in which they can impact your assets, and nobody knows the value of them better than you.
- Ensuring Security Post-Change - Whether you have developed a new function or component of an application, or have expanded your corporate network, or doubled in company size, or whatever, these kinds of significant changes should be in consideration the next time you do a test. Many organizations want to run it back with tests, and sometimes that’s appropriate, but often these significant changes lead to new vulnerabilities. Again, your testers should know.
- Gaining An Outsider or Real Life Attacker Perspective - Good internal testing is important, but sometimes a goal can be to get an independent view of security with an attacker’s mentality. It’s not that your internal team isn’t smart or doesn’t know anything about security, but defensive security and offensive security are different. Many of the clients we do testing for find that their internal teams grow in security knowledge as a result of working with us, which is a side benefit of performing a good test with a 3rd party outside of the results.
- Test Your Resilience - There is no 100% secure system. Everything can be broken. But it’s good to know how far your defense can bend before it breaks. It’s also impossible to invest unlimited time and resources into testing and security, but these kinds of exercises can help right size investments in security to match a reasonable risk profile.
There are definitely more goals and perhaps there are some that only you could even know. Spend some time thinking about it before looking for a pen test. It’s worth it.
Limited Documentation and Information During Scoping
When you visit the doctor, they pull your chart. They want to see your medical history, know what medications you are taking, and so on. That’s because they want to focus on treating you and not asking a million questions, as well as making sure they don’t kill you with the wrong medication!
Many people looking for pen tests don’t want to share or gloss over important information. They think it should be the tester’s job to figure out the target. You can do that, but really you are just slowing down the process or potentially setting up your assessment for failure. While there is a place for black box testing methodology (very limited or no documentation), for most of the goals I mentioned previously, it’s better to go with a white box test where your testers understand how the target works.
Beyond just deciding on a methodology, it’s important that you truly articulate the details of what is being tested. Don’t assume your testers will be able to identify every function, endpoint, API, or device. Don’t assume that your testers will understand things like how data is segregated in a system, or what data in that system is actually sensitive. If there is redundancy in place, don’t assume that your testers will identify that. The way to get the best results from your pen test is transparency and clarity. When I do a scoping discussion, the way I usually put it to the person I am talking with is to imagine that they just hired me to work on their team, and not as someone scoping a pen test. If it was my first day on the job, what would I need to know about the system I am working on? Usually, this puts the people I work with in the right mindset.
Maybe you have reasons for being less transparent. That’s fine, but if one of the major objectives of your pen test is to find the most significant vulnerabilities in the most efficient way, you want to be as open as possible.
Access Misunderstandings, Unclear Boundaries and False Assumptions
I wanted to group these together, because fundamentally they are all about the testers and the person wanting the test being on the wrong wavelength. It’s miscommunication. Once again, we shouldn’t be surprised this happens. This is what people do. The good news is that this is everybody’s fault! Just kidding (sorta).
It is worth the time for testers, up front, to ask a lot of questions about the target. If there are areas where testing needs to be limited, those challenges should be shared. Keep in mind that many people who decide to hire a pen testing firm don’t know about ethical limitations, like what is OK and not OK for a tester to mess with. A good tester should point these things out up front to avoid any confusion.
On the stakeholder side, don’t be afraid to ask questions. You should try to understand as best as possible, even if you are a layperson, what your testers are doing. Perhaps you won’t understand the finer technical points of ethical hacking, but your testers should be able to give you simple language to distill what steps they are going to take as part of their test.
When I review a scope of work directly with a client, I take about 15 minutes or so to cover the following areas to try and avoid misunderstandings:
- Key objectives
- A brief description of the general methodology we will be taking
- Assets, both broadly as well as calling out any specific assets that have particular impact
- A broad review of targets including components of a web application such as APIs, DBs, mobile apps, etc., or, in the case of a network, a general summary of endpoints, locations, and key infrastructure components
- Areas of testing and major steps, while still giving a little bit of latitude to my team to improvise based on how the test goes
- How remediation testing works to validate fixes
- Any caveats about areas we won’t be covering or other potential “gotchas”
- The team working on the project, their roles, and a broad schedule to set timing
- Access and documentation needs
- Expectations on deliverables, including any custom needs for reporting, such as Executive Summaries or particular call outs
That might seem like a lot but it’s not difficult to get through this in a brief discussion. I encourage lots of questions and emphasize that this can all change based on feedback. This scope of work is MY idea of what I think a test should look like, but I am not perfect, and I also don’t know what they care about as much as they do. It’s GOOD to change the scope of work up front. Maybe this is my bias, but these discussions are as important as testing itself.
If you are working with testers long term, you should make sure to have these discussions regularly. What makes sense for testing today may not make sense a month from now. The only constant is change when it comes to security and pen testing.
Budget Driven Decision Making Leading To Shallow Scoping
Let me be very clear: You should establish a budget for pen testing and other security efforts. It shouldn’t be a blank check. But like anything else we spend significant money on, we should look at Return on Investment (ROI). Establishing a budget is outside the scope of this blog, but to keep it simple, the budget should reflect the value of the assets you wish to protect (you can read this blog to learn more about budgeting specifically.
Here’s where the problem often starts. Someone responsible for the decision on hiring a pen tester starts shopping for testers, and they go to each of them and ask for a bid. They might share some details I mentioned earlier, but it’s something like “I have this SaaS product I need tested in September. Here’s the URL. How much would it cost and when can you get started?” which they go on to ask 5 different companies through 5 identical web forms.
By all means, get multiple bids. That’s a good idea. The problem is the framing of the request. Depending on who is on the other side of that question, you are going to get a different reaction. You’ve just asked a very broad question that provides no context, and if you are having this conversation over email, you are going to be subject to the perspective of the person answering. Some will be order takers and give you no questions or friction. Others (like me) will approach it far more consultatively.
Even if you are trying to get a sense of price up front, it’s better to ask for a range, but really you should be trying to have a broader conversation about the items I mentioned earlier. Talk with the companies you are considering working with about what the goals are, some broad details about the tech, and so on.
If budget needs to be a part of the conversation, that’s fine. Establish what your budget is (you can also throw out a range if you don’t want to give too much away) as a constraint. I welcome this kind of information. There is always more hacking that can be done, but when we establish a budget, along with goals, we can hammer in on what kind of testing should actually be done, instead of just racing to the bottom on a bid, or misplacing focus in order to hit a budget number. This is about ROI. How can we take this investment and get the most out of it for security?
Your real goal should be to match your budget with the offering that best hits the goals and targets of the test, not whatever is cheapest. Besides, there are clever ways to stretch out a limited budget, like shifting focus of each test in order to cover a broader attack surface over a long period of time. To put it another way, you just don’t want to shop for pen tests like it is a commodity. You don’t want a rice and beans pen test. You want an experience that is like a dish crafted to your specific taste. You can still do that on a budget. There’s nothing wrong with rice and beans, but after eating rice and beans when you wanted steak, you probably feel like you didn’t get exactly what you wanted.
Conclusion
There are other problems that occur during the scoping process, but these were a few big ones that I see all the time. While achieving a perfect test is probably impossible, the more important point is to have a test that is in sync with your goals. Sometimes bad outcomes happen with security. Novel vulnerabilities can be discovered that are exploited before you have a chance to defend against them. You want to avoid the consequences that ARE avoidable, like missed targets, lack of depth of testing, or misused effort. All of this can be avoided through proper scoping.
Yes, scoping takes time and effort not only from your testers but also your internal team. But I promise you that this investment of mental energy, communication, and a little bit of time is well worth it. You will have a better test as a result! If you want to learn more about how internal and external teams can work together to get the best results from testing, check out this blog.
To recap how to avoid scoping mishaps:
- Before looking for a pen test resource, spend the time internally to figure out what your goals are. It doesn’t have to be complicated, but your goals should be beyond “find the vulnerabilities”. Think of it in business terms.
- Bring as much information to your testers as possible. The more transparency and detail, the more time will be focused on testing and identifying potential vulnerabilities.
- Miscommunications are scope killers. Don’t assume that your testers understand every last detail about your targets. Spend the time to go through boundaries, unique elements of your target, points of emphasis, and so on. It’s better to communicate too much than not enough. I’ve never complained that someone gave me too many details in scoping. I welcome it.
- While having a budget is good, look at this as an investment first. Treating a pen test like a commodity to get the best price on, like you are shopping different dealerships for the same car, is going to inevitably lead to shortcomings. At best, it will lead to a generic test. At worst, it will leave areas of testing uncovered that could lead to an incident. Put your money in the areas that will lead to the most ROI for security, not the cheapest provider.
Hopefully you found something of value from this blog. Have you had any testing challenges from scoping problems? Want to talk about it? I’d like to learn more and see how ISE can help. Check out our Ask An Expert form and we can have a chat. Thanks for reading!
