Effective security management relies on consistent disciplines like maintaining technology, incorporating security into design, and regular assessments to identify vulnerabilities. But every so often the security world is shocked by a newly identified critical vulnerability that requires security teams to snap into action and react quickly. In 2021, the Log4Shell vulnerability within the Log4j Java logging framework shocked the world, right before the holiday season. Many security experts called it the worst vulnerability of all time because it was both easy to exploit and very impactful.
Depending on your perspective and role in the security management landscape, Log4Shell probably had a major impact on your organization, as it reportedly impacted 93% of enterprise cloud environments. At ISE, we not only had to identify quickly how this vulnerability could impact us, but also how it could affect our clients.
An important element of proper security management is building your skills by learning lessons from past incidents. So, one year later, we want to take this opportunity to go back and revisit this historic vulnerability and come up with some takeaways to make us all more secure in the future.
What is the Log4Shell vulnerability? Why was it so bad?
For the purposes of this article, we will keep this explanation at a high level. Originally written in 2001, Log4j is an open-source logging framework that developers use to log data within Java based applications, which is extremely common in the world of application development. It is important to recognize that this vulnerability goes after devices that are practically everywhere, from personal devices to enterprise applications, smart TVs, and other non-traditional computing devices.
Log4Shell allowed an attacker to inject arbitrary, malicious code into an application, which could be used for a variety of things, such as gaining control of a device for follow up attacks. This could include ransomware deployment, cryptocurrency mining, data exfiltration, and other attacks.
The vulnerability was privately disclosed to the Apache Software Foundation on November 24, 2021, with the first fixes coming on December 6, 2021, followed by public disclosure of the vulnerability three days later on December 9th. It is thought that this vulnerability had existed since 2013 and it is impossible to know to what extent it was exploited until late 2021. Several security researchers, including some at Cloudflare, found evidence that testing for the exploit was happening before public disclosure, so it is reasonable to believe that damage had been done well in advance of the response by the security community .
The concern and confusion about Log4Shell extended to businesses, governments, and organizations all around the world. From our perspective in the cybersecurity field, this was something that needed to be addressed immediately for everyone impacted. For large corporations and governments, security incidents because of Log4Shell could be devastating on an entirely different scale. After all, this vulnerability came not too long after the Colonial Pipeline incident, which caused significant problems and had many question the security of the United States’ infrastructure.
The United States government set a deadline for December 24, 2021, for contractors to patch vulnerabilities, and the Federal Trade Commission said they would pursue companies that did not take reasonable steps to address Log4j in early 2022. Similar measures were taken by other governments across the planet.
Large technology companies like Amazon, Google, and Microsoft noticed impacts as well. Microsoft said on January 3, 2022 “Exploitation attempts, and testing have remained high during the last weeks of December. We have observed many existing attackers adding exploits of these vulnerabilities in their existing malware kits and tactics, from coin miners to hands-on-keyboard attacks” and “Sophisticated adversaries (like nation-state actors) and commodity attackers alike have been observed taking advantage of these vulnerabilities. There is high potential for the expanded use of the vulnerabilities.” 
If you want to understand why the security world was so alarmed at the time, step back a moment and consider the implications. By late 2021, even without the pandemic, we were heavily reliant on our devices and applications for our day to day lives, and it is almost guaranteed that among the dozens of apps we use on our PCs, smartphones, tablets, and connected devices that something would be impacted by this vulnerability. Extend that logic out and think about the partners and vendors you use along with the information and control they have over your assets. They are impacted too. So, this issue is out there, it impacts everyone, we all need to act on it immediately, and even if we try our hardest, we can’t control all of the ways it may impact us.
What was it like to experience?
As mentioned earlier, ISE works in many areas where Log4Shell has impact, including:
- Our own application, Start, a Vendor Risk and Assessment Management tool used by many of our clients to manage their vendor risk. The Java-based application potentially had the vulnerability. Disruption to the application itself, or extraction of data within the application, or using the application to launch a follow up attack would be disastrous for everyone involved. Start is a major part of the risk management solution, so the product itself has to be both secure and effective during a time of incident response.
- Our clients that we perform assessments for, particularly those who work with us on Application Security Assessments. Many of them may have recently had assessments, were due for one, or perhaps were even in the middle of testing as the vulnerability was disclosed. An application that was once thought to be effectively assessed for vulnerabilities will need to be reviewed.
- Our other clients and business partners that we share information with. We just don’t operate in a vacuum. To work in this space, you are inherently interconnected to other technology providers.
I interviewed several people at ISE to get a sense of what it was like at the time. I wanted to know what it was like on the ground floor, so to speak. How did the people who code applications and identify vulnerabilities deal with this huge news that required immediate attention?
Like many in the security space, we were prepared to respond and understood the ramifications of the vulnerability. “We found out about it through the analysts” recalls Tom Cullison, Director of Software Development at ISE. He described hearing chatter about it on Slack on December 9th, as the public disclosure occurred, followed by the team splitting into two groups. One group spent time reading about the vulnerability, exploits, how to mitigate, and focused on making sure the core vulnerability was addressed. Fortunately, we found that as it pertained to Start, Log4Shell was not impactful. We reviewed our code and found we were not susceptible. While the existence of the vulnerability itself informed future design, we did not have to take drastic steps to secure the product at the time.
The other group was focused on working with our clients utilizing Start to manage the incident with their vendors. Security questionnaires about your posture are a common practice with vendors and third party business partners, because you don’t want another company’s data breach or vulnerability to become your liability. The Log4j news created an impromptu demand from many stakeholders for immediate transparency.
“Within a day, we had clients reaching out asking us to help facilitate their mass communications via Start through the use of our ad-hoc questionnaires, which allow users to put together a security questionnaire at any time. Using our automation tools, they were then able to craft an email notification and send it out to thousands of vendors within minutes” said Katie Pickrell, Senior Product Manager for Start. “This helped to mitigate the top-down requests from CISOs our clients were getting quickly. They needed a timely response and the critical nature of the vulnerability needed to be made clear.”
“The questionnaires were very direct. What are you doing? Yes or No?” Tom remembered not much mincing of words. At this point, it’s not just a matter of managing the security issue but keeping your clients confident in your business and the way you manage it. Everything needed to be secure, but nothing makes people feel less secure than confusion and a lack of confidence in who you are working with.
As far as support for our clients was concerned, it was hectic but manageable. “At ISE, it was a big deal, but it wasn’t dramatic” said Cullison.
Let’s pivot a moment to the assessment side of ISE and our analysts who focus on identifying vulnerabilities during testing and determining how impactful they are to the security of our clients. I interviewed Josh Meyer, Principal Security Analyst, to hear his thoughts.
“Companies freaked out about it”, he said. “There was a lot of confusion. Some of our clients were worried that they were exposed and, in many cases, it wasn’t even relevant if they used, for example, Python instead of Java.”
I asked Josh if he was worried at the time about the vulnerability and how it would have impacted recent assessments we performed or how we approach them in the future. Would it have been possible that recent assessment information was no longer valid or that we missed something in a test? “As far as I know, none of our clients we have done assessments for were ever exploited for Log4Shell. Once the vulnerability was publicized, it became something we tested on specifically.”
Talking to my colleagues about their different experiences as it happened was striking. On the one side, we had people supporting clients on Start, and everyone was reacting with concern and worry (but not panic). Things were unknown but potentially very impactful. But from the analyst perspective it almost seemed like there was a kind of serenity and calmness about it. I asked Josh why his attitude is a bit different. “When I find a critical vulnerability during an assessment, I am very calm and relaxed. I have to deal with it professionally by writing it up, explaining what it means and how to deal with it. My approach must be low emotion.”
I came to the realization during our discussion that this is all a matter of perspective and experience. While the developers and product owners are sensitive to how a newly disclosed vulnerability can impact their application and by extension their entire business, analysts who test for these things see Log4Shell as an important vulnerability…just not one that carries more weight than other critical vulnerabilities that are perhaps not as widely publicized. The analogy we talked about was open heart surgery. For the layperson, open heart surgery is a terrifying prospect. It’s an incredible personal vulnerability that has literal life or death consequences. But to the heart surgeon, who has potentially done this a hundred times, this is just another operation. It’s critically important to handle it correctly but freaking out about it will not make the outcome better. Josh explained it quite simply. “Why should I treat Log4Shell any differently than another critical vulnerability I find during an assessment? It needs to be remediated.”
Some might consider avoiding an incident from the Log4Shell vulnerability to be “dodging a bullet”. But is that the right way to look at it? When the security experts warned about the impact of this vulnerability, many feared the worst. Yet one year later, for those who could have been impacted by Log4j/Log4Shell and weren’t, the whole thing may seem like a distant memory.
The point is that for professionals who manage security risk the right way, surprise vulnerabilities requiring action shouldn’t be a problem. It shouldn’t feel like dodging a bullet because luck has very little to do with it. Tom Cullison said it best. “It was a big deal, but it wasn’t dramatic.” That’s really the goal of any security program. Keep the security risk under control and manageable. How do you get there?
- Be Ready to Respond – To keep it simple, have an Incident Response Plan, both for actual attacks as well as to remediate critical vulnerabilities that are identified. Everyone on your team should know what their role is and how to move forward as these unexpected events occur.
- Communicate – Know who is impacted both in your business as well as vendors, customers, partners, service providers, and so on. Your data can be touched by so many entities, so having effective risk management practices across all stakeholders is critical. Start can help you track all of your vendor risks in the supply chain in one place.
- Layer Security into Design – You should never assume any component of technology is secure, so as you build out your systems, try to avoid creating single points of security failure.
- Maintenance – Josh Meyer put it best in our discussion. “The best practice is to always be updating your stuff. It’s hard. It takes work. It takes time to do that. But that’s what it takes to be a secure software developer. You have thousands of dependencies, and you need to be on top of them.” You could apply this ethos to other aspects of security outside of development.
- Documentation – It can be very difficult to know exactly how vulnerabilities like this can impact you as the news breaks, but having proper documentation so that you understand your environment and where to begin investigations will make a difference in how effectively you manage these kinds of issues. If you are developing an application, figure out if you have a Software Bill of Materials (SBOM). If an incident occurs, you will be glad you had it.
- Assess – You don’t need to wait for news like this to make sure your applications, networks, devices, and any other part of your business are secure. We are talking about Log4j now, but there are numerous critical vulnerabilities in technology deployed all over the world, some of which are known and others that are yet to go public. Effective vulnerability assessments, penetration tests and other security evaluations will lower your risks, regardless of the latest news. On top of that, threats and vulnerabilities will change over time, so an assessment performed one year ago will naturally have different results than one completed today. Consistent assessment disciplines keep vulnerabilities at a manageable level. While vulnerability assessments can help identify issues on your systems, you need a product like Start to assess the risk posture of your third party vendors, as well as contact them to gather information quickly the next time a major vulnerability requires immediate action.
Strong security takes work and there is a direct relationship between the investment of time, planning, effort, and resources to the security successes and failures of any organization. ISE is here to help, whether it is performing a White Box Application Security Assessment, Penetration Testing, supporting our clients third party vendor risks with Start, IoT Device Hacking and more! Contact us to learn about how we can help turn security into an advantage for your business.