Effective security management relies on consistent disciplines like maintaining technology, incorporating security into design, and regular assessments to identify vulnerabilities. But every so often the security world is shocked by a newly identified critical vulnerability that requires security teams to snap into action and react quickly. In 2021, the Log4Shell vulnerability within the Log4j Java logging framework shocked the world, right before the holiday season. Many security experts called it the worst vulnerability of all time because it was both easy to exploit and very impactful.
Depending on your perspective and role in the security management landscape, Log4Shell probably had a major impact on your organization, as it reportedly impacted 93%[1] of enterprise cloud environments. At ISE, we not only had to identify quickly how this vulnerability could impact us, but also how it could affect our clients.
An important element of proper security management is building your skills by learning lessons from past incidents. So, one year later, we want to take this opportunity to go back and revisit this historic vulnerability and come up with some takeaways to make us all more secure in the future.
For the purposes of this article, we will keep this explanation at a high level. Originally written in 2001, Log4j is an open-source logging framework that developers use to log data within Java based applications, which is extremely common in the world of application development. It is important to recognize that this vulnerability goes after devices that are practically everywhere, from personal devices to enterprise applications, smart TVs, and other non-traditional computing devices.
Log4Shell allowed an attacker to inject arbitrary, malicious code into an application, which could be used for a variety of things, such as gaining control of a device for follow up attacks. This could include ransomware deployment, cryptocurrency mining, data exfiltration, and other attacks.
The vulnerability was privately disclosed to the Apache Software Foundation on November 24, 2021, with the first fixes coming on December 6, 2021, followed by public disclosure of the vulnerability three days later on December 9th. It is thought that this vulnerability had existed since 2013 and it is impossible to know to what extent it was exploited until late 2021. Several security researchers, including some at Cloudflare, found evidence that testing for the exploit was happening before public disclosure, so it is reasonable to believe that damage had been done well in advance of the response by the security community [2].
The concern and confusion about Log4Shell extended to businesses, governments, and organizations all around the world. From our perspective in the cybersecurity field, this was something that needed to be addressed immediately for everyone impacted. For large corporations and governments, security incidents because of Log4Shell could be devastating on an entirely different scale. After all, this vulnerability came not too long after the Colonial Pipeline incident, which caused significant problems and had many question the security of the United States’ infrastructure.
The United States government set a deadline for December 24, 2021, for contractors to patch vulnerabilities, and the Federal Trade Commission said they would pursue companies that did not take reasonable steps to address Log4j in early 2022. Similar measures were taken by other governments across the planet.
Large technology companies like Amazon, Google, and Microsoft noticed impacts as well. Microsoft said on January 3, 2022 “Exploitation attempts, and testing have remained high during the last weeks of December. We have observed many existing attackers adding exploits of these vulnerabilities in their existing malware kits and tactics, from coin miners to hands-on-keyboard attacks” and “Sophisticated adversaries (like nation-state actors) and commodity attackers alike have been observed taking advantage of these vulnerabilities. There is high potential for the expanded use of the vulnerabilities.” [3]
If you want to understand why the security world was so alarmed at the time, step back a moment and consider the implications. By late 2021, even without the pandemic, we were heavily reliant on our devices and applications for our day to day lives, and it is almost guaranteed that among the dozens of apps we use on our PCs, smartphones, tablets, and connected devices that something would be impacted by this vulnerability. Extend that logic out and think about the partners and vendors you use along with the information and control they have over your assets. They are impacted too. So, this issue is out there, it impacts everyone, we all need to act on it immediately, and even if we try our hardest, we can’t control all of the ways it may impact us.
As mentioned earlier, ISE works in many areas where Log4Shell has impact, including:
I interviewed several people at ISE to get a sense of what it was like at the time. I wanted to know what it was like on the ground floor, so to speak. How did the people who code applications and identify vulnerabilities deal with this huge news that required immediate attention?
Like many in the security space, we were prepared to respond and understood the ramifications of the vulnerability. “We found out about it through the analysts” recalls Tom Cullison, Director of Software Development at ISE. He described hearing chatter about it on Slack on December 9th, as the public disclosure occurred, followed by the team splitting into two groups. One group spent time reading about the vulnerability, exploits, how to mitigate, and focused on making sure the core vulnerability was addressed. Fortunately, we found that as it pertained to Start, Log4Shell was not impactful. We reviewed our code and found we were not susceptible. While the existence of the vulnerability itself informed future design, we did not have to take drastic steps to secure the product at the time.
The other group was focused on working with our clients utilizing Start to manage the incident with their vendors. Security questionnaires about your posture are a common practice with vendors and third party business partners, because you don’t want another company’s data breach or vulnerability to become your liability. The Log4j news created an impromptu demand from many stakeholders for immediate transparency.
“Within a day, we had clients reaching out asking us to help facilitate their mass communications via Start through the use of our ad-hoc questionnaires, which allow users to put together a security questionnaire at any time. Using our automation tools, they were then able to craft an email notification and send it out to thousands of vendors within minutes” said Katie Pickrell, Senior Product Manager for Start. “This helped to mitigate the top-down requests from CISOs our clients were getting quickly. They needed a timely response and the critical nature of the vulnerability needed to be made clear.”
“The questionnaires were very direct. What are you doing? Yes or No?” Tom remembered not much mincing of words. At this point, it’s not just a matter of managing the security issue but keeping your clients confident in your business and the way you manage it. Everything needed to be secure, but nothing makes people feel less secure than confusion and a lack of confidence in who you are working with.
As far as support for our clients was concerned, it was hectic but manageable. “At ISE, it was a big deal, but it wasn’t dramatic” said Cullison.
Let’s pivot a moment to the assessment side of ISE and our analysts who focus on identifying vulnerabilities during testing and determining how impactful they are to the security of our clients. I interviewed Josh Meyer, Principal Security Analyst, to hear his thoughts.
“Companies freaked out about it”, he said. “There was a lot of confusion. Some of our clients were worried that they were exposed and, in many cases, it wasn’t even relevant if they used, for example, Python instead of Java.”
I asked Josh if he was worried at the time about the vulnerability and how it would have impacted recent assessments we performed or how we approach them in the future. Would it have been possible that recent assessment information was no longer valid or that we missed something in a test? “As far as I know, none of our clients we have done assessments for were ever exploited for Log4Shell. Once the vulnerability was publicized, it became something we tested on specifically.”
Talking to my colleagues about their different experiences as it happened was striking. On the one side, we had people supporting clients on Start, and everyone was reacting with concern and worry (but not panic). Things were unknown but potentially very impactful. But from the analyst perspective it almost seemed like there was a kind of serenity and calmness about it. I asked Josh why his attitude is a bit different. “When I find a critical vulnerability during an assessment, I am very calm and relaxed. I have to deal with it professionally by writing it up, explaining what it means and how to deal with it. My approach must be low emotion.”
I came to the realization during our discussion that this is all a matter of perspective and experience. While the developers and product owners are sensitive to how a newly disclosed vulnerability can impact their application and by extension their entire business, analysts who test for these things see Log4Shell as an important vulnerability…just not one that carries more weight than other critical vulnerabilities that are perhaps not as widely publicized. The analogy we talked about was open heart surgery. For the layperson, open heart surgery is a terrifying prospect. It’s an incredible personal vulnerability that has literal life or death consequences. But to the heart surgeon, who has potentially done this a hundred times, this is just another operation. It’s critically important to handle it correctly but freaking out about it will not make the outcome better. Josh explained it quite simply. “Why should I treat Log4Shell any differently than another critical vulnerability I find during an assessment? It needs to be remediated.”
Some might consider avoiding an incident from the Log4Shell vulnerability to be “dodging a bullet”. But is that the right way to look at it? When the security experts warned about the impact of this vulnerability, many feared the worst. Yet one year later, for those who could have been impacted by Log4j/Log4Shell and weren’t, the whole thing may seem like a distant memory.
The point is that for professionals who manage security risk the right way, surprise vulnerabilities requiring action shouldn’t be a problem. It shouldn’t feel like dodging a bullet because luck has very little to do with it. Tom Cullison said it best. “It was a big deal, but it wasn’t dramatic.” That’s really the goal of any security program. Keep the security risk under control and manageable. How do you get there?
Strong security takes work and there is a direct relationship between the investment of time, planning, effort, and resources to the security successes and failures of any organization. ISE is here to help, whether it is performing a White Box Application Security Assessment, Penetration Testing, supporting our clients third party vendor risks with Start, IoT Device Hacking and more! Contact us to learn about how we can help turn security into an advantage for your business.
[1] https://www.wiz.io/blog/10-days-later-enterprises-halfway-through-patching-log4shell
[2] https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/
[3] https://www.microsoft.com/en-us/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/#NightSky