Disrupting the SOC: Empowering your analysts to reduce burnout
According to Richard Pethia from Carnegie Mellon University, DARPA (the Defense Advanced Research Projects Agency) established the first Security Operations Center (SOC), the CERT/CC (Computer Emergency Response Team/Coordination Center) in 1988 in the aftermath of the infamous Morris Worm propagation.
From that day until present, the biggest problem that SOC analysts faced was the high volume of alerts they had to process daily. Today, it is not uncommon to have over a billion alerts show up in the SOC over a period of 90 days. If you are a Fortune 500 size company, it’s way bigger than that. Microsoft processes 15 billion security events on a given day and manage more then 600,000 devices in more than 100 countriesi
SOC burn out is very common – as we all know, a SOC under attack can be very stressful and 96% of analysts recently surveyed say they feel significant personal impact after a cybersecurity attack. We all know that this level of high stress leads to burnout, energy depletion and mental distance from one’s role and reduced professional efficiency.
In a typical SOC environment, there are generally three tiers of analysts. Tier 1 analysts are typically the newbies, less than a year or two of experience. These are the console jockeys. Their job is to review each and every alert that comes into the SOC and decide what to do about it. They spend a lot of their time “swiping left” not because the alert isn’t important, but because the volume is so high that they can’t possibly look at all of them in any meaningful way. If they discover something they are not sure about, they bump it up to the Tier 2 analysts. The Tier 2 analysts typically have 3-5 years of hard-won experience and know how to handle most of the events that come into the SOC. If they can’t, they bump it up to the Tier 3 team. When you think of Tier 3 analysts, think of the stereotypical Unix greybeards. They don’t all look like that but most everybody has one or two of these people in their organization who have all been there, done that, and wear the scars on their back because of it. They’re the go-to people for all issues out of the ordinary. This essential design of the SOC team hasn’t changed that much since we invented it back in the 1980s.
Today the average tenure for a SOC analyst is between 1-3 years and the average size of a SOC is 10 employeesii. This represents a significant turnover, cost to the business and loss of experience on an annual basis with many of these departures being preventable.
With disruptive innovation occurring every couple of years in the IT and security community, it’s strange that we haven’t seen it in the basic design of the SOC since inception. That might be starting to change though with the philosophies of DevOps and DevSecOps gaining traction, tools like SOAR designed specifically for this purpose, and with organizations being conscious of these concerns taking time to invest in their talent.
DevOps
DevOps began to emerge as an industry best practice out of three converging ideas sometime in late 2009: The Agile development method, a 2009 Velocity Conference talk called “10+ Deploys per Day” by John Allspaw and Paul Hammond, and Eric Ries’ book, “Lean Startup” which influenced many Silicon Valley companies between 2007 and 2010. It is the idea that there needs to be a much tighter integration between software developers and information technology operations (IT Ops); that once the developers, the quality assurance teams, and the security analysts pass any new code or maintenance updates to IT Ops for deployment, their jobs aren’t done.
Instead of creating artificial black boxes within each team where updates come in, get worked on, and then are passed to the next black box, DevOps is the recognition that update creation, deployment, and maintenance is one big system of systems that needs to be managed that way. It is the idea that organizations would use the same Agile methodology they use today with their software development teams but expand it across all organizations in the deployment cycle: product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers, and security professionals. In other words, DevOps uses the Agile philosophy across the entire lifecycle of deployed systems from design, to development, to testing, to deployment, to maintenance, and finally to end-of-life.
By 2014 or so, the big internet giants had become who they were, stand alone leaders in their industry, due in no small part to their adoption of the DevOps philosophy. Their competitors who traditionally used the old waterfall software development method might take years to deploy a new service for their customers. The DevOps companies were doing 10 deployments a day.
SOAR is DevSecOps
SOAR (Security Orchestration, Automation and Response) tools started to emerge for SOC use in 2018. These tools purported to solve the telemetry volume problem coming into the SOC for Tier 1 analysts to deal with. Through APIs (Application interfaces through software), the SOAR tool can connect to practically any kind of networking equipment and security stack tool deployed on your network, automatically collect telemetry from that device in terms of health and welfare and security alerts, and provide the SOC team with the ability to easily write code that handles that telemetry in a prescribed way; things like, save for the future, delete, or escalate to the a human for review. This has enormous implications.
With SOAR tools in place, we can essentially eliminate the need for humans to do tier 1 and tier 2 functions by replacing them with automation.
SOAR helps to reduce false positives and allow analysts to focus on real events. In my previous CSO job, we used SOAR to reduce the volume down to 500 alerts in a quarter that a human had to look at, about 5 events a day that Tier 3 analysts spent time on. That seems much more manageable. Of course, we don’t fire the old Tier 1 and Tier 2 analysts. We change their jobs.
Some we turn into DevSecOps developers and put them on the task of automating the other things we need to automate. For others, we might put them on the threat hunting teams or purple teams or other teams trying to solve a myriad of SOC problems. The benefits of such a change are many. A recent IDG survey concluded that deployed SIEM/SOAR technology reduced the number of missed threats, increased employee moral and reduced analyst attrition.
The point is that tier 1 analysts don’t learn a thing manning a desk and ”swiping left” eight hours a day in the old SOC design. You could make a strong argument that the tier 2 analysts, once they settle in, are not learning a lot of new things either. You take those two sets of employees and put them in the DevSecOps team and the threat hunting team and they become cybersecurity experts in six months. That is the power of automation.
Using SOAR tools to automate your Tier 1 tasks is a stepping stone to reduce SOC burnout. It also moves your security team closer to adopting the DevOps philosophy.
Investing in your talent
Investing in your team with technology solutions designed for the SOC like SOAR, and other DevOps projects, may reduce analyst burnout within your own SOC.
Other ways may contribute too. Leaders should consider continual skills development programs that will help grow confidence, improve morale and increase the team’s sense of purpose. While these ideas seem like a no brainer, the previously mentioned IDG survey says that 50% of organizations don’t have formal programs in place.
Fostering creativity is key as many tasks within a SOC can be repetitive and mundane. Having the team frequently reflect on experiences, approaches, playbooks, tooling, etc will help them create new methods to automate and develop new use case content that will provide deeper visibility. This creativity helps build a desire to find new ways to be efficient and increases overall role satisfaction and with new efficiencies. This is also an opportunity for management to recognize employees for their contributions and make them feel proud of their hard work and accomplishments.
Rotation of roles within the SOC, or even within the broader IT organization, can both support the reduction of the mundane and increase security knowledge across the organization. At the same time, employees will likely increase their desire to be part of the broader team and organization.
Combining these approaches of investing in skill development, creativity, empowerment, and individual growth, helps create a culture that is inclusive and with a sense of purpose across the team.
Reducing SOC burnout is complex. There is no easy button here. One place to start is to put your people first for career development and provide them with the right integrated tooling that will both lengthen their tenure within an organization and reduce risks.
Learn more about DevSecOps – Cybersecurity first principals: DevSecOps
Learn more about Microsoft’s Azure Sentinel Cloud Native SIEM/SOAR
