One week wonder: How Etsy mastered the art of SIEM migration

For decades, organizations have been falling in and out of love with their existing security information and event management (SIEM) platforms. Some organizations migrate rarely, others go through three SIEMs in four years. However often it happens, SIEM migrations are notoriously complex.

Many security operations teams choose to put up with underperforming and often costly SIEMs for longer than ideal for the simple fact that they want to avoid tackling yet another migration. Other teams jump from one SIEM to the next, seemingly without having learned any lessons, doomed to repeat the same mistakes.

Grappling with an outdated SIEM is more than just an operational burden. It can also make it harder to protect modern computing environments, as teams spend more time struggling to navigate and maintain outdated systems than detecting, investigating, and responding to threats.

On top of all that, the rapid disruption of the SIEM market may soon make migrations an imperative, as more legacy providers sell or merge to stay competitive against newer, innovative players. There’s no doubt that change is in the air for security operations, which is why many SecOps leaders are rethinking their current SIEMs and preparing to move to a modern platform, despite the pain of migration.

What if we told you that “pain” doesn’t have to be the defining buzzword of SIEM migrations, and that it’s possible to conquer your next migration in a matter of weeks, not months?

That might sound like a SecOps fairytale, but in this case the truth is better than fiction. Just ask global online marketplace Etsy, which recently migrated the majority of its existing SIEM to Google Security Operations — in just one week.

Manan Doshi, who leads SIEM and SecOps engineering initiatives at Etsy, shed some light during a recent Cloud Security podcast and Security Summit session on core best practices that helped set the stage for Etsy’s swift and efficient one-week migration.

Though your path to SecOps modernization may end up looking different, Doshi’s insights can be invaluable for any organization looking to accelerate their own SIEM migration journey. Ready to ditch the outdated and embrace the modern? Let’s unpack Doshi's top three lessons for a swift and successful SIEM migration.

1. Stick to the basics

One of the more daunting aspects of SIEM migration is finding a platform that can meet your current and future needs. Many migration efforts can go off the rails before they even start, especially if organizations fall into the trap of prioritizing flashy capabilities instead of considering what teams actually need to do their work.

In previous migrations, for example, Etsy’s security operations team used a complex set of criteria to evaluate and score potential SIEMs, such as product integrations, search and detection language, threat intelligence dashboards, and training resources. However, Doshi noted this approach often led the team down a path to optimizing for things they didn’t actually need.

The team discovered that while a laundry list of fancy features looks good on paper, it’s often not worth the effort to set up or maintain them in the long run — not to mention a robust feature set usually comes with a heftier price tag.

“We got way too carried away trying to focus on a million features. A lot of SIEMs, especially older ones, have more features — that doesn't necessarily mean they're good or even if they are good, it doesn’t mean you’ll necessarily use them,” Doshi said.

At Etsy, for example, the security operations team spends the majority of its time on searching, writing detections, and API capabilities. Rather than trying to classify all the different things they might need in a SIEM, Doshi said it’s better to focus on the features that will support the activities where the most time is spent.

“We’ve learned, keep it simple, focus on the core functionality of a SIEM — search and detection writing — and make sure that works extremely well,” he said. “Our new motto is just keep it simple.”

2. Invest time in detection-as-code

Another critical component to Etsy’s success was that the team took steps after its previous migrations to simplify the process of moving platforms again. The migrations happening today will not be the last; SIEMs have been a staple of SecOps for years, and they (or whatever future form they take) will likely remain a vital tool for teams working to identify, investigate, and respond to threats quickly.

As such, it’s important to lay the groundwork now to make subsequent migrations faster and more efficient.

“The first time we migrated, we really started from scratch again. That was a bad time,” Doshi said. “We had a frame of reference for what our detections were, but we had to rewrite them in the new tool completely in a new language. For future migrations, we really started to invest heavily in Detection-as-Code and certainly this has helped us greatly in this recent migration.”

Detection-as-code is a set of principles that use code and automation to create, manage, and deploy detection rules and logic in an agile, continuous model. Doshi compared managing detections “as code” to working with unified DevOps platforms like Github, where detections are stored in a centralized repository that enables engineers to share and review rules, track changes, and revert to earlier versions if needed.

Preparations for Etsy’s migration were complex, and sometimes took longer than a week. These practices not only enhance collaboration, change management, and auditing but also significantly increases the portability of detections during migrations.

“Our previous platform, we had Detection-as-Code set up and in our current one, we have Detection-as-Code set up. That’s a lot easier to translate,” he said. “Of course, the actual syntax has to change, but the way we store our rules, I’d say half stays the same, as far as metadata, MITRE category tagging, all that good stuff. You just have to learn the new syntax of the actual detection and change that part.”

Organizations can use Gemini in Security Operations to build detections in plain English that will craft the structure of the detection. It can help lighten a significant portion of the detection-building workload. “With AI, 70-80% of the work is already done for you,” Doshi said. “The other great thing is that it adds a lot of uniformity to how everyone on the team writes detections.”

3. Lean into your expertise to plan your migration

When it comes to migrations — any migration — the right platform and practices won’t make as much of a difference without the right approach. In Etsy’s case, the security operations team chose to make a SIEM migration plan that played to its natural strengths.

“We have such a strong engineering culture at Etsy, so we did the engineering thing and turned it into a hackathon project. That’s why we did it in a week,” Doshi said.

The migration took place as part of Etsy’s annual internal hackathon that allows engineers to build and pilot bold new ideas that can deliver big wins or improvements to internal systems and workflows. While the majority of the transition occurred during the hackathon, there was still plenty of planning and preparation beforehand that enabled the team to reach its final goal.

“We did a lot of pre-planning ahead of time, so I guess it wouldn’t necessarily be fair to say the process was only a week,” Doshi explained. “We knew we were making this transition. We knew the hackathon was a couple months out, so we spent a lot of time preparing.”

The team listed out all the different data sources and content they had like detections, documentation links, and in some cases, even writing infrastructure configuration files for tools like Terraform ahead of time. To make this process most effective, Doshi recommended using migrations as an opportunity to clean house, focusing efforts on the best detection rules and log sources instead of migrating everything wholesale.

“When the week came, we had the spreadsheet of all the different log types, we already had Terraform code prepared, so we were really just applying things and making sure they come in properly,” Doshi said.

All the smart planning and careful preparation paid off: By the end of the week, the security operations team had all its log types migrated and its entire Detection-as-Code pipeline set up.

At the same time, Doshi was careful to caution SecOps leaders to think carefully about the approach that will work best for their organization before setting aggressive targets. The key to success, he said, is understanding your requirements, the strengths of your team, and finding an approach that will help you start extracting value from your new platform as quickly as possible.

“We have a strong engineering culture at Etsy. We have a very strong engineering team,” Doshi said. “That’s why things like this are possible and our management made space for us to make this project. If you’re not doing all of the above, if you haven’t built that culture, you can’t just expect magic to happen on its own.”

For more details on SIEM migrations, please read our guide to moving on from legacy SIEM systems.