Software logs by themselves aren’t very useful. This isn’t a secret. Developers around the world spend hours building quality logging into their applications, but those logs by themselves aren’t very helpful. To leverage those logs effectively, you need great search. When you’re troubleshooting an issue, you don’t have time to comb through logs line by line. That’s why lots of teams turn to software solutions to filter their logs. Two of the most popular are Elasticsearch and Splunk. When you’re comparing the two, you’ll see some big differences. Not just in their cost, which is substantial, but in the ways that you’ll need to work with them as a team. In this post, we’ll go over the primary attributes of each and help you figure out the best log search functionality for your team.
All About Elasticsearch
Elasticsearch is part of a bundle of open source products provided by Elastic, a distributed company devoted to open source software. Elastic does lots more than log analytics; it is dedicated to making search easier in every way. As we’ve noted, Elastic’s products are open source. This means that the barrier to entry for trying them out could not be lower. For the low cost of free, any team can add Elasticsearch to its operations stack. Elastic also provides high-quality integrations into all sorts of cloud environments, so setting up a search cluster on AWS or Azure is trivially easy. The company also provides a log-focused toolkit in Elastic Observability that focuses specifically on software logging.
What Does Elasticsearch Do Well?
In addition to the aforementioned cost benefits of adopting Elasticsearch, there are a few things it does tremendously well. For starters, it’s outrageously configurable. You can adapt it to nearly any environment, and it’ll help you find better information more quickly. That’s a powerful tool to have in your toolbelt. One-size-fits-all tools generally fit all the sizes they’re targeted toward poorly. Instead of giving you one size and hoping it will fit everything, Elastic’s approach is to give you a set of tools. You’ll then use them to tailor a tool for your team’s needs. That kind of do-it-yourself ability gives teams the freedom they need to build the tool that works perfectly for them.
What Doesn’t Elasticsearch Do Well?
If you’ve read the preceding paragraph, you probably won’t be too surprised by this answer. The kind of customizability Elasticsearch provides is a two-edged sword that it cuts both ways. Any team that decides to dive into adapting Elasticsearch to its environment is going to find that it’s a real project. Elastic’s business model is providing software for free and then selling services and support for its product. This isn’t a criticism; every company charges you for services and support. But it’s worth understanding the context. Elasticsearch provides incredibly powerful, customizable capability. It also requires a real commitment by your team to adopt it. The time you spend customizing Elasticsearch is time you’re not spending on building other features.
Additionally, it’s worth recognizing that your developers probably don’t spend all day thinking about observability or search. Elastic (and Splunk) have engineers who do that as their day-to-day job. The solutions that they create are often going to be more robust than those of developers for whom observability is a secondary or tertiary concern.
All About Splunk
Splunk is almost the polar opposite of Elasticsearch. Splunk is part of a suite of products made by the company of the same name that is laser-focused on log monitoring and observability. Instead of providing an open source platform that teams customize to their needs, Splunk provides a very focused piece of software designed to work well specifically for clients with lots of log data. It integrates with both cloud and on-premise software stacks, making it easy to plug and play.
What Does Splunk Do Well?
Splunk’s biggest benefit is that it provides a quick way to plug log visibility into your ecosystem with minimal work by your team. Just pipe your log output to Splunk’s processing server and log into the web interface a few minutes later. You’ll see your logs and can start digging into them using Splunk’s incredibly powerful search interface. As noted above, Splunk has engineers dedicated to working with logs and log searching, so they’re always refining their interfaces and workflows. Compared to something that’s home-rolled by a team of internal developers, it’s likely that Splunk provides something far easier to use out of the box than Elasticsearch. This value is even more apparent when we compare Elasticsearch versus Splunk for less technical users.
What Doesn’t Splunk Do Well?
As with the strength section, Splunk is the polar opposite of Elasticsearch. It works with a lot of environments, but it doesn’t work with every environment. If your environment is outside Splunk’s operating bounds, you’re going to struggle to get great value from it. Which is a big problem, because Splunk has a pretty steep price tag. For a team whose operating parameters match Splunk’s, it’s a great value. If yours don’t, trying to wedge those two pieces together is likely to cause pain and frustration. What’s more, Splunk can often be overkill for smaller or newer teams. It’s designed to work in enterprise environments with lots of data. If your team is serving 5,000 requests per day, plugging in Splunk can be a little like trying to use an elephant to squash a mosquito.
Elasticsearch Versus Splunk: Who Wins?
If you’ll permit me a car analogy, Splunk and Elasticsearch are like different kinds of race cars. Splunk is like a supercar: fast with terrific handling. But it’s not the sort of car that you want to take off the track. It needs to color within the lines. It does that very, very well, but if your use case is outside the world Splunk lives in, you’re going to have a bad time. It’s also very pricey. Elasticsearch is more like a home brew rally car. It’s designed to operate in any environment you put it in. It’ll take a beating and doesn’t care if you want to drive through mud or straight off the road. But as a consequence, there’s a decent chance you’re going to need to get out a wrench and repair something yourself.
First Generation Versus Second Generation
When comparing Elasticsearch and Splunk, it’s useful to think of them as representative of different generations of log visibility. Splunk is kind of the first generation of log visibility tools. It’s hyper-focused on enterprise clients and provides a complete tool kit out of the box. It’s also very expensive and might not fit your organization’s specific needs. If it does fit your needs, it’s a terrific answer, assuming the cost isn’t a blocker.
Elasticsearch upended that first generation by going in the opposite direction. The company found enough partners who had learned that tools like Splunk didn’t work for them. Instead of trying to build a tool that focused on those needs, Elasticsearch instead built a tool that was adaptable to a variety of situations. They invited people into the garage and gave them the skills to adapt Elasticsearch to their needs.
In reality, there is no best log visibility tool. Whether you’d be better off picking Elasticsearch or Splunk is a matter of your team’s needs, not which is the best software.
The Next Generation of Log Visibility
Wherever you land on the topic of Elasticsearch versus Splunk, it’s important to go into that new relationship understanding what you are and aren’t getting. If you haven’t already, it’s worth visiting Scalyr’s video on the topic, which provides a lot of insight about the technical ways these tools work. It also shows how Scalyr can be thought of as the next generation of log visibility tools. By rethinking how data is stored, Scalyr gets data from your logs to you with better context, a key requirement for almost anyone investigating logs. Whether you’re considering Scalyr or Elasticsearch for your team, you’re likely to find that Scalyr’s technology fits your needs better and provides better ease of use and speed for your team.
