It’s all about the data. Turns out it always was.
When we started Scalyr, we initially focused on log analytics – providing engineering teams with a detailed, reliable view into their distributed systems. But our core vision was always about data; specifically, event data.
The original Scalyr team previously worked together at Google. Google, of course, has incredible technology for working with data at scale, most famously the contents of the public web. Built on keyword indexes, Google returns high-quality search results in a fraction of a second. And yet, the internal tools used by Google’s engineering teams weren’t remotely up to the task of providing interactive access to logs and other machine event data.
The problem was that event data is fundamentally different than the natural language text that keyword indexes were designed for. The data is different, the structure is different, the usage patterns are different. Solutions designed for text struggle with event data, especially at scale.
That’s where Scalyr came in. Instead of looking for a cleverer set of compromises that would let us jam a slightly better analytics solution onto existing data management architectures, we built a new architecture from scratch, designed for large-scale, high-cardinality, highly dimensional machine data. We made some big bets: aggressive multi-tenancy, columnar layout even for poorly-structured data, a query engine that dispenses with indexes, a closely integrated streaming analytics engine that offloads repetitive queries from the main engine.
Our early offering had gaps. The user interface looked like a couple of backend engineers had built it – because that’s who we were. But it still inspired love. Users came, apparently, for the questionable UI. But they stayed for the scalability and performance. (Actually, they mostly came because they had read about our unusual architecture and wanted to try it out; and later, through word of mouth.)
That early response was everything we had hoped for. What we hadn’t anticipated was how users would keep finding new use cases, stretching far beyond log analytics. It turns out that if you give people a solution that can scale to their event data, new use cases will come out of the woodwork. And many of those use cases had nothing to do with our UI; customers were building new applications directly on our APIs.
And thus was born the Event Data Cloud: the event analytics engine at the heart of Scalyr, now available to power customer applications, analytics services, and dashboards. We found immediate interest from multiple sectors, including cybersecurity. We quickly realized that cybersecurity has much in common with log analytics: large data volumes; a mix of continuous monitoring of complex rules with bursty, ad-hoc analysis; and the ever-present tradeoff between scale, cost, and performance.
SentinelOne Meets Scalyr
Several months back, Scalyr was contacted by several passionate engineers looking to solve an interesting data problem in the realm of XDR. The folks at SentinelOne had aptly recognized that for a security company, data analytics is a strategic core competency, and long-term success requires building that competency in-house rather than relying on third-party solutions. They had been exploring the market, and saw that Scalyr’s Event Data Cloud was a perfect fit for their vision.
One of those “only in Silicon Valley” whirlwind romances ensued. Tests on real-world data showed that Scalyr’s unique architecture delivers groundbreaking cost, performance, and scalability for XDR use cases, just as it has for log analytics. Even more important, the flexibility of our architecture will power the next generation of solutions. Scalyr can ingest, correlate, and search data from any source. SentinelOne has industry-leading AI technology for analyzing and acting on data.
Today, I’m thrilled to celebrate that Scalyr is becoming part of SentinelOne. Together, we are poised to deliver the industry’s most advanced integrated and real time data lake that can ingest structured and unstructured data from any technology product or platform. This is a huge step for us; and yet, it’s precisely aligned with our existing course. The reason Scalyr and SentinelOne are such a good fit is that we share precisely the same vision around the value of event data. We will continue to develop our log analytics and Event Data Cloud solutions; but now, at a whole new level of scale and impact.
The Expanding Circle: A Bright Future
Scalyr started out by merging traditionally distinct circles of data in the log analytics world into a single, larger circle. That circle keeps on growing. What we’ve learned along the way is that the more data you have, the more use cases you find; and the more use cases you find, the more data you add. Our architecture incorporates a network effect – the farther we scale, the better it performs – meaning that as the circle grows, our customer experience only improves. As part of SentinelOne, we’ll be taking another huge step up that curve. I couldn’t be more excited!
