The Scalyr Elasticsearch Connector

Elasticsearch is angry again… What do I do?

Four months and seven days ago I joined Scalyr. Scalyr was of interest to me because of how different it was from other observability solutions. Created to solve issues that confound other observability systems, Scalyr excels at scale, performance and cost using global optimizations such as being cloud native, separating storage and compute, and being aggressively multi-tenant, while avoiding index challenges through the use of a columnar event data store. 

Scalyr’s architecture is completely different from Elasticsearch which tons of companies have utilized. I was at one of these companies, building and managing a large solution on Elasticsearch. As I reached out to my network, it was apparent that every Elasticsearch engineer I spoke to was having similar scaling issues to the ones I was having. Whether it was ingestion latency, retention challenges, or stability, so many I spoke with were frustrated with Elasticsearch’s rough edges, and, even worse, had no easy path forward. 

I came to Scalyr with an agenda to solve this problem and I am happy to announce that the Scalyr Elasticsearch Connector is now available for Early Access.

Problems Solved

Elasticsearch allows you to scale by adding more computers. The computers that store data are simply referred to as data nodes and are limited by a lot of variables including physical RAM, disk size, cpu cores, and network speed among several others. These variables all play together in an intricate dance to determine service delivery. Get one variable wrong and you will have a very bad day. 

For smaller Elasticsearch clusters, with low velocity ingestion, the problems are easy to tune away. For larger clusters, with a largely non-deterministic workload, it can be a nightmare. Once you hit that scale, backing out of Elasticsearch can be complex. You selected Elasticsearch, trained your SREs and Devs, and built an ingestion, query, and visualization ecosystem around it. At this point, switching solutions could be detrimental to end-user productivity, morale and, potentially, your career. 

This was the problem I wanted to solve: how to help customers who have hit scale limits with Elasticsearch move to a solution like Scalyr. The goal was to deliver a drop-in replacement for Elasticsearch that allows you to keep Kibana, Logstash, and any other application and tooling investments you have made in-place. This could solve the very real problems of Elasticsearch durability, scalability, and cost with the least amount of effort, time, and disruption.

Our new Elasticsearch Connector does just that. It masquerades as an Elasticsearch Cluster. By interfacing in this way, Elasticsearch queries can be dispatched without modification; be translated into Scalyr’s native query language; be executed in Scalyr, and have the results returned to your application and dashboards. This lets you leverage Scalyr for what it is best at without re-training or re-tooling obstacles.

Getting Started is Easy

Appearing as an Elasticsearch Cluster gives us tons of flexibility. Our Elasticsearch Connector is delivered as a Docker container. Running with Scalyr API keys within your infrastructure, the Connector provides the translation glue between the Elasticsearch and Scalyr worlds. Once running, add the new Scalyr endpoint as a remote cluster and now Scalyr is queryable through Elasticsearch!

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "scalyr": {
          "seeds": [
            "<scalyr_es_connector>:9300"
          ],
          "transport.ping_schedule": "60s"
        }
	}
    }
  }
}
GET http://<my_es_cluster_endpoint>/scalyr:*/_search

Feeding data to Scalyr is also easy as Scalyr already supports many common ingest systems such as Kafka, Logstash and FluentD. Feed all your data or just a subset; the decision is entirely up to you. Once data is ingested into Scalyr, updating and rolling back Kibana configurations occur with Scalyr-provided scripts. Our goal is to make your transition easy and minimize the risk. Your users will not notice a difference and, for many queries, will actually see improvements.

Bright Future

The first step to a brighter tomorrow with Elasticsearch is to stabilize problems caused by scaling. Once these are stabilized, you gain the flexibility to move at your own pace. Scalyr will help you re-write queries, visualizations and dashboards to capture the full Scalyr SaaS experience when you are ready. We think you’ll like that, but that part of the migration can be done at your own pace, or even not at all. So if you are having Elasticsearch grief or are just ready for something new and modern, feel free to reach out to me directly and let’s see how we can improve scale, save you money, and get you more sleep.