Today, enormous amounts of machine data is being generated. This data is of significant value to organizations that can store, search, and analyze this information. Fortunately, there are robust search analytics technologies in the market that can aid in managing this data. Two of the most notable solutions are Splunk, a leader in the space since 2003, and ELK (Elasticsearch, Logstash, and Kibana) as an open-source alternative.
It’s no secret that IT operations and security teams struggle to control the cost of their Splunk installation. At roughly $1,000 per GB per year, some savings can be achieved by negotiating multi-year licenses, but the issues around operating the infrastructure at scale and discarding data to minimize storage constraints will always be present. As Splunk shifts away from perpetual licenses, it also becomes difficult for their customers to convert to term licensing without an incentive to do so. To achieve the same log analytics goals at a lower price, why not consider moving to Elasticsearch instead?
Elastic is the company behind Elasticsearch. Elasticsearch is a distributed, schemaless, full-text search engine. At nearly 10 years old, it has been utilized by companies large and small across many different industries. As the popularity of centrally hosted subscription-based offerings has increased, Elastic’s SAAS offering “Elastic Cloud” has emerged as a powerful search tool for enterprise workloads.
It’s important to understand cost and engineering expertise when evaluating log and metric analysis tools. Splunk is a traditionally used, powerful tool to investigate, monitor, and act on data, but it is expensive. Organizations that employ Splunk engineers in their security operations teams might find it difficult to convince the business to switch to Elasticsearch, even when significant cost savings can be achieved. To meet your business and technology objectives, a careful, detailed comparison of Elasticsearch offerings is highly recommended. The outcome of this effort is a scalable, cost-effective log analytics solution.
Vertical Trail has had the opportunity to implement and operate many of the solutions with our enterprise clients and have recently evaluated two market leaders – AWS Elasticsearch and Elastic Cloud. Both of the solutions are similar but have some notable differences. If you’re considering Elasticsearch for your company, this blog post aims to describe the differences so that you can present an informed decision to your stakeholders.
Splunk to Elasticsearch
If you have an existing Splunk logging implementation and you are considering switching to Elasticsearch, it’s important to understand how any custom development in your Splunk implementation will carry over. Some organizations have found it quite difficult to “sell” the move to Elasticsearch due to years of internal development work and staff expertise within that platform. When considering a switch to Elasticsearch, stakeholders should carefully consider the effort and time requirements in migrating away from the existing service.
Splunk Enterprise software is priced by how much data you send into your Splunk installation in a day. Therefore, you must guess at the maximum amount of data you expect to send to Splunk in one day and purchase a license size that aligns with that guess. With ingest limits essentially removed from the cost equation when moving to Elasticsearch, there is no longer a need to guess, and there’s no need to decide which events to retain or discard. Data retention can also be increased at a much lower cost point.
Management and Operations
Launched in 2015, Amazon’s managed Elasticsearch service makes it easy to deploy an Elasticsearch cluster in AWS. From metrics analytics to enterprise search, AWS Elasticsearch is a managed service that works well for many use cases with small data ingest rates. It’s important to note that because AWS utilizes the blue-green deployment model when updating domains, an entirely new cluster is created, so data is copied from the old cluster to the new cluster and the old cluster is destroyed while the new cluster becomes the primary. In large clusters, this can take days to complete.
For organizations seeking to transform their monolithic infrastructure into many microservices, comprehensive logging – and more of it – is a requirement. The transition is made easier with each new microservice logging to Elasticsearch, as it slowly separates the old and expensive logging strategy from the monolith application.
Many organizations have mandates around data access and authentication. Elasticsearch satisfies compliance obligations in this area through role-based access control, encrypted communications, IP-filtering and auditing. For environments requiring secure implementation of cryptographic modules, FIPS 140-2 is supported in the JVM.
With AWS Elasticsearch, operators won’t have direct access to their clusters, which results in consequences such as having less control over the environment or experiencing more difficulty troubleshooting. Similarly, operators will not have access to several features. (A full list of the features AWS does not and cannot provide is here.) Therefore, with an on-premise / self-managed installation of Elasticsearch, you’ll need to ensure engineers have the required expertise to operate the cluster. But, in this scenario, features that were not made available by AWS are now fully within your control.
As logging analytics matures in the coming years, these offerings will certainly change. Organizations that adapt to this change should also be sure to retain flexibility to pivot away, if necessary, to avoid vendor lock-in. Amazon’s CloudWatch Logs Insights is becoming a strong offering, especially for organizations that are looking for a quick and easy-to-deploy service. Logs Insights allows users to examine logs in serverless applications, containerized environments and much more, including the ability to see multiple log groups as one. If AWS develops alerting capabilities within Logs Insights, it could emerge as a fully integrated pay-as-you-go IT operations service.
In terms of pricing, expandability, support and security, Elasticsearch is a leader in the log event management space. This scalable, cost-effective log analytics solution reduces costs and improves management and operations, making it possible to meet your business and technology objectives. To reduce the cost of your log and metrics collection and analysis and stop worrying about discarding data, please contact us.
- Big Data
- Data Science