WebPerf Top 10 Log Analysis Tools – Making Data-Driven Decisions

We live in a time where data is recorded and logged for just about everything. Whether it is for web server uptime, JavaScript errors, database queries, CDN traffic, Nginx errors, etc. You name it, and you can probably log it. In a previous post, we discussed the importance of integrating a log management service into your DevOps workflow. Having centralized logs can be great, but you also need to be able to analyze them efficiently. So today we want to cover the top 10 log analysis tools which you can use to better parse your logs, run live tail searches, and query the specific log data you need.


Log Analysis Tools


As more and more companies move to the cloud, log analytics, log analysis, and log management tools and services are becoming more critical. DevOps engineers, system administrators, site reliability engineers, and web developers can all use logs to make better data-driven decisions.

There are a lot of different log analysis tools out there, below are 10 of the most popular ones. Each of them offers their own unique search features, live tail queries, etc. Some of the tools listed are more log management services, but they also offer more efficient ways to analyze your logs compared to looking at the raw data. Note: These are in no particular order and include both free and paid tools.

“Funnel all logs, metrics and machine events into a single hub. Get a clear overview of what is happening across your distributed environments, and spot the needle in the haystack in no time.” – logmatic

1. Loggly


Loggly is a cloud based logging management and analytics service provider founded in 2009. Their main focus is that log management needs to be much simpler and that DevOps, SysOps, and Engineers should not have to worry about log management. Some of their customers include EA, Autodesk, SendGrid, Atlassian, Sony Pictures, and Citrix.



Loggly features


Loggly has both free and paid plans starting at $49 per month. Some of loggly’s analyzing features include the ability to see a bird’s eye view of your logs with their dynamic field explorer. You can weed out the noise with a few simple clicks. They also boast powerful full-text searches, ranges, and booleans. You can also easily spot trends in your logs by using their rich views and graphs. If you see a spike, you can quickly narrow it down to that point in time in the log.

2. Logentries


Logentries is also a cloud-based logging management and intelligence service founded in 2010, with a strong community behind them. Their main focus is to eliminate the complexity of managing applications and systems in today’s distributed environments and deliver valuable log data insight to your team and to your business. Some of their customers include the Discovery Channel, Macy’s, LogMeIn, Yummly, and Parallels.



Logentries Features


Logentries has both free and paid plans starting at $39 per month. Some of Logentries analyzing features aggregated live-tail searches, custom tags, and context views. With aggregated live-tail searches you can filter logs in real-time, giving you much deeper insights into web application and system events as the occur. You can also custom tag important events so that your team can see these events in the log at a later time.

3. GoAccess

---

GoAccess is designed to be a fast, terminal-based log analyzer. Its core idea is to quickly analyze and view web server statistics in real-time without needing to use your browser. It is open source and available on Github with over 2,900 stars and 200+ forks.



GoAccess Features


We love GoAccess because it is open source, and because of that, it is completely free to use. This is a tool for those who love using terminal and SSH and want quick access to their data. You can easily generate reports with this tool on the fly by generating them to real-time HTML, JSON or CSV. You can view a live demo here.



Other features include the ability to process logs incrementally, picking your log format, real-time stats, and predefined custom log format strings.

4. logz.io


logz.io offers you real-time, actionable insights into your log analytics data with hosted ELK as a service. ELK is a simple but robust log analysis platform that costs a fraction of the price. Some of their customers include Netflix, Facebook, LinkedIn, Cisco, and Microsoft. logz.io is probably one of the biggest competitors to Splunk, which is mentioned further below.



logz.io Features


logz.io has both free and paid plans starting at $89 per month. Troubleshooting production issues is perhaps the most common use-case of log analytics. Their interface, which is powered by Kibana, lets you search through millions of records to investigate and pinpoint potential issues. You can filter results by server, application, or any custom parameter until you reach the source of the problem. If you are looking for a Splunk alternative you might want to give logz.io a try.

5. Graylog


Graylog is an open source log management platform which allows you to search, analyze, and alert you across all your log files. Some of their customers include BCBS, eBay, SAP, Cisco, LinkedIn, and Twilio. It is available on Github with over 2,000 stars and 300+ forks.



Graylog Features


Because it is open source, Graylog is completely free to use. They also have an enterprise version where they charge you per license on a server. Some of its features include a REST API and Graylog’s flexible processing engine which makes it easy to parse and enrich logs from any data source. You can search through terabytes of data instantly and even save search queries to be shared later with your colleagues. Their powerful drill-down analysis and charts make it easy to pinpoint issues in your logs.



6. Splunk


Splunk is a big name in the log and application management space. They have been around since 2003 are no newcomers when it comes to analyzing and monitoring data. They offer great solutions for larger enterprise customers.



Splunk Features


Splunk has both free and paid plans. Their free plan, Spunk light, allows you up to log up to 500MB data per day. There pricing for paid plans depends on the volume of data you are processing. Splunk offers a great way to collect, store, index, search, correlate, visualize, analyze and report on any machine-generated data to identify and resolve operational and security issues in a faster, repeatable, and more affordable way. They offer powerful drill-downs which allow you to easily go back in time using ad-hoc queries. Their dashboard and trend charts also provide a great way to spot and visualize possible trends.

7. Logmatic.io


Logmatic.io is a log analysis tool designed specifically to help improve software and business performance. The founders have more than 10 years experience in real-time and big data software. Their emphasis is on analyzing your “machine data.”



Logmatic Features


Logmatic.io has paid plans starting at $49 per month. Some of their features include the ability to create your own custom parsing rules which allows the software to automatically recognize patterns. This is supported in Apache, Nginx, syslogs, JSON events, etc. You have faceted and full-text granular searches down to the log level as well as real-time logs and real-time searches. You can use complex queries such as AND, OR, wildcards. etc. They provide you with all sorts of graphs to spot trends, everything from geo maps, flow charts, and pivot tables!

8. Logstash


Logstash is a free open source tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use. This tool goes hand in hand with both Elasticsearch and Kibana. Using these together can be a powerful combination for a log analysis tool. They are all three available on Github.

• elastic / logstash
• elastic / elasticsearch
• elastic / kibana

Logstash Features


Because it is open source, Logstash is completely free to use. They also have a paid license though available for those needing additional features. You can use elasticsearch, kibana, and logstash together. Kibana allows you to more easily explore and visualize the log data you bring in with logstash, and elasticsearch gives you the powerful real-time search and analytics capabilities.

9. Sumo Logic


Sumo Logic focuses on machine learning for unified logs and metrics to uncover real-time insights into application needs and new customer opportunities. They were founded in 2010 and their cloud-native service analyzes more than 100 petabytes of data per day.



Sumo Logic Features


Sumo Logic has both free and paid plans starting at $90 per month. They offer a unique feature that they call machine learning. This allows you to analyze petabytes of data and learn from this to uncover patterns more quickly. Their tool uses advanced analytics which helps make sense of large amounts of data using indexing and filtering. Their intuitive dashboard allows you to spot anomalies by setting up customer predefined metric filters.



10. Papertrail


Papertrail is more of a log management service, but they also offer some great features which make analyzing your logs fast! Some of their customers include GitHub, Instacart, Product Hunt, and DNSimple.



Papertrail Features


Papertrail has both free and paid plans starting at $7 per month. Some of their features include an intuitive web-based log viewer and powerful command-line tools. They have a REST API and long-term archives with S3. If you are a developer, you will probably like what Papertrail is doing! You can run live tail searches, seek by time, content, elegant searches, save your searches, and even colorize your logs differently. They even have an OS X dashboard widget and integrations with your favorite tools such as Slack and Zendesk.

Summary


There are plenty of log analysis tools to help you better understand your log data and parse it in a more efficient manner. This can help streamline your DevOps workflow and save you time the next time a problem pops up. The last thing you want to do is spend hours digging through unorganized log data trying to find what you need.

Did we miss any really important log analysis tools? If so, feel free to comment below. We especially love new open source tools!

Related Articles

• Configure KeyCDN Log Forwarding
• Integrating Log Management into Your DevOps Workflow

The post Top 10 Log Analysis Tools – Making Data-Driven Decisions appeared first on KeyCDN Blog.

Continue reading...