WebPerf Integrating Log Management into Your DevOps Workflow

We live in a day where large amounts of data is constantly being distributed and recorded and sometimes that can be overwhelming. One way to stay on top of everything is to integrate a well thought out log management system into your current devops or IT workflow. Log data can help you become aware of application errors, slow web performance and other operational issues before they turn into a larger scale issue or start costing your company revenue.


What is Log Management?


Log management can be defined as the way you deal with large volumes of log messages and or log data, sometimes also referred to as audit trails, event logs, etc. This could include processes and policies you use to store them, generate them, archive them, and even how you dispose of them. Log data provides a record of what is happening in your business, from recording transactions, database queries, distributed services, CDN traffic, Nginx errors, etc. Whether it is an information system, service, or web server, data is being logged for just about everything.

Below are just a few examples of what you can use log management with:

• Linux
• Windows
• HTTP/S
• Nginx
• Apache
• Node.js
• Docker
• PHP
• JavaScript
• Python
• File Monitoring
• .NET
• CDN (such as KeyCDN)

​

The real problem when you don’t have a log management system or process in place is that it is easy to let the ball drop as they say because of not having a good handle on all the countless sources of data. Sven Dummer over at Loggly wrote a great article about what he calls the logging conundrum. He gives the advice of asking yourself three questions for every component or distributed service you have running.

1. Does it log?
2. If it doesn’t log, should it?
3. Should its logs be centralized in a log management service or software?

​

Advantages to Using Centralized Log Management


Obviously one of the biggest advantages to a centralized log management system is the ability to see all of the data. While it is possible to get an overload of data, many of the logging tools now have powerful filters in place which you can use to only see what you need to see. You can even run live-tail searches. Below are some additional advantages to a centralized log management service.

• Ability to identify key trends and anomalies and correlate events across your systems.
• Using your log management service as a monitoring tool can help more cohesively connect the data with downtime.
• Spend less time bouncing between different data log locations and dashboards. Big advantage to having everything under one dashboard.
• Turn searches across different devices and platforms into real-time alerts, reports or dashboards with a few mouse clicks.
• Troubleshoot problems faster and notify the right people.

​

As long as you maintain “log silos,” you will not be able to get a cohesive, in-context view of everything that makes your application run. — Sven
​

Log Management Services


When it comes to log management there are a lot of tools and services to help you centralize and improve your workflow. We will be covering a larger list of log management tools in an upcoming post, but below are recommended ones.

Loggly


Loggly is a cloud based logging management and analytics service provider founded in 2009. There main focus is that log management needs to be much simpler and that DevOps, SysOps, and Engineers should not have to worry about log management. Some of their customers include EA, Autodesk, SendGrid, Atlassian, Sony Pictures, and Citrix.



Some Common Use Cases

---

Below are some common uses cases in which Loggly’s customers are using log management in their workflows.

• Troubleshooting periodic slowness on a website and searching logs to discover it was a low memory issue.
• Notifications sent to team when the payment processing service on an eCommerce site falls out of the SLA of 500ms.
• Identifying increased processing time trends due to a recent bug introduced from the last software update from the development team.
• Using New Relic and Loggly together to more quickly see why the CPU is maxing out.

Logentries


Logentries is also a cloud based logging management and intelligence service founded in 2010, with a strong community behind them. Their main focus is to eliminate the complexity of managing applications and systems in today’s distributed environments, and deliver valuable log data insight to your team and to your business. Some of their customers include the Discovery Channel, Macy’s, LogMeIn, Yummly, and Parallels.



Some Common Use Cases


Below are some common uses cases in which Logentries’ customers are using log management in their workflows.

• Automate collection of all log formats and types (application, database, security, system, etc), across multiple platforms and cloud PaaS and IaaS services.
• Tail logs in real-time to pinpoint issues and drill down to understand root cause of the problem.
• Correlate server metrics (CPU, Memory, Network, Disk), system log data and application log data to gain deep insights to your system health and performance.
• Set custom log retention policies and intervals to meet individual compliance and auditing requirements.

CDN Log Management With KeyCDN


KeyCDN provides you with real-time logs in your dashboard in which you can analyze your CDN traffic at any time. You can run a live tail on your raw logs and define custom filters to drill down on your traffic pattern. This could be very helpful to identify broken links (Status: 404) or if your content has been cached (Cache: HIT).



But the whole idea behind centralized log management is to actually save time by not having to login to separate dashboards. And because of that, KeyCDN can export your raw-logs in real-time as syslog messages over UDP. Normally this implies that you could receive such messages with a syslog server, but that would again mean your creating a “log silo.” So you can easily export your KeyCDN logs to a external log management service like Logentries or Loggly.

1. See our tutorial on how to enable log forwarding.
2. See our tutorial on how to export your CDN logs to Logentries.

We also have a complete list of KeyCDN log formats.

Summary


Having a good system in place for log management can be a great way to improve your DevOps workflow, save time, and help troubleshoot issues with power search capabilities that extend across all of your devices and services. Using a 3rd party log management service like Loggly or Logentries can be a great cost effective solution.

Related Articles

• Configure your own Syslog Server
• Analyzing CDN Traffic to Your Website

The post Integrating Log Management into Your DevOps Workflow appeared first on KeyCDN Blog.

Continue reading...