It’s hard to overstate the importance of logging in software development. If your application misbehaves in production and you don’t have logging, then you’re seriously out of luck. It’ll be nothing short of a miracle to be able to diagnose and fix whatever is wrong in your program. However, logs can be much more than simple troubleshooting mechanisms. If you employ the proper techniques, tools, and processes, your logging approach can become a first-class helper in business decision-making. As it turns out, there’s a shorter and catchier term for “the proper techniques, tools, and processes”: log management.
In this post, we’ll offer you a complete guide on log management. You’ll understand what it means, why your organization should care about it, and more. By the end of the post, you’ll be ready to start implementing log management in your organization (and eager to do so).
Let’s begin.
Why Are Logging and Monitoring Important?
As promised, we’ll define log management so that we’re on the same page in regards to what this term means. But before we get there, it makes sense to take a brief detour to talk about why logging and, on a general scale, monitoring are so crucial.
It all boils down to the nature of software itself. Most non-trivial applications tend to be quite complicated things. As soon as you release a piece of software, it’s anyone’s guess what’s going to happen. The only certainty is that, sooner or later, it’s going to break somehow. When it does, you’ll need to know two things: (a) that something has gone wrong and (b) details about how it went wrong. This way you can hopefully fix it.
Someone might argue that you could wait for the customer to complain about the problem. That would be a terrible idea for at least two reasons. The first reason is that there are many issues that are invisible or silent. This means that the software is giving the users incorrect results, but they’re none the wiser about it. The other (and more important) reason is that fixing a bug is cheaper the sooner you find it. It’s in the best interest of the organization to fix defects before they have the opportunity to impact the customer’s work. That will not only protect your organization’s reputation but also prevent the bug from creating havoc and leading to higher losses.
That’s why you should peek through the curtain, so to speak, and closely watch how your software behaves in production. Logging and monitoring are two sides of the same coin, and they allow you to do just that.
What Is Log Management?
Having explained why logging and monitoring are vital for software development, we’re now ready to define log management. So what is it?
To understand what log management is, we have to know what kinds of problems it solves. In other words, we must get a glimpse of what a logging approach without log management looks like.
Cargo Cult Logging
The opposite of proper log management is something that could be called “cargo cult logging.” Ever heard of cargo cult programming? This is the same thing, applied specifically to logging.
Organizations will start to log because they’ve heard or read somewhere that it’s important, without really understanding why. They hope that this will somehow lead them to profit. Then they’re surprised when it doesn’t—because hope isn’t a strategy.
One could argue that any logging, even if half-baked, is better than no logging at all. I beg to differ. Failure to follow logging best practices—be they general best practices or specific to some language or platform—can lead to dire consequences. Not employing logging levels correctly can put you in a needle-in-a-haystack kind of situation and prevent your team from finding the right information when they need it. Not applying log rotation can also do that, besides filling up your disks with gigabytes worth of log files.
Ignorance with regards to security concerns could lead you to include sensitive data like PII in your logs, potentially causing you big financial and even legal headaches.
In short, that’s what lack of log management looks like: It consists of logging “just because,” without a clear goal or road map, hoping that things will work out in the end.
Log Management to the Rescue
Log management is the way to avoid the scenario described above. It’s now time to finally define it. What is log management?
Well, Wikipedia has its usual formal, sounds-straight-out-of-a-textbook definition. Mine is simpler:
Log management is dealing with lots of log data in the most efficient way possible.
Of course, the “most efficient way” is somewhat subjective. It might mean different things for different organizations. That’s why, in order to avoid the scenario described in the previous section, you need to have a clear goal for your logging approach. Only then you’ll be ready to apply the correct techniques and processes to achieve that goal. In other words, log management is deciding to get the most out of your logging approach and then taking the right steps to ensure you get there.
What Does Log Management Include?
I keep talking about the correct techniques and processes. What are those? What’s included in log management?
There’s no standard one-size-fits-all version of what log management should look like. We could say, though, that the most common components are as follows: log collection, aggregation, log store and retention strategies, log rotation, log analysis, and search and reporting.
Log Collection
Your log management approach or tool should be able to take logs from all your relevant sources and put them into a single place. Going further, you should be able to intersperse them. That way, you can have a unified chronological perspective on all your logs.
Log Analysis
Bringing all of your logs together is the first step. A great first step, mind you, but a first step nonetheless. Where should you go next?
Well, having tons of data that you can’t make sense of is just useless. So the next logical step for your log management approach should be making sense of all your logs. Your log management solution should be able to parse your logs, organizing the elements in such a way that allows you to extract valuable knowledge from them.
Long-Term Log Storage and Retention
You probably can’t keep all of your logs forever. Even if you could, you probably shouldn’t. Whatever tool you use for log management should allow you to define your strategy when it comes to long-term retention of your log files. Questions about how long to store logs are ones that you need to answer, and your log management solution should empower you to answer those questions.
Log Rotation
This item is closely related to the previous one. Since you won’t be keeping all of your logs forever, you have to decide how to dispose of them and which method is going to be used for that. Log files that become too old or too big can be deleted. Other alternatives include compressing, archiving, or renaming the log files. A log management solution worth its salt should give you the ability to set the policies governing log rotation.
Log Search and Reporting
A good log management solution should provide the means for you to search through its data in real time. It should allow you to monitor issues and set up alerts so that you can stay ahead of problems and can act preemptively on them.
Additionally, a log management tool should have advanced visualization capabilities, allowing you to interpret the data on your log files in ways that help your decision-making process.
The Benefits of Log Management
As discussed, log management is important in avoiding the scenario of aimless and harmful logging, which we called “cargo cult logging.” But what are some practical benefits of adopting a proper log management strategy?
First, as we’ve mentioned before, proper log management and monitoring will help you catch defects earlier when fixing them is cheaper. Additionally, log management can be priceless when it comes to security and compliance. By having 24/7 monitoring, log management tools can immediately alert you of breaches or suspicious activities in general, so you can take action ASAP.
Log management will also improve operations across the organization by helping you collect data and metrics that you can then improve upon. That can also improve the overall user experience and user satisfaction since log management metrics can help you diagnose and fix performance bottlenecks.
Say “No” to Cargo Cult Logging!
Today we’ve covered log management, explained what it is, and shown how grim logging looks without it. But how does one go about preparing their organization to adopt log management?
The first step is to have a goal. Decide what your organization wants out of its logs. After you set up application logging, the next step is to start collecting and aggregating your log data into a centralized place. Even though you might be able to pull that off manually, the solid choice is employing a log management tool.
Implement log management and say no to cargo cult logging!
Carlos Schults wrote this post. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
