You probably regard application logging the way you think of buying auto insurance. You sigh, do it, and hope you never need it. And aren’t you kind of required to do it anyway, or something? Not exactly the scintillating stuff that makes you jump out of bed in the morning.
It feels this way because of how we’ve historically used log files. You dutifully instrument database calls and controller route handlers with information about what’s going on. Maybe you do this by hand, or maybe you use a mature existing tool. Or maybe you even use something fancy, like aspect-oriented programming (AOP). Whatever your decision, you probably make it early and then further information becomes rote and obligatory. You forget about it.
At least, you forget about it until, weeks, months, or years later, something happens. Something in production blows up. Hopefully, it’s something innocuous and easily fixed, like your log file getting too big. But more likely some critical and maddeningly intractable production issue has cropped up. And there you sit, scrolling through screens filled with “called WriteEntry() at 2017-04-31 13:54:12,” hoping to pluck the needle of your issue from that haystack.
This represents the iconic use of the log file, dating back decades. And yet it’s an utterly missed opportunity. Your log file can be so much more than just an afterthought and a hail mary for addressing production defects. You just need the right tooling.
Log Monitoring To the Rescue
I’ve talked in the past about one form of upgrade from this logging paradigm: log aggregation. A log aggregation tool brings your log files into one central place, parses them, and allows you to search them rapidly. But you can do even more than that, making use of log monitoring via dashboards.
Log monitoring is what it sounds like. Or at least, it’s what it sounds like if you envision automated tooling and not some poor sap sitting and staring at the tail of a log file for eight hours straight. Log monitoring provides you with intelligent means to process your log files as they come in and glean insight from them.
This may sound strange if you’re familiar only with the “production behavior archaeology” use case of the log file. But you really do have many valid reasons to want to process your logs intelligently in real time. I’ll take you through some of those today.
Know About Issues Before Users Call Support
I’ll start with the use case that requires the smallest mental leap from the default archaeology one. The default case is inherently reactive. You go scrambling for the log file in response to users reporting an error. But log monitoring lets you turn that around — turn it into something proactive.
To understand what I mean, let’s consider the simplest sort of scenario. Let’s say that you don’t even have an application log and you’re just monitoring your application’s web server logs. On a normal day, this involves a lot of 200 responses, with a smattering of 300s, and the occasional 400 when a user makes a mistake. A normal day does not involve 500 responses, indicating server errors.
From years of blogging, e-commerce, and entrepreneurial activities, I’ve learned that users can be funny and strange. You might assume that their response to a 500 code would be to file a bug report. But in reality, they probably just shrug and leave your site. You might be throwing 500 codes for hours or even days before anyone bothers to tell you. But if you have a log monitoring strategy in place, you’ll see these errors right away. You might even correct the problem before any of your users have a chance to get upset.
Detect Suspicious Activity Before It Blows Up On You
Avoiding upsetting users with bugs and outages is critical to your application’s success. But it’s not the only thing that matters. You have other perils to avoid as well.
If you put it on the internet, it’s only a matter of time before someone tries to hack it, compromise it, or otherwise take advantage of it. This ranges from the annoying, like link spam in comments, to the malicious, like calculated attempts to compromise financial data. Sooner or later, someone’s going to try it.
Log monitoring can help here as well. Take the same simple example of logging the web server’s activity. But now imagine your monitoring logic is keeping an eye for something slightly different as well: a huge spike in 401 and 403 codes, indicating unauthorized and forbidden requests. This represents a crude, brute force example, but hopefully, you get the idea. Log monitoring shows you that someone is trying to break in before they actually succeed. This gives you the ability to engage in prevention instead of damage control.
Regulatory Compliance, Needed Now or Later
Let’s switch gears a bit now. Instead of considering the early intelligence log monitoring gives you, consider what the act itself gives you. The act of log monitoring itself is vital for regulated industries.
If you work in such an industry, you probably know it, at least implicitly. Without going into too much detail, suffice it to say that regulated industries are ones that must comply with governmental regulations. Manufacturers have to comply with safety regulations, while financial institutions have plenty of regulations on both handling your money and acting ethically.
Where does log monitoring fit? Well, consider HIPAA, for example, which ensures medical data privacy. HIPAA regulations not only require extensive logging but also monitoring of log files to look for certain discrepancies. If you’re in a regulated industry or plan to break into one, adopting log monitoring now may save you headaches later.
Baseline System Performance For Future Early Detection
You don’t need to be subject to any regulations to benefit from this next point. Log monitoring can help you track your application’s production performance in unique ways.
I can’t offer a simple case of HTTP response codes for this one. Realizing this benefit will depend on what you put into your log files and how you put it there. But let’s imagine a different, relatively simple scenario. Say that you have a site that logs both inbound requests and outbound responses, easily allowing you to tie those together. In other words, given a user or session ID, you can reconstruct the request’s cycle time from the log.
Using log monitoring, you can establish a known baseline of those cycle times. In other words, you can look at your logs for a period of time where you know the system to be behaving well. Then, you can take the average time and record it. Finally, you can implement a log monitoring scheme where you receive notification if the average response time, say, doubles. Slowdown can sometimes cost an e-commerce site just as dearly as outages, and log monitoring can help you detect it, even when there’s not an error to be found.
Look For Patterns To Help Your Business
Finally, let’s speak to something a little higher up on the Maslow’s Hierarchy of Needs for a business. So far, we’ve looked at getting out in front of errors, hacks, and regulatory agencies before they hurt you. But you can use log monitoring to gather intelligence and improve your business as well.
Your log file contains data, and you can use that data for strategic and tactical decisions. Let’s say you set up an alert that informs you about spikes in traffic to a page on your site. Maybe you see that alert for a page and realize you could really help yourself by adding a promo or discount coupon there for another offering. Or, maybe you find that you’re getting weekly alerts about a drop in traffic on Sunday mornings. You then use this information to work on a strategy to boost Sunday morning traffic.
The idea of using your logs as a form of business intelligence should tell you just how far we’ve come from the days of log files as an inefficient tool for hunting down production defects. Gather your logs in one place, turn them into data, and then monitor them. This will give you a leg up on your competition, and it will make life easier for everyone involved in application development and support.
