You have a problem. But you don’t just have an ordinary problem. You have one of the most frustrating kinds of problems in the technical world.
In the most basic terms, you’re trying to open a log file that’s too big to open. But “log file too big” doesn’t fully capture the frustration or the problem.
You need something out of your log file, so you go to open it.
- Then you wait.
- And wait.
- And wait.
After some amount of time, your text editor just crashes.
Hoping it’s a fluke, you try again, waiting 15 minutes before another crash. So you’re half an hour in and not only have you not solved your actual problem—you haven’t even successfully taken what should be the simplest imaginable step toward solving it.
This combination of a long feedback loop with a non-deterministic outcome is what makes this so maddeningly frustrating.
But fear not. We’ll take a look at how you can solve this, starting with the quickest and most superficial route and working toward root cause analysis.
Pick a Different Tool at Your Disposal
If opening this log file is crashing an editor, like, say, Notepad, then your easiest step is to use a different editor. At least that way you can know that fate will reward your waiting with an opened file rather than with a crash.
Your path of least resistance here is to use something you already have installed. So consider the following utilities for each application.
- For Windows, you can use WordPad. If you have enough memory to cover the size of the file you want to edit, WordPad will load it. So these days, that’s quite likely to apply to files even topping a gig in size.
- For Mac, use Vim. It should be able to handle as big a file as you have memory, and with good search besides.
- There are a lot of different flavors of Linux out there, so it’s a little harder to talk about default installations. But if you have it, you can also use Vim here. If not, you can install it easily, and you can use tail -X at the command line, where X is the number of lines you’d like to see.
That should at least get you started. You should be able to see your file without needing to wait for something to maybe or maybe not crash.
Download and Use a Text Editor Meant for This
If you have a little more patience, you should ask yourself whether your current need is a one-time-only situation or if you’ll be viewing and editing a lot of large files. If the latter, you’ll want to get more deliberate about the tools in your toolbox.
I might suggest this even if you think this is a one-time need. Familiarizing yourself with a new, powerful text editor can’t hurt anything.
The number of text editors available to you is FAR too large to enumerate here. But Wikipedia has an extensive page on them, including specific information about file size.
If the problem solved by opening your large file isn’t too pressing, you could always engage in some yak-shaving. But you should probably solve that problem first, using a tool at your disposal.
Then come back and spend some time evaluating your text editor options. Find one that can open large files and that has other features you like besides. I’d say even try out a few of them.
If large files figure to be part of your life going forward, you should have a plan of attack for them.
Splitting Up Your Log File
We’re now getting out of the realm of superficial solutions and into your more meaningful ones.
So far, we’ve just looked at a variety of tactical approaches that involve finding different software with which to view your file.But you can also do stuff to your file.
First, you should make a copy of the file (and then go get some coffee while that copy happens). After all, you don’t want to mess with the original.
Once you’ve done that, you can set about splitting your file into bite-sized chunks. This Super User Stack Exchange question offers a number of suggestions for doing exactly that.
You have both downloadable tool options you can explore or command line magic that you can enlist.
However you do it, you simply want to turn your one large file into something like 10 or 100 smaller ones that will work with your text editor.
Obviously you’ll have to search these in sequence, which detracts a bit from convenience. But it’s more convenient than waiting for something to crash.
One other thought about splitting up your log file. Maybe in the future you should think about doing this deliberately as part of your application’s runtime behavior.
A lot of logging frameworks and appenders will make it easy for you to cap your log file size. You should take advantage of this.
Parse the Log File
I’ve previously discussed the importance of making logs that both humans and machines can parse. Broadly speaking, techniques for this include:
- Normalizing the format of your log’s entries.
- Including timestamps.
- Tagging and categorizing entries.
- Using key-value pair schemes for recording information.
If the log in question uses this approach, you can take advantage of that. Instead of opening it in an editor and doing a search, you could write a utility to parse it, looking for the information you need.
It doesn’t have to be anything fancy to get the job done.
Even better, you can use existing tooling to parse your logs into meaningful information. There are solutions out there that specialize in doing exactly this.
Use a Log Aggregator
Speaking of third-party solutions, their potential goes way beyond just parsing log files. You can aggregate all of your logs into one place, parse them, gather metrics on them, and even visualize them. And this exists as a pretty turnkey offering.
When you get here, you’re fundamentally changing the game. Developers and operations folks have, since time immemorial, cracked open log files and used their guts to figure out what was going wrong at runtime.
And you can still do this. There’s nothing wrong with that approach, per se.
But it’s important to understand that your logs really contain data about runtime behavior. And you can use this data like you’d use any data in a database.
As with a database, you wouldn’t just log in and dig through the raw text. You’d take advantage of tooling to do things like writing queries and visualizing the contents. You have that option with your logs.
Log File Too Big? Get to the Why
I’ve given you a lot of possible approaches, from the immediately tactical to the deeply strategic.
But whichever of these approaches you employ, it’s worth doing a bit of examination as to what landed you in your situation in the first place.
Waiting impatiently for a three-gigabyte log file to open so you can chase down some production issue is hardly ideal. If you’re doing it once, doesn’t it stand to reason that you might need to do it again?
And if so, are your actions today making it easier for you to fix the problem next time around?
Your log file being too big is much more of a symptom than a disease. Get the right text editor and you’ll fix the symptom. Start treating your logs as data, and you’ll address the disease.
