So, you want to have visibility over your EC2 memory usage but happen to be staring at an AWS dashboard similar to this:
Clearly, you have no way of monitoring the EC2 memory of your instances using this dashboard because it doesn’t give you a graph for that. Trust me, I’ve been there before, and I know your pain.
In this guide, I’ll go step by step through the process of monitoring the memory usage of an EC2 instance and everything it entails. I’ll even throw in some tips on how to improve your EC2 performance. So, if monitoring EC2 memory is your dilemma, sit tight right until the end.
Note that this is not a very beginner-friendly post, as it assumes some familiarity with the AWS environment. However, I’ll try to keep everything as bare-bones as possible.
CPU vs. Memory
Just to make sure that we’re on the same page, the metric we’re interested in is memory usage, which is the memory computer programs use to store their working data. This is not to be confused with CPU usage, the processor that computes instructions for every task the instance executes.
How Do I Monitor EC2 Memory Usage?
With that out of the way, let’s better understand what can be monitored on an EC2 instance. These are what are referred to as metrics, and monitoring them is done with the help of a fantastic service called CloudWatch. For an exhaustive list of metrics available for monitoring, consult the official documentation here.
By default, AWS gives you visibility into metrics like CPU load logs, network latency, request volume, etc., but not EC2 memory usage. For other metrics like EC2 memory usage, you’ll have to install and configure a CloudWatch agent on the instance. And that’s exactly what we’ll be doing next.
Giving Permissions
An agent is a background task that collects specified metrics on an EC2 instance and sends them to CloudWatch. If you’re familiar with AWS, you may already know that roles and permissions are needed for one service to access another.
We will create an EC2 role and give it permission to access CloudWatch. Once we do that, we can attach this role to one or more EC2 instances that need to be monitored.
To do that, go to the Identity and Access Management (IAM) dashboard. Then, click on Roles in the side menu to the left.
Next, click the Create Role button, which will bring you to the following page. Here, select EC2 as the service that will be needing permissions, and then press the Next button at the bottom of the page.
We want to give access to CloudWatch. So, type “CloudWatchFullAccess” in the search box, and select it by clicking on the checkbox circled below. Click Next at the bottom of the page.
Optionally, you can add some tags for organization. That totally depends on your needs.
Next, you should put in a nice descriptive name and short description, and then click the Create Role button to create this role. Our role should now be ready for use.
Now that we’ve created this role, you can attach it to any instance whose memory usage you want to monitor. This will most likely be an instance your applications are running on.
To attach a role to an EC2 instance, simply go to the EC2 dashboard. Click on the instance you wish to attach your newly created role to, and then click on Actions > Security > Modify IAM role.
On the new page, select the newly created role from the drop-down menu, and then click apply.
With these done, your instance is now ready to have an agent installed on it.
Installing a CloudWatch Agent
For this part, we will need to SSH into the EC2 instance. If you’re having difficulties accessing your server via SSH, Google is your friend. Once we’re in our terminal, move into the home directory and download the appropriate CloudWatch agent.
cd ~ wget download-link
Depending on your instance’s operating system, you can find the appropriate download link here.
For example, if you have an Amazon Linux OS, the command to download the agent will be this:
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
When the download is complete, you can install the agent by running the following command:
sudo rpm -U ./amazon-cloudwatch-agent.rpm
Again, for other platforms, the appropriate command can be found at the link above.
Configuring the Agent
With our agent now installed, let’s configure it. The configuration file specifies two key things:
- The metric to be monitored
- The monitoring interval or frequency
To specify those, create a file at /opt/aws/amazon-cloudwatch-agent/bin/config.json using your preferred method and paste in the following configuration:
{
"metrics":{
"metrics_collected":{
"mem":{
"measurement":[
"mem_used_percent"
],
"metrics_collection_interval":30
}
}
}
}
(If you’re on a Windows server, use this path instead: $Env:ProgramDataAmazonAmazonCloudWatchAgentamazon-cloudwatch-agent.json.)
The file is quite self-explanatory. Here, we’re telling the agent to collect memory usage data in percentage format every 30 seconds .Of course, you can specify whatever interval you prefer.
Now, start the CloudWatch agent with this command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
And voila! We’re up and running.
Visualizing Your EC2 Memory Usage
Everything we’ve done so far will be for nothing if we can’t visualize it. So, let’s see what we have to do next. Oh, wait. We have absolutely nothing to do.
Go to the CloudWatch dashboard in your AWS console and click on Metrics. You’ll see a new CWAgent card. After you click on this card, you should see your EC2 memory usage in a graph. Be sure to give it some time for the collection process to start properly.
If, like myself, you’re big fan of great presentation, try out DataSet. Their integration with AWS EC2 allows for even better organization and presentation of metric graphs all in one place.
Improving EC2 Performance
Since you’re looking to monitor your EC2 memory usage, I take it you’re keen on the performance of your EC2 instances. Running out of memory in an EC2 instance is a very common problem that many people face. This can cause your application to freeze completely and therefore should be avoided. Here are some things I do to avoid this:
- Monitor memory usage: Luckily, this is exactly what we have already done. Having information on memory usage in itself is crucial because high memory consumption is progressive. If detected early, it can be avoided altogether.
- Manually killing processes: If particular processes are taking up a lot of memory, you can kill them off to free up memory and prevent the whole application from crashing.
- Autoscaling: This is something I used a lot and recommend for most applications in production. With autoscaling, redundant servers can be started when one crashes, keeping your app running at all times.
At this point, you have your CloudWatch agent sending your memory usage in percentages, which you can visualize in CloudWatch or via a third-party tool like Scalyr. And that’s it. We’re done.
However, I must mention that there is much more that a CloudWatch agent can do. For example, a very simple tweak to our configuration file above can monitor access logs on an Apache server:
{
"logs":{
"logs_collected":{
"files":{
"collect_list":[
{
"file_path":"/var/log/httpd/access_log",
"log_group_name":"access_log"
}
]
}
}
},
"metrics":{
"metrics_collected":{
"mem":{
"measurement":[
"mem_used_percent"
],
"metrics_collection_interval":30
}
}
}
}
But although the possibilities a CloudWatch agent offers are interesting, that’s not the purpose of this post. For that, feel free to explore on your own—the AWS documentation is your friend.
A CloudWatch Agent Makes It Easy to Monitor EC2
So, in summary, monitoring EC2 memory usage is now possible, contrary to what many people still believe. All you really have to do is carefully install and configure an agent on your instances. Once you have that in place, you can monitor memory usage and many other metrics not displayed by default on the AWS monitoring dashboard.
And if you want even better visualization of your EC2 memory usage, try out DataSet‘s demo, which includes an EC2 monitoring widget.
