Sameh Attia: Delete clean cache to free up memory on your slow Linux server, VPS

http://www.blackmoreops.com/2014/10/28/delete-clean-cache-to-free-up-memory-on-your-slow-linux-server-vps

Many Linux systems, servers and VPS’s run on low memory and over time you will see a degradation of speed and responsiveness. By default, Linux got excellent Memory Management and it knows when to clean up cache to free up enough Memory to execute the next command. However, saying that, more new features being added to Linux everyday and when you are playing games, running a Web Server, a Database (i.e. MySQL, PostgreSQL, MariaDB etc.), Network Storage (NAS / SAN ), you will see there’s a drop on speed and responsiveness. By deleting and cleaning pagecache, dentries and inodes related cache data from Memory, you can get free up some of your Memory (RAM) which then makes rest of system work bit faster. This article will show you 3 different options to delete and clean cache to free up memory on your slow Linux server and small VPS’s.

Using drop_caches to clean cache to free up memory

Starting Linux Kernel v2.6.16 ono we have a new mechanism to have the kernel drop the page cache and/or inode and dentry caches on command, which can help free up a lot of memory. However, before we do that, we need to discuss about clean and dirty caches.

Clean and dirty caches

When you run something on a Linux system or server, Kernel will try to cache the response for a period of time so that the next time the same request is made, instead of running a complex lookup in disk/process, it can just fetch that info directly from Memory/RAM and send back a response. This is one of the main reasons Linux systems are so much faster and responsive. Alternatively, Linux systems will store data/info in Memory first before writing it to disk. So it goes both ways. Ideally, the data in Disk/database should be the same in Memory. But when you’re playing games, or it’s a busy Linux server, there will be some delay before these two (disk-data and memory-data) can sync up.
Cleaning cache is easy. But in Linux we have what we call clean and dirty cache.Let’s have a quick look at the definition of these two types of caches and later I will discuss why they are important when you clean cache.

Dirty Cache

Dirty Cache refers to data which has not yet been committed to the database (or disk), and is currently held in computer memory. In short, the new/old data is available in Memory and it is different to what you have in database/disk.

Clean Cache

Clean cache refers to data which has been committed to database (or disk) and is currently held in computer memory. This is what we desire where everything is in sync.

Kernel options for dropping cache

Following explanation is taken from Linux Kernel docu:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt

drop_caches

Writing to this will cause the kernel to drop clean caches, as well as reclaimable slab objects like dentries and inodes.  Once dropped, their memory becomes free.

To free pagecache:
    echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
    echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
    echo 3 > /proc/sys/vm/drop_caches

This is a non-destructive operation and will not free any dirty objects.

To increase the number of objects freed by this operation, the user may run `sync' prior to writing to /proc/sys/vm/drop_caches.  This will minimize the number of dirty objects on the system and create more candidates to be dropped.

This file is not a means to control the growth of the various kernel caches (inodes, dentries, pagecache, etc...)  These objects are automatically reclaimed by the kernel when memory is needed elsewhere on the system.

Use of this file can cause performance problems.  Since it discards cached objects, it may cost a significant amount of I/O and CPU to recreate the dropped objects, especially if they were under heavy use.  Because of this, use outside of a testing or debugging environment is not recommended.

Based on that explanation, we can say that drop_caches doesn’t touch dirty caches and only drops clean caches. That means after you’ve used the standard “echo 3 > /proc/sys/vm/drop_caches” command, it will only drop clean caches and all the dirty caches are left behind. This isn’t too bad but we want to free up maximum Memory / RAM for our server.
A nice explanation from stackexchange regarding the importance of syncing cache:

“sync” only makes dirty cache to clean cache. cache is still preserved. drop_caches doesn’t touch dirty caches and only drops clean caches. So to make all memory free, it is necessary to do sync first before drop_caches in case flushing daemons hasn’t written the changes to disk.

So in this article, I will use sync and drop_caches together to ensure we are syncing dirty caches (hence making them clean cache) and then dropping the whole lot:

Three drop_caches options to clean caches

Option 1: Free pagecache

This is suitable for webserver as they tend to store lots of pagecaches.
To free pagecache:

sync; echo 1 > /proc/sys/vm/drop_caches

Option 2: Free dentries and inodes info

To free dentries and inodes:

sync; echo 2 > /proc/sys/vm/drop_caches

Option 3: Free the whole lot – pagecache, dentries and inodes

This the maximum memory cleanup you can do on any Linux system without killing a process.
To free pagecache, dentries and inodes:

sync; echo 3 > /proc/sys/vm/drop_caches

When do you run drop_caches? Is it safe?

drop_caches is not a means to control the growth of the various kernel caches (inodes, dentries, pagecache, etc…) These objects are automatically reclaimed by the kernel when memory is needed elsewhere on the system.
If you use drop_caches command on a busy server, it can cause performance problems. Since drop_caches discards cached objects, it may cost a significant amount of I/O and CPU to recreate the dropped objects, especially if they were under heavy use. Running this command is always a bad idea when your server is under heavy usage.
Depending on your location (geo-location) your server traffic will increase decrease on different time. For example: US users login and browse between 8am – 9pm, that mean you don’t want to run this command on your server if your main userbase is from US. At the sametime, when it’s 12am in US, it’s 12pm in China (I am guessing here). So if your userbase is from China, you might want to reconsider when you want to run this command on your Linux server or VPS to free up memory. Personally, I prefer to do it just before my server gets busier (I get US, India and Chinese users) that means there is little down-period, but I can still squeeze it somewhere.

WARNING / CAUTION

Because just after your run drop_caches, your server will get busy re-populating memory with inodes and dentries, original Kernel documentation recommends not to run this command outside of a testing or debugging environment. But what if you are a home user or your server is getting too busy and almost filling up it’s memory. You need to be able trade the benefits with the risk. Read below for my explanation.

Home users vs Servers

There are two types of Linux users:

Home Users – i.e. Desktop
Servers – i.e. WebServers

For Home users, duh! do whatever you want … you can just do much damage. You clean cache, and the system will be slowed for about 3 seconds and then re-populate memory with necessary files. You can safely run echo 3 to cleanup maximum memory:

sync; echo 3 > /proc/sys/vm/drop_caches

For Servers, (i.e. WebServer, A VPS with Low memory), it is much safer to use option 1, cleaning pagecaches only.

sync; echo 1 > /proc/sys/vm/drop_caches

Why? cause pagecaches fills up massive part of your servers memory and in case of a Apache webserver, 50% of memory is used for pagecaches.

Example scenarios

You need to run free -m and drop_caches to see the differences:

My Desktop

Run the following command to see how much memory is being committed:

free -m

Then run

sync; echo 3 > /proc/sys/vm/drop_caches

As you can see, my desktop was doing next to nothing, so there’s not much to cleanup.

Delete clean cache to free up memory on your slow Linux server VPS - blackMORE Ops -1

My server

But because my server is always busy and it’s got a small memory pool (for a server), running drop_caches makes a massive difference:

super@myserver [~]# 
super@myserver [~]# free -m
             total       used       free     shared    buffers     cached
Mem:         12792      12151       1640          0       1177      11263
-/+ buffers/cache:       1711      12080
Swap:        11999       1131      11868
super@myserver [~]# 
super@myserver [~]# sync; echo 1 > /proc/sys/vm/drop_caches
super@myserver [~]# 
super@myserver [~]# free -m
             total       used       free     shared    buffers     cached
Mem:         12792       1831      11960          0          0       1132
-/+ buffers/cache:        697      12094
Swap:        11999       1131      11868
super@myserver [~]#

How awesome is that? In an instant I recovered a massive pool of memory and my terminal to the server became lot more responsive.

Use cron to drop caches regularly

Its a good idea to schedule following in crontab to automatically flushing cache on regular interval.

Option 1: drop_caches using cron

Edit crontab file and add the following line in red at the end of it.

# crontab -e
* 4 * *  * sync; echo 1 > /proc/sys/vm/drop_caches

The command sync; echo 1 > /proc/sys/vm/drop_caches will execute every minute of 4am every day.

Option 2: drop_caches using sysctl

Alternative you can use also sysctl to change the drop_caches value:

# crontab -e
* 4 * *  * sync; /sbin/sysctl vm.drop_caches=1

Option 3: drop_caches using a bash script

Create a shell script say clearcache.sh under root partition and enter following contents

#!/bin/sh
sync; echo 1 > /proc/sys/vm/drop_caches

Now, set the permission of script /root/clearcache.sh to 755

chmod 755 /root/clearcache.sh

Now edit crontab file

crontab -e

Enter following line to set cronjob for clearing cache everyday at 4am

* 4 * * * /root/clearcache.sh

Restart crond service

Restart crond service for each options (you only need to choose any of the options above)

/etc/init.d/crond restart

Conclusion

I don’t recommend using a cron to run this command as bad things might happen. Let’s just say your server had some unusual user activity in an odd period from a different geo-location than you see (i.e. your post went viral in reddit) and cron is trying to run this command at the same time. The result: instant server crash and possibly a corrupted database. Use it with caution and only when you are logged in. But then again, a properly configured 1GB VPS can take 20-40 concurrent users and run this command at any time.
Home users, go crazy, your online games or low powered Computer can only work better with this command. Server and VPS users, use with caution, only use it when you are unable to increase Memory or you have a good timeframe with low user activities. Whether to use echo 1, echo 2 or echo 3, is upto you and your judgment.

NOTE: Found a better way of maintaining these using crontab and some additional scripts. I will post them later when I am sure they are suitable for every Linux blends.

Thanks for reading. Share and spread.

Tuesday, January 13, 2015

Delete clean cache to free up memory on your slow Linux server, VPS