Tuesday, December 11, 2012

Monitoring your server with tmux


There are lots of systems and utilities available to monitor your system. Many of these are web-based, or they run as a client-server system.
Unfortunately, there are several instances where the only allowed connection to the server of interest is over SSH. This might be for several reasons, the least of which being security. In these cases, you will likely still want some way of easily monitoring what is going on with your server. Using tmux, you can create a session which will run all of your monitoring software and
keep it running, regardless of whether you lose your connection or not.
This article will cover the basics of creating such a session, which you should be able to tune and tweak to fit your specific requirements. This way, you can simply log in using any available SSH connection and see, in an instant, all of the information that is of interest to you. Also, since you need to log into the system over SSH, you don’t need to worry about the problems of locking down other software, such as a web server.
server monitor
You can monitor a lot of processes and logs with tmux



Step by Step

Step 01

Getting tmux
Tmux originated as part of the OpenBSD system. It should be available in most distributions. For example, you can get it in Ubuntu with ‘sudo apt-get install tmux’. If you need the latest and greatest features, you can download the source code from SourceForge.

Step 02

Building tmux
The build system uses the usual ‘./configure; make; make install’ steps to build tmux. The reason you may want to build your own is that many distributions are behind one or more versions on the software provided by their respective repositories.

Step 03

Starting tmux
Starting tmux is as simple as typing ‘tmux’ and hitting Enter. Your console will clear for a split second, and then you will be presented with a Bash prompt again, along with a status bar located at the bottom of your screen. This status bar will contain information about your current tmux session.

Step 04

Starting top – CPU sorted
One of the things you will be interested in monitoring is which processes are using up the most CPU cycles on your server. A good tool for this is ‘top’. The default when you first start it is to sort processes based on CPU usage, so that is fine.

Step 05

Getting a new window
Here we come to one of the features of tmux; we need to create a new window in this tmux session. There are two ways to handle this: first, you can use the shortcut ‘C-b c’, or you can enter the complete command ‘new-window’. To enter commands, you need to enter ‘C-b :’ and then the command. This will put your current window into the background and open a new window in the foreground.

Step 06

Starting top – memory sorted
With a new window, you can start a new instance of top, sorting it on some other criteria. One of interest to most system administrators is which processes are using up memory. To sort the processes in this way, you will need to enter ‘M’. This may vary for other versions of top, so always check your version’s man page. You might also want to change the refresh rate by entering ‘d’ and setting the number seconds between each display.

Steo 07

Navigating windows
Now that you have multiple windows, you need to be able to navigate between them. The simplest way is to use the shortcut navigation keys. To move to a specific window, you can use ‘C-b’ and then the window number. Remember that window numbering starts at 0. If you simply want to move to the next or previous window, use ‘C-b n’ or ‘C-b p’.

Step 08

Creating new panes
The next great feature of tmux is the ability to break up windows into panes. This lets you have multiple programs running in the same window. To split the current pane horizontally, use ‘C-b %’ to get two panes, left and right. If you wish to split the current pane vertically, you would use ‘C-b “’.

Step 09

Navigating panes
Once you end up with multiple panes, you need to be able to navigate them. To move to the next pane in the current window, you would use the shortcut ‘C-b o’.
You can also rearrange panes within a window. To swap the current pane with the previous pane, use the ‘C-b {’ keyboard shortcut. To do so with the next pane, use ‘C-b }’.

Step 10

Using tail
Now that you have tmux essentials under your belt, it’s time to add some systems monitoring. You’ll want to monitor system logs, and you can do so in multiple panes, giving you an overall view. For example, navigate to an empty pane and enter:
tail -f /var/log/syslog
…in order to get a continually updating view of system messages.

Step 11

Following dmesg
Kernel messages can be followed by using the program dmesg. The problem is that it doesn’t do automatic refreshing. You can accomplish this with ‘watch -n 3 “dmesg | tail -n 15” ’, where the 3 is the number of seconds between refreshes, and the 15 is the number of lines to display.

Step 12

Network statistics
The next area you will want to monitor is networking. One utility you can use is netstat. To see all of the current connections on your server, you can use ‘netstat -at | grep -v LISTEN’. This is non-refreshing, so again you will likely want to pass it to watch in order to get an updating output.

       With tmux, you can create a monitoring system allowing you to check on your server remotely and get the perfect overview of what’s happening. Joey Bernard explains how…

Step 13

Disconnecting tmux
The next powerful feature of tmux is the ability to take your session and detach it from the console that you are currently using. To do this, you can use the shortcut key ‘C-b d’. This puts tmux into the background, allowing you to logout of the server if you wish. The great thing is this also works if your connection simply dies, too.

Step 14

Reconnecting tmux
Now that you have a tmux session set up that is monitoring all of the parts of your server that you are interested in, you may want to check in on it. You can log into your server and simply reattach to the existing tmux session with ‘tmux attach-session’.

Step 15

There is an alternative program available called Byobu. This program is actually a wrapper around both tmux and screen. It provides a prettier interface to tmux, including a more detailed, two-line status bar at the bottom of the screen. This improved status bar will give you more information, like battery level, CPU frequency and temperature, and even whether there are updates available for your system. These extras are all configurable, and there is even the option of creating a custom notification. You should consider checking Byobu out as a ‘tmux+’ option for your monitoring setup.

Step 16

Naming windows
Once you have your monitoring windows set up, you will likely want to name them so that they are easier to manage. You can do so with the ‘C-b ,’ shortcut. This will rename the current window, and this new name will appear in the list at the bottom of the screen.

Step 17

Configuration files
All of the commands you have used so far to create your monitoring session manually can be done automatically through the use of a configuration file. Each of the shortcuts has an equivalent long command which can be used in the configuration file.

Step 18

Creating windows
To create a new window, you need to add the line ‘new-window’ to the configuration file. When you create this new window, you can give it a target of a current window whose index is where your new window will be inserted.

Step 19

Starting top
Another important option to the ‘new- window’ command is a shell command to execute upon launching the new window. This is where you would place the command to start up ‘top’ within your new window.

Step 20

Naming windows
Naming windows is done through the option ‘-n NAME’. This is important as it makes managing the windows easier. This name gets used to label the window, and it also gets used when you target a window with some particular command through the ‘-t TARGET’ option.

Step 21

Creating panes
To create a pane, you will need to know which window you want to do so in. You can use the ‘split-window’ command, with either the ‘-h’ option for horizontal splitting or ‘-v’ for vertical splitting. Panes are identified through their 0-based index in the current window.

Step 22

Starting tail in a pane
To start up a program in your new pane, you can add the command to the end of your ‘split-window’ tmux command. You can change this at runtime with the tmux command ‘respawn-pane -k -t TARGET-PANE command’, which will kill the current process and start up your new one.

Step 23

Loading a configuration file
After all of this work, you should have a configuration file that will load your entire monitoring session. To do so, you can save it to the default filename ‘~/.tmux.conf’, or you can save it to another filename and load it with ‘tmux -f filename’.

Step 24

What else can you do?
This has only been a start. You can take this and add your own monitoring programs to your tmux session to help in your system administrator duties. You can now connect to your system on a whim and see what is going on in a matter of moments.

No comments:

Post a Comment