Saturday, July 2, 2016

Monitoring Network Load with Nload

http://www.linux-server-security.com/linux_servers_howtos/linux_monitor_network_nload.html

On a continually changing network it is often difficult to spot issues due to the amount of noise generated by expected network traffic. Even when communications are seemingly quiet a packet sniffer will display screeds of noisy data. That data might be otherwise unseen broadcast traffic being sent to all hosts willing to listen and respond on a local network. Make no mistake, noise on a network link can cause all sorts of headaches because it can be impossible to identify trends quickly, especially if a host or the network itself is under attack. Packet sniffers will clearly display more traffic for the busiest connections which ultimately obscures the activities of less busy hosts.
You may have come across the excellent nearly real-time networking monitoring tool, “iftop” (http://www.ex-parrot.com/pdw/iftop/), in the past. It uses ncurses via a console to display a variety of highly useful bar graphs and even accommodates regular expressions. An alternative to iftop is a powerful console-based tool is called “nload” (http://www.roland-riegel.de/nload). Such network monitoring tools can really save the day if you need to analyse traffic on your networks in a hurry.

   Background


In the past when I’ve been tasked with maintaining critical hosts I’ve left the likes of iftop and nload running in a console throughout my working day. Spotting real-time spikes is essential if you’re struggling for bandwidth capacity or suspect that a specific host might be attacked thanks to historical attempts.
Thankfully by having nearly real-time graphical interfaces, even displayed over a standard console there’s little eyestrain involved either. During times of heightened stress, such as when a host was being attacked, I’ve used these tools in one window alongside other console windows. That way I can simultaneously show the continually-changed output of network-sniffing tools in both a text and a graphical form. I find that by running different filters through each tool and changing them periodically as my field of focus evolves means that digging down into the data which is of most interest is much easier. Ultimately I end up with a clear picture of who is using the network and most importantly for what purpose.

   Installation


The nload packages can be found in a number of software repositories. On Debian derivatives you can use this command to trigger your package manager’s installation process:
# apt-get install nload
On Red Hat derivatives you can use this command:
# yum install nload
In the same way that iftop uses the “ncurses” package to output “graphics” to your console without the need of a GUI  the flexible nload switched to using ncurses too, back in 2001. Your package manager should take care of any dependencies in this regard so there’s no extra package installation work involved.

   Look And Feel


Now that we have a working installation all we need to do in order to run the package is use this command:
# nload
The results of such a command is the simple but useful output as shown in Figure One:

Linux Server Security

Figure One: The load on “eth0” interface using ingress and egress displays
The ability to split a single console window into two parts, one half for ingress (inbound) traffic and the other for egress (outbound) traffic is clearly of significant use. The clarity you immediately gain is invaluable, especially if you’re trying to diagnose an attack of some description. Also shown in Figure One, at the top, are the adjustable options to alter your display on the fly, without stopping nload in its tracks.
If we only wanted to focus on one specific network interface then we could run nload as follows:
# nload eth1
You can however add more than one network interface to the same console window for a useful comparison too. In this case we would use the following command:
# nload -m eth0 eth1
As we can see in Figure Two this can give a useful insight into which of our network interfaces are under the highest load and without squinting at the output of a packet sniffing tool.

Linux Server Security

Figure Two: Nload can get highly graphical but here we're just seeing two network interface displayed at once
Now that we’ve got an idea of how malleable nload is let’s look at some of the configurations options available to us.

   Refresh Frequency


Years ago I remember debating the effectiveness of making your traffic statistics update faster than the default setting. Tools which use SNMP (Simple Network Monitoring Protocol) such as RRDTool and MRTG (and indeed tools which use the “pcap” library such as iftop) use averages to populate the display which you are presented with. To cut a long story short: a quicker refresh frequency may lower the accuracy of the output from such tools. If you are interested in the intricacies of such a statement I would encourage you to read a little more into such matters online.
I mention this for good reason. Before we continue one caveat is that the “screen refresh” frequency is a different animal altogether to the “averages” used during the collection of statistics. When it comes to nload however the two are separated for clarity (unlike with other applications). The clever author keeps things simple which is very welcome.
From what I can tell from the manual the “-a” option is for changing the period used for measuring “averages” which (I am guessing) affects the calculations behind the scenes. Whereas the refresh frequency is for “screen display”, in terms of tweaking it to suit your display needs as we’ve just mentioned. Both ultimately relating to the how the majority of real-time statistics are collected and then displayed between them. In case it causes confusion with nload the screen refresh period is referred to as the “refresh interval”. The manual goes to some length to explain that lowering the refresh interval to less than the default 500ms is not that wise, as we’ve just discussed. It states: “Specifying refresh intervals shorter than about 100 milliseconds makes traffic calculation very unprecise.”
This confirms my experience with other traffic collection software too. The sophisticated nload goes on to reassure us with this additional comment: “nload tries to balance this out by doing extra time measurements, but this may not always succeed.”
I couldn’t confirm for certain what its exact effect is on other settings but within the Options Window (the box we saw at the top of Figure One) there is a “Smoothness of  average” setting which you may have some success in changing to affect your accuracy. Fear not, following some trial and error you shouldn’t have too many problems now that you’re armed with the terminology and the difference between the two key adjustable parameters.

   Battle Hardening


When I’m learning a new tool, in order to remedy such headaches, I tend to run a few tests between machines and monitor how the tool reacts at both ends of a connection.
Usually I would send a small amount of data, stop and start the connection and then try and saturate the network link (or at least put it under significant levels of stress) and repeat this a few times during my tests.
Coupling your newly-found knowledge of how a tool will ultimately react to differing scenarios, having tuned the refresh frequency (and potentially the averages used for traffic calculations) to suit your needs, is the best way to battle-harden a tool in my experience.

   Runtime Options


We can launch nload with a number of options, let’s explore some of those made available by the well-considered piece of software now.
We just discussed the effect of making changes to the “refresh interval” and “traffic averages” settings.
The setting which you probably shouldn’t drop much lower than the default for fear of losing accuracy is the “-t” option which affects the “refresh interval”. Should you decide to do so then you can move away from the default five hundred milliseconds to using a quarter of a second (two hundred and fifty milliseconds) as follows by launching nload with this option:
# nload -t 250
When it comes to “traffic averages” we can adjust the period used in the calculations by using the “-a” option as follows. Take note that this value is in fact in seconds and not milliseconds, it defaults at five minutes (300s).
# nload -a 60
Consider another option now. Picture the scene, your internet connection is via a gigabit network link but your ISP only allows you to use 100Mbit of that connection. Any network tool querying your network link will see a gigabit link speed as being available. However clearly this isn’t of any use to you. The clever nload lets you configure the throughput ceiling which you will monitor. As you continue to use the tool you need to bear in mind that you’ve altered this setting just in case you see unusual spikes above that ceiling. Otherwise it’s as simple as altering the setting like this:
# nload -i 100000
The “100000” value above is in kilobits-per-second (as scaling settings in nload generally are) and represents 100Mbit if my calculator is working properly. Note the “-i” option is only for inbound (ingress) traffic and the “-o” option is for outbound (egress) traffic.
On that note should you wish (as I almost always do when moving between different network capacities) to alter the default unit of measurement for traffic then we can launch nload in a variety of differing ways to achieve that. An example of making nload use kilobits-per-second (kbps) units by using the “-u” option is as follows:
# nload -u k
In the case of nload and the above example actually there’s no need to run that command however because that’s the default setting (“kbps” being a very good choice on all but very fast networks in my opinion). Looking at Table One we can see the other available options for unit measurements.

Bits
Bytes
Throughput units of measurement
h
H
Human readable formatted (otherwise known as auto mode)
b
B
Bits per second or Bytes per second
k
K
Kilobits per second or KiloBytes per second (the default is “k” or “kbps”)
m
M
Megabits per second or MegaBytes per second
g
G
Gigabits per second or GigaBytes per second
Table One: Unit measurement traffic throughput options
Let’s continue looking at another group of runtime options available to nload. Along the same vein as our unit measurements in terms of network throughput we can also change how the amount of data transferred is presented to you.
In Table Two we can see the possible upper and lowercase options. Note that this time we use the uppercase “-U” option to effect transfer data measurements and that the Bytes and Bits columns are in a different order due to the default setting being for megabytes (or “M”). This is almost the same as Table One but there’s no per-second measurement as it relates to file sizes essentially.

Bytes
Bits
Data transfer units of measurement
H
h
Human readable format (auto mode)
B
b
Bytes of data or Bits of data
K
k
KiloBytes of data transferred or Kilobits
M
m
MegaBytes of data or MegaBits of data
G
g
GigaBytes of data transferred or Gigabits
Table Two: The available nload data transfer unit measurement options
For clarity here’s a quick example of altering the data transfer measurement is a follows:
# nload -U K
The above option changes moves off the default megabytes to displaying data collection values in kilobytes.

   Live Options


There’s also a few commands which you can use while nload is running.
We mentioned having more than one device displayed on the console at once but you additionally have the ability to quickly move between devices. You can do this by simply pressing the Left and Right arrow keys (the cursor keys) on your keyboard. You won’t get lost because the number of windows available to you are paginated. How many pages can be accessed and which page you are currently on is dutifully displayed at the top of the window. Alternatively you can achieve the same functionality by hitting the Enter key or the Tab key to cycle through the network interfaces visible to your machine.
In Figure One we can see the available options displayed in a box at the top of the console. To toggle this Options Window on and off we simply hit the F2 key. To move around the Options Window thankfully there’s not much to learn, it’s very intuitive. Simply use the cursor keys on your keyboard to move around the box. Once you’re over the setting which you wish to adjust simply use the plus and minus keys on your keyboard to increment and decrement the setting. Once happy it’s just a case of hitting the F2 key again to hide the Options Window.
If you make a mistake and your display isn’t as you would like any longer then you can load up any saved settings (we’ll look at this further in a moment) by using the F6 key. If you’ve hit the sweet spot with your config settings and want to overwrite your saved config file then it’s as simple as hitting the F5 key. A minor word of warning is that I have to admit that (probably because I associate the F5 key with reloading a Browser’s page) I got the F5 and F6 keys the wrong way around at first. Just create a backup of your saved config to an unrelated filename if you’re worried that you’ll do this and lose configs.
If you wanted to quit nload then you can either reach for the ever-present Ctrl+C key combination or additionally simply hit the lowercase “q” key.

   Saved Config


There are two main files which nload uses for saving its config. The system-wide configuration file is called “/etc/nload.conf”. We can affect all users by editing options within this file, as opposed to an individual user’s settings. To change options for an individual it’s as simple as creating and editing a file in your home directory such as:
# pico -w /home/chrisbinnie/.nload
Follow the options that we’ve discussed and any in the system-wide config file to populate this file.

   Troubleshooting


Fear not if you get stuck. In addition to running this command below there are of course other routes to receiving assistance:
# nload --help
There’s a useful mailing list available from here which you could ask questions on:
Archives of previous mailing list discussions were formerly found here on this link but sadly it appears to be a 404 now:
http://sourceforge.net/mailarchive/forum.php?forum_name=nload-user
The useful netiquette applies. Be courteous and respectful and don’t expect every list member to immediately jump to your rescue if you haven’t made any efforts yourself.

   Summary


Watching the mighty nload in action can be a little mesmerising at times. Running it alongside other console windows nload is a real lifesaver however and during periods of heightened stress it launches almost instantly and you can usually easily discern the required information.
Hopefully it goes without saying that I would recommend trying it a few times prior to an outage or some other stressful interlude. Bosses leaning over your shoulder during a problem and witnessing your lack of understanding of your tool of choice isn’t ideal I’m sure that you’ll agree.
When you're able to retrieve the information that you need instantly from nload, seemingly by osmisis, at 4am during a callout, you will be glad of trying it out beforehand.




   Linux Books


If you've enjoyed reading the technical content on this site then please have a look at my Linux books which were both published in 2016 and some of my articles in Linux Magazine and Admin Magazine are available on their respective websites.

No comments:

Post a Comment