Sunday, November 23, 2014

How to visualize memory usage on Linux

http://xmodulo.com/visualize-memory-usage-linux.html

Lack of sufficient physical memory can significantly hamper the performance of Linux desktop and server environments. When your desktop is sluggish, one of the first things to do is to free up RAMs. Memory usage is even more critical in multi-user shared hosting or mission-critical server environments, where different users or application threads constantly compete for more memory.
When it comes to monitoring any type of system resources such as memory or CPUs, visualization is an effective means to help understand quickly how they are consumed by different processes and users. In this tutorial, I describe how to visualize memory usage in Linux environment using a command-line tool called smem.

Physical Memory Usage: RSS vs. PSS vs. USS

In the presence of virtual memory abstraction, accurately quantifying physical memory usage of a process is actually not straightforward. The virtual memory size of a process is not meaningful because it does not tell how much of it is actually allocated physical memory.
Resident set size (RSS), reported by top command, is one popular metric which captures what portion of a process' reported memory is residing in RAM. However, aggregating RSS of existing processes can easily overestimate the overall physical memory usage of the Linux system because the same physical memory page can be shared by different processes. Proportional set size (PSS) is a more accurate measurement of effective memory usage of Linux processes since PSS properly discounts the memory page shared by more than one process. Unique set size (USS) of a process is a subset of the process' PSS, which is not shared by any other processes.

Install Smem on Linux

The command-line tool smem can generate a variety of reports related to memory PSS/USS usage by pulling information from /proc. It comes with built-in graphical chart generation capabilities, so one can easily visualize overall memory consumption status.

Install Smem on Debian or Ubuntu

$ sudo apt-get install smem

Install Smem on Linux Mint

$ sudo apt-get install smem python-matplotlib python-tk

Install Smem on Fedora or CentOS/RHEL

On CentOS/RHEL, you need to enable EPEL repository first.
$ sudo yum install smem python-matplotlib

Check Memory Usage with Smem

When you run smem as a unprivileged user, it will report physical memory usage of every process launched by the current user, in an increasing order of PSS.
$ smem

If you want to check the overall system memory usage for all users, run smem as the root.
$ sudo smem

To view per-user memory usage:
$ sudo smem -u

smem allows you to filter memory usage results based on mapping, processes or users in the following format:
  • -M
  • -P
  • -U
For a complete usage of smem, refer to its man page.

Visualize Memory Usage with Smem

Visualized reports are often easier to read to identify the memory hogs of your system quickly. smem supports two kinds of graphical reports for memory usage visualization: bar and pie graphs.
Here are examples of memory usage visualization.
The following command will generate a bar graph that visualizes the PSS/USS memory usage of a user alice.
$ sudo smem --bar name -c "pss uss" -U alice

The next command will plot a pie graph of the overall PSS memory usage of different processes.
$ sudo smem --pie name -c "pss"

As a summary, smem is a simple and effective memory analysis tool that comes in handy in various circumstances. Using its formatted output, you can run smem to identify any memory issues and take an action in an automatic fashion. If you know of any good memory monitoring tool, share it in the comment.

How to create dialog boxes in an interactive shell script

http://xmodulo.com/create-dialog-boxes-interactive-shell-script.html

When you install new software in the terminal environment, you may often see informative dialog boxes popping up, accepting your input. The type of dialog boxes ranges from simple yes/no dialog to input box, password box, checklist, menu, and so on. The advantage of using such user-friendly dialog boxes is obvious as they can guide you to enter necessary information in an intuitive fashion.

When you write an interactive shell script, you can actually use such dialog boxes to take user's input. Pre-installed on all modern Linux distributions, a program called whiptail can streamline the process of creating terminal-based dialogs and message boxes inside a shell script, similar to how Zenity or Xdialog codes a GUI for scripts.
In this tutorial, I describe how to create user-friendly dialog boxes in a shell script by using whiptail. I also show Bash code snippets of various dialog boxes supported by whiptail.

Create a Message Box

A message box shows any arbitrary text message with a confirmation button to continue.
whiptail --title "" --msgbox ""  
Example:
1
2
#!/bin/bash
whiptail --title "Test Message Box" --msgbox "Create a message box with whiptail. Choose Ok to continue." 10 60

Create a Yes/No Box

One common user input is Yes or No. This is when a Yes/No dialog box can be used.
whiptail --title "" --yesno ""  
Example:
1
2
3
4
5
6
#!/bin/bash
if (whiptail --title "Test Yes/No Box" --yesno "Choose between Yes and No." 10 60) then
    echo "You chose Yes. Exit status was $?."
else
    echo "You chose No. Exit status was $?."
fi

Optionally, you can customize the text for Yes and No buttons with "--yes-button" and "--no-button" options.
Example:
1
2
3
4
5
6
#!/bin/bash
if (whiptail --title "Test Yes/No Box" --yes-button "Skittles" --no-button "M&M's"  --yesno "Which do you like better?" 10 60) then
    echo "You chose Skittles Exit status was $?."
else
    echo "You chose M&M's. Exit status was $?."
fi

Create a Free-form Input Box

If you want to take any arbitrary text input from a user, you can use an input box.
whiptail --title "" --inputbox ""   
Example:
1
2
3
4
5
6
7
8
9
#!/bin/bash
PET=$(whiptail --title "Test Free-form Input Box" --inputbox "What is your pet's name?" 10 60 Wigglebutt 3>&1 1>&2 2>&3)
 
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "Your pet name is:" $PET
else
    echo "You chose Cancel."
fi

Create a Password Box

A password box is useful when you want to take a sensitive input from a user.
whiptail --title "" --passwordbox ""  
Example:
1
2
3
4
5
6
7
8
9
#!/bin/bash
PASSWORD=$(whiptail --title "Test Password Box" --passwordbox "Enter your password and choose Ok to continue." 10 60 3>&1 1>&2 2>&3)
 
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "Your password is:" $PASSWORD
else
    echo "You chose Cancel."
fi

Create a Menu Box

When you want to ask a user to choose one among any arbitrary number of choices, you can use a menu box.
whiptail --title "" --menu ""    [   ] . . .
Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
OPTION=$(whiptail --title "Test Menu Dialog" --menu "Choose your option" 15 60 4 \
"1" "Grilled Spicy Sausage" \
"2" "Grilled Halloumi Cheese" \
"3" "Charcoaled Chicken Wings" \
"4" "Fried Aubergine"  3>&1 1>&2 2>&3)
 
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "Your chosen option:" $OPTION
else
    echo "You chose Cancel."
fi

Create a Radiolist Dialog

A radiolist box is similar to a menu box in the sense that you can choose only option among a list of available options. Unlike a menu box, however, you can indicate which option is selected by default by specifying its status.
whiptail --title "" --radiolist ""    [    ] . . .
Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
DISTROS=$(whiptail --title "Test Checklist Dialog" --radiolist \
"What is the Linux distro of your choice?" 15 60 4 \
"debian" "Venerable Debian" ON \
"ubuntu" "Popular Ubuntu" OFF \
"centos" "Stable CentOS" OFF \
"mint" "Rising Star Mint" OFF 3>&1 1>&2 2>&3)
 
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "The chosen distro is:" $DISTROS
else
    echo "You chose Cancel."
fi

Create a Checklist Dialog

A checklist dialog is useful when you want to ask a user to choose more than one option among a list of options, which is in contrast to a radiolist box which allows only one selection.
whiptail --title "" --checklist ""    [    ] . . .
Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
DISTROS=$(whiptail --title "Test Checklist Dialog" --checklist \
"Choose preferred Linux distros" 15 60 4 \
"debian" "Venerable Debian" ON \
"ubuntu" "Popular Ubuntu" OFF \
"centos" "Stable CentOS" ON \
"mint" "Rising Star Mint" OFF 3>&1 1>&2 2>&3)
 
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "Your favorite distros are:" $DISTROS
else
    echo "You chose Cancel."
fi

Create a Progress Bar

Another user-friendly dialog box is a progress bar. whiptail reads from standard input a percentage number (0 to 100) and displays a meter inside a gauge box accordingly.
whiptail --gauge ""   
1
2
3
4
5
6
7
#!/bin/bash
{
    for ((i = 0 ; i <= 100 ; i+=20)); do
        sleep 1
        echo $i
    done
} | whiptail --gauge "Please wait while installing" 6 60 0

By now, you must see how easy it is to create useful dialog boxes in an interactive shell script. Next time you need to write a shell script for someone, why don't you try whiptail and impress him or her? :-)

Thursday, November 20, 2014

Smartctl – Monitoring & Analysis tool for Hard drive

http://www.linuxtechi.com/smartctl-monitoring-analysis-tool-hard-drive

Smartctl (Self-Monitoring, Analysis and Reporting Technology) is a command line utility or a tool in UNIX like operating system that perform SMART tasks such as printing the SMART self-test and error logs, enabling and disabling SMART automatic testing, and initiating device self-tests.
Smartctl is useful on physical Linux servers where smart disks can be checked for errors and extract info regarding the disks that are used behind the Hardware RAID.
In this post we will discuss some practical examples of smartctl command, Use below steps to install smartctl , if it is not installed on your linux box.

Installation of Smartcl in Ubuntu

$ sudo apt-get install smartmontools

Installation of Smartcl in Redhat / CentOS

# yum install smartmontools
Start the Smartctl Service
For Ubuntu
$ sudo /etc/init.d/smartmontools start
For CentOS & RHEL
# service smartd start ; chkconfig smartd on
Example:1 Check Whether Smart Capability is enabled or not for the disk
root@linuxtechi:~# smartctl -i /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Momentus 5400.6
Device Model:     ST9320325AS
Serial Number:    5VD2V59T
LU WWN Device Id: 5 000c50 020a37ec4
Firmware Version: 0002BSM1
User Capacity:    320,072,933,376 bytes [320 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 1.5 Gb/s
Local Time is:    Sun Nov 16 12:32:09 2014 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Where ‘/dev/sdb’ is your hard disk. Last Two lines in above output shows that SMART  capability is enable for the device.
Example:2 Enable Smart Capability for the disk
root@linuxtechi:~# smartctl -s on /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.
Example:3 Disable Smart Capability for the disk
root@linuxtechi:~# smartctl -s off  /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Disabled. Use option -s with argument 'on' to enable it.
Example:4 Display Detailed Smart info for the disk
root@linuxtechi:~# smartctl -a /dev/sdb              // For IDE drive
root@linuxtechi:~# smartctl -a -d ata /dev/sdb       // For SATA drive
Example:5 Display Overall health of the Disk
root@linuxtechi:~# smartctl -H  /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
190 Airflow_Temperature_Cel 0x0022   067   045   045    Old_age   Always   In_the_past 33 (Min/Max 25/33)
Example:6 Test Hard drive using long & short option.
Long Test
root@linuxtechi:~# smartctl --test=long /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 102 minutes for test to complete.
Test will complete after Sun Nov 16 14:29:43 2014

Use smartctl -X to abort test.
OR We can redirect test output to a log file , as shown below
root@linuxtechi:~# smartctl --test=long /dev/sdb > /var/log/long.text
Short Test
root@linuxtechi:~# smartctl --test=short /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Sun Nov 16 12:51:45 2014

Use smartctl -X to abort test.
OR
root@linuxtechi:~# smartctl --test=short /dev/sdb > /var/log/short.text
Note : Short test will take maximum 2 minutes whereas in long test there is no time restriction because it read & verify every segment of the entire disk.
Example:7 To View Drive’s Self Test result
root@linuxtechi:~# smartctl -l selftest /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%       492         210841222
# 2  Extended offline    Completed: read failure       90%       492         210841222
Example:8 Calculate Estimate time to perform test
root@linuxtechi:~# smartctl -c  /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121)    The previous self-test completed having
                    the read element of the test failed.
Total time to complete Offline 
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x73) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 102) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x103b)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
Example:9 Display Error logs of the disk
root@linuxtechi:~# smartctl -l error  /dev/sdb

Sample Output

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-32-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 5
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 da 08 e7 e5 a5 4c 00      00:30:44.515  READ DMA EXT
  25 da 08 df e5 a5 4c 00      00:30:44.514  READ DMA EXT
  25 da 80 5f e5 a5 4c 00      00:30:44.502  READ DMA EXT
  25 da f0 5f e6 a5 4c 00      00:30:44.496  READ DMA EXT
  25 da 10 4f e6 a5 4c 00      00:30:44.383  READ DMA EXT