Monday, September 15, 2014

Perform Multiple Operations in Linux with the ‘xargs’ Command

http://www.maketecheasier.com/mastering-xargs-command-linux

Xargs is a useful command that acts as a bridge between two commands, reading output of one and executing the other with the items read. The command is most commonly used in scenarios when a user is searching for a pattern, removing and renaming files, and more.
In its basic form, xargs reads information from the standard input (or STDIN) and executes a command one or more times with the items read.
As an illustration, the following xargs command expects the user to enter a file or a directory name:
xargs ls -l
Once a name is entered, the xargs command passes that information to the ls command.
Here is the output of the above shown xargs command when I executed it from my home directory by entering “Documents” (which is a sub-directory in my Home folder) as an input:
xargs-basic-example
Observe that in this case, the xargs command executed the ls command with the directory name as a command line argument to produce a list of files present in that directory.
While the xargs command can be used in various command line operations, it comes in really handy when used with the find command. In this article, we will discuss some useful examples to understand how xargs and find can be used together.
Suppose you want to copy the contents of “ref.txt” to all .txt files present in a directory. While the task may otherwise require you to execute multiple commands, the xargs command, along with the find command, makes it simple.
Just run the following command:
find ./ -name "*.txt" | xargs -n1 cp ../ref.txt
To understand the command shown above, let’s divide it into two parts.
The first part is find ./ -name "*.txt" , which searches for all the .txt files present in the current directory.
The second part xargs -n1 cp ../ref.txt will grab the output of the first command (the resulting file names) and hand it over to the cp (copy) command one by one. Note that the -n option is crucial here, as it instructs xargs to use one argument per execution.
When combined together, the full command will copy the content of “ref.txt” to all .txt files in the directory.
One of the major advantages of using xargs is its ability to handle a large number of arguments. For example, while deleting a large number of files in one go, the rm command would sometimes fail with an “Argument list too long” error. That’s because it couldn’t simply handle such a long list of arguments. This is usually the case when you have too many files in the folder that you want to delete.
rm-arg-list-too-long
This can be easily fixed with xargs. To delete all these files, use the following command:
find ./rm-test/  -name "*" -print | xargs rm
Software developers as well as system administrators do a lot of pattern searching while working on the command line. For example, a developer might want to take a quick look at the project files that modify a particular variable, or a system administrator might want to see the files that use a particular system configuration parameter. In these scenarios, xargs, along with find and grep, makes things easy for you.
For example, to search for all .txt files that contain the “maketecheasier” string, run the following command:
$ find ./ -name "*.txt" | xargs grep "maketecheasier"
Here is the output the command produced on my system:
find-xargs-grep
Xargs, along with the find command, can also be used to copy or move a set of files from one directory to another. For example, to move all the text files that are more than 10 minutes old from the current directory to the previous directory, use the following command:
find . -name "*.txt" -mmin +10 | xargs -n1  -I '{}' mv '{}' ../
The -I command line option is used by the xargs command to define a replace-string which gets replaced with names read from the output of the find command. Here the replace-string is {}, but it could be anything. For example, you can use “file” as a replace-string.
find . -name "*.txt" -mmin 10 | xargs -n1  -I 'file' mv 'file' ./practice
Suppose you want to list the details of all the .txt files present in the current directory. As already explained, it can be easily done using the following command:
find . -name "*.txt" | xargs ls -l
But there is one problem; the xargs command will execute the ls command even if the find command fails to find any .txt file. Here is an example:
find-xargs
So you can see that there are no .txt files in the directory, but that didn’t stop xargs from executing the ls command. To change this behaviour, use the -r command line option:
find . -name "*.txt" | xargs -r ls -l
Although I’ve concentrated here on using xargs with find, it can also be used with many other commands. Go through the command’s main page to learn more about it, and leave a comment below if you have a doubt/query.

Thursday, September 11, 2014

Find HorizSync VertRefresh rates to fix Linux display issue – Why my display is stuck at 640×480?

http://www.blackmoreops.com/2014/08/29/fix-linux-display-issue-find-horizsync-vertrefresh-rates

I had this problem a few days back and it took me sometime to figure out what to do.
I have a NVIDIA GTX460 Graphics card on my current machine and a Acer 22" Monitor. After installing NVIDIA driver, my display was stuck at 640x480 and no matter what I do, nothing fixed it. This is an unusual problem with NVIDIA driver. I am assuming Intel and ATI driver might have similar issues.

Fix Linux display issue

So if you are having problem with your display or if your display is stuck at 640x480 then try the following:
Edit /etc/X11/xorg.conf file
root@kali:~# vi /etc/X11/xorg.conf

You will see something like this
Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Acer X223W"
    HorizSync       28.0 - 33.0
    VertRefresh     43.0 - 72.0
    Option         "DPMS"
EndSection

Now the lines that control display in monitor is the following two:
    HorizSync       28.0 - 33.0
    VertRefresh     43.0 - 72.0
Depending on your monitor size, you have to find the correct HorizSync VertRefresh rates.

Find supported HorizSync VertRefresh rates in Linux

This took me quite some time to determine exactly what I am looking for. I obviously tried xrandr command like anyone would do..
root@kali:~# xrandr --query

This gave me an output like the following
root@kali:~# xrandr --query
Screen 0: minimum 8 x 8, current 1680 x 1050, maximum 16384 x 16384
DVI-I-0 disconnected (normal left inverted right x axis y axis)
DVI-I-1 disconnected (normal left inverted right x axis y axis)
DVI-I-2 connected 1680x1050+0+0 (normal left inverted right x axis y axis) 474mm x 296mm
   1680x1050      60.0*+
   1600x1200      60.0  
   1440x900       75.0     59.9  
   1400x1050      60.0  
   1360x765       60.0  
   1280x1024      75.0  
   1280x960       60.0  
   1152x864       75.0  
   1024x768       75.0     70.1     60.0  
   800x600        75.0     72.2     60.3     56.2  
   640x480        75.0     72.8     59.9  
HDMI-0 disconnected (normal left inverted right x axis y axis)
DVI-I-3 disconnected (normal left inverted right x axis y axis)


Fix display issue in Linux - after installing NVIDIA driver, display stuck - blackMORE Ops -1
Bugger all, this doesn’t help me to find supported HorizSync VertRefresh rates. I went around looking for options and found this tool that will do exactly what you need to find.

Find monitor HorizSync VertRefresh rates with ddcprobe

First we need to install xresprobe which contains ddcprobe.
root@kali:~# apt-get install xresprobe
Fix display issue in Linux - after installing graphics driver, display stuck - Detect supported VertRefresh and HorizSync values - blackMORE Ops -2

Once xresprobe is installed, then we can run the following command to find all supported monitor HorizSync VertRefresh rates including supported Display Resolution … well the whole lot .. some even I wasn’t aware.
root@kali:~# ddcprobe 
vbe: VESA 3.0 detected.
oem: NVIDIA
vendor: NVIDIA Corporation
product: GF104 Board - 10410001 Chip Rev
memory: 14336kb
mode: 640x400x256
mode: 640x480x256
mode: 800x600x16
mode: 800x600x256
mode: 1024x768x16
mode: 1024x768x256
mode: 1280x1024x16
mode: 1280x1024x256
mode: 320x200x64k
mode: 320x200x16m
mode: 640x480x64k
mode: 640x480x16m
mode: 800x600x64k
mode: 800x600x16m
mode: 1024x768x64k
mode: 1024x768x16m
mode: 1280x1024x64k
mode: 1280x1024x16m
edid: 
edid: 1 3
id: 000d
eisa: ACR000d
serial: 7430d0b5
manufacture: 43 2007
input: analog signal.
screensize: 47 30
gamma: 2.200000
dpms: RGB, active off, suspend, standby
timing: 720x400@70 Hz (VGA 640x400, IBM)
timing: 720x400@88 Hz (XGA2)
timing: 640x480@60 Hz (VGA)
timing: 640x480@67 Hz (Mac II, Apple)
timing: 640x480@72 Hz (VESA)
timing: 640x480@75 Hz (VESA)
timing: 800x600@60 Hz (VESA)
timing: 800x600@72 Hz (VESA)
timing: 800x600@75 Hz (VESA)
timing: 832x624@75 Hz (Mac II)
timing: 1024x768@87 Hz Interlaced (8514A)
timing: 1024x768@70 Hz (VESA)
timing: 1024x768@75 Hz (VESA)
timing: 1280x1024@75 (VESA)
ctiming: 1600x1200@60
ctiming: 1152x864@75
ctiming: 1280x960@60
ctiming: 1360x850@60
ctiming: 1440x1440@60
ctiming: 1440x1440@75
ctiming: 1400x1050@60
dtiming: 1680x1050@77
monitorrange: 31-84, 56-77
monitorserial: LAV0C0484010
monitorname: X223W
root@kali:~#
Now  the line I am interested is this:
monitorrange: 31-84, 56-77
That’s the highest supported HorizSync VertRefresh rates for my monitor.
Fix display issue in Linux - after installing graphics driver, display stuck - Detect supported VertRefresh and HorizSync values with ddcprobe - blackMORE Ops -3
ddcprobe also gave me few more useful info, like MonitorName and Monitor Serial.
monitorserial: LAV0C0484010
monitorname: X223W
Now time to put it all together.

Edit xorg.conf file to with correct HorizSync VertRefresh rates

So now we know the exact values we need to know. We can now edit our /etc/X11/xorg.conf file with the values we want. So I’ve edited my xorg.conf file to look like the following:
root@kali:~# vi /etc/X11/xorg.conf

Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Acer X223W"
    HorizSync       31.0 - 84.0
    VertRefresh     56.0 - 77.0
    Option         "DPMS"
EndSection
Save and exit xorg.conf file, restart and I am now enjoying 1680x1050 display on my Monitor. Here’s the xorg.conf file I have right now:
# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig:  version 304.48  (pbuilder@cake)  Wed Sep 12 10:54:51 UTC 2012

# nvidia-settings: X configuration file generated by nvidia-settings
# nvidia-settings:  version 304.88  (pbuilder@cake)  Wed Apr  3 08:58:25 UTC 2013

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"

    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Acer X223W"
    HorizSync       31.0 - 84.0
    VertRefresh     56.0 - 77.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 460"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "metamodes" "nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection
This fixed my problem quite well. It might be useful to someone else out there.

Reference Websites and posts

The biggest help is always X.org website for any display related issues.
http://www.x.org/wiki/FAQVideoModes
I also later realized the that Eddy posted a similar problem in one of my posts where he fixed this problem too in exactly similar way.
doh! I should’ve just searched my own posts and readers comments. Eddy’s post doesn’t outline how to find the HorizSync VertRefresh rates though. Either way, Eddy’s post was the most accurate I found related with my problem.

How to harden Apache web server with mod_security and mod_evasive on CentOS

 http://xmodulo.com/2014/09/harden-apache-web-server-mod_security-mod_evasive-centos.html

Web server security is a vast subject, and different people have different preferences and opinions as to what the best tools and techniques are to harden a particular web server. With Apache web server, a great majority of experts -if not all- agree that mod_security and mod_evasive are two very important modules that can protect an Apache web server against common threats.
In this article, we will discuss how to install and configure mod_security and mod_evasive, assuming that Apache HTTP web server is already up and running. We will perform a demo stress test to see how the web server reacts when it is under a denial-of-service (DOS) attack, and show how it fights back with these modules. We will be using CentOS platform in this tutorial.

Installing mod_security & mod_evasive

If you haven't enabled the EPEL repository in your CentOS/RHEL server, you need to do so before installing these packages.
# yum install mod_security
# yum install mod_evasive
After the installation is complete, you will find the main configuration files inside /etc/httpd/conf.d:

Now you need to make sure that Apache loads both modules when it starts. Look for the following lines (or add them if they are not present) in mod_security.conf and mod_evasive.conf, respectively:
1
2
LoadModule security2_module modules/mod_security2.so
LoadModule evasive20_module modules/mod_evasive20.so
In the two lines above:
  • The LoadModule directive tells Apache to link in an object file (*.so), and adds it to the list of active modules.
  • security2_module and evasive20_module are the names of the modules.
  • modules/mod_security2.so and modules/mod_evasive20.so are relative paths from the /etc/httpd directory to the source files of the modules. This can be verified (and changed, if necessary) by checking the contents of the /etc/httpd/modules directory.

Now restart Apache web server:
# service httpd restart

Configuring mod_security

In order to use mod_security, a Core Rule Set (CRS) must be installed first. Basically, a CRS provides a web server with a set of rules on how to behave under certain conditions. Trustwave's SpiderLabs (the firm behind mod_security) provides the OWASP (Open Web Application Security Project) ModSecurity CRS.
To download and install the latest OWASP CRS, use the following commands.
# mkdir /etc/httpd/crs
# cd /etc/httpd/crs
# wget https://github.com/SpiderLabs/owasp-modsecurity-crs/tarball/master
# tar xzf master
# mv SpiderLabs-owasp-modsecurity-crs-ebe8790 owasp-modsecurity-crs
Now navigate to the installed OWASP CRS directory.
# cd /etc/httpd/crs/owasp-modsecurity-crs
In the OWASP CRS directory, you will find a sample file with rules (modsecurity_crs_10_setup.conf.example).

We will copy its contents into a new file named (for convenience) modsecurity_crs_10_setup.conf.
# cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_setup.conf
To tell Apache to use this file for mod_security module, insert the following lines in the /etc/httpd/conf/httpd.conf file. The exact paths may be different depending on where you unpack the CRS tarball.
1
2
3
4
    Include crs/owasp-modsecurity-crs/modsecurity_crs_10_setup.conf
    Include crs/owasp-modsecurity-crs/base_rules/*.conf
</IfModule>
Last, but not least, we will create our own configuration file within the modsecurity.d directory where we will include our chosen directives. We will name this configuration file xmodulo.conf in this example. It is highly encouraged that you do not edit the CRS files directly but rather place all necessary directives in this configuration file. This will allow for easier upgrading as newer CRSs are released.
# vi /etc/httpd/modsecurity.d/xmodulo.conf
1
2
3
4
5
6
7
    SecRuleEngine On
    SecRequestBodyAccess On
    SecResponseBodyAccess On
    SecResponseBodyMimeType text/plain text/html text/xml application/octet-stream
    SecDataDir /tmp
</IfModule>
  • SecRuleEngine On: Use the OWASP CRS to detect and block malicious attacks.
  • SecRequestBodyAccess On: Enable inspection of data transported request bodies (e.g., POST parameters).
  • SecResponseBodyAccess On: Buffer response bodies (only if the response MIME type matches the list configured with SecResponseBodyMimeType).
  • SecResponseBodyMimeType text/plain text/html text/xml application/octet-stream: Configures which MIME types are to be considered for response body buffering. If you are unfamiliar with MIME types or unsure about their names or usage, you can check the Internet Assigned Numbers Authority (IANA) web site.
  • SecDataDir /tmp: Path where persistent data (e.g., IP address data, session data, and so on) is to be stored. Here persistent means anything that is not stored in memory, but on hard disk.
You can refer to the SpiderLabs' ModSecurity GitHub repository for a complete guide of configuration directives.
Don't forget to restart Apache to apply changes.

Configuring mod_evasive

The mod_evasive module reads its configuration from /etc/httpd/conf.d/mod_evasive.conf. As opposed to mod_security, we don't need a separate configuration file because there are no rules to update during a system or package upgrade.
The default mod_evasive.conf file has the following directives enabled:
1
2
3
4
5
6
7
8
    DOSHashTableSize    3097
    DOSPageCount        2
    DOSSiteCount        50
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   10
</IfModule>
  • DOSHashTableSize: The size of the hash table that is used to keep track of activity on a per-IP address basis. Increasing this number will provide a faster look up of the sites that the client has visited in the past, but may impact overall performance if it is set too high.
  • DOSPageCount: The number of identical requests to a specific URI (for example, a file that is being served by Apache) a visitor can make over the DOSPageInterval interval.
  • DOSSiteCount: similar to DOSPageCount, but refers to how many overall requests can be made to the site over the DOSSiteInterval interval.
  • DOSBlockingPeriod: If a visitor exceeds the limits set by DOSSPageCount or DOSSiteCount, he/she will be blacklisted for the DOSBlockingPeriod amount of time. During this interval, any requests coming from him/her will return a 403 Forbidden error.
You may want to change these values according to the amount and type of traffic that your web server needs to handle. Please note that if these values are not set properly, you may end up blocking legitimate visitors.
Here are other useful directives for mod_evasive:
1) DOSEmailNotify: Sends an email to the address specified whenever an IP address becomes blacklisted. It needs a valid email address as argument. If SELinux status is set to enforcing, you will need to grant the user apache SELinux permission to send emails. That is, run this command as root:
# setsebool -P httpd_can_sendmail 1
Then add this directive in the mod_evasive.conf file:
1
DOSEmailNotify you@yourdomain.com
2. DOSSystemCommand: Executes a custom system command whenever an IP address becomes blacklisted. It may come in handy to add firewall rules to block offending IPs altogether.
1
DOSSystemCommand <command>
We will use this directive to add a firewall rule through the following script (/etc/httpd/scripts/ban_ip.sh):
1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/sh
# Offending IP as detected by mod_evasive
IP=$1
# Path to iptables binary executed by user apache through sudo
IPTABLES="/sbin/iptables"
# mod_evasive lock directory
MOD_EVASIVE_LOGDIR=/tmp
# Add the following firewall rule (block IP)
$IPTABLES -I INPUT -s $IP -j DROP
# Unblock offending IP after 2 hours through the 'at' command; see 'man at' for further details
echo "$IPTABLES -D INPUT -s $IP -j DROP" | at now + 2 hours
# Remove lock file for future checks
rm -f "$MOD_EVASIVE_LOGDIR"/dos-"$IP"
Our DOSSystemCommand directive will then read as follows:
1
DOSSystemCommand "sudo /etc/httpd/scripts/ban_ip.sh %s"
Don't forget to update sudo permissions to run our script as apache user:
# vi /etc/sudoers
1
2
apache ALL=NOPASSWD: /usr/local/bin/scripts/ban_ip.sh
Defaults:apache !requiretty

Simulating DoS Attacks

We will use three tools to stress test our Apache web server (running on CentOS 6.5 with 512 MB of RAM and a AMD Athlon II X2 250 Processor), with and without mod_security and mod_evasive enabled, and check how the web server behaves in each case.
Make sure you ONLY perform the following steps in your own test server and NOT against an external, production web site.
In the following examples, replace http://centos.gabrielcanepa.com.ar/index.php with your own domain and a file of your choosing.

Linux-based tools

1. Apache bench: Apache server benchmarking tool.
# ab -n1000 -c1000 http://centos.gabrielcanepa.com.ar/index.php
  • -n: Number of requests to perform for the benchmarking session.
  • -c: Number of multiple requests to perform at a time.
2. test.pl: a Perl script which comes with mod_evasive module.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/usr/bin/perl
 
# test.pl: small script to test mod_dosevasive's effectiveness
 
use IO::Socket;
use strict;
 
for(0..100) {
  my($response);
  my($SOCKET) = new IO::Socket::INET( Proto   => "tcp",
                                      PeerAddr=> "192.168.0.16:80");
  if (! defined $SOCKET) { die $!; }
  print $SOCKET "GET /?$_ HTTP/1.0\n\n";
  $response = <$SOCKET>;
  print $response;
  close($SOCKET);
}

Windows-based tools

1. Low Orbit Ion Cannon (LOIC): a network stress testing tool. To generate a workload, follow the order shown in the screenshot below and DO NOT touch anything else

Stress Test Results

With mod_security and mod_evasive enabled (and the three tools running at the same time), the CPU and RAM usage peak at a maximum of 60% and 50%, respectively for only 2 seconds before the source IPs are blacklisted, blocked by the firewall, and the attack is stopped.
On the other hand, if mod_security and mod_evasive are disabled, the three tools mentioned above knock down the server very fast (and keep it in that state throughout the duration of the attack), and of course, the offending IPs are not blacklisted.

Conclusion

We can see that mod_security and mod_evasive, when properly configured, are two important tools to harden an Apache web server against several threats (not limited to DoS attacks) and should be considered in deployments exposed on the Internet.