Thursday, November 27, 2014

How to use inotify-tools to trigger scripts on filesystem events

http://techarena51.com/index.php/inotify-tools-example

In my last example of Bash if statements we created a backup script which would run “rsync” as per the time set in a cron.

But wouldn’t it be nice if we could run our “rsync” script only if there was a change to the files in our directory instead of running a cron every hour despite no change? With inotify-tools you can do just that.

What is inotify-tools?
Inotify tools are a set of command line programs based on inotify a Linux kernel (2.6.13 or later) feature which can be used to monitor filesystem events.

Installing inotify-tools
Software versions :

inotify-tools.x86_64 3.14-1.el6
CentOS 6.5
Linux kernel 2.6.32-042stab085.20
#Add the epel repo
[leo@linux-vps ~]$sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
#Install inotify-tools
[leo_g@vps ~]$ sudo yum install inotify-tools

Inotify-tools has two commands

Inotifywait
This command simply blocks for inotify events, making it appropriate for use in shell scripts. It can watch any set of files and directories, and can recursively watch entire directory trees.

inotifywatch
inotifywatch collects filesystem usage statistics and outputs counts of each inotify event.
We will use inotifywait, since we do not need statistics.

inotifywait syntax
inotifywait filename

inotifywait example
[leo@linux-vps ~]$ inotifywait /tmp
Setting up watches.
Watches established.
/tmp/ MODIFY test

As you can see in the above example an event (in this case a “modify action” was performed on a file named “test” inside /tmp) triggered the output.

Now “inotifywait” by default checks for all events including if a file was opened but not written to, Since we only want “rsync” to trigger on change events like when a file is modified, we will need to specify the “-e” flag along with the list of events we want to be notified about.

[leo@linux-vps ~]$ inotifywait -m -r -e \ modify,attrib,close_write,move,create,delete /tmp
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ MODIFY a
/tmp/ CLOSE_WRITE,CLOSE a

The -m flag is for continuous monitoring as by default inotifywait will exit on the first event and -r means recursively or check through sub-directories as well.

List of inotifywait events
 
access
A watched file or a file within a watched directory was read from.
 
modify
A watched file or a file within a watched directory was written to.
 
attrib
The metadata of a watched file or a file within a watched directory was modified. This includes timestamps, file permissions, extended attributes etc.
 
close_write
A watched file or a file within a watched directory was closed, after being opened in writeable mode. This does not necessarily imply the file was written to.
 
close_nowrite
A watched file or a file within a watched directory was closed, after being opened in read-only mode.
 
close
A watched file or a file within a watched directory was closed, regardless of how it was opened. Note that this is actually implemented simply by listening for both close_write and close_nowrite, hence all close events received will be output as one of these, not CLOSE.
 
open
A watched file or a file within a watched directory was opened.
 
moved_to
A file or directory was moved into a watched directory. This event occurs even if the file is simply moved from and to the same directory.
 
moved_from
A file or directory was moved from a watched directory. This event occurs even if the file is simply moved from and to the same directory.
 
move
A file or directory was moved from or to a watched directory. Note that this is actually implemented simply by listening for both moved_to and moved_from, hence all close events received will be output as one or both of these, not MOVE.
 
move_self
A watched file or directory was moved. After this event, the file or directory is no longer being watched.
 
create
A file or directory was created within a watched directory.
 
delete
A file or directory within a watched directory was deleted.
 
delete_self
A watched file or directory was deleted. After this event the file or directory is no longer being watched. Note that this event can occur even if it is not explicitly being listened for.
 
unmount
The filesystem on which a watched file or directory resides was unmounted. After this event the file or directory is no longer being watched. Note that this event can occur even if it is not explicitly being listened to.
Now let’s use inotifywait with our script.
[leo@linux-vps~]$ vim inotify-example
while true #run indefinitely
do 
inotifywait -r -e modify,attrib,close_write,move,create,delete /dir && /bin/bash backup-script
done

Since we want to continuously monitor changes, we use an infinite while loop and the Logic “&&” operator will ensure that our backup script is only triggered on a successful completion of the inotifywait event
 
[leo_g@vps scripts]$bash inotify-example 
+ true
+ inotifywait -r -e modify,attrib,close_write,move,create,delete /
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/dir DELETE a
+ /bin/bash backup-script 
+ rsync -avz  -e “ssh ” /path/to/yourfile user@backupserver.com:/backup/ 2> \ /tmp/error.txt
+ mail -s “backup complete” user@youremail.com
+ echo “backup for $(date) “
+ true
+ inotifywait -r -e modify,attrib,close_write,move,create,delete /dir
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

If you have a suggestion feel free to let me know. The complete source code to the file syncing daemon with rsync and inotify can be found at https://github.com/Leo-g/backup-bash

You mat also like How to display a changing output like top
source https://github.com/rvoicilas/inotify-tools/wiki

Encrypt Everything: How to encrypt the disk to protect the data

http://blog.adityapatawari.com/2014/11/encrypt-everything-how-to-encrypt-disk.html

Recently, at BrowserStack.com, some of our services got compromised. We use Amazon Web Services extensively. The person (or group) who attacked us mounted one of our backups and managed to steal some of the data. We could have prevented this simply by ensuring that we use encrypted disks which would have made this attack useless. Learning from our mistakes, we have recently started encrypting everything and I am going to show you how to do that. One point worth noting here is that Amazon AWS does provide encryption support for the EBS volumes but that is transparent and would not help in case of the account getting compromised. I am going to use dm-crypt which is supported by Linux kernel so the steps are quite generic and would work on any kind of disk, on any kind of environment, including Amazon AWS, Google Compute Engine, physical disks in your datacenter.

Our goal is to encrypt /home. To achieve this, we'll attach a disk, encrypt it, move the entire /home data to this disk and create a symbolic link to /home.

Step1: We are going to use Linux Unified Key Setup. For that we need to install cryptsetup package.
# yum install cryptsetup

Step2: While using AWS, never attach the volume to be encrypted while launching the instance. If we do so, the instance will fail to boot up next time because it'll ask for decryption password while booting up which is not possible to supply in AWS. Still if it is absolutely mandatory to do this then I suggest trying to remove entries from fstab and crypttab but it is much easier to just attach the disk after the launching of the instance is done. Assuming that the attached disk is available at /dev/xvdf, we'll setup the encryption now.
# cryptsetup -y -v luksFormat /dev/xvdf
WARNING!
========
This will overwrite data on /dev/xvdf irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:

Command successful.


We can verify the encryption parameters as well. Default is AES 256 bit.
# cryptsetup luksDump /dev/xvdf

Step3: We'll open the device and map it to /dev/mapper/home so that we can use it.
# cryptsetup luksOpen /dev/xvdf home
Enter passphrase for /dev/xvdf:


Step4: This step is optional. To further protect our data, we can zero out the entire disk before even creating the filesystem.
# dd if=/dev/zero of=/dev/mapper/home

Step5: Now we'll create a filesytem
# mkfs.ext4 /dev/mapper/home

Step6: Let us mount and copy the data from /home
# mkdir /myhome
# mount /dev/mapper/home /myhome
# cp -a /home/* /myhome/
# rm -rf /home
# ln -s /myhome /home

Great! Our /home directory is encrypted. But wait a minute.. this approach has a short coming. We have deliberately designed it so that the disk won't auto-mount during the boot because there is no way to give it a password in cloud environment during the boot. Since the disk won't mount, we won't be able to ssh into the machine because the authorized_keys file is kept inside the home directory of the user. To address this problem, either change the "AuthorizedKeysFile" in sshd_config or create a user with home directory in /var/lib or /opt and grant sudo for cryptsetup and mount commands. So after reboot, if we take the first approach, we would be able to ssh without any problem or we'll ssh via other user, mount the encrypted drive and then use it normally.

$ ssh mountuser@
$ sudo /sbin/cryptsetup luksOpen /dev/xvdf home
$ sudo /bin/mount /dev/mapper/home /myhome/


Couple of points to remember:

  • Do not forget the LUKS password. It cannot be retrieved, if lost.
  • Try it a couple of times on staging machines before doing it on the machines that matter.

How to install Docker on CentOS 7

http://www.linuxtechi.com/install-docker-on-centos-7

Docker is an open-source tool that makes creating & managing Linux containers(LXC) easy. Containers are like lightweight VMs which can be started & stopped in milliseconds. Dockers help the system admin & coders to develop their application in a container and can further scale up to 1000 of nodes.
The main difference between container and VM(Virtual machine) is that dockers provide process based isolation , whereas VM provides full isolation of resources. Virtual machine takes a minute to start where as container can be started in a second or less than a second. Container uses the Kernel of host OS , whereas VM uses the separate Kernel.
One of the limitation of Docker is that it can be used only on 64bit hosts OS.
In this post we will discuss how to install docker in CentOS 7.x

Installation of Docker on CentOS 7

Docker package is included in the default CentOS-Extras repository. So to install docker , simply run below yum command :
[root@localhost ~]# yum install docker

Start the Docker Service

Once the Installation is finished , start docker service and enable it at boot using below commands
[root@localhost ~]# service docker start
[root@localhost ~]# chkconfig docker on
Download the official Centos images Locally
[root@localhost ~]# docker pull centos
Pulling repository centos
192178b11d36: Download complete 
70441cac1ed5: Download complete 
ae0c2d0bdc10: Download complete 
511136ea3c5a: Download complete 
5b12ef8fd570: Download complete
 Verify CentOS images that have been fetched :
[root@localhost ~]# docker images centos
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
centos              centos5             192178b11d36        2 weeks ago         466.9 MB
centos              centos6             70441cac1ed5        2 weeks ago         215.8 MB
centos              centos7             ae0c2d0bdc10        2 weeks ago         224 MB
centos              latest              ae0c2d0bdc10        2 weeks ago         224 MB

Run a Docker Container :

[root@localhost ~]# docker run -i -t centos /bin/bash
[root@dbf66395436d /]#
As we can see centos container has been started and we got the bash shell. In docker command we have used options like ‘-i attaches stdin and stdout’ and ‘-t allocates a terminal or console’ . To disconnect from container type exit .
[root@cd05639b3f5c /]# cat /etc/redhat-release 
CentOS Linux release 7.0.1406 (Core) 
[root@cd05639b3f5c /]# exit
exit
[root@localhost ~]#
We can also search Containers based on fedora & ubuntu OS.
[root@localhost ~]# docker search ubuntu
[root@localhost ~]# docker search fedora
Display the list of running containers
docker-ps