Mastering The Linux Shell : Standard In, Out, and Error

http://marcelgagne.com/content/mastering-linux-shell-standard-out-and-error

Welcome to part four in my Mastering The Linux Shell series where we will wander wistfully into a land where three terribly under-appreciated but vitally important files live. You will remember that everything is a file, including directories which some people call folders. What you may not know is that your keyboard is a file, as is the screen on which you read these words. And if all hell breaks loose, all that goes into a file too. Those special files, and variations on those files, are always in use, unlike those other, less hard-working files that seem to get all the credit. They are called STDIN, STDOUT, and STDERR.
Standard in (STDIN) is where the system expects to find its input. This is usually the keyboard, although it can be a program or shell script. When you change that default, you call it redirecting from STDIN.
Standard out (STDOUT) is where the system expects to direct its output, usually the terminal screen. Again, redirection of STDOUT is at the discretion of whatever command or script is executing at the time. The chain of events from STDIN to STDOUT looks something like this:

standard in  -> Linux command  ->  standard out

STDIN is often referred to as fd0, or file descriptor 0, while STDOUT is usually thought of as fd1. There is also standard error (STDERR), where the system reports any errors in program execution. By default, this is also the terminal. To redirect STDOUT, use the greater-than sign (>). As you might have guessed, to redirect from STDIN, you use the less-than sign (<). But what exactly does that mean? Let’s try an experiment. Randomly search your brain and pick a handful of names. Got them? Good. Now type the cat command and redirect its STDOUT to a file called random_names.

cat > random_names

Your cursor will just sit there and wait for you to do something, so type those names, pressing Enter after each one. What’s happening here is that cat is taking its input from STDIN and writing it out to your new file. You can also write the command like this:

cat - 1> random_names

The hyphen literally means standard in to the command. The 1 stands for file descriptor 1. This is good information, and you will use it later. Finished with your random names list? When you are done, press to finish. , by the way, stands for EOF, or end of file.

Marie Curie
Albert Einstein
Mark Twain
Wolfgang Amadeus Mozart
Stephen Hawking
Hedy Lamarr
^D

If you cat this file, the names will be written to STDOUT, in this case, your terminal window. You can also give cat several files at the same time. For instance, you could do something like this:

cat file1 file2 file3

Each file would be listed one right after the other. That output could then be redirected into another file. You could also have it print out the same file over and over (cat random_names random_names random_names). cat isn’t fussy about these things and will deal with binary files (programs) just as quickly. Beware of using cat to print out the contents of a program to your terminal screen. At worst, your terminal session will lock up or reward you with a lot of beeping and weird characters.

If you get caught in such a situation and all the characters on your screen appear as junk, try typing echo and then pressing and . If you can still type, you can also try typing stty sane and then pressing . Some systems also provide a command called reset which will return your terminal session to some kind of sane look.

Redirecting STDIN works pretty much the same way, except that you use the less-than sign instead. Using the sort command, let’s take that file of random names and work with it. Many commands that work with files can take their input directly from that file. Unless told otherwise, cat and sort will think that the word following the command is a filename. That’s why you did the STDIN redirection thing. Yes, that’s right: STDIN is just another file. Sort of.

sort random_names

The result, of course, is that you get all your names printed out in alphabetical order. You could have also specified that sort take its input from a redirected STDIN. It looks a bit strange, but this is perfectly valid.

$ sort < random_names
Albert Einstein
Hedy Lamarr
Marie Curie
Mark Twain
Stephen Hawking
Wolfgang Amadeus Mozart

One more variation involves defining your STDIN (as you did previously) and specifying a different STDOUT all on the same line. In the following example, I am redirecting from my file and redirecting that output to a new file:

sort < random_names > sorted_names

Pipes and Piping

Sometimes the thing that makes the most sense is to feed the output from one command directly into another command without having to resort to files in between at every step of the way. This is called piping. The symbolism is not that subtle: Imagine pieces of pipe connecting one command with another. Not until you run out of pipe does the command’s output emerge. The pipe symbol is the broken vertical bar on your keyboard usually located just below or (depending on the keyboard) just above the Enter key and sharing space with the backslash key. Here’s how it works:

cat random_names | sort | wc –w > num_names

In the preceding example, the output from the cat command is piped into sort, whose output is then piped into the wc command (that’s "word count"). The -w flag tells wc to count the number of words in random_names. So far, so good.
That cat at the beginning is actually redundant, but I wanted to stack up a few commands for you to give you an idea of the power of piping. Ordinarily, I would write that command as follows:

sort random_names | wc –w > num_names

The cat is extraneous because sort incorporates its function. Using pipes is a great timesaver because you don’t always need to have output at every step of the way.

tee: A Very Special Pipe

Suppose that you want to send the output of a command to another command, but you also want to see the results at some point. Using the previous word count example, if you want a sorted list of names, but you also want the word count, you might have to use two different commands: one to generate the sorted list and another to count the number of words. Wouldn’t it be nice if you could direct part of the output one way and have the rest continue in another direction? For this, use the tee command.

sort random_names | tee sorted_list | wc –w > num_names

The output from sort is now sitting in a file called sorted_list, while the rest of the output continues on to wc for a word count.

STDERR

What about STDERR? Some commands (many, in fact) treat the error output differently than the STDOUT. If you are running the command at your terminal and that’s all you want, there is no problem. Sometimes, though, the output is quite wordy and you need to capture it and look at it later. Unfortunately, using the STDOUT redirect (the greater-than sign) is only going to be so useful. Error messages that might be generated (such as warning messages from a compilation) will go to the terminal as before. One way to deal with this is to start by redirecting STDERR to STDOUT, and then redirect that to a file. Here’s the line I use for this:

command_name 2>&1 > logfile.out

Remember that file descriptor 2 is STDERR and that file descriptor 1 is STDOUT. That’s what that 2>&1 construct is all about. You are redirecting fd2 to fd1 and then redirecting that output to the file of your choice. Using that program compilation example, you might wind up with something like this:

make –f Makefile.linux 2>&1 > compilation.output

The final greater-than sign in the preceding example could be eliminated completely. When using the 2>&1 construct, it is assumed that what follows is a filename.

The Road to Nowhere

If the command happens to be verbose by nature and doesn’t have a quiet switch (that's usually a -q option), you can redirect that STDOUT and STDERR noise to what longtime  Linux users like to call the bit bucket, a special file called /dev/null, literally, a road to nowhere. Anything fed to the bit bucket takes up no space and is never seen or heard from again. When I was in school, way back in the before time, we would tell people to shut up by saying, "Dev null it, will you?" As you can see, we were easily amused.
To redirect output to the bit bucket, use the STDOUT redirection.

command –option > /dev/null

If, for some strange reason, you want to sort the output of the random_names files and you do not want to see the output, you can redirect the whole thing to /dev/null in this way:

sort random_names > /dev/null

Using the program compilation example where you had separate STDOUT and STDERR streams, you can combine the output to the bit bucket.

make –F makefile.linux 2>&1 /dev/null

That’s actually a crazy example because you do want to see what goes on, but redirecting both STDOUT and STDERR to /dev/null is quite common when dealing with automated processes running in the background.
And once again, it's time to wrap it up. Look for part 5 of this series coming your way on Monday. And that is where I will leave this discussion for today. Leave any comments you may have here on Google Plus or over here on Facebook. Join me this coming Thursday for part four in this series. Remember that you can also follow the action on CookingWithLinux.com where it's all Linux, all the time. Except for the occasional wine review.

4 gui applications for installing Linux from USB key

http://www.linuxbsdos.com/2013/03/26/linux-live-usb-best-tool-to-install-linux-on-usb


The traditional and most common method of installing Linux is by burning the installation ISO image to a CD or DVD. But with many laptops, notebooks, ultra notebooks, subnotebooks shipping without an optical drive, installation via USB flash stick has become the most common method for installing Linux on these types of computers. That’s were a Linux USB installer come into play.
In our neck of the woods, where virtually all computers running a Linux or UNIX-like operating system did not ship with them pre-installed, users are very familiar with the tools to burn or transfer their favorite distribution to removable media. The same cannot be said for those on the other side.
With this article, four of the best graphical applications that Windows users may use to transfer most actively-developed Linux distributions to a USB key, and, therefore, use it to install the Linux distribution to a computer that does not have an optical drive are presented.
Note that these applications do not just make it easy to install a Linux distribution from a USB key, but also allow you to run supported distributions in Live mode, that is, run it from the USB stick without installing it to the computer’s hard disk drive (HDD). In that (Live) mode, they also allow you to have persistent storage on the USB stick, such that any data you store on the USB key is not lost when the computer is rebooted.
They also have other features in common. For example, you can install an installation image of supported distributions stored locally, or they can automatically download a supported distribution from the nearest download mirror. Because they don’t have support for the very latest of many distributions, I find it more convenient to manually download my distribution of choice and point the application to the download directory.
In no particular order, the applications are:
1. LinuxLive USB Creator: This has a beautiful and flowery graphical interface. It is a Windows-only application and has supported for many popular distributions. I find it kinda fun to use. LinuxLive USB Creator or LiLi, may be downloaded from here.

A distribution can be downloaded from the nearest mirror.

2. Unetbootin: This runs on Windows and Linux. Its interface is not as fancy as the LinuxLive USB Creator’s, but it gets the job done.

Its list of supported distributions tend to be one or two revisions removed from the current stable release.

For example, the last “stable” release of BackTrack is BackTrack 5 R3, which has since been replaced by Kali Linux. Unetbootin is available for download here.

3. Universal USB Installer: This runs only on Windows. Universal USB Installer (UUI). Like Unetbootin and LinuxLive USB Creator, UUI does not support all distributions, but it does give you the option of attempting to install an unsupported distribution on a USB Flash drive. I tried it with Mageia 2, and it did not work. You may download UUI from Universal USB Installer.

4. Your Universal Multiboot Installer: This is from the same author as UUI, and it, too, runs only on Windows. The only feature that sets Your Universal Multiboot Installer (YUMI) apart from the others is the claim that it can install more than one distribution on the same USB key.

I haven’t tried it yet, but I’ll attempt it before the end of the day and let you know what happened. YUMI is available for download here.

Being able to install a Linux distribution from a USB key or USB flash drive is neat, but when it comes to running it in Live mode and storing persistent data on it, I’d rather install the distribution on an SSD or a small, external HDD. Then I don’t have to worry about how much persistent storage space I can allocate to it.

Linux processes explained

http://mylinuxbook.com/linux-processes-part1


Generally, on any operating system, we say we have so many programs running. These running programs introduce the concept of processes. Let’s Define as to what is a process – A process is a program in execution.
Robert Love expresses his definition of a process in one of his books as :

The Process is one of the fundamental abstractions in Unix Operating Systems, the other fundamental abstraction being files

Linux is a multi-user and multi-tasking operating system(seemingly, discussed later in the article). A Linux process is a program in execution on a Linux system. Therefore, whenever a program is executed, a new process is created. A process also consumes resources like the file system, memory or other CPU resources. This gives rise to the need of process management in Linux.

Identifier for Linux Processes

In Linux, every process has a unique process Identifier(ID) associated to it. A process ID( i.e. PID) is a number which is uniquely assigned as soon as the process is created. The PID’s are allocated sequentially as the processes are being created. However, it generally starts from 2, as PID=1 is reserved for ‘init process. As we always expect, there is a maximum limit to the PID value. In a system, the way one can get to know the maximum limit to PID is

$ cat /proc/sys/kernel/pid_max

For me, I got the following value

32768

Hence, whenever the sequentially allocated PID reaches the maximum value, it wraps to the lower limit(generally 300) and the next PID’s allocated are the available ones starting from the lower limit.
The PID of the process, as the name suggests is its identifier. Hence, most of the operations being done on a process needs the PID to be mentioned.
We shall see in the following sections, how do we see display all the processes with their PID’s and various operations that can be performed on a process.

Listing Processes

At any moment, the Linux user can view the list of all the processes which have been created and not terminated. There is a reason I don’t say all the processes which are running, as once a process has been started, it can be in any state, not necessarily running. More about process states in further sections.
The Linux command used to view list of processes is ‘ps’ which means ‘process status’ (Some authors also interpret it as ‘process snapshot’).
To see what all this Linux command has to offer in detail, the best source is the man-page.
Let us try out running the command

$ ps

 PID TTY          TIME CMD

1779 pts/0    00:00:00 bash

2176 pts/0    00:00:00 ps

We just see two processes! Although we are sure there are other processes being running as well. Well, the ‘ps’ command without any options just lists the processes which are created by the current terminal. The first one is the ‘bash’ which is the running linux shell by the terminal and other is the process created by ‘ps’ command itself.
Getting to know from the ‘ps’ output, let’s walk through it column by column.

• PID – The process Identifier which is ‘1779’ for ‘bash’ and ‘2176’ for ‘ps’.
• TTY – stands for terminal-type and is the name of the console/terminal, the process is associated to.

Note: To determine the name of your terminal, use command ‘tty’.

• TIME – The CPU time since the process has started. It is confusing that why the CPU time for ‘bash’ process is ‘00:00:00’? This is because, CPU time is the time for which the process is being executed by the processor. However, when bash runs commands,lets say ‘ls command’ , a child process ‘ls’ is spawned and whatever execution and cpu utilization takes place, goes under the ‘ls’ process and not ‘bash’ Bash process is just the parent process.
• CMD – Command run to create the process.

There are many more options offered by the Linux command ‘ps’ to explore the various processes being launched in a system. Kindly go back to the referred man-page of ‘ps’ to get familiar with each and every available option. Here we shall be playing around with a few.

List All processes

$ps -e

If we look at the ‘ps’ man page, ‘-e’ option means “Select all processes”, which implies now our list of displayed processes is not limited to the ones by the current terminal. Instead, we’ll be able to see all the currently running processes. Note, this option is identical to ‘-A’.
Running the above command, we see a huge list of processes. Hence, to read them through reasonably, we pipe the output to ‘more’

$ ps -e | more

PID TTY          TIME CMD

1 ?        00:00:00 init

2 ?        00:00:00 kthreadd

3 ?        00:00:01 ksoftirqd/0

5 ?        00:00:00 kworker/u:0

6 ?        00:00:00 migration/0

7 ?        00:00:00 watchdog/0

8 ?        00:00:00 cpuset

9 ?        00:00:00 khelper

10 ?        00:00:00 kdevtmpfs

11 ?        00:00:00 netns

12 ?        00:00:00 sync_supers

13 ?        00:00:00 bdi-default

14 ?        00:00:00 kintegrityd

15 ?        00:00:00 kblockd

16 ?        00:00:00 ata_sff

17 ?        00:00:00 khubd

18 ?        00:00:00 md

21 ?        00:00:00 khungtaskd

22 ?        00:00:01 kswapd0

23 ?        00:00:00 ksmd

24 ?        00:00:00 fsnotify_mark

25 ?        00:00:00 ecryptfs-kthrea

26 ?        00:00:00 crypto

34 ?        00:00:00 kthrotld

35 ?        00:00:00 kworker/u:2

36 ?        00:00:00 scsi_eh_0

38 ?        00:00:00 scsi_eh_1

39 ?        00:00:00 scsi_eh_2

61 ?        00:00:00 devfreq_wq

200 ?   00:00:01 jbd2/sda1-8

201 ?   00:00:00 ext4-dio-unwrit

219 ?   00:00:05 flush-8:0

368 ?   00:00:00 upstart-udev-br

375 ?   00:00:00 udevd

503 ?   00:00:00 kpsmoused

628 ?   00:00:00 upstart-socket-

656 ?   00:00:05 rsyslogd

680 ?   00:00:00 dbus-daemon

708 ?   00:00:00 modem-manager

--more--

Observe, that for each process we have just following four details in the output, similar to what has been discussed in the section above:

PID     TTY          TIME        CMD

In order to get more details about each process, we use option ‘-f’ alongwith. This option will add more details in the form of more columns.

$ps -ef

Again, to read through the huge list of output, page by page, we pipe the output to ‘more’ looks like:

$ps -ef | more

UID        PID  PPID  C STIME TTY          TIME CMD

root         1     0  0 Mar08 ?        00:00:00 /sbin/init

root         2     0  0 Mar08 ?        00:00:00 [kthreadd]

root         3     2  0 Mar08 ?        00:00:01 [ksoftirqd/0]

root         5     2  0 Mar08 ?        00:00:00 [kworker/u:0]

root         6     2  0 Mar08 ?        00:00:00 [migration/0]

root         7     2  0 Mar08 ?        00:00:00 [watchdog/0]

root         8     2  0 Mar08 ?        00:00:00 [cpuset]

root         9     2  0 Mar08 ?        00:00:00 [khelper]

root        10     2  0 Mar08 ?        00:00:00 [kdevtmpfs]

root        11     2  0 Mar08 ?        00:00:00 [netns]

root        12     2  0 Mar08 ?        00:00:00 [sync_supers]

root        13     2  0 Mar08 ?        00:00:00 [bdi-default]

root        14     2  0 Mar08 ?        00:00:00 [kintegrityd]

root        15     2  0 Mar08 ?        00:00:00 [kblockd]

root        16     2  0 Mar08 ?        00:00:00 [ata_sff]

root        17     2  0 Mar08 ?        00:00:00 [khubd]

root        18     2  0 Mar08 ?        00:00:00 [md]

root        21     2  0 Mar08 ?        00:00:00 [khungtaskd]

root        22     2  0 Mar08 ?        00:00:01 [kswapd0]

root        23     2  0 Mar08 ?        00:00:00 [ksmd]

root        24     2  0 Mar08 ?        00:00:00 [fsnotify_mark]

root        25     2  0 Mar08 ?        00:00:00 [ecryptfs-kthrea]

root        26     2  0 Mar08 ?        00:00:00 [crypto]

root        34     2  0 Mar08 ?        00:00:00 [kthrotld]

root        35     2  0 Mar08 ?        00:00:00 [kworker/u:2]

root        36     2  0 Mar08 ?        00:00:00 [scsi_eh_0]

root        38     2  0 Mar08 ?        00:00:00 [scsi_eh_1]

root        39     2  0 Mar08 ?        00:00:00 [scsi_eh_2]

root        61     2  0 Mar08 ?        00:00:00 [devfreq_wq]

root       200     2  0 Mar08 ?        00:00:01 [jbd2/sda1-8]

root       201     2  0 Mar08 ?        00:00:00 [ext4-dio-unwrit]

root       219     2  0 Mar08 ?        00:00:05 [flush-8:0]

root       368     1  0 Mar08 ?        00:00:00 upstart-udev-bridge --daemon

root       375     1  0 Mar08 ?        00:00:00 /sbin/udevd --daemon

root       503     2  0 Mar08 ?        00:00:00 [kpsmoused]

root       628     1  0 Mar08 ?        00:00:00 upstart-socket-bridge --daemon

syslog     656     1  0 Mar08 ?        00:00:05 rsyslogd -c5

102        680     1  0 Mar08 ?        00:00:00 dbus-daemon --system --fork --activation=upstart

root       708     1  0 Mar08 ?        00:00:00 /usr/sbin/modem-manager

--more--

Well, we have a new set of details of the running processes.

• UID – The User ID,is the username of the user, which owns the process.
• PID – Already discusses the Process ID.
• PPID – It is the Parent Process ID. Mostly, all the processes, except the first one, have a parent process i.e. the process which has created our relevant process. Therefore, every process retains the PID of its parent process it calls as the parent process ID i.e. PPID, in its process descriptors. We shall learn about process descriptors in the next part of this article series. Meanwhile, one can understand process descriptors as some related parameters describing the process.

Therefore, we can conclude from here that, process can be represented as a tree (hierarchical) structure in Linux. To view the complete tree structure, linux provides a command – pstree.
It gives an interesting output, showcasing the first ‘init’ process and the other processes spawned out of it.
The process tree snapshot on my ubuntu system is:

$ pstree

init─┬─NetworkManager─┬─dhclient

    │                ├─dnsmasq

    │                └─2*[{NetworkManager}]

    ├─accounts-daemon───{accounts-daemon}

    ├─acpid

    ├─at-spi-bus-laun───2*[{at-spi-bus-laun}]

    ├─atd

    ├─avahi-daemon───avahi-daemon

    ├─bamfdaemon───2*[{bamfdaemon}]

    ├─bluetoothd

    ├─colord───2*[{colord}]

    ├─console-kit-dae─┬─63*[{console-kit-dae}]

    │                 └─{console-kit-dae}Dt\267

    ├─cron

    ├─cupsd───dbus

    ├─2*[dbus-daemon]

    ├─dbus-launch

    ├─dconf-service───2*[{dconf-service}]

    ├─deja-dup───2*[{deja-dup}]

    ├─firefox───22*[{firefox}]

    ├─gconfd-2

    ├─geoclue-master

    ├─6*[getty]

    ├─gnome-keyring-d─┬─4*[{gnome-keyring-d}]

    │                 └─{gnome-keyring-d}Dt\267

    ├─gnome-terminal─┬─bash───pstree

    │                ├─gnome-pty-helpe

    │                └─3*[{gnome-terminal}]

    ├─goa-daemon───{goa-daemon}

    ├─gvfs-afc-volume───{gvfs-afc-volume}Dt\267

    ├─gvfs-fuse-daemo───3*[{gvfs-fuse-daemo}]

    ├─gvfs-gdu-volume

    ├─gvfs-gphoto2-vo

    ├─gvfsd

    ├─gvfsd-burn

    ├─gvfsd-trash

    ├─hud-service───2*[{hud-service}]

    ├─indicator-appli───{indicator-appli}

    ├─indicator-datet─┬─{indicator-datet}

    │                 └─{indicator-datet}Dt\267

    ├─indicator-messa───{indicator-messa}Dt\267

    ├─indicator-print─┬─{indicator-print}

    │                 └─{indicator-print}Dt\267

    ├─indicator-sessi───2*[{indicator-sessi}]

    ├─indicator-sound─┬─{indicator-sound}

    │                 └─{indicator-sound}Dt\267

    ├─lightdm─┬─Xorg

    │         ├─lightdm─┬─gnome-session─┬─bluetooth-apple─┬─{bluetooth-apple}

    │         │         │               │                 └─{bluetooth-apple}Dt\267

    │         │         │               ├─deja-dup-monito───2*[{deja-dup-monito}]

    │         │         │               ├─gdu-notificatio───2*[{gdu-notificatio}]

    │         │         │               ├─gnome-fallback-───2*[{gnome-fallback-}]

    │         │         │               ├─gnome-screensav───2*[{gnome-screensav}]

    │         │         │               ├─gnome-settings-───2*[{gnome-settings-}]

    │         │         │               ├─metacity───3*[{metacity}]

    │         │         │               ├─nautilus───2*[{nautilus}]

    │         │         │               ├─nm-applet───2*[{nm-applet}]

    │         │         │               ├─polkit-gnome-au─┬─{polkit-gnome-au}

    │         │         │               │                 └─{polkit-gnome-au}Dt\267

    │         │         │               ├─ssh-agent

    │         │         │               ├─telepathy-indic───2*[{telepathy-indic}]

    │         │         │               ├─unity-2d-panel───2*[{unity-2d-panel}]

    │         │         │               ├─unity-2d-shell───5*[{unity-2d-shell}]

    │         │         │               ├─update-notifier─┬─{update-notifier}

    │         │         │               │                 └─{update-notifier}Dt\267

    │         │         │               └─3*[{gnome-session}]

    │         │         └─{lightdm}

    │         └─2*[{lightdm}]

    ├─mission-control─┬─{mission-control}

    │                 └─{mission-control}Dt\267

    ├─modem-manager

    ├─polkitd───{polkitd}

    ├─pulseaudio─┬─gconf-helper

    │            └─2*[{pulseaudio}]

    ├─rsyslogd───3*[{rsyslogd}]

    ├─rtkit-daemon───2*[{rtkit-daemon}]

    ├─system-service-

    ├─ubuntu-geoip-pr

    ├─udevd───2*[udevd]

    ├─udisks-daemon─┬─udisks-daemon

    │               └─{udisks-daemon}

    ├─unity-applicati─┬─{unity-applicati}

    │                 └─{unity-applicati}Dt\267

    ├─unity-files-dae─┬─{unity-files-dae}

    │                 └─{unity-files-dae}Dt\267

    ├─unity-lens-vide───{unity-lens-vide}

    ├─unity-music-dae───{unity-music-dae}Dt\267

    ├─unity-musicstor───{unity-musicstor}Dt\267

    ├─unity-panel-ser─┬─{unity-panel-ser}

    │                 └─{unity-panel-ser}Dt\267

    ├─unity-scope-vid─┬─{unity-scope-vid}

    │                 └─{unity-scope-vid}Dt\267

    ├─upowerd───2*[{upowerd}]

    ├─upstart-socket-

    ├─upstart-udev-br

    ├─whoopsie───{whoopsie}

    ├─zeitgeist-daemo───{zeitgeist-daemo}

    ├─zeitgeist-datah───{zeitgeist-datah}

    └─zeitgeist-fts─┬─cat

                    └─{zeitgeist-fts}

More information about the ‘pstree’ command can be fetched from its man page

• C – The CPU usage and scheduling information. The value is incremented with every tick of the system clock, however degraded by the scheduler by dividing it by two in every second. Therefore, A higher value indicates CPU intensive process.
• STIME – The start time of the process.
• TTY – The terminal type associated with the process. If this value is ‘?’, then it means the process is not associated with any terminal. These are daemon process, which we shall be discussing in the next section.
• TIME – The cumulative CPU time since the process is running.
• CMD – The command which launched the process.

We shall see more usage of ‘ps’ command in further sections, as we come to know about other dimensions of the linux processes.

Types of Processes

Although there is no standard classification of types of processes in Linux. The segregation could be in interactive and non-interactive processes, foreground and background processes or daemon or batch processes. It can also be classified based on the status of the processes such as as zombie processes. It is good enough if we comprehend all these various terminologies in the linux system.

Interactive processes

An interactive process is one which needs user’s interaction while it is active. For example, when we launch a vi-editor, it is an interactive process. Another example could be the telnet command. Hence, the interactive processes have to be associated to a terminal.
Under the umbrella of interactive processes, we have Foreground and Background processes.
Lets discuss them one by one :

Foreground Process

A process is a foreground process if it is in focus and can be given input from the standard input. It blocks the shell until the foreground process is complete. When we run our commands on the terminal, they generally run as foreground processes. They block the terminal until it is complete. Although most of our linux commands are quick enough for us to even realise that.
Let us create our own program which sleeps for 10 seconds and then ourselves experience what waiting for the foreground process feels like.
The C source looks like:

#include 

#include 

int main()

{

    int time = 10;

    sleep(time);

    printf("Slept for 10 secs\n");

    return 0;

}

Now compile and run the program,

$ gcc wait_process.c -Wall -o wait_process

$ ./wait_process

What do you experience? The terminal is blocked by the running process, and not letting the user to do anything until the program is complete.
To check another foreground blocking, open the geditor through the terminal. The terminal launches the geditor, won’t let you input anything to the terminal until we terminate the geditor.

Background Process

Background processes are ones, that are running, but in the background, not taking any user input from the terminal. It doesn’t block the terminal, and allows us to use the terminal irrespective of the background process is complete or not. They key-character to make any new process to be run in background is ‘&’.
How we use this character, is by suffixing it with the command, as in,

 &

It is time, to run our ‘wait_process’ program to run as a background, so that we can avoid the terminal to get blocked while the process is sleeping.

$ ./wait_process &

[1] 2534

$

Whoahh! we get the command line back, to be able to use it and not to worry about the sleeping process.
To get these background and foreground handy, linux provides certain commands to view what is running and also switch any foreground process to background and vice versa.
One can use command ‘jobs’ to see what all is running associated with the terminal.

$gedit wait_process.c&

[1] 2538

$./wait_process&

[2] 2546

$jobs

[1]-  Running                 gedit wait_process.c &

[2]+  Running                 ./wait_process &

$

In the above exercise, we started two background processes – gedit and the wait_process program. Hence, the command ‘jobs’ lists both of them along with their PID’s.
If we want to switch the ‘geditor’ as a foreground process, use linux command ‘fg’

$fg

We see following after running the ‘fg’ command

gedit wait_process.c

And we again lose the command prompt. No points for guessing, now gedit is running as a foreground process. However, to switch it back as a background process, use key combinations ‘Ctrl + Z’ to suspend the process and then run the linux command ‘bg’
Note: When we suspend the process, it is not running and hence on resuming it will start with the same status, when it was stopped.

^Z

[1]+  Stopped                 gedit wait_process.c

$ bg

[1]+ gedit wait_process.c &

$

We got the command prompt back. We need to confirm if the geditor process is still up and running. It can be done by again using the command ‘jobs’

$jobs

[1]+  Running                 gedit wait_process.c &

Batch processes

These are the processes which are queued in and executed one by one in FIFO (First In First Out). Batch processes are not associated with any terminal instead given to the system to run, preferably when the system load is low or at a specific time. Low system load is a relative term, and hence it depends on the system and the type and requirements of the batch processes.
There are two linux commands which are provided to create the batch processes:

Linux at command

Usage, taken from man page

at [-V] [-q queue] [-f file] [-mldbv] TIME

at [-V] [-q queue] [-f file] [-mldbv] -t time_arg

at -c job [job...]

atq [-V] [-q queue]

atrm [-V] job [job...]

The ‘at’ command is used to schedule a process at a specific date/time. The time of today’s day is taken in the format HH::MM. However, it also accepts phrases like ‘today’, ‘tomorrow’, ‘teatime’, etc.
When we execute the ‘at’ command, we reach the ‘at’ prompt, where we can queue all the tasks/commands/programs in an order. When we are done with the queueing of the task, press key combination ‘Ctrl +d’. On pressing the key combination ‘Ctrl + d’, we see ‘EOT’ displayed at the standard output, following which we are back to our command prompt.
As an example,

$ at tomorrow

warning: commands will be executed using /bin/sh

at> ./wait_process

at> 

job 1 at Sun Mar 10 20:41:00 2013

Linux batch command

Usage, taken from man page

batch

This command launches the process when the system load is low i.e. load average drops below 0.8, or the value set by atd
It’s usage example is similar to the command ‘at’ (explained above).

Daemon Processes

A daemon process in Linux is also one of its kind which runs in background. However, what is different here is, daemon processes are not associated to any terminal in any way. Therefore such processes don’t take interact with the user. A widespread example of daemon process is a server service of any kind. For example, if we consider a mail server, it just have to listen to the relevant ports and respond with its protocol routines on receiving packages. So, such kind of processes can be run as daemon processes, independent of terminal and user interaction.
Generally, when we code any program in C in Linux and execute it, the terminal becomes its parent process. Hence, in order to develop a daemon process service, the programmer needs to detach the process from its parent process. This is done by killing its parent process, which makes the process independent of the terminal, but controlled by its grandparent process i..e init process.
We shall learn more about coding a daemon process in the second part of the article.

Zombie Processes

When a process terminates, there is a proper exit and cleanup routine to be done by the developer of the program. If there is a bug, such that cleanup could not happen appropriately, though the process has been killed. Now, this process do occupy some memory, but will never be scheduled by the scheduler as the process status is ‘terminated’.
Such processes are called zombie processes, which are killed but still exist.
Zombie processes are generally harmless if there are not much. However, if we have a whole lot of zombie processes lingering, then it could be a pain. Since, the PID’s still taken up by the zombie processes are not available for re-allocation to new processes. Hence, soon the system would be out of available PID’s if the zombie processes keep on increasing, and no new process would be able be launch.

The init process

The init process is the one which initiates system processes taken from the script

/etc/inittab

and has been assigned PID = 1. So, these system processes include setting up the user space, mounting file systems, set up everything to get the system up and running. Worth mentioning, it is the init process which is at the apex of the complete process tree. It is run as root, and is the parent process of a user shell. It is the last sequence in the booting and is the one, which launches and controls the shutdown.

Process States

Linux processes generally go through six major states, which are listed below:
1. Running or Runnable ( R ) – A running state has a broader concept here. Running always does not mean utilising the CPU. Even while a process is ready to run, the state is running state.
Hence, there are two sub-states, when the process is queued in the ready queue to run and when the process is actually being executed, it is in the executing sub-state as has been scheduled by the scheduler.
2. Stopped (T) – If a running process receives a stop signal, it is moved to the stopped state. A process can also be in stopped state if it has been halted by a trace while debugging. .
3. Uninterruptible sleep (D) – It is a sleeping state, process has been blocked. Mostly, process goes into an uninterruptible sleep during an IO operation.
4. Interruptible sleep (S) – It is a sleeping state i.e. a blocking state where the process is waiting for an event to occur.
5. Zombie/Defunct state(Z) – It is the process state in which process has been terminated but not reaped by its parent process.
6. Dead (X) – A process never reaches this state, as as soon as it is dead, it is gone.
Note: For BSD formats and when the stat keyword is used, additional characters may be displayed:

• < high-priority (not nice to other users)
• N low-priority (nice to other users)
• L has pages locked into memory (for real-time and custom IO)
• s is a session leader
• l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
• + is in the foreground process group.

A simplified life cycle of a Linux process is illustrated through following diagram:













In order to check the current status of the active processes at a moment on the linux system, we again use the ‘ps’ command but with a different set of options.

$ps ax

In my system, I again get a huge list of processes. However, we are interested in the status details of the listed processes, therefore here is a snapshot of the output

 PID TTY      STAT   TIME COMMAND

1779 pts/0    Ss     0:00 bash

1797 ?        Sl     0:00 telepathy-indicator

1805 ?        Sl     0:00 /usr/lib/telepathy/mission-control-5

1809 ?        Sl     0:00 /usr/lib/gnome-online-accounts/goa-daemon

1851 ?        Sl     0:03 gnome-screensaver

1891 ?        Sl     0:04 update-notifier

1936 ?        S      0:00 /usr/bin/python /usr/lib/system-service/system-service-d

1940 ?        Sl     0:00 /usr/lib/deja-dup/deja-dup/deja-dup-monitor

2091 ?        SNl    0:00 deja-dup --prompt

2129 ?        Sl    21:54 /usr/lib/firefox/firefox

2145 ?        Sl     0:00 /usr/lib/i386-linux-gnu/at-spi2-core/at-spi-bus-launcher

2724 ?        S      0:00 /usr/lib/cups/notifier/dbus dbus://

2801 ?        SNl    0:22 /usr/bin/python /usr/bin/update-manager --no-focus-on-map

3157 ?        S      0:00 [kworker/0:1]

3168 ?        S      0:00 [kworker/0:2]

3207 ?        S      0:00 [kworker/0:0]

3208 pts/0    R+     0:00 ps ax

Observer the status of the list of processes under column ‘STAT’. Although, we have more confidence on Linux commands than anything, but it is exciting to confirm from the process ‘ps’ (at the end). Its status is ‘R+’ as we can read from the above output snapshot, which indicates it is running and running as a foreground process. So true!

Real time snapshot of processes

The ‘ps’ command that we just discussed in the previous section, evinces the active process list at a particular moment when the command is executed. In many cases, it is a need of the hour to view dynamic real time running of the processes. For such circumstances, linux comes with the ‘top’ command.
The ‘top’ usage looks like

top -hv | -bcHisS -d delay -n iterations -p pid [, pid ...]

More details can be found at its man page
To see how and what all it provides, here is an output snapshot from my ubuntu system
The command:

$top

Output:

top - 10:16:34 up 1 day, 17:36,  1 user,  load average: 1.51, 0.99, 0.84

Tasks: 136 total,   1 running, 135 sleeping,   0 stopped,   0 zombie

Cpu(s): 28.8%us,  3.3%sy,  0.0%ni, 67.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:    507536k total,   498516k used,     9020k free,    10488k buffers

Swap:   521212k total,   187996k used,   333216k free,    91572k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                      

2129 ubuntu    20   0  727m 258m  10m S 27.1 52.1 113:12.30 firefox                                      

 918 root      20   0  116m  35m 2128 S  3.3  7.1   7:33.90 Xorg                                         

1772 ubuntu    20   0 89352 5264 2940 S  0.7  1.0   0:08.13 gnome-terminal                               

1496 ubuntu    20   0  142m 3036 1920 S  0.3  0.6   1:20.55 metacity                                     

3782 root      20   0     0    0    0 S  0.3  0.0   0:00.57 kworker/0:0                                  

3800 ubuntu    20   0  2836 1152  880 R  0.3  0.2   0:00.05 top                                          

    1 root      20   0  3512 1160  604 S  0.0  0.2   0:00.66 init                                         

    2 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kthreadd                                     

    3 root      20   0     0    0    0 S  0.0  0.0   0:07.52 ksoftirqd/0                                  

    5 root      20   0     0    0    0 S  0.0  0.0   0:00.36 kworker/u:0                                  

    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                  

    7 root      RT   0     0    0    0 S  0.0  0.0   0:02.89 watchdog/0                                   

    8 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 cpuset                                       

    9 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 khelper                                      

  10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kdevtmpfs                                    

  11 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 netns                                        

  12 root      20   0     0    0    0 S  0.0  0.0   0:00.79 sync_supers                                  

  13 root      20   0     0    0    0 S  0.0  0.0   0:00.02 bdi-default                                  

  14 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kintegrityd                                  

  15 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kblockd                                      

  16 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 ata_sff                                      

  17 root      20   0     0    0    0 S  0.0  0.0   0:00.03 khubd                                        

  18 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 md                                           

  21 root      20   0     0    0    0 S  0.0  0.0   0:00.12 khungtaskd                                   

  22 root      20   0     0    0    0 S  0.0  0.0   0:08.45 kswapd0                                      

  23 root      25   5     0    0    0 S  0.0  0.0   0:00.00 ksmd                                         

  24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 fsnotify_mark                                

  25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ecryptfs-kthrea                              

  26 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 crypto                                       

  34 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kthrotld                                     

  35 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/u:2                                  

  36 root      20   0     0    0    0 S  0.0  0.0   0:00.08 scsi_eh_0                                    

  38 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_1                                    

  39 root      20   0     0    0    0 S  0.0  0.0   0:00.03 scsi_eh_2

Checking out running the linux ‘top’ command, we’ll observe that the output keeps on changing with every moment, as the process dynamics keep on changing at real time. This is possible, as the ‘top’ command is running all this while and monitoring the running tasks. Hence, we need to terminate the ‘top’ process to get back the command prompt.
Analysing and trying to understand the ‘top’ command output, let us go line by line.

top - 10:16:34 up 1 day, 17:36,  1 user,  load average: 1.51, 0.99, 0.84

Going in order from left to right, it indicates -

• current time
• time since the machine is on
• the number of users logged in
• average load on the system specifying three values for last one minute, last five minutes and last fifteen minutes respectively.

Coming to the next row of output, which looks like

Tasks: 136 total,   1 running, 135 sleeping,   0 stopped,   0 zombie

It helps us know how many total processes are there, which are 136 in our case, which includes one in the running state, 135 are in the sleeping state, zero in the stopped and zero zombie processes.
Now we know what all processes have been launched and are in which states.
Moving on,

Cpu(s): 28.8%us,  3.3%sy,  0.0%ni, 67.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

This row indicates the CPU usage in percentages. As in,

• 28.8%us – CPU percentage for user processes
• 3.3%sy – CPU percentage for system processes
• 0.0%ni – CPU percentage for processes with ‘nice’ priority
• 67.9%id – idle CPU percentage
• 0.0%wa – CPU percentage for processes waiting for I/O
• 0.0%hi – CPU percentage for processes serving hardware interrupts
• 0.0%si – CPU percentage for processes serving software interrupts
• 0.0%st – It is the steal time i.e. percentage CPU time stolen from a virtual machine i.e. time in an involuntary wait by a virtual machine, while the hypervisor is servicing another process.

The next row in the output specifies physical memory usage

Mem:    507536k total,   498516k used,     9020k free,    10488k buffers

The information is in terms of total physical memory available, where how much has been used, how much is free and how much is used for buffers.
On similar lines, the next output states usage of swap space

Swap:   521212k total,   187996k used,   333216k free,    91572k cached

Then following rows list all the details of all the launched processes at real time. What each detail means is:

• PID – The process ID
• USER – The user which owns the process
• PR – Process priority value
• NI – The nice value of a process.
• VIRT – The virtual memory used by the process
• RES – Physical memory used
• SHR – Shared memory of the process
• S – The status of the process where S – Sleeping : R – Running : Z- Zombie
• %CPU – Percentage CPU utilization of the process
• %MEM – Percentage memory usage of the process
• (TIME+)  – The total activity time of the process
• COMMAND – The command used to launch the process.

Terminating the processes

The linux system allows its users to terminate any process, of course with due considerations to access permissions. The linux user can terminate a process using command ‘kill’.
The usage syntax

kill [ -signal | -s signal ] pid ...

      kill [ -L | -V, --version ]

      kill -l  [ signal ]

More details found in its man page
The ‘kill’ linux command sends a signal to the specified process. Which process to send the signal is specified by its PID.
There are standard numbers assigned to each set of signals. We can get the information of what number is corresponding to which signal using the same ‘kill’ command through ‘-l’ option.

$ kill -l

1) SIGHUP     2) SIGINT     3) SIGQUIT     4) SIGILL     5) SIGTRAP

6) SIGABRT     7) SIGBUS     8) SIGFPE     9) SIGKILL    10) SIGUSR1

11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM

16) SIGSTKFLT    17) SIGCHLD    18) SIGCONT    19) SIGSTOP    20) SIGTSTP

21) SIGTTIN    22) SIGTTOU    23) SIGURG    24) SIGXCPU    25) SIGXFSZ

26) SIGVTALRM    27) SIGPROF    28) SIGWINCH    29) SIGIO    30) SIGPWR

31) SIGSYS    34) SIGRTMIN    35) SIGRTMIN+1    36) SIGRTMIN+2    37) SIGRTMIN+3

38) SIGRTMIN+4    39) SIGRTMIN+5    40) SIGRTMIN+6    41) SIGRTMIN+7    42) SIGRTMIN+8

43) SIGRTMIN+9    44) SIGRTMIN+10    45) SIGRTMIN+11    46) SIGRTMIN+12    47) SIGRTMIN+13

48) SIGRTMIN+14    49) SIGRTMIN+15    50) SIGRTMAX-14    51) SIGRTMAX-13    52) SIGRTMAX-12

53) SIGRTMAX-11    54) SIGRTMAX-10    55) SIGRTMAX-9    56) SIGRTMAX-8    57) SIGRTMAX-7

58) SIGRTMAX-6    59) SIGRTMAX-5    60) SIGRTMAX-4    61) SIGRTMAX-3    62) SIGRTMAX-2

63) SIGRTMAX-1    64) SIGRTMAX

The most commonly used signal to terminate processes is

9) SIGKILL

Now, as an example experience, we shall be creating a process and terminate it.
We’ll use our old program source, and its executable.

$ ./wait_process &

[1] 3315

$ ps

 PID TTY          TIME CMD

1779 pts/0    00:00:00 bash

3315 pts/0    00:00:00 wait_process

3316 pts/0    00:00:00 ps

$ kill -9 3315

[1]+  Killed                  ./wait_process

$ ps

 PID TTY          TIME CMD

1779 pts/0    00:00:00 bash

3317 pts/0    00:00:00 ps

In the above practical exercise, we ran our waiting process, and killed it before it got complete.

References

http://www.advancedlinuxprogramming.com
http://careers.directi.com/display/tu/Understanding+Processes+in+Linux