Mastering The Linux Shell : Standard In, Out, and Error

http://marcelgagne.com/content/mastering-linux-shell-standard-out-and-error

Welcome to part four in my Mastering The Linux Shell series where we will wander wistfully into a land where three terribly under-appreciated but vitally important files live. You will remember that everything is a file, including directories which some people call folders. What you may not know is that your keyboard is a file, as is the screen on which you read these words. And if all hell breaks loose, all that goes into a file too. Those special files, and variations on those files, are always in use, unlike those other, less hard-working files that seem to get all the credit. They are called STDIN, STDOUT, and STDERR.
Standard in (STDIN) is where the system expects to find its input. This is usually the keyboard, although it can be a program or shell script. When you change that default, you call it redirecting from STDIN.
Standard out (STDOUT) is where the system expects to direct its output, usually the terminal screen. Again, redirection of STDOUT is at the discretion of whatever command or script is executing at the time. The chain of events from STDIN to STDOUT looks something like this:

standard in  -> Linux command  ->  standard out

STDIN is often referred to as fd0, or file descriptor 0, while STDOUT is usually thought of as fd1. There is also standard error (STDERR), where the system reports any errors in program execution. By default, this is also the terminal. To redirect STDOUT, use the greater-than sign (>). As you might have guessed, to redirect from STDIN, you use the less-than sign (<). But what exactly does that mean? Let’s try an experiment. Randomly search your brain and pick a handful of names. Got them? Good. Now type the cat command and redirect its STDOUT to a file called random_names.

cat > random_names

Your cursor will just sit there and wait for you to do something, so type those names, pressing Enter after each one. What’s happening here is that cat is taking its input from STDIN and writing it out to your new file. You can also write the command like this:

cat - 1> random_names

The hyphen literally means standard in to the command. The 1 stands for file descriptor 1. This is good information, and you will use it later. Finished with your random names list? When you are done, press to finish. , by the way, stands for EOF, or end of file.

Marie Curie
Albert Einstein
Mark Twain
Wolfgang Amadeus Mozart
Stephen Hawking
Hedy Lamarr
^D

If you cat this file, the names will be written to STDOUT, in this case, your terminal window. You can also give cat several files at the same time. For instance, you could do something like this:

cat file1 file2 file3

Each file would be listed one right after the other. That output could then be redirected into another file. You could also have it print out the same file over and over (cat random_names random_names random_names). cat isn’t fussy about these things and will deal with binary files (programs) just as quickly. Beware of using cat to print out the contents of a program to your terminal screen. At worst, your terminal session will lock up or reward you with a lot of beeping and weird characters.

If you get caught in such a situation and all the characters on your screen appear as junk, try typing echo and then pressing and . If you can still type, you can also try typing stty sane and then pressing . Some systems also provide a command called reset which will return your terminal session to some kind of sane look.

Redirecting STDIN works pretty much the same way, except that you use the less-than sign instead. Using the sort command, let’s take that file of random names and work with it. Many commands that work with files can take their input directly from that file. Unless told otherwise, cat and sort will think that the word following the command is a filename. That’s why you did the STDIN redirection thing. Yes, that’s right: STDIN is just another file. Sort of.

sort random_names

The result, of course, is that you get all your names printed out in alphabetical order. You could have also specified that sort take its input from a redirected STDIN. It looks a bit strange, but this is perfectly valid.

$ sort < random_names
Albert Einstein
Hedy Lamarr
Marie Curie
Mark Twain
Stephen Hawking
Wolfgang Amadeus Mozart

One more variation involves defining your STDIN (as you did previously) and specifying a different STDOUT all on the same line. In the following example, I am redirecting from my file and redirecting that output to a new file:

sort < random_names > sorted_names

Pipes and Piping

Sometimes the thing that makes the most sense is to feed the output from one command directly into another command without having to resort to files in between at every step of the way. This is called piping. The symbolism is not that subtle: Imagine pieces of pipe connecting one command with another. Not until you run out of pipe does the command’s output emerge. The pipe symbol is the broken vertical bar on your keyboard usually located just below or (depending on the keyboard) just above the Enter key and sharing space with the backslash key. Here’s how it works:

cat random_names | sort | wc –w > num_names

In the preceding example, the output from the cat command is piped into sort, whose output is then piped into the wc command (that’s "word count"). The -w flag tells wc to count the number of words in random_names. So far, so good.
That cat at the beginning is actually redundant, but I wanted to stack up a few commands for you to give you an idea of the power of piping. Ordinarily, I would write that command as follows:

sort random_names | wc –w > num_names

The cat is extraneous because sort incorporates its function. Using pipes is a great timesaver because you don’t always need to have output at every step of the way.

tee: A Very Special Pipe

Suppose that you want to send the output of a command to another command, but you also want to see the results at some point. Using the previous word count example, if you want a sorted list of names, but you also want the word count, you might have to use two different commands: one to generate the sorted list and another to count the number of words. Wouldn’t it be nice if you could direct part of the output one way and have the rest continue in another direction? For this, use the tee command.

sort random_names | tee sorted_list | wc –w > num_names

The output from sort is now sitting in a file called sorted_list, while the rest of the output continues on to wc for a word count.

STDERR

What about STDERR? Some commands (many, in fact) treat the error output differently than the STDOUT. If you are running the command at your terminal and that’s all you want, there is no problem. Sometimes, though, the output is quite wordy and you need to capture it and look at it later. Unfortunately, using the STDOUT redirect (the greater-than sign) is only going to be so useful. Error messages that might be generated (such as warning messages from a compilation) will go to the terminal as before. One way to deal with this is to start by redirecting STDERR to STDOUT, and then redirect that to a file. Here’s the line I use for this:

command_name 2>&1 > logfile.out

Remember that file descriptor 2 is STDERR and that file descriptor 1 is STDOUT. That’s what that 2>&1 construct is all about. You are redirecting fd2 to fd1 and then redirecting that output to the file of your choice. Using that program compilation example, you might wind up with something like this:

make –f Makefile.linux 2>&1 > compilation.output

The final greater-than sign in the preceding example could be eliminated completely. When using the 2>&1 construct, it is assumed that what follows is a filename.

The Road to Nowhere

If the command happens to be verbose by nature and doesn’t have a quiet switch (that's usually a -q option), you can redirect that STDOUT and STDERR noise to what longtime  Linux users like to call the bit bucket, a special file called /dev/null, literally, a road to nowhere. Anything fed to the bit bucket takes up no space and is never seen or heard from again. When I was in school, way back in the before time, we would tell people to shut up by saying, "Dev null it, will you?" As you can see, we were easily amused.
To redirect output to the bit bucket, use the STDOUT redirection.

command –option > /dev/null

If, for some strange reason, you want to sort the output of the random_names files and you do not want to see the output, you can redirect the whole thing to /dev/null in this way:

sort random_names > /dev/null

Using the program compilation example where you had separate STDOUT and STDERR streams, you can combine the output to the bit bucket.

make –F makefile.linux 2>&1 /dev/null

That’s actually a crazy example because you do want to see what goes on, but redirecting both STDOUT and STDERR to /dev/null is quite common when dealing with automated processes running in the background.
And once again, it's time to wrap it up. Look for part 5 of this series coming your way on Monday. And that is where I will leave this discussion for today. Leave any comments you may have here on Google Plus or over here on Facebook. Join me this coming Thursday for part four in this series. Remember that you can also follow the action on CookingWithLinux.com where it's all Linux, all the time. Except for the occasional wine review.