http://linux-server-security.com/linux_servers_howtos/linux_process_table.html
nodev rootfs
nodev bdev
nodev proc
nodev cgroup
nodev cpuset
nodev tmpfs
nodev devtmpfs
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev inotifyfs
nodev devpts
nodev ramfs
nodev hugetlbfs
nodev pstore
nodev mqueue
If you've enjoyed reading the technical content on this site then please have a look at my Linux books which were both published in 2016 and some of my articles in Linux Magazine and Admin Magazine are available on their respective websites.
A fundamental design feature of
Unix-like Operating Systems is that many of a system’s resources
are accessible via the filesystem, as a file. For example the
“procfs” pseudo-filesystem offers us access to all kinds of
valuable treasures.
By querying files present in
“/proc” you can quickly find out the intricacies of your network
connections, the server’s CPU vendor and look up a mountain of
useful information such as the command line parameters passed to
your latest-and-greatest application when it fired up. This is
thanks to the fact that many of the functions of a server, such
as a network connection are really just another stream of data
and in most cases can be presented as a file on your
filesystem.
Let’s jump ahead of ourselves
for a moment in case you’re not too familiar with the “/proc”
filesystem. If you knew that your Process ID was “16651” for
example then you could run this command in order to find out what
was sent to the Puppet process to start it up:
# xargs -0 < /proc/16551/cmdline
The output from that command
is:
/usr/bin/ruby /usr/bin/puppet
agent
As you can see Puppet’s agent
is using the Ruby programming language in this case and the
binary “/usr/bin/puppet” is passed the parameter “agent” to run
it as an “agent” and not a “master”.
The “Everything Is A File”
philosophy makes total sense if you think about it. The power
harnessed within Unix’s standard tools (usually used for
manipulating data held in the more common text files) such as
“grep”, “awk” and “sed” are a major strength of the Operating
System. But most importantly you can have a system’s components
integrate very efficiently and easily if many things are simply
files.
If you have you ever tried to
look into a process running on a Unix-like machine then you’ll
know that if anything the abundance of information adds
confusion, rather than assists, if you don’t know where to look.
There are all sorts of things to consider when you are eagerly
trying to track down a rogue process on production
machine.
In this chapter I will attempt
to offer a broad insight into how the Process Table can be
accessed by the ps command and in combination with “/proc” and
“/dev” how it can help you manipulate your systems.
Legacy
There are a few legacy
stumbling blocks when it comes to looking up a process on
different types of Unix boxes but thankfully we can rely on the
trusty "ps" command to mitigate some of these headaches
automatically for us.
For example, Unix used the "ps"
command by grouping its parameters together and prepending a
hyphen. Whereas BSD enjoyed grouping switches together but, for
one reason or another, fell out with the hyphen
entirely.
Throwing another spanner in the
works however was good, old GNU's preference; where its long
options used two dashes. Now that you've fully committed those
confusing differences to memory let's assume that the ps command
does as much as it can for us by mixing and matching the
aforementioned options in an attempt to keep everyone
happy.
Be warned that occasionally
sometimes oddities can occur so keep an eye out for them just in
case. I’ll try and offer alternative commands as we continue to
act as a reminder that not all is to be taken exactly as read.
For example a very common use of the ps command is:
# ps -aux
It should be noted however that
this is indeed different to:
# ps aux
You might suspect, and would be
forgiven for thinking as much, that this is purely to keep
everyone on their toes. However according to the ps command’s
manual this is apparently because POSIX and UNIX insist that they
should cater for processes owned by a user called "x". However,
if I’m reading the information correctly, then should the user
"x" not exist then "ps aux" is run. I love the last sentence of
the manual’s definition and draw your attention to it as a gentle
warning: “It is fragile, subject to change, and thus should not
be relied upon.”
Process Tree
Enough eye-strain for a moment
and let’s begin by looking at the ps command and what it is
capable of helping with in relation to querying the Process
Table.
For starters (and I won’t be
betting the ranch on this statement) it's relatively safe to
assume that upper and lower case mean the same thing.
If you’ve never seen the output
of a Process Tree then it might help with understanding “child”
threads which live under a “parent” process. In this case the
command is simply:
# pstree
The
not-so-impossible-to-decipher output from that command is shown
in Listing One (on a server not running
“systemd” but good, old “init” (which is always Process ID
(PID) number one, as an aside):
init-+-auditd---{auditd}
|-certmonger
|-crond
|-dbus-daemon---{dbus-daemon}
|-hald---hald-runner-+-hald-addon-acpi
|
`-hald-addon-inpu
|-apache2---8*[apache2]
|-master-+-pickup
|
`-qmgr
|-6*[mingetty]
|-oddjobd
|-rpcbind
|-rsyslogd---3*[{rsyslogd}]
|-sshd---sshd---sshd---bash---pstree
|-udevd---2*[udevd]
Listing One: Output from the
“pstree” command showing parent processes and their
children
You can make your screen output
much messier by adding the “-a” switch. Doing so will add command
line arguments (pulled from the /proc filesystem in the same way
that our example did earlier). This is very useful but you might
want to do something like “grep” a specific process name from the
output, as follows:
# pstree -ap | grep
ssh
|-sshd,29076
|
`-sshd,32365
|
`-sshd,32370
|
|-grep,8401 ssh
| |-sssd_ssh,1143
--debug-to-files
Listing Two: The command
“pstree” showing only SSH processes with command line arguments
and PIDs
As we can see from Listing Two,
the command which we are querying with is also shown (starting
with “grep”) in the output so try not to let that trip you up on
your travels. I’ve added the “-p” switch to display the PIDs too
for ease.
One final look at that example
is seen in Listing Three. Here the all-pervasive “-Z” switch
offers us any SELinux config associated with the parent and child
detail displayed in our process table tree. That command for
reference was:
# pstree -aZp | grep
ssh
|-sshd,29076,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'
|
`-sshd,32365,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'
|
`-sshd,32370,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023'
|
|-grep,8406,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023'
ssh
|
|-sssd_ssh,1143,`system_u:system_r:sssd_t:s0'
--debug-to-files
Listing Three: The output
now includes SELinux detail, command line arguments and
PIDs
Alternative View
Back to school for a moment and
we’ll pick up some of the basics that will help us increase our
knowledge later. We will (very loosely) mimic the Process Tree
output we saw earlier with just the ps command now. You might
stumble across “forest” puns in manuals with regards to “trees”
in case it’s confusing.
The “-e” switch is also equal
to “-A” and it will dutifully display ALL of the processes. We’re
going to combine that with “-j” which should give us a “jobs
format”. I’m sorry to potentially give you more headaches and I
encourage to try a few of these of alternative options but one
small caveat would be that running “j” and “-j” gives different
levels of information. The non-hyphenated version giving more
output in this case.
We will also add the “-H” to
display the hierarchical output and we can achieve this by
running:
# ps -ejH
Or, in a parallel universe,
alternatively you can also use for vaguely similar
results:
# ps axjf
Try it yourself. It’s far from
identical to the “pstree” command but there are similarities in
the output due to the hierarchies I’m sure that you will
agree.
Day-To-Day
It is easy to get overwhelmed
by the level of detail that the ps command can provide. My
favourite process command is:
# ps -ef
With “-e” this gives us a
display of all the processes in the Process Table and the newly
added “-f” switch gives us what’s known as full-format listing.
Apparently we can add “-L” to this concoction in order to offer
us process thread information. I find this command very easy to
remember around Christmas time.
# ps -eLf
Et voilà, as requested two new
columns NLWP (number of threads) and LWP (thread ID) are now
added to the now busying display.
If you only wanted to print the
name of a process (which might be ideal for scripting) then you
can sort the wheat from the chaff by using this command for
process number "37":
# ps -p 37 -o comm=
An option that I've used in the
past a few times is checking what command my terminal is
currently
responsible for. This is useful
if you're stressing your server or worrying your workstation with
lots of extra load. The simple (non-hyphenated) "T" switch lets
us view this.
# ps T
You can test this by running
something unusual, like a “for” loop with a pause in it
using“sleep”, or anything odd that stands out such as this
command.
# echo {1..999999} &
This simply runs a counter
process in the background and when we run “ps T” we can see
processes associated with the terminal in question.
Built-In Features
Let’s look at a few other
built-in basics of the pliable ps command.
You can reverse the output of a
command with a non-hyphenated “N” switch, which stands for
negate. Or more precisely from the manual “Select all processes
except those that fulfill the specified conditions. (negates the
selection) Identical to --deselect.” All is revealed in Listing
Four. As we can see, there isn’t any mention of “ntp” except from
our ps command.
# ps N ntp
PID TTY STAT
TIME COMMAND
1414 tty1
Ss+ 0:00 /sbin/mingetty /dev/tty1
1416 tty2
Ss+ 0:00 /sbin/mingetty /dev/tty2
1418 tty3
Ss+ 0:00 /sbin/mingetty /dev/tty3
1420 tty4
Ss+ 0:00 /sbin/mingetty /dev/tty4
1426 tty5
Ss+ 0:00 /sbin/mingetty /dev/tty5
1430 tty6
Ss+ 0:00 /sbin/mingetty /dev/tty6
9896 pts/1 S
0:00 sudo -i
9899 pts/1 S
0:00 -bash
10040 pts/1 R+
0:00 ps N ntp
Listing Four: Demonstrates
running processes without including “ntp” (note our “ps” command
again being seen)
Imagine that you again wanted
to see what SSH activity was going on and also the Hardware
Abstraction Layer daemon, “hald”. Hardly related I agree but you
can never account for strange scenarios when it comes to
computers.
The command which follows is a
way of searching for a list of processes with a certain name.
Separated without a space and just a comma in this
case.
# ps -C sshd,hald
If you need to check any
processes run by a particular system group you can achieve this
with the
following command:
# ps -G ntp
Compatability
Although not always the case
the modern ps command cleverly mitigates our migraine-inducing
compatibility headaches by letting us run a simple command in a
number of ways where possible.
If you wanted to select a list
of processes which belonged to the superuser, "root", then you
can achieve this with the following three commands (which
admittedly display ever-so-slightly different
outputs):
# ps -u root
# ps U root
# ps --user root
The above commands dutifully
offer us what's known as the "EUID" or “effective ID” of a user
but
not the "real user ID". In
reality, no pun intended, every process actually has two user
IDs; just to keep things simple. This also applies to groups too
but let's not worry about that.
Apparently the kernel is most
concerned with the "effective user ID" for activities such as
writing to a file and if a user is allowed to complete a request
to do something that requires a privilege.
And, although that’s required
for much of the time, there's an important scenario within which
the "real user ID" needs to be paid attention to. If someone or
something wants to alter the owner of "effective user ID" on an
already running process then the kernel needs to check up on both
the "real user ID" and the "effective user ID".
Changing the ownership of a
process is particularly useful if a new user wants to do
essentially the same thing (like write to a file) as the existing
owner does. Rather than duplicating the process (adding extra
strain to the system and potentially introducing more security
aspects to consider) we can avoid duplicating the process and
simply reassign it.
What about after a user is
finished with their short task? The answer is that we only
temporarily give access and then swap it back again to its
original owner. If you want to reveal the somewhat elusive "real
user IDs" then you can do so with system groups like this (which
is the same as "-G"):
# ps --Group ntp
And, not surprisingly, we can
do exactly the same thing for users as follows (we do this with
"-U"):
# ps --User chrisbinnie
If you want to query a very
specific Process ID (because you've spotted it in "top" or a
script
has complained about it) then
to all intents and purposes these commands all the do the same. I
have included some output for comparison as it’s short and easy
to read in this case:
# ps -p 1248
PID TTY TIME
CMD
1248 ?
00:00:08 ntpd
# ps p 1248
PID TTY STAT TIME
COMMAND
1248 ?
Ss 0:08 ntpd -u ntp:ntp -p
/var/run/ntpd.pid -g
# ps --pid 1248
PID TTY TIME
CMD
1248 ?
00:00:08 ntpd
If you’ve ever wondered about
the efficiencies of a system, here’s something interesting. The
kernel has to be able to tidy up when a user logs out (otherwise
there would be a plethora of useless processes clogging up the
pipes) so Unix-like systems dutifully groups processes into
“sessions”. You can test for session IDs by using "--sid" or as
below with “-s”:
# ps -s 1526
If it’s of interest a session
can have an associated terminal (of "tty" or "Teletype"
varieties) controlling it however only one process can be at
running in the foreground. All these components are given numbers
to keep the system nice and orderly. A a result we have thread
IDs, process IDs, process group IDs and session IDs. Here was you
thinking that the ps command didn't have much to do.
If you’re interested in
reading a little more about sessions this book’s excerpt is
intriguing with sentences such as “Consequently, sessions are
largely the business of shells. In fact, nothing else really
cares about them.”:
Children
Now picture the scene where we
want to check for the parents of a process (we'll look at this in
more detail in a second). You can achieve this by using this
command:
# ps --ppid 21201
This shows us the processes
with a parent process of that ID. In other words we can pinpoint
processes that are children of process "21201"in this
case.
Having said earlier that
usually case-sensitivity shouldn't cause too many headaches I'm
going to completely contradict myself with a few examples of why
that statement isn’t always true.
Try running my favourite ps
command again, its abbreviated output is shown
underneath:
# ps -ef
UID
PID PPID C STIME TTY TIME
CMD
apache 23026 22856
0 Feb26 ? 00:00:00
/usr/sbin/apache2
Now try running the full fat
version by using an uppercase "F":
# ps -eF
UID
PID PPID C SZ RSS PSR STIME
TTY TIME CMD
apache 23026 22856
0 44482 3116 0 Feb26 ?
00:00:00 /usr/sbin/apache2
The differences are that the
latter includes SZ, RSS and PSR fields. The first two are memory
related whereas PSR shows which CPU the process is using. For
more information there’s lots more in the manual:
# man ps
Onwards we can look at another
alternative to the “-Z” option which we briefly touched on
before:
# ps -efM
unconfined_u:system_r:apache2_t:s0 apache 23031
22856 0 Feb26 ? 00:00:00
/usr/sbin/apache2
A useful BSD throwback I quite
like the look of is possibly one of the shortest commands known
to
mankind. Have a look at Listing
Five.
# ps l
F UID PID
PPID PRI NI VSZ RSS WCHAN
STAT TTY TIME COMMAND
4 0 1414
1 20 0 4064 584 n_tty_
Ss+ tty1 0:00 /sbin/mingetty
/dev/tty1
4 0 1416
1 20 0 4064 588 n_tty_
Ss+ tty2 0:00 /sbin/mingetty
/dev/tty2
4 0 1418
1 20 0 4064 584 n_tty_
Ss+ tty3 0:00 /sbin/mingetty
/dev/tty3
4 0 1420
1 20 0 4064 580 n_tty_
Ss+ tty4 0:00 /sbin/mingetty
/dev/tty4
4 0 1426
1 20 0 4064 584 n_tty_
Ss+ tty5 0:00 /sbin/mingetty
/dev/tty5
4 0 1430
1 20 0 4064 588 n_tty_
Ss+ tty6 0:00 /sbin/mingetty
/dev/tty6
4 0 9896
9558 20 0 191392 2740 poll_s S
pts/1 0:00 sudo -i
4 0 9899
9896 20 0 110496 1960 wait S
pts/1 0:00 -bash
4 0 10776
9899 20 0 108104 980 -
R+ pts/1 0:00 ps l
Listing Five:Shows us the
“long formatted” output which can be embellished with other
options, harking from BSD origins
Clarity
Sometimes even the mighty ps
command struggles to precisely refine its output. Imagine a
scenario where Java processes are filling up the process table
and all you want to do is find their parent so that you can stop
(or "kill") the process abruptly. To summarise your information
you can use the non-hyphenated “S” switch:
# ps S
This helps to find a parent
when its child processes only live for a short period of
time.
What about when your Process
Table is brimming with processes and you need to list a number of
process PIDs at once? As you’d expect there are different ways to
achieve this as Listing Six shows when we run the following
command:
# ps -p "1 2" -p 3,4
PID TTY TIME
CMD
1 ?
00:00:03 init
2 ?
00:00:01 kthreadd
3 ?
00:00:01 migration/0
4 ?
00:00:20 ksoftirqd/0
Listing Six: We can pick and
choose the PIDs that we view in a number of ways
Everything Is A File
As we mentioned earlier the
well-considered principle of basing as much as possible on
Unix-like systems around files is a well-advised approach. It
could be said that this also extends to the Process Table;
consider the treasure trove of gems to be found if you delve
deeply into the “procfs” pseudo-filesystem, located in root level
“/proc” on your filesystem.
Elements of the innards of
/proc can only be read from and not written to. The key file here
is
“/etc/sysctl.conf” where you
can also change many tunable kernel settings so that they persist
following a reboot. One not-so-trivial caveat is that, almost
magically, any freshly entered parameters into “/proc” are
usually set live instantly so be careful!
Clearly this has a number of
advantages. There's no messing about with stopping and starting
daemons but be warned that if you're in the slightest bit unsure
of making a change (especially to servers) then take a deep
breath before doing so. Rest assured that a reboot will revert
any changes that you make if they are not entered into the file
“/etc/sysctl.conf”.
There are zillions of hidden
corridors and secret rooms to explore inside /proc and sadly we
will only be able to look at a tiny percentage of them. Needless
to say that on a test Virtual Machine or development machine you
should spend a long time tweaking, fiddling and breaking your
current kernel’s procfs settings. If you’re like me then you
might even find it vaguely cathartic and it’s safe to say that
the immediacy of the changes really appeal to the
impatient.
You can look further into a
particular process which you've found using the excellent ps
command as we've already seen. The path of Process ID “23022” for
example is simply “/proc/23022” in relation to /proc.
If we enter that directory then
we are shown (after some complaints that we don't have access to
parts of the directory structure if we’re not logged in as root)
the contents is presented as follows in Listing Seven:
dr-xr-xr-x. 8 apache
apache 0 Feb 26 03:15 .
dr-xr-xr-x. 144 root
root 0 Feb 11 13:31 ..
dr-xr-xr-x. 2 apache
apache 0 Feb 26 04:03 attr
-rw-r--r--. 1 root
root 0 Feb 28 08:25 autogroup
-r--------. 1 root
root 0 Feb 28 08:25 auxv
-r--r--r--. 1 root
root 0 Feb 28 08:25 cgroup
--w-------. 1 root
root 0 Feb 28 08:25 clear_refs
-r--r--r--. 1 root
root 0 Feb 26 04:03 cmdline
-rw-r--r--. 1 root
root 0 Feb 28 08:25 comm
-rw-r--r--. 1 root
root 0 Feb 28 08:25 coredump_filter
-r--r--r--. 1 root
root 0 Feb 28 08:25 cpuset
lrwxrwxrwx. 1 root
root 0 Feb 28 08:25 cwd -> /
-r--------. 1 root
root 0 Feb 27 14:01 environ
lrwxrwxrwx. 1 root
root 0 Feb 28 08:25 exe ->
/usr/sbin/apache2
dr-x------. 2 root
root 0 Feb 26 04:03 fd
dr-x------. 2 root
root 0 Feb 28 08:25 fdinfo
-r--------. 1 root
root 0 Feb 28 08:25 io
-rw-------. 1 root
root 0 Feb 28 08:25 limits
-rw-r--r--. 1 root
root 0 Feb 28 08:25 loginuid
-r--r--r--. 1 root
root 0 Feb 28 08:25 maps
-rw-------. 1 root
root 0 Feb 28 08:25 mem
-r--r--r--. 1 root
root 0 Feb 28 08:25 mountinfo
-r--r--r--. 1 root
root 0 Feb 28 08:25 mounts
-r--------. 1 root
root 0 Feb 28 08:25 mountstats
dr-xr-xr-x. 4 apache
apache 0 Feb 28 08:25 net
dr-x--x--x. 2 root
root 0 Feb 28 08:25 ns
-r--r--r--. 1 root
root 0 Feb 28 08:25 numa_maps
-rw-r--r--. 1 root
root 0 Feb 28 08:25 oom_adj
-r--r--r--. 1 root
root 0 Feb 28 08:25 oom_score
-rw-r--r--. 1 root
root 0 Feb 28 08:25 oom_score_adj
-r--r--r--. 1 root
root 0 Feb 28 08:25 pagemap
-r--r--r--. 1 root
root 0 Feb 28 08:25 personality
lrwxrwxrwx. 1 root
root 0 Feb 28 08:25 root -> /
-rw-r--r--. 1 root
root 0 Feb 28 08:25 sched
-r--r--r--. 1 root
root 0 Feb 28 08:25 schedstat
-r--r--r--. 1 root
root 0 Feb 28 08:25 sessionid
-r--r--r--. 1 root
root 0 Feb 28 07:52 smaps
-r--r--r--. 1 root
root 0 Feb 28 08:25 stack
-r--r--r--. 1 root
root 0 Feb 26 03:15 stat
-r--r--r--. 1 root
root 0 Feb 26 03:15 statm
-r--r--r--. 1 root
root 0 Feb 26 04:03 status
-r--r--r--. 1 root
root 0 Feb 28 08:25 syscall
dr-xr-xr-x. 3 apache
apache 0 Feb 27 11:41 task
-r--r--r--. 1 root
root 0 Feb 28 08:25 wchan
Listing Seven: Inside
“/proc/23022” we can see a number of pseudo files and directories
for our Web server
You might want to think of this
content as belonging to runtime system information. It has been
said that /proc is a centralised config system for the kernel and
it’s easy to see that directory is full of a mountain of
information for just one process. As suggested rummaging through
these directories and looking up which file does what when you
have a spare moment might be described as therapeutic. It’s well
worth the effort.
Pseudo Filesystems
It’s hard to dismiss the power
which /proc wields. Be wary though that there’s a lot going on
inside your server when it is running, even if no-one is hitting
your website. As a result wouldn’t it be sensible to separate the
tricksy hardware settings aside from the kernel settings and
Process Table?
Continuing onwards with our
“Everything Is A File” mantra that’s exactly what Unix-type
Operating Systems do. Step forward “/dev”.
When dealing with physical
devices, whether connected to the machine or not, we turn to
“/dev” and not “/proc”.
Here’s an abbreviated directory
listing of “/dev” as shown in Listing Eight.
drwxr-xr-x. 2 root root
740 Feb 11 13:31 block
drwxr-xr-x. 2 root root
80 Feb 11 13:31 bsg
lrwxrwxrwx. 1 root root
3 Feb 11 13:31 cdrom ->
sr0
lrwxrwxrwx. 1 root root
3 Feb 11 13:31 cdrw ->
sr0
drwxr-xr-x. 2 root root
2.5K Feb 11 13:31
char
crw-------. 1 root root
5,1 Feb 11 13:31
console
lrwxrwxrwx. 1 root root
11 Feb 11 13:31 core ->
/proc/kcore
drwxr-xr-x. 4 root root
80 Feb 11 13:31 cpu
crw-rw----. 1 root root
10, 61 Feb 11 13:31
cpu_dma_latency
crw-rw----. 1 root root
10, 62 Feb 11 13:31
crash
drwxr-xr-x. 5 root root
100 Feb 11 13:31 disk
Listing Eight: We can see an
abbreviated list of some of the devices which “/dev” deals
with
What about another example of
what “/dev” can do for us? Lets take an example of the superb
“lsof” utility. If you're not familiar with “lsof” then it's
unquestionably worth a look at, I’m a big fan. The abbreviation
“lsof” stands for “list open files” and its seemingly endless
functionality is exceptionally useful.
Listing Nine shows us output
from “lsof” when looking up information relating to the directory
“/var/log“. We can display this information having run this
command:
# lsof +D /var/log/
COMMAND PID USER
FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd
1103 root 1w REG 253,4
2743 19 /var/log/messages
rsyslogd
1103 root 2w REG 253,4
1906 17 /var/log/cron
rsyslogd
1103 root 4w REG 253,4
747 18 /var/log/maillog
rsyslogd
1103 root 5w REG 253,4
1753 27 /var/log/secure
apache2
22856 root 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
22856 root 6w REG 253,4
0 131104
/var/log/apache2/access_log
apache2
23022 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23022 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23024 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23024 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23026 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23026 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23027 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23027 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23028 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23028 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23029 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23029 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23030 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23030 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
apache2
23031 apache 2w REG 253,4
245 131095 /var/log/apache2/error_log
apache2
23031 apache 6w REG 253,4
0 131104 /var/log/apache2/access_log
Listing Nine: The output
from the mighty “lsof” looks much like that from the ps
command
I am using this “lsof” example
because it highlights how a system weaves in and out referencing
data from “/proc” and “/dev”. I won’t pretend to understand the
nuances.
From its manual the versatile
“lsof” transparently informs us of how it gathered such
information about that directory, by telling us which files it
references:
“/dev/kmem” - the kernel
virtual memory device
“/dev/mem” - the physical
memory device
“/dev/swap” - the system paging
device
From what I can gather these
files change between varying Unix versions but hopefully they at
least give you a taste of which file is responsible for which
task.
As we can see “/dev and “/proc”
are useful for all sorts of things whether they be network
related, devices (real or virtual), disks (loop disks and
physical drives) and much more.
Counting Processes
Now we’ve whet our whistles
with a quick look at the Process Table and pseudo filesystems,
“/dev” and “/proc”, let’s explore a few (hopefully useful)
unrelated command lines which may save the day at some point in
the future.
You might be concerned that a
single member of a cluster of identical machines is beginning to
give up the ghost. You could check how many processes the system
has created since its last reboot by using this
command:
# grep processes /proc/stat
Shovel at the ready now look
inside the file “/proc/$PID/cmdline” (replacing “$PID” for your
Process ID) and there you will find some buried goodies. This
file retains the entire and complete command line for a process
(almost always at least, zombie processes being one of the
exceptions so please don’t fret if your mileage
varies).
Sitting alongside that file, in
the PID’s directory, is the “cwd” symbolic link (or symlink) to
the current working directory of that process.You may need to be
logged in as “root”.
In order to discover the
current-working-directory of that process run this
command:
# cd /proc/$PID/cwd; /bin/pwd
We probably need to be “root”
sometimes because the symlink is hidden to normal users. I won't
claim to understand all of the intricacies of these pseudo files
but if you use the commands “cat” or “less” to view some of these
files usually then a little more light is shed on their raison d’
etre.
One such pseudo file (which is
less mysterious thanks to its name) is the disk input/output
statistics file, named “io”. By running this command and seeing
its output in Listing Ten:
# cat io
rchar: 0
wchar: 0
syscr: 0
syscw: 0
read_bytes: 342
write_bytes: 11
cancelled_write_bytes:
0
Listing Ten: Here we can see
what this process has been up to in relation to disk
activities
Amongst many others there's
another useful addition which is the "maps" file. In Listing
Eleven we can see the memory regions and their access permissions
for a process by using:
# cat maps
7eff839c7000-7eff839de000 r-xp
00000000 fd:01 3221
/lib64/libpthread-2.12.so
7eff839de000-7eff83bde000 ---p
00017000 fd:01 3221
/lib64/libpthread-2.12.so
7eff83bde000-7eff83bdf000 r--p
00017000 fd:01 3221
/lib64/libpthread-2.12.so
7eff83bdf000-7eff83be0000 rw-p
00018000 fd:01 3221
/lib64/libpthread-2.12.so
7eff843ac000-7eff843b3000 r-xp
00000000 fd:01 3201
/lib64/libcrypt-2.12.so
7eff843b3000-7eff845b3000 ---p
00007000 fd:01 3201
/lib64/libcrypt-2.12.so
7eff845b3000-7eff845b4000 r--p
00007000 fd:01 3201
/lib64/libcrypt-2.12.so
7eff845b4000-7eff845b5000 rw-p
00008000 fd:01 3201
/lib64/libcrypt-2.12.so
7eff82fb4000-7eff83025000 r-xp
00000000 fd:01 3478
/lib64/libfreebl3.so
7eff83025000-7eff83224000 ---p
00071000 fd:01 3478
/lib64/libfreebl3.so
7eff83224000-7eff83226000 r--p
00070000 fd:01 3478
/lib64/libfreebl3.so
7eff83226000-7eff83227000 rw-p
00072000 fd:01 3478
/lib64/libfreebl3.so
Listing Eleven: An extremely
abbreviated sample of the “maps” file for a process
Apparently the legend for the
permissions is as follows:
r = read
w = write
x = execute
s = shared
p = private (copy on
write)
The "maps" file can be useful
to see how a process is interacting with the system’s files. Or
maybe you’re curious as to which libraries a process needs and
you have forgotten the correct options to add to a tool like the
super-duper “lsof”.
Any eagle-eyed console
champions will spot that the file sizes, having used "ls" to list
the
files in the directory, all
appear as zero bytes. Clearly these pseudo files are different
animals than we’re used to getting our fingers nipped
by.
Kernel Support
Let's now move onto other
benefits of /proc and not just on a per-process basis.
For example you might look at
the filesystems which were compiled into the kernel by checking
out "/proc/filesystems".
In Listing Twelve you can see
(an abbreviated list) of what filesystems our kernel supports and
even as standard it’s a sizeable list.
nodev sysfsnodev rootfs
nodev bdev
nodev proc
nodev cgroup
nodev cpuset
nodev tmpfs
nodev devtmpfs
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev inotifyfs
nodev devpts
nodev ramfs
nodev hugetlbfs
nodev pstore
nodev mqueue
Listing Twelve: Here we can
see an abbreviated list of the types of file systems supported by
the kernel without having to make any tricky changes
You may have heard of an
excellent lightweight utility called "vmstat" which reports back
dutifully with screeds of useful memory statistics.
You may not, at this juncture,
fall off your seat if I tell you that in order to retrieve this
useful
information the following files
are used. Note the asterisk for all PIDs etc.
/proc/meminfo
/proc/stat
/proc/*/stat
Another aspect that the kernel
deals with on a system is the hardware ports, you know the kind
that accept a keyboard or a mouse. Have a peek at Listing
Thirteen having looked into the file "/proc/ioports", again
abbreviated for ease.
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0071 : rtc0
0080-008f : dma page
reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
Listing Thirteen: Displaying
an abbreviated output from the file "/proc/ioports"
As expected we can also look at
network information. I’ll leave you to trying some of these
yourself. You might want to begin by trying this:
# ls /proc/net
Or for live ARP information
(the Address Resolution Protocol) use the “cat” command in front
of this:
# cat proc/net/arp
If that output looks familiar
then think of how many utilities which you’ve used in the past
that refer to this file.
Why stop with ARP? SNMP
statistics are also readily available:
# cat /proc/net/snmp
There's a whole heap (no memory
puns intended) of other options which we could explore but
instead what about a look at changing some of the "sysctl"
settings I mentioned earlier too.
Sadly the majority of the
powerful changes you can make to a running system's kernel are
for another day (there's simply too many to cover) but let's take
the popular "/proc/sys" directory for a quick look before we
finish.
What about if the system-wide
limit of the number of open files for every process on the system
is holding you up when you’re trying to launch your latest
ground-breaking application?
This setting can cause all
sorts of system headaches if it’s abused (and it appears that not
only the “root” user can change it on occasion). Warnings aside
it’s very useful to arm yourself with this quick fix if it's ever
necessary.
Try this command yourself on a
test box:
# echo 999999 > /proc/sys/fs/file-max
Once you've conquered that
particular outage-inducing issue, taking all the glory as you go,
how about checking that the new setting has stuck? Treat it like
any other file:
# cat /proc/sys/fs/file-max
I should have of course
mentioned that you should check the original value before making
any changes (so you can revert the value if need be). My computer
used "6815744".
What about another setting?
This usually uses a start/stop daemon to make the change, on some
systems, but let's do it on the fly:
# echo "ChrisLinux" > /proc/sys/kernel/hostname
In case that you’re wondering
you will probably struggle to edit these files with an editor
(you may get “file has changed” errors) hence using “echo” as
above.
On other distributions you can
affect the same change with:
# hostname "ChrisLinux"
I'm sure you get the idea; it’s
hardly Computer Science after all.
The End
We have looked in some detail
(but not nearly enough) at the very clever Unix-like process
table, procfs and “/dev” briefly. We covered how to read and
sometimes manipulate tunable settings and how they integrate with
components running on Unix-like systems.
Hopefully this starter-for-ten
chapter will give you enough of an insight to know what to look
if you encounter issues in the future. Above everything else
however it’s important for you to find the time to fire up a test
machine, dig into otherwise buried file paths and tweak the
available settings to further increase your knowledge.
Incidentally should you ever
find yourself at a loss on a rainy afternoon then try running
this command:
# /sbin/sysctl -a
That should definitely offer
you some food for thought. It helpfully lists all of the tunable
kernel parameters on a system, each of which can be altered with
"echo", simply read the “dots” as “slashes” to find them inside
/proc. Just don’t make ill-advised, arbitrary kernel changes on a
production machine or you will likely rue the day!
If you weren’t aware of some of
the settings that we’ve covered then I hope you will enjoy your
newly gained Sysadmin knowledge and use it creatively in the
future.
Linux Books
If you've enjoyed reading the technical content on this site then please have a look at my Linux books which were both published in 2016 and some of my articles in Linux Magazine and Admin Magazine are available on their respective websites.
No comments:
Post a Comment