How to see what's going on with your Linux system right now

https://www.hpe.com/us/en/insights/articles/how-to-see-whats-going-on-with-your-linux-system-right-now-1807.html

Is that service still running? What application is using that TCP port? These questions and more can be answered easily by sysadmins who know simple Linux commands.
If you're a system administrator responsible for Linux servers, it's essential that you master the Bash shell, especially the commands that show what software is running. That's necessary for troubleshooting, obviously, but it is always a good idea to know what's happening on a server for other reasons—not the least of which is security awareness.
Previously, I summarized 16 Linux server monitoring commands you really need to know, what to do when a Linux server keels over, and Linux commands you should never use. Here are some more commands to further your system administration skills—and help you identify the current status of your Linux server. I consider these commands to be fundamentals, whether your servers run on bare iron, a virtual machine, or a container with a lifespan of only a few hours.
To put these in some kind of context, let's follow sysadmin Sammie as she goes through a typical day.

man

First, Sammie wants to check out exactly what a command does, so she turns to man.
Yes, I know I’m starting with the most obvious command. There's a reason that one of the knee-jerk put-downs in tech circles is Read the F(ine) Manual (RTFM). The secrets of the universe, or Linux/Unix anyway, are hidden inside the man command.
You use man searches to search for the manual page for a given command. So, for example,
# man top
shows you in great detail how to run the top command. (While I covered top in an earlier article, it’s worth a short review: top shows all active processes so you can see the most CPU-intensive tasks running on the server, and it provides a fast and easy way to see if any process is starting to lurch out of control.)
If Sammie needed a quick summary of what a command does, she could run apropos or whatis. But, for the down and dirty of what a command does—including its options and flags, and examples of how to use it—man is the command to use. And then, too, you won’t have to put up with someone telling you to read the freaking manual again. Because you have.

head and tail

Sammie finds herself staring at a log file that simply keeps scrolling and scrolling. Her answer? Turn to head and tail, two simple but powerful Linux commands. Head displays the first part of a file, while tail displays its last part. By default, they show the first 10 or last 10 lines.
So what? That's a lot actually!
Say you want to troubleshoot a program that's going wrong. Do you want to read its entire log? Probably not. Often, what matters is the information about the software’s last few actions. So Sammie starts troubleshooting with the command:
# tail /var/log/messages
If something was going awry, Sammie would keep an eye on its log file by using tail with the -f option.
# tail -f /var/log/messages
With this command, every new message that appears in that log file displays at a refresh rate of once per second.
Head isn't as useful, but it can be helpful for peeking at data. For, instance, it may be handy to get a quick look at the first few lines listed in a password file, such as:
# head /etc/passwd
With either command, you're not limited to the first or last 10 lines. The -n flag lets you display as many lines as you'd like. Thus, to see the last 20 lines in that log file:
#head -n 20 /var/log/syslog

cat

Sammie wants to do more with another log file. She knows another powerful, simple command for looking at files is cat, which concatenates and prints files. In other words, it displays all the contents of a given file. So, what's the use of that, you say, as you enter…
#cat /var/log/kern.log
…and your screen is instantly filled with hundreds or thousands of lines? Lots. Usually you use cat in conjunction with other commands. For example, you can pipe data from cat through the more and less commands to view a file's contents more easily:
# cat /etc/passwd | less
Of the pair, less is more functional. For instance, you can go backward from screen to screen.
Sammie can also use cat to redirect a file's standard output into a new file. So, to create a time-dated copy of a password file, she would type:
# cat /etc/passwd > passwd07042018
However, the most practical use for cat is to use it to pipe a file's content through to grep or another pattern-searching program such as awk or sed. For instance, to show any error messages about your PCI Express bus or devices, type:
# cat /var/log/kern.log | grep PCIe
So, as you can see, it's very handy to have a cat in your system.

grep

The grep program is so powerful that it is worth a book about itself—and it has one: the grep Pocket Reference. Sammie has a copy, and so should you.
Sammie wants to track down an active process that's eating more CPU cycles than it should. So she uses grep to find a pattern within a line of text. That line can reside in a file or be the output result of another command. Thus, grep is often used to filter data from another program. For example, one favorite way to use grep is in this one liner:
# ps -ef | grep
In this case, grep is used with ps, which displays data about active processes. The -ef flags tell ps to show all the active processes with standard Linux syntax. (If you were doing this on a BSD system, you'd use -aux rather than -ef.) When you pipe these ps results to grep, you include the name of the text (in this case, the misbehaving program you want to know about). That lets you examine the lines of text that are associated with that application, so you can figure out the source of your pain.

env

Sammie is also fond of env. This is a useful little command that can run another program in a custom environment without modifying the current shell.
For example, Sammie is logged in as Joe to see what's what with his bash_profile file. Joe's default editor is emacs, but Sammie prefers to use vim. So, she simply enters:
env EDITOR=vim
And she can use vim to edit his file.

Unsure how to get started with containers? Yes, we have a guide for that. Get Containers for Dummies.

Download now

More commonly, you use env without any arguments to see the current environment variables and their values. This includes such variables as HOME, SHELL, and PATH. For example, Sammie knows that if Joe's shell script isn't working as expected, there's a decent chance it's because he's calling a command that isn't in his PATH. Sure enough, that proved to be the case.

df and du

Next, Sammie has some containers that are running out of space. She knows there are many ways to learn how much storage your files are consuming. She likes to use du (disk usage) and df (display free disk space) because they're the simplest file system utilities.
To troubleshoot disk space problems, Sammie starts with df. Even though storage is cheap nowadays, this is still relevant. With the rise of containers, it's becoming more common to run into error messages about running out of free space on the container host. With df, it's easy to work if you're indeed out of room.
The simplest way to use df is with the -h option. This shows you the information in human-readable format. For example, to learn the display file system type, run:
# df -T
In this case, our intrepid sysadmin uses the following command to see how much room her DBA's MySQL databases are taking up:
# df /var/lib/mysql
Sammie follows this up by using the related du utility, which shows how much space each file takes up. As with df, you can use it with the -h flag for results that are more useful to the naked eye. When she wants to know if a file or directory is growing out of control (wait, how did that go from 34 MB to 304 GB?!), du is the easiest tool.

ip

There's no rest for a sysadmin. Now someone's workstation is hiccuping on the network. Sammie knows she could turn to any number of useful Linux network tools, such as tcpdump, but she regularly uses one great do-it-all program: ip. This collection of TCP/IP utilities is for networking and traffic control in Linux. Using ip with the command ip address displays the server's interfaces and IP addresses.
For example, if you have a container attached to two networks, the -a flag (which is short for address) shows which interface connects to which network.
# ip a
What's this? Is sjvn connected to a wireless network? That will never do! It's time to knock out that connection with the command:
ip link set dev wlp2s0 down
That will show him! (Ed. Probably not.)

(Image courtesy of the author.)
Sammie also knows ip can do far more than just show addresses. It includes other networking utilities, such as adding and deleting addresses; managing the routing tables; monitoring the state of devices, addresses, and routes; and creating IPv4 tunnels. Really, iproute is a powerful utility worth studying in detail.
If your system doesn't have the ip utility, install it as part of the iproute2 package.

id

Who on the network is running that application? Sammie can find the answer to that question and many more with the id command. By itself, id displays the currently logged-in user, the user ID, group ID and name, and other groups the current user belongs to. You can also feed it another user's name to get their data.
So how is this information useful? Say you're in a container and you want to install a program within it. Are you logged in as root for the container or as the admin for an application? It’s frustrating to try to install a program and fail because you don’t have sufficient user rights. Running id before you get started can save you from that annoyance.

Simple commands, big results

Most of these commands are basic. If you want to become a professional sysadmin like Sammie, you must master them.
True, DevOps tools such as Ansible, Chef, and Puppet make it easy to install, implement, monitor, and manage servers. DevOps makes it easy to define server configurations and maintain them with version control. But at their root, you find shell commands.
Sure, there are sexier sysadmin programs, such as the DevOps SaltStack, but these commands are fundamental—and powerful. Once you have a grip on them, you're well on your way to being a kick-butt sysadmin.