tcpdump fu

Packet capture is one of the most fundamental and powerful ways to do network analysis. You can learn virtually anything about what is going on within a network by intercepting and examining the raw data that crosses it. Modern network analysis tools are able to capture, interpret and describe this network traffic in a human-friendly manner.
tcpdump is one of the original packet capture (or "sniffing") tools that provide these analysis capabilities, and even though it now shares the field with many other utilities, it remains one of the most powerful and flexible.
If you think that tcpdump has been made obsolete by GUI tools like Wireshark, think again. Wireshark is a great application; it's just not the right tool for the job in every situation. As a refined, universal, lightweight command-line utility—much like cat, less and hexdump—tcpdump satisfies a different type of need.
One of tcpdump's greatest strengths is its convenience. It uses a "one-off-command" approach that lends itself to quick, on-the-spot answers. It works through an SSH session, doesn't need X and is more likely to be there when you need it. And, because it uses standard command-line conventions (such as writing to STDOUT, which can be redirected), tcpdump can be used in all sorts of creative, interesting and extremely useful ways.
In this article, I introduce some of the basics of packet capture and provide a breakdown of tcpdump syntax and usage. I show how to use tcpdump to zero in on specific packets and reveal the useful information they contain. I provide some real-world examples of how tcpdump can help put the details of what's happening on your network at your fingertips, and why tcpdump is still a must-have in any admin's toolbox.

Essential Concepts

Before you can begin to master tcpdump, you should understand some of the fundamentals that apply to using all packet sniffers:

• Packet capturing is passive—it doesn't transmit or alter network traffic.
• You can capture only the packets that your system receives. On a typical switched network, that excludes unicast traffic between other hosts (packets not sent to or from your machine).
• You can capture only packets addressed to your system, unless the network interface is in promiscuous mode.

It is assumed that you're interested in seeing more than just your local traffic, so tcpdump turns on promiscuous mode automatically (which requires root privileges). But, in order for your network card to receive the packets in the first place, you still have to be where the traffic is, so to speak.

Anatomy of a tcpdump Command

A tcpdump command consists of two parts: a set of options followed by a filter expression (Figure 1).
Figure 1. Example tcpdump Command
The expression identifies which packets to capture, and the options define, in part, how those packets are displayed as well as other aspects of program behavior.

Options

tcpdump options follow the standard command-line flag/switch syntax conventions. Some flags accept a parameter, such as -i to specify the capture interface, while others are standalone switches and can be clustered, such as -v to increase verbosity and -n to turn off name resolution.
The man page for tcpdump lists all available options, but here are a few of the noteworthy ones:

• -i interface: interface to listen on.
• -v, -vv, -vvv: more verbose.
• -q: less verbose.
• -e: print link-level (Ethernet) headers.
• -N: display relative hostnames.
• -t: don't print timestamps.
• -n: disable name lookups.
• -s0 (or -s 0): use the max "snaplen"—capture full packets (default in recent versions of tcpdump).

None of these are required. User-supplied options simply modify the default program behavior, which is to capture from the first interface, and then print descriptions of matching packets on the screen in a single-line format.

Filter Expression

The filter expression is the Boolean (true or false) criteria for "matching" packets. All packets that do not match the expression are ignored.
The filter expression syntax is robust and flexible. It consists primarily of keywords called primitives, which represent various packet-matching qualifiers, such as protocol, address, port and direction. These can be chained together with and/or, grouped and nested with parentheses, and negated with not to achieve virtually any criteria.
Because the primitives have friendly names and do a lot of the heavy lifting, filter expressions are generally self-explanatory and easy to read and construct. The syntax is fully described in the pcap-filter man page, but here are a few example filter expressions:

• tcp
• port 25 and not host 10.0.0.3
• icmp or arp or udp
• vlan 3 and ether src host aa:bb:cc:dd:ee:ff
• arp or udp port 53
• icmp and \(dst host mrorange or dst host mrbrown\)

Like the options, filter expressions are not required. An empty filter expression simply matches all packets.

Understanding tcpdump Output

How much sense the output makes depends on how well you understand the protocols in question. tcpdump tailors its output to match the protocol(s) of the given packet.
For example, ARP packets are displayed like this when tcpdump is called with -t and -n (timestamps and name lookups turned off):

arp who-has 10.0.0.1 tell 10.0.0.2
arp reply 10.0.0.1 is-at 00:01:02:03:04:05

ARP is a simple protocol used to resolve IPs into MAC addresses. As you can see above, tcpdump describes these packets in a correspondingly simple format. DNS packets, on the other hand, are displayed completely different:

IP 10.0.0.2.50435 > 10.0.0.1.53: 19+ A? linuxjournal.com. (34)
IP 10.0.0.1.53 > 10.0.0.2.50435: 19 1/0/0 A 76.74.252.198 (50)

This may seem cryptic at first, but it makes more sense when you understand how protocol layers work. DNS is a more complicated protocol than ARP to begin with, but it also operates on a higher layer. This means it runs over top of other lower-level protocols, which also are displayed in the output.
Unlike ARP, which is a non-routable, layer-3 protocol, DNS is an Internet-wide protocol. It relies on UDP and IP to carry and route it across the Internet, which makes it a layer-5 protocol (UDP is layer-4, and IP is layer-3).
The underlying UDP/IP information, consisting of the source and destination IP/port, is displayed on the left side of the colon, followed by the remaining DNS-specific information on the right.
Even though this DNS information still is displayed in a highly condensed format, you should be able to recognize the essential elements if you know the basics of DNS. The first packet is a query for linuxjournal.com, and the second packet is an answer, giving the address 76.74.252.198. These are the kind of packets that are generated from simple DNS lookups.
See the "OUTPUT FORMAT" section of the tcpdump man page for complete descriptions of all the supported protocol-specific output formats. Some protocols are better served than others by their output format, but I've found that tcpdump does a pretty good job in general of showing the most useful information about a given protocol.

Capture Files

In addition to its normal behavior of printing packet descriptions to the screen, tcpdump also supports a mode of operation where it writes packets to a file instead. This mode is activated when the -w option is used to specify an output capture file.
When writing to a file, tcpdump uses a completely different format from when it writes to the screen. When writing to the screen, formatted text descriptions of packets are printed. When writing to a file, the raw packets are recorded as is, without analysis.
Instead of doing a live capture, tcpdump also can read from an existing capture file as input with the -r option. Because tcpdump capture files use the universally supported "pcap" format, they also can be opened by other applications, including Wireshark.
This gives you the option to capture packets with tcpdump on one host, but perform analysis on a different host by transferring and loading the capture file. This lets you use Wireshark on your local workstation without having to attach it to the network and location you need to capture from.

Analyzing TCP-Based Application Protocols

tcpdump is a packet-based analyzer, and it works great for connectionless, packet-based protocols like IP, UDP, DHCP, DNS and ICMP. However, it cannot directly analyze "connection-oriented" protocols, such as HTTP, SMTP and IMAP, because they work completely different.
They do not have the concept of "packets". Instead, they operate over the stream-based connections of TCP, which provide an abstracted communications layer. These application protocols are really more like interactive console programs than packet-based network protocols.
TCP transparently handles all of the underlying details required to provide these reliable, end-to-end, session-style connections. This includes encapsulating the stream-based data into packets (called segments) that can be sent across the network. All of these details are hidden below the application layer.
In order to capture TCP-based application protocols, an extra step is needed beyond capturing packets. Because each TCP segment is only a slice of application data, it can't be used individually to obtain any meaningful information. You first must reassemble TCP sessions (or flows) from the combined sets of individual segments/packets. The application protocol data is contained directly within the sessions.
tcpdump doesn't have an option to assemble TCP sessions from packets directly, but you can "fake" it by using what I call "the tcpdump strings trick".

The tcpdump Strings Trick

Usually when I'm capturing traffic, it's just for the purpose of casual analysis. The data doesn't need to be perfect if it shows me what I'm looking for and helps me gain some insight.
In these cases, speed and convenience reign supreme. The following trick is along these lines and is one of my favorite tcpdump techniques. It works because:

1. TCP segments usually are sent in chronological order.
2. Text-based application protocols produce TCP segments with text payloads.
3. The data surrounding the text payloads, such as packet headers, is usually not text.
4. The UNIX command strings filters out binary data from streams preserving only text (printable characters).
5. When tcpdump is called with -w - it prints raw packets to STDOUT.

Put it all together, and you get a command that dumps real-time HTTP session data:

tcpdump -l -s0 -w - tcp dst port 80 | strings

The -l option above turns on line buffering, which makes sure data gets printed to the screen right away.
What is happening here is tcpdump is printing the raw, binary data to the screen. This uses a twist on the -w option where the special filename - writes to STDOUT instead of a file. Normally, doing this would display all kinds of gibberish, but that's where the strings command comes in—it allows only data recognized as text through to the screen.
There are few caveats to be aware of. First, data from multiple sessions received simultaneously is displayed simultaneously, clobbering your output. The more refined you make the filter expression, the less of a problem this will be. You also should run separate commands (in separate shells) for the client and server side of a session:

tcpdump -l -s0 -w - tcp dst port 80 | strings
tcpdump -l -s0 -w - tcp src port 80 | strings

Also, you should expect to see a few gibberish characters here and there whenever a sequence of binary data also happens to look like text characters. You can cut down on this by increasing min-len (see the strings man page).
This trick works just as well for other text-based protocols.

HTTP and SMTP Analysis

Using the strings trick in the previous section, you can capture HTTP data even though tcpdump doesn't actually understand anything about it. You then can "analyze" it further in any number of ways.
If you wanted to see all the Web sites being accessed by "davepc" in real time, for example, you could run this command on the firewall (assume the internal interface is eth1):

tcpdump -i eth1 -l -s0 -w - host davepc and port 80 \
  | strings | grep 'GET\|Host'

In this example, I'm using a simple grep command to display only lines with GET or Host. These strings show up in HTTP requests and together show the URLs being accessed.
This works just as well for SMTP. You could run this on your mail server to watch e-mail senders and recipients:

tcpdump -l -s0 -w - tcp dst port 25 | strings \
  | grep -i 'MAIL FROM\|RCPT TO'

These are just a few silly examples to illustrate what's possible. You obviously could take it beyond grep. You could go as far as to write a Perl script to do all sorts of sophisticated things. You probably wouldn't take that too far, however, because at that point, there are better tools that actually are designed to do that sort of thing.
The real value of tcpdump is the ability to do these kinds of things interactively and on a whim. It's the power to look inside any aspect of your network whenever you want without a lot of effort.

Debugging Routes and VPN Links

tcpdump is really handy when debugging VPNs and other network connections by showing where packets are showing up and where they aren't. Let's say you've set up a standard routable network-to-network VPN between 10.0.50.0/24 and 192.168.5.0/24 (Figure 2).
Figure 2. Example VPN Topology
If it's operating properly, hosts from either network should be able to ping one another. However, if you are not getting replies when pinging host D from host A, for instance, you can use tcpdump to zero in on exactly where the breakdown is occurring:

tcpdump -tn icmp and host 10.0.50.2

In this example, during a ping from 10.0.50.2 to 192.168.5.38, each round trip should show up as a pair of packets like the following, regardless of from which of the four systems the tcpdump command is run:

IP 10.0.50.2 > 192.168.5.38: ICMP echo request, 
 ↪id 46687, seq 1, length 64
IP 192.168.5.38 > 10.0.50.2: ICMP echo reply, 
 ↪id 46687, seq 1, length 64

If the request packets make it to host C (the remote gateway) but not to D, this indicates that the VPN itself is working, but there could be a routing problem. If host D receives the request but doesn't generate a reply, it probably has ICMP blocked. If it does generate a reply but it doesn't make it back to C, then D might not have the right gateway configured to get back to 10.0.50.0/24.
Using tcpdump, you can follow the ping through all eight possible points of failure as it makes its way across the network and back again.

Conclusion

I hope this article has piqued your interest in tcpdump and given you some new ideas. Hopefully, you also enjoyed the examples that barely scratch the surface of what's possible.
Besides its many built-in features and options, as I showed in several examples, tcpdump can be used as a packet-data conduit by piping into other commands to expand the possibilities further—even if you do manage to exhaust its extensive "advertised" capabilities. The ways to use tcpdump are limited only by your imagination.
tcpdump also is an incredible learning tool. There is no better way to learn how networks and protocols work than from watching their actual packets.
Don't forget to check out the tcpdump and pcap-filter man pages for additional details and information.

The tcpdump/libpcap Legacy

tcpdump has been the de facto packet capture tool for the past 25 years. It really did spawn the whole genre of network utilities based on sniffing and analyzing packets. Prior to tcpdump, packet capture had such high processing demands that it was largely impractical. tcpdump introduced some key innovations and optimizations that helped make packet capture more viable, both for regular systems and for networks with a lot of traffic.
The utilities that came along afterward not only followed tcpdump's lead, but also directly incorporated its packet capture functionality. This was possible because very early on, tcpdump's authors decided to move the packet capture code into a separate portable library called libpcap.
Wireshark, ntop, snort, iftop, ngrep and hundreds of other applications and utilities available today are all based on libpcap. Even most packet capture applications for Windows are based on a port of libpcap called WinPcap.

Resources

tcpdump and libpcap: http://www.tcpdump.org
Wireshark: http://www.wireshark.org
TCP/IP Model: http://en.wikipedia.org/wiki/TCP/IP_model