Hack and / - Forensics with Ext4

 Learn from my mistakes as I figure out how to gather forensics data on an ext4 filesystem.
One great thing about writing technical articles is that you have a nice collection of documentation you can turn to. I tell people that I probably reference my books and articles more than anyone else, because although I may not always remember specific steps to perform a task, I do always remember whether I wrote about how to do it.
One article I find myself referring to now and then is the "Introduction to Forensics" article I wrote in Linux Journal back in the January 2008 issue (my first feature article in Linux Journal). In that article, I walk through how to use Autopsy, a front end to Sleuthkit, to perform your own forensics investigation on a server that has been hacked. Recently, I had to perform an investigation on a server that fell victim to an SSH brute-force attack (use SSH keys!) and discovered that my personal documentation no longer worked. In this article, I walk through the symptoms of this problem and ultimately how I was able to work around it.

The Victim

As I mentioned, recently I investigated a server (let's call it alvin to protect the innocent) that had been compromised by a brute-force attack. I followed the procedure I document in my forensics article to a T. I pulled the plug from the server the moment I detected it was compromised, immediately created an image of the entire drive, created a second copy of the image on a separate drive that I would work from, and once my evidence was secure, I re-installed the affected server with a clean OS.
The first big difference about this system I ran into compared to past investigations was the sheer size of the data. It's one thing to image a 10–20GB disk and quite another when the drive is hundreds of gigabytes and every action—creating the initial image, image duplication and md5sums—takes hours. This was a slow process to say the least. Once I had my working image, I fired up Autopsy and started my investigation. Initially, everything was fine—Autopsy saw the image and was able to detect its partitions, and when I started to browse the filesystem, I could see the contents of the root directory.

The Problem

The first problem showed up when I tried to navigate through the filesystem with Autopsy—all of the directories below the root directory appeared to be empty. I knew that couldn't be right, but I wasn't sure what the problem was. At first, I thought I might just have a corrupted copy of the image, but since the md5sum seemed to take almost as long as copying the image, I started the image copy again from my gold master just to be sure. When that still didn't work, I tried to use Sleuthkit from the command line and even tried tools from The Coroner's Toolkit, yet I got the same result. I thought perhaps Autopsy couldn't support such a large disk image. Before I did too much more research, I decided to try to mount the image loopback to see if it was a corrupt image. After all, if it were, I'd need to create a fresh image before I went any further.
The challenge here was that I had created an image of the entire disk, not just individual partitions. Normally, when you mount something loopback, you mount an image of a partition, and the syntax is something like:

$ sudo mount -o loop /path/to/image /mnt/image

Then, you can browse /mnt/image like any other mounted filesystem. In this case, since the image was of a full partition, I had to figure out how to skip ahead in the image and mount only the partition. Mount has an option called offset that allows you to specify how far ahead in the file to skip before it mounts it as a filesystem, but the trick is knowing what that offset should be set to. It turns out that if you have parted installed, it's relatively simple to find the offset. First, I run parted against alvin's full disk image and tell it to print out the full partition table in bytes:

$ sudo parted /media/Forensics2/images/alvin-sda
GNU Parted 2.2
Using /media/Forensics2/images/alvin-sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit                                                             
Unit?  [compact]? B                                                       
(parted) print                                                            
Model:  (file)
Disk /media/Forensics2/images/alvin-sda: 251059544064B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start          End            Size           Type     ...
 1      32256B         242242721279B  242242689024B  primary  ...
 2      242242752512B  249999160319B  7756407808B    extended ...
 5      242242753536B  249999160319B  7756406784B    logical  ...

(parted) quit                         

In this case, I'm interested in the first partition, and from this output, I can see that the partition starts at byte 32256. Now I can use that offset to mount the partition loopback:

$ sudo mount -o loop,ro,offset=32256 alvin-sda /mnt 

Note that I mounted the filesystem read-only here. I didn't want to risk writing any new data to this partition! From this point, I found that I could browse the filesystem under /mnt just fine—the problem somehow had to be with Autopsy and Sleuthkit. I did research, but I couldn't really find any evidence that there was an upper limit to the image size that I was close to, but all the same, I decided to try to create an image of just the initial partition. A few hours later, Autopsy still couldn't use the new image, but strangely enough, I could mount the partition loopback on my filesystem just fine. After trying countless other things, I started to realize that it couldn't be the size of my partition that was a problem. It had to be something else, and it was then that I noticed that this filesystem was ext4.

Like Ext3 Plus One More

To be honest, apart from the fact that ext4 touted faster speeds with large numbers of files while claiming backward compatibility with ext3, I hadn't given the filesystem much thought. I had used it on a few new systems (it is the default on modern Ubuntu installs), and it seemed to work fine. Because it was supposed to be backward-compatible, I initially wrote it off as being a cause of my problem. After more research though, I discovered not only did other users complain about lack of support for ext4 in Sleuthkit, but that the backward compatibility isn't as compatible as you might think. Although it's true that you can mount ext2 and ext3 filesystems as ext4, you can mount ext4 filesystems as ext3 only if the filesystem does not use extents (which is a major new feature of ext4, so it's commonly turned on). Because I couldn't treat this ext4 filesystem as ext3, that meant neither could Sleuthkit.
So if I could mount the partition loopback, what is the big deal if Sleuthkit and Autopsy didn't work? Although being able to browse through the filesystem is a useful feature of forensics tools, another incredibly valuable feature is the ability to create a filesystem timeline. A filesystem timeline organizes all the files into a giant text database where they are ordered according to their MAC times (when the file was last Modified or Accessed or Changed its metadata). With a filesystem timeline, if you have a good idea when the attackers might have been on the system, you can start to track their virtual footprints as they execute programs, browse directories and untar their scripts onto the system. Normally, I would have Autopsy generate this file for me. Luckily, it turned out that a tool from The Coroner's Toolkit called mactime was able to generate the timeline either from an image or from a mounted filesystem. Here's the command I used to create a timeline from the filesystem mounted at /mnt:

$ sudo mactime -d /mnt -R -g /mnt/etc/group -p 
 ↪/mnt/etc/passwd 1/2/1970 > timeline.txt

The -d option specifies the directory where the filesystem is mounted, and -R tells mactime to scan recursively through that directory. The -g and -p options tell mactime to get group and user ID information from the group and passwd files in my mounted filesystem, and finally, the date specifies the date range to start from. The date must be after 1/1/1970 (I'll leave why that is as an exercise for the reader), so I chose the next day.
Although I didn't have the same user experience I was used to with Autopsy, once I could browse the filesystem and view the timeline, I could use the same investigation techniques. Ultimately, I was able to piece together the timeline when the attacker compromised the machine via a weak password, changed the password so no one else could break in the same way, and then downloaded and launched SSH brute-force attack scripts to spread onto other servers.
I admit I was really surprised to see that some of my favorite forensics tools no longer worked. Although it is great that Linux filesystems continue to improve, one unintended downside (or possibly an upside if you are an attacker) is that forensics tools must be upgraded continually to work on modern filesystems. Although I was able to mount the partition and create a timeline, I don't know that I would have been able to recover any deleted files from the partition—another valuable use of forensics tools. That said, at least if I (or you) need to analyze an ext4 image again, I now have all of the steps documented.