WARNING! WARNING! WARNING!
The changes below are NOT RECOMMENDED! See part III for details!
WARNING! WARNING! WARNING!

It’s been several weeks since I touched on this subject last, and I’m sorry to say that the solution I came up with in the earlier article just wasn’t good enough. It was forcing the system to swap, which slowed things down even more. I had to back the Linux virtual machine back down to 2.5GB of memory, which was just barely enough to keep up the fast compiles, if I made sure to always log out and back in each morning and always have my browser shut down before I started compiling.

Yesterday afternoon, while pondering this problem, it suddenly occurred to me that I might already know of a solution! The problem is the slow hard drive, and the amount of memory needed by the file-system cache to mitigate that. But if those files took up less space in the file-system, then presumably they would also take up less space in the cache as well, right? So why not compress the files?

Disk compression is a classic trade-off. You spend more CPU time in exchange for saving disk space. It’s less well known, but if you have a slow drive and a fast CPU, that can also mean faster access to the compressed data.

The next question was, how to do it? Compressing the host drive was chancy… it might help the speed, but it might equally hurt it, and I didn’t want to waste a lot of time on this on no more than a guess. Compressing the entire Linux system drive might be an option too, but I didn’t want to mess with the boot system if I didn’t have to. Then I realized that I didn’t have to — I could just store the files that the compiler would presumably need on a new and compressed virtual drive, and mount it on the file system where the original, uncompressed files were.

Okay, how do I turn on compression on EXT4…? Hm, slight problem: EXT4 doesn’t have any compression. Okay, not a big deal. Linux has lots of file systems. Surely some of them would handle compression.

Some do, but not many. According to Wikipedia’s comparison of file systems page, the only Linux file systems that support transparent compression are Reiser 4 and BTRFS. (There were a few others that support it and could be gotten to work, but they were user-space, or required extra drivers, or were read-only, or weren’t native to Linux.) I’ve considered Reiser 4 before, but last I heard it’s not under development anymore, and its namesake developer is in jail for the foreseeable future, which rather tarnishes it.

BTRFS is another story. I’ve been impatiently waiting for it to mature enough to safely use. After some consideration, I decided that it was sufficient for a mostly-read-only portion of the system.

I still don’t know what files gcc is using, but some poking around the file system suggested that most of them are likely to be under /usr. There were somewhere between 4.2GB and 4.4GB in that tree, depending on what program you believed. To be safe, I made a new 15GB virtual drive, formatted it to BTRFS (via gparted — yeah, I cheated, so sue me 😉 ), and mounted it with the “compress” switch by using a line like this in /etc/fstab…

/dev/sdb1  /mnt/btrfs  btrfs  compress,noatime  0  2

…copied the entire contents of the /usr directory to it (which took less than 15 minutes). df reported that the entire 4.2 or 4.4GB was actually using only 2.5GB, so the compression is working. 🙂 Then changed the /etc/fstab to this instead…

/dev/sdb1  /usr  btrfs  compress,noatime  0  2

…and rebooted. Everything came up properly, and mount reported that /usr was pointing to the new virtual drive, as it should.

Now for the tests.

Initial compile: 3:03 (down from 3:48)! Subsequent compiles, after changing a single source file: 0:17 (a slight increase over the 0:16 listed in the earlier entry, but well within measurement tolerance, as well as my own). And the kicker: after clearing out the entire cache (with sync then sudo su then echo 3 > /proc/sys/vm/drop_caches), and confirming that it was cleared out, I modified the same source file and started a compile. The cache when it was done: 1.1GB!

I hadn’t thought to drop the cache and test it with the older setup, but if it dropped below 1.4GB or 1.5GB, I’d start getting IOWait delays again. So it looks like that gives me a respectable amount of breathing room — enough, at least, that I can safely run a browser while compiling code. 😀

I suspect I could do better than that, if I could figure out what other files to compress. There are /lib, /lib32, and /lib64 directories off of the root, but they contain less than 500MB total. Poking around other directories didn’t show me any other obvious targets. In any case, what I have now should be better than before. We’ll see how it goes.