Google is currently in the middle of upgrading from ext2 to a more up to date file system. We ended up choosing ext4. This thread touches upon many of the issues we wrestled with, so I thought it would be interesting to share.

Just think about this for a moment. Google was still running on ext2 filesystem. Although ext2 remains one super stable filesystem, imagine the size of the impact of the forthcoming upgrade to ext4 filesystem. In case your are not familiar with the ext4 filesystem, ext4 is a journaling file system developed as the successor to ext3. Ext4′s main features range from backward compatibility, bigger filesystem/file sizes and multiblock allocation to fast fsck, online defragmentation and inode-related features. For more detailed information you should check out kernel.org‘s article on ext4.

The driving performance reason to upgrade is that while ext2 had been “good enough” for a very long time the metadata arrangement on a stale file system was leading to what we call “read inflation”. This is where we end up doing many seeks to read one block of data. In general latency from poor block allocation was causing performance hiccups. We spent a lot of time with unix standard benchmarks (dbench, compile bench, et al) on xfs, ext4, jfs to try to see which one was going to perform the best.

–Michael Rubin

Now, back to Google, what do you think this could mean to our future searches? Search results showing up in light speed time? Probably. For example, take a look at Google Goggles. It is an amazing product right? However the current status quo of the speed of its results is limited by the perception we have about the device’s speed and not only. I mean most of us realize it is not an easy thing to match a photo of the Parthenon temple I just took with trillions of random photos indexed somehow. Every query like this one, is narrowed by the network bottleneck and the processing speeds, which include hard disk’s read/write speed.

Imagine a super-fast network, with lets say 10Gbps bandwidth, where on the one end you press a <button> and all it does is erasing 10^100 rows and recreating 10^100 rows in a SQL database on the other end. Given that, apart from the CPU usage which is required for a process like this, there is the limit of read and write speed of the hard disks even if using RAID technology. This is where ext4 comes in to make this whole process a lot faster.

I can only imagine what’s next for our future searches. Fast and responsive holographic search results? I really don’t think there is a limit here, right?