Deploying Oracle Database on EXT File System with iSCSI SAN

In the past, I have always preferred running Oracle on ASM over EXT file system on Linux because I get better performance with raw disks.  Since Oracle ASM stripes across multiple disks in a disk group, this would give me much more IOPS for my database.

One day I decided to test Oracle on EXT file system and see if I would get similar results as I would with Oracle ASM.  The results with EXT file system were astounding.  Less than 1 percent difference between Oracle ASM and EXT4 file system.  The tool I used for my testing is called Silly Little Oracle Benchmark (SLOB) by Kevin Closson.  It can be found here: http://kevinclosson.wordpress.com/2012/02/06/introducing-slob-the-silly-little-oracle-benchmark/.

Oracle Automatic Storage Management (ASM) is a very cool product but not everyone knows how to configure it and administer it.  There is one downside of using Oracle ASM that I know of is the ASM disk cannot be more than 2TB in size unless your backend storage is an Exadata.  NOTE: this 2TB limit applies to each volume that you assign to ASM group.  If your ASM disk group has 5 volumes, with 2TB each, then you’d have 10TB of capacity in your ASM disk group.

Now, let’s talk about EXT file system.  Most *nix Sys Admin out there knows how to create an EXT file system.  If I have a single LUN created on my backend storage and present the LUN to my Linux host, I can just run mkfs –t <fstype> against that LUN and I have myself a file system.  Here’s a brief EXT filesystem 101: EXT file system was implemented in 1992 as the first file system created specifically for the Linux kernel.  It is a metadata based file system.  It was the first implementation that used the virtual file system (VFS) and it could handle up to 2GB in size.  It was quickly superseded by EXT2 and over time by EXT3, and now EXT4.  Enhancements between EXT2, EXT3, and EXT4 include journaling, large file size support, and larger overall file system size just to name a few.  I also found out that most storage vendors out there claim that the data will be read/written across all of the available disks in the backend storage.  Hence, one would think that there is no need to create multiple LUNs to distribute all that I/Os at the operating system level and just let the backend storage does all the work.  While this is true but you will NOT get good performance by just creating a single LUN for your database.  The bottleneck will be at the operating system level.  This is true for both Oracle ASM and EXT file system.

To get more IOPS for your database, you need to create more than one LUNs on your backend storage and present them to your host.  With Oracle ASM, you can put these LUNs in an ASM disk group.  With EXT file system, you need to use Logical Volume Manager (LVM).  Having more LUNs allows the operating system to have more disk queues.  Think of it as having “more lanes to a highway”.  Picture yourself driving down a single lane highway and talking on the phone, you will most likely slow down and the other cars behind you would not be able to go any faster.  Therefore, causing extra latency to your commute and others behind you.  Now, if the highway has multiple lanes, then others could easily pass you and flip you off – their commute won’t get affected negatively.  One might wonder if I/O blocks could flip each other off. J

OK…is it true that one could get a performance boost by creating a file system spanning multiple LUNs?  Yes and no.  It depends on the IO pattern.  To get the most out of it, you need to tune the IO Scheduler, the stride and stripe-width, CPU scaling governor, and the mount options.  For iSCSI SAN, you also need to change some parameters in the /etc/sysctl.conf file.  Please keep in mind that not all backend storage will perform better with the default IO Scheduler.  You need to run some tests to find out which scheduler is suited for your backend storage.  Please note that some of these parameters need to be changed for Oracle ASM as well.  If you are a Nimble customer, I got your backJ.  All of the tuning parameters and instructions are documented in our Oracle + Nimble BPG, available @ http://www.nimblestorage.com/solutions/oracle.php.  If you use other storage, try the settings at your own risk.

In summary, here’s a quick comparison between running Oracle on ASM and EXT filesystem:

Leave a Reply