Tuesday, November 20, 2007

3ware Hardware RAID vs. Linux Software RAID

As part of a project we are building a iSCSI storage server. It has 16 500GB SATA disks and to provide redundancy we needed some sort of RAID controller. So we went with the well known 3ware controllers. More specifically a 9550SXU 16ML. The server itself is running CentOS 5 with 2 Dual-Core Intel Xeon processors. This post is just to share my experiences with this controller.

First, everybody that uses a ext3 filesystem on top of this controller should upgrade to firmware 9.4.2. It improves the write performance. In our case we went from around 55MB/s to around 75 MB/s for sequential I/O. That is not bad for a simple firmware upgrade.

But we were having one issue. On initial tests with a iSCSI exported logical volume we did a copy of a directory tree (on the same volume) with a total size of 1GB. And this test took around 6 minutes, that is even less then 3MB/s. To be fair, this structure contains a lot of smaller files, different directories. So a copy also involves a lot of metadata activity. So we did the same test on the storage server itself and we got around 3 minutes there. Mmm, and this should be a top end RAID controller, but we are only getting 6MB/s in this test.

So, just for fun I decided to let the RAID controller export each disk individually and use the Linux software RAID and see what performance that would give. Well, for this specific test the time was 2'50". And for all the other tests I did software RAID outperformed the hardware RAID.

So what did we learn here. First, do not let the numbers of all the different hardware RAID controller vendors foul you. They are sequential I/O test. But most day-to-day I/O patterns are different.
Second, always test yourself. Benchmarks found online can be indicators, but always test everything yourself for your specific case. Third, when using RAID always give software RAID a change. It may save you some money.

As a final note. If we mounted the filesytems as ext2 in this specific test the copy would only take 1 minute (in both HW and SW RAID). So do not forget about ext2, it still has it's advantages.

32 comments:

Paolo Campegiani said...

This bug is related to RHEL 3, but it could of interest:
https://bugzilla.redhat.com/show_bug.cgi?id=121434

Unknown said...

Can you please post how you're measuring your microbenchmarks as you're progressing? For example: what were you using to measure throughput (for your firmware upgrade)? I've built a 5x500GB SW RAID system with a single-core P4 machine, and I'd love to be able to track my performance against the monster you've put together.

Tim said...

That is a good question indeed. I'll do a writeup of how I tested all this in the coming days.
Thanks for your feedback!

Anonymous said...

RAID5 I'm assuming? I'd be interested to see what the results yield on XFS as well.

Anonymous said...

I have also had horrible performance on 3ware raid cards and was therefore considering LSI. do you have any input on a less expensive card for use with software raid, it seems all of the controllers that support a large number of drives are also hardware raid cards.

Tim said...

I've also been looking for simple SATA controllers with at least 8 ports. There are a couple. But the biggest challenge is finding one that has multilane connectors. So if someone knows any plane SATA controllers with multilane connectors please speak up :-)

Anonymous said...

I too was very disappointed of that 3ware card (with 8 ports) on an AMD Centos 5 sys. But then I tried with XFS, disabled cpu frequency scaling and installed irqbalance. Now the performance rocks, at least with iozone. (seq. writes ~180MB/s and read ~380MB/s)
You see I am not sure what did the trick but you should give XFS a try.

Tim said...

With 8 disks (I presume) those numbers aren't bad. Unfortunately we are more in the need for performance with other IO patterns then sequential. The array will also be used to host the filesystems of Xen guests (using iSCSI) and this makes it a bit more complicated to use XFS so we will be sticking with ext3 (or use it as ext2 when we need more speed).

Anonymous said...

Same here (random IO)

I'm in the process of testing various setups and interested in any idea for this.

Anonymous said...

Once I played AOC, I did not know how to get strong, someone told me that you must have aoc gold.He gave me some conan gold, he said that I could buy age of conan gold, but I did not have money, then I played it all my spare time. From then on, I got some cheap aoc gold, if I did not continue to play it, I can sell aoc money to anyone who want.
I was a new player in Aion, I did not know what is the aion kina. Someone told me that aion online kina is the game gold, he said that I could buy aion kina, but I did not have money, then I played it all my spare time. From then on, I got some aion gold, if I did not continue to play it, I can sell cheap aion kina to anyone who want.

Adi said...

Get the lates information of Electronics Gadgets Tech gadgets Cool Gadgets New Gadgets Best Gadgets Latest Gadgets Cool New Gadgets</a

Unknown said...

Yeah, ext2 still have its benefits. I still support it.

Website SEO

Unknown said...

Website SEO

SEO said...

SEO company

Grand SEO said...

Nice tips.But I think there will be a problem in installing it.

Grand SEO said...

Thanks for the tip. I will give it a try.

Unknown said...

South dakota insurance companies, insurance agents brokers and insurance guide. Comprehensive information about South dakota car insurance, home owner insurance, health insurance, life insurance and more.

South dakota
Disability Insurance
, South dakota
Earthquake insurance
, South dakota
Estate Planning Insurance And Legal Services
, South dakota
Farm, Crop, & Livestock Insurance
, South dakota
Financial Guaranty & Life Insurance
, South dakota
Fire Insurance
, South dakota
Fleet Insurance
, South dakota
Flood Insurance
, South dakota
Foreign Insurance
, South dakota
Franchising
, South dakota
Government Insurance Carriers
, South dakota
Group Insurance
, South dakota
Health Insurance

Unknown said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
auto
body shop
, international labour
organization
, cheap laminate
flooring
, auto tune up, labour
organizations
, cleaning flooring, transmission
repair
, union history

Unknown said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
discount
laminate flooring
, towing
services
, union local, flooring
options
, auto repair help, trade
union
, flooring sales

Anonymous said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
Illinois
Substance Abuse Counselor
, Kansas
Substance Abuse Counselor
, Ohio
Substance Abuse Counselor
, Michigan
Substance Abuse Counselor
, Oregon
Substance Abuse Counselor

Anonymous said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ? wholesale
earring jewelry
, plastic
exporter
, Connecticut Substance Abuse Counseling, Illinois
Substance Abuse Counseling
, Kansas
Substance Abuse Counseling
, Ohio
Substance Abuse Counseling

real estate software said...

Thanks for giving the clear definition about 3ware hardware RAID and Linux software RAID. This is very useful.

Unknown said...

Drum Beat Machine for Beat Makers, Make your own beats. The Best Beat Making Software program, download to make your own original hip hop beats. make beats

Property software said...

Property software comes in different versions to meet different needs with absolute precision

janet li said...

Hello!It is my first time visit your website.It is nice,just like our website.Yes,we are the cheapest ghd outlet.All of the Discount GHD Straighteners on sale here are high quality,and free shipping.I don't know whether you have already have a flat iron,we here have theghd uk for your choice.Australia is our main market as well.ghd australia also on sale.If you want to have a differdent hairstyle with less money,then come to our website,we have the confidence that we will bring you a nice mood everyday.Pick your beauty now,It is suitable for both men and women.Come on,your beauty is in our website waits for you.what's more,with a nice hairstyle,i think you will also need a nice christian louboutin clearance sale and Ghd Australia to match your hairstyle.Ghd Australia sincerely for your service!
At last,best flat iron thanks for sharing your articles.May you happy everyday.
ghd uk
ghd uk

Your Escort Agency said...

Your Escort Agency offers exclusive and most beautiful London escort girls of various nationalities.

London escorts said...

Bestescort4U agency provids best London escorts companionship in the UK.

Escorts London said...

Hot - Collection is a honest and confidential London escort agency which provides genuine London escorts girls for gentlemen of taste.

Learn Chinese said...

Welcome to the leading online Mandarin school that makes you fluent in a short time -- all from the comfort of your home or office!

I am a Chinese teacher,you can learn more about Free Chinese language info: Mandarin Chinese teacher | Online Chinese teachers | Native chinese teacher

Paul Wolf said...

Well, it was a wonderful and constructive post and I am glad to read this posting. Thanks for sharing it. free software download

Unknown said...

I really enjoyed the posting you have share in this article. free software download full version

David said...

The information is very good and very useful.
clipping path service provider
clipping path service
background removal service
image masking service
raster to vector conversion service

color corrections service

neck joint service

Shadow creating service