Thursday, November 29, 2007

How to test and monitor disk performance

In my RAID performance article I've given a lot of performance numbers, but it is indeed a good question on how to exactly measure disk performance. To do this you need 2 tools. One that does the actual benchmark and one that measure the different system parameters so you can know the impact of the benchmark.

For the benchmark tool itself I usually use 3 tools. The first one is good old dd. The problem with dd is that it can only do 1 type of benchmark (sequential read and write), but it can to this to and from many sources and that is its strength.
So how do you use dd for a disk benchmark. Well usually like this :
dd if=/dev/zero of=/dev/sdb bs=64k count=100k
dd of=/dev/null if=/dev/sdb bs=64k count=100k
First a word of warning. Using dd like this will mean that any data on /dev/sdb (including filesystems) will be overwritten. The dd command does a write test and the second dd command does a read test. The write test reads from /dev/zero and writes to /dev/sdb, it uses a blocksize of 64k and it reads 102400 blocks. The read test reads from /dev/sdb and writes to /dev/null, it uses the same amount and size of blocks like the read test.
So in this case we are testing /dev/sdb, but this can be any blockdevice. Like /dev/md1 (a software RAID device) or /dev/VolGroup00/Logvol01 (a LVM logical volume.
The amount of blocks you need to read or write to have a valid is very simple. Make sure you process at least twice the amount of memory you have in your system. Why ? Because the Linux kernel does caching you need to make sure you request more data then the size of the cache to make sure you are actually testing the disk and not the Linux kernel caching.
And then the blocksize remains. Well, this depends on the type of usage you will do on the disks or RAID arrays. There are no real fixed rules for this. But generally speaking when you have a lot of small files take a small blocksize (4 to 16k), for standard fileserving and databases take a medium blocksize (64 to 128k) and for large files or applications that do a lot of sequential I/O use a large blocksize (256k and larger).

One final note on dd, since CentOS 5 dd will, when it has finished a test, tell you the speed if obtained. But on older versions of CentOS you will need to calculate this yourself. Another option there is to use ddrescue, which is a tool similar to dd but also provides the bandwidth used in its output. You can find ddrescue for CentOS in Dag's RPMForge repository.

As I said, dd only does sequential I/O but since it does not need a filesystem it is very useful to get a baseline/indication/magnitude/... for the performance of a disk. The next step is to put a filesystem on the device being tested. Then the second tool I use to test performance is iozone. You can also find prebuild rpms for CentOS in Dag's RPMForge repository.

I usually use iozone like this
iozone -e -i0 -i1 -i2 -+n -r 16k -s8g -t1
Iozone has a look of options (that is part why I like it). If you do iozone -h you will get the complete list. I suggest you go over them when you start testing so you know all your options. But the example above shows enough to get started. The -e option adds the time of the flush operation to the results and the -+n disables the retest feature. The different -i options indicate the tests done by iozone, -i0 is the sequential write test (always needed), -i1 is sequential read and -i0 is random read and write. Another interresting test is the mixed read/write (-i8). The 3 remaining options are : -r 16k is the record (or block) size used, -s8g is the size of the data tested (8 GB in this case, remember the 2x RAM rule here) and -t1 is the amount of threads performing the test (in this case 1).

Depepding in the application you are planning to use you try to use iozone in a way that it mimics as close as possible the IO patterns of your real application. This means playing with the -i (the kind of test), the -r (block size) and -t (number of threads) options.

Other nice features of iozone are the autotest. Here iozone can iterate over different tests using different parameters for file size and record size. Also iozone can output its results in a Excell file, so you later make nice graphs of the tests.

The final test that I perform is a bit less scientific, but still a good indication of performance of a disk or a RAID array. The test is a copy of a directory that contains about 1GB of data. This data is stored in different subdirectories (I think it is about 5 levels deep) and all these subdirs contains either a couple of big files and for the rest a lot of small files.
This means that a copy of this directory involves a lot of metadata activity, and a mix of sequential I/O (the big files) and something that resembles random or mixed I/O (the small files). And the result is usually a big difference compared to the sequential speeds obtained with dd or iozone.

Now that we have all these benchmarks we still need a tool to monitor the system itself and see what is going on. The tool I always use for this is dstat. It is written and maintained by Dag of the RPMForge repo (and of course it also contains CentOS rpms of dstat). The nice thing about dstat is that is has different monitoring modules (and you can also write your own), its output formatting is really good and it can output the results in CSV files so you can also process the results in a spreadsheet. This is a example on how to use dstat :
dstat -c -m -y -n -N eth2,eth3 -d -D sda -i -I 98 3
----total-cpu-usage---- ------memory-usage----- ---system-- --net/eth2- --dsk/sda-- inter
usr sys idl wai hiq siq| used buff cach free| int csw | recv send| read writ| 98
0 0 100 0 0 0| 190M 195M 924M 2644M|1029 92 | 0 0 | 793B 4996B| 14
0 0 99 1 0 0| 190M 195M 924M 2644M|1008 41 | 21B 0 | 0 23k| 1

All the diferent output are different output modules and they get display in the order you have put them on the commandline. With the (for example) -n -N structure you can specifiy for which devices you would like to see output. In this case the -n refers to network-card statistics and with -N I specified I would like to see the output for eth2 and eth3. A similar systems is used for the disk statistics (-d and -D) and number of interrupts (-i and -I).

To get the list of extra modules available next to the standard onces do this :
dstat -M list
/usr/share/dstat:
app, battery, cpufreq, dbus, freespace, gpfs, gpfsop, nfs3, nfs3op, nfsd3, nfsd3op, postfix, rpc, rpcd, sendmail, thermal, utmp, vmkhba, vmkint, vzcpu, vzubc, wifi,
For more information about all the different dstat options do dstat -h or refer to the dstat man-page.

44 comments:

Anonymous said...

Hi Tim,

Would you consider making your rss tab a bit more friendly with Firefox. FF can not figure out what your tag corresponds to:

tag:blogger.com,1999:blog-2299401950681542520.post-585698113837316006

-T

Anonymous said...

Hi Tim,

2 of my servers and one other person's service has reported that they are SLOW after upgrading to Cent OS5. Mine are 3 times slower. I have posted a bug at:

http://bugs.centos.org/view.php?id=2382

But so far no joy. The problem seems to be those using hardware RAID cards. And,
when I test against an identical 4.x server, I get minor differences (like with bonnie++), but the 4.x server is still 3 times faster executing application.

Would you mind taking a peek at the bug posting to see if you can suggest a test? (I am on the mailing list for the bug. I will see anything you post to it.)

Many thanks,
-T

Anonymous said...

Hi Tim,

tag:blogger.com,1999:blog-2299401950681542520.post-58

comes from

Planet Cent OS: http://planet.centos.org/rss20.xml

-T

Anonymous said...

While I certainly find utilities like dd (though I really prefer robin millers dt) and iozone good for summary info, I like to look at what's really going on every second as the one big problem with just looking at total run time or average rate is you never really know what was happing during the test. Was the CPU hammered? Was memory starved? Where there too many context switches? And let's not forget the network for remote types of I/O.

In fact I apply that type of logic to monitoring everything in the system which is one of the reasons I wrote collectl - see http://collectl.sourceforge.net/ for lots of examples and if you like what you see download it and fire it up.

-mark

chr1s said...

Hi Tim,

I have found a 1:1-copy of this blog entry in one other blog...:

http://shell.burgas.org/2009/02/how-to-test-and-monitor-hard-disk-performance/

I thought you are interested in.

Bye,
Chr1s
http://chr1stus.blogspot.com

Anonymous said...

Once I played Asda Story, I did not know how to get strong, someone told me that you must have Asda Story gold. He gave me some Asda Story money, he said that I could buy Asda Story Gold, but I did not have money, then I played it all my spare time. From then on, I got some cheap Asda Story gold.
I think 9Dragons is my favorite game, I did not know how to play at first, someone told me that you must have 9Dragons gold. That he gave me some 9 Dragons gold, and then he said that I could buy 9 Dragons gold, but I did not have so much money, and then I played it in all my spare time. From then on, I have got a lot of cheap 9Dragons gold, if I did not continue to play it, I can sell 9Dragons money to anyone who want.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

corporate branding division said...

I'm glad you're back to posting. I love reading about your shopping adventures and great finds. Your outfits are always super cute too! Very inspirational. Please come visit my site brand marketing when you got time.

industrial goods marketing said...

Hey congrats on the new posting come out
btw i love your blog although i have just stumbled upon it =)
Love the new pictures you got there! Please come visit my site marketing companies give me any valuable feedbacks.

Anonymous said...

There is obviously a lot more than this. Would you mind telling me how long it took you to gather your content? Please come visit my site Wichita Yellow Pages when you got time.

Anonymous said...

There is obviously a lot more than this. Would you mind telling me how long it took you to gather your content? Please come visit my site Newark Business Phone Book when you got time.

Anonymous said...

Very nice information. Thanks for this. Please come visit my site Colorado Springs Phone Numbers when you got time.Denver

Anonymous said...

Very nice information. Thanks for this. Please come visit my site Colorado Springs Business Search when you got time.Denver

Getting Divorced said...

I usually don’t leave comments!!! Trust me! But I liked your blog…especially this post! Would you mind terribly if I put up a backlink from my site to your site? Please come visit my site Divorce Advice give me any valuable feedbacks.

Anonymous said...

Who knows where to download XRumer 5.0 Palladium?
Help, please. All recommend this program to effectively advertise on the Internet, this is the best program!

Anonymous said...

Such interesting read and information, thanks for sharing this post. I will check back to read your other new posts. Please come visit my site Milwaukee Business Services And Classifieds when you got time.

Adi said...

Oes Tsetnoc one of the ways in which we can learn seo besides Mengembalikan Jati Diri Bangsa. By participating in the Oes Tsetnoc or Mengembalikan Jati Diri Bangsa we can improve our seo skills. To find more information about Oest Tsetnoc please visit my Oes Tsetnoc pages. And to find more information about Mengembalikan Jati Diri Bangsa please visit my Mengembalikan Jati Diri Bangsa pages. Thank you So much.
Oes Tsetnoc | Semangat Mengembalikan Jati Diri Bangsal

rr8004 said...

Hello, What enticed you to post this article. It was extremely interesting, especially since I was searching for info on this subject last Thursday. Please come visit my site Fortworth Business Directory when you got time.

rr8004 said...

Hello, What enticed you to post this article. It was extremely interesting, especially since I was searching for info on this subject last Thursday. Please come visit my site Business Directory Fortworth when you got time.

rr8004 said...

I found your site on google, great site, keep it up. Will return in the future. Submitted this post to Google News Reader. Please come visit my site Resource Guide Columbus Ohio OH when you got time.

rr8004 said...

I found your site on google, great site, keep it up. Will return in the future. Submitted this post to Google News Reader. Please come visit my site Directory Of Columbus when you got time.

rr8004 said...

. Since I’m new to blogging, these articles are greatly appreciated; very useful and informative blog and every body must visit this blog. Please come visit my site Garland Yellow Page Business Directory when you got time

rr8004 said...

. Since I’m new to blogging, these articles are greatly appreciated; very useful and informative blog and every body must visit this blog. Please come visit my site Phone Directory Of Garland City Texas TX State when you got time

rr8004 said...

Wow! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog? Please come visit my site Indianapolis Phone Book when you got time.

rr8004 said...

Wow! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog? Please come visit my site Indianapolis Business Directory Forum Blog Classifieds when you got time.

Trades & Services Telephone Directory Of Chatham Ontario said...

If you had some way of rating posts I would for sure give you a high rating my friend!

Chatham Business Directory, Chatham Agriculture, fishing & Forestry, Chatham Apparel & Accessories, business directory listings of Chatham Automotive, Chatham Business & Professional Services, Chatham Computers, Communications & Electronics, home garden furnishing, real estate business finderChatham Construction & Renovation, Chatham Education, Chatham Entertainment & Media, Chatham Family & Community, Lawyers Attorneys & Law Firms Directory Chatham Finance & Legal,dinning restaurants Entertainment serving Chatham Food & Beverages, Chatham Health & Medicine Doctors Hospitals, Chatham Home & Garden, Chatham Industrial supplies & services , Chatham Personal Care, Chatham Public utilities & environment, Chatham Real-Estate & Insurance, Chatham Shopping & Specialty Stores, shopping, retail, department stores company guide of Chatham Sports & Recreation, Chatham Transportation, Chatham Travel & Lodging

Anonymous said...

Check this performance test... they use DSTAT to test OLTP performance test

Unknown said...

South carolina insurance companies, insurance agents brokers and insurance guide. Comprehensive information about South carolina car insurance, home owner insurance, health insurance, life insurance and more.



South carolina
Home Health Care Insurance
, South carolina
Home Insurance
, South carolina
Hospitalization Insurance
, South carolina
Insurance
, South carolina
Insurance Adjusters
, South carolina
Insurance Agents & Brokers
, South carolina
Insurance Annuities
, South carolina
Insurance Bonds
, South carolina
Insurance Claims & Services
, South carolina
Insurance Claims Processing Services
, South carolina
Insurance Consultants & Advisors
, South carolina
Insurance Employment Agencies
, South carolina
Insurance Examinations
, South carolina
Insurance Examiners

Unknown said...

Nice experience shared. Its not less than an interview. Great way of posting such good and informative stuff. vinyl
floors
, auto repair services, security
agencies
, vinyl plank flooring, foreign
car repair
, cork floors

Unknown said...

If you had some way of rating posts I would for sure give you a high rating my friend!
auto
repair costs
, floating flooring, auto
repair advice
, flooring companies, auto
shops
, flooring services

Anonymous said...

If you had some way of rating posts I would for sure give you a high rating my friend!
Connecticut Alcohol Rehab Centers, Illinois
Alcohol Rehab Centers
, Kansas
Alcohol Rehab Centers
, Ohio
Alcohol Rehab Centers
, Michigan
Alcohol Rehab Centers
, Oregon
Alcohol Rehab Centers
, wholesale
men's clothing
wholesale fashion apparel, Wholesale
necklaces,

Anonymous said...

Your summaries are always top-notch. Thanks for keeping us apprised. I’m reading every word here. Wholesale Fashion Earring, exporter
company
, Connecticut Drug And Alcohol Rehabilitation, Illinois
Drug And Alcohol Rehabilitation
, Kansas
Drug And Alcohol Rehabilitation
, Ohio
Drug And Alcohol Rehabilitation

vishal said...

Easy Earning money in online never been this easy and transparent. You would find great tips on how to make that dream amount every Day. Sitting in the home earn around $100 per day. So go ahead and click here for more details and open floodgates to your online income. All the best.
Thanks.

Dylan Hall said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.

eagle parts

Anonymous said...

レンタルサーバー
レンタルサーバー
レンタルサーバー
レンタルサーバー

Anonymous said...

結婚したい
パートナーエージェント 結婚
ブライダルネット 結婚
ミアモーレ
秋田 結婚
神奈川 結婚
三重 結婚
広島 結婚
大分 結婚
結婚相談所
サンマリエ 婚活
エキサイト恋愛結婚 婚活
エヴァ 婚活
秋田 婚活
神奈川 婚活
三重 婚活
広島 婚活
大分 婚活
婚活難民
婚活サイト会員数
自衛隊プレミアクラブ 婚活
婚活女へ
結婚相談所 セレブ
ブライダルネットVIP
婚活時代
婚活 マッチ.com
婚活 サンマリエ
脚やせ
メンズTBC体験コース
メンズTBC体験

Cleveland Art said...

Very Very interesting blog. I just wanted to say that I love your blog. keep up the good job.

Cross Dressing said...

Great tutorial!

Anonymous said...

great tutorial

High speed VPS

study Chinese online said...

The best place to learn mandarin Chinese is in China. However, we understand that it isn't always possible to move here to study Chinese language. The next best thing is to study with our experienced teachers in a virtual classroom. Online students enjoy the same excellent way of Chinese Online Courses and custom designed courseware that we provide for our face to face clients.

Anonymous said...

Great info. I love all the posts, I really enjoyed, I would like more information about this, because it is very nice., Thanks for sharing.
baixar facebook
baixar whatsapp
unblocked games

Unknown said...

I know where I'm going and I know the truth, and I don't have to be what you want me to be. I'm free to be what I want.Thankyou i reaIIy love it..

Unknown said...

I know where I'm going and I know the truth, and I don't have to be what you want me to be. I'm free to be what I want.Thankyou i really love it

sharelink
filehippo
2016 free download latest version
acehfreesoft
umarka
offlineinstallerfilehippo
esoftpedia
filehorse
supportcanon
freedownloadlatetsversion

Unknown said...

level تسليك مجارى بالرياض
افضل شركة تنظيف بالرياض
تنظيف شقق بالرياض
تنظيف منازل بالرياض
شركة غسيل خزنات بالرياض
افضل شركة مكافحة حشرات بالرياض
رش مبيدات بالرياض
شركة تخزين عفش بالرياض
تنظيف مجالس بالرياض
تنظيف فلل بالرياض
شركة تنظيف بالرياض