Tuesday, November 20, 2007

3ware Hardware RAID vs. Linux Software RAID

As part of a project we are building a iSCSI storage server. It has 16 500GB SATA disks and to provide redundancy we needed some sort of RAID controller. So we went with the well known 3ware controllers. More specifically a 9550SXU 16ML. The server itself is running CentOS 5 with 2 Dual-Core Intel Xeon processors. This post is just to share my experiences with this controller.

First, everybody that uses a ext3 filesystem on top of this controller should upgrade to firmware 9.4.2. It improves the write performance. In our case we went from around 55MB/s to around 75 MB/s for sequential I/O. That is not bad for a simple firmware upgrade.

But we were having one issue. On initial tests with a iSCSI exported logical volume we did a copy of a directory tree (on the same volume) with a total size of 1GB. And this test took around 6 minutes, that is even less then 3MB/s. To be fair, this structure contains a lot of smaller files, different directories. So a copy also involves a lot of metadata activity. So we did the same test on the storage server itself and we got around 3 minutes there. Mmm, and this should be a top end RAID controller, but we are only getting 6MB/s in this test.

So, just for fun I decided to let the RAID controller export each disk individually and use the Linux software RAID and see what performance that would give. Well, for this specific test the time was 2'50". And for all the other tests I did software RAID outperformed the hardware RAID.

So what did we learn here. First, do not let the numbers of all the different hardware RAID controller vendors foul you. They are sequential I/O test. But most day-to-day I/O patterns are different.
Second, always test yourself. Benchmarks found online can be indicators, but always test everything yourself for your specific case. Third, when using RAID always give software RAID a change. It may save you some money.

As a final note. If we mounted the filesytems as ext2 in this specific test the copy would only take 1 minute (in both HW and SW RAID). So do not forget about ext2, it still has it's advantages.

69 comments:

Paolo Campegiani said...

This bug is related to RHEL 3, but it could of interest:
https://bugzilla.redhat.com/show_bug.cgi?id=121434

Ken said...

Can you please post how you're measuring your microbenchmarks as you're progressing? For example: what were you using to measure throughput (for your firmware upgrade)? I've built a 5x500GB SW RAID system with a single-core P4 machine, and I'd love to be able to track my performance against the monster you've put together.

Tim said...

That is a good question indeed. I'll do a writeup of how I tested all this in the coming days.
Thanks for your feedback!

Anonymous said...

RAID5 I'm assuming? I'd be interested to see what the results yield on XFS as well.

Anonymous said...

I have also had horrible performance on 3ware raid cards and was therefore considering LSI. do you have any input on a less expensive card for use with software raid, it seems all of the controllers that support a large number of drives are also hardware raid cards.

Tim said...

I've also been looking for simple SATA controllers with at least 8 ports. There are a couple. But the biggest challenge is finding one that has multilane connectors. So if someone knows any plane SATA controllers with multilane connectors please speak up :-)

mjp said...

I too was very disappointed of that 3ware card (with 8 ports) on an AMD Centos 5 sys. But then I tried with XFS, disabled cpu frequency scaling and installed irqbalance. Now the performance rocks, at least with iozone. (seq. writes ~180MB/s and read ~380MB/s)
You see I am not sure what did the trick but you should give XFS a try.

Tim said...

With 8 disks (I presume) those numbers aren't bad. Unfortunately we are more in the need for performance with other IO patterns then sequential. The array will also be used to host the filesystems of Xen guests (using iSCSI) and this makes it a bit more complicated to use XFS so we will be sticking with ext3 (or use it as ext2 when we need more speed).

natmaka said...

Same here (random IO)

I'm in the process of testing various setups and interested in any idea for this.

chenlu said...

runescape money
runescape gold
runescape money
runescape gold

xuemei said...

Once I played AOC, I did not know how to get strong, someone told me that you must have aoc gold.He gave me some conan gold, he said that I could buy age of conan gold, but I did not have money, then I played it all my spare time. From then on, I got some cheap aoc gold, if I did not continue to play it, I can sell aoc money to anyone who want.
I was a new player in Aion, I did not know what is the aion kina. Someone told me that aion online kina is the game gold, he said that I could buy aion kina, but I did not have money, then I played it all my spare time. From then on, I got some aion gold, if I did not continue to play it, I can sell cheap aion kina to anyone who want.

jewelry wholesale said...

wholesale jewelryhandmade jewelryjewelry wholesalefashion jewelry

Anonymous said...

I like play online game, I also gw gold and GuildWars Gold, the Guild Wars Gold is very cheap, and use the GuildWars money can buy many things, I like cheap gw gold, thanks, it is very good.

I like play online game, I also buy habbo gold and habbo credits, the habbo gold is very cheap, and use the habbo coins can buy many things, I like cheap habbo credits, thanks, it is very good.

Anonymous said...

酒店經紀PRETTY GIRL 台北酒店經紀人 ,禮服店 酒店兼差PRETTY GIRL酒店公關 酒店小姐 彩色爆米花酒店兼職,酒店經紀, 酒店上班,酒店工作 PRETTY GIRL酒店喝酒酒店上班 彩色爆米花台北酒店酒店小姐 PRETTY GIRL酒店上班酒店打工PRETTY GIRL酒店打工酒店經紀 彩色爆米花

Anonymous said...

With the perfect combination of healthy sports and fashion trend, people begin to care about the sports suits which may lead to fashion. Then how to choose one functional and beautiful sports spectacles? Chinaamanda introduces to you.From the professional point of view, the material of glasses is very significant. For it may influence our sight to some extent. There are many kinds of sports spectacles for your choice on our website at http://www.amandaiec.com/. Please feel free to search it and choose your ideal style. Finally there will be one to make you satisfied. Meanwhile, wholesale is available at any time here. Online wholesale is acceptable. We can connect with the Yiwu market which is the biggest one in Yiwu about China wholesale directly. Go on visiting our website, we got various promotional gifts for Christmas. Shoes wholesale is our main wholesale. Dollar items are refreshed frequently, so that they are able to meet your needs. Come to Chinaamanda to find whatever you want.

care emergency preparedness said...

Such interesting read and information, thanks for sharing this post. I will check back to read your other new posts. Please come visit my site emergency earthquake when you got time.

kitchen remodel photos said...

I really liked your blog! Please come visit my site kitchen designers give me any valuable feedbacks

tolerance said...

anime cosplay follows Comic-Figures or and also your ideas and desires for cheap cosplay.Our cosplay costumes are not made in the same measurement, but are made with our cosplay special attention to your request.naruto cosplay or cosplay naruto is hot sale now. Furthermore, we will still send you the product Japanese School Uniform after our detailed further check.Yyou'll be surprised at the plentiful varieties such as Kingdom Hearts Cosplay, Bleach Cosplay, Final Fantasy Cosplayand you are sure to find the special anime cosplay costumes you've been looking for wholesale costume in our store.

Adi said...

Get the lates information of Electronics Gadgets Tech gadgets Cool Gadgets New Gadgets Best Gadgets Latest Gadgets Cool New Gadgets</a

Knee Injury Football Injuries said...

sorry to ask this here but… I really love your theme, would it happen to be a free one i can download somewhere, or is it a custom theme you had made? Soon i will be launching my own blog, however i’m not great with designs but i do like the style of your site so it would be excellent if i could find (or buy) something with a similar look as my last designer cannot finish my site. Thanks! Please come visit my site common sports injuries include knee injury when you got time.

Living In Apartment said...

Congratulations, you just earned yourself an entry in my feed reader, great blog. Please come visit my site living in apartment give me any valuable feedbacks.

Luis said...

Yeah, ext2 still have its benefits. I still support it.

Website SEO

lilena said...

Celiac symptoms, celiac disease, celiac diet, gluten-sensitive enteropathy gluten-sensitive enteropathy

lilena said...

Potty training tips, baby potty training. Resource guide on how to potty train, potty training for boys and girls.train potty

Adi said...

Oes Tsetnoc one of the ways in which we can learn seo besides Mengembalikan Jati Diri Bangsa. By participating in the Oes Tsetnoc or Mengembalikan Jati Diri Bangsa we can improve our seo skills. To find more information about Oest Tsetnoc please visit my Oes Tsetnoc pages. And to find more information about Mengembalikan Jati Diri Bangsa please visit my Mengembalikan Jati Diri Bangsa pages. Thank you So much.
Oes Tsetnoc | Semangat Mengembalikan Jati Diri Bangsa

Luis said...

Website SEO

SEO said...

SEO company

Grand SEO said...

Nice tips.But I think there will be a problem in installing it.

St Albert Canada said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

St Albert Business Directory, St Albert Agriculture, fishing & Forestry, St Albert Apparel & Accessories, business directory listings of St Albert Automotive, St Albert Business & Professional Services, St Albert Computers, Communications & Electronics, home garden furnishing, real estate business finderSt Albert Construction & Renovation, St Albert Education, St Albert Entertainment & Media, St Albert Family & Community, Lawyers Attorneys & Law Firms Directory St Albert Finance & Legal,dinning restaurants Entertainment serving St Albert Food & Beverages, St Albert Health & Medicine Doctors Hospitals, St Albert Home & Garden, St Albert Industrial supplies & services , St Albert Personal Care, St Albert Public utilities & environment, St Albert Real-Estate & Insurance, St Albert Shopping & Specialty Stores, shopping, retail, department stores company guide of St Albert Sports & Recreation, St Albert Transportation, St Albert Travel & Lodging

Find Alberta Businesses said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?


Alberta Business Directory, Alberta Agriculture, fishing & Forestry, Alberta Apparel & Accessories, business directory listings of Alberta Automotive, Alberta Business & Professional Services, Alberta Computers, Communications & Electronics, home garden furnishing, real estate business finderAlberta Construction & Renovation, Alberta Education, Alberta Entertainment & Media, Alberta Family & Community, Lawyers Attorneys & Law Firms Directory Alberta Finance & Legal,dinning restaurants Entertainment serving Alberta Food & Beverages, Alberta Health & Medicine Doctors Hospitals, Alberta Home & Garden, Alberta Industrial supplies & services , Alberta Personal Care, Alberta Public utilities & environment, Alberta Real-Estate & Insurance, Alberta Shopping & Specialty Stores, shopping, retail, department stores company guide of Alberta Sports & Recreation, Alberta Transportation, Alberta Travel & Lodging

Port Coquitlam Companies said...

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.
Port Coquitlam Business Directory, Port Coquitlam Agriculture, fishing & Forestry, Port Coquitlam Apparel & Accessories, business directory listings of Port Coquitlam Automotive, Port Coquitlam Business & Professional Services, Port Coquitlam Computers, Communications & Electronics, home garden furnishing, real estate business finderPort Coquitlam Construction & Renovation, Port Coquitlam Education, Port Coquitlam Entertainment & Media, Port Coquitlam Family & Community, Lawyers Attorneys & Law Firms Directory Port Coquitlam Finance & Legal,dinning restaurants Entertainment serving Port Coquitlam Food & Beverages, Port Coquitlam Health & Medicine Doctors Hospitals, Port Coquitlam Home & Garden, Port Coquitlam Industrial supplies & services , Port Coquitlam Personal Care, Port Coquitlam Public utilities & environment, Port Coquitlam Real-Estate & Insurance, Port Coquitlam Shopping & Specialty Stores, shopping, retail, department stores company guide of Port Coquitlam Sports & Recreation, Port Coquitlam Transportation, Port Coquitlam Travel & Lodging

Prince George facts said...

There are certainly a lot of details like that to take into consideration. That’s a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith.
Prince George Business Directory, Prince George Agriculture, fishing & Forestry, Prince George Apparel & Accessories, business directory listings of Prince George Automotive, Prince George Business & Professional Services, Prince George Computers, Communications & Electronics, home garden furnishing, real estate business finderPrince George Construction & Renovation, Prince George Education, Prince George Entertainment & Media, Prince George Family & Community, Lawyers Attorneys & Law Firms Directory Prince George Finance & Legal,dinning restaurants Entertainment serving Prince George Food & Beverages, Prince George Health & Medicine Doctors Hospitals, Prince George Home & Garden, Prince George Industrial supplies & services , Prince George Personal Care, Prince George Public utilities & environment, Prince George Real-Estate & Insurance, Prince George Shopping & Specialty Stores, shopping, retail, department stores company guide of Prince George Sports & Recreation, Prince George Transportation, Prince George Travel & Lodging

Manitoba MB Phone Numbers said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks

Manitoba Business Directory, Manitoba Agriculture, fishing & Forestry, Manitoba Apparel & Accessories, business directory listings of Manitoba Automotive, Manitoba Business & Professional Services, Manitoba Computers, Communications & Electronics, home garden furnishing, real estate business finderManitoba Construction & Renovation, Manitoba Education, Manitoba Entertainment & Media, Manitoba Family & Community, Lawyers Attorneys & Law Firms Directory Manitoba Finance & Legal,dinning restaurants Entertainment serving Manitoba Food & Beverages, Manitoba Health & Medicine Doctors Hospitals, Manitoba Home & Garden, Manitoba Industrial supplies & services , Manitoba Personal Care, Manitoba Public utilities & environment, Manitoba Real-Estate & Insurance, Manitoba Shopping & Specialty Stores, shopping, retail, department stores company guide of Manitoba Sports & Recreation, Manitoba Transportation, Manitoba Travel & Lodging

New Brunswick NB Phone Numbers said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ?

New Brunswick Business Directory, New Brunswick Agriculture, fishing & Forestry, New Brunswick Apparel & Accessories, business directory listings of New Brunswick Automotive, New Brunswick Business & Professional Services, New Brunswick Computers, Communications & Electronics, home garden furnishing, real estate business finderNew Brunswick Construction & Renovation, New Brunswick Education, New Brunswick Entertainment & Media, New Brunswick Family & Community, Lawyers Attorneys & Law Firms Directory New Brunswick Finance & Legal,dinning restaurants Entertainment serving New Brunswick Food & Beverages, New Brunswick Health & Medicine Doctors Hospitals, New Brunswick Home & Garden, New Brunswick Industrial supplies & services , New Brunswick Personal Care, New Brunswick Public utilities & environment, New Brunswick Real-Estate & Insurance, New Brunswick Shopping & Specialty Stores, shopping, retail, department stores company guide of New Brunswick Sports & Recreation, New Brunswick Transportation, New Brunswick Travel & Lodging

Anonymous said...

茨城出会いカフェ栃木出会いカフェ群馬出会いカフェ千葉出会いカフェ東京出会いカフェ

Grand SEO said...

Thanks for the tip. I will give it a try.

Anonymous said...

Do you want to get the most popular sports accessories? Please click our website, we offer many kinds of fashionable Air Jordan Shoes, NFL Jerseys, MBT Shoes, Nike TN and Puma Shoes Men , especially in NFL Jerseys, you will find more than thousand kinds of jerseys in our store. We guarantee the best quality and the best service for every consumer. If you are interest in, please feel free to contact us, we will reply you as soon as possible.

li said...

replica handbags
Handbags
balenciaga replica handbags
chanel replica handbags
chloe replica handbags
fendi replica handbags
gucci replica handbags
hermes replica handbags
louis vuitton replica handbags
miumiu replica handbags
prada replica handbags
ysl replica handbags
chanel replica handbags
gucci replica handbags
louis vuitton replica handbags
balenciaga replica handbags
chanel replica handbags
chloe replica handbags
fendi replica handbags
gucci replica handbags
hermes replica handbags
louis vuitton replica handbags
miumiu replica handbags
prada replica handbags
ysl replica handbags
replica handbags
replica handbags
fake designer handbags
discount replica handbags
replica handbags
replica handbags
replica handbags
designer replica handbags
replica handbags
replica handbags
replica handbags
fake replica handbags
fake replica handbags
replica handbags
replica handbags

zero said...

cell phonesCheap cell phonesChina Cell PhonesDiscount Handbags On Sale-Beautycasa Online : - Hermes Handbags Miu Miu Handbags Fendi Handbags Prada Handbags Balenciaga Handbags Chloe Handbags Marc Jacobs Handbags Jimmy Choo Handbags Mulberry Luxury Handbags Christian Dior Handbags Hermes Wallet Gucci Handbags FREE GUCCI WALLET Difference the amount Designer Handbags Chanel Handbags Burberry Handbags Louis Vuitton Wallet ecommerce, open source, shop, online shopping

Mark said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
Alabama life insurance, Alaska life insurance, Arizona life insurance, Arkansas life insurance, Bail Bonds Referral Services life insurance, California life insurance, Colorado life insurance, Connecticut life insurance, Delaware life insurance, District Of Colombia life insurance,

Mark said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
Florida life insurance, Georgia life insurance, Hawaii life insurance, Idaho life insurance, Illinois life insurance, Indiana life insurance, Iowa life insurance, Kansas life insurance, Kentucky life insurance, Louisiana life insurance, Maine life insurance,

Mark said...

South dakota insurance agencies and brokers. South dakota car insurance, home owner insurance, health insurance, life insurance and more.
South dakota
Accident & Health Insurance
, South dakota
Aircraft & Aviation Insurance
, South dakota
Auto Insurance
, South dakota
Bail Bonds Referral Services
, South dakota
Boat & Yacht Insurance
, South dakota
Business Insurance
, South dakota
Business Interruption Insurance
, South dakota
Casualty insurance
, South dakota
Churches Insurance
, South dakota
Contents insurance
, South dakota
Corporate Insurance Adjusters
, South dakota
Credit Insurance
, South dakota
Crime insurance
, South dakota
Dental Insurance

Mark said...

South dakota insurance companies, insurance agents brokers and insurance guide. Comprehensive information about South dakota car insurance, home owner insurance, health insurance, life insurance and more.

South dakota
Disability Insurance
, South dakota
Earthquake insurance
, South dakota
Estate Planning Insurance And Legal Services
, South dakota
Farm, Crop, & Livestock Insurance
, South dakota
Financial Guaranty & Life Insurance
, South dakota
Fire Insurance
, South dakota
Fleet Insurance
, South dakota
Flood Insurance
, South dakota
Foreign Insurance
, South dakota
Franchising
, South dakota
Government Insurance Carriers
, South dakota
Group Insurance
, South dakota
Health Insurance

Yosef said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
auto
body shop
, international labour
organization
, cheap laminate
flooring
, auto tune up, labour
organizations
, cleaning flooring, transmission
repair
, union history

Yosef said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
discount
laminate flooring
, towing
services
, union local, flooring
options
, auto repair help, trade
union
, flooring sales

raoliming2 said...

louis vuitton
replica handbags
Handbags
balenciaga replica handbags
chanel replica handbags
chloe replica handbags
fendi replica handbags
gucci replica handbags
hermes replica handbags
louis vuitton replica handbags
miumiu replica handbags
prada replica handbags
ysl replica handbags
chanel replica handbags
gucci replica handbags
louis vuitton replica handbags
balenciaga replica handbags
chanel replica handbags
chloe replica handbags
fendi replica handbags
gucci replica handbags
hermes replica handbags
louis vuitton replica handbags
miumiu replica handbags
prada replica handbags
ysl replica handbags
replica handbags
replica handbags
fake designer handbags
discount replica handbags
replica handbags
replica handbags
replica handbags
designer replica handbags
replica handbags
replica handbags
replica handbags
fake replica handbags
fake replica handbags
replica handbags
replica handbags

Kendall said...

What a great post, I actually found it very thought provoking, you just never know sometimes when a golden nugget of information is going to land at your feet, thanks
Illinois
Substance Abuse Counselor
, Kansas
Substance Abuse Counselor
, Ohio
Substance Abuse Counselor
, Michigan
Substance Abuse Counselor
, Oregon
Substance Abuse Counselor

Kendall said...

A fantastic read….very literate and informative. Many thanks….what theme is this you are using and also, where is your RSS button ? wholesale
earring jewelry
, plastic
exporter
, Connecticut Substance Abuse Counseling, Illinois
Substance Abuse Counseling
, Kansas
Substance Abuse Counseling
, Ohio
Substance Abuse Counseling

kiara said...

Costa Rica Manuel Antonio Tours
Costa Rica Carara Biological
Arenal Volcano Costa Rica
Costa Rica Crocodile Aventure
Costa Rica Poas Volcano
Costa Rica Monteverde Tours
Costa Rica Sea Kayaking
Costa Rica Rafting
Costa Rica Atv Adventure
Costa Rica Tours Canopy
Costa Rica Rainforest Tour
Costa Rica Vacation Rentals
Costa Rica Groups Travel
Los Suenos Costa Rica Tours
Costa Rica Zip Line
Costa Rica Vacation Rentals
Costa Rica los Suenos Vacation Rentals
Costa Rica Wedding
Costa Rica los Suenos Fishing
Costa Rica los Suenos Condos
Costa Rica los Suenos Sportfishing
Los Suenos Resort
Costa Rica Whale Watching Tours
Costa Rica Dolphin Watching Tours
Costa Rica Dominical
Costa Rica Dominical Tours
Dominical Tours
Dominical Costa Rica Tours
Costa Rica DMC Incentive Travel

real estate software said...

Thanks for giving the clear definition about 3ware hardware RAID and Linux software RAID. This is very useful.

valentine said...

Drum Beat Machine for Beat Makers, Make your own beats. The Best Beat Making Software program, download to make your own original hip hop beats. make beats

antywong said...

The Louis Vuitton ARTSY MM with a lot of book is along with the famous figure; just we can know the charm of Louis Vuitton Damier Ebene Canvas Sac Plat. The brand always try its best to makes people life better, no matter cheap lv which role you play in the society ,the Louis Vuitton Cabas Rivington will give you the best nengha100911.

Property software said...

Property software comes in different versions to meet different needs with absolute precision

andre12323 said...

christian louboutin shoeschristian louboutin shoes
christian louboutin bootschristian louboutin boots
christian louboutin sandalschristian louboutin sandals
christian louboutin slingbackchristian louboutin slingback
christian louboutin sneakerschristian louboutin sneakers
christian louboutin wedgeschristian louboutin wedges
christian louboutin weddingchristian louboutin wedding
timberland bootstimberland boots
asics shoesasics shoes
asics running shoesasics running shoes
dsquared jeans mendsquared jeans men
dsquared shoes 2011dsquared shoes 2011
karen millen dresseskaren millen dresses
marque t shirtmarque t shirt

elizabeth bernot said...

Costa Rica Tours
Tours Costa Rica
Jaco Tours
Tours Jaco

luck said...

Nike Air Max Shoes unique style and pattern gain very popuplarity from people. This may be the most memorable Air Max generations to see, the weight of the color of the predecessor of the loan, the Nike Air Max Presto Classic weight.All of the characteristics decide thatNike Air Max 1 are durable , smooth , very good to use for the sports events.Nike Air Max zoom Kobe are identified as the plan colors , just like white , black , and so on. The populalr of Nike Air Max 2009 is mainly due to the black color is perfect and rounded material layer.

janet li said...

Hello!It is my first time visit your website.It is nice,just like our website.Yes,we are the cheapest ghd outlet.All of the Discount GHD Straighteners on sale here are high quality,and free shipping.I don't know whether you have already have a flat iron,we here have theghd uk for your choice.Australia is our main market as well.ghd australia also on sale.If you want to have a differdent hairstyle with less money,then come to our website,we have the confidence that we will bring you a nice mood everyday.Pick your beauty now,It is suitable for both men and women.Come on,your beauty is in our website waits for you.what's more,with a nice hairstyle,i think you will also need a nice christian louboutin clearance sale and Ghd Australia to match your hairstyle.Ghd Australia sincerely for your service!
At last,best flat iron thanks for sharing your articles.May you happy everyday.
ghd uk
ghd uk

Nike Mercurial Vapor Superfly said...

Ah good exciting content! Will always come to our attention. To bring you good news-works perfect! Nike Football Cleats best shoes 2011!! And Nike Mercurial Vapor Superfly.Nike soccer cleats with New nike soccer shoes or Nike mercurial soccer cleats .

Your Escort Agency said...

Your Escort Agency offers exclusive and most beautiful London escort girls of various nationalities.

London escorts said...

Bestescort4U agency provids best London escorts companionship in the UK.

Escorts London said...

Hot - Collection is a honest and confidential London escort agency which provides genuine London escorts girls for gentlemen of taste.

redhattie yu said...

I am working at Rsorder.com, which is providing cheap runescape gold ,rs items, runescape money , rs account and rs powerleveling with fast delivery.What you say is very funny and your thought rs gold is so new , which is also easy to be understood. However, i also want to say i am hope for more here. Better to remain silent and be thought a fool that to speak and remove all cheap rs gold doubt.

china tours said...

Thanks. I always enjoy reading your posts - they are always humorous and intelligent.You can learn more: China tour packages | China travel packages | China Travel Agency

Learn Chinese said...

Welcome to the leading online Mandarin school that makes you fluent in a short time -- all from the comfort of your home or office!

I am a Chinese teacher,you can learn more about Free Chinese language info: Mandarin Chinese teacher | Online Chinese teachers | Native chinese teacher

Retrovit ID said...

Thanks for review, it was excellent and very informative.
thank you :)

Paul Wolf said...

Well, it was a wonderful and constructive post and I am glad to read this posting. Thanks for sharing it. free software download

Williams jack said...

I really enjoyed the posting you have share in this article. free software download full version

jaring futsal said...

Nice article, thanks for the information. It's very complete information. I will bookmark for next reference
jaring futsal | jaring golf | jaring pengaman proyek |
jaring pengaman bangunan | jaring pengaman gedung
http://www.jual-jaring.blogspot.com/
http://www.agen-jaring.blogspot.com/
http://www.pancasamudera-safetynet.blogspot.com/
http://www.toko-jaring.blogspot.com/
http://www.pusat-jaring.blogspot.com/
http://jualjaringpengaman.blogspot.com/
https://pancasamudera.wordpress.com/
https://pasangjaringfutsal.wordpress.com/
https://jualtambangmurah.wordpress.com/
https://tokojaring.wordpress.com/
https://jualjaringfutsal.wordpress.com/
https://jaringfutsal.wordpress.com/

baixar facebook gratis said...

Wonderful, very useful information for me, thank you for sharing
----
mobogenie and apply baixar facebook movel | descargar mobogenie gratis