What is up with the CentOS project

Some people may have noticed the open letter to Lance Davis that has appeared on the CentOS website (www.centos.org) or the CentOS mailing list (http://lists.centos.org/pipermail/centos/2009-July/079767.html). Some people may also noticed that the project was not running as it should lately. Since it is in the open now I can talk about all this. These are my personal opinions about this issue.

I only joined the project a little more then two years ago, so I'm was not personally involved in the start of the CentOS project. Lance is one of the co-founders of the project. I really would like to thank Lance, and the other co-founders, for their work and getting this great project started. But from the time I got involved in the project things have gone from bad to worse with Lance.

Now, I don't know him or what is going on at his end. So I'm only going to report facts here and leave the interpretation of these to you, dear reader.

The first issue is that Lance basically is not actively involved in the project anymore. He attends meetings irregularly. If he was present and was asked certain things, he usually said that he need to look that up or he will do that later. But that never happened. And next to the meetings we do not see or hear from him at all.

The second issue is that Lance has registered the centos.org domain using one of his companies and that the rest of the team has no control of it. He is the only one who can make changes to it. It is not registered to he project itself. He recently even put a anonymiser service on the domain, meaning that people can no longer see the real owner of the domain.

The third issue is that Lance is the only one who has access to the PayPal and Google AdSense accounts that are used on the CentOS websites. This basically means that all money that comes in through those channels went directly to him, not to the project. We again have no control over it.
We repeatably asked Lance for a overview of the finances of the project, but he never showed that to the rest. We have no idea how much money flowed into the project.

These three issues basically means that the project depends to much on one person and that a lot of things are invisible for the project and the community. And this is unacceptable. We need to be transparent and Lance is preventing this. We asked repeatably to correct these things in private. But he never did.

So we, the rest of the team, have said that this was enough. That is why the public letter was made and also why we removed, temporarily, the Google ads and the link to PayPal from the CentOS website.

We still like to make things right and give Lance the change to correct all this and open up. If not we will continue without him and get the project back on track. And yes, this might potentially mean that we loose the centos.org domain and all the money already received by the project through the ads and PayPal. This is also what I regret the most, that money that people have donated, thinking they are helping the project, flowed to 1 person. I sincerely apologize to everyone for this.

The open letter and thoughts of other team members can be found on the following none centos.org sites :

http://www.karan.org/blog/index.php/2009/07/30/open-letter-to-lance-davis
http://orcorc.blogspot.com/2009/07/sadly-open-letter-to-lance-davis.html
http://lestighaniker.de/2009/07/30#open-letter-to-lance-davis

And I will also reproduce it here for completeness:

July 30, 2009 04:39 UTC

This is an Open Letter to Lance Davis from fellow CentOS Developers

It is regrettable that we are forced to send this letter but we are left with no other options. For some time now we have been attempting to resolve these problems:

You seem to have crawled into a hole ... and this is not acceptable.

You have long promised a statement of CentOS project funds; to this date this has not appeared.

You hold sole control of the centos.org domain with no deputy; this is not proper.

You have, it seems, sole 'Founders' rights in the IRC channels with no deputy ; this is not proper.

When I (Russ) try to call the phone numbers for UK Linux, and for you individually, I get a telco intercept 'Lines are temporarily busy' for the last two weeks. Finally yesterday, a voicemail in your voice picked up, and I left a message urgently requesting a reply. Karanbir also reports calling and leaving messages without your reply.

Please do not kill CentOS through your fear of shared management of the project.

Clearly the project dies if all the developers walk away.

Please contact me, or any other signer of this letter at once, to arrange for the required information to keep the project alive at the 'centos.org' domain.

Sincerely,

Russ Herrold
Ralph Angenendt
Karanbir Singh
Jim Perrin
Donavan Nelson
Tim Verhoeven
Tru Huynh
Johnny Hughes

CentOS 5.2 - release update

A short update on the release of CentOS 5.2. We are currently in the progress of doing QA testing. All packages have been build. The current plan is to be able to finish all QA test this week so we might be able to release 5.2 next weekend or in the days after it.

As always, these are just indications. It will be ready when it is ready. But we are trying our best and we know that you are waiting for it :-)

I'll post another update in a couple of days.

CentOS 5.2

Before everyone starts asking questions let's get this cleared up here :-) Upstream released version 5.2 of their Enterprise Linux distribution on the 21st. So I understand that you are all wanting to now when CentOS 5.2 will be released.

Well, looking back at 5.1 we see that upsteam released 5.1 on November 7th 2007. CentOS 5.1 was released on December 2th 2007. So there was a delay of 3,5 weeks before the release of CentOS 5.1

So when can we expect CentOS 5.2 then, well using the same 3,5 weeks we end up with a date of around June 14th 2008. This is of course a estimate it can be later or earlier.


For some background information, why does it take 3,5 weeks ? First we need to remove all the logos and trademarks of Upstream. Secondly we need to build everything from source and this for both i386 and x86_64. Then everything that gets build goes past the QA team that verify that everything works as it should. From all the build packages install media will be created and these also need to be tested by the QA team. For each release a set of release notes are created and these are translated in different languages (12 for 5.1). Finally all the packages and media need to be uploaded in distributed to the mirror network so you can download it .

So this is why it takes a couple of weeks for a CentOS release to come out and remember that all this is done by volunteers and we could always use some more. So if you have some spare time and are willing to help you can make yourself known in the centos-devel mailinglist.

How to scan the SCSI bus with a 2.6 kernel

If you are playing with SCSI devices (like Fibre Channel, SAS, ..) you sometimes need to rescan the SCSI bus to add devices or to tell the kernel a device is gone. Well, this is the way to do it in CentOS with versions that have a 2.6 kernel. This means CentOS 5 and CentOS 4 (starting from update 3).

1. Find what's the host number for the HBA:
   

   ls /sys/class/fc_host/

   (You'll have something like host1 or host2, I'll refer to them as host$NUMBER from now on)
   
   
2. Ask the HBA to issue a LIP signal to rescan the FC bus:
   

   echo 1 >/sys/class/fc_host/host$NUMBER/issue_lip

3. Wait around 15 seconds for the LIP command to have effect
   
   
4. Ask Linux to rescan the SCSI devices on that HBA:
   

   echo - - - >/sys/class/scsi_host/host$NUMBER/scan

   The wildcards "- - -" mean to look at every channel, every target, every lun.
   

That's it. You can look for log messages at "dmesg" to see if it's working, and you can look at /proc/scsi/scsi to see if the devices are there. In CentOS 5 there is also the "lsscsi" command that will show you the know devices plus the device entries that point to those devices (very usefull).

For more information about how this works see the the upstream release notes for 4.3.

Anaconda, Kickstart & iSCSI

The previous days I've been trying to get Anaconda to install a Xen domU with it's harddisk being a iSCSI target. And this all automated using kickstart. It is working now, but Anaconda needed a bit of help.

First a quick overview of the seutp. I have set of servers that will host all the virtual machines. The storage comes from 2 servers that have 12 disks in a RAID array (see some of the other articles on this blog). I use LVM to carve up this storage and export each logical volume as a iSCSI target for the different virtual machines.

Since I would like to be able to use the Xen migration feature to move virtual machines between the servers, I basically have 3 options to have a shared storage so the migration works. The first one is to use a cluster/distributed filesystem that is present on all the servers that contain image files of the different virtual machines. The second one is to use CLVM that makes logical volumes available to a set of machines. And the third one is to let the virtual machine import the disk directly. It's a bit to much to discuss the merit of the 3 choices here. Maybe in another post.
Anyway, the choice was made to use the last option. Our virtual machine would act itself as a iSCSI initiator and get a disk to install on that way.

If you read the CentOS 5 docs about kickstart (http://www.centos.org/docs/5/html/5.1/Installation_Guide/s1-kickstart2-options.html) you can see that there are options for iSCSI. So this was supposed to be easy. A first test indicates that it all worked, so that looked good. Now we had some requirements. To prevent the different virtual machines from touching the wrong iSCSI targets we using a username and password on each target.

The documentation indicates that Anaconda should support username and password during the import of the iSCSI targets. But testing indicates that this does not work. Then I looked at the code and see nothing that indicates that it should work.

A second issue that we are seeing is that we have quickly have a lot of targets on our storage server and Anaconda tries to do a login to all targets that it discovers. This slows down the installation process a lot.

So it was time to refresh my Python skills and do some Anaconda hacking. BTW, a good reference for this is the Anaconda wiki page. Using the updates support you can easily use modified version of the anaconda code during a installation. And after some testing but problems are now solved. Anaconda will now use the --user, --password and --target options if they are present.

I've created 2 entries in the upstream bugzilla about this with the patches containing my 402431 and 418781. I have also attached a updates.img to bug 418781. This can be used to start a install with the patches I've made but without the need to recreate all the installation images. See the Anaconda wiki or the docs that come with Anaconda on how to use them (it depends on how you actual do the install).

CentOS 5.1 Released

For those of you that have not been behind there computer this weekend. CentOS 5.1 for the i386 and x86_64 platforms has been released into the wild. As part of the QA team I can say that it looks like a solid release (except the autofs issue, but that is already fixed with a new kernel in the updates repo). Especially Xen looks like it is a lot more stable then the 5.0 version was.

Anyway you can find the official announcement here : http://lists.centos.org/pipermail/centos-announce/2007-December/014476.html.
And the release notes (considered essential reading) are here : http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/.

Enjoy 5.1 and thanks again to all the people involved to make it happen.

How to test and monitor disk performance

In my RAID performance article I've given a lot of performance numbers, but it is indeed a good question on how to exactly measure disk performance. To do this you need 2 tools. One that does the actual benchmark and one that measure the different system parameters so you can know the impact of the benchmark.

For the benchmark tool itself I usually use 3 tools. The first one is good old dd. The problem with dd is that it can only do 1 type of benchmark (sequential read and write), but it can to this to and from many sources and that is its strength.
So how do you use dd for a disk benchmark. Well usually like this :

dd if=/dev/zero of=/dev/sdb bs=64k count=100k
dd of=/dev/null if=/dev/sdb bs=64k count=100k

First a word of warning. Using dd like this will mean that any data on /dev/sdb (including filesystems) will be overwritten. The dd command does a write test and the second dd command does a read test. The write test reads from /dev/zero and writes to /dev/sdb, it uses a blocksize of 64k and it reads 102400 blocks. The read test reads from /dev/sdb and writes to /dev/null, it uses the same amount and size of blocks like the read test.
So in this case we are testing /dev/sdb, but this can be any blockdevice. Like /dev/md1 (a software RAID device) or /dev/VolGroup00/Logvol01 (a LVM logical volume.
The amount of blocks you need to read or write to have a valid is very simple. Make sure you process at least twice the amount of memory you have in your system. Why ? Because the Linux kernel does caching you need to make sure you request more data then the size of the cache to make sure you are actually testing the disk and not the Linux kernel caching.
And then the blocksize remains. Well, this depends on the type of usage you will do on the disks or RAID arrays. There are no real fixed rules for this. But generally speaking when you have a lot of small files take a small blocksize (4 to 16k), for standard fileserving and databases take a medium blocksize (64 to 128k) and for large files or applications that do a lot of sequential I/O use a large blocksize (256k and larger).

One final note on dd, since CentOS 5 dd will, when it has finished a test, tell you the speed if obtained. But on older versions of CentOS you will need to calculate this yourself. Another option there is to use ddrescue, which is a tool similar to dd but also provides the bandwidth used in its output. You can find ddrescue for CentOS in Dag's RPMForge repository.

As I said, dd only does sequential I/O but since it does not need a filesystem it is very useful to get a baseline/indication/magnitude/... for the performance of a disk. The next step is to put a filesystem on the device being tested. Then the second tool I use to test performance is iozone. You can also find prebuild rpms for CentOS in Dag's RPMForge repository.

I usually use iozone like this

iozone -e -i0 -i1 -i2 -+n -r 16k -s8g -t1

Iozone has a look of options (that is part why I like it). If you do iozone -h you will get the complete list. I suggest you go over them when you start testing so you know all your options. But the example above shows enough to get started. The -e option adds the time of the flush operation to the results and the -+n disables the retest feature. The different -i options indicate the tests done by iozone, -i0 is the sequential write test (always needed), -i1 is sequential read and -i0 is random read and write. Another interresting test is the mixed read/write (-i8). The 3 remaining options are : -r 16k is the record (or block) size used, -s8g is the size of the data tested (8 GB in this case, remember the 2x RAM rule here) and -t1 is the amount of threads performing the test (in this case 1).

Depepding in the application you are planning to use you try to use iozone in a way that it mimics as close as possible the IO patterns of your real application. This means playing with the -i (the kind of test), the -r (block size) and -t (number of threads) options.

Other nice features of iozone are the autotest. Here iozone can iterate over different tests using different parameters for file size and record size. Also iozone can output its results in a Excell file, so you later make nice graphs of the tests.

The final test that I perform is a bit less scientific, but still a good indication of performance of a disk or a RAID array. The test is a copy of a directory that contains about 1GB of data. This data is stored in different subdirectories (I think it is about 5 levels deep) and all these subdirs contains either a couple of big files and for the rest a lot of small files.
This means that a copy of this directory involves a lot of metadata activity, and a mix of sequential I/O (the big files) and something that resembles random or mixed I/O (the small files). And the result is usually a big difference compared to the sequential speeds obtained with dd or iozone.

Now that we have all these benchmarks we still need a tool to monitor the system itself and see what is going on. The tool I always use for this is dstat. It is written and maintained by Dag of the RPMForge repo (and of course it also contains CentOS rpms of dstat). The nice thing about dstat is that is has different monitoring modules (and you can also write your own), its output formatting is really good and it can output the results in CSV files so you can also process the results in a spreadsheet. This is a example on how to use dstat :

dstat -c -m -y -n -N eth2,eth3 -d -D sda -i -I 98 3
----total-cpu-usage---- ------memory-usage----- ---system-- --net/eth2- --dsk/sda-- inter
usr sys idl wai hiq siq| used buff cach free| int csw | recv send| read writ| 98
0 0 100 0 0 0| 190M 195M 924M 2644M|1029 92 | 0 0 | 793B 4996B| 14
0 0 99 1 0 0| 190M 195M 924M 2644M|1008 41 | 21B 0 | 0 23k| 1

All the diferent output are different output modules and they get display in the order you have put them on the commandline. With the (for example) -n -N structure you can specifiy for which devices you would like to see output. In this case the -n refers to network-card statistics and with -N I specified I would like to see the output for eth2 and eth3. A similar systems is used for the disk statistics (-d and -D) and number of interrupts (-i and -I).

To get the list of extra modules available next to the standard onces do this :

dstat -M list
/usr/share/dstat:
app, battery, cpufreq, dbus, freespace, gpfs, gpfsop, nfs3, nfs3op, nfsd3, nfsd3op, postfix, rpc, rpcd, sendmail, thermal, utmp, vmkhba, vmkint, vzcpu, vzubc, wifi,

For more information about all the different dstat options do dstat -h or refer to the dstat man-page.