From: <Chris.Phillips_at_email.protected>
Date: Fri, 12 Nov 2004 15:18:47 +1100 (EST)

LBA/Huygens software RAID "howto"

For the LBA support of Huygens and some miscellaneous LBA disk based
tests, we will be using a set of normal IDE disk drives configured
using software raid. Software raid used the operating system to
combine normal disk drives into a single logical volume as compared
with hardware raid which used special disk adapter devices with custom
electronics. The advantage of software raid is that the disks can be
moved from one system to another with a much greater compatibility.

Setup:

Careful thought must be used when adding the drives to the system with
respect to PCI usage. Because the VSIB cards are PCI based they
consume PCI bandwidth transferring data from the card to main memory. If
this data was transfered to a PCI based network card or PCI IDE
adapter, the potential PCI bandwidth will be exceeded (for 512 Mbps
recording rates). To get around this, the system must then

 - Use the mainboard IDE controllers, as long as the motherboard
   design does not have these controllers on the PCI bus (modern
   motherboards should be OK).
 - Use a system with independent PCI bus and an IDE adapter.

Drive installation:
 - Generally you want to use 4 IDE drives to reliably record at 512 Mbps,
   this probably means using a master and slave on each controller which is
   not ideal as they compete for ATA bandwidth. It seems to work OK.
 - Make sure the jumper setting are correct. I find that cable select works
   very well.
 - Do *not* have a cdrom sharing the IDE cable with a data disk this seems
   to limit available DMA modes.\
 - For new fast disks (ATA100/133, maybe ATA66) you *must* use
   80 wire/40 pin IDE connectors. These have very second wire grounded.
   You can tell them from normal IDE cables as the wires are thinner.

General software setup:

 - If using the Promise Ultra133 Tx2 IDE adapter, the Linux kernel needs
   a special module compiled in. This does not seem to be activated in
   a standard kernel config. This card uses the PDC20269 chipset. It can
   be activated using the CONFIG_BLK_DEV_PDC202XX_NEW option
   in the kernel setup (**CHECK**). The module *must* be compiled into
   the kernel, not used as a module as the module doesn't work.

 - Some kernels do not activate DMA transfer by default. For each drive
   this can be checked using the hdparm command (as root). The
   following drive *is* using DMA access.

teem-mm:/home/vlbi# hdparm /dev/hda

/dev/hda:
 multcount = 16 (on)
 I/O support = 1 (32-bit)
 unmaskirq = 1 (on)
 using_dma = 1 (on)
 keepsettings = 0 (off)
 nowerr = 0 (off)
 readonly = 0 (off)
 readahead = 8 (on)
 geometry = 48641/255/63, sectors = 781422768, start = 0
 busstate = 1 (on)

 - ATA133 does not seem to work reliably and can cause major DMA error
  (including kernel panics). This may be trying to use 2 drives per
  controller or over long IDE cables. DO not try and use it. The command
  hdparm -i will tell the available DMA modes. hdparm -X 69 will set the
  drive to ATA100 (udma5). If the drive does not support ata133
  (udma6) the defalt (fastest available) is OK. E.g.:

teem-mm:/home/vlbi# hdparm -i /dev/hda

/dev/hda:

 Model=HDS724040KLAT80, FwRev=KFAOA32A, SerialNo=KRFA01RAG4SWTB
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=52
 BuffType=DualPortCache, BuffSize=7919kB, MaxMultSect=16, MultSect=16
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes: pio0 pio1 pio2 pio3 pio4
 DMA modes: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6
 AdvancedPM=yes: disabled (255) WriteCache=enabled
 Drive Supports : slowATA-2 ATA-3 ATA-4 ATA-5 ATA-6 ATA-7

  This drives supports ATA133 (udma6) but as it does not work
  reliably, it has been set to udma5 (see below).

 - hdparm can also turn on miscellaneous parameters which potentially help
   disk speed or reduce CPU overhead. These include
    -c1 Enable 32 bit i/o across the PCI bus
    -d1 Enable DMA access
    -u1 Set interrupt unmasking. Improves kernels responsiveness
    -m16 Set multiple sector i/o for drive

   An example hdparm line (for each physical disk) might be:

>> hdparm -c1 -d1 -u1 -m16 -X69 /dev/hda

 - To enable hdparm to be run automatically each reboot:
     Debian: Edit /etc/init.d/hwtools
     SuSE: See Appendix
     Fedora: ?????

 - If you get DMA errors while the system is booting use the kernel
   boottime option "ide=nodma". If you still have problems, boot with
   a rescue/live disk and check that the boot scripts are not turning DMA
   on (from experience).

Software RAID setup.

- Make sure each disk to become part of the software raid set is partitioned.
  The best solution is to have one large partition per software raid disk,
  each which is the same size. If you need to have a boot partition on
  the same disks (not recommended), the following recipe will need to
  be modified. The Linux softraid does seem to cope with different
  size disks/partitions, but it might just use the smallest disks size.
  To partition each disk use cfdisk (read manpage) or fdisk. An
  example usage would be

>fdisk /dev/hda
>> p To print the current partition (assumed empty)
>> n To add a new partition
>> p Primary partition
>> 1 First partition
>> <ret> Default start cylinder (first)
>> <ret> Default last cylinder (maximum)
>> w Write the partition table

>reboot

teem-mm:~# fdisk /dev/hda

The number of cylinders for this disk is set to 48641.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/hda: 255 heads, 63 sectors, 48641 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot Start End Blocks Id System

Command (m for help): n
Command action
   e extended
   p primary partition (1-4) p
Partition number (1-4): 1
First cylinder (1-48641, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-48641, default 48641):
Using default value 48641

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

- The software raid is controlled by the file /etc/raidtab.
  An example setup is

raiddev /dev/md0
     raid-level 0
     nr-raid-disks 4
     persistent-superblock 1
     chunk-size 16

     device /dev/hde1
     raid-disk 0
     device /dev/hdf1
     raid-disk 1
     device /dev/hdg1
     raid-disk 2
     device /dev/hdh1
     raid-disk 3

Use raid level "0". This strips data across all drives. Sofrware raid
5 has built in redundency, but it dog slow. Create such an entry for each
softraid disk you want to use.

- To create the softraid disk use:
>> mkraid /dev/md0 -f
  This will give lots of warnings. Read and run as suggested.

- The commands
>> raidstart /dev/md0 start the softraid device
>> raidstop /dev/md0 stops the softraid device.
  You should normally never need to use them unless you need to recreate a disk
  (Raidstart is run automatically when you create the disk).

- You need to create a filesystem on the new softraid disk. Ext2 seems to
  be the fastest.
>> mke2fs -m 0 -T largefile4 /dev/md0
  -m 0 means don't reserve space for the root user when the disk gets full.

 - Add a mount option to /etc/fstab something like:

/dev/md1 /data/disk1 ext2 user,noauto 0 2

   This will *not* mount the disk when the system boots, but allows any
   user to request the disks is mounted. This is best for possibly
   removable disks. As any user, use the command:
>> mount /data/disk1 (or whatever you called it)

 - Mount the new disk and change user permission (as root)
>> mount /data/disk1
>> chown vlbi:vlbi /data/disk1

The disk should be ready to use.

** Trouble shooting

- I don't seem to be able to see one or more disks:
    - Check IDE connectors are in place correctly
    - Check you remembered to plug the power connector in
    - Is the drive jumpered correct? (see above)

- My drive is running very slowly
    - Check with hdparm that DMA is enabled
    - Check with hdparm -i that ATA133 (udma6) is *not* enabled

- I am getting lots of DMA errors on the drives
    - Check with hdparm -i that ATA133 (udma6) is *not* enabled

Appendix:

SuSE hdparm at boottime (Thanks to Craig West):

login as root
>yast

-> goto hardware menu
   -> goto IDE DMA menu
      For each IDE drive press Alt+D
      select UltraDMA/100
      hit tab 4 times and scroll down to next drive
      repeat for all IDE drives

      Press Alt+F
      Press Alt+Q

You drives should now be set, you may wish to reboot the computer and you
can confirm the setting by typing this:

hdparm -i /dev/hde (repeat for hdf, hdg and hdh)

You should see a UDMA line (3rd last line) that has a * next to udma5
Received on 2004-11-12 15:19:07