Configuring SAN access with Linux multipath

From Science IT

Jump to: navigation, search

Here's what you need to do on Debian etch or Ubuntu feisty to get the SAN working properly:

First, install the extra packages you'll need:

$ apt-get install firmware-qlogic multipath-tools multipath-tools-initramfs sysfsutils

Then create the multipathd configuration file:

$ cat <<EOF > /etc/multipath.conf
defaults {
       user_friendly_names     yes
       path_grouping_policy    group_by_serial
       path_checker            tur

blacklist {
       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
       devnode "^sda"
       # blacklist any other locally attached block devices here

Next, you want to stop LVM from trying to rescan every locally attached block device looking for physical volumes because it will create havoc with the controller failover and generally waste a lot of time and also output loads of garbage about duplicate PV's. Open /etc/lvm/lvm.conf in your favourite editor (vim, right?) and find the filter section and change the regexps to include the locally attached SCSI devices but not the FC connected ones. Here's an example that allows LVM to scan /dev/sda (the local hard disk) but ignore the others:

filter = [ "r|/dev/cdrom|", "a|/dev/sda|", "r|/dev/sd.*|" ]

Note that you might want to also specifically include /dev/mapper/mpath[0-9]+ if you're going to run CLVM.

Next, update the initrd images so that the firmware and device mapper modules are present:

$ update-initramfs -u

Now you can reboot the machine. You don't have to, but it's probably a good idea to make sure your changes are reboot safe.

Once you reboot it should load the qlogic SCSI drivers, and then fire up multipath on it's own. You can see that multipath is working by doing the following:

$ multipath -v2 -l
mpath1 (3600a0b8000167eba0000173245e35ce4) dm-4 IBM,1722-600
\_ round-robin 0 [prio=0][active]
\_ 4:0:0:2 sdc 8:32  [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 4:0:1:2 sde 8:64  [active][undef]
mpath0 (3600a0b8000167c50000016cd45e36385) dm-3 IBM,1722-600
\_ round-robin 0 [prio=0][active]
\_ 4:0:0:1 sdb 8:16  [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 4:0:1:1 sdd 8:48  [active][undef]

Note that only one device on each LUN is specified as "active" and the other is "enabled". This shows us that the multipath is using the correct policy. If you see more than one path selector as "active" then you're going to have major problems with our SAN (think iris).

$ ls -l  /dev/mapper/mpath*
brw-rw---- 1 root disk 254,  3 2007-09-17 13:55 mpath0
brw-rw---- 1 root disk 254,  4 2007-09-17 13:45 mpath1

These are your mountpoints. You get one for each LUN.

You also want to go back and change the default timeouts for the FC bus, so that when a path fails it flips over before anything notices (in my case the OCFS2 heartbeat was timing out and panicing the machine). Note that the kernel always adds 5 seconds, so setting it to "1" is actually setting it to "6".

$ ls /sys/class/fc_remote_ports/*/dev_loss_tmo | sed 's/\/sys\///g' | awk '{print $0 " = 1"}' > /etc/sysfs.conf
$ /etc/init.d/sysfsutils restart
This page was last modified on 18 September 2007, at 10:41. This page has been accessed 5,565 times.

Personal tools