Storage‎ > ‎Storage-disk‎ > ‎

Device-Mapper Multipath configuration on linux

Thank you for visiting this page, this page has been update in another link Device mapper multipath configuration on Linux


It has been a lot changed since I started using it many years ago, so I don't want to have it too particular on certain platform, rather, I'd like to share something in general, so once you know what and how multipath works, it's easier for you to manage multipath configuration versions after versions.

In principle, Device mapper multipathing (DM-Multipath) is a tool that allows you to configure multiple I/O paths between server and storage arrays into a single device, mostly used in external storage systems(SAN, iSCSI, tape etc..). Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths.

However, there are many different types of external devices, as well hosts and HBAs, so there has to have functions to manage them, this is also one of the reason it keeps changing in configuration.

Ok, let get started for real meat. I'd like to skip installation part for you can easily find somewhere else.

1. multipath configuration
    on SL6, there is a tool called /sbin/mpathconf, it makes things easier, but I still like to get started with configuration file.
    /etc/multipath.conf and /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults(the case version 0.4.9 )
    .default file has all default settings, and all parameters explicitly set in /etc/multipath.conf will overwrite the default value.
    There are only four sessions in the .conf file, let's start with devices one.
    devices
     This session doesn't show in /etc/multipath.conf, but in conf.default. It can not be simply ignored, for multipathd uses parameters settings the device listed in conf.default. My experience is that find your device setting in .conf.default, then put it to default session.
     To get Vendor and product info, use command
#  /sbin/scsi_id --page=0x80 --whitelisted --device=/dev/sdb
SIBM     1818      FAStTSV14408879     
#  /sbin/scsi_id --page=0x80 --whitelisted --device=/dev/sda
SLSI     Logical Volume   394949227760212329
    sdb device from Vendor IBM and product is 1818, while sda from Vendor LSI and product description is Logical Volume
    So in .conf.default file, you can easily find devices session
#       device {
#               vendor "IBM"
#               product "^1818"
#               product_blacklist "Universal Xport"
#               path_grouping_policy group_by_prio
#               getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
#               path_selector "round-robin 0"
#               path_checker rdac
#               features "0"
#               hardware_handler "1 rdac"
#               prio rdac
#               failback immediate
#               rr_weight uniform
#               no_path_retry queue
#               rr_min_io 1000
#               rr_min_io_rq 1
#       }
     Then, put the settings to default sessions, easier to have a check later, or you can put it into devices session, most of time they do the best for the device.

    defaults
     Default session covers parameters not set in device session, of couse, also for unknown devices.
defaults {
    udev_dir                /dev
    polling_interval        15
    path_selector "round-robin 0"
    path_grouping_policy group_by_prio
    path_checker rdac
    getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
    rr_min_io 1000
    rr_min_io_rq 1
    rr_weight uniform
    failback immediate
    no_path_retry 3
    user_friendly_names     yes
    features "0"
    prio rdac
    max_fds                 4096
    find_multipaths yes
    log_checker_err always
}
          You don't have to go through every parameter, but most of time, you need to pay more attention for parameters below
polling_interval
interval between two path checks in seconds For properly functioning paths, the interval between checks will gradually increase to (4 * polling_interval); default is 5
user_friendly_names
If set to yes , using the bindings file /etc/multipath/bindings to assign a persistent and unique alias to the multipath, in the form of mpath<n>. If set to no use the WWID as the alias. In either case this be will be overriden by any specific aliases in the multipaths section. Default is no
max_fds
Specify the maximum number of file descriptors that can be opened by multipath and multipathd. This is equivalent to ulimit -n. A value of max will set this to the system limit from /proc/sys/fs/nr_open. If this is not set, the maximum number of open fds is taken from the calling process. It is usually 1024. To be safe, this should be set to the maximum number of paths plus 32, if that number is greated than 1024.
find_multipaths
The Red Hat Enterprise Linux 6 release provides a new mode for setting up multipath devices, which you set with the find_multipaths configuration file parameter. In previous releases of Red Hat Enterprise Linux, multipath always tried to create a multipath device for every path that was not explicitly blacklisted. In Red Hat Enterprise Linux 6, however, if the find_multipaths configuration parameter is set to yes, then multipath will create a device only if one of three conditions are met:
  • There are at least two non-blacklisted paths with the same WWID.
  • The user manually forces the device creation, by specifying a device with the multipath command.
  • A path has the same WWID as a multipath device that was previously created (even if that multipath device does not currently exist). For instructions on the procedure to follow if you have previously created multipath devices when the find_multipaths parameter was not set, see
features "0" and no_path_retry

Specify the number of retries until disable queueing, or fail for immediate failure (no queueing), queue for never stop queueing. Default is 0.

Issues with queue_if_no_path feature(this part from RHEL portal)

If features "1 queue_if_no_path" is specified in the /etc/multipath.conf file, then any process that issues I/O will hang until one or more paths are restored. To avoid this, set the no_path_retry N parameter in the /etc/multipath.conf file (where N is the number of times the system should retry a path).
When you set the no_path_retry parameter, remove the features "1 queue_if_no_path" option from the /etc/multipath.conf file as well. If, however, you are using a multipathed device for which the features "1 queue_if_no_path" option is set as a compiled-in default, as it is for many SAN devices, you must explicitly add features "0" to override this default. You can do this by copying the existing devices section for your device from /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults into /etc/multipath.conf and editing it to suit your needs.
If you need to use the features "1 queue_if_no_path" option and you experience the issue noted here, use the dmsetup command to edit the policy at runtime for a particular LUN (that is, for which all the paths are unavailable). For example, if you want to change the policy on the multipath device mpathc from "queue_if_no_path" to "fail_if_no_path", execute the following command.

    blacklist
     Blacklist session tells multipathd which device you don't want to use multipath, for example, you local disk, usb devices etc.. Here is a part of configuration in .conf.default

## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid.  The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line.  Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.

#blacklist {
#       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
#       devnode "^hd[a-z]"
#       devnode "^dcssblk[0-9]*"
#       device {
#               vendor "DGC"
#               product "LUNZ"
#       }
    Here I have an example to blacklist local disk, good for large number of servers using an identical multipath.conf in a large SAN environment

        device {
                vendor "LSI"
                product "Logical Volume"
        }
  
    multipaths
    It works only you set user_friendly_names to yes. Define you real device name here.
multipaths {
          multipath {
                     wwid    360080e50002d26520000067a4f7a3fb8
                     alias   dcsunit01_lun0
          }
...
}
    The device /dev/mapper/dcsunit01_lun0 is the device you want to use in your application.

2. start to use multipathd
On RHEL6, there is a quick tool to get started.
# mpathconf
multipath is enabled
find_multipaths is enabled
user_friendly_names is enabled
dm_multipath module is loaded
multipathd is chkconfiged off
# mpathconf -h
usage: /sbin/mpathconf <command>

Commands:
Enable: --enable
Disable: --disable
Set user_friendly_names (Default n): --user_friendly_names <y|n>
Set find_multipaths (Default n): --find_multipaths <y|n>
Load the dm-multipath modules on enable (Default y): --with_module <y|n>
start/stop/reload multipathd (Default n): --with_multipathd <y|n>
chkconfig on/off multipathd (Default y): --with_chkconfig <y|n>

 
Or, you can just run service start multipathd to start mulltipath daemon.
# multipath -ll
dcsunit03_lun10 (360080e50002d12b4000005814f35cb40) dm-34 IBM,1818      FAStT
size=35T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 5:0:3:10 sdav 66:240  active ready running
| `- 6:0:3:10 sddp 71:112  active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 5:0:4:10 sdbh 67:176  active ready running
  `- 6:0:4:10 sdeb 128:48  active ready running
...
To see one device info
# multipath -ll dcsunit02_lun1
dcsunit02_lun1 (360080e50002d029c000005a74f7a402b) dm-13 IBM,1818      FAStT
size=35T features='1 queue_if_no_path' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 5:0:2:1  sdaa 65:160  active ready running
| `- 6:0:2:1  sdcu 70:32   active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 5:0:1:1  sdo  8:224   active ready running
  `- 6:0:1:1  sdci 69:96   active ready running
Flush device map away(device in use can't be flushed away)
# multipath -F

Create map with verbose output
# multipath -v2
...

2. debug
multipathd supports interactive mode for debug.
       -k     multipathd will enter interactive mode. From this mode, the available commands can be viewed by entering "help".  When  you  are finished entering commands, press CTRL-D to quit.

# multipathd -k
multipathd> help
multipath-tools v0.4.9 (04/04, 2009)
CLI commands reference:
 list|show paths
 list|show paths format $format
 list|show status
 list|show daemon
 list|show maps|multipaths
 list|show maps|multipaths status
 list|show maps|multipaths stats
 list|show maps|multipaths format $format
 list|show maps|multipaths topology
 list|show topology
 list|show map|multipath $map topology
 list|show config
 list|show blacklist
 list|show devices
 list|show wildcards
 add path $path
 remove|del path $path
 add map|multipath $map
 remove|del map|multipath $map
 switch|switchgroup map|multipath $map group $group
 reconfigure
 suspend map|multipath $map
 resume map|multipath $map
 resize map|multipath $map
 disablequeueing map|multipath $map
 restorequeueing map|multipath $map
 disablequeueing maps|multipaths
 restorequeueing maps|multipaths
 reinstate path $path
 fail path $path
 paths count
 forcequeueing daemon
 restorequeueing daemon
 quit|exit
 map|multipath $map getprstatus
 map|multipath $map setprstatus
 map|multipath $map unsetprstatus

The most useful part I used is to check current configuration, blacklist devices, devices not monitored etc..
multipathd> show blacklist
device node rules:
...
wwid rules:
- blacklist:
        <empty>
- exceptions:
        <empty>
device rules:
- blacklist:
        (config file rule) LSI:Logical Volume
        (default rule)     DGC:LUNZ
...

multipathd> show device
available block devices:
    ram0 devnode blacklisted, unmonitored
...
    loop7 devnode blacklisted, unmonitored
    sda
    sdb devnode whitelisted, monitored
...




Comments