Storage‎ > ‎Storage-disk‎ > ‎

mdadm -- A tool for software array in linux

Thank you for visiting this page, this page has been update in another link Mdadm a tool for software array on linux
Nowadays, most servers equiped with raid card that we do not need to create software array, but some servers still do, particularly with lots of CPU resources. For them, mdadm is the best tool that manage software array in linux.
Here is its basic syntax
       mdadm [mode] <raiddevice> [options] <component-devices>
It has has five major modes of operation. The first two modes, Create and Assemble, are used to configure and activate arrays. Manage mode is used to manipulate devices in an active array. Follow or Monitor mode allows administrators to configure event notification and actions for arrays. Build mode is used when working with legacy arrays that use an old version of the md driver. I will not cover build mode in this article. The remaining options are used for various housekeeping tasks and are not attached to a specific mode of operation, although the mdadm documentation calls these options Misc mode. Below is the details for modes, we will go through of them later.

              Assemble  the  components  of  a  previously  created array into an active array.  Components can be explicitly given or can be
              searched for.  mdadm checks that the components do form a bona fide array, and can, on request, fiddle  superblock  information
              so as to assemble a faulty array.

       Build  Build  an  array  that  doesn’t  have per-device metadata (superblocks).  For these sorts of arrays, mdadm cannot differentiate
              between initial creation and subsequent assembly of an array.  It also cannot perform any checks  that  appropriate  components
              have  been  requested.   Because of this, the Build mode should only be used together with a complete understanding of what you
              are doing.

       Create Create a new array with per-device metadata (superblocks).  Appropriate metadata is written to each device, and then the  array
              comprising  those  devices  is  activated.   A ’resync’ process is started to make sure that the array is consistent (e.g. both
              sides of a mirror contain the same data) but the content of the device is left otherwise untouched.  The array can be  used  as
              soon as it has been created.  There is no need to wait for the initial resync to finish.

       Follow or Monitor
              Monitor  one  or  more  md  devices  and act on any state changes.  This is only meaningful for RAID1, 4, 5, 6, 10 or multipath
              arrays, as only these have interesting state.  RAID0 or Linear never have missing, spare, or failed drives, so there is nothing
              to monitor.

       Grow   Grow  (or  shrink)  an  array,  or otherwise reshape it in some way.  Currently supported growth options including changing the
              active size of component devices and changing the number of active devices in Linear and RAID levels  0/1/4/5/6,  changing  the
              RAID level between 0, 1, 5, and 6, and between 0 and 10, changing the chunk size and layout for RAID 0,4,5,6, as well as adding
              or removing a write-intent bitmap.
      Incremental Assembly

              Add a single device to an appropriate array.  If the addition of the device  makes  the  array  runnable,  the  array  will  be
              started.  This provides a convenient interface to a hot-plug system.  As each device is detected, mdadm has a chance to include
              it in some array as appropriate.  Optionally, when the --fail flag is passed in we will remove the device from any active array
              instead of adding it.
               If a CONTAINER is passed to mdadm in this mode, then any arrays within that container will be assembled and started.

       Manage This is for doing things to specific components of an array such as adding new spares and removing faulty devices.

       Misc   This  is  an  ’everything else’ mode that supports operations on active arrays, operations on component devices such as erasing
              old superblocks, and information gathering operations.

              This mode does not act on a specific device or array, but rather it requests the Linux Kernel  to  activate  any  auto-detected

Creating an Array

Create (mdadm --create) mode is used to create a new array. In this example I use mdadm to create a RAID-0 at /dev/md0 made up of /dev/sdb1 and /dev/sdc1:

# mdadm --create --verbose /dev/md1 --level=0
        --raid-devices=2  /dev/sdd1 /dev/sdc1
mdadm: chunk size defaults to 64K
mdadm: array /dev/md1 started.

The --level option specifies which type of RAID to create in the same way that raidtools uses the raid-level configuration line. Valid choices are 0,1,4 and 5 for RAID-0, RAID-1, RAID-4, RAID-5 respectively. Linear (--level=linear) is also a valid choice for linear mode. The --raid-devices option works the same as the nr-raid-disks option when using /etc/raidtab and raidtools.

# mdadm -Cv /dev/md0 -l0 -n2 -c128 /dev/sdd1 /dev/sdc1

-C selects Create mode, and the -v option here to turn on verbose output. -l and -n specify the RAID level and number of member disks. Users of raidtools and /etc/raidtab can see how much easier it is to create arrays using mdadm. You can change the default chunk size (64KB) using the --chunk or -c option. In this previous example I changed the chunk size to 128KB. mdadm also supports shell expansions, so you don't have to type in the device name for every component disk if you are creating a large array. In this example, I'll create a RAID-5 with five member disks and a chunk size of 128KB:

# mdadm -Cv /dev/md0 -l5 -n5 -c128 /dev/sd{a,b,c,d,e}1
mdadm: layout defaults to left-symmetric
mdadm: array /dev/md0 started.

This example creates an array at /dev/md0 using SCSI disk partitions /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1, and /dev/sde1. Notice that I have also set the chunk size to 128 KB using the -c128 option. When creating a RAID-5, mdadm will automatically choose the left-symmetric parity algorithm, which is the best choice.

Use the --stop or -S command to stop running array:

# mdadm -S /dev/md1


It's mdadm configuration file, mdadm will function properly without the use of a configuration file, but this file is useful for keeping track of arrays and member disks. In general, a mdadm.conf file is created, and updated, after arrays are created. This is the opposite behavior of /etc/raidtab which is created prior to array construction.

Unlike the configuration file for raidtools, mdadm.conf is concise and simply lists disks and arrays. The configuration file can contain two types of lines each starting with either the DEVICE or ARRAY keyword. Whitespace separates the keyword from the configuration information. DEVICE lines specify a list of devices that are potential member disks. ARRAY lines specify device entries for arrays as well as identifier information. This information can include lists of one or more UUIDs, md device minor numbers, or a listing of member devices.

A simple mdadm.conf file might look like this:
DEVICE       /dev/sdc1 /dev/sdd1
ARRAY        /dev/md1 devices=/dev/sdc1,/dev/sdd1

In general, it's best to create an /etc/mdadm.conf file after you have created an array and update the file when new arrays are created. Without an /etc/mdadm.conf file you'd need to specify more detailed information about an array on the command in order to activate it. That means you'd have to remember which devices belonged to which arrays, and that could easily become a hassle on systems with a lot of disks. mdadm even provides an easy way to generate ARRAY lines.

#mdadm --detail --scan
ARRAY /dev/md1 metadata=0.90 UUID=a0d44a20:b41dda0a:e9c548a1:a3e98bee

If there were multiple arrays running on the system, then mdadm would generate an array line for each one.

Starting/Checking an Array

Assemble mode is used to start an array that already exists. If you created an /etc/mdadm.conf you can automatically start an array listed there with the following command:

Since using mdadm -A in this way assumes you have some prior knowledge about how arrays are arranged, it might not be useful on systems that have arrays that were created by someone else. So you may wish to examine some devices to gain a better picture about how arrays should be assembled.

The examine options (-E or --examine) allows you to print the md superblock (if present) from a block device that could be an array component.

#mdadm -E /dev/sdc1
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a0d44a20:b41dda0a:e9c548a1:a3e98bee
  Creation Time : Mon Jun 20 16:08:13 2011
     Raid Level : raid10
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
     Array Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Sun Oct 27 22:29:50 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3ac939ff - correct
         Events : 614

         Layout : near=2
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1

#mdadm --detail /dev/md1
        Version : 0.90
  Creation Time : Mon Jun 20 16:08:13 2011
     Raid Level : raid10
     Array Size : 244195904 (232.88 GiB 250.06 GB)
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sun Oct 27 22:30:05 2013
          State : clean, checking
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

   Check Status : 54% complete

           UUID : a0d44a20:b41dda0a:e9c548a1:a3e98bee
         Events : 0.614

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1

Managing Arrays

Using Manage mode you can add and remove disks to a running array. This is useful for removing failed disks, adding spare disks, or adding replacement disks. Manage mode can also be used to mark a member disk as failed. Manage mode replicates the functions of raidtools programs such as raidsetfaulty, raidhotremove, and raidhotadd.

For example, if one hard driver failed, here is steps for disk replacement.
1. check md disk status
    you can either use mdadm -E, or cat /proc/mdstat

2. fail disk partition first, repeat steps below if there are more partitions.
  #mdadm /dev/md1 --fail /dev/sdc1
  followed by remove command, or you can put them together
  #mdadm /dev/md1 --fail /dev/sdc1 --remove /dev/sdc1

3. replace physical disk and partition new disk
  #sfdisk -d /dev/sdd | sfdisk /dev/sdc

add new disk partitions to array
  #mdadm /dev/md1 --add /dev/sdc1

Monitoring Arrays

Follow, or Monitor, mode provides some of mdadm's best and most unique features. Using Follow/Monitor mode you can daemonize mdadm and configure it to send email alerts to system administrators when arrays encounter errors or fail. .

The following command will monitor /dev/md0 (polling every 300 seconds) for critical events. When a fatal error occurs, mdadm will send an email to sysadmin. You can tailor the polling interval and email address to meet your needs.

# mdadm --monitor --mail=root --delay=300 /dev/md0

You can also specify -y option to let mdadm write log into syslog.

In linux, commonly people just enable mdmonitor, which is a put mdadm to backgroup  daemon mode, it uses the following options.
OPTIONS="--monitor --scan -f --pid-file=$PIDFILE"

#chkconfig mdmonitor on
#chkconfig --list mdmonitor
mdmonitor          0:off    1:off    2:on    3:on    4:on    5:on    6:off

To test monitoring function, try
mdadm --monitor  -t /dev/md1