I recently used mdadm to setup RAID6 on my GNU/Linux file servers. The following is a tutorial on the commands I ran on debian 9 stretch to accomplish this task.
Prerequisites
You will need to install the parted, mdadm, and xfsprogs packages. GNU parted is used to create partition tables. mdadm is used to create and maintain multi devices. The xfsprogs package contains several xfs related utilities including mkfs.xfs and xfs_info.
$ sudo apt-get install parted mdadm xfsprogs
Create Partition Table
The first step in preparing your disk drives is to create the GPT partition table.
$ sudo parted /dev/sdb mklabel gpt
Create Partition
Next you need to create the partitions. While this step is optional since the RAID array can be built from devices, it is recommended that you use partitions so that down the road you have the option of replacing a drive with one of equal or greater capacity. For my purposes, I just created a single partition that consumes all of the available space.
The “-a optimal” option tells parted to use optimal alignment.
$ sudo parted -a optimal /dev/sdb mkpart primary 0% 100%
Enable the RAID Flag
Next you need to set the raid flag on the partition.
$ sudo parted /dev/sdb set 1 raid on
Verify the Configuration
You can verify that the Partition Table is type gpt, the primary partiton has been created with the appropriate size and that the raid flag is set on that partition.
$ sudo parted /dev/sdb print Model: ATA HGST HDN724040AL (scsi) Disk /dev/sdb: 4001GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 4001GB 4001GB primary raid
Create the Multi Device
After you have completed the previous steps on all disks, the next step is to create the multi device (md). In my case, I am creating a RAID6 multi device from 7 partitions.
$ sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=6 /dev/sd{b,c,d,e,f,g,h}1
The multi device is created, but it will not be preserved on reboot. To preserve the configuration on reboot we need to append to the mdadm configuration file. You may also need to update initramfs and grub to ensure the multi device is automatically assembled on reboot.
$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf $ sudo update-initramfs -u $ sudo update-grub
Create the XFS Filesystem
Now that we have a multi device, we can create the filesystem. I recommend using xfs if the effective capacity of your multi device may eventually grow to be more than 16TB of hard drive space.
The “-d” option allows us to specify one or more disk parameters.
The sunit disk parameter stands for Stripe Unit. It is optimal when assigned to the Stripe Size in bytes divided by 512. In my case the Stripe Size is 512KB, and thus the optimal Stripe Unit is 1024.
The swidth disk parameter stands for Stripe Width. It is defined by the number of non-parity disks times Stripe Unit. In my case, I am using RAID6 with a total of 7 disks. Since RAID6 requires 2 parity disks, that leaves me with 5 non-parity disks. Thus the optimal Stripe Width is 5120 (5 non-parity disks times the 1024 Stripe Unit).
$ sudo mkfs.xfs -d sunit=1024,swidth=5120 /dev/md0
You can verify the sunit and swidth settings aftewards, but it can be tricky because the units are different. With mkfs.xfs they are defined as a multiple of 512-bytes. With xfs_info, they are defined as a multiples of the block size (bsize). In this case bsize is 4096. So the values appear as 1/8th of the value we specified in the mkfs.xfs command.
$ sudo xfs_info /dev/md0 meta-data=/dev/md0 isize=256 agcount=32, agsize=152612736 blks = sectsz=4096 attr=2, projid32bit=1 = crc=0 finobt=0 spinodes=0 rmapbt=0 = reflink=0 data = bsize=4096 blocks=4883607040, imaxpct=5 = sunit=128 swidth=640 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count
Alternative: Create the ext4 Filesystem
If you are certain you will never exceed the 16TB limit, then ext4 is a reasonable alternative choice for a filesystem. Note that you should only run this command if you are using ext4 instead of xfs.
$ sudo mkfs.ext4 -F /dev/md0
Mount the Multi Device
A mount point is simply the directory to which the device is mounted. So we just need to make sure that directory exists and mount the multi device to the newly created mount point.
$ sudo mkdir -p /mnt/md0 $ sudo mount /dev/md0 /mnt/md0
Next we update fstab to preserve the changes on reboot.
$ echo '/dev/md0 /mnt/md0 xfs defaults,nofail 0 0' | sudo tee -a /etc/fstab
Finally df can be used to verify the disk is mounted and displays the usage.
$ df -h -t xfs Filesystem Size Used Avail Use% Mounted on /dev/md0 19T 0T 0T 0% /mnt/md0
Verify the Setup
You can verify the setup with lsblk. It shows the disk device and capacity, the raid partition, and multi device and its capacity, filesystem type, raid level, and mount point.
$ lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT /dev/sd{b,c,d,e,f,g,h}
NAME SIZE FSTYPE TYPE MOUNTPOINT
sdb 3.7T disk
└─sdb1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sdc 3.7T disk
└─sdc1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sdd 3.7T disk
└─sdd1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sde 3.7T disk
└─sde1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sdf 3.7T disk
└─sdf1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sdg 3.7T disk
└─sdg1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
sdh 3.7T disk
└─sdh1 3.7T linux_raid_member part
└─md0 18.2T xfs raid6 /mnt/md0
You can view the state of multi devices using the proc virtual filesystem.
$ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md0 : active raid6 sdc1[4] sdg1[3] sde1[2] sdd1[7] sdf1[6] sdh1[0] sdb1[5] 19534428160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU] bitmap: 0/30 pages [0KB], 65536KB chunk unused devices: <none>
Lastly, you can view detailed information about a multi device using mdadm.
$ sudo mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon May 22 21:28:46 2017 Raid Level : raid6 Array Size : 19534428160 (18629.48 GiB 20003.25 GB) Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB) Raid Devices : 7 Total Devices : 7 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Sep 8 20:07:07 2017 State : clean Active Devices : 7 Working Devices : 7 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : euclid:0 (local to host euclid) UUID : a7faccce:58914d5e:b26cce7e:795b773b Events : 48719 Number Major Minor RaidDevice State 0 8 113 0 active sync /dev/sdh1 6 8 81 1 active sync /dev/sdf1 7 8 49 2 active sync /dev/sdd1 3 8 97 3 active sync /dev/sdg1 4 8 1 4 active sync /dev/sdc1 5 8 17 5 active sync /dev/sdb1 2 8 65 6 active sync /dev/sde1