I recently used mdadm to setup RAID6 on my GNU\/Linux file servers. The following is a tutorial on the commands I ran on debian 9 stretch to accomplish this task.<\/p>\n
You will need to install the parted, mdadm, and xfsprogs packages. GNU parted is used to create partition tables. mdadm is used to create and maintain multi devices. The xfsprogs package contains several xfs related utilities including mkfs.xfs and xfs_info.<\/p>\n
$ sudo apt-get install\u00a0parted mdadm xfsprogs<\/pre>\nCreate Partition Table<\/h5>\n
The first step in preparing your disk drives is to create the GPT partition table.<\/p>\n
$ sudo parted \/dev\/sdb\u00a0mklabel gpt<\/pre>\nCreate Partition<\/h5>\n
Next you need to create the partitions. While this step is optional since the RAID array can be built from devices, it is recommended that you use partitions so that down the road you have the option of replacing a drive with one of equal or greater capacity. For my purposes, I just created a single partition that consumes all of the available space.<\/p>\n
The “-a optimal” option tells parted to use optimal alignment.<\/p>\n
$\u00a0sudo parted -a optimal \/dev\/sdb mkpart primary 0% 100%<\/pre>\nEnable the\u00a0RAID Flag<\/h5>\n
Next you need to set the raid flag on the partition.<\/p>\n
$ sudo parted \/dev\/sdb set 1 raid on<\/pre>\nVerify the Configuration<\/h5>\n
You can verify that the Partition Table is type gpt, the primary partiton has been created with the appropriate size and that the raid flag is set on that partition.<\/p>\n
$ sudo parted \/dev\/sdb print\nModel: ATA HGST HDN724040AL (scsi)\nDisk \/dev\/sdb: 4001GB\nSector size (logical\/physical): 512B\/4096B\nPartition Table: gpt\nDisk Flags:\n\nNumber Start End Size File system Name Flags\n1 1049kB 4001GB 4001GB primary raid<\/pre>\nCreate the Multi Device<\/h5>\n
After you have completed the previous steps on all disks, the next step is to create the multi device (md). In my case, I am creating a RAID6 multi device from 7 partitions.<\/p>\n
$ sudo mdadm --create --verbose \/dev\/md0 --level=6 --raid-devices=6 \/dev\/sd{b,c,d,e,f,g,h}1<\/pre>\nThe multi device is created, but it will not be preserved on reboot. To preserve the configuration on reboot we need to append to the mdadm configuration file. You may also need to update initramfs and grub to ensure\u00a0the multi device is automatically assembled on reboot.<\/p>\n
$ sudo mdadm --detail --scan | sudo tee -a \/etc\/mdadm\/mdadm.conf\n$ sudo update-initramfs -u\n$ sudo update-grub<\/pre>\nCreate the XFS Filesystem<\/h5>\n
Now that we have a multi device, we can create the filesystem. I recommend using xfs if the effective capacity of your multi device may eventually grow to be more than 16TB of hard drive space.<\/p>\n
The “-d” option allows us to specify one or more disk parameters.<\/p>\n
The sunit disk parameter stands for Stripe Unit. It is optimal when assigned to\u00a0the Stripe Size in bytes divided by 512. In my case the Stripe Size is 512KB, and thus the optimal Stripe Unit is 1024.<\/p>\n
The\u00a0swidth disk parameter stands for Stripe Width. It is defined by the number of non-parity disks times Stripe Unit. In my case, I am using RAID6 with a total of 7 disks. Since RAID6 requires 2 parity disks, that leaves me with 5 non-parity disks. Thus the optimal Stripe Width is 5120 (5 non-parity disks times the 1024 Stripe Unit).<\/p>\n
$ sudo mkfs.xfs -d sunit=1024,swidth=5120 \/dev\/md0<\/pre>\nYou can verify\u00a0the sunit and swidth settings aftewards, but it can be tricky because the units are different. With mkfs.xfs they are defined as a multiple of 512-bytes. With xfs_info, they are defined as a\u00a0multiples of the block size (bsize). In this case bsize is 4096. So the values appear as 1\/8th of the value we specified in the\u00a0mkfs.xfs command.<\/p>\n
$ sudo xfs_info \/dev\/md0\nmeta-data=\/dev\/md0 isize=256 agcount=32, agsize=152612736 blks\n = sectsz=4096 attr=2, projid32bit=1\n = crc=0 finobt=0 spinodes=0 rmapbt=0\n = reflink=0\ndata = bsize=4096 blocks=4883607040, imaxpct=5\n = sunit=128 swidth=640 blks\nnaming =version 2 bsize=4096 ascii-ci=0 ftype=0\nlog =internal bsize=4096 blocks=521728, version=2\n = sectsz=4096 sunit=1 blks, lazy-count<\/pre>\nAlternative: Create the ext4 Filesystem<\/h5>\n
If you are certain you will never exceed the 16TB limit,\u00a0then ext4 is a reasonable alternative choice for a filesystem. Note that you should only<\/strong> run this command if you are using ext4 instead of xfs.<\/p>\n
$ sudo mkfs.ext4\u00a0-F \/dev\/md0<\/pre>\nMount the Multi Device<\/h5>\n
A mount point is simply the directory to which the device is mounted. So we just need to make sure that directory exists and\u00a0mount the multi device to the newly created mount point.<\/p>\n
$ sudo mkdir -p \/mnt\/md0\n$ sudo mount \/dev\/md0 \/mnt\/md0<\/pre>\nNext\u00a0we update fstab to preserve the changes on reboot.<\/p>\n
$ echo '\/dev\/md0 \/mnt\/md0 xfs\u00a0defaults,nofail 0 0' | sudo tee -a \/etc\/fstab<\/pre>\nFinally df can be used to\u00a0verify the disk is mounted and displays the usage.<\/p>\n
$ df -h -t xfs\nFilesystem Size Used Avail Use% Mounted on\n\/dev\/md0 19T 0T 0T 0% \/mnt\/md0<\/pre>\nVerify the Setup<\/h5>\n
You can verify the setup with lsblk. It shows the disk device and capacity, the raid partition, and multi device and its capacity, filesystem type, raid level, and mount point.<\/p>\n
$ lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT \/dev\/sd{b,c,d,e,f,g,h}\nNAME SIZE FSTYPE TYPE MOUNTPOINT\nsdb 3.7T disk\n\u2514\u2500sdb1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsdc 3.7T disk\n\u2514\u2500sdc1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsdd 3.7T disk\n\u2514\u2500sdd1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsde 3.7T disk\n\u2514\u2500sde1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsdf 3.7T disk\n\u2514\u2500sdf1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsdg 3.7T disk\n\u2514\u2500sdg1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0\nsdh 3.7T disk\n\u2514\u2500sdh1 3.7T linux_raid_member part\n \u2514\u2500md0 18.2T xfs raid6 \/mnt\/md0<\/pre>\nYou can view the state of multi devices using the proc virtual filesystem.<\/p>\n
$ cat \/proc\/mdstat\nPersonalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]\nmd0 : active raid6 sdc1[4] sdg1[3] sde1[2] sdd1[7] sdf1[6] sdh1[0] sdb1[5]\n 19534428160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7\/7] [UUUUUUU]\n bitmap: 0\/30 pages [0KB], 65536KB chunk\n\nunused devices: <none><\/pre>\nLastly, you can view detailed information about a multi device using mdadm.<\/p>\n
$ sudo mdadm --detail \/dev\/md0\n\/dev\/md0:\n Version : 1.2\n Creation Time : Mon May 22 21:28:46 2017\n Raid Level : raid6\n Array Size : 19534428160 (18629.48 GiB 20003.25 GB)\n Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)\n Raid Devices : 7\n Total Devices : 7\n Persistence : Superblock is persistent\n\nIntent Bitmap : Internal\n\nUpdate Time : Fri Sep 8 20:07:07 2017\n State : clean\n Active Devices : 7\nWorking Devices : 7\n Failed Devices : 0\n Spare Devices : 0\n\nLayout : left-symmetric\n Chunk Size : 512K\n\nName : euclid:0 (local to host euclid)\n UUID : a7faccce:58914d5e:b26cce7e:795b773b\n Events : 48719\n\nNumber Major Minor RaidDevice State\n 0 8 113 0 active sync \/dev\/sdh1\n 6 8 81 1 active sync \/dev\/sdf1\n 7 8 49 2 active sync \/dev\/sdd1\n 3 8 97 3 active sync \/dev\/sdg1\n 4 8 1 4 active sync \/dev\/sdc1\n 5 8 17 5 active sync \/dev\/sdb1\n 2 8 65 6 active sync \/dev\/sde1<\/pre>\n","protected":false},"excerpt":{"rendered":"I recently used mdadm to setup RAID6 on my GNU\/Linux file servers. The following is a tutorial on the commands I ran on debian 9 stretch to accomplish this task. Prerequisites You will need to install the parted, mdadm, and xfsprogs packages. GNU parted is used to create partition tables. mdadm is used to create […]<\/p>\n","protected":false},"author":1,"featured_media":400,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,9,15],"class_list":["post-122","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sysadmin","tag-debian","tag-mdadm","tag-xfs"],"_links":{"self":[{"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/posts\/122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=122"}],"version-history":[{"count":1,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/posts\/122\/revisions"}],"predecessor-version":[{"id":504,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/posts\/122\/revisions\/504"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=\/wp\/v2\/media\/400"}],"wp:attachment":[{"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codemonkeyden.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}