Building 6 Bay NAS with Software RAID

  • Only Rev.B boards are capable with M.2 to SATA adapter.

Since the new H2 marked Rev.B supports M.2 to SATA adapter, we are able to build a NAS with more handy looks.

Our environment is,

  • ODROID-H2 Rev.B
  • Samsung DDR4 4GB * 2
  • 64GB eMMC
  • IOCREST M.2 to 4 Port SATA 6G Adapter (IO-M2F9235-4I, Marvell 88SE9235 Controller)
  • WD RED 1TB HDD * 6
  • HCiPC P401-1 HCNAS(Node6) Mini-ITX Case
  • Micronics Cyclone III 350W ATX Power Supply
  • Basic engineering kits for DIY tasks

In this note, I will introduce to you how to set the RAID 6 system up and how the performance of this is. And will think together whether H2 is worth for NAS use with RAID-ed hard drives or not.

We decided to use Mini-ITX PC case for its enclosure.

That case has to have at least 6 hard drive bays and have space for attaching a standard ATX power supply. Also having extra case fan would be good.

And definitely, it must have compact and solid looking.

So we chose this one, called HCiPC P401-1 HCNAS(Node6) Mini-ITX, having all of the requirements with a solid look.

But H2 onboard screw-holes ain't fit into the ITX cases. So we had to 3D-print a guide for it.

Here is the download link for the mounting adapter 3D model: https://www.thingiverse.com/thing:3686128

This assembly steps had been posted one by one on our forum. To see the whole assembly process, please look into this thread: https://forum.odroid.com/viewtopic.php?f=172&t=35309

With its default setup, SATA adapter on M.2 slot will not work. To use properly, we have to configure BIOS settings.

Enter to BIOS by press delete key during just after the power button pushed. Then you should face the screen like the below.

You have to set 2 options.

  • Chipset - South Cluster Configuration - Miscellaneous Configuration - State After G3 to S0 State
    • To make it turn on automatically when the power plugged.
  • Chipset - South Cluster Configuration - PCI Express Configuration - PCI Express Clock Gating to Disabled
    • To make M.2 to SATA adapter working properly.

Move using arrow keys on the keyboard to them and set each option.

To its host operating system, we've decided to install Ubuntu 19.04.

Download the latest OS image from Ubuntu download page (https://ubuntu.com/download), and flash that image into a USB stick using Etcher. Etcher is a multi-platform flashing tool.

Plug the USB memory stick to H2 and power on. Then press F7 to enter choosing boot media screen. If the screen shows up, select the inserted USB boot media.

Then install Ubuntu as your taste. Just be aware of selecting a hard drive when you select a root media during installation.

And make the system up-to-date is highly recommended in any case of application use, in most cases.

To check whether the hard drives installed or not, you can look into the outputs of dmesg.

odroid@ODROID-H2:~$ dmesg | grep scsi
[    2.067831] scsi host0: ahci
[    2.068684] scsi host1: ahci
[    2.080796] scsi host2: ahci
[    2.080964] scsi host3: ahci
[    2.084816] scsi host4: ahci
[    2.084976] scsi host5: ahci
[    2.548364] scsi 0:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.549029] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    2.549321] scsi 1:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.549627] sd 1:0:0:0: Attached scsi generic sg1 type 0
[    2.563013] scsi 2:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.563292] sd 2:0:0:0: Attached scsi generic sg2 type 0
[    2.563497] scsi 3:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.563693] sd 3:0:0:0: Attached scsi generic sg3 type 0
[    2.563875] scsi 4:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.564070] sd 4:0:0:0: Attached scsi generic sg4 type 0
[    2.564268] scsi 5:0:0:0: Direct-Access     ATA      WDC WD10EFRX-68F 0A82 PQ: 0 ANSI: 5
[    2.564444] sd 5:0:0:0: Attached scsi generic sg5 type 0

This shows that those 6 drives are recognized normally. If yours ain't, you should check the hardware connection or BIOS settings.

Check the installed hard drives using the following commands.

odroid@ODROID-H2:~$ sudo fdisk -l | grep sd
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sde: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdf: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors

Then create a partition using parted tool.

You can use fdisk to partition them but GPT partition table should be created in order to use more than 2TB hard drive.

Partition the disks respectively referring to the following procedures.

odroid@ODROID-H2:~$ sudo parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt                                                      
(parted) print free                                                       
Model: ATA WDC WD10EFRX-68F (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 
Number  Start   End     Size    File system  Name  Flags
        17.4kB  1000GB  1000GB  Free Space
(parted) mkpart primary 1M 1000GB                                         
(parted) p                                                                
Model: ATA WDC WD10EFRX-68F (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 
Number  Start   End     Size    File system  Name     Flags
 1      1049kB  1000GB  1000GB               primary
(parted) q                                                                
Information: You may need to update /etc/fstab.

Check the changes applied.

odroid@ODROID-H2:~$ lsblk | grep sd
sda            8:0    0 931.5G  0 disk 
└─sda1         8:1    0 931.5G  0 part 
sdb            8:16   0 931.5G  0 disk 
└─sdb1         8:17   0 931.5G  0 part 
sdc            8:32   1 931.5G  0 disk 
└─sdc1         8:33   1 931.5G  0 part 
sdd            8:48   1 931.5G  0 disk 
└─sdd1         8:49   1 931.5G  0 part 
sde            8:64   1 931.5G  0 disk 
└─sde1         8:65   1 931.5G  0 part 
sdf            8:80   1 931.5G  0 disk 
└─sdf1         8:81   1 931.5G  0 part 

We can see each all hard drive has one partition.

After partitioning all of the drives, now it's time to set RAID 6 up on our NAS.

Here are 2 big facts why RAID level 6 we are going to use.

  • It is more robust than RAID 5, because it uses one more disk for parity.
  • So there will be no data loss even after 2 disk fails. We can rebuild it after replacing the failed disk.

But it also has its kind of overhead. Double parity can verify its stability but also comes with poor writing performance.

And minimum 4 disks required to build with RAID 6. Since we have 6 hard drives which have 1TB capacity each, we can build using RAID 6 and there will be 4TB capacity we can use.

To setup RAID without physical RAID controller on Linux system, we have to use mdadm tool. It is provided by package manager on each Linux distros.

Install mdadm using apt.

odroid@ODROID-H2:~$ sudo apt install mdadm

We already know that those 6 drives are allocated as /dev/sda to /dev/sdf.

Create an array using the following command.

odroid@ODROID-H2:~$ sudo mdadm --create /dev/md0 --level=6 --raid-devices=6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Then /dev/md0 device file will be created.

You should use that device like as a one hard drive partition. E.g, if you about to mount that array to a directory, you just can mount with that device file.

Do format that partition and mount to the /media/storage.

odroid@ODROID-H2:~$ sudo mkfs.ext4 /dev/md0
mke2fs 1.44.6 (5-Mar-2019)
Creating filesystem with 976628736 4k blocks and 244162560 inodes
Filesystem UUID: 100a470d-96f1-47d2-8cf0-a211c010e8b9
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
	102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done       
odroid@ODROID-H2:~$ sudo mkdir /media/storage
odroid@ODROID-H2:~$ sudo mount /dev/md0 /media/storage/

Check if it mounted properly.

odroid@ODROID-H2:~$ cat /proc/mounts | grep md0
/dev/md0 /media/storage ext4 rw,relatime,stripe=512 0 0

You also can see the RAID configurations.

odroid@ODROID-H2:~$ sudo mdadm --detail /dev/md0
           Version : 1.2
     Creation Time : Mon Jun 17 18:08:26 2019
        Raid Level : raid6
        Array Size : 3906514944 (3725.54 GiB 4000.27 GB)
     Used Dev Size : 976628736 (931.39 GiB 1000.07 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent
     Intent Bitmap : Internal
       Update Time : Mon Jun 17 18:27:08 2019
             State : active, resyncing 
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0
            Layout : left-symmetric
        Chunk Size : 512K
Consistency Policy : bitmap
     Resync Status : 8% complete
              Name : ODROID-H2:0  (local to host ODROID-H2)
              UUID : d4759dbb:65fd2b07:d5f4f9c3:0fba55cc
            Events : 253
    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1
       5       8       81        5      active sync   /dev/sdf1

Since this RAID array is just created, resync process that makes synchronized with the other devices is performed at the very first time.

You can see the status of the resync process.

odroid@ODROID-H2:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid6 sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      3906514944 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      [=>...................]  resync =  9.6% (93909272/976628736) finish=120.2min speed=122360K/sec
      bitmap: 8/8 pages [32KB], 65536KB chunk
unused devices: <none>

If the rsync process has been done, you can see that message on dmesg.

odroid@ODROID-H2:~$ dmesg | grep resync
[  199.311304] md: resync of RAID array md0
[10093.988694] md: md0: resync done.

And you also would set your system environment. Adding an entry to /etc/fstab and configure SAMBA or SFTP server to share your data by using that RAID-ed hard drives.

There are many management guides for further use of RAID built system including adding spare drives or dealing with failed devices. But this guide will not handle it.

I performed iozone3 to evaluate its performance of the H2 with RAID level 6 on 6 hard drives.

I had been checked that the resync process is done successfully.

odroid@ODROID-H2:/media/storage$ sudo iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
sudo iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.429 $
		Compiled for 64 bit mode.
		Build: linux-AMD64 
	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa.
	Run began: Tue Jun 18 10:03:49 2019
	Include fsync in write timing
	O_DIRECT feature enabled
	Auto Mode
	File size set to 102400 kB
	Record Size 4 kB
	Record Size 16384 kB
	Command line used: iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     9241    14364    27027    29648    15326     4404                                                          
          102400   16384   208414   209245   521260   669540   731096   177565                                                          
iozone test complete.

Amazing. Since the 6 hard drives linked each other, its performance is far better than from the results with the only 1 hard drive.

One Disk RAID 6 with 6 Disks
4K R/Read 979 27027
4K R/Write 1937 9241
16M S/Read 147202 (147 MB/s) 521260 (521 MB/s)
16M S/Write 132406 (132 MB/s) 208414 (208 MB/s)

RAID 6 uses double parity but 6 hard drives perform like a kind of multiprocessing in CPU literally. So the overall speed can be hugely increased. If with RAID level 5, the performance would be a little bit better than this since RAID 5 uses single parity.

Therefore, there's no reason to do not use RAID 6 when you have more than 4 hard drives.

Especially, this also clear that H2 is powerful host hardware for NAS use.

With its reasonable price, without something being lazy tasks such as buying a Mini-ITX case and DIY-ing it, H2 with software RAID configuration is worth. Furthermore, H2 has two numbers of Gigabits LAN ports.

  • odroid-h2/application_note/6_bay_nas_with_sw_raid.txt
  • Last modified: 2022/10/13 12:22
  • by justin