BTRFS RAID 1 array setup with encryption and monitoring

March 27, 2022

Overview

This post will describe the process for creating an encrypted BTRFS RAID 1 array. I will also cover automatic decrypting at boot using a key file. This assumes we have an encrypted root as well which is where the key file will be stored. Lastly, I’ll provide some scripts that I use to monitor my BTRFS RAID array as well as my BTRFS root along with the associated service and timer files. My particular OS is Fedora Workstation. Here are a few links that may be helpful for reference during this process:

Creating the array and decrypting at boot

First, we’ll create an empty GPT container on both disks and encrypt them with a passphrase as well as a key file. The key file can be stored on the encrypted root and allow for automatic decryption of the array at boot. The actual BTRFS RAID array can then be created and mounted. Either disk can be mounted since they are both part of the array.

# use fdisk to create an empty GPT partition table on both disks
# encrypt each disk with a passphrase
sudo cryptsetup -v luksFormat /dev/sdx
# add a key file to each disk
sudo cryptsetup luksAddKey /dev/sdx /home/user/mykeyfile
# open both disks then create the RAID array
sudo cryptsetup -d /home/user/mykeyfile open /dev/sdx cryptsdx
sudo mkfs.btrfs -m raid1 -d raid1 /dev/mapper/cryptsdx /dev/mapper/cryptsdx
# before mounting, set the immutable flag to prevent anything being written if the array is not mounted
sudo chattr +i /mntpoint
# get UUIDs
sudo blkid
# add a line to the /etc/crypttab file for each disk. this is the encrypted disk, not the /dev/mapper disk
cryptsdx    UUID=device-uuid    /home/user/mykeyfile
# add a line to the /etc/fstab file for only one of the disks. this is the /dev/mapper disk
UUID=device-uuid    /mntpoint    btrfs    defaults    0 0
# mount and change ownership
sudo mount -a
sudo chown -R user:user /mntpoint

Maintenance and monitoring

Be sure to backup the LUKS header in case you ever need to restore it.

The BTRFS filesystem has a feature known as scrubbing that checks for data integrity issues. This can be automated with scripts available in the btrfsmaintenance package. The default configuration automatically scrubs the root filesystem. Additional BTRFS filesystems can be added in the configuration file at /etc/sysconfig/btrfsmaintenance. To enable the scrub timer, simple run # systemctl enable --now btrfs-scrub.timer.

I created two scripts to perform some basic monitoring of my BTRFS devices. I then automated these scripts using systemd with timers. These scripts also take advantage of how I setup postfix and mailx to send email externally for notifications. If you use these scripts, be sure to change the mount points to reflect your setup. The .sh files should be placed in /usr/local/bin and made executable with # chmod +x *.sh. The .service and .timer files should be placed in /etc/systemd/system/. Don’t forget to enable and start the timers.

Daily error check

btrfs-check.sh

#!/bin/bash
#
# must run as root
#
day=`date +%D`
time=`date +%T`

if ! btrfs device stats -c /storage >> /dev/null
then
    echo "Email sent on $day at $time." > /etc/btrfs-stats-storage-email
    echo "" >> /etc/btrfs-stats-storage-email
    btrfs device stats /storage >> /etc/btrfs-stats-storage-email
    mail -Ssendwait -s "BTRFS ERROR ON STORAGE" [email protected] < /etc/btrfs-stats-storage-email
    rm /etc/btrfs-stats-storage-email
fi

if ! btrfs device stats -c / >> /dev/null
then
    echo "Email sent on $day at $time." > /etc/btrfs-stats-root-email
    echo "" >> /etc/btrfs-stats-root-email
    btrfs device stats / >> /etc/btrfs-stats-root-email
    mail -Ssendwait -s "BTRFS ERROR ON ROOT" [email protected] < /etc/btrfs-stats-root-email
    rm /etc/btrfs-stats-root-email
fi

btrfs-check.service

[Unit]
Description=Check BTRFS devices for reported errors

[Service]
Type=oneshot
ExecStart=/usr/local/bin/btrfs-check.sh

btrfs-check.timer

[Unit]
Description=Check BTRFS devices for reported errors

[Timer]
OnCalendar=Mon..Fri *-*-* 13:30:00
OnCalendar=Sat,Sun *-*-* 06:30:00

[Install]
WantedBy=timers.target

Weekly summary

btrfs-summary.sh

#!/bin/bash
#
# must run as root
#
day=`date +%D`
time=`date +%T`

echo $'Weekly BTRFS status summary.\n ' > /etc/btrfs-summary-email
echo "Sent on $day at $time." >> /etc/btrfs-summary-email
echo $'\n----------------------------------------------------\n' >> /etc/btrfs-summary-email
echo $'/\n' >> /etc/btrfs-summary-email
btrfs filesystem usage / >> /etc/btrfs-summary-email
echo $'\nScrub status:' >> /etc/btrfs-summary-email
btrfs scrub status / >> /etc/btrfs-summary-email
echo $'\nErrors:' >> /etc/btrfs-summary-email
btrfs device stats / >> /etc/btrfs-summary-email
echo $'\n----------------------------------------------------\n\n/storage\n' >> /etc/btrfs-summary-email
btrfs filesystem usage /storage >> /etc/btrfs-summary-email
echo $'\nScrub status:' >> /etc/btrfs-summary-email
btrfs scrub status /storage >> /etc/btrfs-summary-email
echo $'\nErrors:' >> /etc/btrfs-summary-email
btrfs device stats /storage >> /etc/btrfs-summary-email
echo $'\n----------------------------------------------------' >> /etc/btrfs-summary-email

mail -Ssendwait -s "BTRFS WEEKLY SUMMARY" [email protected] < /etc/btrfs-summary-email
rm /etc/btrfs-summary-email

btrfs-summary.service

[Unit]
Description=Generate BTRFS status summary

[Service]
Type=oneshot
ExecStart=/usr/local/bin/btrfs-summary.sh

btrfs-summary.timer

[Unit]
Description=Generate BTRFS status summary

[Timer]
OnCalendar=Sat *-*-* 07:30:00

[Install]
WantedBy=timers.target

Conclusion

There are definitely variations of this setup process that would ultimately lead to the same result. This is just my documentation of how I setup my BTRFS RAID 1 array.