FreeBSD System Disk Mirroring
How to establish a RAID-1 for the system partitions
Ralf S. Engelschall <rse@FreeBSD.org>
Written: 2005-01-09, Last Modified: 2007-05-17
Version: 2.2

Problem

RAID-1 (mirroring) is a popular approach to protect the system from a harddisk failure. It is either done in hardware or software. The usual hardware solution is to buy a RAID disk controller like the popular 3ware ATA RAID controllers and then not having to deal with any software incompatibilities because the system just sees one large physical disk.

The software solution is less expensive and more flexible, but usually makes more trouble during booting. Because an important point in establishing a software RAID-1 for the system disk partitions is whether one can directly boot from the resulting mirror setup without ugly BIOS and/or boot loader tricks. Additionally, IMHO it is also important that in case of a major problem with the RAID-1 software driver, one is still able to easily rescue boot from the mirror setup by treating it like a plain disk setup again.

Possibilities

For software solutions in FreeBSD 6 one can choose between the raw ata(4), the good old ccd(4), the GEOM gvinum(8) and the GEOM gmirror(8). Each one has its pros and cons:

I currently prefer GEOM mirror for establishing such a RAID-1 solution for the system disk partitions because it both provides the possibility to directly boot from the mirror without tricks and is a really clean and modern solution. Additionally, GEOM based solutions can be stacked in an arbitrary way, i.e., the resulting GEOM mirror is just another GEOM "provider" on which one can base any other GEOM "consumers", including GEOM striping (gstripe), GEOM based disk encryption (gbde) or my most favorite: GEOM labeling (glabel) of UFS filesystems (see tunefs(8) for "tunefs -L").

Solutions

The FreeBSD 6 geom(4) framework is an excellent abstraction layer for disk I/O. The GEOM gmirror(8) class is a software RAID-1 implementation which can be used even for establishing a mirror for the system disk partitions. GEOM mirror is very flexible and actually can be established for any GEOM "providers", including whole disks or just particular slices on a disk. Unfortunately each approach has again pros and cons and hence should be carefully understood before using one of them:

As the first approach "GEOM mirror on whole disk" is usually not reasonable in practice (where disks are often just nearly equal in size), and the third approach "GEOM mirror on partitions of a slice on a disk" is usually total overkill (both from the requirement, configuration and understanding point of view), I personally prefer the second approach "GEOM mirror on slices of a disk" as it provides good enough flexibility but still easy enough management.

Implementation

The following is a step-by-step command list for remotely converting a production FreeBSD 6.2-STABLE (i386 or amd64 PC architecture based) system from a single-disk/single-slice (ad0s1) to a two-disk/single-slice (ad0s1 & ad1s1) GEOM mirror (gm0) based setup without the need for console access or Fixit/LiveFS CDROM:
# switch to /bin/sh as this script uses Bourne-Shell syntax
exec /bin/sh

# make step-by-step list more reusable by allowing the two disk devices
# and the resulting mirror device to be adapted to the local situation
d1=ad0; d2=ad1; gm=gm0

# make sure the second disk is treated as a really fresh one
dd if=/dev/zero of=/dev/${d2} bs=512 count=79

# place a PC MBR onto the second disk with a single FreeBSD slice
# /dev/mirror/${gm} as large as the /dev/${d1}s1 We reduce the size by
# one block because ${d1} and ${d1}s1 else would share the same last sector
# which could lead to ${d1} be recognized as the GEOM provider instead of
# ${d1}s1. Notice also that fdisk(8) usually further reduces the size by
# more blocks to make sure the size is a multiple of the cylinder size.
size=`fdisk /dev/${d1} | grep ', size ' |\
    head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
size=`expr $size - 1`
fdisk -v -B -I /dev/${d2}
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -f- -i /dev/${d2}

# place a GEOM mirror label onto first slice of second disk (actually on
# the last block of the disk slice) and activate the GEOM mirror kernel
# layer (which makes the /dev/mirror/${gm} device available).
gmirror label -v -n -b round-robin ${gm} /dev/${d2}s1
gmirror load

# place a BSD disklabel onto /dev/mirror/${gm} which is nearly identical
# to the BSD disklabel on /dev/${d1}s1 but has the slightly reduced total
# slice size (partition "c" because of the additional GEOM mirror label
# block) and hence a slightly smaller last partition. NOTICE: partition
# "a" always should start at offset 16 because of the BSD disklabel
# and partition "c" at offset 0 and notice that partition "c" has to
# cover the whole disk.
bsdlabel -w -B /dev/mirror/${gm}
( bsdlabel /dev/mirror/${gm} | grep 'c:'; \
  bsdlabel /dev/${d1}s1 | grep -v 'c:' ) | \
awk 'BEGIN { d = 0; n = 0; } \
    /c:/ { \
        d = $2; \
        printf("%s %d %s %s %s %s\n", $1, d, $3, $4, $5, $6); next; \
    } \
    /[abdefgh]:/ { \
        p = $2; if (d > 0 && n+p > d) { p = d-n }; \
        if ($1 == "a:") { n += $3 }; n += p; \
        printf("%s %d %s %s %s %s\n", $1, p, $3, $4, $5, $6); \
    } \
' >/tmp/bsdlabel.txt
bsdlabel -R /dev/mirror/${gm} /tmp/bsdlabel.txt

# dump & restore filesystem data from first to second disk
for p in `bsdlabel /dev/${d1}s1 |\
          egrep '^ *[adefgh]:' | sed -e 's/^ *\([adefgh]\).*/\1/'`; do \
    fs=`egrep "^ */dev/${d1}s1$p" /etc/fstab | awk '{ print $2; }'`; \
    newfs -U /dev/mirror/${gm}$p && \
    mount /dev/mirror/${gm}$p /mnt$fs && \
    dump -L -0 -f- $fs | (cd /mnt$fs && restore -r -v -f-); \
done

# adjust new system configuration for GEOM mirror based setup
cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
sed -e "s;dev/${d1}s1;dev/mirror/${gm};g" \
    </mnt/etc/fstab.orig >/mnt/etc/fstab
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf

# instruct boot stage 2 loader on first disk to boot
# with the boot stage 3 loader from the second disk
# (mainly because BIOS might not allow easy booting from second
# ATA disk or at least requires manual intervention on the console)
# In general make sure your BIOS boot order is configured to try to
# boot of both ATA disks.
echo $d2 | sed -e 's;^\([^0-9]*\)\([0-9][0-9]*\)$;1:\1(\2,a)/boot/loader;' \
    >/boot.config

# reboot system
# (for running on top of new GEOM mirror on second disk)
shutdown -r now
    

# switch to /bin/sh as this script uses Bourne-Shell syntax
exec /bin/sh

# make step-by-step list more reusable by allowing the two disk devices
# and the resulting mirror device to be adapted to the local situation
d1=ad0; d2=ad1; gm=gm0

# make sure the first disk is treated as a really fresh one again
dd if=/dev/zero of=/dev/${d1} bs=512 count=79

# place a new PC MBR onto the first disk with a single FreeBSD slice
# /dev/${d1}s1 _exactly_ as large as the /dev/${d2}s1
size=`fdisk ${d2} | grep ', size ' |\
    head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/${d1}

# switch GEOM mirror to auto-synchronization and add first disk (with
# higher priority). The first disk is now immediately synchronized with
# the second disk content.
gmirror configure -a ${gm}
gmirror insert -p 1 ${gm} /dev/${d1}s1

# wait for the GEOM mirror synchronization to complete
while [ ".`gmirror list ${gm} | grep SYNCHRONIZING`" != . ]; do \
    gmirror list ${gm} | grep Synchronized: |\
    awk '{ printf("\r%s", $0); }'; sleep 5; \
done

# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on first disk
# as it was synchronized from second disk)
shutdown -r now
    
Final notice: In case of a drive failure, after you've replaced the broken drive with a new one just remember that you have to apply the second part of the step-by-step list for this drive again. Especially do not forget to fdisk(8) it again before adding to the GEOM mirror.

References

See the following URLs for further details: