FreeBSD UFS/ZFS Snapshot
Management Environment
How to Deliver FreeBSD UFS/ZFS Snapshots to End-Users
Ralf S. Engelschall <rse@FreeBSD.org>
Written: 29-Aug-2004
Last Modified: 11-Nov-2007

Problem

The FreeBSD UFS (both UFS1 and UFS2) and ZFS filesystems provides the possibility to create snapshots of live filesystems. On UFS is already best known (and can be easily used) for allowing fsck(8) to run in the background (see rc.conf variable background_fsck) and to create consistent filesystem dumps (see dump(8) option -L). In ZFS snapshots even can be used with the zfs(8) send and receive commands to easily replicate ZFS filesystems.

Additionally, with the help of md(4) and mdconfig(8) you can mount UFS snapshots as read-only filesystems, too. For illustration purposes, look at this sample session where a snapshot of /var is created on UFS: 

# echo "before" >/var/tmp/foo.txt
# ls -l /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  7 Sep  3 16:19 /var/tmp/foo.txt

# test -d /var/.snap || mkdir /var/.snap
# mount -u -o snapshot /var/.snap/test /var
# ls -l /var/.snap/test
-r--------  1 root  operator  2147483784 Sep  3 16:19 /var/.snap/test

# echo "after" >/var/tmp/foo.txt
# ls -l /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  6 Sep  3 16:20 /var/tmp/foo.txt

# mdconfig -a -t vnode -f /var/.snap/test -u 1
# mount -o ro /dev/md1 /mnt
# ls -l /mnt/tmp/foo.txt /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  7 Sep  3 16:19 /mnt/tmp/foo.txt
-rw-r--r--  1 root  wheel  6 Sep  3 16:20 /var/tmp/foo.txt
# cat /mnt/tmp/foo.txt /var/tmp/foo.txt
before
after

# umount /mnt
# mdconfig -d -u 1
# rm -f /var/.snap/test

The same session with /var on ZFS could look like this: 

# echo "before" >/var/tmp/foo.txt
# ls -l /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  7 Sep  3 16:19 /var/tmp/foo.txt

# zfs snapshot `zfs list -H -o name /var`@test
# ls -ld /var/.zfs/snapshot/test
drwxr-xr-x 23 root  operator  23 Sep  3 16:19 /var/.zfs/snapshot/test

# echo "after" >/var/tmp/foo.txt
# ls -l /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  6 Sep  3 16:20 /var/tmp/foo.txt

# ls -l /var/.zfs/snapshot/test/tmp/foo.txt /var/tmp/foo.txt
-rw-r--r--  1 root  wheel  7 Sep  3 16:19 /mnt/tmp/foo.txt
-rw-r--r--  1 root  wheel  6 Sep  3 16:20 /var/tmp/foo.txt
# cat /var/.zfs/snapshot/test/tmp/foo.txt /var/tmp/foo.txt
before
after

# zfs destroy `zfs list -H -o name /var`@test

As you can see, one can easily create filesystem snapshots on both UFS and ZFS and use it to recover old states of files. Unfortunately, this is all the FreeBSD 7 base system currently provides out-of-the-box. There is neither an easy way to regularily create new and expire old snapshots nor an easy -- or even unpriviledged -- way to access the snapshot data. Additionally, although the involved steps and commands are not really complicated, they are not convenient enough for daily work. Although ZFS snapshots are certainly already easier to use than UFS snapshots, one nevertheless have to remember the particular commands. Additionally, the differences in snapshot management between UFS and ZFS makes the daily work complicated, too.

Keep in mind that regular filesystem backup snapshots are a great solution for easily recovering old file states within a small time frame. But only if the interface for the end-user follows the KISS principle. And at least this is IMHO not the case under our FreeBSD until now.

On the other hand, at my previous work my home directory was placed on an NFS-mounted WAFL® filesystem provided by an ONTAP® based Filer from Network Appliance. For instance, after you have configured scheduled snapshots with the ONTAP® command "snap sched 0 0 4", you can easily and at any time recover old states (as a regular user) in hourly steps for the last 4 hours: 

$ echo "before" >foo.txt
$ sleep 3601
$ echo "middle" >foo.txt
$ sleep 3601
$ echo "after" >foo.txt

$ ls -la | fgrep .snapshot
$ cat .snapshot/hourly.1/foo.txt .snapshot/hourly.0/foo.txt foo.txt
before
middle
after

As you have seen above, ZFS more or less already provides this approach. Unfortunately, UFS is far away from this approach. So, we need at least an abstraction layer to let us handle UFS and ZFS identicially...

Solution

An abstracted snapshot managemebt service should be provided to our users under FreeBSD. We achieve this by implementing three solutions:
  1. Snapshot Management Frontend:
    The involved commands should be wrapped by a frontend utility snapshot(8) which provides a more simple and convenient way to create, expire, mount and unmount snaphots (independent whether this is on UFS or ZFS). 
    # snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
# snapshot make -g4 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
/var              1172MB   14.8%      4MB    0.1%  test.0
# snapshot make -g4 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
/var              1176MB   14.8%      4MB    0.1%  test.0
/var              1176MB   14.8%      4MB    0.1%  test.1
# snapshot make -g4 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
/var              1181MB   14.9%      4MB    0.1%  test.0
/var              1181MB   14.9%      4MB    0.1%  test.1
/var              1181MB   14.9%      4MB    0.1%  test.2
# snapshot make -g4 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
/var              1186MB   15.0%      4MB    0.1%  test.0
/var              1186MB   15.0%      4MB    0.1%  test.1
/var              1186MB   15.0%      4MB    0.1%  test.2
/var              1186MB   15.0%      4MB    0.1%  test.3
# snapshot make -g4 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot
/var              1186MB   15.0%      4MB    0.1%  test.0
/var              1186MB   15.0%      4MB    0.1%  test.1
/var              1186MB   15.0%      4MB    0.1%  test.2
/var              1186MB   15.0%      4MB    0.1%  test.3
# snapshot mount /var:test.2 /mnt
# ls /mnt
.snap      crash      games      lost+found obj        rwho
account    cron       heimdal    mail       opkg       spool
at         db         lib        msgs       preserve   tmp
backups    empty      log        named      run        yp
# snapshot umount /mnt
# snapshot make -g0 /var:test
# snapshot list /var
Filesystem          User   User%     Snap   Snap%  Snapshot

  2. Periodic and Flexible Backup Snapshot Creation
    There has to be a flexible and easy way to configure periodically created backup snapshots. For instance, to have available two "daily" snapshots for / and /usr, two daily and four hourly (created every hour) for /var and two weekly, seven daily and eight hourly (created at times 08:00, 12:00, 16:00 and 20:00) snapshots for /home/ all which is required should be (similar to the syntax of the ONTAP "snap sched" command): 
    $ grep ^snapshot_ /etc/periodic.conf
snapshot_enable="YES"
snapshot_schedule="/,/usr:0:2:0 /var:0:2:4 /home:2:7:8@8,12,16,20"

  3. Easy Access to Backup Snapshot Data
    There has to be an easy way to access the data of an arbitrary backup snapshot as a regular user. Mostly similar to the .snapshot sub-directory feature on WAFL (we use a top-level /snap directly here because this is what can be provided by amd(8) easily and without any performance penalties and stability issues): 
    $ df | egrep '(/snap|/.am)'; ls -l /snap
pid1562@en2:/snap           0       0        0   100%    /snap

$ cd /home/rse
$ echo "before" >foo.txt
$ sleep 3601
$ echo "middle" >foo.txt
$ sleep 3601
$ echo "after" >foo.txt
$ cat /snap/home:hourly.1/rse/foo.txt /snap/home:hourly.0/rse/foo.txt foo.txt
before
middle
after

$ df | egrep '(/snap|/.am)'; ls -l /snap
pid1562@en2:/snap           0       0        0   100%    /snap
/dev/md1              2026030    2464  1861484     0%    /.am/en2/snap/var:hourly.0
lrwxrwxrwx  1 root  wheel  26 Sep  3 16:38 var:hourly.0 -> /.am/en2/snap/var:hourly.0
$ sleep 3600
$ df | egrep '(/snap|/.am)'; ls -l /snap
pid1562@en2:/snap           0       0        0   100%    /snap
    As the surrounding df(1) and ls(1) calls show, amd(8) on demand mounts the snapshot files and after about 5 minutes of no access it automatically unmounts them again.

With those three solutions, FreeBSD snapshots are available to our unprivileged users as a short-time backup solution allowing them to easily recover their older file states.

Implementation

The implementation is straight-forward and based on mount(8), mdconfig(8), amd(8) and cron(8) and zfs(8):
  1. Snapshot Management Frontend:
    First, I've implemented a little utility snapshot(8) (implementation peek: /usr/sbin/snapshot and /usr/share/man/man8/snapshot.8) which provides the convenience frontend for creating, expiring, mounting and unmounting snapshots.

  2. Periodic and Flexible Backup Snapshot Creation
    Second, I've implemented a periodic scheduler periodic-snapshot(8) (implementation peek: periodic-snapshot and /usr/share/man/man8/periodic-snapshot.8) which is called from /etc/crontab (periodic(8) cannot be used here because we need hourly runs and at exact times to not confuse users) and runs the "snapshot make" commands according to the configuration provided in /etc/periodic.conf.

  3. Easy Access to Backup Snapshot Data
    Third, I've setup a amd(8) map /etc/amd.map.snap in /etc/rc.conf which performs the "snapshot mount" and "snapshot umount" commands when a user accesses directories under /snap.
In order to easily integrate this implementation into a FreeBSD 7 installation, DOWNLOAD the latest tarball HERE and install it as following: 
# tar zxf freebsd-snapshot-*.tar.gz
# cd freebsd-snapshot-*
# make install
# /etc/rc.d/amd start
(If you later want to remove it again, just run "make uninstall")

References

See the following URLs for further details: