Random Musings

O for a muse of fire, that would ascend the brightest heaven of invention!

FreeBSD ZFS notes

Thursday, 8 Oct 2020 Tags: freebsdzfs


Get, and read, these books:

  • Storage Essentials
  • Specialty Filesystems
  • ZFS & Advanced ZFS

Cleaning up ZFS snapshots on low disk space

Snapshots are taken automatically, of both jails (zroot/iocell) and of the main system.

In particular the main system database and logfiles are the files that change the most, and consume the most space. These can be cleaned up easily. As we have a DB cluster, and also backups, the snapshots are not particularly important after a few days of use.


First we check the general pool usage and health, and apply a temporary snapshot, as this simplifies cleaning up all snapshots in a given dataset.

# zpool list -v
zroot        230G   191G  39.4G        -         -    68%    82%  1.00x  ONLINE  -
  ada0p3     230G   191G  39.4G        -         -    68%    82%

# zpool status -v
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 00:13:39 with 0 errors on Sat Oct  3 03:45:56 2020

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

# zfs snapshot -r zroot@cleanup

Where is the space going

List datasets, with space sorted by snapshot, excluding the iocell jails as these change rarely, and are trivial to regenerate using ansible, and any datasets that have zero snapshot usage:

#  zfs list -Ho usedbysnapshots,name | sort -h |egrep -v 'iocell|^0'
48K     zroot/var/log/rsyslog
664K    zroot/var/mail
768K    zroot/var/tmp
2.34M   zroot/usr/ports
14.1M   zroot/var/spool/rsyslog
340M    zroot/ROOT/default
794M    zroot/var/db/kyototycoon
996M    zroot/var/log
1.39G   zroot/usr/home
29.7G   zroot/var/db/couchdb/views
52.2G   zroot/var/db/couchdb

clean up

We do cleanup in two phases - first with -n which informs us of changes without actually making them - and then without, to trash stuff.

zfs destroy can annihilate a lot of data very fast, so always use the -n flag before proceeding.

Here, we are particularly interested in ensuring that there are no cloned/parent datasets being pulled in, as a result of this destroy command. This is apparent as you’ll see a different dataset/snapshot name in the list compared to the one you request. If that happens, stop, and sort out the other dataset first before proceeding.

Note that the -r flag is recursive - we want that here for the database and the materalised views in /var/db/couchdb/views. If you’re doing a different dataset, the -r flag may not be desirable. It will clean up all datasets underneath this one — not by mountpoint, but by how they are organised in the zfs dataset tree.

# zfs destroy -vrn zroot/var/db/couchdb@%cleanup
would destroy zroot/var/db/couchdb@20200421-1714:11.3-RELEASE-p7
would destroy zroot/var/db/couchdb@20200423-1355:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb@20200513-1150:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb@20200609-0757:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb@20200609-0804:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb@20200709-0557:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb@20200709-0710:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb@monthly-2020-09
would destroy zroot/var/db/couchdb@weekly-2020-35
would destroy zroot/var/db/couchdb@20200910-0845:12.1-RELEASE-p7
would destroy zroot/var/db/couchdb@weekly-2020-36
would destroy zroot/var/db/couchdb@weekly-2020-37
would destroy zroot/var/db/couchdb@weekly-2020-38
would destroy zroot/var/db/couchdb@monthly-2020-10
would destroy zroot/var/db/couchdb@20201001-0809:12.1-RELEASE-p9
would destroy zroot/var/db/couchdb@daily-2020-10-02
would destroy zroot/var/db/couchdb@daily-2020-10-03
would destroy zroot/var/db/couchdb@weekly-2020-39
would destroy zroot/var/db/couchdb@daily-2020-10-04
would destroy zroot/var/db/couchdb@daily-2020-10-05
would destroy zroot/var/db/couchdb@daily-2020-10-06
would destroy zroot/var/db/couchdb@daily-2020-10-07
would destroy zroot/var/db/couchdb@daily-2020-10-08
would destroy zroot/var/db/couchdb@hourly-2020-10-08-05
would destroy zroot/var/db/couchdb@hourly-2020-10-08-06
would destroy zroot/var/db/couchdb@hourly-2020-10-08-07
would destroy zroot/var/db/couchdb@hourly-2020-10-08-08
would destroy zroot/var/db/couchdb@hourly-2020-10-08-09
would destroy zroot/var/db/couchdb@hourly-2020-10-08-10
would destroy zroot/var/db/couchdb@cleanup
would destroy zroot/var/db/couchdb/views@20200318-2345:11.2-RELEASE-p15
would destroy zroot/var/db/couchdb/views@20200319-2046:11.3-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200421-1714:11.3-RELEASE-p7
would destroy zroot/var/db/couchdb/views@20200423-1355:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb/views@20200513-1150:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb/views@20200609-0757:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb/views@20200609-0804:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb/views@20200709-0557:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200709-0710:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200910-0845:12.1-RELEASE-p7
would destroy zroot/var/db/couchdb/views@20201001-0809:12.1-RELEASE-p9
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-05
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-06
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-07
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-08
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-09
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-10
would destroy zroot/var/db/couchdb/views@cleanup
would reclaim 81.9G

OK this looks good, so let rip:

# zfs destroy -vr zroot/var/db/couchdb@%cleanup
will destroy zroot/var/db/couchdb@20200421-1714:11.3-RELEASE-p7
will destroy zroot/var/db/couchdb@20200423-1355:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb@20200513-1150:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb@20200609-0757:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb@20200609-0804:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb@20200709-0557:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb@20200709-0710:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb@monthly-2020-09
will destroy zroot/var/db/couchdb@weekly-2020-35
will destroy zroot/var/db/couchdb@20200910-0845:12.1-RELEASE-p7
will destroy zroot/var/db/couchdb@weekly-2020-36
will destroy zroot/var/db/couchdb@weekly-2020-37
will destroy zroot/var/db/couchdb@weekly-2020-38
will destroy zroot/var/db/couchdb@monthly-2020-10
will destroy zroot/var/db/couchdb@20201001-0809:12.1-RELEASE-p9
will destroy zroot/var/db/couchdb@daily-2020-10-02
will destroy zroot/var/db/couchdb@daily-2020-10-03
will destroy zroot/var/db/couchdb@weekly-2020-39
will destroy zroot/var/db/couchdb@daily-2020-10-04
will destroy zroot/var/db/couchdb@daily-2020-10-05
will destroy zroot/var/db/couchdb@daily-2020-10-06
will destroy zroot/var/db/couchdb@daily-2020-10-07
will destroy zroot/var/db/couchdb@daily-2020-10-08
will destroy zroot/var/db/couchdb@hourly-2020-10-08-05
will destroy zroot/var/db/couchdb@hourly-2020-10-08-06
will destroy zroot/var/db/couchdb@hourly-2020-10-08-07
will destroy zroot/var/db/couchdb@hourly-2020-10-08-08
will destroy zroot/var/db/couchdb@hourly-2020-10-08-09
will destroy zroot/var/db/couchdb@hourly-2020-10-08-10
will destroy zroot/var/db/couchdb@cleanup
will destroy zroot/var/db/couchdb/views@20200318-2345:11.2-RELEASE-p15
will destroy zroot/var/db/couchdb/views@20200319-2046:11.3-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200421-1714:11.3-RELEASE-p7
will destroy zroot/var/db/couchdb/views@20200423-1355:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb/views@20200513-1150:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb/views@20200609-0757:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb/views@20200609-0804:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb/views@20200709-0557:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200709-0710:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200910-0845:12.1-RELEASE-p7
will destroy zroot/var/db/couchdb/views@20201001-0809:12.1-RELEASE-p9
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-05
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-06
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-07
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-08
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-09
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-10
will destroy zroot/var/db/couchdb/views@cleanup
will reclaim 81.9G

Looking Good

Note that free space is returned lazily to the pool depending on io usage, so don’t be surprised if 100+GiB takes a while to re-appear.

Once you’ve finished tidying up, feel free to trash the temporary cleanup dataset, recursively.

# zpool list -v
zroot        230G   109G   121G        -         -    56%    47%  1.00x  ONLINE  -
  ada0p3     230G   109G   121G        -         -    56%    47%

# zfs destroy -vr zroot@cleanup

Cleaning up a dataset with dependent clones

The zroot/ROOT/default dataset contains the / filesystem, and after every upgrade or update, will have a cloned and snapshotted “boot environment” left behind.

# zfs destroy -vnr zroot/ROOT/default@%cleanup
cannot destroy 'zroot/ROOT/default@2020-10-01-08:38:52-0': snapshot has dependent clones
use '-R' to destroy the following datasets:
would destroy zroot/ROOT/default@2019-10-21-14:12:40
would destroy zroot/ROOT/default@hourly-2020-10-08-10
would destroy zroot/ROOT/default@cleanup

These come from the boot environment - an amazing way to switch between boot systems in case of failure.

# bectl list
BE               Active Mountpoint Space Created
12.1-RELEASE-p10 -      -          132K  2020-10-01 08:38
default          NR     /          14.7G 2020-02-17 12:52

The default BE has NR flags set - Now and Reboot - its the current BE and will also be used on next reboot.

We can safely destroy the previous one, but note that the snapshot used to clone the BE still remains:

# bectl destroy 12.1-RELEASE-p10
bectl destroy: leaving origin 'zroot/ROOT/default@2020-10-01-08:38:52-0' intact

Let’s re-try that destroy:

# zfs destroy -vnr zroot/ROOT/default@%cleanup
would destroy zroot/ROOT/default@2019-10-21-14:12:40
would destroy zroot/ROOT/default@monthly-2019-12
would destroy zroot/ROOT/default@cleanup
would reclaim 6.32G