FreeBSD ZFS notes
Thursday, 8 Oct 2020
References
Get, and read, these books:
- Storage Essentials
- Specialty Filesystems
- ZFS & Advanced ZFS
Cleaning up ZFS snapshots on low disk space
Snapshots are taken automatically, of both jails (zroot/iocell) and of the main system.
In particular the main system database and logfiles are the files that change the most, and consume the most space. These can be cleaned up easily. As we have a DB cluster, and also backups, the snapshots are not particularly important after a few days of use.
Pre-requisites
First we check the general pool usage and health, and apply a temporary snapshot, as this simplifies cleaning up all snapshots in a given dataset.
# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 230G 191G 39.4G - - 68% 82% 1.00x ONLINE -
ada0p3 230G 191G 39.4G - - 68% 82%
# zpool status -v
pool: zroot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0 days 00:13:39 with 0 errors on Sat Oct 3 03:45:56 2020
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
errors: No known data errors
# zfs snapshot -r zroot@cleanup
#Where is the space going
List datasets, with space sorted by snapshot, excluding the iocell jails as these change rarely, and are trivial to regenerate using ansible, and any datasets that have zero snapshot usage:
# zfs list -Ho usedbysnapshots,name | sort -h |egrep -v 'iocell|^0'
48K zroot/var/log/rsyslog
664K zroot/var/mail
768K zroot/var/tmp
2.34M zroot/usr/ports
14.1M zroot/var/spool/rsyslog
340M zroot/ROOT/default
794M zroot/var/db/kyototycoon
996M zroot/var/log
1.39G zroot/usr/home
29.7G zroot/var/db/couchdb/views
52.2G zroot/var/db/couchdbclean up
We do cleanup in two phases - first with -n which informs us of
changes without actually making them - and then without, to trash stuff.
zfs destroy can annihilate a lot of data very fast, so always use
the -n flag before proceeding.
Here, we are particularly interested in ensuring that there are no cloned/parent datasets being pulled in, as a result of this destroy command. This is apparent as you’ll see a different dataset/snapshot name in the list compared to the one you request. If that happens, stop, and sort out the other dataset first before proceeding.
Note that the -r flag is recursive - we want that here for the
database and the materalised views in /var/db/couchdb/views. If you’re
doing a different dataset, the -r flag may not be desirable. It will
clean up all datasets underneath this one – not by mountpoint, but by
how they are organised in the zfs dataset tree.
# zfs destroy -vrn zroot/var/db/couchdb@%cleanup
would destroy zroot/var/db/couchdb@20200421-1714:11.3-RELEASE-p7
would destroy zroot/var/db/couchdb@20200423-1355:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb@20200513-1150:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb@20200609-0757:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb@20200609-0804:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb@20200709-0557:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb@20200709-0710:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb@monthly-2020-09
would destroy zroot/var/db/couchdb@weekly-2020-35
would destroy zroot/var/db/couchdb@20200910-0845:12.1-RELEASE-p7
would destroy zroot/var/db/couchdb@weekly-2020-36
would destroy zroot/var/db/couchdb@weekly-2020-37
would destroy zroot/var/db/couchdb@weekly-2020-38
would destroy zroot/var/db/couchdb@monthly-2020-10
would destroy zroot/var/db/couchdb@20201001-0809:12.1-RELEASE-p9
would destroy zroot/var/db/couchdb@daily-2020-10-02
would destroy zroot/var/db/couchdb@daily-2020-10-03
would destroy zroot/var/db/couchdb@weekly-2020-39
would destroy zroot/var/db/couchdb@daily-2020-10-04
would destroy zroot/var/db/couchdb@daily-2020-10-05
would destroy zroot/var/db/couchdb@daily-2020-10-06
would destroy zroot/var/db/couchdb@daily-2020-10-07
would destroy zroot/var/db/couchdb@daily-2020-10-08
would destroy zroot/var/db/couchdb@hourly-2020-10-08-05
would destroy zroot/var/db/couchdb@hourly-2020-10-08-06
would destroy zroot/var/db/couchdb@hourly-2020-10-08-07
would destroy zroot/var/db/couchdb@hourly-2020-10-08-08
would destroy zroot/var/db/couchdb@hourly-2020-10-08-09
would destroy zroot/var/db/couchdb@hourly-2020-10-08-10
would destroy zroot/var/db/couchdb@cleanup
would destroy zroot/var/db/couchdb/views@20200318-2345:11.2-RELEASE-p15
would destroy zroot/var/db/couchdb/views@20200319-2046:11.3-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200421-1714:11.3-RELEASE-p7
would destroy zroot/var/db/couchdb/views@20200423-1355:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb/views@20200513-1150:11.3-RELEASE-p8
would destroy zroot/var/db/couchdb/views@20200609-0757:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb/views@20200609-0804:11.3-RELEASE-p9
would destroy zroot/var/db/couchdb/views@20200709-0557:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200709-0710:12.1-RELEASE-p6
would destroy zroot/var/db/couchdb/views@20200910-0845:12.1-RELEASE-p7
would destroy zroot/var/db/couchdb/views@20201001-0809:12.1-RELEASE-p9
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-05
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-06
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-07
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-08
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-09
would destroy zroot/var/db/couchdb/views@hourly-2020-10-08-10
would destroy zroot/var/db/couchdb/views@cleanup
would reclaim 81.9GOK this looks good, so let rip:
# zfs destroy -vr zroot/var/db/couchdb@%cleanup
will destroy zroot/var/db/couchdb@20200421-1714:11.3-RELEASE-p7
will destroy zroot/var/db/couchdb@20200423-1355:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb@20200513-1150:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb@20200609-0757:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb@20200609-0804:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb@20200709-0557:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb@20200709-0710:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb@monthly-2020-09
will destroy zroot/var/db/couchdb@weekly-2020-35
will destroy zroot/var/db/couchdb@20200910-0845:12.1-RELEASE-p7
will destroy zroot/var/db/couchdb@weekly-2020-36
will destroy zroot/var/db/couchdb@weekly-2020-37
will destroy zroot/var/db/couchdb@weekly-2020-38
will destroy zroot/var/db/couchdb@monthly-2020-10
will destroy zroot/var/db/couchdb@20201001-0809:12.1-RELEASE-p9
will destroy zroot/var/db/couchdb@daily-2020-10-02
will destroy zroot/var/db/couchdb@daily-2020-10-03
will destroy zroot/var/db/couchdb@weekly-2020-39
will destroy zroot/var/db/couchdb@daily-2020-10-04
will destroy zroot/var/db/couchdb@daily-2020-10-05
will destroy zroot/var/db/couchdb@daily-2020-10-06
will destroy zroot/var/db/couchdb@daily-2020-10-07
will destroy zroot/var/db/couchdb@daily-2020-10-08
will destroy zroot/var/db/couchdb@hourly-2020-10-08-05
will destroy zroot/var/db/couchdb@hourly-2020-10-08-06
will destroy zroot/var/db/couchdb@hourly-2020-10-08-07
will destroy zroot/var/db/couchdb@hourly-2020-10-08-08
will destroy zroot/var/db/couchdb@hourly-2020-10-08-09
will destroy zroot/var/db/couchdb@hourly-2020-10-08-10
will destroy zroot/var/db/couchdb@cleanup
will destroy zroot/var/db/couchdb/views@20200318-2345:11.2-RELEASE-p15
will destroy zroot/var/db/couchdb/views@20200319-2046:11.3-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200421-1714:11.3-RELEASE-p7
will destroy zroot/var/db/couchdb/views@20200423-1355:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb/views@20200513-1150:11.3-RELEASE-p8
will destroy zroot/var/db/couchdb/views@20200609-0757:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb/views@20200609-0804:11.3-RELEASE-p9
will destroy zroot/var/db/couchdb/views@20200709-0557:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200709-0710:12.1-RELEASE-p6
will destroy zroot/var/db/couchdb/views@20200910-0845:12.1-RELEASE-p7
will destroy zroot/var/db/couchdb/views@20201001-0809:12.1-RELEASE-p9
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-05
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-06
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-07
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-08
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-09
will destroy zroot/var/db/couchdb/views@hourly-2020-10-08-10
will destroy zroot/var/db/couchdb/views@cleanup
will reclaim 81.9GLooking Good
Note that free space is returned lazily to the pool depending on io usage, so don’t be surprised if 100+GiB takes a while to re-appear.
Once you’ve finished tidying up, feel free to trash the temporary
cleanup dataset, recursively.
# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 230G 109G 121G - - 56% 47% 1.00x ONLINE -
ada0p3 230G 109G 121G - - 56% 47%
# zfs destroy -vr zroot@cleanupCleaning up a dataset with dependent clones
The zroot/ROOT/default dataset contains the / filesystem, and after
every upgrade or update, will have a cloned and snapshotted “boot
environment” left behind.
# zfs destroy -vnr zroot/ROOT/default@%cleanup
cannot destroy 'zroot/ROOT/default@2020-10-01-08:38:52-0': snapshot has dependent clones
use '-R' to destroy the following datasets:
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-03
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-10
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-09
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-04
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-07
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-07
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-08
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-05
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-02
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-08
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-05
zroot/ROOT/12.1-RELEASE-p10@daily-2020-10-06
zroot/ROOT/12.1-RELEASE-p10@weekly-2020-39
zroot/ROOT/12.1-RELEASE-p10@cleanup
zroot/ROOT/12.1-RELEASE-p10@hourly-2020-10-08-06
zroot/ROOT/12.1-RELEASE-p10
would destroy zroot/ROOT/default@2019-10-21-14:12:40
...
would destroy zroot/ROOT/default@hourly-2020-10-08-10
would destroy zroot/ROOT/default@cleanupThese come from the boot environment - an amazing way to switch between boot systems in case of failure.
# bectl list
BE Active Mountpoint Space Created
12.1-RELEASE-p10 - - 132K 2020-10-01 08:38
default NR / 14.7G 2020-02-17 12:52
#
The default BE has NR flags set - Now and Reboot - its the current
BE and will also be used on next reboot.
We can safely destroy the previous one, but note that the snapshot used to clone the BE still remains:
# bectl destroy 12.1-RELEASE-p10
bectl destroy: leaving origin 'zroot/ROOT/default@2020-10-01-08:38:52-0' intact
# Let’s re-try that destroy:
# zfs destroy -vnr zroot/ROOT/default@%cleanup
would destroy zroot/ROOT/default@2019-10-21-14:12:40
would destroy zroot/ROOT/default@monthly-2019-12
...
would destroy zroot/ROOT/default@cleanup
would reclaim 6.32G
#