How to setup kerberized NFS on FreeBSD (particularily NFSv4) Setting up Kerberized NFS (more correctly: NFS using RPCSEC_GSS authentication via the kerberos5 mechanism) on FreeBSD: For the example: nfs-server.my.domain - The FQDN of the NFS server nfs-client.my.domain - The FQDN of the NFS client MY.REALM - The default Kerberos realm ricktst - A user assigned uid 502, with a Kerberos principal name ricktst@MY.REALM /export - The single file system being exported to the client In the GSSAPI there are two kinds of principal names, User and Host Based. These names are translated to Kerberos principal names for Kerberos mechanism, which is the only mechanism used. A User principal name normally refers to a user and looks like: ricktst <-- which becomes ricktst@MY.REALM in KerberosV It normally has a TGT in a credentials cache file (/tmp/krb5cc_502) created by kinit or during the user's login process. A Host Based principal name also has the fully qualified host domain name (FQDN) in it (so it can only be used on that host) and looks like: nfs@nfs-server.my.domain <-- which becomes nfs/nfs-server.my.domain@MY.REALM in KerberosV It normally has credentials in a keytab file created on the KDC and copied to the machine's /etc/krb5.keytab via some secure mechanism. (These are sometimes referred to as a machine credentials, or in Kerberos, a service principal. Note that the NFS client can (mis)use a service principal as a client machine credential, referred to as a host based initiator credential, below.) (Since this principal will only work on the system called nfs-server.my.domain, the damage done if the keytab file is compromised on nfs-server.my.domain it is limited to that system.) These principals have the advantage that their credentials do not expire, because they can be refreshed indefinitely. For the NFS server, the following principal will need to be added to the KDC and a keytab entry for it will need to be generated and copied to /etc/krb5.keytab on the NFS server (the first component must be "nfs"): nfs/nfs-server.my.domain@MY.REALM Section 14.5 of the FreeBSD handbook does a nice job of describing how to set up the Heimdal kdc that is distributed with FreeBSD and the use of kadmin to acquire the above keytab entry. (However, note that "service kdc start" is used to start the kdc and not what the handbook currently says.) The following should work for Heimdal and MIT KDCs, but does not work for Windows AD KDCs. Once this is done, the credential may be tested by doing: # kinit -k nfs/nfs-server.my.domain on the NFS server as su/root. For the NFS client, the same is often done to create a host based initiator credential, although the first component does not need to be "nfs" and is usually "host". For example, create the principal and a keytab entry for: host/nfs-client.my.domain@MY.REALM on the KDC and then copy the keytab to /etc/krb5.keytab on nfs-client and test it via: # kinit -k host/nfs-client.my.domain on the client, running as su/root. (There are ways to do mounts without this host based initiator credential. See below.) For RPCSEC_GSS, there are 3 types of service which, for KerberosV are: krb5 - Use KerberosV for user authentication, but only protect the RPC header from compromise. krb5i - Use KerberosV for user authentication, but also use encrypted checksums on the RPC data to protect against "man in the middle" attacks involving replacement of the RPC data. krb5p - Use KerberosV for user authentication, but also encrypt the RPC data, so that it isn't on the wire in clear text. As an alternative to krb5i/krb5p, NFS-over-TLS may be used with krb5 to provide integrity/privacy. See: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Since Kerberized NFS does not use uid/gids, these do not need to be uniform across the client(s) and server(s). However, uniform user/group name databases across all client(s) and server(s) is strongly recommended. Example server setup (after confirming the above Kerberos config): Create a /etc/exports file with the following lines in it: /export -sec=krb5:krb5i:krb5p -network 192.168.1.0 -mask 255.255.255.0 V4: /export -sec=krb5:krb5i:krb5p -network 192.168.1.0 -mask 255.255.255.0 This allows all the client(s) to use any of the three RPCSEC_GSS types of service. The client(s) are limited to the local 192.168.1.0 subnet. (Prior to FreeBSD 13.2, krb5i is always needed in the V4: line for Linux client mounts, since the Linux client always uses krb5i for state management operations.) Add the following lines to your /etc/rc.conf: nfsuserd_enable="YES" gssd_enable="YES" nfs_server_enable="YES" nfsv4_server_enable="YES" mountd_enable="YES" If you are running FreeBSD 12.3 or 13.0 (and maybe others?), you should apply this simple patch to your /etc/rc.d/gssd file. (All client(s) and server(s).) --- etc/rc.d/gssd.old 2022-02-13 07:53:46.456616000 -0800 +++ etc/rc.d/gssd 2022-02-13 13:34:01.662048000 -0800 @@ -4,7 +4,8 @@ # # PROVIDE: gssd -# REQUIRE: root mountcritlocal +# REQUIRE: root mountcritlocal NETWORKING kdc +# BEFORE: mountcritremote # KEYWORD: nojail shutdown . /etc/rc.subr Then reboot the server and it should be ready to handle Kerberized NFS mounts. (If you are restarting the daemons without rebooting, you must stop both the gssd and nfsd and then start the gssd before nfsd.) Example client setup (after confirming the above Kerberos config): Add the following lines to your /etc/rc.conf: nfsuserd_enable="YES" gssd_enable="YES" # -h is only needed for the host based initiator credential mounts gssd_flags="-h" nfs_client_enable="YES" nfscbd_enable="YES" Apply the above patch to /etc/rc.d/gssd, as required. If your DNS is misconfigured, there may be a long delay of over 25 seconds during the mount attempt. Unless you have a kernel with the following patch, the 25 second timeout on the kernel upcall to the gssd(8) will happen and the gssd(8) daemon will terminate. A similar delay will occur when a user's TGT has expired. (The GSSAPI library seems to attempt to contact a DNS service even if /etc/nsswitch.conf specifies "files" for the hosts line, if there is a /etc/resolv.conf file on the system.) If you run into this problem, you can: - Delete the /etc/resolv.conf file. or - Fix DNS so that it works. or - Apply this patch to your kernel and rebuild it from sources. (After applying the patch, the mount can take 30 seconds, but it will succeed and the gssd will not get terminated.) https://people.freebsd.org/~rmacklem/gssdtimeout.patch Reboot and the client should be ready to do a Kerberized NFS mount. There are multiple ways the mount can be done. To do a mount using a host based initiator credential, as su/root: (This is the one where the client has a host/nfs-client.my.domain@MY.REALM principal entry in its /etc/krb5.keytab file.) # mount -t nfs -o nfsv4,sec=krb5,gssname=host nfs-server.my.domain:/ /export Since an entry for the principal host/nfs-client.my.domain@MY.REALM is being used for the client's machine credential, this mount should continue to work until unmounted. It does require that the client machine have a fixed, well known DNS host name (nfs-client.my.domain for this example). It also requires an entry for host/nfs-client.my.domain@MY.REALM be in the client system's /etc/krb5.keytab. To do a mount without a host based initiator credential: For FreeBSD 14 (main) or an up-to-date stable/13 system, you may use the "syskrb5" option on NFSv4.1/4.2 mounts so that system maintenance operations are done using AUTH_SYS. As such, the client does not need to have a host/nfs-client.my.domain@MY.REALM entry in the system's /etc/krb5.keytab nor a fixed, well known DNS name. Use of the "tls" mount option with this mount option is strongly recommended, to provide privacy and, optionally, client machine credentials via a X.509 certificate. (See https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt.) # mount -t nfs -o nfsv4,sec=krb5,syskrb5 nfs-server.my.domain:/ /export Alternately, the mount can be done using a TGT held by the user doing the mount. # kinit ricktst - Input ricktst's Kerberos password at the prompt. # klist - Should show a valid TGT for "ricktst". # mount -t nfs -o nfsv4,sec=krb5 nfs-server.my.domain:/ /export So long as there is a valid TGT for any principal in the KDC in root's credential cache, this mount should work until that TGT expires. It is possible to do this kind of mount as non-root. First, as root, do: # sysctl vfs.usermount=1 Then, login as "ricktst" and: $ kinit - Input ricktst's Kerberos password at the prompt. $ klist - Should show a valid TGT for "ricktst". $ mount -t nfs -o nfsv4,sec=krb5 nfs-server.my.domain:/ /export - Note that for this to work, /export must be owned by "ricktst". Also, "umount /export" might take 1 minute, since it must time out the Mount protocol Unmount RPC, if you have a NFSv4 only server. Mounts without "gssname" and "syskrb5" will stop working the TGT expires. However, mounts using a user's TGT or done with the "syskrb5" option does not require the client to have a fixed, well known DNS host name (a laptop using WIFI, for example) nor an entry in /etc/krb5.keytab. If you, for some reason, choose to switch between these two types of Kerberized mounts, you will need to do: # umount /export # kdestroy before doing the other style of mount, so that there is no "cruft" in the credentials cache. Once the mount is done, users will need to have valid TGTs to access files on the /export mount. Note that, if there are multiple valid TGTs in a user's credential cache, I do not know which one will be used. I recommend that only one valid TGT be in a user's credential cache when accessing Kerberized NFS mount points. (A "kdestroy" followed by a "kinit" should fix any case of multiple TGTs.) Here are some gotchas to be aware of: - If you are doing a NFSv4.1/4.2 mount to a pNFS server, you cannot use the "pnfs" mount option with "sec=krb5[ip]" unless you have a client system built from sources at least as recent as Nov. 2023. (A bug that resulted in the "service principal" for the MDS being used for the DSs was fixed in main late Oct. 2023.) - Linux clients use krb5i for state maintenance operations, no matter which variant of sec=krb5[ip] you specify in the mount. As such, krb5i will need to be in the export V4: line for the mount to work unless you are running a FreeBSD 13.2 or newer server. - The time of day clocks for all systems must be synchronized to within the clock skew specified in /etc/krb5.conf. - KerberosV principal names are case sensitive, although DNS names are not. The simplest way to avoid grief is to use all lower case characters in your DNS host names and all upper case characters for your KerberosV REALM name. - The host name resolver functions must return the fully qualified host name i.e. nfs-server.my.domain and not nfs-server. If you are using /etc/hosts, put the fully qualified name first, like: 192.168.1.12 nfs-server.my.domain nfs-server If the mount is not working: Add the "-v" command line option to gssd. The gssd daemon should now log information wherever your /etc/syslog.conf directs daemon.info. If there are large negative numbers in the logged messages, they are Kerberos errors that can be found in /usr/include/krb5_err.h. If you can access it, the KDC should log information that may be useful. Thats about all I can think of w.r.t. diagnostics. Good luck with it, rick --- This is an email that Peter Eriksson posted with good information. Copied here with his permission. We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos environment). There are a few differences in behaviour (from a Solaris perspective), mainly regarding the pseudo NFSv4 filesystem but not something that can't be handled. In the process of moving to FreeBSD based servers I've also been testing lots of different clients in order to make sure everything works as expected. Here are some notes: General stuff: 1. Have a _kerberos.my.zone.com DNS TXT record containing the Kerberos realm (nice to have) 2. Have a _nfsv4domain.my.zone.com DNS TXT record containing the NFSv4 “domain” (not all clients use it, but it’s nice to have) * FreeBSD server (with ZFS filesystems), things below /export is NFS-shared as (for example) server:/staff/user1 1. /etc/exports (we _only_ allow sec=krb for home directories): V4: /export -sec=krb5:krb5i:krb5p Or (on a server also serving some (read-only) sec=sys filesystems below /exports) V4: /export -sec=krb5:krb5i:krb5p:sys 2. /etc/zfs/exports (generated from set sharenfs=xxx on the ZFS filesystems) Home-server: /export/staff -sec=krb5:krb5i:krb5p /export/staff/user1 -sec=krb5:krb5i:krb5p /export/staff/user2 -sec=krb5:krb5i:krb5p /export/students -sec=krb5:krb5i:krb5p /export/students/user3 -sec=krb5:krb5i:krb5p Software-server: /export/db/icons -sec=sys -ro /export/old/ifm/solaris-x86 -sec=krb5:krb5i:krb5p:sys -ro /export/old/ifm/windows-86 -sec=krb5:krb5i:krb5p:sys -ro /export/software -sec=krb5:krb5i:krb5p -ro 3. Make sure you have host/FQDN@REALM and nfs/FQDN@REALM Kerberos principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts like: 1.2.3.4 foo.bar.com foo 4. rc.conf (we have a lot of users in our AD so we have to use a large number for usermax, replace liu.se with your NFSv4 domain for nfsuserd_flags) gssd_enable="YES" nfs_server_enable="YES" nfsv4_server_enable="YES" nfscbd_enable="YES" mountd_enable="YES" nfsuserd_enable="YES" nfsuserd_flags="-manage-gids -domain liu.se -usertimeout 10 -usermax 100000 16" 5. Make sure you use NTPD so the clock is correct. * All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested): 1. Make sure FQDN is in /etc/hosts 2. Make sure you use NTPD so the clock is correct. 3. Have a host/FQDN@REALM Kerberos host principal in /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work) 4. We use a fairly default /etc/krb5.conf, sort of like: [libdefaults] default_realm = REALM dns_lookup_realm = true ticket_lifetime = 24h renew_lifetime = 7d forwardable = true default_ccache_name = KEYRING:persistent:%{uid} KEYRING probably only works on Linux and there are some problems with KEYRING in Debian & Ubuntu since not everything in it supports it due to them using Heimdal instead of MIT (like for smbclient) but it mostly works. Works fine in CentOS 7 though - in general CentOS 7 feels more enterprise-ready than Debian & Ubuntu. The old classic FILE-ccaches should work fine though. For mounting we use the automounter and a "executable map (perl script) that looks up records in DNS (Hesiod-style) since the built-in Hesiod support in most automounters is a bit.. lacking. Works quite well. You can find the scripts we use here: http://www.grebo.net/~peter/nfs (The dns-update scripts use data from an SQL database so probably isn't directly usable to anybody else. We use the same SQL database to populate a locally developed BerkeleyDB-based NSS-database on each FreeBSD server in order to speed things up since AD/LDAP-looks with ~90k users and silly amounts of AD groups takes forever, even with cacheing). Some Linux-specific stuff: Packages needed: CentOS: - nfs-utils - libnfsidmap - nfs4-acl-tools - autofs Debian: - keyutils - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian Ubuntu: - keyutils Other nice-to have packages: - hesiod - autofs-hesiod Some settings to check for: /etc/default/nfs-common: NEED_IDMAPD=yes NEED_GSSD=yes /etc/idmapd.conf (replace liu.se with your NFSv4 domain): Domain=liu.se /etc/request-key.d/id_resolver.conf (should be there already if using a modern Linux and you've added the packages above): create id_resolver * * /usr/sbin/nfsidmap %k %d MacOS: Basically require the latest - 10.14 (Mojave) - for things to work smoothly. In theory 10.12 & 10.13 should work but there is some bug in them that causes the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier doesn't support good enough encryption for us. But with 10.14 you just need to get a Kerberos ticket and then you can mount things just fine. /etc/nfs.conf should contain (replace liu.se with your NFSv4 domain): nfs.client.default_nfs4domain=liu.se (There are a lot of problems you can run into with Microsofts AD implementation of Kerberos too that we've had to be fighting with, but that's a whole other topic) - Peter