Setting up the NFS server in a vnet jail.

This is an early draft of a document that will
definitely change.

At this time, it is possible to enable NFSv3 and/or NFSv4
over TCP, but not NFSv3 over UDP.  I do not see this as
a major issue, since NFSv3 over UDP is deprecated these
days anyhow.  This does mean that the "-u" option
cannot be specified for nfs_server_flags in the prison's
/etc/rc.conf.  pNFS server configurations (-p, -m) are not supported.
Also, you cannot run lockd, so if you
need file locking to be visible across multiple clients,
you should use NFSv4.

You will need to install an up to date FreeBSD14/main
system or an up-to-date stable/13 system.

Your kernel configuration will need to include
options VIMAGE
to enable running of nfsd(8) and friends in a vnet jail.
If "options NFSD" is not in your kernel configuration,
you will need to have "nfsd" in a kld_list line in
/etc/rc.conf outside of the prison.
Also, if you are going to support Kerberized NFS in
the prison you either need "options KGSSAPI" in
your kernel configuration or you need to put
"kgssapi kgssapi_krb5" in kld_list in /etc/rc.conf
outside of the jails.
If you are going to support nfs-over-tls in the prison
and you are running a FreeBSD13 system, you also need
"ktls_ocf" in kld_list in /etc/rc.conf outside of the
jails.
(This is because kld_loads cannot be done inside the
prison during startup so the load must be done before jail startup.)

Then, on the system (outside of the jail), the
following needs to be done:
- The jail must be in a separate file system, or put
  another way, the root directory of the jail must
  be a file system mount point.  This is necessary so
  that mountd can hang exports on the file system
  mount point.

- Add the line "allow.nfsd;" to your /etc/jail.conf,
  which must be a vnet jail.

- If you wish to export file systems mounted below
  the root of the prison, you need to set enforce_statfs
  to "1" in your /etc/jail.conf.
  If enforce_statfs is "2", the prison's file system may
  be exported, but no file system mounted below the prison's
  root can be exported.  Setting enforce_statfs to "0"
  will not work.

Once the above is done, configuring a NFS server in the jail
is no different that outside of the jails, except that the
"-u, -p and -m" command line options for nfsd(8) cannot be used.
For example..
A /etc/exports file is needed.
A simple example for a jail in a single file system,
might look like:
/ -alldirs -sec=sys -network 192.168.1.0/24
V4: / -sec=sys -network 192.168.1.0/24

Now, add the following lines to /etc/sysctl.conf,
so that user/group strings are just the uid/gid
numbers on the wire.
vfs.nfs.enable_uidtostring=1
vfs.nfsd.enable_stringtouid=1
Alternately, you can run the nfsuserd(8) daemon in the vnet jail.
To run the nfsuserd(8) daemon, add:
nfsuserd_enable="YES"
to the /etc/rc.conf in the jail.
Usually running nfsuserd(8) is not required and is just unnecessarily complex,
for sec=sys NFSv4 mounts.  It is needed for Kerberized NFS mounts.

In the jail's /etc/rc.conf (for a NFSv4 only server):
  nfs_server_enable="YES"
  nfsv4_server_only="YES"
  nfs_server_flags="-t"

For a server configuration that supports NFSv3 as
well as NFSv4, replace:
  nfsv4_server_only="YES"
with
  nfsv4_server_enable="YES"

For NFS-over-TLS and/or Kerberized NFS you will also
need lines to enable tlsservd and/or gssd.  Same goes
for nfsuserd, if you are using it.

Note that "nfsstat -E -s" will report stats for the nfsds in the jail
you are in.  Also, any "vfs.nfsd.*" sysctls
that need to be set must be done in /etc/sysctl.conf
outside the jails and affect all nfsd.
The only vfs.nfsd sysctls set within the jail(s) at
this time are:
   vfs.nfsd.server_min_nfsvers
   vfs.nfsd.server_max_nfsvers
   vfs.nfs.enable_uidtostring
   vfs.nfsd.enable_stringtouid
   vfs.nfsd.fha.enable
   vfs.nfsd.fha.read
   vfs.nfsd.fha.write
   vfs.nfsd.fha.bin_shift
   vfs.nfsd.fha.max_nfsds_per_fh
   vfs.nfsd.fha.max_reqs_per_nfsd
--> Some more of these may be moved into the jails, if
    experience dictates a need to do so.

Normally a client mount to the vnet jail nfsd should
The client mount to a vnet jail nfsd will need to use
a path relative to the jail's root.  For example:
# mount -t nfs -o nfsv4 nfs-jail-server:/ /mnt

Good luck with it and feel free to post suggestions for
improvement of either this document or changes to the
nfsd in a vnet jail work.