Setting up the nfsd in a vnet prison.

This is an early draft of a document that will
definitely change.

At this time, it is possible to enable NFSv3 and/or NFSv4
over TCP, but not NFSv3 over UDP.  I do not see this as
a major issue, since NFSv3 over UDP is deprecated these
days anyhow.  This does mean that the "-u" option
cannot be specified for nfs_server_flags in the prison's
/etc/rc.conf.  pNFS server configurations (-p, -m) are not supported.
Also, you cannot run lockd, so if you
need file locking to be visible across multiple clients,
you should use NFSv4.

You will need to install an up to date FreeBSD14/main
system.

Your kernel configuration will need to include
options VIMAGE
to enable running of nfsd(8) and friends in a prison.
If "options NFSD" is not in your kernel configuration,
you will need to have "nfsd" in a kld_list line in
/etc/rc.conf outside of the prison.
Also, if you are going to support Kerberized NFS in
the prison you either need "options KGSSAPI" in
your kernel configuration or you need to put
"kgssapi kgssapi_krb5" in a kld_list entry in /etc/rc.conf
outside of the prisons.
(This is because kld_loads cannot be done inside the
prison during startup.)

Then, on the system (outside of the prison), the
following needs to be done:
- The prison must be in a separate file system, or put
  another way, the root directory of the prison must
  be a file system mount point.  This is necessary so
  that mountd can hang exports on the file system
  mount point.

- Add the line "allow.nfsd;" to your /etc/jail.conf,
  which must be a vnet prison.

- If you wish to export file systems mounted below
  the root of the prison, you need to set enforce_statfs
  to "1" in your /etc/jail.conf.
  If enforce_statfs is "2", the prison's file system may
  be exported, but no file system mounted below the prison's
  root can be exported.  Setting enforce_statfs to "0"
  will not work.

Once the above is done, configuring a NFS server in the prison
is no different that outside the prison, except that the
"-u, -p and -m" command line options for nfsd(8) cannot be used.
For example..
A /etc/exports file is needed.
A simple example for a jail in a single file system,
might look like:
/ -alldirs -sec=sys -network 192.168.1.0/24
V4: / -sec=sys -network 192.168.1.0/24

Now, add the following lines to /etc/sysctl.conf,
so that user/group strings are just the uid/gid
numbers on the wire.
vfs.nfs.enable_uidtostring=1
vfs.nfsd.enable_stringtouid=1
Alternately, you can run the nfsuserd(8) daemon in the vnet prison.
To run the nfsuserd(8) daemon, add:
nfsuserd_enable="YES"
to the /etc/rc.conf in the prison.
Usually running nfsuserd(8) is not required and is just unnecessarily complex,
for sec=sys NFSv4 mounts.  It is needed for Kerberized NFS mounts.

In the prison's /etc/rc.conf (for a NFSv4 only server):
  nfs_server_enable="YES"
  nfsv4_server_only="YES"
  nfs_server_flags="-t"

For a server configuration that supports NFSv3 as
well as NFSv4, replace:
  nfsv4_server_only="YES"
with
  nfsv4_server_enable="YES"

For NFS-over-TLS and/or Kerberized NFS you will also
need lines to enable tlsservd and/or gssd.  Same goes
for nfsuserd, if you are using it.

Note that "nfsstat -E -s" will report stats for the nfsds in the prison
you are in.  Also, any "vfs.nfsd.*" sysctls
that need to be set must be done in /etc/sysctl.conf
outside the prisons and affect all nfsd.
The only vfs.nfsd sysctls set within the prison(s) at
this time are the following ones manipulated by the
scripts in etc/rc.d:
   vfs.nfsd.server_min_nfsvers
   vfs.nfsd.server_max_nfsvers
   vfs.nfsd.nfs_privport
If you applied the vnetsmall-nfsuserd.patch:
   vfs.nfs.enable_uidtostring
   vfs.nfsd.enable_stringtouid
are also in the vnet prisons.
--> Some more of these may be moved into the prisons, if
    experience dictates a need to do so.

Normally a client mount to the vnet prison nfsd should
use "/" as its path.  For example:
# mount -t nfs -o nfsv4 nfs-prison-server:/ /mnt

Good luck with it and feel free to post suggestions for
improvement of either this document or changes to the
nfsd in a vnet prison work.

rick