FreeBSD Test Labs Application Note 1010.R-DISKLESS ====================================================================== (c) 1998 FreeBSD Test Labs, all rights reserved. Summary ------- This FTL Application Note describes the configuration process for diskless FreeBSD workstations and presents some simple techniques for minimising the administrative workload involved in managing large groups of such systems. FreeBSD systems are typically operated diskless in situations where the computing hardware is shared amongst multiple users with different operating system requirements, and in other environments where disk space on a system is unavailable or undesirable. In all cases, at least one server system must be available. This system must be able to provide BOOTP and NFS services. TFTP can also be optionally used by the client to obtain some configuration information. This Application Note will focus on the use of FreeBSD servers, however other platforms capable of providing the above services may also be used (eg. NetBSD, HP/UX, Solaris, AIX, etc.), with some reservations as detailed below. Applicable FreeBSD Versions --------------------------- FreeBSD 2.2.6 and above. Server Configuration -------------------- Servers for diskless clients must provide two separate service types: - Boot-time configuration information (BOOTP, TFTP). - Runtime filesystems and swap (NFS). There is no requirement that these two service types be provided only by a single server, nor any reason that they may not be. Further, if multiple NFS filesystems are used, they may be served by different servers. See the Application Note 1000-BOOTP_TFTP for information on configuring the BOOTP and TFTP services required to boot a diskless workstation. Server Filesystem Configuration ------------------------------- Minimal operation of a diskless workstation requires a root filesystem. In the simplest case, each diskless workstation has its own NFS-exported root filesystem in order to have workstation-specific configuration data (normally kept in /etc) unique on a per-workstation basis. To produce a suitable workstation root filesystem, a number of approaches can be taken. The easiest is to clone the root filesystem from the server, however this often results in undesired extra or incorrect information being included. - The recommended approach is to mount a FreeBSD CDROM, or obtain otherwise the collection of bin.?? files from the FreeBSD distribution which will be run on the workstations. These should be unpacked in a scratch directory using the following command: # cat /bin.* | tar xvzpf - DO NOT use the 'install.sh' script included with the distribution, as it will overwrite the running system, which is rarely the desired effect. The extraction command, and all others in this section, must be executed as root in order to preserve the permissions on extracted and moved files. The result of this extraction will be the following directories: bin dev etc lkm mnt proc root sbin usr var - Move the usr directory to a suitable location and arrange for it to be NFS exported read-only. This directory can be safely shared amongst multiple diskless workstations, greatly reducing the required disk space. The other directories should be moved into a safe storage location for use as a template for creating client root filesystems. If only one workstation is being supported these steps are not required, and the entire extracted tree can be used as-is as a root filesystem. Care should be taken when moving these filesystems to ensure that permission and ownership information on files is not lost. Failure to maintain this information may lead to poor security on client systems. - Create a directory to hold the root filesystem for each client, and copy the directories from the template into each. Note that because the dev directory contains device nodes, cp(1) cannot be used for this. Instead, use the tar(1) command in a pipeline to populate the client directory thus: # cd # (cd ; tar cpf - .) | tar xvpf - If the root and usr filesystems are to be exported from separate machines, they can be bundled using tar(1) in the normal fashion. - Edit the configuration files in each client's etc directory to reflect the desired client configuration. At least the following files will need to be adjusted: /etc/rc.conf - Set local hostname, IP address, options. /etc/fstab - Add location of /usr filesystem. - Create a swapfile for each client. FreeBSD clients are passed the location of the swapfile directory via BOOTP/TFTP during the early boot phase, and look for a swap file named swap. in that directory. The normal rules for sizing swap apply to diskless clients, although swapping on network connections imposes such a performance hit that it is desirable to avoid it wherever possible. To create a 32MB swapfile for a client whose IP address is 10.2.3.20, use the command: # dd if=/dev/zero of=swap.10.2.3.20 bs=1m count=32 Note that swapfiles cannot be shared between clients. - Install a kernel suitable for each client system in the client root directories. - Arrange for the client root and usr filesystems, any other NFS filesystems and the swapfile directory to be exported for the clients. Sample NFS Exports ------------------ The following configuration file excerpts describe a server providing filesystems to a set of clients, and the configuration of these clients. - There are 5 clients, named client1-5 - Each client has a private root filesystem, each mounts a shared /usr filesystem, and each mounts a shared work filesystem from the server. Server, /etc/exports: /a/root1 -maproot=0 client1 /a/root2 -maproot=0 client2 /a/root3 -maproot=0 client3 /a/root4 -maproot=0 client4 /a/root5 -maproot=0 client5 /a/usr -maproot=0 -ro client1 client2 client3 client4 client5 /b/swapfiles -maproot=0 client1 client2 client3 client4 client5 /c/public -maproot=nobody client1 client2 client3 client4 client5 Client1, /etc/fstab: (other clients substitute client number) server:/a/root1 / nfs rw,soft,bg,intr 0 0 server:/a/user /usr nfs ro,soft,bg,intr 0 0 server:/c/public /work nfs rw,soft,bg,intr 0 0 Advanced Configuration Techniques --------------------------------- Reducing Client Root Filsystem Space Usage: It is possible to use hard links to greatly reduce the amount of disk space required for multiple client root filesystems. In order to achieve this, the template directory and all client root directories must be on the same filesystem. Because most files are shared between all clients, this requires much less space (typically not much more than a single copy of the template). Begin as normal with the template directory and one directory per client. Populate the client directory as above, in order to ensure the correct permissions for device nodes and directories. Execute the following commands (as root): # cd # ln -f /bin/* bin/ # ln -f /sbin/* sbin/ Note that this has the side effect that changes to permissions of any file in /bin or /sbin in the template or on any client will affect all other clients. This procedure can be modified to share common configuration files amongst diskless clients. For example, the procedure # cd # ln -f /etc/resolv.conf etc/resolv.conf will result in the client sharing the template copy of the /etc/resolv.conf file. For a site with large numbers of clients this can greatly reduce the amount of manual maintenance required In many cases, the entire /etc/ directory with the exception of rc.conf can be shared, as for the bin and sbin directories above. To take a file that has been shared and make it private to a given client, proceed as follows: # cd # cp -p temp.file # rm # mv temp.file This can be used to make a private copy of the rc.conf file after sharing the etc directory. Likewise, kernels can be shared. If more than one kernel type is in use, place a copy of each in the template root filesystem, naming each such that they can be readily identified. For each client, make a link to the appropriate kernel, eg.: # cd # ln -f /kernel.type1 . There are a number of points to consider when doing this: - The 'dset' program run during client startup will try to modify the booted kernel if changes were made at boot time using userconfig. If kernels are shared, this can cause problems. Removing the USERCONFIG option from the kernel configuration or removing the /sbin/dset program from the clients' root filesystems are effective workarounds. - The kernel is normally installed with the schg (system immutable) flag set. It is not possible to create a link to a file which has this attribute, so it should be temporarily removed from the kernel images while links are being made for clients. Once the links have been made, the flag should be reset. eg.: # cd # chflags noschg kernel.* [... make client kernel links ...] # chflags schg kernel.* Note that due to the way that the root filesystems are mounted symbolic links cannot be used outside of a client's filesystem. Sharing Root Filesystems: It is feasible to share a single root filesystem between multiple clients. The exact details of the procedure involved are outside the scope of this document, however the basic steps required are: - Have separate /var and /tmp filesystems for each client. - Supply the hostname via BOOTP or TFTP and use it early in the boot process to locate client-specific configuration data (normally /etc/rc.conf and /etc/fstab). Implementing this requires minor modifications to the client startup file /etc/rc. Using Local Swap: If a small local disk partition is available on a client workstation it can be used as an alternative to swapping across the network. In many cases this will lead to improved client performance. Each local disk must be prepared for swapping as follows. Note that local swap (unlike boot filesystems) may be located anywhere, on any disk. In this example, a small partition has been created on the first IDE disk on the workstation. This can be achieved using the freeware FIPS utility, or a commercial package such as Partition Magic. The disk was previously entirely occupied by another operating system. This partition is referred to as /dev/wd0s2, and it is possible to swap directly to the partition without requiring a filesystem at all. Remove any reference to swap from the BOOTP/TFTP-loaded configuration, and in the client's /etc/fstab file, add the entry /dev/wd0s2 none swap sw 0 0 Cautions for Non-FreeBSD Servers -------------------------------- Some non-FreeBSD systems are unable to correctly represent the device numbers used for some FreeBSD devices. This normally prevents the use of local disks, as the device nodes affected are those representing disk devices. Thus, use of local swap is incompatible with these systems. Servers known to suffer from this problem include Digital Unix (all versions). References ---------- The Application Note 1000-BOOTP_TFTP discusses the configuration of BOOTP and TFTP services on a FreeBSD host for the purpose of booting diskless FreeBSD clients. dd(1) ln(1) tar(1) exports(5) fstab(5) dset(8)