Unofficial Advisory Title: Regression in FreeBSD >8.x BIOS boot code Author: Bruce Simpson Date: 2014-12-09 (Rev 1.4) Summary ------- We have observed that with FreeBSD releases after 8.4, the system may fail to boot on certain machines. Affected systems --------------- This has been observed with the following systems and configurations: * HP EliteBook 2540p laptop; * SYSLINUX memdisk driver (e.g. used by OpenStack Cobbler and other provisioning systems) with a combination of hardware; * Cisco UCS C220 M3 rack servers; * Gateway GR380 F1 rack servers. All hardware configurations included 64-bit capable Intel processors. Whilst booting FreeBSD USB images from SYSLINUX is not a configuration which is officially supported by the FreeBSD Project, we have found that this is necessary in order to use FreeBSD with provisioning and deployment systems with wide uptake and acceptance (e.g. OpenStack Cobbler), as is required in a heterogenous technology environment. Affected releases ----------------- This issue has been observed to affect: * FreeBSD 9.2-RELEASE * FreeBSD 9.3-RELEASE * The system fails to boot from USB or CDROM media as shipped. * The system cannot be booted using PXELINUX the memdisk shim. * FreeBSD 10.0-RELEASE * The system fails to boot from USB or CDROM media as shipped. * FreeBSD 10.1-RELEASE * The system fails to boot from USB or CDROM media as shipped. * In UEFI mode, the system boots the installer successfully, however the console is unusable and cannot be redirected to another I/O medium (e.g. physical or virtual serial port such as Intel AMT). * PC-BSD 10.1 * The system boots from USB or CDROM media, but cannot boot successfully once installed. Specifically, the system may be unable to progress further in the boot process beyond the "boot2" stage or early in the "loader" stage. This may trigger an immediate system reset in either case. With certain versions of SYSLINUX, a BIOS exception may be logged before the system is reset. Where the system is being booted from a USB CDROM, the cdboot code may display a message that it is unable to find /boot/loader. This is not believed to be a regression but a known limitation of the FreeBSD cdboot loader. Workaround ---------- For Cisco UCS and Gateway systems, we have found that USB images prepared using FreeBSD 10.0-RELEASE and later do not have the interoperability issue with SYSLINUX memdisk. For affected HP laptops, one workaround is to prepare the system by booting from a mfsBSD 8.3 ISO image and installing the new release from within the mfsBSD root image by manually extracting its '*.txz' distribution files into a root filesystem created manually or by its "zfsinstall" script. This ensures that "known good" boot sectors will be installed, and should not affect system operation. It does, however, mean that automated installers will not function on affected systems. mfsBSD 8.3 can be downloaded from: http://mfsbsd.vx.sk/files/images/8/ The HP EliteBook 2540p may require booting mfsBSD 8.3 from ISO media and/or disabling UEFI mode in the firmware. UEFI issues with HP EliteBook 2540p ----------------------------------- The FreeBSD-10.1 UEFI mode installer is not usable on this system. A screen capture obtained using the integrated Intel AMT KVM can be found at: https://people.freebsd.org/~bms/dump/freebsd-10.1-uefi-hp-2540p.png The system is able to progress to single-user mode and is able to launch the installer, however the console cannot be redirected to a serial device. The commands used for this redirection are known to work with a combination of FreeBSD 8.3 boot sectors and the FreeBSD 10.0 loader. This has been exercised both with a dedicated PCIe ExpressCard-RS232 device and the onboard Intel AMT Serial-over-LAN (SOL) redirection. We have not attempted to reproduce this issue on Cisco UCS hardware at this time. Contents of this directory -------------------------- . |-- README.txt This file |-- comparisons Visual diff, 8.3 by "known good" system | |-- gptzfsboot-knowngood.png | `-- pmbr-knowngood.png |-- mfsbsd8dot3 FreeBSD boot sectors from mfsBSD 8.3 | |-- gptzfsboot | `-- pmbr |-- rel10dot1 FreeBSD 10.1-RELEASE boot sectors | |-- gptzfsboot | `-- pmbr `-- sut "Known good" HP EliteBook 2540p sectors |-- gptzfsboot `-- pmbr Diagnosis --------- During diagnosis of the issue, we extracted the /boot/pmbr and /boot/gptzfsboot sectors (installed within a GPT+ZFS disk configuration) from a 10.0-RELEASE system which was installed using this workaround. Included here are screen captures from a visual binary comparison performed using Hex Fiend v2.1.2 under Mac OS X 10.9 "Mavericks". The purpose of this comparison is to confirm the source of the boot blocks in the "known good" configuration which was able to boot successfully on the affected systems. Note: It is known that certain bytes may differ when boot sectors are installed (specific to their installation on a particular disk) and these are clearly indicated in the lower pane. The window panes on the left contain a hex dump of the /boot/pmbr and /boot/gptzfsboot sectors from FreeBSD 8.3-RELEASE. The panes on the right contain the data extracted from the working install. We observe that the only difference within the /boot/pmbr view are bytes specific to its installation on a particular disk drive. Whilst the /boot/gptzfsboot view includes the additional padding bytes from embedding it within a 64KiB GPT partition, it is otherwise unmodified from 8.3-RELEASE. However, there are significant differences between these "known good" working boot sectors and the boot sectors installed by subsequent FreeBSD / PC-BSD releases. A disassembly is not included here. Root cause ---------- The root cause of the issue is currently unknown. There have been a significant number of changes to the boot subsystem since FreeBSD 8.3 was released, including the migration from GCC to LLVM/clang by the FreeBSD Project as a whole. Therefore, determining the root cause will require following up on these changes and their possible impact on BIOS compatibility with a variety of x86 systems.