FreeBSD SysVR4 Emulation

Mark Newton, newton@atdot.dotat.org

Contents

Description

This page describes an SysVR4 emulator for FreeBSD. It is currently capable of running (or walking, in some cases) a wide-ish variety of SysV executabls taken from Solaris/x86 2.5.1 and 2.6 systems. I have reason to believe that it will also run SCO UnixWare and SCO OpenServer binaries.

The emulator has been produced by a process of distillation, starting with the Linux and iBCS2 emulators from FreeBSD and the SVR4 emulator from NetBSD. HUGE thanks to Christos Zoulas and Todd Vierling for the NetBSD stuff, and David Greenman and Soren Schmidt for the FreeBSD stuff. I haven't so much written this stuff as integrated and debugged it, and couldn't have done it without the efforts of those individuals.

The emulator has some limitations, and needs a lot of work. The most most pressing items at the top of my to-do list are presented below; if you are interested in lending a hand, it'd probably be best to start with those.

If you use it, please provide feedback. If something doesn't work, ktrace output describing the failure would also be appreciated (although I'll get very rude if you bombard my mailbox with 50Mbyte ktrace.out files, so please be considerate).

I've tested the emulator with a variety of hand-rolled test programs and actual utilities from the Solaris 2.5.1 CD. The most complicated shell-based program which runs is "vi". Networking programs such as "telnet" and "ftp" work fine too, which means the SysVR4 STREAMS emulation is at least pretending to be functional.

News

1999-Jan-30: Committed the rest of the emulator to FreeBSD 4.0-CURRENT. Blaz Zupan kindly sent me info about a glitch he had with it shortly thereafter: The emulator will fail to load with an "Exec format error" if your kernel hasn't been compiled with options KTRACE. Since you'd be mad to attempt to debug an emulator without options KTRACE anyway, I don't see that this is a huge problem at present (although we'll have to fix it eventually).

1999-Jan-28: Committed the LDT patch as a lead-up to committing the whole emulator. Also cleaned up a crapload of warnings. There are still other cosmetic nits and major functionality botches, but it's better now than it was. svr4_sys_getdents64() and poll() are still broken, incidentally.

1999-Jan-11: I've merged most of Christos Zoulas' changes to the NetBSD SysVR4 emulator into this one so that it can be relatively functional with Solaris 2.6 libraries. Changes involved unstaticizing dosetrlimit() in sys/kern/kern_resource.c, which is taken care of by the updated installation patch. Despite the update, svr4_sys_poll() and svr4_sys_getdents64() syscalls don't seem to be working as expected, so more work needs to be done here. NetBSD's VOP_READDIR and native poll() syscalls are different enough from FreeBSD's that they can't JustWork(tm).
Lots of #if defined(NOTYET) stuff was removed too -- Solaris 2.6 libraries need a reasonable number of 64-bit system calls to work, so I've had to bring 'em in. DNS name resolution doesn't work at present, but I think that's related to the svr4_sys_poll() problem outlined above.

1999-Jan-06: The emulator is now a KLD ready for ELF 3.0.x (rather than an LKM, the FreeBSD 2.2.x kernel loadable object format). See the download section to get it. You'll need to re-fetch the svr4.tar.gz source module if you have an earlier version of the emulator already installed.

Downloading

IGNORE THIS SECTION IF YOU'RE RUNNING 4.0-CURRENT. ALTERNATIVELY, IF YOU'RE NOT RUNNING -CURRENT THE CODE BELOW MAY NOT NECSSARILY WORK ANYMORE WITHOUT A BIT OF MAINTENANCE, SO ONLY ATTEMPT TO INSTALL THE EMULATOR IF YOU KNOW WHAT YOU'RE DOING.

The distribution is broken into three parts. You'll need all three parts if you expect to get this bucket of bolts working:

Each distribution contains a README file in some semi-obvious location; The README files document things you'll need to do to make that sub-part do its thing (for instance, things you'll need to change in your kernel source tree to make the "streams" pseudo-device driver work). That information is reproduced below in the INSTALLATION section.

To make the emulator work you'll need a SysVR4 CD-ROM. I've been using Solaris CDs for testing, but there shouldn't be anything preventing SCO binaries from running (crosses fingers -- I've only tried a small subset of SCO binaries because SCO's habit of putting billions of symlinks all over the filesystem makes it all too hard to do large-scale testing at this point). SCO and Sun both give their x86 versions away for free for educational or private individual use, so it shouldn't be all that difficult to come across them. See http://www.sco.com and http://www.sun.com to find out how to get free copies of licenses and cheap installation media.

If you have everything working, you should be able to run the "hello" program provided in /usr/src/sys/modules/svr4.

Developers are extremely welcome, and actively encouraged. I've set up a majordomo mailing list to discuss the technical issues of SVR4 emulation; You can put yourself onto it by sending email to majordomo@atdot.dotat.org with the text "subscribe freebsd-svr4" in the BODY of the message.

Eventually this distribution will get rolled into the -current CVS repository. That'll make installing it sooo much easier!

To-Do list

Current priorities are listed below, in what I think is their order of importance (YMMV, though).

  1. Solaris 2.6
    I'm told that 2.6 executables used to fail utterly with Bus Error signals shortly after startup. Sun changed the syscall conventions used by Solaris between 2.5.1 and 2.6. We've dealt with that problem by patching FreeBSD's LDT table, but other Solaris 2.6 changes in userland exercise bugs in the emulator (so hostname resolution doesn't work right and poll() acts bogusly). getdents64() doesn't work right either - test prints reveal that the emulator copies dirent structures into process address space once, but many filenames show up twice! Bizarre. Oh well, at least filemgr runs now...
    NOTE: X windows clients do work, you just can't use :0.0 as your $DISPLAY. Change it to unix:0.0; Solaris wants to use a named pipe to talk to the X server if you use :0.0, but XFree86 provides a socket (which can't be open()'ed). If you use unix:0.0 the Solaris libX11 will use the socket.
  2. USER_LDT
    Currently disabled with #ifdef (NOTYET) bits. Needs to be fixed so we can see if Wabi will work...
  3. Signals.
    Signals seem to work pretty well: I've tested alternate signal stacks, catching and ignoring signals, etc. Part of the implementation of svr4_sys_context() required redesign of the sendsig() routine and the sigcode() assembly code, however, and that can probably do with some verification. That's the most recent bit of work I've done on it, so it's the bit that has had the least amount of testing. It seems to work for most cases (i.e.: all the ones I've tried), though.
  4. Backporting
    There should be no reason why a single source tree can't compile on NetBSD and FreeBSD. I've done something a bit dumb by making my code diverge from Christos', and to repent I really should fold my differences back into the NetBSD codebase.

    I have some ideas on how to make that simple, but I'll need time. If anyone thinks they can assist, let me know. It should be a relatively light weight project technically speaking, and could be an ideal way for someone who hasn't hacked kernels before to make a contribution.

  5. Slightly more exhaustive testing than I've done.
    Related to (2) above.

    My development methodology has been something like, "xxx doesn't work. Find out what services xxx needs. Emulate them. xxx works now, therefore those services must be emulated correctly." While the first three steps are ok, the fourth represents a false conclusion. More extensive testing than I've carried out really needs to be done. Call me lazy :-)

    In most cases I'm building on a fantastic foundation provided by Christos Zoulas from the NetBSD Project (this is really just a FreeBSD port of his code). In some cases, however (parts of signal delivery, the LKM framework, etc) there is code I've written which has been tested somewhat less rigourously.

  6. SysV shared memory
    Interface BSD's SysV shared memory emulation with SysVR4 system calls
  7. LKM streams pseudo-device
    Daniel O'Connor informs me that it's simple, so I'll let him do it. :-) It's built-in to the kernel at the moment, which means you need a customized kernel to run SysVR4 binaries.
  8. "Getting the tabs right."
    Can't offend the style(9) gods, can we? :-p (hi, Bruce!)
  9. ???
    Probably lots of other crap. SysV is wierd compared to BSD, after all.

THIS SOFTWARE WILL NOT BUILD CLEANLY!!

THIS SOFTWARE WILL NOT BUILD CLEANLY!!

THIS SOFTWARE WILL NOT BUILD CLEANLY!!

You'll need to make other changes
to your kernel source code before
it'll work. See "INSTALLATION" below
for more information.

Mini-FAQ

If it doesn't build at all
If it complains about undefined symbols, they're probably declared static in the kernel sources. Grep through /sys/kern/* and /sys/sys/* for the symbols involved, and change static definitions to non-static. Rebuild your kernel, recompile the emulator, and see if it works.

"All the SysVR4 programs I try say, `ELF binary type not known'"
A peculiarity of the FreeBSD ELF subsystem is that ELF programs need to be "branded". Use the following command:

brandelf -f -t SVR4 program

[ substitute the pathname of your SysV program in place of "program" ]

yes, that means you can't run things straight off a SysV CD-ROM. Cope.

"X11 clients hang or tell me they can't open server :0.0"
X windows clients do work, you just can't use :0.0 as your $DISPLAY. Change it to unix:0.0; Solaris wants to use a named pipe to talk to the X server if you use :0.0, but XFree86 provides a socket (which can't be open()'ed). If you use unix:0.0 the Solaris libX11 will use the socket.
Unfortunately the aforementioned poll() bug means TCP connections to X servers hang. Let's just fix poll(), ok?

"Crudbox 3.1 from Foobar Enterprises doesn't work!"
Tell the mailing list you're working on it; Modify the emulator to make it work; Submit patches to me.

I don't have huge numbers of SysVR4 apps, so I can't test everything. At this stage I expect people who submit bug reports to be able to send 'em in with patches. Sorry if this sounds harsh, but if I spend 24 hours per day working on everyone else's bugs I'll end up like Mike Smith :-)

kdump(1) doesn't seem to be reporting the right system calls
That's because ktrace(1) is seeing FreeBSD syscall numbers when you ktrace a SysV executable. To find out what the SysV executable is really doing, grep for the syscall name provided by ktrace in the file /sys/kern/syscalls.c. That'll give you the system call number. Then grep for that number in /sys/svr4/syscalls.master -- that'll tell you the SysV system call that was really running.

Example: kdump tells you that a Solaris/x86 program has executed getrusage(0x28113000,0x1ea8). If you grep for "getrusage" in /sys/kern/syscalls.c, you get:

"getrusage", /* 117 = getrusage */

So grep for 117 in /sys/svr4/syscalls.master:

117 NOPROTO SVR4 { int munmap(void *addr, int len); }

... which tells you that the Solaris program was really running an "munmap" system call.

It sucks, but I've spent so much time on the emulator itself that I haven't finished porting linux_ktrace. I've done most of it; if anyone wants to take the bits I've done and finish them off, let me know.

Installation

Most of this is already done if you're running 4.0-CURRENT. If uname -r reports 4.0-CURRENT you should only carry out the steps marked 4.0.

Extract svr4.tar.gz and streams.tar.gz into /usr/src. Then follow the steps below. They take about half an hour to run through (including kernel building), and only need to be done once.

You can simplify your task by taking this patch, which was generated against 3.0-RELEASE. It promotes static symbols to global scope and edits /sys/conf/files as described below, and ends up modifying /sys/kern/kern_exit.c, , /sys/sys/socketvar.h and /sys/conf/files to do its work (so you'll need write permission on those files and should expect them to change as a result of running the patch). You can install it by running patch -p < svr4patch in /sys.

If you don't trust foreign patches and want to do things by hand, see below.

  1. Some static symbols need to be "promoted" to global scope so that they can be used by the emulator LKM.

    The symbols concerned are listed below, along with the files they belong to. Edit each of the files and "unstaticize" the appropriate symbols.

    SymbolSource File
    M_ZOMBIE sys/kern/kern_exit.c
    soo_read() sys/kern/sys_socket.c
    soo_write() sys/kern/sys_socket.c
    soo_close() sys/kern/sys_socket.c
    dosetrlimit() sys/kern/kern_resource.c

    Note that function call declarations need to be declared non-static both in their prototypes and the actual definition of the function.

    It's easier and quicker than it sounds.

  2. Modify /sys/sys/socketvar.h. Add a new field to the end of struct socket like so:

    void *so_emuldata; /* private data for emulators */

    (it should go just underneath "so_gen_t so_gencnt")

  3. Modify /sys/conf/files. Add a line which says:

    dev/streams/streams.c optional streams device-driver

  4. Modify /sys/i386/conf/majors.i386. Add a line which says:

    103 streams SysV STREAMS network driver

  5. Install this patch by executing patch -p < segpatch in /usr/src. The patch will modify sys/i386/include/segments.h and sys/i386/i386/machdep.c, and will have the effect of changing LUDATA_SEL from 4 to 5, adding LSOL26CALLS_SEL as LDT entry 4, and modifying the ldt_segs table to insert a new blank entry for LSOL26CALLS_SEL. This patch is required to allow executables from Solaris 2.6 and Solaris 7 to run on the emulator, and is a fix for the problem described here.

  6. Modify your kernel config file. Add a line which says:

    pseudo-device streams 1

  7. Build a new kernel. Boot from it. Look depressed, because you now (kinda) support streams, and they're REALLY ugly.
  8. Run the following commands:

    freebsd% cd /sys/modules/svr4
    freebsd% make depend
    freebsd% make

    Expect it to compile with quite a few warnings at this stage (anyone who wants to fix them and submit patches is welcome to do so).

  9. As root:

    freebsd# make install
    freebsd# kldload /modules/svr4.ko

    4.0 Type svr4 instead of the commands above to load the emulator. To make it automatically load at boot time, set the svr4_enable flag to YES in rc.conf

  10. 4.0 Extract compat_svr4.tar.gz into /compat. It'll create /compat/svr4 right alongside /compat/linux. Execute the following as root:

    freebsd# cd /compat/svr4/dev
    freebsd# sh SVR4_MAKEDEV all

    The tar archive contains symlinks which expect you to have a Solaris for Intel CD-ROM mounted on /cdrom -- If that's what you actually have, congratulations, installation is completed and you can start seeing what applications you can actually run. If you only have v2.6 or v7 media, stay with us, we're working to improve support for that (2.5.1 and earlier is presently more "capable").

    If you have some other OS CD, remove the symlinks and install that OS's /usr/lib, /lib, and so on within the /compat/svr4 heirarchy. Make sure you run the SVR4_MAKEDEV script as shown above, though, otherwise code which tries to do networking or UNIX-domain sockets won't work.

  11. 4.0 Optionally set the kern.fallback_elf_brand sysctl to the value SVR4.

    This will remove the need to brand executables, because it will make the kernel assume that unbranded executables belong to the SVR4 emulator.

    If you get "ELF binary type not known" messages, see the mini-FAQ above.