Library versioning under FreeBSD Symbol versioning - simple explanation -------------------------------------- Symbol versioning allows a library to define its exported symbols through the use of a map file. It also allows one library to provide multiple versions of the same symbol. In the past, without symbol versioning, this was accomplished by bumping the shared library version (libfoo.so.1, libfoo.so.2, etc.). Now one library version can provide multiple versions of the same symbol. Symbol versioning - example --------------------------- As a simple example, consider a hypothetical library libvector that provides the following interface: struct __vector; typedef struct __vector *vector_t; vector_t v_create(int initial, int max); int v_add(vector_t v, const void *obj); int v_remove(vector_t v, const void *obj); int v_elements_in(vector_t v); void *v_element_at(vector_t v, int index); int v_size_current(vector_t v); int v_size_max(vector_t v); The corresponding symbol map file would be: VER_1.0 { global: v_add; v_create; v_element_at; v_elements_in; v_remove; v_size_current; v_size_max; local: *; }; This symbol map defines the desired vector interfaces as global symbols. The asterisk (*) in the local section is a wildcard and specifies that all other symbols, regardless of whether they are declared static or not, are defined as local (hidden) symbols. When linking an application (or library) to libvector, only the global symbols are visible. Now suppose, after a release of libvector (as libvector.so.1), new interfaces are added: int v_remove_at(vector_t v, int index); int v_insert_at(vector_t v, int index, const void *obj); A new map file that defines the new version of libvector would be: VER_1.0 { global: v_add; v_create; v_element_at; v_elements_in; v_remove; v_size_current; v_size_max; }; VER_1.1 { global: v_remove_at; v_insert_at; local: *; } VER_1.0; The new symbol map defines VER_1.1 to provide the two new additional interfaces, and also to provide the same interfaces as VER_1.0. The new version of libvector is also released as libvector.so.1 and is compatible with the previous version. Older binaries or libraries that have linked to the previous version of libvector will continue to behave as before. Applications or libraries that are linked to the new version of libvector will be able to access the new interfaces. Now suppose an incompatible interface change is made to libvector: vector_t v_create(int initial, int extent, int max); The original v_create() created a vector with an initial size that could grow to a max size. When it needed to grow, it used a default size by which it grew. The new interface allows the application to specify what this size is, but the new interface is not compatible. Without symbol versioning, this would normally require releasing the library with a subsequent version, libvector.so.2. But symbol versioning allows the same library to support the original v_create() interface to older binaries, while providing the modified interface to new binaries. This is done using the assembler .symver pseudo opcode: .symver __v_create_old, v_create@VER_1.0 .symver __v_create_new, v_create@@VER_1.2 The library implementation provides two versions of v_create, __v_create_old() that provides the VER_1.0 and VER_1.1 interface, and __v_create_new() that provides the new (VER_1.2) interface. The second .symver statement above specifies the default interface for v_create() (using '@@' instead of '@'). There can be only one default interface, while there may be several non-default interfaces. The map file corresponding to VER_1.2 would be: VER_1.0 { global: v_add; v_create; v_element_at; v_elements_in; v_remove; v_size_current; v_size_max; }; VER_1.1 { global: v_remove_at; v_insert_at; } VER_1.0; VER_1.2 { global: v_create; local: *; } VER_1.1; Symbol versioning in libc ------------------------- We take a similar approach to symbol versioning as Linux does. There is a master version file that defines the libc versions. The master version file lives in src/lib/libc (or perhaps in src/lib if a more general approach to version definiitons is desired). This file does not contain any symbols, only the complete list of versions. It is f similar form as the map file in the example above: $ cat /usr/src/lib/libc/Versions.def FreeBSD_1.0 { }; FreeBSD_1.1 { } FreeBSD_1.0; FreeBSD_1.2 { } FreeBSD_1.1; Each sub-directory of libc that exports symbols defines a symbol map file (Symbol.map or whatever) that lists its symbols in a similar way as the example above. For example, libc/stdlib would contain: $ cat /usr/src/lib/libc/stdlib/Symbol.map FreeBSD_1.0 { abort; abort2; abs; atexit; ... system; wctomb; wcstombs; }; It should be noted that these symbol map files do not specify version successors; they only define the list of exported symbols. When multiple versions of the same symbol need to be exported for compatibility reasons, they are listed multiple times in the symbol map file, just like the example above shows. is modified to provide the macro: #define __sym_compat(sym,impl,ver) \ __asm__(".symver " #impl ", " #sym "@" #ver) #define __sym_default(sym,impl,ver) \ __asm__(".symver " #impl ", " #sym "@@" #ver) that source files can use to define multiple versions of the same symbol: int __foo_ver1(FILE *); /* original */ int __foo_ver2(FILE *); /* FILE changed format */ int __foo_ver3(FILE *, int); /* added argument */ __sym_compat(foo, __foo_ver1, FreeBSD_1.0); __sym_compat(foo, __foo_ver2, FreeBSD_1.1); __sym_default(foo, __foo_ver3, FreeBSD_1.2); Each subdirectory of libc that defines a symbol map file, also needs to add this file to SYM_MAPS in its appropriate Makefile.inc. SYM_MAPS+= Symbol.map Tying it all together, there is a magical awk script that parses the master version file (libc/Versions.def), then each symbol map file to produce a master map file that can be used by the linker. It is installed in /usr/share/mk/version_gen.awk and run as: awk -f version_gen.awk -v vfile=Versions.def db/Symbol.map \ gdtoa/Symbol.map gen/Symbol.map ... with the master map being written to standard output. This is then provided as an argument to the linker's --version-script option. Version naming conventions -------------------------- Symbol versioning is incorporated into FreeBSD starting with release 7.0 with the first namespace defined as "FBSD_1.0". Any subsequent ABI changes use the next namespace "FBSD_1.1". Multiple ABI changes can be added to the same namespace as long as the same ABI only changes once in that namespace. For instance, if FTS (see ) changes after 7.0, compat fts_*() functions are added to the FBSD_1.0 namespace and the new fts_*() functions are added to FBSD_1.1. If FBSD_1.1 doesn't exist yet, Version.defs in libc is updated to add it: FBSD_1.0 { }; FBSD_1.1 { <- new } FBSD_1.0; <- new src/lib/libc/gen/Symbol.map is updated to add the new fts_*() symbols to the FBSD_1.1 namespace: FBSD_1.0 { __xuname; pthread_atfork; pthread_attr_destroy; ... fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; ... wordexp; wordfree; }; FBSD_1.1 { fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; }; Note that the fts symbols are listed in both versions. If sometime later, FILE changes, a compat version of any affected function is added to libc along with its new counterpart. Since there is no overlap in ABI change between functions using FTS and those using FILE, the new functions are added to the existing FBSD_1.1. We continue to use FBSD_1.1 for any ABI changes until an existing and released ABI in FBSD_1.1 changes. So if FILE or FTS were changed after a release, you would add yet another set of compat functions for FBSD_1.1 and add the new functions in FBSD_1.2. Dealing with major releases --------------------------- Since FreeBSD typically has more than one major release being developed and maintained, there needs to be a procedure for how to deal with versioning and MFC'ing to other branches. As stated above, for release 7.0 we use FBSD_1.0 and FBSD_1.1 for subsequent ABI changes along the 7.x branch. For 8.x branches, we also use FBSD_1.0 and FBSD_1.1 in the same manner, but all changes to FBSD_1.1 in 7.x must first be tested and added FBSD_1.1 in 8.x. MFC'able changes to 8.x ----------------------- Let's suppose we change pthread_mutex_t and pthread_cond_t from pointers to structs (to allow them to be used from different processes in shared memory). We first make this change in -current (8.x in this example) and modify the symbol map in libc/gen/Symbol.map to look like this: FBSD_1.0 { __xuname; pthread_atfork; pthread_attr_destroy; ... fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; ... wordexp; wordfree; }; FBSD_1.1 { fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; pthread_cond_broadcast; pthread_cond_destroy; ... pthread_mutex_destroy; pthread_mutex_init; ... pthread_mutex_unlock; }; If these changes are MFC'd, then libc/gen/Symbol.map is modified in the same way as in 8.x. Non-MFC'able changes to 8.x --------------------------- Now suppose we add an interface or an ABI change that is not MFC'able. We could add the new or changed interface to the existing FBSD_1.1 without any problem since we don't need to run 8.x binaries using 7.x libraries. But the opposite always has to be true; we always have to be able to run 7.x binaries on 8.x. So whatever versions exist in the 7.x libraries must also exist (and with the same ABI) in 8.x. This also means that we are not strictly required to MFC something that is added to FBSD_1.1 in 8.x. Just consider versions in 8.x to be supersets of the same versions in 7.x. When new versions become necessary ---------------------------------- Let's suppose we change the fts ABI again in 8.x. We need to create another version since the fts symbols already exist in FBSD_1.1. So we use FBSD_1.2 for the new fts interfaces: FBSD_1.1 { fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; ... pthread_cond_broadcast; pthread_cond_destroy; ... pthread_mutex_destroy; pthread_mutex_init; ... }; FBSD_1.2 { fts_open; fts_close; ... fts_get_stream; fts_set_clientptr; }; This is done in 8.x. If the changes are MFC'd, then a new version FBSD_1.2 can be created in 7.x. TBD: Do all new ABI changes go to FBSD_1.2 from now on? Development in -current ----------------------- Symbol versioning does solve some of the problems that development in -current has. In the past without symbol versioning, ABI changes may have forced a rebuild of all installed ports. This is mostly gone away with symbol versioning as long as an ABI only changes at most once in a version. An example of where this would break and force a ports rebuild is if the layout of FILE were to change more than once between releases. The first change to FILE would add fopen(), fread(), fwrite(), etc, to FBSD_1.1 (or possibly 1.2 - see TBD above). If a few weeks later it was determined that FILE had to change again, a new version is not created to handle this; -current users would be forced to rebuild ports. Versions are not a crutch to aid development in -current, but with proper review and test before committing, problems like this can be avoided.