The new unionfs implementation for FreeBSD and status of merging

Daichi GOTO (daichi@freebsd.org)
First edition Wed Jan 4 17:20:11 2006
Last modified Tue Oct 23 13:32:10 2007

[Japanese]

1 Introduction

There were several known problems in unionfs implementation of FreeBSD up until 6.2-RELEASE. The specification is ambiguous and its locking implementation was buggy.

Because of these issues, mounting unionfs cd9660 file system as a lower layer had caused problems. This article describes these problems and provides patches to prevent and improve them. The most valuable codes of new unionfs has already been merged into -current/-stable. Our going improvements are getting merged into -current/-stable step by step.

2 What is unionfs

Unionfs makes it possible to mount one file system on top of another. For example, you can mount a memory file system on top of a CD-ROM. As a result, it looks as if you could write to files on the CD-ROM.

Changes are only made to the upper file system layer and no changes are made to the lower one. Therefore, you can use it to keep modifications without changing the lower layers.

For a more detailed explanation have a look at Section 6.7 on page 256 of "The Design and Implementation of the FreeBSD Operating System" by Marshall Kirk McKusick and George V. Neville-Neil.

3 Problems of original(old) unionfs implementation of FreeBSD

The problems of FreeBSD unionfs implementation up until 6.2-RELEASE are the following:

  1. [Implementation Problem] The original(old) unionfs locks upper and/or lower layer locks as necessary for VOP_* calls. In this case, unionfs needs to lock up to 3 locks. Because it locks and unlocks multiple times independently of the kernel, dead-lock happens easily. In fact, the original(old) implementation is unstable and dead-locks happen quite often, especially on systems with high load.
  2. [Implementation Problem] Parts of VOP_* call unionfs vnodes directly, others call upper/lower layer vnodes. If this happens in LOOKUP situations, strange problems occur. One of the known results is that pwd returns an absolute path of the upper/lower layer.
  3. [Implementation Problem] Readdir is not fully implemented and only works with upper layer's list. As a result, files that only exist on lower layer cannot be seen.
  4. [Implementation Problem] Because there is no white-out, if files or directories with the same name exist on both upper and lower layers, it looks like they were not removed.
  5. [Implementation Problem] If files or directories exit on only lower layer, the upper layer does not have corresponding permissions. As a result, there are problems like you cannot write onto a writable file.
  6. [Specification Problem] The specification of unionfs does not specify permission control when creating a shadow file or directory. In the original(old) implementation, "umask-ed 0777" was used. However, it produces undesirable situations. For example, executable permission is assigned to a file when it shouldn't.

And the original(old) unionfs implementation of FreeBSD had led some issues as follow:

4 Solution

We have made up new unionfs of FreeBSD. The most valuable codes of our new implementation has already been merged up until FreeBSD 8/FreeBSD 7.0/FreeBSD 6.3 fixing the following problems:

  1. [Implementation Problem] The lock object within the unionfs vnode have been replaced by a pair of pointers to the lock objects in the upper/lower layers, and locking and unlocking of these vnodes are now controlled by unionfs. This strategy was chosen to introduce minimal changes to the other parts of the kernel. The locks on the upper/lower layers are managed to always keep the same state, so as not to interfere with the other parts of the kernel.
  2. [Implementation Problem] Fixes http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/84107 (FIXED).
  3. [Implementation Problem] Fixes http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/84498 (FIXED).
  4. [Implementation Problem] Fixes http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/89755 (FIXED).
  5. [Implementation Problem] Fixes http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/73094 (FIXED).
  6. [Implementation Problem] Fixes any other known problems.

Solving [Ambiguous Specification Problems] involves discussions what the appropriate behaviour is.

As a result of the old implementation of FreeBSD unionfs up until 6.2-RELEASE, applications which check and determine their behavior accordingly do not run.

Because the specification of unionfs has ambiguity of its behavior, so it is difficult to implement appropriately. Therefore, I have proposed different options for different situations. This patch includes an option that allows unionfs to change its behavior on three ways.

Three solutions for solving [Ambiguous Specification Problems]:

As a result, the following problem is also be fixed in [transparent mode].

5 How to Install

prompt5.1 shows how to apply the patches.

# cd /usr/src/
# patch -p3 < /anywhere/unionfs-p19-20070504.diff
# cp /usr/src/sys/fs/unionfs/union.h /usr/include/fs/unionfs/
# cd /usr/src/sbin/mount_unionfs/
# make obj depend && make && make install
# make clean
# cd /usr/src
# make buildkernel
# make installkernel
# shutdown -r now
prompt5.1 how to install

6 How to Use

As much as anything else, see mount_unionfs(8). It is all in there.

7 Download

Patches for FreeBSD 8-current. They are numbered in chronological order, the newest patch has the highest number.

Patches for FreeBSD 7-stable.

Patches for FreeBSD 7-current.

Patches for FreeBSD 6-stable

7.1 Changes in unionfs-p19-20070504.diff / unionfs-p19-20070524.diff

7.2 Changes in unionfs-p18.diff

7.3 Changes in unionfs-p17.diff

7.4 Changes in unionfs-p16.diff

7.5 Changes in unionfs-p15.diff

7.6 Changes in unionfs-p14.diff

7.7 Changes in unionfs-p13.diff

7.8 Changes in unionfs-p12.diff

7.9 Changes in unionfs-p11.diff

7.10 Changes in unionfs-p10.diff

7.11 Changes in unionfs-p9.diff

7.12 Changes in unionfs-p8.diff

2006/02/13 - Added unionfs6-p8-fix1.diff fixes 6.x build fail

7.13 Changes in unionfs-p7.diff

7.14 Changes in unionfs-p6.diff

7.15 Changes in unionfs-p5.diff

7.16 Changes in unionfs-p4.diff

7.17 Changes in unionfs-p3.diff

7.18 Changes in unionfs-p2.diff

7.19 Changes in unionfs-p1.diff

8 Known Problems

9 Committed 1

The unionfs-17.diff (without patch for sys/kern/vfs_lookup.c) was committed to FreeBSD 7-current 2006-12-02 19:35:56 UTC.

リスト9.1 unionfs-17.diff (without patch for sys/kern/vfs_lookup.c) was committed to -current
rodrigc     2006-12-02 19:35:56 UTC

  FreeBSD src repository

  Modified files:
    sys/fs/unionfs       union.h union_subr.c union_vfsops.c 
                         union_vnops.c 
    sbin/mount_unionfs   Makefile mount_unionfs.8 mount_unionfs.c 
  Log:
  Many, many thanks to Masanori OZAWA <ozawa@ongs.co.jp>
  and Daichi GOTO <daichi@FreeBSD.org> for submitting this
  major rewrite of unionfs.  This rewrite was done to
  try to solve many of the longstanding crashing and locking
  issues in the existing unionfs implementation.  This
  implementation also adds a 'MASQUERADE mode', which allows
  the user to set different user, group, and file permission
  modes in the upper layer.
  
  Submitted by:   daichi, Masanori OZAWA
  Reviewed by:    rodrigc (modified for minor style issues)
  
  Revision  Changes      Path
  1.13      +1 -1        src/sbin/mount_unionfs/Makefile
  1.21      +184 -23     src/sbin/mount_unionfs/mount_unionfs.8
  1.25      +107 -74     src/sbin/mount_unionfs/mount_unionfs.c
  1.33      +87 -92      src/sys/fs/unionfs/union.h
  1.87      +948 -1080   src/sys/fs/unionfs/union_subr.c
  1.79      +394 -332    src/sys/fs/unionfs/union_vfsops.c
  1.135     +1864 -1613  src/sys/fs/unionfs/union_vnops.c

The most valuable codes of new unionfs have been committed to FreeBSD 6-stable 2007-02-13 05:56:43 UTC.

リスト9.2 The most valuable codes of new unionfs have been committed to FreeBSD 6-stable 2007-02-13 05:56:43 UTC
rodrigc     2007-02-13 05:56:43 UTC

  FreeBSD src repository

  Modified files:        (Branch: RELENG_6)
    sys/fs/unionfs       union.h union_subr.c union_vfsops.c 
                         union_vnops.c 
    sbin/mount_unionfs   Makefile mount_unionfs.8 mount_unionfs.c 
  Log:
  MFC: New unionfs implementation from Daichi GOTO and Masanori OZAWA,
       which fixes many locking and crashing problems in the previous
       implementation.  kib@ helped a lot by eliminating a source
       of deadlocks encountered with unionfs with these commits:
         rev. 1.50 of src/sys/fs/deadfs/dead_vnops.c
         rev. 1.97 of src/sys/kern/vfs_lookup.c
  Submitted by:   daichi, Masanori OZAWA <ozawa ongs co jp>
  
  Revision   Changes      Path
  1.12.8.1   +1 -1        src/sbin/mount_unionfs/Makefile
  1.20.2.1   +198 -25     src/sbin/mount_unionfs/mount_unionfs.8
  1.24.2.1   +108 -74     src/sbin/mount_unionfs/mount_unionfs.c
  1.31.2.1   +88 -94      src/sys/fs/unionfs/union.h
  1.86.2.1   +937 -1080   src/sys/fs/unionfs/union_subr.c
  1.76.2.2   +398 -334    src/sys/fs/unionfs/union_vfsops.c
  1.132.2.1  +1879 -1607  src/sys/fs/unionfs/union_vnops.c

10 Committed 2

The unionfs-p19-20070524.diff was committed to FreeBSD 8-current at 2007-10-14 JST, to RELENG_7 at 2007-10-22 JST and to RELENG_6 (without default mode change patch) at 2007-10-23 JST.

11 Appendix

11.1 FAQ

Q: Why did you rewrite from scratch?

A: The original(old) unionfs implementation of FreeBSD up until 6.2-RELEASE had many dead-lock scenes easily. Fixing of it was harder than rewriting it.

Q: What about backward compatibility?

A: If the traditional option is used, the new implementation behaves like the old one. In addition, it fixes http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/84107, http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/84498, http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/89755, http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/86596, http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/73094 and the other issues of unionfs related.

-c transparent option seems to be the most reasonable default behavior. It fixes most of the problems in the original implementation.

Q: Does the implementation of the patch meet the intended unionfs behavior?

A: I think it does. See pp.256-258, Section 6.7 `The Union Filesystem' of ``The Design and Implementation of the FreeBSD Operating System'' by Marshall Kirk McKusick and George V. Neville-Neil. for the unionfs specification.

Q: Why are shadow files created even for read access?

A: Because the system cannot modify the access time of the lower layer filesystem. Thus, it uses shadow file to update access time. You can use -o noatime option to prevent that behavior. See mount(8) manual page for detail.

Q: vi sometimes gets some messages as follow. Is it okey?

'filename: file modified more recently than this copy; use ! to override.'

A: It is no problem. vi reports above message when catched inode changes. It is correct.

11.2 Call for help

If you are interested in translating the Japanese text into English, please do so and send it to me :) My mail address is daichi AT-MARK freebsd.org or daichi AT-MAKR jp.freebsd.org.

11.3 Thanks!

Some guys have helped me with the unionfs work and I would like to thank all of them.

Yoshihiro OTA-san wrote most of the English text, Hiroo ONO-san and Hiroharu TAMARU-san translated the other parts. This page was born by those guys efforts. The manual page of new unionfs has been rewritten by Hiroki Sato-san (aka hrs). Good quality manual page was created by him.

Danny Braniss did some unionfs tests with his diskless unionfs system. Dario Freni, one of the key FreeSBIE developers, did some unionfs tests while developing FreeSBIE. Alexander Leidinger, one of the FreeBSD committers, is going to help to merge the unionfs patch to FreeBSD 7-current. Craig Rodrigues, daichi's src mentor and one of the FreeBSD committers, has very hard work around unionfs to get merge into FreeBSD 7-current. Especially by hard work of Craig Rodrigues, we could merge patches into FreeBSD 7-current.

Ed Schouten contributed whiteout-mode-switch ability. Stanislav Sedov has pointed out Coverity NULL check issue. Kris Kennaway always reports deadlock issues faster than anybody. Jeff Roberson and Ken Smith have reviewed our vast unionfs patches bunch.

And Fabian Keil gave me a patch that fixes my broken English of this page:)