Multi-processor locking in 4.4BSD Lite2

Abstract

In a multi-processor system locking between processors to serialize access to critical data structures and variables is neccecary. In 4.4BSD Lite2 this has been implemented by the addition of a generic lock manager, and the existing file system code has been updated to use it. In this paper we will attempt to describe the lock manager implementation thoroughly, and look at how some of the file system code uses it.

In particular, we will examine how vnode reclamation is handled in situations where more than one CPU may be trying to free a vnode. We will start by examining the low-level atomic locks in detail. After that we will see how the atomic locks are used by the generic locking package, and examine how multiple, shared and exclusive locks are implemented.

The locking package also supports upgrading a shared lock to an exclusive lock, by draining out other shared locks. This will also be examined in detail.

We will also examine how the locking code is currently used in the updated file system code, specifically in the functions vgone(), vclean(), vrele(), vget() and getnewvnode() to protect against multiple CPUs trying to reclaim a vnode at the same time.

Annotated pseudo-code

The following is pseudo-code describing the lock manager, as well as part of the file system code. Following that there will be annotations of each part of the pseudo-code, describing in detail what the code does.

Low-level locking code

All multi-processor systems need some amount of locking between processors to make sure access to some data structures or hardware is done atomically. The low-level locking code described here is responsible for serializing such access. Note that the low-level locking code is using spin-locks (a processor will busy-wait while trying to acquire such a lock); an operation that can tie up resources on some architectures. Because of this, and also to implement shared locks, the low-level locking routines described here is usually used only by a high-level lock manager and to protect short critical sections of code.

simple_lock_init

The simple_lock_init() function is used to initialize a low-level spinlock. This must be done before any other operation on a lock can be performed. The contents of a struct simplelock is highly architecture dependant, and usually consist of the smallest data type that can be atomically modified.

	/*
	 * Initialize a simple lock structure so it can be used
	 */
	void
	simple_lock_init(struct simplelock *lock)
	{
[1]		clear lock->data
	}
[1]
Initialize simple lock state by setting it to the equivalent of unlocked. This operation is machine dependant, as is the structure of simple locks.

simple_lock

The simple_lock() function is used to acquire a low-level spinlock. It will spin waiting for a lock to be released if it is currently held by another processor. This is the only kind of lock that can be used inside an interrupt handler. For a low-level lock to be used in an interrupt handler, care must be taken to disable the interrupt, acquire the lock, do any processing, release the lock and re-enable the interrupt. This is neccecary to avoid deadlock between the interrupt handler and other code executing on the same processor. Note that if there are no other processors in a system and the lock has already obtained, the function will never return as the lock cannot be released.

The test_and_set() function must be able to test and set the contents of a struct simplelock atomically across processors regardless of what bus is used to connect them, and return TRUE as long as the lock is already being held by a process.

	/*
	 * Attempt to acquire a lock, spin until lock obtained
	 */
	void
	simple_lock(struct simplelock *lock)
	{
[2]		while (test_and_set returns TRUE)
			;
	}
[2]
Spin, trying to acquire a simple lock. This operation will busy-wait until such time as the lock could be obtained.

simple_lock_try

The simple_lock_try() function is used to acquire a low-level spinlock. If the lock has already been acquired the function will return failure immediately. Look at simple_lock() for a description of test_and_set().

	/*
	 * Attempt to acquire a lock
	 *
	 * Return TRUE if success, FALSE otherwise
	 */
	int
	simple_lock_try(struct simplelock *lock)
	{
[3]		if (test_and_set returns FALSE)
			return TRUE
		return FALSE
	}
[3]
Call test_and_test() once, attempting to acquire a lock. If the lock is acquired, return TRUE, otherwise FALSE. This operation will never block.

simple_unlock

simple_unlock() will essentially perform the same operation as simple_lock_init() above. However, some architectures might need to have the state of a low-level spinlock propagated to other processors in ways that would not be needed for the initialization of a lock, but only when setting or clearing it.

Many multiprocessor architectures, like systems based on Intel®'s MP specification do not need to do this, but need only clear the contents of the struct simplelock.

	/*
	 * Unlock
	 */
	void
	simple_unlock(struct simplelock *lock)
	{
[4]		clear lock->data
	}
[4]
Reset a lock previously acquired with simple_lock() or simple_lock_try(). This will set the state of the lock to the same as when the lock was first initialized with simple_lock_init().

High-level locking code

The following section will describe the lock manager as found in 4.4BSD Lite2. This lock manager supplies both exclusive and shared locks, with recursive exclusive locks within a single process. It also allows upgrading a shared lock to an exclusive lock, as well as downgrading an exclusive lock to a shared lock.

lockinit

As with the low-level spinlocks, a high-level lock needs to be initialized before use. In the 4.4BSD Lite2 implementation, this will initialize the associated struct simplelock and save some parameters for the lock.

The parameters that can be specified for a high-level lock are priority, which is used to determine a process' priority when it is woken up after sleeping on a lock, a message used when a process goes to sleep waiting for a lock, so that the exact reason a process is sleeping can easily be identified, a timeout, and flags specifying options governing the lock's behaviour.

	/*
	 * Initialize a lock
	 */
	void
	lockinit(struct lock *lock, int pri, char *msg, int timeout, int flags)
	{
[5]		call simple_lock_init on lock->simplelock
		set lock->priority to pri
		set lock->wmesg to msg
		set lock->timo to timeout
		set lock->flags to flags
	}
[5]
First call simple_lock_init() on the embedded low-level lock. Initialize the remaining fields in a lock structure from the arguments provided; the priority to wake up at after being put to sleep on a lock, the message to pass to the sleep routing when blocking on a lock, the timeout and any auxilliary flags on a lock.

lockmgr

The lockmgr() function is used to perform all operations on a high-level lock after it has been initialized with lockinit(). Operations supported are LK_SHARED to obtain a shared lock, LK_DOWNGRADE to downgrade an exclusive lock to a shared lock, LK_EXCLUPGRADE and LK_UPGRADE to upgrade a shared lock to an exclusive lock, LK_EXCLUSIVE to obtain an exclusive lock, LK_RELEASE to free a lock and LK_DRAIN to drain out all locks active locks.

The LK_EXCLUPGRADE and LK_UPGRADE calls are essentially the same, but LK_EXCLUPGRADE will fail if any other process has already requested an upgrade to an exclusive lock.

A process can also specify that it does not want to be put to sleep waiting for a lock, such that if a lock request can not be honored immediately, the lock manager will return an EBUSY error to the calling function. This is described by the boolean condition polling in the following pseudo-code.

If at any time the process is put to sleep waiting for a lock to become available, the message specified in lockinit() will be used to provide information about the reason the process was put to sleep. When a process wakes up after sleeping on a lock, it's priority will be set to that specified when the lock was first initialized.

	/*
	 * Acquire, change or release a lock
	 *
	 * Return 0 if successful, error code if not
	 */
	int
	lockmgr(struct lock *lock, u_int flags, struct simplelock *slock, struct proc *p)
	{
		clear error;
[6]		call simple_lock on lock->interlock
[7]		if (flags & LK_INTERLOCK)
			call simple_unlock on slock
[8]		save combined new and old lock flags
		switch (lock request) {
			case LK_SHARED:
[9]				if (current process does not own a lock) {
					if (someone else is waiting)
[10]						if (polling) {
							set error to EBUSY
							break;
						}
[11]					sleep until exclusive locks and upgrades are cleared;
[12]					if (error is set)
						break;
[13]					increment shared lock count;
					break;
				}
[14]				increment shared lock count;
			case LK_DOWNGRADE:
[15]				increment shared lock count by exlusive lock count;
[16]				reset exclusive lock count and flag;
[17]				if (anyone waiting for exlusive locks to clear)
					wake them up;
				break;
			case LK_EXCLUPGRADE:
[18]				if (anyone else is waiting for exclusive upgrade) {
					reduce shared lock count by one;
					set error to EBUSY;
					break;
				}
			case LK_UPGRADE:
[19]				reduce shared lock count by one;
				if (we would block)
[20]					if (polling) {
						set error to EBUSY;
						break;
					}
[21]				if (noone is waiting for upgrade) {
					set WANT_UPGRADE flag
					sleep until shared locks are drained;
					if (error is set)
						break;
[22]					set exclusive lock flag and count;
					break;
				}
[23]				if (anyone else is waiting for upgrade)
					wake them up;
			case LK_EXCLUSIVE:
[24]				if (we have exlusive lock) {
					increment exclusive lock count;
					break;
				}
				if (we would block)
[25]					if (polling) {
						set error to EBUSY;
						break;
					}
[26]				if (anyone is waiting for exclusive lock) {
					sleep until WANT_EXCLUSIVE is clear;
					if (error is set)
						break;
				}
[27]				set WANT_EXCLUSIVE flag;
				sleep until shared locks and upgrades are drained;
				reset WANT_EXCLUSIVE flag;
				if (error is set)
					break;
[28]				set exclusive lock count to one;
				break;
			case LK_RELEASE:
[29]				if (exclusive locks exist) {
					decrement exclusive lock count;
[30]					if (lock count zero)
						clear exclusive lock flag;
				}
[31]				else if (shared locks exist)
					decrement shared lock count;
[32]				if (anyone waiting for locks to drain)
					wake them up;
				break;
			case LK_DRAIN:
				if (we would block)
[33]					if (polling) {
						set error to EBUSY;
						break;
					}
[34]				sleep until waits for exlusive locks and upgrades are drained;
				reset error;
[35]				while (any exclusive locks, upgrades or shared locks exist) {
					set WAIT_DRAIN flag;
[36]					call simple_unlock on lock->simplelock;
[37]					sleep until a lock has drained;
					if (error is set)
						return error;
[38]					call simple_lock on lock->simplelock;
				}
[39]				set draining and exclusive lock flag;
				set exclusive lock count to one;
				break;
		}
[40]		if (we are draining locks and all locks are gone)
			wake them up anyone waiting for locks to change;
[41]		call simple_unlock on lock->interlock
		return error;
	}
[6]
Make sure the low-level interlock is acquired to avoid other changes being done to the lock while we are changing it.
[7]
If we were passed a low-level lock that was held, release it now that we have made sure no changes will be made to the locking structure.
[8]
Store externally settable flags on the lock, combining the currently set flags and the flags passed into the lock manager.
[9]
If the current process is holding an exclusive lock already, changing the lock to a shared lock is simple.
[10]
Another process is holding an exclusive lock or is waiting to acquire one. If we would block waiting for the lock to clear and were told not to wait, return EBUSY to indicate this.
[11]
Go to sleep until exclusive locks are cleared and any upgrades are completed. This will loop until a lock could be acquired, incrementing the count of sleepers and releasing the interlock immediately before going to sleep, acquiring the interlock and decrementing the count of sleepers (in that order) when woken.
[12]
If we got an error while sleeping, waiting for the locks to clear, stop processing the lock request.
[13]
Increment the count of shared locks and break out of the switch statement to finish processing the lock request.
[14]
Increment the number of shared locks held and fall through to the code handling the downgrading of exclusive locks to shared locks.
[15]
Increment the shared lock count by the number of exclusive locks held by the calling process. Note that it is only possible to have more than one exclusive lock by the same process acquiring recursive locks.
[16]
Reset the count of exclusive locks held and clear the exclusive lock flag, changing the lock to a shared lock.
[17]
If anyone is waiting for the exclusive lock to be cleared, wake them up, then break out of the switch statement to finish processing the lock request.
[18]
If anyone else is waiting to upgrade a lock from shared to exclusive, decrement the shared lock count (releasing the shared lock) and return EBUSY. Otherwise fall through to the generic upgrade code.
[19]
Decrement the shared lock count, releasing the shared lock held by the current process.
[20]
If we are just polling for the upgrade and someone else has an exclusive lock or is waiting for an upgrade to exclusive lock, return EBUSY instead of blocking.
[21]
If no other process is waiting for a lock upgrade, we can do the upgrade to exclusive lock in a simpler and faster way. Otherwise, set the flag indicating that we are waiting to do an upgrade of the lock and go to sleep until all shared locks have been drained. If we get an error, abort processing and return the error code.
[22]
Set the exclusive lock flag and count of all currently held exclusive locks to 1 and break out of the switch statement to complete processing of the lock request.
[23]
If anyone else is waiting for an upgrade to exclusive lock, and we held the last shared lock (released above), wake them up now. Then fall through to the exclusive lock code to obtain the exclusive lock.
[24]
If we already have an exclusive lock, this is a recursive lock request. Just increase the count of exclusive locks held and break out of the switch statement to finish lock processing.
[25]
If we would block waiting for the lock (someone already has a shared or exclusive lock) and we are polling, just return EBUSY.
[26]
If anyone else has already started processing an exclusive lock request, go to sleep until is has been cleared. If we get an error while waiting, abort processing.
[27]
Set the WANT_EXCLUSIVE flag on the lock and go to sleep until all other upgrades and shared locks have been drained. When we wake up, clear the WANT_EXCLUSIVE flag. If we get an error while sleeping, abort processing. Note that we clear the WANT_EXCLUSIVE flag regardless of whether we get an error or not.
[28]
Set exclusive lock flag, set exclusive lock count to one and assert that we are holding the exclusive lock. Break out of the switch to finish processing the lock request.
[29]
If we are holding any exclusive locks, decrement the exclusive lock count.
[30]
If the exclusive lock count reaches zero, clear the exclusive lock flag.
[31]
If we do not hold any exclusive locks, but we do hold shared locks, decrement the shared lock count.
[32]
If anyone is waiting for locks to drain, wake them up before breaking out of the switch statement and finish processing the lock request.
[33]
If we would block waiting for locks or upgrades to clear, and we are just polling, return EBUSY.
[34]
Go to sleep until all upgrades and requests waiting for exclusive locks have been drained.
[35]
Clear the error status and repeat steps [36] through [38] while there are any exclusive locks, shared locks or lock upgrades in progress.
[36]
Set the WAIT_DRAIN flag and release the interlock on the lock structure.
[37]
Go to sleep until the lock flags have been changed, signalling a change in upgrades or exclusive locks. If we get an error while sleeping, abort processing.
[38]
Acquire the interlock on the lock structure again, making sure we hold it while we continue the processing of lock structure members which might otherwise change.
[39]
Set the draining and exclusive lock flags, making sure no other process squires a lock until we have finished processing the request.
[40]
If we have been draining locks, and there is no one waiting for exclusive locks or upgrades, clear the draining flag on the lock and wake up any processes waiting for the lock to change.
[41]
Release the interlock we held throughout processing the request and return any error status that might have been set.

lockstatus

The lockstatus() function is used to retrieve the status of a lock. The lock must have been previously initialized by lockinit(), but need not otherwise have been used.

The function will return a status value indicating whether any locks are held, and if so, what kind of lock (shared or exclusive). All accesses to internal lock data are protected by asserting the high-level lock's interlock while examining the lock data.

	/*
	 * Get the status of a lock
	 */
	int
	lockstatus(struct lock *lock)
	{
[42]		call simple_lock on lock->simplelock;
[43]		if (lock->exclusive_count) {
			call simple_unlock on lock->simplelock;
			return exclusive status;
		}
[44]		if (lock->shared_count) {
			call simple_unlock on lock->simplelock;
			return shared status;
		}
[45]		call simple_unlock on lock->simplelock;
		return unlocked status;
	}
[42]
Call simple_lock() on the embedded interlock to guarantee the lock does not change out from under us.
[43]
If there are any exclusive locks held, unlock the embedded interlock and return a value indicating that exclusive locks are held.
[44]
If there are any shared locks held, unlock the embedded interlock and return a value indicating that shared locks are held.
[45]
Unlock the embedded interlock and return a value indicating that no locks are currently held.

File system code

The file system code in 4.4BSD Lite2 has been modified to be multi-processor safe. In the following section we will examine how the vnode reclamation code uses locks around critical sections to serialize access to vnodes.

vgone

The system calls vgone() to eliminate all activity on a vnode. This is done whenever a vnode will be re-used, possibly for a different file system. The operation needs to be protected to avoid race conditions where the system might attempt to re-use the vnode due to a process re-opening the file previously associated with it on a different processor or another processor attempts to reclaim the vnode at the same time.

We do this by utilizing the struct simplelock embedded in the vnode and using the low-level lock functions to serialize access to it. Note that a low-level spinlock may not be held for the duration of the reclamation process. We also have to verify that the vnode is not already being reclaimed after acquiring the spinlock. If we determine that reclamation is already in progress, we release the spinlock and go to sleep until the reclamation process has completed.

We will also move the vnode to the head of the free list if it is already on it, to force the system to re-use it as soon as possible, since it is no longer associated with any underlying file system. Access to the vnode free list also needs to be serialized using a struct simplelock.

At the end of processing, the vnode will be marked as being "BAD". We have already associated the vnode with the "dead" file system by calling vclean() on it.

	/*
	 * Eliminate all activity associated with a vnode
	 * in preparation for re-use
	 */
	void
	vgone(struct vnode *vp)
	{
[46]		call simple_lock on vp->interlock;
[47]		if (vgone or vclean in progress) {
			call simple_unlock on vp->interlock;
[48]			wait for vgone or vclean to finish;
			return;
		}
[49]		call vclean on vp to close file;
[50]		remove from mount list if on it;
		if (vnode on freelist) {
			call simple_lock on free_list_lock;
[51]			move vnode to head of free list;
			call simple_unlock on free_list_lock;
		}
[52]		set vnode type to BAD;
	}
[46]
Acquire the vnode interlock to ensure the vnode does not change out from under us.
[47]
If the vnode is already being reclaimed we must let that complete and then release the vnode interlock.
[48]
Sleep until the concurrent vgone() or vclean() is done and return.
[49]
Use vclean() to close the underlying file and clear out any underlying file system specific data from the vnode.
[50]
If the vnode is on the mount point vnode list, remove it.
[51]
If the vnode is already on the free list, we want to move it to the head of the list so that it will be re-used sooner. Lock the free list, move the vnode to the head of the list and unlock the free list.
[52]
Set the vnode type to "BAD". It has already been associated with the "dead" file system by the call to vclean().

vclean

The vclean() function is used to disassociate the attached underlying file system from a vnode. Note that the embedded struct simplelock is always held on entry to the function. We also set the VX_LOCK flag on the vnode, marking it as currently being reclaimed before we continue processing the reclamation.

If we have been asked to close the file associated with the vnode, we will invalidate all associated buffers, forcing them to be written to the underlying file system. If the file associated with the vnode was open on entry, we call the underlying file system, closing the file, and deactivating the vnode. This operation will release the embedded struct simplelock. If the vnode is not currently active, we release the embedded lock.

We then call the underlying file system to reclaim the file system specific portion of the vnode, and if the vnode was active we call vrele() to release it and invalidate any references to the vnode in the buffer cache.

If there were any lock structures associated with the vnode, we free them, and point the vnode at the "dead" file system so that any attempt to do operations on the vnode will fail.

The vnode has now been completely reclaimed, and we can clear the lock flag we set at the start of processing and wake up any processes waiting for the reclamation to complete.

	/*
	 * Disassociate the underlying file system from a vnode.
	 * The vnode interlock is held on entry
	 */
	void
	vclean(struct vnode *vp, int flags, struct proc *p)
	{
[53]		save vnode active count;
[54]		obtain lock on vnode;
[55]		if (vnode is closing)
			invalidate associated buffers;
[56]		if (vnode is active) {
[57]			if (vnode is closing)
				call close routine;
[58]			inactivate vnode;
		}
		else
			release lock on vnode;
[59]		reclaim vnode;
		if (vnode was active)
[60]			call vrele to release vnode;
[61]		purge cache information about vnode;
[62]		free lock structures associated with vnode;
[63]		make vnode reference dead filesystem;
[64]		clear the reclaimation lock;
		wake up any sleepers on vnode;
	}
[53]
Save away the current active count on the vnode so we can use this to decide how to proceed after we have deactivated it. Also increment the reference count on the vnode so that no other process tries to recycle it before we have completed.
[54]
Mark the vnode as being locked for reclamation. Note that the low-level vnode interlock has already been acquired at this point. This lock ensures that no new processes will attempt to acquire any locks on the vnode. Thus, once all shared and exclusive high-level locks has been drained, it will be safe to deallocate the high-level lock structures in the VOP_RECLAIM() routine.
[55]
If the file associated with the vnode is being closed, invalidate all associated buffers in the buffercache.
[56]
Use the active count saved above to determine whether or not the vnode is currently active. If it is not, release the vnode interlock.
[57]
If we are closing the associated file, call the underlying file system to close it. This will release the vnode interlock that was acquired on entry.
[58]
Mark the vnode as inactive.
[59]
Call the underlying file system to reclaim the file system specific portion of the vnode.
[60]
If the active count saved above indicated that the vnode was currently active, call vrele() to release it.
[61]
Purge all information about the vnode from the vnode cache.
[62]
Free all lock structures associated with the vnode.
[63]
Point the vnode at the "dead" file system so that any references to the vnode will fail.
[64]
Clear the lock flag that was previously set in [52]. If any processes are waiting for the lock to be cleared, wake them up.

vrele

vrele() is called to remove references to a vnode. Changing the reference count has to be done inside a spinlock to serialize access to the vnode. If we removed the last reference to the vnode, acquire a spinlock on the vnode free list, append the vnode to the free list, and, release the free list spinlock. If we can get an exclusive lock on the vnode, call the underlying file system to inform it that we removed the last reference to the vnode.

	/*
	 * Release a vnode.  If count drops to zero, call inactive
	 * routine and return to free list
	 */
	void
	vrele(struct vnode *vp)
	{
		call simple_lock on vp->interlock;
[65]		decrement vp->usecount;
[66]		if (vp->usecount > 0) {
			call simple_unlock on vp->interlock;
			return;
		}
[67]		call simple_lock on free_list_lock;
[68]		append vp to free list;
		call simple_unlock on free_list_lock;
[69]		if (exclusive lock on vp succeeds)
			call inactive on vp;
	}
[65]
After setting the vnode interlock, decrement the count of active references to the vnode.
[66]
If there are still active references to the vnode, release the vnode interlock and return.
[67]
Acquire the lock on the free list to make sure no other processor attempts to add or remove vnodes from it.
[68]
Add the vnode to the vnode free list and release the lock on the free list.
[69]
Try to acquire an exclusive lock on the vnode; if successful mark it as inactive. Note that if the exclusive lock fails, the vnode interlock is released. The vnode inactivation will release the vnode interlock.

vget

The vget() function is called when we want to reuse a specific vnode, ie. when we open a file that has been open recently and for which the associated vnode has not been reclaimed for a different file.

The first thing that is done is to acquire the vnode interlock unless it was already set before vget() was called. We then check to see whether reclamation is in progress by vclean(). If it is, we flag that we want to be woken up once reclamation has finished, after which we release the vnode interlock. We then go to sleep until the reclamation process has finished and return ENOENT to indicate the vnode is no longer usable.

If there are currently no references to the vnode, it is on the vnode free list. Acquire a lock on the free list, remove the vnode and release the lock.

Increment the reference count on the vnode, and see of we were told to acquire a high-level lock on it. If so, try acquiring the requested locks, and call vrele() is not successful. Note that acquring the high-level lock has the side effect of clearing the vnode interlock. Return the status from attempting to acquire the high level vnode lock.

All processing has now been completed, release the vnode interlock and return.

	/*
	 * Grab a particular vnode from the free list, increment its
	 * reference count and lock it. If the vnode lock bit is set the
	 * vnode is being eliminated in vgone. The process is awakened
	 * when the transition is completed, and an error returned to
	 * indicate that the vnode is no longer usable (possibly having
	 * been changed to a new file system type).
	 */
	int
	vget(struct vnode *vp, int flags, struct proc *p)
	{
[70]		if (vnode interlock not set)
			acquire interlock;
[71]		if (vnode reclamation in progress) {
			flag we want it;
			release vnode interlock;
[72]			sleep until vnode reclamation done;
			return ENOENT status;
		}
[73]		if (no references to vnode) {
			acquire vnode free list lock;
[74]			remove vnode from free list;
			release vnode free list lock;
		}
		increment number of references to vnode;
[75]		if (high-level locks wanted) {
			if (we cannot acquire the high-level lock)
				call vrele on vnode;
			return status from acquiring high-level lock;
		}
[76]		release vnode interlock;
		return OK status;
	}
[70]
Unless we are told that the vnode interlock was already set at entry, acquire the vnode interlock now.
[71]
If the VXLOCK flag is set in the vnode, reclamation is in progress. We then set the VXWANT flag indicating that we will be sleeping until reclamation is complete and release the interlock.
[72]
Go to sleep on the vnode until the reclamation has been completed. When we wake up, return ENOENT indicating that the vnode is no longer usable.
[73]
If there were currently no references to the vnode, it has been put on the free list and we need to take it off.
[74]
Acquire a lock on the vnode free list, remove the specified vnode from it and release the lock.
[75]
If we were told to acquire a high-level lock on the vnode, attempt to do so now. Note that attempting to acquire the high-level lock will release the vnode interlock we acquired at the start. If we were unable to acquire the high-level lock, call vrele() to put the vnode back on the free list. Return the status code from attempting to acquire the high-level lock on the vnode.
[76]
All processing has been completed, and the vnode is ready for reuse. Release the interlock and return with the good news.

getnewvnode

The getnewvnode() is used to allocate a new vnode. It will attempt to reclaim vnodes from the vnode free list unless the free list is currently empty or the current number of vnodes is less than the desired number of vnodes on the system.

Start out by acquiring the lock on the vnode free list. Then, if the free list is empty and the current number of vnodes is less than twice the desired maximum number of vnodes on the system, or the current count is less than the desired number of vnodes, release the lock on the vnode free list. Allocate a new vnode and clear it and increment the total number of vnodes on the system.

If we did not allocate a new vnode as outlined above, we need to find one on the vnode free list. Walk down the free list, attempting to acquire the interlock on each vnode in turn. If we find a vnode we could lock, stop the process. If we walk to the end of the free list without being able to lock a vnode, release the lock on the free list and return ENFILE status.

Having found a vnode on the free list, we take it off the list. We also mark the vnode so that vgone() will not move it on the free list. We are now done with the free list, and release the lock we acquired at the start of processing.

If the vnode found above has not been completely cleared out, we now call vgone() on it to release any file system specific data associated with it. Note that this will release the vnode interlock. If the vnode had already been cleared out, we simply release the interlock.

Regardless of wether we allocated a new vnode or picked one out from the free list, we now set values according to the arguments passed. We start by setting the type to NONE, indicating that it is not currently associated with any file. We call cache_purge() to make sure any old data associated with the vnode has removed from the cache. We then set up the vnode operations table according to the vops argument passed and insert the vnode on the specified mount queue.

Last, we set the current number of references to the vnode to one and return status indicating success.

	/*
	 * Return the next vnode from the free list
	 */
	int
	getnewvnode(enum vtagtype tag, struct mount *mp, int (**vops)(), struct vnode **vpp)
	{
		acquire lock on vnode free list;
[77]		if ((vnode free list empty && current vnode count < twice the desired count) ||
		    current vnode count < desired vnode count) {
			release lock on vnode free list;
[78]			allocate new vnode and clear it;
			increment the vnode count;
		}
		else {
[79]			walk vnode free list until we can acquire interlock on vnode;
[80]			if (no vnode could be locked) {
				release lock on vnode free list;
				return ENFILE status;
			}
[81]			remove vnode from vnode free list;
[82]			mark vnode so vgone() will skip it over;
			release lock on vnode free list;
[83]			if (vnode not pointing to dead file system)
				call vgone to release associated file system data;
			else
				release vnode interlock;
[84]			clear vnode data;
		}
[85]		set vnode type to NONE;
[86]		call cache_purge on vnode;
		initialize vnode from arguments;
[87]		insert vnode into the specified mount queue;
		set reference count on vnode to 1;
		return OK;
	}
[77]
Start out by acquiring a lock on the vnode free list. Then, if the current number of vnodes is less than the desired number of vnodes or the free list is empty and the current number of vnodes is less than twice the number of desired number, allocate a new vnode.
[78]
Release the lock on the vnode free list and allocate a new vnode. Clear the newly allocated vnode and increment the total count of vnodes we have.
[79]
We have found that we need to get a vnode from the free list. Start walking down the free list, attempting to acquire an interlock on each vnode visited. If we find one we are able to lock, stop walking the free list and continue processing.
[80]
If we could not find and acquire the interlock on any vnode above, release the lock on the free list and return ENFILE, indicating that no more new vnodes can be allocated or opened.
[81]
Having found a usable vnode on the free list, take it off the free list.
[82]
Mark the vnode with the value 0xdeadb which is special-cased in vgone(), making sure it does not get get reordered on the free list.
[83]
If the vnode from the free list was not maked as "BAD", indicating it has been disassociated from the underlying file system, call vgone() on it to release any associated file system data. This has the side effect of releasing the interlock on the vnode. If it had already been disassociated from the underlying file system, just release the interlock acquired above.
[84]
Now that we are ready to reuse the vnode, clear out fields that might have been used previously.
[85]
Set the vnode type to "NONE", indicating it is not currently associated with any specific file type.
[86]
Call cache_purge() on the vnode, ensuring that any data previously associated with it is invalidated. Then use the arguments passed into getnewvnode() to associate the vnode with the specified file system.
[87]
Use insmntque() to insert the vnode onto the specified mount queue and set the reference count to one. Return success to the calling function.