Locking notes for network stack

1. the second argument to the mutext_*() calls will always be consistant.
2. The argument to XXX_LOCK() can be something more practical, like the structure that contains the mutex, rather than the mutex itself.
3. To add debugging code into the lock calls, there is only one place to add it for each lock type.

So now you all can see what sort of things I'm doing, and can offer up comments if you see something that worries them. (This is too much information to put into a status report.)

-David Borman, dab@bsdi.com

SOCKET BUFFERS

    Sockets have a send and receive socket buffer.  Each socket buffer
    has a lock.

    SOCKBUF_LOCK(sb)	Locks the socket buffer via sb->sb_mtx
    SOCKBUF_LOCK(sb)	Unlocks the socket buffer via sb->sb_mtx

    sbwait(sb)		"sb" must already be locked when called.
    sblock(sb)		"sb" must already be locked when called.

    sorflush(so)		so->so_rcv will get locked and unlocked.

SOCKET LISTEN/ACCEPT QUEUES

    The socket accept queue (so_q0, so_q, so_q0len, so_qlen, so_qlimit)
    is protected by the lock on the receive socket buffer (so_rcv).
    The functions soqinsque() and soqremque() assume that the lock
    is held before they are called.  This is so that a socket can
    be moved from the half-open queue to the established queue with
    one lock/unlock around the pair of soqremque()/soqinsque() calls.

INTERFACE QUEUES

    Interface queues are operated on by a set of macros:
	IF_QFULL()	Check and see if the queue is full.
	IF_DROP()	Mark the queue that a packet was dropped.
	IF_ENQUEUE()	Add a packet to the end of the queue.
	IF_PREPEND()	Add a packet to the front of the queue.
	IF_DEQUEUE()	Remove a packet from the front of the queue.

    Three new macros are added:
	IF_QLOCK(ifq)	Lock the interface queue.
	IF_QUNLOCK(ifq)	Unlock the interface queue.

    Most of the calls to IF_QLOCK()/IF_QUNLOCK() are buried in
    the macros.  IFQ_PREPEND() and IF_DEQUEUE() call IF_QLOCK()
    and IF_QUNLOCK() as necessary.  For the other macros, the
    typical coding scenario is:
		if (IF_QFULL(ifq))
			m_free(m);
			IF_DROP(ifq);
		} else {
			IF_ENQUEUE(ifq, m);
		}
    So, we can still bury the locking macros by having IF_QFULL()
    call IF_QLOCK(), and having IF_DROP() and IF_ENQUEUE() call
    IF_QUNLOCK().

    Also, for code that flushes an interface queue by looping
    calling IF_DEQUEUE(), rather than doing:
	for (;;) {
		struct mbuf *m;
		IF_DEQUEUE(ifq, m);
		if (m == NULL)
			break;
		m_freem(m);
	}

    the lock can be pulled outside the loop by doing:

	IF_QLOCK(ifq);
	for (;;) {
		struct mbuf *m;
		IF_DEQUEUE_NOLOCK(ifq, m);
		if (m == NULL)
			break;
		m_freem(m);
	}
	IF_QUNLOCK(ifq);

    which will avoid doing a lock/unlock for each packet that is
    pulled off the queue.

IPv4 PROTOCOL CONTROL BLOCKS

    Each protocol control block (struct inpcb) contains a lock.
    The lock covers everything in the block and chained off of
    it (via inp_ppcb), except for the pcb linkage pointers (inp_next,
    inp_prev, inp_hash).

    There are multiple chains of PCBs (tcb, udb rawinpcb).  The
    heads used to be "struct inpcb", but now there is a new
    structure, "struct inpcb_head" which is strictly for defining
    the head of a chain.  The lock in the head protects all the
    linkage pointers in the chain (forward & back pointers, and
    hash queues). 

    The locks in the control blocks only need to be write locks,
    the lock on the chain head should be a read/write lock.  Hence,
    the new structure for the head of a PCB chain.

	INP_LOCK(inpcb)			Lock an inpcb
	INP_UNLOCK(inpcb)		Unlock an inpcb

	INP_HEAD_RLOCK(inpcb_head)	Read lock on an inpcb_head
	INP_HEAD_WLOCK(inpcb_head)	Write lock on an inpcb_head

	INP_HEAD_RUNLOCK(inpcb_head)	Read unlock on an inpcb_head
	INP_HEAD_WUNLOCK(inpcb_head)	Write unlock on an inpcb_head

IP REASSEMBLY QUEUE

    The IP reassembly queue (ipq) had an "ip_doing_reass"
    variable which protected ipq if ip_drain() got called
    at interrupt level.  There are now two new macros:
	IPQ_LOCK()	Lock ipq
	IPQ_UNLOCK()	Unlock ipq
    that protect all references to ipq and the "ip_doing_reass"
    variable is no longer needed.

LOCK HIERARCHY

    The locking hierarchy is defined to allow an interrupt
    thread to allocate locks in a natural order.  Hence, the
    locking order is:
	"pcb head" -> "pcb" -> "socket send/recv queue"
    Read/write calls will have the recv/send socket queue
    locked when they call down to the protocol specific code.
    That code will drop the lock on the recv/send queue,
    lock the pcb head, then the pcb, and then reget the lock
    on the recv/send queue.

    Interface queues are leaf locks, meaning no other locks
    are gotten while holding an interface queue lock.

IP FILTERS
    The ipfw_t structure now has a mutex.  Two new macros are
    used for locking/unlocking:
	IPFW_LOCK(struct ipfw_t *)	Lock a filter
	IPFW_UNLOCK(struct ipfw_t *)	Unlock a filter
    The caller checks for a function, the if there is one it
    locks the filter and rechecks for a function, if there is
    still one it calls it, and then it unlocks the filter.
    The sysctl code which munges with the filters also locks the
    appropriate filter before munging with it.

    The function ip_rateoutput() is not real clean, the current
    version can return bogus error codes and send mbufs into
    a black hole...

LOCAL DOMAIN SOCKETS

    To-Be-Done

ISO - sys/netiso
    I am not planning on doing any multithreading work for ISO.