Unfinished snapshot of my current project, which aims to reimplement
and generalize the existing package build cluster into a general
purpose distributed compute cluster for batch jobs as well as
interactive users.

ZFS is used as the filesystem layer, with snapshot replication from a
server used to instantiate virtual OS instances (via jail(8)).  ZFS
clone instantiation is extremely cheap, requiring 2-3 seconds to
dynamically add a new OS instance.

A queue manager maps job requests onto available machines, and is
implemented using an SQLite backend that supports a simple resource
description language used to specify the machine constraints that can
execute a job.

Various services will be (re-)implemented on top of this, including:

* The package build infrastructure, which already uses a similar but
  much less flexible system for dispatching build jobs onto clients
  machines

* Various existing parallelizable jobs that can benefit from running
  on the dozens of machines we have available (ghetto Map/Reduce ;-)

* Other existing computational jobs that currently are confined to
  dedicated hardware with (or without!) their own management
  infrastructure

* Future planned computational jobs including the performance
  monitoring cluster framework currently in development

* Interactive sessions from FreeBSD.org developers who want access to
  test machine images for compilation testing/development on foreign
  architectures

Directory:

zfs/	   : zfs snapshot replication layer

freebsd.py : module providing UNIX domain socket credential passing
	     (missing in python!)

qmanager/ : queue manager for mapping job requests to build machines.
	    Incomplete but minimally working.

package-build : First pass rewriting the core package build dispatcher
	      	in python to remove some limitations of the previous
	      	system.

mapindex/  : proof of concept distributed index build