Sean Chittenden @ FreeBSD

seanc
e-mail | home | pics | pgp

datacenter

It's trendy to have a page with a TODO file of things that you'd like to do, but infrequently gets worked on due to errno == ENOTIME. servers
Links:
TODO:
  • Add detection for X11 to the net/cvsup port
  • Add backtick support for postgresql.conf
  • Add gnats support to kbugbuster
  • Implement setval(TEXT,BIGINT) so that when BIGINT is NULL, resets is_called to NULL so next call to CURRVAL() returns an error.
  • Implement recvfrom(2)
  • Write a KDE taskbar widget that queries available SSIDs and lets users set the SSID/change wireless configs
  • Finish my doc/en_US.ISO8859-1/articles/spam-filter article for FreeBSD.org and submit for review from the community.
  • Update internal/committers-guide (ala peter), "it's out of date"
  • Finish bde's patch in PR kern/25777 so that the atime of a file isn't updated when any user exec()'s the program in question
  • Track down the bug where tsleep() ("sbwait") has sb_timeo is set to 0 and blocks forever inside the kernel. This only happens on SMP machines as far as I can tell. I think this system call should be interuptable.
  • Update sendfile(2) so that there are more statistics exported via sysctl(8) to the administrator
  • Update the 4.x NFS lockd code so that NFS lockd works on the client side.
  • Update PostgreSQL to use kqueue()
  • Update PostgreSQL to use sendfile(2)/recvfile(2)
  • Track and commit the O(1) search for appending mbufs to an incoming or outgoing socket buffer
  • Dynamically allocate the number of groups that a user can have (kern.ngroups -> softlimit)
  • Begin adding EXAMPLE sections to FreeBSD's man pages
  • Add libxml/libxslt to the base system
  • Build ports with the "ports" user/group and install data into chroot
  • Finish the DTD/XML file for security advisories
  • Geographic ports distribution
  • Commit the VPN via pppd and ssh article
  • centralize documentation of kernel configuration tunables
  • intergrate the doc build process between DocBook and man pages
  • create a tool that'll automatically alert an admin if they have vulnerable software installed on their system via the ports
  • update the ports system to use something more advanced than Makefiles
  • make FreeBSD have soft limits for everything (load once, tune forever and only restart to add hardware or update the kernel: never to change kernel tunables or the user/kernel memory split)
  • Add kqueue() support to Ruby instead of using select()
  • Finish updating ruby-snmp to support NET-SNMP 5.0
  • Continue to work on/finish ruby libxml/libxslt, specifically SAX/TextReader/Schema
  • Update sysinstall to set autoboot in /boot/loader.conf to 2-3 seconds for workstations and 10sec for servers
  • Rewrite cvsup in C using KSE/kqueue()
  • Add a feature to shutdown(8) so that shutdown(8) will spawn a thread that will sleep for a given amount of time and if the box hasn't been shutdown, exec 'reboot -n' as a last ditch effort to reboot the box (useful on headless servers where admins are far away from home and there is faulty hardware).

SPAM - A Necessary Byproduct of a Free Market

See my note on SPAM: SPAM is a necessary byproduct of a free market economy and suggests the existence of idiots on the Internet.


Patches

This is a cluttered up version of what you can find on my page 'o patches. I've automated this system sooo, if anyone would like to lift some of these bits, let me know and I'll give you a quick intro to XML/XSL.

DateTypeStatusPatchDescription
2006-07-04srcOutstandingRELENG_6 HEAD Have adjkerntz(8) exit cleanly and silently when run in a jail(8)'ed environment.
2006-06-20srcOutstandingRELENG_6 HEAD Add the appropriate tunables to give jail IPs a netmask that way a jail+if_vlan can be the only IP/alias from a host in a given subnet. Ex:
cloned_interfaces="vlan598"
ifconfig_bge0="media 100baseTX mediaopt full-duplex up"
ifconfig_vlan598="vlan 598 vlandev bge0 up"
jail_foo_rootdir="/jails/foo.example.com"
jail_foo_hostname="foo.example.com"
jail_foo_ip="10.42.42.254"
jail_foo_netmask="255.255.255.0"
jail_foo_interface="vlan598"
jail_foo_devfs_enable="YES"
jail_foo_mount_enable="YES"
...
2005-06-02srcOutstandingHEAD Sometimes there are applications that you have to run in Linux compat mode and sometimes those programs are so braindead that they use back asswards syscalls such as setfsuid16() or setfsgid16() that are no-op's on FreeBSD and don't change the execution of the program, but, it's less than desirable to have dmesg(1) filled with these worthless messages. This patch adds a sysctl(8) value compat.linux.syscall_warn_enable that lets one disable all compat warnings.
2005-03-22srcOutstandingHEAD Fixes the use of "sufficient" for authentication PAM modules. Required for some pam configurations. Ex:

auth sufficient pam_rootok.so no_warn
auth sufficient pam_self.so no_warn
auth required pam_deny.so no_warn

2004-12-09FreeBSD 5.XOutstandingRELENG5 This patch fixes getcwd(3) on RELENG5 on unionfs's.
2004-01-13PostgreSQLOutstandingHEAD On machines that have TCP_NOPUSH defined in netinet/tcp.h, this will change all libpq connections to use setsockopt() TCP_NOPUSH. This hasn't been widely tested in terms of performance, but it _should_ make receiving data faster.
2003-09-18portsOutstandingHEAD The patch supposed by ISC doesn't apply cleanly to the ports/dns/bind9 port. This patch, however, does and is functionally identical.
2003-07-20srcOutstandingHEAD Adds a kern.nbuf sysctl, making it possible to see what the nbufs are set to. Also documents kern.nbuf in loader(8) and /boot/loader.help
2003-04-16PostgreSQLOutstandingHEAD Implements the various SHOW commands in psql to appease the whining MySQL masses. PostgreSQL_usability++, MySQL--
2003-03-13docOutstandingHEAD White space reduction: noticed while fixing a link. Gratuitous tabbing in emacs to indent properly is fine so long as you remove your newly added trailing whitespace.
2003-03-12docOutstandingHEAD Add the appropriate #include where environ is defined. It's actually stand.h but unistd.h is the correct header to include.
2003-03-10docCommitted - 2003-03-12HEAD This patch updates sendfile(2) to include a TUNING section with some advice along with an update to tuning(7) to reference the updated sendfile(2) man page.
2003-03-10src/docOutstandingHEAD For a system administrator trying to learn about the kernel and its limits/parameters, it can be quite problematic to have to trudge through the handbook and various man pages scattered about to learn about a specific sysctl MIB that may or may not be documented. This patch addresses this by creating a new man page, sysctl(5), that is used as an index for searching for what man pages contain various references to sysctl MIBs. This man page scales well in that there can be multiple Xr macros and the references will wrap nicely. Maybe tomorrow night I'll create the stub of a man page that can be used for the kernel config and start pushing kernel config bits into various man pages.
2003-03-09srcOutstandingHEAD whitespace

Updates the case of sendfile() when there aren't any sf_buf's available. Instead of calling msleep() and blocking the caller on a socket that has been marked non-blocking, return instantly with EAGAIN. This doesn't provide a mechanism for identifying that there aren't any sf_buf's available. At some point a read only sysctl should be added that lets an administrator know how many sf_buf's are free (max number already exists so it should be trivial for an admin to figure out how many are in use), but that will come at a later date (it's 5:30am, ENOSLEEP). Returning control to the program should dramatically improve the responsiveness of a non-blocking application that uses sendfile() since it gives the application the chance to continue to work. Ideally sf_buf's are freed up for use making it possible to deal with the load described. Currently when this limit is hit, it kills the concurrency of the server that non-blocking IO affords and the throughput of a system drops from __Mbps down to near 0Mbps (I have some cool bandwidth graphs to prove this). With this patch, at the very least the server should be able to continue to send data at __Mbps.

There is a race with this code in that if a non-blocking socket that has had sendfile() called on it where there wasn't an sf_buf available, and it is set back to being a blocking socket, sf_buf_alloc_want will never reach zero and as a result, wakeup_one(&sf_freelist) will be called every time in sf_buf_free(). I'm not sure how best to fix this if it should be, or even what the behavior of wakeup_one(&sf_freelist) will be if it is called when there isn't anything msleep()'ing on sf_freelist.

Another something that I'd like to do is to alert the server admin that he/she's out of sf_buf's the same way that the kernel does when it runs out of nmbclusters.

If someone doesn't believe this is a problem, look at the bandwidth graph from a single box of mine when it hits this limit (remaining traffic is from servers that use writev(), not sendfile()).

UPDATED (2003-03-10): This patch is updated to include Alan Cox's changes to re-use sf_buf's for pages that are already in use by the server. This patch enables re-use of sf_buf's, which is a likely common case for sendfile(2).

2003-03-04docCommitted - 2003-03-04HEAD Chump update to note that kern.ipc.nsfbufs can be adjusted at boot time.
2003-03-04srcCommitted - 2003-03-04HEAD Updates loader.8 and /boot/loader.help (actually help.common) to indicate that kern.ipc.nsfbufs can be updated at boot time.
2003-03-03docCommitted - 2003-03-04HEAD +whitespace Updates the handbook to include a few bits about sf_buf exhaustion and tuning NSFBUFS.
2003-02-16docCommittedHEAD This updates the firewall rules in the Dialup Firewalling article to use IPFW2 and to be substantially more secure (as well as less error prone and simpler).
2003-02-14docCommittedHEAD This patch fundamentally provides a small update to the example .emacs file to add highlighting of whitespace. It also nukes some whitespace and americanizes the spelling of two common words: intepreted -> interpreted and optimise -> optimize.
2003-02-11portsOutstandingHEAD Along with an updated random(6) utility, the following patch is used to randomize the MASTER_SITES for downloading distfiles for ports
2003-02-09srcCommitted - 2003-03-05HEAD Updates src/sbin/ping to be WARNS=2 compliant
2003-02-08srcCommitted - 2003-02-15-CURRENT Patch for -CURRENT (should work for 4.X and even probably 3.X). Updates random(6) to be able to completely reorder a file based off of lines or words as specified by isspace(3).
2001-02-07srcOutstandingHEAD on all *BSD version, socket buffers contain a list of incoming and/or outgoing mbufs. Unfortunately the list only has a pointer to the head, meaning that all append operations require to scan the full list. The overhead can be very bad in some cases (e.g. small UDP packets), and becomes worse and worse as the socket buffer size increases (which is what one would commonly do when expecting a lot of traffic!). The attached patch implements a tail pointer to the list, so that you can append in constant time. By default, the code works exactly as before -- the tail of the list is reached with the usual linear scan, and the pointer to the tail is only used for comparison purposes to make sure that it yields the same value. If you enable the fast behaviour with sysctl -w kern.ipc.fastscan=1 then the new code takes over and linear scans are replaced by dereferences of the tail pointer. Apart from the obvious benefits of using O(1) instead of O(n) algorithms, your mileage may vary. When the socket buffer is almost always empty (fast receivers) then you have no gain. When the socket buffer is almost always full you also have no gain, because the decision to drop the packet only requires a comparison. However, this code can really avoid trashing in those cases where the queue size oscillates. I'd like to commit this (or similar) code after proper testing, so i'd like people to try it out -- I am reasonably confident about it, and have done a fair amount of testing under heavy udp load, but want to be sure that there are no side effects.
1999-06-01srcOutstandingA patch to add NEWALIAS and NETBIND to the kernel. Makes adding entire cidr blocks as aliases possible.

OSSAL - Open Source Software Alliance License

I'm a firm believer in BSD Licensed software and think the GPL is the antithesis of a good software license. Please read the discussion section of the OSSAL license for details and feel free to send any comments you may have regarding OSSAL to <seanc at FreeBSD dot org>.


BIND9

Here's a quick ditty on getting bind9 to run in a chroot() jail.


PostgreSQL

Ten times out of ten, PostgreSQL will eat MySQL for lunch (and twice on Sunday).

PostgreSQL vs. MySQL

I don't benefit from this at all, but, if you're a PostgreSQL user, DBA, programmer, hacker, etc., and like that logo (someone at Command Prompt, Inc. did it), then you may be interested to know that you can get a baseball jersey, golf shirt, stainless steel travel mug, stein, or a few other odds and ends at http://www.cafeshops.com/linuxports.

PostgreSQL - Because MySQL, ORA, & DB2 suck.FreeBSD - Because Linux & Win32 suck.