Descrypture

I've started writing an article about PAM for the FreeBSD Documentation Project. It's far from done, but I've checked in what I've written so far (doc/en_US.ISO8859-1/articles/pam/) and an HTML rendition is available online here. It's looking pretty good so far, if I may say so myself.

I was in Bergen on R&R from Thursday until early this afternoon, and came back nicely rested from a weekend with my fiancée. This doesn't mean I've been idle, though. We spent the first day and night babysitting a friend's son, and he (the father, not the son) has a microwave link to the University of Bergen, so I got some work done, mostly on a Perl script for indexing Majordomo archives, which I wrote for the buggers archive.

Upon my return, I had quite a bit of catch-up to do, most of it Bugmeister-related. I started by implementing something I've always missed: a tool to perform simple PR operations without going through the hassle of edit-pr(1). I ended up with a pair of scripts, one of which sets some environment variables and invokes edit-pr(1) with EDITOR pointing to the second script, which makes the required changes to the PR and fills in the audit trail entries with the reason provided. Very nifty:

des@freefall ~% pradm close 32294 'MFCed, thanks!'
vipr: editing PR 32294...
vipr: editing audit trail entry...
edit-pr: filing PR docs/32294 back into the database...
    

I've also decided to hand out bugbuster hats to the following persons:

• Ian Dowse
• Sheldon Hearn
• Ruslan Ermilov
• Pete Fritchman
• Arnaud Launay
• Ashley Penney

Oh, and I closed or feedback'ed a handful of PRs. Attaboy.

I miss my cats though.

I've also spent some time reading Andrew Morgan's PAM whitepaper and browsing through the original OSF RFC and The Open Group's preliminary draft for PAM.. The OSF and X/Open documents describe PAM as it was originally defined and implemented by Sun, while Andrew Morgan tries to go one step further by separating out the conversation functions so they can run on the client rather than on the server. It's a good idea, but the design is flawed (one problem is that the proposed protocol does not include mechanisms for exchanging information about library versions and supported features; the paper suggests that this should be negotiated out of band!) and it does not address or even acknowledge the event vs. callback problem I discussed in an earlier issue. One very good thing about Morgan's paper, though, is that it establishes terminology for most of the various components and actors involved in a PAM session. Where previous writers used user and remote user, for instance, which is confusing not only because remote user includes the word user but also because the remote user is not necessarily remote, Morgan chooses to use user and applicant. Not quite perfect, but certainly a huge step in the right direction.

I tried to submit patches to Linux-PAM through their SourceForge site, and came away quite rankled and frustrated. Using SourceForge's bug- and patch-tracking system is a gruesome experience. There's no way to preview a ticket before it's filed, so you can't tell if you goofed, and there's no way to correct a ticket after you've filed it either; though if you're logged in, SourceForge will helpfully present you with the full ticket-editing interface with lots of controls for modifying the ticket, removing attached files etc., which—unless you're an administrator for the project to which you submitted the ticket—do absolutely nothing except clutter your screen and render the ticket essentially unreadable. Never mind the fact that SourceForge can easily verify that you are the original submitter, since you were logged in all along. To top it all off, if you screw up while submitting a ticket, SourceForge will present you with a nice, big, red error message, but it will not return you to the submission form and let you correct your mistake—nor will it tell you that it has in fact already filed the erroneous ticket! You are then left to choose between: filing a new (hopefully correct) ticket, and hoping the project admins won't be ticked off at your spammage; submitting a follow-up to the incorrect ticket explaining what is wrong, and hoping the admins won't be ticked off at having to manually effect the changes you specify; or throwing up your arms in disgust and going out for a beer.

I used to think SourceForge was a neat idea—then I started using it.

Yes, you may quote me on that.

In the past few days, I've looked (or at least glanced cursorily) at four different open-source bug-tracking systems: Request Tracker, Mantis, Bugzilla and Scarab. They all have their strength and weaknesses. Here's a summary of my impressions:

• Request Tracker has an impressive feature list and a nice web interface. Unfortunately, it won't even let you view tickets without logging in. It seems to be geared towards centralized organizations, where the set of people who submit requests is clearly distinct from the set of people who handle them. This does not apply very well to an massively cooperative open source project like FreeBSD.
• Mantis also suffers from the login syndrome, but has a simpler and cleaner web interface and some very nifty features like user-selectable interface language. On the downside, it's very beta, and it doesn't seem to support (or even plan to support) email submissions.
• Bugzilla seems to be all the rage these days. It's free from the login disease, and supports email submissions (not natively, though - it's a third-party add-on). However, it emulates SourceForge in that it presents users with the full bug-editing interface, even when the user is not authorized to edit anything. I also found the search page quite disorganized, crowded and intimidating—I had to look hard to find the submit query button.
• Scarab seems like an ambitious, but well thought-out project. It does however suffer from a terminal case of login disease; you can't even see a summary without logging in, and unlike Request Tracker and Mantis, there is no guest account. I didn't bother registering to try it out.

So where does this all lead to? I think my feelings about these systems can be summarized as follows: if I needed a ticket tracking system for my company, I'd use Request Tracker. If I were picking a bug tracking system for an open source project that didn't already have one, I'd choose Mantis, or Bugzilla if I were worried about not knowing PHP well enough to make local hacks to Mantis. But if I were picking a bug tracking system for an open source project that had previously relied on Gnats and had a database of 30,000 problem reports, I wouldn't pick any of these systems. They are all, in different ways, so much of a departure from Gnats that the transition would be too painful - and they would all require a significant amount of glue code to integrate with the existing infrastructure (such as the installed base of FreeBSD systems with Gnats' send-pr(1) utility). FreeBSD's problem report database is such an important part of the project's workflow and methodology that replacing Gnats with a system designed to fit somebody else's workflow and methodology is simply not an option.

I won't even try to explain. Just read the paper.

I spent some time today writing a man page for the pam_ssh(8) module, and updating the pam_unix(8) man page. Next on my list is a quick scan of the other existing man pages, just to check for any glaring errors or omissions, then PR docs/32294, then any other missing man pages, then a pam_module(8) man page explaining how to write a PAM module. That done, I'll return to PAMifying userland programs like passwd(1) etc.

Long time no write! I've spent the last couple of days playing around with neural networks and writing a neural network engine in Java which I've tentatively christened Jinn. It's far from done yet, but I'm going to have to shelve it for a little while and get back to the harsh reality of PAM and problem reports.

The next item on my Bugmeister to-do list is to write a set of guidelines for handling problem reports; I'm going to air this on the list and see what the buggers come up with. As for PAM, I'm going to stray from my to-do list for a while once I've committed the stuff I did last week; first because I've received some external input I want to look at and reflect upon, and second because I want to take some time to look at the newest Linux-PAM and see if they've done anything worth nabbing since we imported PAM into our tree.

I've been tinkering with a script that converts Gnats PRs (either single PRs or entire directories) to some sort of XML. I haven't decided on the schema yet, though, so it's still quite unfinished. Next on the list will be to import new PRs into an XML database (a related yet slightly different problem).

Oh, and I'm starting to think Rik Rose was right about using an SQL database as a backend for the PR system. It does simplify things a lot. The requirement for easy distribution can be satisified by regularly exporting the database to some human-readable format and distributing that.

Finally, memo to self: I still have to make slides for my NUUG presentation, and try to write that PAM-NG paper I've been thinking about.

I've created a mailing list at buggers@ofug.org for informal discussions about the PR system. The charter is:

It's open, so go ahead and subscribe, if you think you have something to contribute. To subscribe, send mail to majordomo@ofug.org with subscribe buggers in the body.

The article will eventually reside under the the FreeBSD documentation page, but I haven't connected it to the doc or www build yet, so for the time being it's here.

From: Jordan Hubbard <jkh@winston.freebsd.org>
Subject: Announcing the creation of "The Bugmeister"
To: developers@freebsd.org
Date: Wed, 21 Nov 2001 08:14:30 -0800
Message-ID: <38374.1006359270@winston.freebsd.org>
Sender: owner-developers@FreeBSD.org

Core is pleased to announce that as of Tuesday, November 20, we've
created the position of "Bugmeister", along with a really outlandish
looking fruit-covered hat to go with it, with the following charter:

  The Bugmeister manages the Project's bug tracking system.  His
  primary responsibilities are to define a consistent policy for
  handling problem reports, and to develop and maintain tools that
  will encourage and facilitate handling problem reports in accordance
  with this policy.  The Bugmeister may delegate some of his tasks to
  a team of Bugbusters, whom he will appoint from the developer and user
  communities.

  The Bugmeister and Bugbuster positions are intended to supplement,
  rather than replace, the important part played by CVS committers in
  handling problem reports.

Dag-Erling Smørgrav (des@FreeBSD.org) has kindly volunteered for,
helped define, and rammed through core with sheer bloody-minded
persistence, the Bugmeister position.  We thank him for his
perseverance and willingness to serve!

- Core
    

1. Document the position in all the right places in the Handbook etc.
2. Create a -bugbusters mailing list and start recruiting Bugbusters.
3. Write a set of guidelines for handling PRs.
4. Write a set of guidelines for submitting PRs.
5. Organize a systematic sweep of the PR database to close stale or bogus PRs.

Tony Finch sent me a link to a very interesting essay on how to write good problem reports.

• We need a good set of guidelines for PR submitters — a document we can point them to when we close a PR because it contains too little information or whatnot. Michael Lucas has written a nice article about PRs, but he's planning to use it in a forthcoming BSD book, so he can't donate it to the project. I'm going to try to write something up in the near future.
• We should also have a set of form letters we can use when closing PRs, especially for "rejected" PRs (PRs that were misfiled, or do not contain sufficient information to reproduce and debug the problem, or just ask a question which should have gone to one of the mailing lists, etc.)

Short entry today, my girlfriend is visiting and has set a quota on tube time, but I'd like to discuss some of the feedback I've received on yesterday's entry.

One thing everybody I've heard from seems to agree on is that a web interface (not just for queries, but for actual maintenance and administration) is a strong requirement, whereas I had placed it in the "would be nice" category. Actually, this suits me fine, as the framework I was thinking of using for the PR system is one I originally wrote for web applications.

I'd like to elaborate on the idea of to-do lists or progress reports. I like the idea of a PR system that allows a committer or PR worker to define a list of tasks that need to be performed before the PR is closed (for instance, "reproduce the bug", "develop a patch", "get feedback from originator", "commit to -CURRENT", "commit to -STABLE"). The system might even automatically close the PR when the last task is completed. One obvious extension of this idea is that a committer can open a PR (assigned to himself) that describes a project he is working on, and use the task list feature to set up a list of milestones, and close each task as the milestone is reached.

Richard Rose suggested a hybrid model for the PR database — an SQL database with references to separate files that contain the actual PR text. XML would only be used as a transport format. Personally, I think distributability (if such a word exists) is more important than performance or ease of implementation. Also, we're looking at a very simple schema, with practically no relations between tables, so SQL won't really give us much. To me, the ability to just add gnats release=current prefix=/usr/local/gnats to your supfile and then use grep(1) and less(1) to browse PRs, or to quickly write simple PR-manipulation tools in Perl without having to install MySQL and a heap of DBI and DBD modules, is more than I'm willing to give up.

There are a few synchronization and performance issues an SQL database would help with, and we'll have to handle those manually, so to say, but they're well-understood problems with straightforward solutions (lock files for synchronization, separate index files for faster searches).

Richard also suggested writing the system in Java, though he mentioned a few caveats that might tip the balance in favor of Perl. I've also thought about Java, which is a great language for certain purposes (and has very good XML support), but the facts that Perl is in our base system whereas Java is not, that Perl is almost consistently faster than Java, and that Perl CGI scripts can run as-is on the FreeBSD Project's web server while Java programs would require installing a JDK plus either the addition of JServ or wrapper scripts to start the virtual machine, all speak in Perl's favor. Not to mention that I already have a non-trivial amount of Perl code I may want to reuse for this project :-)

Some people also suggested doing away with -core altogether. Tcha, you know what? Uh-uh. We need -core, even if it's only to appoint "hats" and serve as a conflict-resolution body. The only alternative I see is holding a referendum every other week or so, and that gets old real quick, as anyone who's ever tried to get more than three committers to agree on something should know. We need a body of elected officials, and if we really don't like the way they're running things, well, we have bylaws which spell out the mechanism for calling an early election: just get one-third of all active committers to sign a petition. But don't do it on a whim. The whole point with electing leaders is continuity and stability, and we'll never get that if we keep calling early elections every six months.

• The new system must support the old send-pr(1) script, or we'll strand everybody who isn't running the latest and greatest. This is an absolute, non-negotiable requirement.
• There should be front-end CGI scripts with the same names as the ones we currently have for GNATS, and they should accept the same arguments — at the very least, query-pr.cgi?pr=12345 has to work.
• The backing database should store each PR in a separate, human-readable text file. This is necessary so mirrors of the database can still be distributed using CVSup and inspected (and if need be repaired) with standard Unix tools such as less(1) and vi(1). The best storage format for this purpose is almost undoubtedly XML.

Then there's everything we'd like to be able to do, except GNATS won't let us:

• Proper escalation levels: the submitter should not be allowed to set the severity of a PR. Instead, the screener who first handles the PR will set an initial severity, and the assignee (and / or the Bugmeister) will be allowed to adjust that severity.
• Access levels: the PR system must have a concept of users, and of who is allowed to do what with PRs. For instance, only "screeners" should be allowed to modify unassigned PRs, and then only to close them or assign them to someone. Once a PR is assigned to someone, only that person (or an administrator) should be allowed to close or unassign it.
• Timed events: the system should have provisions for timed events, such as closing a PR in "feedback" state if no-one replies within a set time, or prompting the person responsible for an open PR if it sees no activity in a certain time, or even unassigning the PR if the assignee doesn't react to such prompting.
• Simpler maintenance: no more edit-pr(1). Instead, a series of simple command-line utilities (or even a single utility with an easily memorizable option syntax) would be available to safely change a PR's various editable properties (state, severity etc.)

Finally, there are a few features that Would Be Nice, but we can live without:

• Web interface: a fully-featured web interface so we don't need to give PR workers a freefall account.
• Originator empowerment: the originator should be allowed to perform certain maintenance tasks on PRs, such as change her own contact address, or mark the PR as "resolved to originator's satisfaction".
• Extensibility: the ability to easily extend the system by adding new object classes, new properties to existing object classes, and new policies and transformations to manipulate those.

Of course, different people will have different opinions of which features are "must have" and which would just be "nice to have", but you can never please everybody. If I get a majority of users and committers to agree that the new system is better than the old, it'll be a huge step in the right direction.

On a lighter note, I just thought of a great name for a bug tracking system: Bugger. The command-line interface would of course be bugger(1), and the command to list all open PRs would be bugger --all. Whaddyall think? :-)

# Each final entry must say "required" -- otherwise, things don't
# work quite right.  If you delete a final entry, be sure to change
# "sufficient" to "required" in the entry before it.
    

                actions[PAM_SUCCESS] = _PAM_ACTION_DONE;
                actions[PAM_NEW_AUTHTOK_REQD] = _PAM_ACTION_DONE;
                _pam_set_default_control(actions, _PAM_ACTION_IGNORE);
    

                actions[PAM_SUCCESS] = _PAM_ACTION_OK;
                actions[PAM_NEW_AUTHTOK_REQD] = _PAM_ACTION_OK;
                actions[PAM_IGNORE] = _PAM_ACTION_IGNORE;
                _pam_set_default_control(actions, _PAM_ACTION_BAD);
    

su      auth    sufficient      pam_rootok.so   no_warn
su      auth    requisite       pam_wheel.so    no_warn auth_as_self noroot_ok
su      auth    sufficient      pam_unix.so     no_warn try_first_pass nullok
    

So how do we fix this? Well, having consulted the original PAM paper (available from Sun's PAM page), I'm not even sure it's a bug. But it's certainly non-intuitive, and can have catastrophic consequences if the admin is not aware of the exact semantics of "sufficient". I see four ways to avoid this pitfall:

1. Don't change the code, but document the pitfall, and recommend that module chains be terminated with a "requisite" pam_deny.
2. Hack the dispatcher to recognize this situation and return PAM_AUTH_ERR.
3. Change pam_wheel to return PAM_IGNORE instead of PAM_SUCCESS.
4. Add a new control flag (called "necessary", maybe?) that means "terminate on failure, ignore success", to mirror "sufficient" (which has the semantics of "ignore failure, terminate on success").

The root of the problem is that the PAM control flags are poorly named and too large-grained to provide sufficient control of the flow of execution. Like everything in life, it's a trade-off: simplicity versus flexibility. In this case, simplicity has, in my opinion, been carried too far.

Unfortunately, it is very difficult to implement an event model on top of the existing callback model. Eivind Eklund (eivind@freebsd.org) has had some success with sshd(1), using a wrapper library that forks off a child process that handles the callback stuff and communicates with the parent process using IPC, but this is a horrible hack.

On the other hand, had the PAM library been based on the event model in the first place, it would have been trivial to implement the callback model on top of it (for simplicity and compatibility with legacy applications), but no such luck.

I could, of course, rewrite PAM from scratch using an event model, but that would be a lot of work and totally outside the scope of my contract, so unless I can get additional funding (whether from DARPA or from another source) or convince the University of Oslo to let me do it for my master's degree, it won't happen in the foreseeable future. Would be nice, though, especially when you consider what a mess our current PAM implementation (which was derived from Linux-PAM) is.

I write the diary in XML using a schema I created specially for this purpose, basically by just starting to type and marking up the text in what seemed a reasonable way. Then I transform the XML source into XHTML using Sablotron and a custom XSLT stylesheet. The resulting HTML code is totally devoid of layout information — it simply uses DIV and SPAN elements with different CLASS attributes to group parts of the text together and indicate their semantic role, and relies on a CSS1 stylesheet to tell the browser how the various elements and classes are to be formatted.

The net result is that the diary not only looks great in a modern browser, but it should also look OK in any browser without CSS support (including text-only browsers), and present no significant trouble for screen readers or HTML-to-speech converters. Unfortunately, there are browsers out there — like Netscape 4 — that claim to support CSS, but make a total hash out of it. Amongst other bugs, Netscape 4 gets confused by DIV elements and ends up rendering successive DIVs as if they were nested, so if you read this diary using Netscape 4, you're better off with the "Enable Style Sheets" option in the "Advanced" preferences dialog turned off. Sucks to be you. Get a real browser.

I went to BSDCon Europe this weekend, which was a ball — it was great to finally meet all those people I've known for so long but, with a few exceptions, never actually met — Paul, Nik, Brian and Josef, of course, and Poul-Henning, Greg and Mark, and many others. Too bad Robert couldn't make it — I was really looking forward to meeting him too! The conference was a smash hit, three days of non-stop fun and fascinating presentations and most of all lots of socializing. Oh, and it finished with a flourish — I stood up in the middle of a near-plenary session and volunteered to take on responsibility for the FreeBSD bug tracking system. I'm still wondering if that was really such a good idea :-)

I arrived late on Thursday night with a growing headache and a horrible pain down the left side of my neck and down to my left shoulder blade. Joe met me in the lobby and brought me to the terminal room, where Paul, Nik, Brian and he were busy setting up the network. Having played around with MacOS X a little, I offered to assist with the network, and they set me to crimping network cables, as I seemed to be the only one around who had done it before — that, or the others had more important things to do.

Two Ibuprofen, a few hundred feet of Cat 5, ten RJ45 connectors and half a roll of Gaffer tape later, the terminal room was fully wired up and connected to the Internet through twelve bundled ISDN channels, so I hooked up my laptop and sat down to read my email. The first item in my FreeBSD mailbox was the contract from NAI Labs! Finally, more than four months late, all the paperwork had been sorted out and we could get to work.

A couple of months ago, I agreed to hold a FreeBSD-related presentation at the December meeting of NUUG, the Norwegian Unix User Group, and selected "FreeBSD as a security platform" as the topic of my presentation. Then, when it became clear that I would be able to afford going to BSDCon Europe (mostly thanks to Josef Karthauser, who generously offered me to crash at his place!), I used that as the topic of my BSDCon Europe talk (and ended up rewriting it from scratch on the morning of the day I was to give it). Since NUUG hadn't yet announced my presentation on their web site, I decided to change the topic and mailed them a new abstract for a presentation entitled "PAM in FreeBSD". I expect that I'll have more interesting things to say about that than I had about "FreeBSD as a security platform".

I'm also considering submitting a paper to BSDCon which is coming up next February, but the deadline for submitting the complete paper is December 4, which might be tight, and I'm not even sure they're still accepting new papers. I won't be able to afford going to BSDCon unless I'm presenting (and possibly not even then), but there are other interesting events coming up in the same timeframe, such as NordU, and later on there's SANE, USENIX and the Security Symposium. The latter is the only one where the deadline for initial submissions hasn't yet expired.

I'll be documenting my work in this diary, both so the community can track my progress and as documentation for the invoices I send to NAI. If it works out well, I'll probably start writing diary entries about FreeBSD work I do as well, and maybe even other Open Source work (I'm mightily tempted to volunteer for Conglomerate... especially as I use XML a lot these days, for papers, slides, reports and even this diary!) I don't know how many people will actually be interested in this diary, but if I get much of a readership, I'm also quite willing to take requests from my readers for topics they'd like to see me discuss.