Disclaimer: this started out as a private rant. Then I started talking with people about it. It was written in 3 seperate sittings with some discontinuities in it. It was not written for publication, so please do not. It has had no wordsmithing etc. -- Just what is this FreeBSD thing? FreeBSD is an outgrowth of a spare-time project started by people who wanted to have some fun and volunteered their time to maintain a patched version of 386BSD. Over the years it has become many things, but the core is still that volunteer, supposedly fun project. People were attracted to FreeBSD in order to get enjoyment from doing something right. Many of the first developers had day jobs where their creativity was directed according to feature lists, marketing goals or customer demands. FreeBSD was attractive because it had to answer to none of those, and was a good place to do something because it was the right thing to do. As time passed more people became involved. Commercial interests and sponsored development became increasingly common. Some developers worked on their employer's time in order to add features that their employer needed. For some people, it wasn't about 'fun', 'volunteer' or 'right' any more. Most of the time however, developers were pulling in the same direction so it mostly worked out ok. After a while, we found ourselves with our first fundamental conflicts. The original core developers found themselves in a defacto management role, even though few were suited to the task. With an increasingly large group of people, inter-personal friction, conflicting goals, the feeling of needing to compete with other projects lead to the realization that the overall project's goals were no longer that simple. The folks who were doing things because it was fun or right found themselves pressured into doing something because Project X has it. Some felt the need to court users as a validation of their efforts. The competitive drive became rather pervasive and we found ourselves in situations like 'Project Y has this nasty feature, but some users like it, so if we dont find somebody to hack it in then we might lose users!' or 'Commercial operation Z needs feature X, if we dont provide it, they might switch hundreds or thousands of machines to another operating system'. We ended up with release schedule timetables, code freezes, feature todo checklists, a massive bug database, a fiercly competitively focussed environment, political management committees of doom, and somewhere in all of this it seems the words 'volunteer', 'fun' and 'right' seem to have fallen out of focus. The lure of validation of our efforts (releases etc) seems to be able to cooerce people to work on feature todo lists and work around feature freezes etc. What have we become? Have we become the embodiment of everthing that we didn't like that caused FreeBSD to be started in the first place? Is the 'product-at-any-cost' focus the thing that is burning out our old school developers? What can we do about this? Should we? Do we admit that FreeBSD has become something completely different and stop pretending that it is a fun volunteer project? We need to keep in mind that a great number of people work on FreeBSD because it is product oriented and useful to their employers. We can't just say "oh, lets get back to our roots and screw the product goals". Or can we? Or is there a better middle ground that we can occupy with clearly defined boundaries so that we know who stands where? Lets explore the latter one for a moment. Suppose we recognize the conflict of fun vs product and create a mechanism to deal with the different goals. Such a mechanism should take care to ensure that we still have a viable project as a whole, but keep the playgrounds or sandboxes seperate enough to stop the butting of heads. Part of the problem is the current vs stable split. Current exists because of gross inadequacies in our source control system. If we retarget development into an ongoing 'development' vs 'release' track system then we have some really interesting possibilities. Suppose we keep the development track stable and functional at any given day (give or take a few days to allow tidying up loose ends). In that case, it should be possible to fork off or import a 'for release' update at pretty much any time, give or take a few days. For this to work, it means features *have* to be developed off the mainline and only committed when they are ready for prime time, give or take the few days to patch up loose ends. / release / / --------------+-------------------------------------+--> release track ^ ^ / / / / -----------+-------------------------------------+----------> development track \ ^ \ \ ^ \ | \ \| features, merged \| + back when ready The idea being that the 'stable' concept is replaced by the release track, which is a time delay of the development track. *All* destabilizing development is done offline and only goes into the development track when its tested, complete and ready. Minor low risk development happens directly without needing to using side branches. Big things (KSE, thr, ULE, SMPng, acpipci) are to be done in seperate sandboxes and merged back in only when they're ready. Some of this is already done in the perforce side tree, but we can make other facilities available for those who are alergic to perforce. If a feature doesn't work out, or is not of general interest, it stays in its side branch rather than being a lame duck in the current tree that needs to be backed out or worked around. I would imagine that the release track would pick a time when the development track is in reasonably good shape and do a complete import. Think of the X11 spinoffs that took periodic code dumps and sync'ed up. Some of the most important effects that this would cause are: * End the extremely long lag time on big new features. ie: no more need to wait 3+ years for something in -current to get released. * Ensure that we have a reasonably robust baseline at all times. * End the MFC burden except as patchups for the release branch. MFC's and bugfixes in the release branch would be done by the release oriented camp. * Give a reasonable seperation of the sandboxes between those who are looking for a robust production oriented or sellable system and those who are doing development. * Because we have a continuous uninterrupted development stream, decisions about what to call a minor or major version bump are made at the time of branching. We look at the actual new features and decide if it warrants a minor or major bump. * Because all the major features exist as side branches, we have a convenient checkpoint of deltas to enable a backout of a feature in the release track if it really isn't ready for prime time afterall. * The goals and motivations of the development track and the release track branches and people can be different. The development track folks can get back to 'doing stuff because its right or fun' without having to worry about 12 month long feature freezes/slushes. The release track folks can apply more of a critical reality check to their timing and feature list. * Give more places to insert finer grained policy decisions. An arch type body would be ideal to vette/oversee the development track. A releng type body would oversee the release track. Obviously the groups need to work together to a degree to make sure the directions are compatable. core doesn't have a development function in this model and would be more of a representitive point of contact and take care of the non-development processes (can you say: management?) The major downside of all this is that it completely turns our moderately well understood model upside down. But maybe its time for more drastic measures? We need to stop having big suprises landing in the tree and causing turmoil. We need better communication to make sure that we're all pulling in the same (or compatable) direction and what kind of things to expect coming down the pipeline. Duplication of work just sucks. -- How can we make something like this work? Some specifics would be in order. First of all, to tackle the bikeshed and suprise problem, we need to encourage people to be more forthright about what they're working on and have a way of ending debates that are going nowhere and just causing wheelspin and wasted energy. One suggestion is to have developers who are working on a Big Project(TM) write up a reasonable description describing what they want to do and how they think they can do it and get it into the tree. It would describe milestones or checkin checkpoints if the project is big enough to warrant multiple code drops. This covers things non-trivial things including new features and big cleanups. The arch entity receives these and can approve, reject or request changes to the plan. For example, it may feel that the project is simply too big for one person to tackle or that it affects other people sufficiently that they need some input as well. It may decide that there has not been enough public exposure yet (arch@ mailing list if it was appropriate). Care needs to be taken to ensure that hidden agendas are not being slipped in so that one developer can derail another developer's work. If the change has implications that affect things like release capability or ABI changes, then other key groups such as re@ need to be given a chance to comment before a final green light is given. For longer term projects, the lead developer needs to keep everybody informed about what the current status is. There is no need to be exhaustively detailed, just a brief statement that gives a fair idea of where the project stands. The arch entity would also check on missing status updates. If a developer finds they need to change their plans, then that's negotiable with the arch entity. In return for the developer doing this legwork, we get several big benefits. * Visibility: everybody knows what is in progress and there should be no more suprises landing in the tree that destroy your works-in-progress. * Communication: because of the minimum documentation requirement, there is a good base of information for the monthly status reports to mine. * If the developer sticks to what they said they'd do, it essentially amounts to a 'get out of bikeshed for free' card. The last point is the key. Once it has been blessed, the developer can go off and just do it and know that it's been pre-approved. Yes, "pre-approved" as in the blasted credit card offers in the USA. The work can be done knowing that its expected to be acceptable for going into the tree. Depending on the circumstances and the project in question, the arch entity may say 'commit when you're finished' or require a final sanity check. At face value, this might sound like a lot of work. However, we do not have such large projects every day that would warrant all this work. The scope and impact of the project would determine what kind of measures the arch entity would feel would be necessary to ensure the smooth running of the developer track tree. Adding an option to ls is hardly a major project and the amount of legwork would consist of little more than 'can I add --foo to ls(1)' and a reply of 'sure, why not?'. Ok, that was a bad example. :-) On the other hand, a major project like KSE or a device configuration subsystem rewrite or interrupt API changes would require more measures. For example, the arch entity would likely require a KSE-sized project be broken up into logical checkpointable steps and a fairly coherent plan of attack. Not an white-paper style "here's what it should look like" but 'how to get from here to there' roadmap that details what gets done when. A KSE-sized project will naturally require refinement to the plan as it goes, but changes should be done in coordination with the arch entity. In return for this legwork, the KSE-sized project dont have to be questioned every few minutes. People will know where its up to and what comes next, and when the code drops happen. And no bikeshedding about whether its the right thing to do or not. Any arguments about direction are with the arch entity. So what is this ``arch entity'' and how would it work? The TRB idea was doomed from the start. More committees isn't the solution to scalability problems. Instead of a core-sized arch committee, we need an ultra-responsive 'ask a question, and you will get a prompt answer' mechanism. One suggestion is to have three people be the arch entity, but have one person be the spokesperson for the group at any given time and makes the easy decisions on the spot. There is a fallback 'chain of command' so to speak so if the primary person fails to answer in a reasonable amount of time, the second or even third person can give the definitive answer. Because the group is a party to all requests, they should all know whats going on and have a compatable set of expectations for FreeBSD's future development. The spokesperson responsibility can be rotated or whatever. The important thing isn't so much how somebody gets to be on it, but instead that questions get prompt decisive answers so developers are not left in limbo for days at a time. But what about core? Where does core fit in to this grand scheme? If an arch entity takes over this role from core+TRB, then perhaps we need other core reforms too? Remember the complaints about developers being expected to be effective managers? More committees is probably not the answer to the scalability problem. A solution to the 'core problem' is something like this: 1) Tear up the rulebook and status quo. 2) Form a couple of focused [sub?]groups, along the lines of the arch entity. a) Group to actively recruit and prescreen committers. This group makes their recommendation 'this person should be a committer' and if there are no meaningful objections or requests to hold within 24 hours, its a done deal. b) Conflict resolution group. Again, to quickly respond to problems before they snowball. Each group answers to core but would be expected to otherwise function as a unit. 3) The core folks take on the project management role as needed. This means mundane things like filtering incoming core@ email and redirecting as necessary. It means being the final 'buck stops here' if one of the focus groups goes off the rails. It means answering questions like "Can I use FreeBSD for my university's coursework and can I reprint the handbook for the students to use?". 4) With arch, committer and conflict responsibilities delegated out, the remaining function of core really isn't all that much. The email load should go way down and it should only have to deal with serious matters, hopefully on an irregular basis. 5) re@ becomes the 'arch entity' of the 'release track'. The arch entity will be in regular contact with re and vice versa about what re@ needs to be coming down the pipeline. re@ decides when the development branch is in good enough shape to sync up with. 5) Perhaps pull some of the existing groups back together a bit. Working independently is fine, but its essential that everybody is on the same page. For example, doceng, re, portmgr. Not that there is a problem with these groups, but new-core would need to make sure that the groups groups are all in sync. How do we deal with conflicts? There are various ways to do this, but we have to keep the reality of a volunteer organization in mind. Responsiveness is also a critical thing here. One possible tactic is that if two developers fight, put both of them on ice until they've sorted it out amongst themselves. If they can't find a way to settle the problem or disagree peacefully, then resort to other measures. Another thing that has been moderately successful is to pick a (patient) person to act as a mediator or mentor for developers who are regularly in or near the trouble areas. Above all, the best way to deal with conflicts is to try and avoid them happening in the first place. Improving the size and quality of the sandboxes in the playground would go a long way to help here. If we can avoid the situations where substandard or contentious things go in the tree in the first place, a good number of irritants for certain people become less of an issue. Above all, people need to be kept informed about what is going on. This means status reports. In return for somebody being given a 'hat', they will be expected to keep everybody up to date about what is going on. This means core, arch, and the various other groups. Especially so for the arch entity. People need to be able to see at a moment's notice what is in the pipeline and how it is coming along. Perhaps the perfect monthly core report would be "Everything's running smoothly, core didn't have to do exceptional". If we split core's functionality so that committer pre-screening and dispute resolution gets relocated, then there may not even be a need for a closed core list. I'd be keen to have things changed so that core@ was a readable list and core-private@ was only used for super-sensitive stuff (with a note going to core@ that the core-private@ list was used) -- Core elections Assuming core becomes a management role, do we still need elections? It is probably premature to do away with them at this point, but we do need to make a few improvements to the process. Right now, people who are thinking about leaving core tend not to because of reluctance to bringing an election forwards or leaving core shorthanded. Regardless of whether the proposals above are acted apon, we should do something about this situation. We need to refine the mechanism to make it as easy as possible to leave. The best way to do this is to allow core to nominate a replacement to fill a seat left by somebody mid-term. To avoid the scenario of a core sub-cabal being able to completely stack the rest of core, elections should be more often and at fixed times. In other words, 1 year fixed term with seats being self selected over the year. Having a fixed election date (october was suggested) that is not in the US or european vacation season means that we can have far shorter election periods, possibly as little as 1 weeks to get it over and done with. (ie: suppose nominations automatically close at october 1st, then there is no need for a specific nomination period) Afterthoughs A couple of people have said to me "Hang on a second, this sounds a lot like a corporate development model. Didn't you start out with 'volunteer' and 'fun' at the beginning of this?". Yes, it does have similarities, but I ask you this.. is the friction and head-butting in the current pseudo free-for-all model causing the right kind of 'fun' to keep people from burning out? The current model isn't scaling well. If nothing changes, then the burnout is likely to continue. Once FreeBSD loses its key developers who want to do things because it is good or right, then we lose what makes FreeBSD different. It becomes reduced to playing catch-up with other operating systems. Creating sufficient infrastructure to make nice large sandboxes in the playground and keeping a roadmap within easy reach can't hurt. It will certainly reduce friction when developer X runs into a suprise landmine in the tree left by developer Y. For that to work it depends on an arch entity being on the ball.