****************************************************************************************** NOTE: Start here first: http://wiki.freebsd.org/SubversionPrimer I'll be attempting to keep the wiki page up to date. ****************************************************************************************** ****************************************************************************************** Last update 3rd June 2008 (x5) ****************************************************************************************** ** WARNING ** This is in unformatted "brain dump" mode. There is no structure, no internal consistency, and probably no hope for anybody trying to make sense of it. I will be adding more material as we go, so don't doc-ify it just yet please. ** WARNING ** http://svnbook.red-bean.com/ - check the nightly build for 1.5 docs. This is a very good primer and easy to read. Keep a copy handy. Getting started. ------------------------ Install devel/subversion-freebsd I recommend the following options: - enable NEON, not SERF. (optional, allows access to http://) - python bindings (optional, allows python tools to talk to svn) - perl bindings (optional, allows perl to talk to svn) The python/perl bindings are important if you intend to use svk or trac or anything that integrates with subversion. If you do not enable the bindings here, then those ports will try and use svn-1.4 from ports and there will be conflicts. This port gives you a slightly hacked subversion-1.5-rc7 installation. The changes are small, but potentially significant. * $FreeBSD$ is expanded in our client. Unpatched clients will not do this. * merge conflicts are presented closer to p4-style than cvs style. * a commit template is generated (and removed in the client if empty) YOU MAY USE UNPATCHED or third party clients to check out from freebsd repo. They will not have keywords expanded. You can commit like this so long as the keywords are in canonical unexpanded form. You will have to supply your own log message tags if needed. You may *NOT* check out a tree with a keyword-aware client (which will expand $FreeBSD$) and THEN commit with an unpatched client. This would cause the stale expanded keywords to be uploaded to the server and a whole world of hurt will follow. There is a pre-commit check to enforce this. In other words, YOU MUST NOT MIX AND MATCH PATCHED/UNPATCHED CLIENTS. You may use svn-1.4, git-svn, hgsvn, svk, TortiseSVN, etc. You can telnet to the http server and speak WebDAv/delta-V for all I care. It should be ok to commit from these, with a caveat that none of these recognize the svn-1.5 mergeinfo records. If merges in a part of the tree are being managed by svn-1.5, then only 1.5 should commit merges. If you check out with something that does not have the keyword patches, or something that uses the svn API bindings (perl, python), then the keywords will not be expanded. This is expected. Somewhat inconvenient, but should be harmless. Be aware and use caution. Merges. ------------------------ svn-1.5 remembers merge history. The metadata for this is stored as properties called 'mergeinfo'. pre-mergeinfo clients will not maintain this. However, our tree came from cvs which has no idea about merges, so the history is blank. The 7.x and earlier branches don't have merge history, so we'll have to do it the old way. svn does help with cherry-picking though, see the 'svn merge -c ' commands to merge a particular commit across. 8.x will likely be managed by svn's mergeinfo and would have a pre-commit check to ensure that commits to 8.x are done with a mergeinfo-aware client. Layout. ------------------------ /head - think of /trunk if you're used to svn. Or 'cvs update -A' or HEAD in cvs. This is -current. src/bin/ls/ls.c is now head/bin/ls/ls.c If enough people yell at me before it is too late, I will change /head to /trunk to be more svn-ish. Check out via svn co $ROOT/head (or $ROOT/head/bin/ls, as desired). The directory ends up being placed in the last path component. svn co $ROOT/head -> ./head/{bin,sbin,...} svn co $ROOT/head src -> ./src/{bin,sbin,...}. You probably want this. RELENG_[4567] is /stable/[4567]/{bin,sbin,...} RELENG_[4567]_[0-9] is /releng/[4567]_.[0-9]/{bin,sbin,...} RELENG_[4567]_[0-9]_[0-9]_RELEASE is /release/[4567]_[0-9]_[0-9]/{bin,sbin,...} eg: RELENG_7 -> /stable/7/* RELENG_7_0 -> /releng/7.0/* (release eng, errata, security, etc) RELENG_7_0_0_RELEASE -> /release/7.0.0/* (static release tag) Branch and code flow structure head -> stable/* -> releng/*.* -> release/*.*.* Yes, there will be a stable/8 before it announces itself as -STABLE. :( I chose these names rather than a more traditional svn-style layout because our $FreeBSD$ expansion includes the path name. I tried to keep the paths as short as practical. repository access. ------------------------ http://svn.freebsd.org/viewvc/base/ - like cvsweb, but smarter in some ways, dumber in others. $ROOT above is one of: http://svn.freebsd.org/base/ - WebDAV root. Probably not going to be public, depending on load. You can anonymously checkout from here. This would be like allowing people to cvs check out from repoman. You have to have NEON or SERF enabled in your svn client to access this. svn://svn.freebsd.org:53690/base/ - like webdav, but faster. uses streaming protocol. Temporary port. svn://svn.freebsd.org/base/ - permanent address of above once firewall is updated. svn+ssh://svn.freebsd.org/base/ - authenticated committer access over ssh. yes, I know its a PITA to type. Thats what aliases or environment variables are for. :) committing. ------------------------ (assuming $ROOT is set as above) svn co $ROOT/head/usr.bin/calendar cd calendar Update Attilla's birthday to compensate for date shift in 916AD svn commit cvs exporter: ------------------------ A script runs that exports changes to cvs. If you make a commit to /head, it will appear in cvs within a minute on the mainline (-A, HEAD). Likewise for /stable/[4567]. The commit will be exported to RELENG_[4567]. Likewise for /releng/*. /release/* is NOT exported. This requires cvs operations by re@ behind the scenes to compensate. Branches older than 4.x will be (at least temporarily) locked to avoid commits that are not exported. Cross-branch commits should be correctly handled. ie: a security fix to 4 locations in /releng, and /stable/* and /head, will be handled. The exporter is designed to fail "safe". If it finds something it doesn't know how to handle, it will abort on the spot rather than risk damaging the cvs tree. The exporter does NOT handle vendor branches. cvs exporter vs $FreeBSD$. ------------------------ I have not decided how to handle this yet. There are pros and cons on each option available. Choice #1: Let cvs and svn expand it differently. This causes interesting problems for trying to track a bug report that says "the file version is 1.212". we then have to go to cvs, to find the log message for rev 1.212 to find the svn change number. Choice #2: Let the exporter change the expansion of $FreeBSD$ as it exports. I have code that has it replace it with $FreeBSD SVNid: blah blah rev 123456 $ before committing to cvs. The problem is that is technically an illegal keyword and ident(1) will not find it. We cannot use $FreeBSD$ for this because cvs and cvsup need to expand the old keywords, and expansion is a global setting. Choice #3: Use something other than $FreeBSD$ for svn, and export that to cvs as-is. I kinda want to use $BSD$, and to hell with the arrogance that implies. :) For the initial transition, we're going with choice #1. We can change at any time, when the right choice is apparent. vendor branches: ------------------------ There is one hell of a mess in /vendor*/* right now. NO VENDOR IMPORTS YET PLEASE! Having said that, it is organized by vendor or import group. Take tcsh as an example. While there isn't a vendor called "tcsh", we tend to import tcsh as a unit, so it gets its own directory - /vendor/tcsh/*. Below that, there are directories. ./dist - is like the 1.1.1 branch in cvs. ./6.02 - is a "tag", like "tcsh_6_02:1.1.1.6" in cvs. Imports are done by replacing the contents of "dist" with the new vendor files. (extract tarball, svn add, delete etc to make it right). After committing, it is 'tagged' by "svn cp $ROOT/vendor/tcsh/dist $ROOT/vendor/tcsh/6.03". This is done server-side to maintain mergeinfo, and to avoid having to check it out. ***SEE ME*** before even thinking of attempting to do an import. I want to walk through with you for the first few times and make sure it is documented correctly etc. Once the 'dist' stuff is imported and "tagged" with the 'svn cp', then it is merged into /head. "cd head/contrib; svn merge $ROOT/vendor/tcsh/dist tcsh". (not exact command) Resolve conflicts, commit. (svn remembers metadata about merge once it has been primed to be told about previous imports done in cvs) There is no concept of a cvs-style "-b branch" in svn. It is done with a flat namespace. /stable/7 is a branch of /head. If you've used p4, you're already used to this. Moving directories. ------------------------ svn knows about directories and heirarchies, unlike cvs. To move subtrees around, you just move the directories. All the files underneath follow. Directory moves are very low cost. Cool stuff. ------------------------ svn log -g svn diff -c somechange $ROOT (beware) svn log -c somechange $ROOT (beware) (svn isn't as clean as p4's describe commands. The -c switches are relative to the server root or your checked out copy. If you ask for a diff -c that is in /head while you're in /stable/7, it will helpfully give you absolutely nothing.) (serf has a crash bug with operations on http forms of $ROOT due to mod_dontdothat.so) See note below about Peg revisions. log -c and diff -c may surprise you otherwise. feature branch areas: ------------------------ $ROOT/projects - this is a first class analog of what we do in perforce. All commits are public, and published. If you merge changes into head, then 'svn log' will follow the merges back and track merge history in your "project" area. This means, write first class log messages for everything here. Layout: /base/projects//stuff $ROOT/user - like projects, but slightly less public. Commit emails happen but will not be mailed to the whole world. There will be a user commit mail list that can be subscribed to. Keep in mind that if you're intending to do a svn merge to bring your changes back into the mainline, the 'svn log' command will find your commit messages. Intended difference between /user and /projects: projects is for things that you intend to be merged back into the base tree. user is for experiments, exploratory changes, etc, that probably won't be merged as-is. You can still merge from user/* though. ** See notes below about policy on feature branches. Danger Will Robinson! DANGER!! ------------------------ Don't check out all of $ROOT. You will find out why this is bad if you try it. (Expect to need more than a few TB of disk space, and your network cable will cool down and stop glowing eventually. You'll be checking out several hundred variations of freebsd and individual copies of every import over the last 15 years). There is an apache block module (mod_dontdothat.so) to try and avoid this in the http / DAV server, but it causes problems with serf. svnserve won't protect you. Problems with the conversion. ------------------------ * Plenty. :) * /cvs2svn - stuff I couldn't figure out where to place in advance. My cvs2svn configuration is roughly 12,000 lines long, created by hand. Keeping track of everything, including what happened 15 years ago was just too much. The goal is to either move stuff out of /cvs2svn into the right place, or blow it away as it isn't worth keeping. In either case, it will be in history. * vendor tree is a mess. It has too many directories. I'll be fixing this online after the switch. it isn't exposed to cvs exporter, so it should be harmless. Multiple repositories. ------------------------ svn.freebsd.org/base/* ^^^^ - this is a repository. Everything below here shares the change number sequence and can be merge freely. /base/head, /base/user, /base/projects, /base/vendor etc. This corresponds to what is in ncvs right now, with feature branches like we do on perforce. This is the 'base system' repository. I'm not too fond of the name, but it can't be "src" because that creates even more confusing path potential. Given bikeshed potential, I'm choosing the colour. If there's any arguments, I'll change it to "red". svn.freebsd.org/ports/* - set aside for ports, should they use it too. Separate ACL's, commit numbers, etc. svn.freebsd.org/doc/* - set aside for doc people. Just like ports. I'm not forcing a choice here. svn.freebsd.org/projects/ - not to be confused with svn.freebsd.org/base/projects/. It will have to have a different name to avoid confusion. It has no ACL. It is equivalent to what is currently in /home/projcvs. It could probably be /proj/ or /pg (playground), or other variants like sandbox, scratch, work, etc. Commitment to svn, and the 'oops!' factor. ------------------------ As long as the exporter runs, and we dont do anything incompatible with cvs, then we have the "oops!" option available. If svn crashes and burns, we can turn it off and turn commits back on with cvs and pretend it never happened. All our commits will be carefully replicated to cvs as we go. However.. once we start taking real advantage of svn, the exporter will start to destroy some constructs in cvs that we're used to. In particular, it cannot maintain vendor branches. Once we start doing vendor imports in svn, we're increasingly committed as we would be destroying the vendor branch setup in cvs. (imports would be committed as deltas on the mainline, with no corresponding 'cvs import' for a baseline) Therefore, no imports for a week or two, until its clear that this will work out. ie: an ACL will prevent all commits to /vendor* Multiple vendor trees. ------------------------ There is /vendor, /vendor-crypto, /vendor-cddl and /vendor-sys. This is because of some overlap that I can't resolve at cvs2svn time. I expect to move most of /vendor-sys into /vendor. We'll probably develop /vendor-gpl and /vendor-gpl3 at some point too. However!! (and this is important) - we do not ship these trees to end users via the release process. Partitioning in the /vendor* area has absolutely no effect on /usr/src. It might help us keep our code partitioned as developers though. Release process. ------------------------ 6.x and 7.x will be released from the exported CVS trees. 8.x may come from svn if we're ready to do that. Scope and visibility: ------------------------ I intend that svn is an internal thing for the committers. It isn't our public interface to the world. cvs/cvsup/cvsweb will be our public front for at least the short term. I do not want to tell people to check out from svn yet as we don't have mirrors set up. svn repository mirrors: ------------------------ Soon, but not yet. Svn's protocol is streaming, but is still sensitive to round-trip times. Particularly with http and NEON. (Serf is faster!). Therefore, we will need to have a mirror network. The svn:// protocol is less sensitive to this, but it is still a factor. The actual mechanism of mirroring is yet to be determined. We have two choices. We can either let people run uncoordinated svnsync mirrors, or we can rsync the repository. There are pros and cons both ways. rsync has the advantage that we can do retroactive edits and obliterates and have all the official mirrors automagically propagate the edits. svnsync will not be aware of these and will not propagate "fixes". My inclination at this point is to use rsync for official mirrors, regional mirrors and committers, while have have uncoordinated private mirrors use svnsync. svn allows you to check out from a local mirror, and switch to the freebsd.org one for commit, with relative ease. The repositories have a uuid, so svn can tell if you're trying to do something bad. eg, a rough example: svn co file://local/mirror/head/bin/ls cd ls hack hack svn switch --relocate svn+ssh://svn.freebsd.org/base/head/bin/ls svn commit I can imagine that this would get unwieldy though. Whining: ------------------------ Not allowed. I'm sorry, but I've spent 10 to 15 hours a day on this for the last few weeks chasing a moving target. I've been working on it for months. I lost my magic wand, so I can't make it perfect. However, I think it is "good enough" and we're all mature and grown up enough to get over the imperfections and fix the problems as we go. If you have no idea how and why it took that much time, consider yourselves lucky. Sleep peacefully tonight. svn is far from perfect, but offers enough of an improvement that it will be worth it over time. Bikeshed appearance ------------------------ I've made a bunch of calls to pick a colour. I know better than to ask "what name should the repository root have?" on a mailing list. The last 14 years have taught me that 1) this sort of thing really doesn't matter, 2) you can't please everybody, and 3) I've done the work, so I'm taking the liberties that go with it. ****************************************************************************************** Update (x3) 3rd June 2008 ****************************************************************************************** Peg revisions ------------------------ Assume: $ setenv BASE svn://svn.freebsd.org:53690/base I svn rm'ed $BASE/head/crypto/openssl/demos/engines/rsaref in rev 179526 This leads to some surprises, which make for a worked example. $ svn log $BASE/head/crypto/openssl/demos/engines/rsaref svn: File not found: revision 179529, path '/head/crypto/openssl/demos/engines/rsaref' Wait.. wtf? $ svn log -v $BASE/head/crypto/openssl/demos/engines =================== r179526 | peter | 2008-06-03 12:27:02 -0700 (Tue, 03 Jun 2008) | 2 lines Changed paths: D /head/crypto/openssl/demos/engines/rsaref cvs2svn did not delete this, even though it is empty. =================== There. it was removed in -r179526. One might expect the -r switch to work. $ svn log -r 179526 $BASE/head/crypto/openssl/demos/engines/rsaref svn: File not found: revision 179529, path '/head/crypto/openssl/demos/engines/rsaref' $ svn log -r 179525 $BASE/head/crypto/openssl/demos/engines/rsaref svn: File not found: revision 179529, path '/head/crypto/openssl/demos/engines/rsaref' Wait.. what?? Where did 179529 come from? Apparently, what I was asking for was 'tell me about 179525, relative to the HEAD revision of /head/crypto/openssl/demos/engines/rsaref'. And of course, it doesn't exist there. What we actually need is: $ svn log -v $BASE/head/crypto/openssl/demos/engines/rsaref@179525 (yes, perforce-ish syntax. This is called a PEG revision. It turns up in svn help a lot) And the output of that command is about what one would expect. svk ------------------------ SVK is a little confused about svn-1.5.0, it seems. It doesn't understand something about how 1.5 plus cvs2svn-2.1.0 construct branches. If you try and mirror a small part of the tree, you'll likely hit a problem with unknown revs or undefined variables. The solution at this point is to just mirror the entire tree. Ouch, sorry, but in the end you've got everything. To ease the pain, I'm creating a freshly initialized .svk "seed" tarball. If you haven't used svk before, and want to, you can grab the tarball to get started. I'll update this with the location. freefall: -rw-r--r-- 1 peter devel 1.0G Jun 4 00:44 /home/peter/dot_svk_r179527.tbz2 This is intended for extracting to ~/.svk, where you would then edit ~/.svk/config to correct the path. /scratch/tmp/peter/.svk/local needs to be corrected to your home dir.. expanded form of ~/.svk/local I've imported the freebsd tree to //freebsd/base. You update your mirror with svk sync //freebsd/base. It is set up to use svn+ssh for ease of pushing your local commits upstream. Do read the docs to be sure you know what you're getting yourself into. http://svk.bestpractical.com/view/HomePage The idea of svk is to give you full disconnected operation and local branches and development. Do make sure you build subversion-freebsd with the Perl bindings enabled before attempting to build or install svk. svnsync ------------------------ svnsync (like svk) take a long time to download everything. You can get a seed tarball to help get started. $ rsync rsync://repoman.freebsd.org:50873/svnseed -rw-rw-r-- 1073359330 2008/06/01 12:33:04 svnmirror-base-r179480.tbz2 You can grab that file, extract it to (say) /home/svnmirror/base, and then: $ svnsync sync file:///home/svnmirror/base running that from cron will keep it up to date. Also available: ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r179480.tbz2 freefall:/home/peter/svnmirror-base-r179480.tbz2 I don't think there is a compelling reason to maintain a local mirror for developers unless network latency demands it. From the perspective of svn.freebsd.org, I believe we can handle developer traffic. In any case, see the "svn switch --relocate" commands. This allows you to check out from a local mirror, then "svn switch --relocate svn+ssh://svn.freebsd.org/base" This is kinda slow though. Policy on /base/projects, /base/user vs /proj (or whatever) ------------------------ Everything in /base/* is globally mirrored. Every single consumer of the svn tree will get a copy of the changes. All the changes are in full public view, including the log messages. Therefore, these areas should not a 'free for all' or we'll end up with a huge mess (and drama) before we know it. Before deciding on creating a project here, you should consider the following questions: * Will placing the work here plausibly benefit FreeBSD? * Is the work likely to be merged into the main tree? (yes indicates /base/projects) * Is the work an experiment that could concievably be merged, but it isn't clear at this point? (A "yes", would tend to indicate /base/user) * Is the project related to FreeBSD, and of public interest? (a 'yes' here would tend to indicate a 'MAYBE put it here, but ASK FIRST'. Think of bsdtar as an example, if it were looking for a home - which it isn't.) * /proj is for stuff that should be public, but isn't part of (or directly related to) freebsd. It is a separate repo, with separate change numbers. It will not support direct merge tracking across to the base repo. Things like sccs might live here. Not limited to only committers. * /scratch is for pure experimental stuff. If you want to try out commands with svn that you're not sure about, here's the place to do it. Experiment with merges here. No commit mail. Not limited to only committers. There are extra points to be awarded for asking.