Index: article.xml =================================================================== --- article.xml (revision 40057) +++ article.xml (working copy) @@ -62,7 +62,7 @@ The codebase Most of the package building magic occurs under the - /var/portbuild directory. Unless + /var/portbuild/ directory. Unless otherwise specified, all paths will be relative to this location. ${arch} will be used to specify one of the package architectures @@ -167,7 +167,7 @@ interesting data (ports and src trees, bindist tarballs, scripts, etc.) to disconnected nodes during the node-setup phase. Then, the disconnected portbuild directory is - nullfs-mounted for chroot builds. + nullfs-mounted for jail builds. The ports-${arch} @@ -191,29 +191,31 @@ - Chroot Build Environment Setup + Jail Build Environment Setup Package builds are performed in a - chroot populated by the + jail populated by the portbuild script using the ${arch}/${branch}/builds/${buildid}/bindist.tar file. - The following command builds a world from the + The makeworld command builds a world from the ${arch}/${branch}/builds/${buildid}/src/ tree and installs it into - ${worlddir}. The tree will + ${arch}/${branch}/builds/${buildid}/bindist.tar. + The tree will be updated first unless -nocvs is - specified. + specified. It should be run as root: - /var/portbuild&prompt.root; scripts/makeworld ${arch} ${branch} ${buildid} [-nocvs] + &prompt.root; cd /var/portbuild +&prompt.root; scripts/makeworld ${arch} ${branch} ${buildid} [-nocvs] The bindist.tar tarball is created from the previously installed world by the mkbindist - script. It should be run as root with the following - command: + script. It should be also be run as root: - /var/portbuild&prompt.root; scripts/mkbindist ${arch} ${branch} ${buildid} + &prompt.root; cd /var/portbuild +&prompt.root; scripts/mkbindist ${arch} ${branch} ${buildid} The per-machine tarballs are located in ${arch}/clients. @@ -280,7 +282,7 @@ (For this case, the contents are identical for both server and client.) - RUBY_DEFAULT_VER= 1.9 + RUBY_DEFAULT_VER= 1.9 @@ -291,8 +293,7 @@ (For this case, the contents are also identical for both server and client.) - -.if !defined(CC) || ${CC} == "cc" + .if !defined(CC) || ${CC} == "cc" CC=clang .endif .if !defined(CXX) || ${CXX} == "c++" @@ -304,29 +305,29 @@ # Don't die on warnings NO_WERROR= WERROR= - + Sample <filename>make.conf.server</filename> for <application>pkgng</application> - WITH_PKGNG=yes -PKG_BIN=/usr/local/sbin/pkg + WITH_PKGNG=yes +PKG_BIN=/usr/local/sbin/pkg Sample <filename>make.conf.client</filename> for <application>pkgng</application> - WITH_PKGNG=yes + WITH_PKGNG=yes Sample <filename>src.conf.server</filename> to test new <application>sort</application> codebase - WITH_BSD_SORT=yes + WITH_BSD_SORT=yes @@ -416,7 +417,7 @@ dopackages.wrapper. These scripts take a number of arguments. For example: - dopackages.7 ${arch} ${buildid} [-options] + &prompt.root; dopackages.7 ${arch} ${buildid} [-options] @@ -425,7 +426,7 @@ The symlinks go away, and you just use dopackages.wrapper directly. For example: - dopackages.wrapper ${arch} ${branch} ${buildid} [-options] + &prompt.root; dopackages.wrapper ${arch} ${branch} ${buildid} [-options] @@ -628,7 +629,7 @@ -nocleanup, you need to clean up clients by running - build cleanup ${arch} ${branch} ${buildid} -full + &prompt.root; build cleanup ${arch} ${branch} ${buildid} -full errors/, logs/, packages/, and so @@ -667,7 +668,7 @@ ${arch}/${branch}/make.[0|1]: - don't know how to make dns-all(continuing) + don't know how to make dns-all(continuing) To correct this problem, simply comment out or remove the SUBDIR entries that point to empty @@ -685,22 +686,22 @@ Update the i386-7 tree and do a complete build - dopackages.7 i386 -nosrc -norestr -nofinish - dopackages.wrapper i386 7 -nosrc -norestr -nofinish + &prompt.root; dopackages.7 i386 -nosrc -norestr -nofinish +&prompt.root; dopackages.wrapper i386 7 -nosrc -norestr -nofinish Restart an interrupted amd64-8 build without updating - dopackages.8 amd64 -nosrc -noports -norestr -continue -noindex -noduds -nofinish - dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish + &prompt.root; dopackages.8 amd64 -nosrc -noports -norestr -continue -noindex -noduds -nofinish +&prompt.root; dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish Post-process a completed sparc64-7 tree - dopackages.7 sparc64 -finish - dopackages.wrapper sparc64 7 -finish + &prompt.root; dopackages.7 sparc64 -finish +&prompt.root; dopackages.wrapper sparc64 7 -finish Hint: it is usually best to run the dopackages @@ -767,7 +768,7 @@ package set. This can be accomplished with the following invocation: - path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz + &prompt.root; path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz @@ -911,11 +912,12 @@ Cleaning up a Build To free up resources, you will need to clean up client machines by - running build cleanup command. For example: - &prompt.user; /var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full + running build cleanup command. For example: + &prompt.root; /var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full + If you forget to do this, then the old build - chroots will not be cleaned up for 24 hours, and no + jails will not be cleaned up for 24 hours, and no new jobs will be dispatched in their place since pointyhat thinks the job slot is still occupied. @@ -930,21 +932,22 @@ You may have problem with the umount commands hanging. If so, you are going to have to use the allgohans script to run an &man.ssh.1; - command across all clients for that buildenv. For example: -ssh -l root gohan24 df + command across all clients for that buildenv. For example: - will get you a df, and + &prompt.user; ssh -l root gohan24 df -allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports" + will get you a df, and + + &prompt.user; allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports" allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src" - are supposed to get rid of the hanging mounts. You will have to - keep doing them since there can be multiple mounts. + are supposed to get rid of the hanging mounts. You will have + to keep doing them since there can be multiple mounts. Ignore the following: -umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory + umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: unknown file system umount: Cleanup of /x/tmp/8-exp/chroot/53837/compat/linux/proc failed! /x/tmp/8-exp/chroot/53837/compat/linux/proc: not a file system root directory @@ -1006,8 +1009,8 @@ You can use qclient command to monitor the status of build nodes, and to list the currently scheduled jobs: - python path/qmanager/qclient jobs - python path/qmanager/qclient status + &prompt.root; python path/qmanager/qclient jobs +&prompt.root; python path/qmanager/qclient status The scripts/stats ${branch} @@ -1564,9 +1567,13 @@ Pick a directory to hold ports configuration and - chroot subdirectories. It may be + chroot subdirectories. It may be best to put it this on its own partition. (Example: /usr2/.) + + The filename chroot is a + a historical remnant. + @@ -1610,14 +1617,15 @@ GENERIC (or, if you are using more than 3.5G on &i386;, PAE). - - Required options: -options NULLFS -options TMPFS - - - Suggested options: -options GEOM_CONCAT + + Required options: + + options NULLFS +options TMPFS + + Suggested options: + + options GEOM_CONCAT options GEOM_STRIPE options SHMMAXPGS=65536 options SEMMNI=40 @@ -1625,32 +1633,33 @@ options SEMUME=40 options SEMMNU=120 -options ALT_BREAK_TO_DEBUGGER - +options ALT_BREAK_TO_DEBUGGER For PAE, it is not currently possible - to load modules. Therefore, if you are running an architecture - that supports Linux emulation, you will need to add: -options COMPAT_LINUX -options LINPROCFS - + to load modules. Therefore, if you are running an + architecture that supports Linux emulation, you will need to + add: + options COMPAT_LINUX +options LINPROCFS + Also for PAE, as of 20110912 you need - the following. This needs to be investigated: -nooption NFSD # New Network Filesystem Server + the following. This needs to be investigated: + + nooption NFSD # New Network Filesystem Server options NFSCLIENT # Network Filesystem Client -options NFSSERVER # Network Filesystem Server - +options NFSSERVER # Network Filesystem Server - As root, do the usual build steps, e.g.: -make -j4 buildworld -make buildkernel KERNCONF=${kernconf} -make installkernel KERNCONF=${kernconf} -make installworld - The install steps use DESTDIR. - + As root, do the usual build steps, e.g.: + + &prompt.root; make -j4 buildworld +&prompt.root; make buildkernel KERNCONF=${kernconf} +&prompt.root; make installkernel KERNCONF=${kernconf} +&prompt.root; make installworld + + The install steps use DESTDIR. @@ -1690,10 +1699,11 @@ - Also add the following users: -squid:*:100:100::0:0:User &:/usr/local/squid:/bin/sh -ganglia:*:102:102::0:0:User &:/usr/local/ganglia:/bin/sh - + Also add the following users: + + squid:*:100:100::0:0:User &:/usr/local/squid:/bin/sh +ganglia:*:102:102::0:0:User &:/usr/local/ganglia:/bin/sh + Add them to etc/group as well. @@ -1703,9 +1713,9 @@ - In etc/crontab: add - * * * * * root /var/portbuild/scripts/client-metrics - + In etc/crontab add: + + * * * * * root /var/portbuild/scripts/client-metrics @@ -1716,26 +1726,28 @@ - In etc/inetd.conf: add - infoseek stream tcp nowait nobody /var/portbuild/scripts/reportload - + In etc/inetd.conf add: + + infoseek stream tcp nowait nobody /var/portbuild/scripts/reportload - We run the cluster on UTC: - cp /usr/share/zoneinfo/Etc/UTC etc/localtime - + We run the cluster on UTC. If you have not set the + clock to UTC: + + &prompt.root; cp /usr/share/zoneinfo/Etc/UTC etc/localtime Create the appropriate - etc/rc.conf. (If you are using + /etc/rc.conf. (If you are using pxeboot, and have multiple, different, machines, you will need to put those in the override directories.) - Recommended entries for physical nodes: -hostname="${hostname}" + Recommended entries for physical nodes: + + hostname="${hostname}" inetd_enable="YES" linux_enable="YES" nfs_client_enable="YES" @@ -1750,17 +1762,17 @@ squid_enable="YES" squid_chdir="/usr2/squid/logs" squid_pidfile="/usr2/squid/logs/squid.pid" - - + - Required entries for VMWare-based nodes: -vmware_guest_vmmemctl_enable="YES" + Required entries for VMWare-based nodes: + + vmware_guest_vmmemctl_enable="YES" vmware_guest_guestd_enable="YES" - - + - Recommended entries for VMWare-based nodes: -hostname="" + Recommended entries for VMWare-based nodes: + + hostname="" ifconfig_em0="DHCP" fsck_y_enable="YES" @@ -1775,8 +1787,7 @@ squid_enable="YES" squid_chdir="/usr2/squid/logs" squid_pidfile="/usr2/squid/logs/squid.pid" - - + &man.ntpd.8; should not be enabled for VMWare instances. @@ -1797,8 +1808,9 @@ - Modify etc/sysctl.conf: -9a10,30 + Modify etc/sysctl.conf: + + 9a10,30 > kern.corefile=/usr2/%N.core > kern.sugid_coredump=1 > #debug.witness_ddb=0 @@ -1819,8 +1831,7 @@ > security.jail.chflags_allowed=1 > security.jail.enforce_statfs=1 > -> vfs.lookup_shared=1 - +> vfs.lookup_shared=1 @@ -1829,11 +1840,8 @@ @pointyhat.freebsd.org. - - - @@ -1841,13 +1849,13 @@ - Install the following ports: -net/rsync + Install the following ports: + + net/rsync security/openssh-portable (with HPN on) security/sudo sysutils/ganglia-monitor-core (with GMETAD off) -www/squid (with SQUID_AUFS on) - +www/squid (with SQUID_AUFS on) There is a WIP to create a meta-port, but it is not yet complete. @@ -1855,7 +1863,7 @@ - Customize files in usr/local/etc/. + Customize files in usr/local/etc/. Whether you do this on the client itself, or another machine, will depend on whether you are using pxeboot. @@ -1864,7 +1872,7 @@ The trick of using conf override subdirectories is less effective here, because you would need to copy over all subdirectories of - usr/. This is an implementation + /usr/. This is an implementation detail of how the pxeboot works. @@ -1873,8 +1881,9 @@ Modify - usr/local/etc/gmond.conf: -21,22c21,22 + usr/local/etc/gmond.conf: + + 21,22c21,22 < name = "unspecified" < owner = "unspecified" --- @@ -1883,8 +1892,7 @@ 24c24 < url = "unspecified" --- -> url = "http://pointyhat.freebsd.org" - +> url = "http://pointyhat.freebsd.org" If there are machines from more than one cluster in the @@ -1897,8 +1905,9 @@ Create usr/local/etc/rc.d/portbuild.sh, using the appropriate value for - scratchdir: -#!/bin/sh + scratchdir: + + #!/bin/sh # # Configure a package build system post-boot @@ -1915,14 +1924,14 @@ done # Flag that we are ready to accept jobs -touch /tmp/.boot_finished - +touch /tmp/.boot_finished Modify - usr/local/etc/squid/squid.conf: -288,290c288,290 + usr/local/etc/squid/squid.conf: + + 288,290c288,290 < #auth_param basic children 5 < #auth_param basic realm Squid proxy-caching web server < #auth_param basic credentialsttl 2 hours @@ -1937,8 +1946,7 @@ 2007a2011 > maximum_object_size 400 MB 2828a2838 -> negative_ttl 0 minutes - +> negative_ttl 0 minutes Also, change usr/local to usr2 in @@ -1967,18 +1975,17 @@ Modify - usr/local/etc/sudoers: -38a39,42 + /usr/local/etc/sudoers: + + 38a39,42 > > # local changes for package building > %wheel ALL=(ALL) ALL -> ports-${arch} ALL=(ALL) NOPASSWD: ALL - +> ports-${arch} ALL=(ALL) NOPASSWD: > ALL - @@ -1993,16 +2000,16 @@ - As root: -mkdir portbuild -chown ports-${arch}:ports-${arch} portbuild -mkdir pkgbuild -chown ports-${arch}:ports-${arch} pkgbuild -mkdir squid -mkdir squid/cache -mkdir squid/logs -chown -R squid:squid squid - + As root: + + &prompt.root; mkdir portbuild +&prompt.root; chown ports-${arch}:ports-${arch} portbuild +&prompt.root; mkdir pkgbuild +&prompt.root; chown ports-${arch}:ports-${arch} pkgbuild +&prompt.root; mkdir squid +&prompt.root; mkdir squid/cache +&prompt.root; mkdir squid/logs +&prompt.root; chown -R squid:squid squid @@ -2011,7 +2018,7 @@ between boots then they must either preserve their /tmp, or revalidate their available builds at boot time (see the script on the amd64 - machines). They must also clean up stale chroots from previous + machines). They must also clean up stale jails from previous builds before creating /tmp/.boot_finished. @@ -2022,11 +2029,11 @@ As root, initialize the squid - directories: - squid -z + directories: + + &prompt.root; squid -z - @@ -2073,15 +2080,15 @@ of MASTER_SITE_BACKUP must be common to all nodes, but the first entry in MASTER_SITE_OVERRIDE should be the - nearest local FTP mirror. Example: -.if defined(FETCH_ORIGINAL) + nearest local FTP mirror. Example: + + .if defined(FETCH_ORIGINAL) MASTER_SITE_BACKUP= .else MASTER_SITE_OVERRIDE= \ ftp://friendly-local-ftp-mirror/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR}/ \ ftp://${BACKUP_FTP_SITE}/pub/FreeBSD/distfiles/${DIST_SUBDIR}/ -.endif - +.endif tar it up and move it to the right @@ -2092,19 +2099,20 @@ Hint: you will need one of these for each machine; however, if you have multiple machines at one site, you should create a site-specific one (e.g. in - /var/portbuild/conf/clients/) + /var/portbuild/conf/clients/) and symlink to it. Create -/var/portbuild/${arch}/portbuild-${hostname} +/var/portbuild/${arch}/portbuild-${hostname} using one of the existing ones as a guide. This file contains overrides to /var/portbuild/${arch}/portbuild.conf. - Suggested values: -disconnected=1 + Suggested values: + + disconnected=1 http_proxy="http://localhost:3128/" squid_dir=/usr2/squid scratchdir=/usr2/pkgbuild @@ -2113,17 +2121,16 @@ rsync_gzip=-z infoseek_host=localhost -infoseek_port=${tunelled-tcp-port} - +infoseek_port=${tunelled-tcp-port} - Possible other values: -use_md_swap=1 + Possible other values: + + use_md_swap=1 md_size=9g use_zfs=1 scp_cmd="/usr/local/bin/scp" ssh_cmd="/usr/local/bin/ssh" - - + @@ -2141,12 +2148,11 @@ Add an appropriate data_source entry to /usr/local/etc/gmetad.conf: - - data_source "arch/location Package Build Cluster" 30 hostname - - You will need to restart gmetad. - + data_source "arch/location Package Build Cluster" 30 hostname + + You will need to restart + gmetad. @@ -2195,18 +2201,11 @@ Tell qmanager about the node. Example: - python path/qmanager/qclient add - name=uniquename - arch=arch - osversion=osversion - numcpus=number - haszfs=0 - online=1 - domain=domain - primarypool=package - pools="package all" maxjobs=1 - acl="ports-arch,deny_all" - + &prompt.root; python path/qmanager/qclient \ +add name=uniquename arch=arch osversion=osversion \ +numcpus=number haszfs=0 online=1 \ +domain=domain primarypool=package pools="package all" \ +maxjobs=1 acl="ports-arch,deny_all" @@ -2215,11 +2214,14 @@ How to configure a new &os; branch + + Steps necessary before <application>qmanager</application> is started + + + When a new branch is created, some work needs to be done to specify that the previous branch is no longer - equivalent to HEAD. The following - instructions apply to the previous - branch number: + equivalent to HEAD. @@ -2235,7 +2237,8 @@ For what was previously head, change SRC_BRANCH_branch_TAG to - RELENG_branch_0. + RELENG_branch_0 + (literal zero). @@ -2254,16 +2257,16 @@ (Only for old codebase) Create a new zfs filesystem - for sources: - zfs create a/snap/src-branch - + for sources: + + &prompt.root; zfs create a/snap/src-branch (Only necessary for old codebase): - Checkout a src tree in the new filesystem: - cvs -Rq -d /r/ncvs co -d src-branch-r RELENG_branch - + Checkout a src tree in the new filesystem: + + &prompt.root; cvs -Rq -d /r/ncvs co -d src-branch-r RELENG_branch @@ -2307,26 +2310,56 @@ (Only necessary for old codebase): In the /var/portbuild/errorlogs/ - directory, create links for the webserver: -ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full -ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs -ln -s ../arch/branch/builds/latest/errors arch-branch-latest -ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs -ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full -ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest - + directory, create links for the webserver: + + &prompt.root; ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full +&prompt.root; ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs +&prompt.root; ln -s ../arch/branch/builds/latest/errors arch-branch-latest +&prompt.root; ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs +&prompt.root; ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full +&prompt.root; ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest + + + + + + + Steps necessary after <application>qmanager</application> is started + + + + + Again, as + ports-arch: + + + - Kick-start the build for the branch with - build create arch branch - + For each branch that will be supported, do the following: + - - Create bindist.tar - . + + + Kick-start the build for the branch with the following + command: + + &prompt.root; build create arch branch + + + + Create + bindist.tar. + + + + + + + @@ -2354,10 +2387,11 @@ - (both): -umount a/snap/src-old-branch/src; -umount a/snap/src-old-branch; -zfs destroy -r a/snap/src-old-branch + (both): + + &prompt.root; umount a/snap/src-old-branch/src +&prompt.root; umount a/snap/src-old-branch +&prompt.root; zfs destroy -r a/snap/src-old-branch @@ -2418,7 +2452,7 @@ with the following changes: - Change + Change the value of SRC_BRANCH_branch_TAG to RELENG_branch_N where N is the newest 'oldest' release @@ -2456,6 +2490,11 @@ How to configure a new architecture + + Steps necessary before <application>qmanager</application> is started + + + The initial steps need to be done using sudo. @@ -2469,20 +2508,19 @@ - mkdir /var/portbuild/arch + &prompt.root; mkdir /var/portbuild/arch - Create a new zfs filesystem: - zfs create -o mountpoint=/a/portbuild/arch a/portbuild/arch - + Create a new zfs filesystem: + + &prompt.root; zfs create -o mountpoint=/a/portbuild/arch a/portbuild/arch - -chown ports-arch:portmgr /var/portbuild/arch; -chmod 755 /var/portbuild/arch; -cd /var/portbuild/arch + &prompt.root; chown ports-arch:portmgr /var/portbuild/arch +&prompt.root; chmod 755 /var/portbuild/arch +&prompt.root; cd /var/portbuild/arch @@ -2490,10 +2528,14 @@ - Create a directory for buildlogs and errorlogs: - mkdir /dumpster/pointyhat/arch/archive - + Create a directory for buildlogs and errorlogs: + + (Specific to pointyhat; fix this! (XXX MCL) + + + &prompt.root; mkdir /dumpster/pointyhat/arch/archive + It is possible that /dumpster/pointyhat will not have enough space. In that case, create the archive @@ -2505,9 +2547,13 @@ - Create a link to the above for the webserver: - ln -s /dumpster/pointyhat/arch/archive archive - + Create a link to the above for the webserver: + + + Specific to pointyhat; fix this! (XXX MCL) + + + &prompt.root; ln -s /dumpster/pointyhat/arch/archive archive @@ -2520,7 +2566,9 @@ In the /var/portbuild/arch - directory:mkdir clients + directory: + + &prompt.root; mkdir clients @@ -2528,17 +2576,18 @@ - mkdir loads + &prompt.root; mkdir loads - mkdir lockfiles + &prompt.root; mkdir lockfiles Create a local make.conf. In the - most trivial case, you can - ln ../make.conf ./make.conf + most trivial case, you can + + &prompt.root; ln ../make.conf ./make.conf @@ -2549,9 +2598,9 @@ (Only necessary for old codebase) Create pnohang.arch. (The easiest way may be to do the following on a client, and - then copy it back): - cc pnohang.c -o pnohang-arch - + then copy it back): + + &prompt.root; cc pnohang.c -o pnohang-arch @@ -2566,7 +2615,7 @@ - cd .ssh && ssh-keygen + &prompt.root; cd .ssh && ssh-keygen @@ -2575,9 +2624,10 @@ - Make the private configuration directory: - mkdir /var/portbuild/conf/arch - + If needed, make the private configuration + directory: + + &prompt.root; mkdir /var/portbuild/conf/arch @@ -2592,11 +2642,6 @@ - Tell qmanager about the arch: - python path/qmanager/qclient add_acl name=ports-arch uidlist=ports-arch gidlist=portmgr sense=1 - - - (Only necessary for new codebase): Add arch to SUPPORTED_ARCHS in /var/portbuild/arch/server.conf. @@ -2616,27 +2661,63 @@ (Only necessary for old codebase): As with the procedure for creating a new branch: - in the /var/portbuild/errorlogs/ - directory, create links for the webserver: -ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full -ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs -ln -s ../arch/branch/builds/latest/errors arch-branch-latest -ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs -ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full -ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest - + in the /var/portbuild/errorlogs/ + directory, create links for the webserver: + + &prompt.root; ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full +&prompt.root; ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs +&prompt.root; ln -s ../arch/branch/builds/latest/errors arch-branch-latest +&prompt.root; ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs +&prompt.root; ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full +&prompt.root; ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest - In the /var/portbuild/errorlogs/ - directory, create two more links for the webserver: -ln -s ../arch/archive/buildlogs arch-buildlogs -ln -s ../arch/archive/errorlogs arch-errorlogs + In the /var/portbuild/errorlogs/ + directory, create two more links for the webserver: + + &prompt.root; ln -s ../arch/archive/buildlogs arch-buildlogs +&prompt.root; ln -s ../arch/archive/errorlogs arch-errorlogs + One last time using sudo: + + + + + Add an appropriate arch entry for + /var/portbuild/scripts/dologs to the root + crontab. (This is a hack and should go away.) + + + + + + + + + Steps necessary after <application>qmanager</application> is started + + + + + Again using sudo: + + + + + Tell qmanager about the + arch: + + &prompt.root; python path/qmanager/qclient add_acl name=ports-arch uidlist=ports-arch \ +gidlist=portmgr sense=1 + + + + Again, as ports-arch: @@ -2648,8 +2729,9 @@ - Kick-start the build for the branch with - build create arch branch + Kick-start the build for the branch with + + &prompt.root; build create arch branch @@ -2673,19 +2755,20 @@ arch: add the arch to /var/portbuild/scripts/dopackagestats. + - - Add an appropriate arch entry for - /var/portbuild/scripts/dologs to the root - crontab. (This is a hack and should go away.) - - - + + + + How to configure a new head node (pointyhat instance) + This section only applies to the new codebase. New installations + using the old codebase are not supported. + This section is in progress. Please talk to Mark Linimon before making any changes. @@ -2700,60 +2783,91 @@ - For each supported arch, add a - ports-${arch} - user and group. Add them to the wheel - group. They should have the '*' password. - Also, similarly, create the ports and - portmgr users. + Create a user to own the portbuild + repository, such as portmgr, and add it + to the wheel group. It should have the + '*' password. - For each supported arch, create - /home/ports-${arch}/.ssh/ - and populate authorized_keys. + Add the following to + /boot/loader.conf: + + console="vidconsole,comconsole" - - Create the appropriate files in - /etc/.ssh/. + Create the appropriate + etc/rc.conf. + + Required entries: + + hostname="${hostname}" +sshd_enable="YES" + + + Recommended entries: + + background_fsck="NO" +clear_tmp_enable="YES" +dumpdev="AUTO" +fsck_y_enable="YES" + +apache22_enable="YES" +apache_flags="" +apache_pidfile="/var/run/httpd.pid" +gmetad_enable="YES" +gmond_enable="YES" +inetd_enable="YES" +# XXX MCL these don't work! +#inetd_flags="-R 10000 -lw" +mountd_enable="YES" +nfs_server_enable="YES" +nfs_server_flags="-u -t -n 12" +nfs_remote_port_only="YES" +ntpd_enable="YES" +ntpdate_enable="YES" +ntpdate_flags="north-america.pool.ntp.org" +rpcbind_enable="YES" +rpc_lockd_enable="NO" +rpc_statd_enable="YES" +sendmail_enable="NONE" +smartd_enable="YES" + - Add the following to /boot/loader.conf: -console="vidconsole,comconsole" - + Create etc/resolv.conf, if + necessary. - Add the following to /etc/sysctl.conf: -kern.maxfiles=40000 -kern.maxfilesperproc=38000 - + Create the appropriate files in + /etc/ssh/. - Make sure the following change is made to - /etc/ttys: -ttyu0 "/usr/libexec/getty std.9600" vt100 on secure - + Add the following to + /etc/sysctl.conf: + + kern.maxfiles=40000 +kern.maxfilesperproc=38000 - TBA + Make sure the following change is made to + /etc/ttys: + + ttyu0 "/usr/libexec/getty std.9600" vt100 on secure - @@ -2764,39 +2878,43 @@ Create a zfs volume named a and mount it on - /a: -# zpool create a mirror da1 da2 mirror da3 da4 mirror da5 da6 mirror da7 da8 - + /a. An example: + + &prompt.root; zpool create a mirror da1 da2 mirror da3 da4 mirror da5 da6 mirror da7 da8 - Set up the base portbuild directory: -# mkdir -p /a/portbuild -# cd /a/portbuild -# chown portmgr:portmgr . -# chmod 775 . - + Create the portbuild directory: + + &prompt.root; mkdir -p /a/portbuild - TBA + Create and mount a new zfs + filesystem on it: + + &prompt.root; zfs create -o mountpoint=/a/portbuild a/portbuild + + Set up the portbuild directory: + + &prompt.root; cd /a/portbuild +&prompt.root; chown portmgr:portmgr . +&prompt.root; chmod 775 . +&prompt.root; ln -sf /a/portbuild /var/portbuild + - Configuring <literal>src</literal> - TBA - - @@ -2804,9 +2922,11 @@ - The following ports (or their latest successors) are required: -databases/py-pysqlite23 -databases/py-sqlalchemy + The following ports (or their latest successors) are + required: + + databases/py-sqlite3 +databases/py-sqlalchemy (only SQLITE is needed) devel/git (WITH_SVN) devel/py-configobj devel/py-setuptools @@ -2815,21 +2935,22 @@ net/rsync sysutils/ganglia-monitor-core (with GMETAD off) sysutils/ganglia-webfrontend (WITHOUT_X11) -www/apache22 (with EXT_FILTER and THREADS) - +www/apache22 (with EXT_FILTER) - Expect those to bring in: -databases/sqlite3 -lang/perl-5.12 -lang/python27 - + Expect those to bring in, among others: - The following ports (or their latest successors) are strongly suggested: -benchmarks/bonnie++ + databases/sqlite3 +lang/perl-5.14 (or successor) +lang/python27 (or sucessor) + + The following ports (or their latest successors) are + strongly suggested: + + benchmarks/bonnie++ devel/ccache mail/postfix net/isc-dhcp41-server -ports-mgmt/pkg_cutleaves +ports-mgmt/pkg ports-mgmt/pkg_tree ports-mgmt/portaudit ports-mgmt/portmaster @@ -2837,31 +2958,240 @@ shells/bash shells/zsh sysutils/screen -sysutils/smartmontools - +sysutils/smartmontools - Configure mail by doing the following: TBA. - + If not already created, add the following users: + + squid:*:100:100::0:0:User &:/usr/local/squid:/bin/sh +ganglia:*:102:102::0:0:User &:/usr/local/ganglia:/bin/sh + + Add them to /etc/group as + well. + + + Configure mail by doing the following: + + TBA. + + + + Setting up and configuring the portbuild repository + + + + + Set up an account with subversion config files + installed. Again, we generally use + portmgr. + + + + As that user, set up the repository: + + &prompt.user; svn checkout svn://svn.freebsd.org/base/projects/portbuild /a/portbuild + + + + Configure how build slaves will talk to your server + by making the following changes to + /var/portbuild/conf/client.conf: + + + + + Set CLIENT_NFS_MASTER to wherever + your build slaves will PXE boot from. (Possibly, the + hostname of your server.) + + + + Set CLIENT_BACKUP_FTP_SITE + to a backup site for FTP fetches; again, possibly + the hostname of your server. + + + + Set CLIENT_UPLOAD_HOST to + where completed packages will be uploaded. + + + + + Most of the other default values should be fine. + + + + Most of the default values in + /var/portbuild/conf/common.conf + should be fine. This file holds definitions used by + both the server and all its clients. + + + + /var/portbuild/conf/make.conf + is most likely obsolete. + + + + Configure the server by making the following changes to + /var/portbuild/conf/server.conf: + + + + + Set SUPPORTED_ARCHS to the + list of architectures you wish to build packages for. + + + + For each source branch you will be building for, set + SRC_BRANCHES and + SRC_BRANCH_branch_TAG + as detailed in . + You should not need to change + SRC_BRANCHES_PATTERN. + + + + Set UPLOAD_DIRECTORY, + UPLOAD_TARGET, and + UPLOAD_USER as approprite + for your site. + + + + Set MASTER_URL to the http + URL of your server. This will be stamped into the + package build logs and the indices thereof. + + + + + Most of the other default values should be fine. + + + + - - Other + + pre-<application>qmanager</application> - TBA + For each architecture, follow the steps in + . + + <application>qmanager</application> + + + + + Copy the following files from + /var/portbuild/conf/etc/rc.d to + /usr/local/etc/rc.d: + + buildproxy.sh +pollmachine.sh +qmanager.sh + + As root, start each one of them. You may find it handy + to start each under screen for + debugging purposes. + + + + Initialize the qmanager + database's acl list: + + &prompt.root; python /var/portbuild/qmanager/qclient add_acl name=deny_all uidlist= gidlist= sense=0 + + + + + + + + Creating src and ports repositories + + + + + Run the following commands manually to create the + src and ports + repositories, respectively: + + &prompt.user; sudo /var/portbuild/scripts/updatesnap.ports +&prompt.user; sudo /var/portbuild/scripts/updatesnap + + + These will be periodically run from the root + crontab, which you will + install below. + + + + + + Other services + + + + + Configure + /usr/local/etc/apache22/httpd.conf + as appropriate for your site. + + + + Copy /var/portbuild/conf/apache.conf + to the appropriate Includes subdirectory, e.g., + /usr/local/etc/apache22/Includes/apache.conf. + Configure it as appropriate for your site. + + + + Set up /var/portbuild/crontab/root as + the root crontab via crontab -e. If you do + not support all the archs listed there, make sure to comment out + the appropriate dologs entries. + + + + If your build slaves will be pxebooted, make sure to + enable the tftp entries in + /etc/inetd.conf. + + + + + + + + Finishing up + + + + + For each architecture, follow the steps in . + + + + You should now be ready to build packages. + @@ -2894,11 +3224,11 @@ Take the machine out of mlist and (for ata disks) run smartctl on the drive: - smartctl -t long /dev/ad0 + &prompt.root; smartctl -t long /dev/ad0 It will take about 1/2 hour: - gohan51# smartctl -t long /dev/ad0 + &prompt.root; smartctl -t long /dev/ad0 smartctl version 5.38 [i386-portbld-freebsd8.0] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/