%
% Slide template for BSD-Day@2010.
% Take it or leave it :-)
%
% It requires LaTeX and LaTeX Beamer [1] to compile.
% pdfLaTeX is recommended for compilation as it produces
% PDF file immediately.
%
% $ pdflatex some.latex
%
%
% It is also recommended to convert images to PDF
% by using ImageMagick ("convert") before including them.
%
% [1] http://latex-beamer.sourceforge.net
%

\documentclass{beamer}

\usepackage{url}
\usepackage[english]{babel}
\usepackage{verbatim}
\usepackage{graphicx}

\mode<presentation>
{
  \definecolor{beamer@gker}{rgb}{0.95,0.40,0.06}
  \setbeamercolor*{structure}{fg=beamer@gker}
  \logo{\includegraphics[scale=0.1]{logo.pdf}}
}

\title{FreeBSD Ports Collection, Autumn 2010}
\author{Mark Linimon \\ \url{linimon@FreeBSD.org}}
\institute{\includegraphics[scale=0.2]{logo.pdf}}
\date{E\"{o}tv\"{o}s Lor\'{a}nd University, \\
Budapest, Hungary \\
November 20, 2010}

\begin{document}

\begin{frame}[plain]
  \titlepage
\end{frame}

\begin{frame}
\frametitle{Ports Collection overall}

Main URL: \url{http://www.freebsd.org/ports}

\bigskip

Important statistics:

\begin{itemize}
\item 22334 ports
\item several architectures:
\begin{itemize}
\item now: amd64, i386, ia64, powerpc (32-bit), sparc64
\item future: mips, powerpc64
\end{itemize}
\item 4 (!) major branches:
\begin{itemize}
\item 6, 7, 8, 9
\item 6 EOL is imminent
\end{itemize}
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Social Structure}

\begin{itemize}
\item Ports Management team (\url{http://www.FreeBSD.org/portmgr}) is responsible for the state of the Ports Collection
\begin{itemize}
\item approves new committers for ports bits
\item creates packages for upload
\begin{itemize}
\item -stable branches
\item -current branch
\item releases
\end{itemize}
\item runs "test builds" for infrastructure changes
\end{itemize}
\item 160 ports committers
\begin{itemize}
\item 30 very active
\item another 70 fairly active
\end{itemize}
\item many hundreds of maintainers
\item mailing lists:
\begin{itemize}
\item \url{freebsd-ports@FreeBSD.org}
\item \url{freebsd-ports-bugs@FreeBSD.org}
\end{itemize}
\item IRC channels:
\begin{itemize}
\item bsdports (EFNet)
\end{itemize}
\end{itemize}

\end{frame}

\begin{frame}[fragile]
\frametitle{Technical Details}

\begin{itemize}
\item 22334 ports in 73 categories
\item wc -l of files in ports/Mk/ :
{\small
\begin{verbatim}
    6458 bsd.port.mk
    1566 bsd.sites.mk
     842 bsd.gnome.mk
     653 bsd.gecko.mk
     593 bsd.ruby.mk
     592 bsd.wx.mk
     572 bsd.lua.mk
     547 bsd.gnustep.mk
     545 bsd.java.mk
     524 bsd.efl.mk
     ...
   21671 total
\end{verbatim}
}
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Hardware}

portmgr has a number of machines at its disposal to help do package builds.

\begin{itemize}
\item dispatch machine: \url{pointyhat.FreeBSD.org/errorlogs/}
\begin{itemize}
\item only set up for automated builds
\end{itemize}
\item package buiding machines:
\begin{itemize}
\item Yahoo: 7 amd64 machines, some older i386 machines, 2 ia64 machines
\item ISC: 2 amd64 machines, 5 i386 machines, 1 powerpc machine, 8 sparc64 machines
\item on loan: \~12 sparc64 machines
\end{itemize}
\end{itemize}

\bigskip

In general, these machines are constantly busy.

It is portmgr's job to filter the false positives out of the failure logs,
and forward the rest in emails to the maintainers.

\end{frame}

\begin{frame}
\frametitle{New Hardware: In Progress}

\begin{itemize}
\item new dispatch machine: \url{pointyhat-west.isc.FreeBSD.org/errorlogs/}
\item new dispatch machine: \url{pointyhat-east.isc.FreeBSD.org/errorlogs/}
\item package buiding machines:
\begin{itemize}
\item NYI: 11 amd64/i386 machines (new)
\item possibly, some VMWare instances
\end{itemize}
\end{itemize}

\bigskip

The intention is that we can now dispatch many more times the amount of
builds that we do now.

\end{frame}

\begin{frame}
\frametitle{How This Will Help}

In the past, we have been so constrained by hardware, that we have been
unable to keep up with demand:

\begin{itemize}
\item incremental package builds for upload (-stable, -current)
\item full package builds for upload (-release)
\item regression-tests for bsd.*.mk changes
\item regression-tests for other sweeping changes
\begin{itemize}
\item gettext
\item autotools
\item xorg
\item (and so on.)
\end{itemize}
\end{itemize}

Result: PRs stack up, and people get discouraged.

\end{frame}

\begin{frame}
\frametitle{The Plan For This Hardware}

\begin{itemize}
\item allow for -exp builds even during release cycles
\item allow for multiple -exp runs simultaneously
\item start doing -exp runs for nonstandard options:
\begin{itemize}
\item non-standard LOCALBASE
\item NO\_GETTEXT
\item NO\_X11
\end{itemize}
\end{itemize}

See also \url{http://www.freebsd.org/cgi/cvsweb.cgi/ports/KNOBS}

\end{frame}

\begin{frame}
\frametitle{Other Machines}

\begin{itemize}
\item \url{http://portsmon.freebsd.org}
\begin{itemize}
\item inputs: CVS commits, GNATS PRs, pointyhat errorlogs, other http/ftp pages
\item outputs: web pages per-port, per-maintainer, per-build; also, periodic emails
\end{itemize}
\item Quality Assurance Tinderbox (QAT)
\begin{itemize}
\item performs test-builds of packages immediately on commit
\item performs test-builds of packages in batch (like pointyhat), but can use non-standard options
\item inputs: CVS commits
\item outputs: emails
\item optimized for automated builds
\end{itemize}
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Related Technology (1)}

Ports tinderbox (see \url{http://www.freebsd.org/cgi/cvsweb.cgi/ports/ports-mgmt/tinderbox/})
\begin{itemize}
\item owned by individual committers
\item intended for pre-commit screening
\item optimized for user-started builds
\item outputs: error logs
\end{itemize}

We need more people to be using these!

\end{frame}

\begin{frame}
\frametitle{Related Technology (2)}

\begin{itemize}
\item Emanuel Haupt's distfile survey (see \url{http://distilator.org}
\begin{itemize}
\item iterates over all ports
\item iterates over all sources of distfiles
\end{itemize}
\item XXX's portscout (see \url{http://portscout.org/}
\begin{itemize}
\item iterates over all ports looking for updated distfiles
\end{itemize}
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Challenges}

There are a number of challenges.

\begin{itemize}
\item keeping the ports tree up-to-date
\item making sure ports build correctly in all build environments
\item dealing with regressions
\end{itemize}

One strategy: staging areas for large commits:
\begin{itemize}
\item xorg
\item kde
\item gnome
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Common Sources Of Port Errors (1)}

\begin{itemize}
\item missing or extra items vs. packing list
\item failure to configure
\item failure to compile
\item compiler bug
\item error in dependency (2nd-order effect)
\item fetch error
\begin{itemize}
\item transient
\item actual distfile update
\item distfile no longer available
\end{itemize}
\end{itemize}

See \url{http://portsmon.freebsd.org/chartsandgraphs/uniqueerrorcounts.html}:

\end{frame}

\begin{frame}
\frametitle{Common Sources Of Port Errors (2)}

\includegraphics[scale=0.5]{uniqueerrorcounts.png}

\end{frame}

\begin{frame}
\frametitle{How You Can Help}

We can always use more help:

\begin{itemize}
\item send port updates for the ports you use
\item send error reports for the ports you use
\item adopt an unmaintained port (please!)
\end{itemize}

send-pr(1) is your friend!

\end{frame}

\begin{frame}
\frametitle{Frequently Asked Questions}

\begin{itemize}
\item is the ports tree branched?
\item is it possible to cross-build ports?
\item will ports switch from CVS to SVN?
\item how can I become a committer?
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Handy Links}

Here are some handy links.

\bigskip

\begin{itemize}
\item \url{http://pointyhat.freebsd.org/errorlogs}
\item \url{http://portsmon.freebsd.org}
\item \url{http://freshports.org}
\item \url{http://people.freebsd.org/~linimon/presentations/bsdday-ports-2010.pdf} (this presentation)
\end{itemize}

\bigskip
\bigskip

\end{frame}

\begin{frame}
\frametitle{Acknowledgements}

This presentation would not have been possible without:

\begin{itemize}
\item a travel grant sponsored by the FreeBSD Foundation: \url{http://www.freebsdfoundation.org/}
\item assistance from our organizer Gabor Pali
\end{itemize}

Thanks to both of them!

\bigskip
\bigskip

\end{frame}

\begin{frame}
\begin{center}
{\Large And now, time for your questions!} \\
\emph{remember, there is supposed to be an award ...}
\end{center}
\end{frame}

\end{document}