FreeBSD recently grew support for profiling in many different ways through use of the hardware's performance monitoring counters (PMCs). This lets you sample things like instructions retired, cache hits/misses, branch mispredicts, etc. either for a single program (and its descendents) or the entire system.

This has been a great aid for me while working on graphics performance in FreeBSD. The usual problem here is that some command is slow, and I want to know what's responsible, whether it's cairo, liboil, X.Org, the DRM, etc., and I want to know what functions are responsible within those pieces of software. The pmccontrol/pmcstat programs will give me a pile of .gmon files of profiles for each of these when I run a command under them. So, I made a set of simple perl/sh scripts for personal use to post-process these .gmon files into a nice flat output for programs I use.

These scripts are very under-developed. We really need someone to write a unified program to handle hwpmc results, but for now I've managed to become much more productive using just this.

If you get a complaint when using these scripts (or sampling mode in general) while using a UP i386 about Operation not supported, this is a limitation of hwpmc at the moment, according to its author. The problem is that many UP i386 motherboards disable the local apic, which is required for hwpmc to deliver the interrupts how it wants to.

If you get a crash at the end of the pmccontrol run, this may be a current issue that jkoshy is working on. For now, apply this patch: hwpmc-logging-dontpanic.diff (from jkoshy)