GENERIC HEAD from 2018-09-04 19:28:46 UTC, r338461M, vmcore.1106 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-ALPHA4 #0 r338461M: Tue Sep 4 22:10:43 CEST 2018 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3519209472 (3356 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... uhub2: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. eval: limits: not found /etc/rc: WARNING: failed to start ddb warning: total configured swap (4194304 pages) exceeds maximum recommended amount (1721060 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 619924 free (2172 frags, 77219 blocks, 0.2% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 810261 free (11221 frags, 99880 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 33964063 free (43191 frags, 4240109 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1940976 free (382936 frags, 194755 blocks, 3.8% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2944613 free (136941 frags, 350959 blocks, 0.5% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc49 /usr/local/lib/libav /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: uhub1: 10 ports with 10 removable, self powered . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0xbc7249c bufwait (bufwait) @ kern/vfs_bio.c:3916 2nd 0x1b56c600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1157091 at witness_debugger+0x81 #1 0x1156de9 at witness_checkorder+0xa19 #2 0x10f4504 at _sx_xlock+0x64 #3 0x1432703 at ufsdirhash_add+0x53 #4 0x1437250 at ufs_direnter+0x4e0 #5 0x1440355 at ufs_mkdir+0x855 #6 0x171e58c at VOP_MKDIR_APV+0xac #7 0x11d448b at kern_mkdirat+0x1eb #8 0x11d4291 at sys_mkdir+0x31 #9 0x16e8ec3 at syscall+0x363 #10 0xffc033a7 at PTDpde+0x43af Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Sep 4 22:48:32 x4 smartd[632]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Sep 4 22:48:32 x4 smartd[632]: see the following Seagate web pages: Sep 4 22:48:32 x4 smartd[632]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Sep 4 22:48:32 x4 smartd[632]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Tue Sep 4 22:48 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: $ CFLAGS="-O0 -g" make Password: Login incorrect login: root Password: Sep 4 22:48:53 x4 login[775]: ROOT LOGIN (root) ON ttyu0 Last login: Tue Sep 4 16:14:59 on ttyu0 FreeBSD 12.0-ALPHA4 (PHO) #0 r338461M: Tue Sep 4 22:10:43 CEST 2018 You have new mail. root@x4:~ # bash $ cd ~pho/stress2/misc $ sh -x ./pmc.sh + id -u + [ 0 -ne 0 ] + . ../default.cfg + hostname + hostname=x4.osted.lan + [ -f ./x4.osted.lan ] + [ -f ../x4.osted.lan ] + . ../x4.osted.lan + testuser=stress + candidates='x4 t1 t2' + hostname + sed 's/\..*//' + me=x4 + echo x4 t1 t2 + sed s/x4// + candidates=' t1 t2' + ping -c 2 -t 1 t1 + continue + ping -c 2 -t 1 t2 + BLASTHOST=t2 + nfs_export=t2:/tmp + break + export 'RUNDIR=/tmp/stressX' + export 'RUNTIME=2m' + export 'VERBOSE=1' + export 'LOAD=20' + export 'INCARNATIONS=20' + export 'BLASTHOST=t2' + export 'runRUNTIME=3d' + export 'runINCARNATIONS=1' + export 'runLOAD=100' + export 'swapINCARNATIONS=40' + export 'swapLOAD=80' + export 'syscallKILL=1' + export 'swapKILL=1' + export 'rwLOAD=70' + export 'mkdirLOAD=80' + export 'creatLOAD=80' + export 'symlinkLOAD=20' + export 'tcpKILL=1' + export 'shmINCARNATIONS=5' + [ -z '' ] + export 'TESTPROGS= testcases/rw/rw testcases/swap/swap testcases/creat/creat testcases/mkdir/mkdir testcases/thr1/thr1 testcases/udp/udp testcases/tcp/tcp ' + export 'diskimage=/tmp/diskimage' + export 'mntpoint=/mnt' + export 'testuser=stress' + export 'mdstart=10' + export 'part=a' + export 'newfs_flags=-U' + pwd + export 'stress2origin=/home/pho/stress2/misc' + dirname /home/pho/stress2/misc + export 'stress2tools=/home/pho/stress2/tools' + CC=cc + pwd + dirname /home/pho/stress2/misc + top=/home/pho/stress2 + STRESS2BIN=/home/pho/stress2/bin + id stress + kldstat -v + grep -q hwpmc + kldload hwpmc hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff + loaded=1 + jot 2 + pmcstat -P instructions -O /tmp/sample.out.1 find -x /var -name not.there + sleep 1 + pmcstat -P instructions -O /tmp/sample.out.2 find -x /var -name not.there + export 'runRUNTIME=5m' + pgrep -q pmcstat + cd .. + ./run.sh vfs.cfg run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:00:15 Loop #1 mkdir: run time 0+00:02:00, incarnations 10, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 7, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 27, load 80, verbose 1 Sep 4 23:00:41 x4 kernel: pid 1241 (swap), uid 0, was killed: out of swap space 23:02:21 Loop #2 swap: run time 0+00:02:00, incarnations 12, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 Sep 4 23:02:41 x4 kernel: pid 1262 (swap), uid 0, was killed: out of swap space Sep 4 23:02:59 x4 kernel: pid 1271 (swap), uid 0, was killed: out of swap space 23:04:29 Loop #3 creat: run time 0+00:02:00, incarnations 10, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 ./all.sh -m 180 ./pmc.sh + wait pmcstat: pmcstat: WARNING: sampling was paused at least 5172 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 5172 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. + [ 1 ] + kldunload hwpmc + rm /tmp/sample.out.1 /tmp/sample.out.2 + exit 0 $ ./all.sh -m 180 ./pmc.sh 20180904 23:06:57 all: pmc.sh 20180904 23:06:57 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:07:05 Loop #1 swap: run time 0+00:02:00, incarnations 21, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 6, load 80, verbose 1 lock order reversal: 1st 0x2140a80 cpslk (cpslk) @ i386/i386/copyout.c:128 2nd 0xbc18438 bufwait (bufwait) @ vm/vm_pager.c:374 stack backtrace: #0 0x1157091 at witness_debugger+0x81 #1 0x1156de9 at witness_checkorder+0xa19 #2 0x10bd05b at lockmgr_xlock_hard+0x5b #3 0x10bd82f at __lockmgr_args+0x37f #4 0x1474d5c at getpbuf+0x22c #5 0x14439a9 at swap_pager_getpages+0x2f9 #6 0x1474873 at vm_pager_get_pages+0x53 #7 0x14525ca at vm_fault_hold+0x1a2a #8 0x1450b4e at vm_fault+0x5e #9 0x16e87fe at trap_pfault+0xfe #10 0x16e7cfc at trap+0x3dc #11 0xffc0315d at PTDpde+0x4165 #12 0x16cd331 at copyout_slow0+0x21 #13 0x16ccea2 at cp_slow0+0x152 #14 0x16cd2da at copyout+0xba #15 0x114fd13 at uiomove_faultflag+0x163 #16 0x114fb9f at uiomove+0x1f #17 0x115dfaf at pipe_read+0x24f 23:09:10 Loop #2 swap: run time 0+00:02:00, incarnations 1, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 20, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 11, load 80, verbose 1 23:11:15 Loop #3 swap: run time 0+00:02:00, incarnations 2, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 14, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 6130 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 6130 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. freework leak: 20/31. newblk leak: 373/383. 20180904 23:13:41 all: pmc.sh 20180904 23:13:41 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:13:48 Loop #1 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 23:15:54 Loop #2 creat: run time 0+00:02:00, incarnations 2, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 14, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 28, load 80, verbose 1 23:17:59 Loop #3 swap: run time 0+00:02:00, incarnations 24, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 8, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 6168 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 6168 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. freework leak: 196/227. 20180904 23:20:08 all: pmc.sh 20180904 23:20:08 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:20:16 Loop #1 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 23:22:16 Loop #2 swap: run time 0+00:02:00, incarnations 30, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 23:24:21 Loop #3 swap: run time 0+00:02:00, incarnations 7, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 2757 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 2757 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180904 23:26:30 all: pmc.sh 20180904 23:26:30 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:26:38 Loop #1 swap: run time 0+00:02:00, incarnations 9, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 6, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 23:28:43 Loop #2 swap: run time 0+00:02:00, incarnations 37, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 17, load 80, verbose 1 Sep 4 23:29:17 x4 kernel: pid 49431 (swap), uid 0, was killed: out of swap space 23:30:50 Loop #3 mkdir: run time 0+00:02:00, incarnations 5, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 2654 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.pmcstat: WARNING: sampling was paused at least 2654 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180904 23:32:54 all: pmc.sh 20180904 23:32:54 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:33:02 Loop #1 creat: run time 0+00:02:00, incarnations 13, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 17, load 80, verbose 1 23:35:05 Loop #2 swap: run time 0+00:02:00, incarnations 1, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 15, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 2, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3009 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 3019 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 23:37:10 Loop #3 swap: run time 0+00:02:00, incarnations 13, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 6, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 15, load 80, verbose 1 20180904 23:39:24 all: pmc.sh 20180904 23:39:24 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:39:31 Loop #1 mkdir: run time 0+00:02:00, incarnations 11, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 17, load 80, verbose 1 23:41:36 Loop #2 creat: run time 0+00:02:00, incarnations 7, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 16, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 17, load 80, verbose 1 23:43:44 Loop #3 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 9, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 5359 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.pmcstat: WARNING: sampling was paused at least 5359 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. newblk leak: 175/558. 20180904 23:46:26 all: pmc.sh 20180904 23:46:26 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:46:33 Loop #1 creat: run time 0+00:02:00, incarnations 3, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 14, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 2688 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 2688 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 23:48:33 Loop #2 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 14, load 80, verbose 1 23:50:41 Loop #3 swap: run time 0+00:02:00, incarnations 2, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 freework leak: 318/545. 20180904 23:52:50 all: pmc.sh 20180904 23:52:50 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:52:57 Loop #1 mkdir: run time 0+00:02:00, incarnations 6, load 80, verbose 1 23:54:57 Loop #2 swap: run time 0+00:02:00, incarnations 39, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 8, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 14, load 80, verbose 1 23:57:02 Loop #3 creat: run time 0+00:02:00, incarnations 19, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 2610 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.pmcstat: WARNING: sampling was paused at least 2610 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180904 23:59:42 all: pmc.sh 20180904 23:59:42 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 23:59:49 Loop #1 swap: run time 0+00:02:00, incarnations 27, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 6, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 15, load 80, verbose 1 00:01:56 Loop #2 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 26, load 80, verbose 1 00:04:01 Loop #3 swap: run time 0+00:02:00, incarnations 18, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 5946 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 5946 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. Note: pmc.sh's runtime was 465 seconds. N Min Max Median Avg Stddev x 15 368 450 382 392.66667 22.432969 20180905 00:07:31 all: pmc.sh 20180905 00:07:31 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:07:38 Loop #1 mkdir: run time 0+00:02:00, incarnations 3, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 2, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 21, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 5647 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.pmcstat: WARNING: sampling was paused at least 5647 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 00:09:44 Loop #2 swap: run time 0+00:02:00, incarnations 29, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 20, load 80, verbose 1 00:11:49 Loop #3 creat: run time 0+00:02:00, incarnations 1, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 10, load 80, verbose 1 20180905 00:13:58 all: pmc.sh 20180905 00:13:58 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:14:05 Loop #1 swap: run time 0+00:02:00, incarnations 7, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 16, load 80, verbose 1 00:16:11 Loop #2 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 19, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 10, load 80, verbose 1 Sep 5 00:16:36 x4 kernel: pid 60172 (swap), uid 0, was killed: out of swap space 00:18:17 Loop #3 swap: run time 0+00:02:00, incarnations 32, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 2, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 6293 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 6293 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 00:20:26 all: pmc.sh 20180905 00:20:26 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:20:34 Loop #1 creat: run time 0+00:02:00, incarnations 15, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 20, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 2, load 80, verbose 1 00:22:39 Loop #2 swap: run time 0+00:02:00, incarnations 13, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 16, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 6, load 80, verbose 1 00:24:44 Loop #3 swap: run time 0+00:02:00, incarnations 12, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 5510 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 5510 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 00:27:12 all: pmc.sh 20180905 00:27:12 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:27:19 Loop #1 creat: run time 0+00:02:00, incarnations 13, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 30, load 80, verbose 1 00:29:24 Loop #2 creat: run time 0+00:02:00, incarnations 1, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 14, load 80, verbose 1 00:31:29 Loop #3 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 31, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 5924 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 5924 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 00:33:53 all: pmc.sh 20180905 00:33:53 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:34:00 Loop #1 swap: run time 0+00:02:00, incarnations 21, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 11, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 Sep 5 00:34:32 x4 kernel: pid 90497 (swap), uid 0, was killed: out of swap space 00:36:05 Loop #2 creat: run time 0+00:02:00, incarnations 6, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 11, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 23, load 80, verbose 1 Sep 5 00:36:31 x4 kernel: pid 91778 (swap), uid 0, was killed: out of swap space 00:38:11 Loop #3 mkdir: run time 0+00:02:00, incarnations 12, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 19, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 16, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3594 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.pmcstat: WARNING: sampling was paused at least 3594 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 00:40:33 all: pmc.sh 20180905 00:40:33 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:40:40 Loop #1 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 13, load 80, verbose 1 00:42:45 Loop #2 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 5, load 80, verbose 1 00:44:50 Loop #3 swap: run time 0+00:02:00, incarnations 3, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 5, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 4774 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 4774 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 00:47:00 all: pmc.sh 20180905 00:47:00 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:47:08 Loop #1 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 2, load 80, verbose 1 00:49:14 Loop #2 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 3, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 7771 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 7771 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 00:51:19 Loop #3 swap: run time 0+00:02:00, incarnations 5, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 18, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 1, load 80, verbose 1 20180905 00:53:28 all: pmc.sh 20180905 00:53:28 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:53:35 Loop #1 creat: run time 0+00:02:00, incarnations 20, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 2685 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 2685 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 00:55:35 Loop #2 creat: run time 0+00:02:00, incarnations 12, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 10, load 80, verbose 1 00:57:35 Loop #3 creat: run time 0+00:02:00, incarnations 8, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 20180905 00:59:38 all: pmc.sh 20180905 00:59:38 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 00:59:46 Loop #1 swap: run time 0+00:02:00, incarnations 34, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 10, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 18, load 80, verbose 1 01:01:51 Loop #2 swap: run time 0+00:02:00, incarnations 17, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 16, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3005 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 3079 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 01:03:59 Loop #3 creat: run time 0+00:02:00, incarnations 12, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 4, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 36, load 80, verbose 1 20180905 01:06:08 all: pmc.sh 20180905 01:06:08 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:06:15 Loop #1 swap: run time 0+00:02:00, incarnations 5, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 3, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 5117 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 5117 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 01:08:20 Loop #2 swap: run time 0+00:02:00, incarnations 30, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 18, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 Sep 5 01:08:59 x4 kernel: pid 48069 (swap), uid 0, was killed: out of swap space Sep 5 01:09:13 x4 kernel: pid 48106 (swap), uid 0, was killed: out of swap space 01:10:28 Loop #3 mkdir: run time 0+00:02:00, incarnations 6, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 9, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 1, load 80, verbose 1 freework leak: 276/821. newblk leak: 235/793. 20180905 01:12:37 all: pmc.sh 20180905 01:12:37 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:12:44 Loop #1 swap: run time 0+00:02:00, incarnations 25, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 12, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 11, load 80, verbose 1 01:14:49 Loop #2 mkdir: run time 0+00:02:00, incarnations 2, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 3, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 2535 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 2535 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 01:16:49 Loop #3 swap: run time 0+00:02:00, incarnations 10, load 80, verbose 1 20180905 01:18:59 all: pmc.sh 20180905 01:18:59 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:19:06 Loop #1 swap: run time 0+00:02:00, incarnations 35, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 13, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 01:21:12 Loop #2 mkdir: run time 0+00:02:00, incarnations 10, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 6, load 80, verbose 1 01:23:18 Loop #3 creat: run time 0+00:02:00, incarnations 16, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 32, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 4110 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 4110 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 01:25:32 all: pmc.sh 20180905 01:25:32 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:25:38 Loop #1 swap: run time 0+00:02:00, incarnations 39, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 11, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 10, load 80, verbose 1 Sep 5 01:26:31 x4 kernel: pid 79863 (swap), uid 0, was killed: out of swap space 01:27:46 Loop #2 mkdir: run time 0+00:02:00, incarnations 9, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 19, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 7, load 80, verbose 1 01:29:54 Loop #3 mkdir: run time 0+00:02:00, incarnations 7, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 2076 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 2076 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 01:31:59 all: pmc.sh 20180905 01:31:59 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:32:07 Loop #1 swap: run time 0+00:02:00, incarnations 36, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 20, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 1, load 80, verbose 1 01:34:13 Loop #2 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 4, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 29, load 80, verbose 1 01:36:18 Loop #3 creat: run time 0+00:02:00, incarnations 14, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 5614 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 5614 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 01:39:11 all: pmc.sh 20180905 01:39:11 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:39:18 Loop #1 creat: run time 0+00:02:00, incarnations 8, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 01:41:18 Loop #2 swap: run time 0+00:02:00, incarnations 29, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 6, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 3, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3247 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 3248 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 01:43:23 Loop #3 creat: run time 0+00:02:00, incarnations 15, load 80, verbose 1 20180905 01:45:26 all: pmc.sh 20180905 01:45:26 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:45:33 Loop #1 mkdir: run time 0+00:02:00, incarnations 3, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 6, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 3, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3368 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 3431 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 01:47:38 Loop #2 swap: run time 0+00:02:00, incarnations 29, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 5, load 80, verbose 1 01:49:43 Loop #3 swap: run time 0+00:02:00, incarnations 4, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 19, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 9, load 80, verbose 1 20180905 01:51:52 all: pmc.sh 20180905 01:51:52 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:51:59 Loop #1 swap: run time 0+00:02:00, incarnations 18, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 13, load 80, verbose 1 01:54:04 Loop #2 swap: run time 0+00:02:00, incarnations 40, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 1, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 18, load 80, verbose 1 Sep 5 01:54:41 x4 kernel: pid 42794 (swap), uid 0, was killed: out of swap space 01:56:10 Loop #3 creat: run time 0+00:02:00, incarnations 11, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 2852 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 2852 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 01:58:14 all: pmc.sh 20180905 01:58:14 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 01:58:22 Loop #1 creat: run time 0+00:02:00, incarnations 12, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 2, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 3003 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 3003 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 02:00:22 Loop #2 swap: run time 0+00:02:00, incarnations 38, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 10, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 5, load 80, verbose 1 02:02:28 Loop #3 creat: run time 0+00:02:00, incarnations 12, load 80, verbose 1 20180905 02:04:32 all: pmc.sh 20180905 02:04:32 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 02:04:39 Loop #1 swap: run time 0+00:02:00, incarnations 28, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 17, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 17, load 80, verbose 1 02:06:45 Loop #2 mkdir: run time 0+00:02:00, incarnations 4, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 18, load 80, verbose 1 swap: run time 0+00:02:00, incarnations 20, load 80, verbose 1 02:08:51 Loop #3 swap: run time 0+00:02:00, incarnations 12, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 4, load 80, verbose 1 pmcstat: pmcstat: WARNING: sampling was paused at least 2754 times. Please consider tuning the "kern.hwpmc.nsamples" tunable.WARNING: sampling was paused at least 2754 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 20180905 02:11:07 all: done 20180905 02:11:07 all: done $ ./all.sh -ona pmc* Note: including known problem tests. 20180905 07:44:15 all: pmc.sh 20180905 07:44:15 all (1/6): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff run: run time 0+00:05:00, incarnations 1, load 100, verbose 1 07:44:22 Loop #1 swap: run time 0+00:02:00, incarnations 7, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 10, load 80, verbose 1 07:46:27 Loop #2 swap: run time 0+00:02:00, incarnations 39, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 5, load 80, verbose 1 pmcstat: WARNING: sampling was paused at least 4392 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. pmcstat: WARNING: sampling was paused at least 4392 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. 07:48:32 Loop #3 swap: run time 0+00:02:00, incarnations 40, load 80, verbose 1 creat: run time 0+00:02:00, incarnations 10, load 80, verbose 1 mkdir: run time 0+00:02:00, incarnations 16, load 80, verbose 1 20180905 07:50:45 all: pmc2.sh 20180905 07:50:45 all (2/6): pmc2.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff pmcstat: WARNING: sampling was paused at least 471090 times. Please consider tuning the "kern.hwpmc.nsamples" tunable. freework leak: 9/21. newblk leak: 2291/2302. 20180905 08:01:00 all: pmc3.sh 20180905 08:01:00 all (3/6): pmc3.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff lock order reversal: 1st 0x25fcb750 ufs (ufs) @ kern/vfs_mount.c:1343 2nd 0x208238bc devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1376 stack backtrace: #0 0x1157091 at witness_debugger+0x81 #1 0x1156de9 at witness_checkorder+0xa19 #2 0x10bcbe9 at lockmgr_lock_fast_path+0x189 #3 0x11b13bb at vop_stdlock+0x3b #4 0x171f400 at VOP_LOCK1_APV+0x90 #5 0x11d7961 at _vn_lock+0x61 #6 0x142771d at ffs_flushfiles+0xcd #7 0x1409f07 at softdep_flushfiles+0x67 #8 0x142a428 at ffs_unmount+0xe8 #9 0x11bb970 at dounmount+0x580 #10 0x11bb2e5 at sys_unmount+0x315 #11 0x16e8ec3 at syscall+0x363 #12 0xffc033a7 at PTDpde+0x43af mount leak: 1/98 20180905 08:07:11 all: pmc4.sh 20180905 08:07:11 all (4/6): pmc4.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 08:07:18 all: pmc5.sh 20180905 08:07:18 all (5/6): pmc5.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 08:07:24 all: pmc6.sh 20180905 08:07:24 all (6/6): pmc6.sh pmcstat: ERROR: Initialization of the pmc(3) library failed: No such file or directory FAIL pmc6.sh exit code 69 20180905 08:07:26 all: done 20180905 08:07:26 all: done Summary: Test which failed within the last two days. 20180905 08:07:26 pmc6.sh $ 20180905 09:00:16 all (1/5): pmc6.sh 20180905 09:00:21 all (2/5): pmc3.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff mount leak: 2/100 20180905 09:06:32 all (3/5): pmc5.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 09:06:39 all (4/5): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff freework leak: 47/58. newblk leak: 17/29. 20180905 09:12:59 all (5/5): pmc2.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff newblk leak: 1547/1576. 20180905 09:23:13 all (1/5): pmc5.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 09:23:20 all (2/5): pmc3.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff mount leak: 2/102 20180905 09:29:33 all (3/5): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 09:35:58 all (4/5): pmc2.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff newblk leak: 104/1680. 20180905 09:46:12 all (5/5): pmc6.sh 20180905 09:46:16 all (1/5): pmc2.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff 20180905 09:54:40 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff freework leak: 212/222. newblk leak: 184/194. 20180905 10:00:59 all: done 20180905 10:07:49 all (1/1): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff panic: [pmc,4980] pm=0x9e81380 runcount 0 cpuid = 1 time = 1536134874 KDB: stack backtrace: db_trace_self_wrapper(1847866,1dac740,0,18501fa,1511fa20,...) at db_trace_self_wrapper+0x2a/frame 0x1511f9f0 kdb_backtrace(180ea1d,5b8f8eda,0,1511faac,d,...) at kdb_backtrace+0x2d/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x147/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 KDB: enter: panic [ thread pid 11 tid 100004 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 11 tid 100004 td 0xbfe9380 kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 db:1:pho1> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x6134c0 curthread = 0xbfe9700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x1511ad40 fpcurthread = none idlethread = 0xbfe9700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x1511ad30 common_tssp = 0xffc01260 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xa0634c0 curthread = 0xbfe9380: pid 11 tid 100004 "idle: cpu1" curpcb = 0x1511fd40 fpcurthread = none idlethread = 0xbfe9380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x1511fd30 common_tssp = 0xffc012c8 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xa0684c0 curthread = 0xbfe9000: pid 11 tid 100005 "idle: cpu2" curpcb = 0x15124d40 fpcurthread = none idlethread = 0xbfe9000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x15124d30 common_tssp = 0xffc01330 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xa06d4c0 curthread = 0xbfe8a80: pid 11 tid 100006 "idle: cpu3" curpcb = 0x15129d40 fpcurthread = none idlethread = 0xbfe8a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x15129d30 common_tssp = 0xffc01398 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 9305 (find) thread 0x27a41a80 (100305) exclusive lockmgr bufwait (bufwait) r = 0 (0xbd0f004) locked @ kern/vfs_bio.c:1670 shared lockmgr ufs (ufs) r = 0 (0x26a4e478) locked @ kern/vfs_syscalls.c:3969 Process 9304 (find) thread 0x2162a700 (100316) shared lockmgr ufs (ufs) r = 0 (0x26a4e478) locked @ kern/vfs_syscalls.c:3969 Process 41759 (bash) thread 0x26c1d700 (100288) exclusive lockmgr bufwait (bufwait) r = 0 (0xbd4f468) locked @ kern/vfs_bio.c:1670 shared lockmgr ufs (ufs) r = 0 (0x20c64b94) locked @ kern/vfs_subr.c:2590 db:1:pho1> show lockedvnods Locked vnodes vnode 0x20c64b60: tag ufs, type VREG usecount 14, writecount 0, refcount 19 flags (VI_ACTIVE) v_object 0x208daaf8 ref 13 pages 16 cleanbuf 4 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0x10bcfba at lockmgr_slock_hard+0x2fa #1 0x10bd864 at __lockmgr_args+0x3b4 #2 0x142cc14 at ffs_lock+0x94 #3 0x171f400 at VOP_LOCK1_APV+0x90 #4 0x11d7961 at _vn_lock+0x61 #5 0x11c3b85 at vget+0xa5 #6 0x14524fc at vm_fault_hold+0x195c #7 0x1450b4e at vm_fault+0x5e #8 0x16e87fe at trap_pfault+0xfe #9 0x16e7e44 at trap+0x524 #10 0xffc0315d at PTDpde+0x4165 ino 94768, on dev ada0s1a vnode 0x26a4e444: tag ufs, type VDIR usecount 4, writecount 0, refcount 6 mountedhere 0 flags (VI_ACTIVE) v_object 0x2527b09c ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 2) #0 0x10bcfba at lockmgr_slock_hard+0x2fa #1 0x10bd864 at __lockmgr_args+0x3b4 #2 0x142cc14 at ffs_lock+0x94 #3 0x171f400 at VOP_LOCK1_APV+0x90 #4 0x11d7961 at _vn_lock+0x61 #5 0x11d49b7 at kern_getdirentries+0x177 #6 0x11d4cb6 at sys_getdirentries+0x46 #7 0x16e8ec3 at syscall+0x363 #8 0xffc033a7 at PTDpde+0x43af ino 1790555, on dev ada0s1e db:1:pho1> show allchains chain 1: thread 100294 (pid 9320, ping) sleeping on 0x28587e64 "select" chain 2: thread 100206 (pid 9317, ping) sleeping on 0x1b03aba4 "select" chain 3: thread 100321 (pid 9307, sh) sleeping on 0x2604b000 "wait" chain 4: thread 100305 (pid 9305, find) sleeping on 0xbd0efa8 "biord" chain 5: thread 100316 (pid 9304, find) blocked on lockmgr bufwaitEXCL thread 100305 (pid 9305, find) sleeping on 0xbd0efa8 "biord" chain 6: thread 100299 (pid 9303, hwpmc: proc(9300)) sleeping on 0x28d4c800 "pmcloop" chain 7: thread 100244 (pid 9302, hwpmc: proc(9299)) sleeping on 0x28d4c100 "pmcloop" chain 8: thread 100122 (pid 9300, pmcstat) sleeping on 0x25036a00 "kqread" chain 9: thread 100313 (pid 9299, pmcstat) sleeping on 0x23f99d00 "kqread" chain 10: thread 101127 (pid 9275, tee) sleeping on 0x24804000 "piperd" chain 11: thread 100251 (pid 9274, sh) sleeping on 0x256709fc "wait" chain 12: thread 100254 (pid 9273, sh) sleeping on 0x25670354 "wait" chain 13: thread 100208 (pid 9163, sleep) sleeping on 0x208740c "nanslp" chain 14: thread 100202 (pid 7573, sh) sleeping on 0x20d1d354 "wait" chain 15: thread 100288 (pid 41759, bash) sleeping on 0xbd4f40c "biord" chain 16: thread 100265 (pid 41758, sshd) sleeping on 0x214d90a4 "select" chain 17: thread 100310 (pid 41756, sshd) sleeping on 0x1b80cd24 "select" chain 18: thread 100257 (pid 1441, sh) sleeping on 0x220bcb98 "piperd" chain 19: thread 100201 (pid 1439, awk) sleeping on 0x1bb33000 "piperd" chain 20: thread 100252 (pid 1438, sh) sleeping on 0x256706a8 "wait" chain 21: thread 100223 (pid 1417, sh) sleeping on 0x246ef354 "wait" chain 22: thread 100205 (pid 1003, csh) sleeping on 0xbfe170c "pause" chain 23: thread 100210 (pid 1002, su) sleeping on 0x1fbbd354 "wait" chain 24: thread 100233 (pid 994, bash) sleeping on 0x20d1c354 "wait" chain 25: thread 100195 (pid 993, sshd) sleeping on 0xbe6f4e4 "select" chain 26: thread 100232 (pid 991, sshd) sleeping on 0xbe6f4a4 "select" chain 27: thread 100197 (pid 797, bash) sleeping on 0x1f33b324 "select" chain 28: thread 100174 (pid 790, csh) sleeping on 0x1b3e9a60 "pause" chain 29: thread 100192 (pid 775, login) sleeping on 0x1fba0354 "wait" chain 30: thread 100191 (pid 774, getty) sleeping on 0x1b56d874 "ttyin" chain 31: thread 100190 (pid 773, getty) sleeping on 0x1b56da74 "ttyin" chain 32: thread 100189 (pid 772, getty) sleeping on 0x1b56dc74 "ttyin" chain 33: thread 100188 (pid 771, getty) sleeping on 0x1b56de74 "ttyin" chain 34: thread 100187 (pid 770, getty) sleeping on 0x1b56e074 "ttyin" chain 35: thread 100186 (pid 769, getty) sleeping on 0x1b56e274 "ttyin" chain 36: thread 100182 (pid 768, getty) sleeping on 0x1b56e474 "ttyin" chain 37: thread 100183 (pid 767, getty) sleeping on 0x1b56e674 "ttyin" chain 38: thread 100121 (pid 750, inetd) sleeping on 0x1f33b3a4 "select" chain 39: thread 100118 (pid 716, nfscbd: master) sleeping on 0x1b597850 "rpcsvc" chain 40: thread 100177 (pid 716, nfscbd: service) sleeping on 0x1f687c50 "rpcsvc" chain 41: thread 100178 (pid 716, nfscbd: service) sleeping on 0x1f687bd0 "rpcsvc" chain 42: thread 100179 (pid 716, nfscbd: service) sleeping on 0x1f687b50 "rpcsvc" chain 43: thread 100110 (pid 715, nfscbd) sleeping on 0x1f7778e8 "accept" chain 44: thread 100171 (pid 682, cron) sleeping on 0x208740c "nanslp" chain 45: thread 100172 (pid 678, sendmail) sleeping on 0x1b54f064 "pause" chain 46: thread 100167 (pid 675, sendmail) sleeping on 0x1b43dba4 "select" chain 47: thread 100170 (pid 672, sshd) sleeping on 0x1f33b424 "select" chain 48: thread 100120 (pid 634, smartd) sleeping on 0x208740c "nanslp" chain 49: thread 100165 (pid 604, powerd) sleeping on 0x1b03ac64 "select" chain 50: thread 100164 (pid 601, ntpd) sleeping on 0xbe6f5e4 "select" chain 51: thread 100163 (pid 563, NLM: master) sleeping on 0x1acb2d50 "rpcsvc" chain 52: thread 100106 (pid 560, rpc.statd) sleeping on 0x1b43dbe4 "select" chain 53: thread 100131 (pid 559, nfsd: master) sleeping on 0x1f687ed0 "rpcsvc" chain 54: thread 100132 (pid 559, nfsd: service) sleeping on 0x1acb2250 "rpcsvc" chain 55: thread 100133 (pid 559, nfsd: service) sleeping on 0x1acb22d0 "rpcsvc" chain 56: thread 100134 (pid 559, nfsd: service) sleeping on 0x1acb21d0 "rpcsvc" chain 57: thread 100135 (pid 559, nfsd: service) sleeping on 0x1acb2350 "rpcsvc" chain 58: thread 100136 (pid 559, nfsd: service) sleeping on 0x1b597dd0 "rpcsvc" chain 59: thread 100137 (pid 559, nfsd: service) sleeping on 0x1acb23d0 "rpcsvc" chain 60: thread 100138 (pid 559, nfsd: service) sleeping on 0x1acb2450 "rpcsvc" chain 61: thread 100139 (pid 559, nfsd: service) sleeping on 0x1acb24d0 "rpcsvc" chain 62: thread 100140 (pid 559, nfsd: service) sleeping on 0x1b597d50 "rpcsvc" chain 63: thread 100141 (pid 559, nfsd: service) sleeping on 0x1acb2550 "rpcsvc" chain 64: thread 100142 (pid 559, nfsd: service) sleeping on 0x1b597cd0 "rpcsvc" chain 65: thread 100143 (pid 559, nfsd: service) sleeping on 0x1acb25d0 "rpcsvc" chain 66: thread 100144 (pid 559, nfsd: service) sleeping on 0x1b597c50 "rpcsvc" chain 67: thread 100145 (pid 559, nfsd: service) sleeping on 0x1acb2650 "rpcsvc" chain 68: thread 100146 (pid 559, nfsd: service) sleeping on 0x1acb26d0 "rpcsvc" chain 69: thread 100147 (pid 559, nfsd: service) sleeping on 0x1acb2750 "rpcsvc" chain 70: thread 100148 (pid 559, nfsd: service) sleeping on 0x1b597bd0 "rpcsvc" chain 71: thread 100149 (pid 559, nfsd: service) sleeping on 0x1acb27d0 "rpcsvc" chain 72: thread 100150 (pid 559, nfsd: service) sleeping on 0x1acb2850 "rpcsvc" chain 73: thread 100151 (pid 559, nfsd: service) sleeping on 0x1acb28d0 "rpcsvc" chain 74: thread 100152 (pid 559, nfsd: service) sleeping on 0x1b597b50 "rpcsvc" chain 75: thread 100153 (pid 559, nfsd: service) sleeping on 0x1acb2950 "rpcsvc" chain 76: thread 100154 (pid 559, nfsd: service) sleeping on 0x1acb29d0 "rpcsvc" chain 77: thread 100155 (pid 559, nfsd: service) sleeping on 0x1acb2a50 "rpcsvc" chain 78: thread 100156 (pid 559, nfsd: service) sleeping on 0x1acb2ad0 "rpcsvc" chain 79: thread 100157 (pid 559, nfsd: service) sleeping on 0x1acb2b50 "rpcsvc" chain 80: thread 100158 (pid 559, nfsd: service) sleeping on 0x1acb2bd0 "rpcsvc" chain 81: thread 100159 (pid 559, nfsd: service) sleeping on 0x1b597ad0 "rpcsvc" chain 82: thread 100160 (pid 559, nfsd: service) sleeping on 0x1b597a50 "rpcsvc" chain 83: thread 100161 (pid 559, nfsd: service) sleeping on 0x1acb2c50 "rpcsvc" chain 84: thread 100162 (pid 559, nfsd: service) sleeping on 0x1acb2cd0 "rpcsvc" chain 85: thread 100130 (pid 556, nfsd) sleeping on 0xbe6f6a4 "select" chain 86: thread 100128 (pid 550, mountd) sleeping on 0x1f33b5e4 "select" chain 87: thread 100109 (pid 544, nfsuserd) sleeping on 0x1b43dc24 "select" chain 88: thread 100117 (pid 542, nfsuserd) sleeping on 0x1f33b624 "select" chain 89: thread 100112 (pid 541, nfsuserd) sleeping on 0x1b43dc64 "select" chain 90: thread 100119 (pid 540, nfsuserd) sleeping on 0x1f33b664 "select" chain 91: thread 100127 (pid 539, nfsuserd) sleeping on 0x1babc3b8 "pause" chain 92: thread 100125 (pid 502, watchdogd) sleeping on 0x208740c "nanslp" chain 93: thread 100111 (pid 418, rpcbind) sleeping on 0x1b43dca4 "select" chain 94: thread 100123 (pid 409, syslogd) sleeping on 0x1b03ae24 "select" chain 95: thread 100124 (pid 337, devd) sleeping on 0xbe6f8a4 "select" chain 96: thread 100101 (pid 25, syncer) sleeping on 0x2101ba0 "syncer" chain 97: thread 100100 (pid 24, vnlru) sleeping on 0x1af9c000 "vlruwt" chain 98: thread 100099 (pid 23, bufdaemon) sleeping on 0x210152c "qsleep" chain 99: thread 100103 (pid 23, bufspacedaemon-0) sleeping on 0x1dcf380 "-" chain 100: thread 100113 (pid 23, /home worker) sleeping on 0x1b572084 "sdflush" chain 101: thread 100114 (pid 23, /tmp worker) sleeping on 0x1b571e84 "sdflush" chain 102: thread 100115 (pid 23, /usr worker) sleeping on 0x1b571c84 "sdflush" chain 103: thread 100116 (pid 23, /var worker) sleeping on 0x1b571a84 "sdflush" chain 104: thread 100098 (pid 22, vmdaemon) sleeping on 0x21170b8 "psleep" chain 105: thread 100097 (pid 21, dom0) sleeping on 0x213df5c "psleep" chain 106: thread 100102 (pid 21, laundry: dom0) sleeping on 0x213df68 "launds" chain 107: thread 100104 (pid 21, uma) sleeping on 0x144d400 "umarcl" chain 108: thread 100091 (pid 20, rand_harvestq) sleeping on 0x1eecb14 "-" chain 109: thread 100090 (pid 19, sctp_iterator) sleeping on 0x213d044 "waiting_for_work" chain 110: thread 100089 (pid 18, soaiod4) sleeping on 0x2101158 "-" chain 111: thread 100088 (pid 17, soaiod3) sleeping on 0x2101158 "-" chain 112: thread 100087 (pid 16, soaiod2) sleeping on 0x2101158 "-" chain 113: thread 100086 (pid 9, soaiod1) sleeping on 0x2101158 "-" chain 114: thread 100083 (pid 8, fdc0) sleeping on 0x1ac9d03c "-" chain 115: thread 100048 (pid 15, usbus0) sleeping on 0x1acd6acc "-" chain 116: thread 100049 (pid 15, usbus0) sleeping on 0x1acd6afc "-" chain 117: thread 100050 (pid 15, usbus0) sleeping on 0x1acd6b2c "-" chain 118: thread 100051 (pid 15, usbus0) sleeping on 0x1acd6b5c "-" chain 119: thread 100052 (pid 15, usbus0) sleeping on 0x1acd6b8c "-" chain 120: thread 100054 (pid 15, usbus1) sleeping on 0x1afa4acc "-" chain 121: thread 100055 (pid 15, usbus1) sleeping on 0x1afa4afc "-" chain 122: thread 100056 (pid 15, usbus1) sleeping on 0x1afa4b2c "-" chain 123: thread 100057 (pid 15, usbus1) sleeping on 0x1afa4b5c "-" chain 124: thread 100058 (pid 15, usbus1) sleeping on 0x1afa4b8c "-" chain 125: thread 100060 (pid 15, usbus2) sleeping on 0x1afb0acc "-" chain 126: thread 100061 (pid 15, usbus2) sleeping on 0x1afb0afc "-" chain 127: thread 100062 (pid 15, usbus2) sleeping on 0x1afb0b2c "-" chain 128: thread 100063 (pid 15, usbus2) sleeping on 0x1afb0b5c "-" chain 129: thread 100064 (pid 15, usbus2) sleeping on 0x1afb0b8c "-" chain 130: thread 100065 (pid 15, usbus3) sleeping on 0x1afb4acc "-" chain 131: thread 100066 (pid 15, usbus3) sleeping on 0x1afb4afc "-" chain 132: thread 100067 (pid 15, usbus3) sleeping on 0x1afb4b2c "-" chain 133: thread 100068 (pid 15, usbus3) sleeping on 0x1afb4b5c "-" chain 134: thread 100069 (pid 15, usbus3) sleeping on 0x1afb4b8c "-" chain 135: thread 100070 (pid 15, usbus4) sleeping on 0x1afbdacc "-" chain 136: thread 100071 (pid 15, usbus4) sleeping on 0x1afbdafc "-" chain 137: thread 100072 (pid 15, usbus4) sleeping on 0x1afbdb2c "-" chain 138: thread 100073 (pid 15, usbus4) sleeping on 0x1afbdb5c "-" chain 139: thread 100074 (pid 15, usbus4) sleeping on 0x1afbdb8c "-" chain 140: thread 100076 (pid 15, usbus5) sleeping on 0x1acc1ca4 "-" chain 141: thread 100077 (pid 15, usbus5) sleeping on 0x1acc1cd4 "-" chain 142: thread 100078 (pid 15, usbus5) sleeping on 0x1acc1d04 "-" chain 143: thread 100079 (pid 15, usbus5) sleeping on 0x1acc1d34 "-" chain 144: thread 100080 (pid 15, usbus5) sleeping on 0x1acc1d64 "-" chain 145: thread 100043 (pid 7, doneq0) sleeping on 0x1e39ec0 "-" chain 146: thread 100096 (pid 7, scanner) sleeping on 0x1e39dec "-" chain 147: thread 100042 (pid 14, sequencer 00) sleeping on 0x9f8984c "seqstate" chain 148: thread 100041 (pid 6, crypto returns 3) sleeping on 0x9f790c4 "crypto_ret_wait" chain 149: thread 100040 (pid 5, crypto returns 2) sleeping on 0x9f7908c "crypto_ret_wait" chain 150: thread 100039 (pid 4, crypto returns 1) sleeping on 0x9f79054 "crypto_ret_wait" chain 151: thread 100038 (pid 3, crypto returns 0) sleeping on 0x9f7901c "crypto_ret_wait" chain 152: thread 100037 (pid 2, crypto) sleeping on 0x2115ed8 "crypto_wait" chain 153: thread 100029 (pid 13, g_event) sleeping on 0x21393fc "-" chain 154: thread 100030 (pid 13, g_up) sleeping on 0x2139404 "-" chain 155: thread 100031 (pid 13, g_down) sleeping on 0x21393f8 "-" chain 156: thread 100002 (pid 1, init) sleeping on 0xbfe09fc "wait" chain 157: thread 100001 (pid 10, audit) sleeping on 0x213d81c "audit_worker_cv" chain 158: thread 100000 (pid 0, swapper) sleeping on 0x213941c "swapin" chain 159: thread 100007 (pid 0, kqueue_ctx taskq) sleeping on 0x9e18680 "-" chain 160: thread 100008 (pid 0, aiod_kick taskq) sleeping on 0x9e18600 "-" chain 161: thread 100011 (pid 0, thread taskq) sleeping on 0x9e18480 "-" chain 162: thread 100013 (pid 0, config_0) sleeping on 0x9e83000 "-" chain 163: thread 100014 (pid 0, if_config_tqg_0) sleeping on 0x9e82f00 "-" chain 164: thread 100015 (pid 0, softirq_0) sleeping on 0x9e82e80 "-" chain 165: thread 100016 (pid 0, softirq_1) sleeping on 0x9e82e00 "-" chain 166: thread 100017 (pid 0, softirq_2) sleeping on 0x9e82d80 "-" chain 167: thread 100018 (pid 0, softirq_3) sleeping on 0x9e82d00 "-" chain 168: thread 100019 (pid 0, if_io_tqg_0) sleeping on 0x9e82c80 "-" chain 169: thread 100020 (pid 0, if_io_tqg_1) sleeping on 0x9e82c00 "-" chain 170: thread 100021 (pid 0, if_io_tqg_2) sleeping on 0x9e82b80 "-" chain 171: thread 100022 (pid 0, if_io_tqg_3) sleeping on 0x9e82b00 "-" chain 172: thread 100032 (pid 0, firmware taskq) sleeping on 0x9e18300 "-" chain 173: thread 100033 (pid 0, crypto_0) sleeping on 0x9ea1f00 "-" chain 174: thread 100034 (pid 0, crypto_1) sleeping on 0x9ea1f00 "-" chain 175: thread 100035 (pid 0, crypto_2) sleeping on 0x9ea1f00 "-" chain 176: thread 100036 (pid 0, crypto_3) sleeping on 0x9ea1f00 "-" chain 177: thread 100085 (pid 0, mca taskq) sleeping on 0x1b294100 "-" chain 178: thread 100092 (pid 0, acpi_task_0) sleeping on 0x1b58f100 "-" chain 179: thread 100093 (pid 0, acpi_task_1) sleeping on 0x1b58f100 "-" chain 180: thread 100094 (pid 0, acpi_task_2) sleeping on 0x1b58f100 "-" chain 181: thread 100095 (pid 0, CAM taskq) sleeping on 0x9ea1c00 "-" db:1:pho1> show mount 0x1ba78000 /dev/ada0s1a on / (ufs) 0x1b9be000 devfs on /dev (devfs) 0x1bb6f000 /dev/ada0s1f on /home (ufs) 0x1bb6e000 /dev/label/tmp on /tmp (ufs) 0x1bb6d000 /dev/ada0s1d on /usr (ufs) 0x1bbff000 /dev/ada0s1e on /var (ufs) 0x1bbfe000 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 93 Buf domain 0 freebufs 383 lofreebufs 311 hifreebufs 466 bufspace 73222144 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 36 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5310 (72423424) dirtyq count 36 (550912) wakeup 0 lim 28 CPU 22, 0, 10, 16, Locked buffers: 2 space 18432 Total buffers: 5779 space 73222144 db:1:pho1> show page vm_cnt.v_free_count: 769111 vm_cnt.v_inactive_count: 8217 vm_cnt.v_active_count: 1800 vm_cnt.v_laundry_count: 48 vm_cnt.v_wire_count: 80589 vm_cnt.v_free_reserved: 1186 vm_cnt.v_free_min: 5487 vm_cnt.v_free_target: 18390 vm_cnt.v_inactive_target: 27585 db:1:pho1> show pageq pq_free 769111 dom 0 page_cnt 860434 free 769111 pq_act 1800 pq_inact 8217 pq_laund 48 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000002 08 (001024K) | 000083 07 (000512K) | 000869 06 (000256K) | 002875 05 (000128K) | 005177 04 (000064K) | 007160 03 (000032K) | 010645 02 (000016K) | 014906 01 (000008K) | 012005 00 (000004K) | 000001 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000002 04 (000064K) | 000002 03 (000032K) | 000003 02 (000016K) | 000004 01 (000008K) | 000003 00 (000004K) | 000002 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0xbfe9700 (CPU 0) cpustop_handler(0,1511aba0,1511ab94,16e79ec,2094640,...) at cpustop_handler+0x25/frame 0x1511aa80 ipi_nmi_handler(2094640,1acfda80,2094650,2094640,0,...) at ipi_nmi_handler+0x36/frame 0x1511aa90 trap(1511aba0,8,28,28,1ac50800,...) at trap+0xcc/frame 0x1511ab94 calltrap() at PTDpde+0x4165/frame 0x1511ab94 --- trap 0x13, eip = 0x1700155, esp = 0x1511abe0, ebp = 0x1511abe0 --- acpi_cpu_c1(1511ac34,16f0412,2428d90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x1511abe0 acpi_cpu_idle(125b84bf,0,125b84bf,1511ac50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x1511ac20 cpu_idle_acpi(125b84bf,0,2094640,2,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511ac34 cpu_idle(0,1511ac80,17ddfd6,adb,2,...) at cpu_idle+0xab/frame 0x1511ac50 sched_idletd(0,1511ace8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511acac fork_exit(111c240,0,1511ace8,0,0,...) at fork_exit+0x7e/frame 0x1511acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511acd4 --- trap 0, eip = 0, esp = 0x1511ad20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfe9380 (CPU 1) kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfe9000 (CPU 2) cpustop_handler(2,15124ba0,15124b94,16e79ec,213aa50,...) at cpustop_handler+0x25/frame 0x15124a80 ipi_nmi_handler(213aa50,bfe9000,15124ad0,10c85c0,213aa40,...) at ipi_nmi_handler+0x36/frame 0x15124a90 trap(15124ba0,8,28,28,1ac50400,...) at trap+0xcc/frame 0x15124b94 calltrap() at PTDpde+0x4165/frame 0x15124b94 --- trap 0x13, eip = 0x1700155, esp = 0x15124be0, ebp = 0x15124be0 --- acpi_cpu_c1(15124c34,16f0412,be7dd90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x15124be0 acpi_cpu_idle(a1727d4,0,a1727d4,15124c50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x15124c20 cpu_idle_acpi(a1727d4,0,20953c0,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15124c34 cpu_idle(0,15124c80,17ddfd6,adb,3,...) at cpu_idle+0xab/frame 0x15124c50 sched_idletd(0,15124ce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x15124cac fork_exit(111c240,0,15124ce8,0,0,...) at fork_exit+0x7e/frame 0x15124cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15124cd4 --- trap 0, eip = 0, esp = 0x15124d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfe8a80 (CPU 3) cpustop_handler(3,15129ba0,15129b94,16e79ec,214aa0bc,...) at cpustop_handler+0x25/frame 0x15129a80 ipi_nmi_handler(214aa0bc,415,15129ae8,23011830,1aa97f20,...) at ipi_nmi_handler+0x36/frame 0x15129a90 trap(15129ba0,8,28,28,1ac50200,...) at trap+0xcc/frame 0x15129b94 calltrap() at PTDpde+0x4165/frame 0x15129b94 --- trap 0x13, eip = 0x1700155, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,16f0412,be82d90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(361a971c,0,361a971c,15129c50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x15129c20 cpu_idle_acpi(361a971c,0,2095a80,4,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17ddfd6,adb,4,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(111c240,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 9320 1 9320 0 Ss select 0x28587e64 ping 9317 9307 7573 0 S+ select 0x1b03aba4 ping 9307 9274 7573 0 S+ wait 0x2604b000 sh 9305 9300 7573 0 D+ biord 0xbd0efa8 find 9304 9299 7573 0 D+ getblk 0xbd0f004 find 9303 0 0 0 DL pmcloop 0x28d4c800 [hwpmc: proc(9300)] 9302 0 0 0 DL pmcloop 0x28d4c100 [hwpmc: proc(9299)] 9300 9274 7573 0 S+ kqread 0x25036a00 pmcstat 9299 9274 7573 0 S+ kqread 0x23f99d00 pmcstat 9275 7573 7573 0 S+ piperd 0x24804000 tee 9274 9273 7573 0 S+ wait 0x256709fc sh 9273 7573 7573 0 S+ wait 0x25670354 sh 9163 1438 1417 0 S nanslp 0x208740c sleep 7573 1003 7573 0 S+ wait 0x20d1d354 sh 41759 41758 41759 1001 Ds+ biord 0xbd4f40c bash 41758 41756 41756 1001 S select 0x214d90a4 sshd 41756 672 41756 0 Ss select 0x1b80cd24 sshd 1441 1417 1417 0 S piperd 0x220bcb98 sh 1439 1417 1417 0 S piperd 0x1bb33000 awk 1438 1417 1417 0 S wait 0x256706a8 sh 1417 1 1417 0 SWs wait 0x246ef354 sh 1003 1002 1003 0 S+ pause 0xbfe170c csh 1002 994 1002 1001 SW+ wait 0x1fbbd354 su 994 993 994 1001 SWs+ wait 0x20d1c354 bash 993 991 991 1001 S select 0xbe6f4e4 sshd 991 672 991 0 Ss select 0xbe6f4a4 sshd 797 790 797 0 S+ select 0x1f33b324 bash 790 775 790 0 SW+ pause 0x1b3e9a60 csh 775 1 775 0 SWs+ wait 0x1fba0354 login 774 1 774 0 Ss+ ttyin 0x1b56d874 getty 773 1 773 0 Ss+ ttyin 0x1b56da74 getty 772 1 772 0 Ss+ ttyin 0x1b56dc74 getty 771 1 771 0 Ss+ ttyin 0x1b56de74 getty 770 1 770 0 Ss+ ttyin 0x1b56e074 getty 769 1 769 0 Ss+ ttyin 0x1b56e274 getty 768 1 768 0 Ss+ ttyin 0x1b56e474 getty 767 1 767 0 Ss+ ttyin 0x1b56e674 getty 750 1 750 0 Ss select 0x1f33b3a4 inetd 716 715 715 0 S (threaded) nfscbd 100118 S rpcsvc 0x1b597850 nfscbd: master 100177 S rpcsvc 0x1f687c50 nfscbd: service 100178 S rpcsvc 0x1f687bd0 nfscbd: service 100179 S rpcsvc 0x1f687b50 nfscbd: service 715 1 715 0 SWs accept 0x1f7778e8 nfscbd 682 1 682 0 Ss nanslp 0x208740c cron 678 1 678 25 SWs pause 0x1b54f064 sendmail 675 1 675 0 Ss select 0x1b43dba4 sendmail 672 1 672 0 Ss select 0x1f33b424 sshd 634 1 633 0 SW nanslp 0x208740c smartd 604 1 604 0 Ss select 0x1b03ac64 powerd 601 1 601 0 Ss (threaded) ntpd 100164 S select 0xbe6f5e4 ntpd 563 1 563 0 Ss rpcsvc 0x1acb2d50 NLM: master 560 1 560 0 Ss select 0x1b43dbe4 rpc.statd 559 556 556 0 S (threaded) nfsd 100131 S rpcsvc 0x1f687ed0 nfsd: master 100132 S rpcsvc 0x1acb2250 nfsd: service 100133 S rpcsvc 0x1acb22d0 nfsd: service 100134 S rpcsvc 0x1acb21d0 nfsd: service 100135 S rpcsvc 0x1acb2350 nfsd: service 100136 S rpcsvc 0x1b597dd0 nfsd: service 100137 S rpcsvc 0x1acb23d0 nfsd: service 100138 S rpcsvc 0x1acb2450 nfsd: service 100139 S rpcsvc 0x1acb24d0 nfsd: service 100140 S rpcsvc 0x1b597d50 nfsd: service 100141 S rpcsvc 0x1acb2550 nfsd: service 100142 S rpcsvc 0x1b597cd0 nfsd: service 100143 S rpcsvc 0x1acb25d0 nfsd: service 100144 S rpcsvc 0x1b597c50 nfsd: service 100145 S rpcsvc 0x1acb2650 nfsd: service 100146 S rpcsvc 0x1acb26d0 nfsd: service 100147 S rpcsvc 0x1acb2750 nfsd: service 100148 S rpcsvc 0x1b597bd0 nfsd: service 100149 S rpcsvc 0x1acb27d0 nfsd: service 100150 S rpcsvc 0x1acb2850 nfsd: service 100151 S rpcsvc 0x1acb28d0 nfsd: service 100152 S rpcsvc 0x1b597b50 nfsd: service 100153 S rpcsvc 0x1acb2950 nfsd: service 100154 S rpcsvc 0x1acb29d0 nfsd: service 100155 S rpcsvc 0x1acb2a50 nfsd: service 100156 S rpcsvc 0x1acb2ad0 nfsd: service 100157 S rpcsvc 0x1acb2b50 nfsd: service 100158 S rpcsvc 0x1acb2bd0 nfsd: service 100159 S rpcsvc 0x1b597ad0 nfsd: service 100160 S rpcsvc 0x1b597a50 nfsd: service 100161 S rpcsvc 0x1acb2c50 nfsd: service 100162 S rpcsvc 0x1acb2cd0 nfsd: service 556 1 556 0 Ss select 0xbe6f6a4 nfsd 550 1 550 0 Ss select 0x1f33b5e4 mountd 544 539 539 0 S select 0x1b43dc24 nfsuserd 542 539 539 0 S select 0x1f33b624 nfsuserd 541 539 539 0 S select 0x1b43dc64 nfsuserd 540 539 539 0 S select 0x1f33b664 nfsuserd 539 1 539 0 SWs pause 0x1babc3b8 nfsuserd 502 1 502 0 Ss nanslp 0x208740c watchdogd 418 1 418 0 Ss select 0x1b43dca4 rpcbind 409 1 409 0 Ss select 0x1b03ae24 syslogd 337 1 337 0 Ss select 0xbe6f8a4 devd 25 0 0 0 DL syncer 0x2101ba0 [syncer] 24 0 0 0 DL vlruwt 0x1af9c000 [vnlru] 23 0 0 0 DL (threaded) [bufdaemon] 100099 D qsleep 0x210152c [bufdaemon] 100103 D - 0x1dcf380 [bufspacedaemon-0] 100113 D sdflush 0x1b572084 [/home worker] 100114 D sdflush 0x1b571e84 [/tmp worker] 100115 D sdflush 0x1b571c84 [/usr worker] 100116 D sdflush 0x1b571a84 [/var worker] 22 0 0 0 DL psleep 0x21170b8 [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100097 D psleep 0x213df5c [dom0] 100102 D launds 0x213df68 [laundry: dom0] 100104 D umarcl 0x144d400 [uma] 20 0 0 0 DL - 0x1eecb14 [rand_harvestq] 19 0 0 0 DL waiting 0x213d044 [sctp_iterator] 18 0 0 0 DL - 0x2101158 [soaiod4] 17 0 0 0 DL - 0x2101158 [soaiod3] 16 0 0 0 DL - 0x2101158 [soaiod2] 9 0 0 0 DL - 0x2101158 [soaiod1] 8 0 0 0 DL - 0x1ac9d03c [fdc0] 15 0 0 0 DL (threaded) [usb] 100048 D - 0x1acd6acc [usbus0] 100049 D - 0x1acd6afc [usbus0] 100050 D - 0x1acd6b2c [usbus0] 100051 D - 0x1acd6b5c [usbus0] 100052 D - 0x1acd6b8c [usbus0] 100054 D - 0x1afa4acc [usbus1] 100055 D - 0x1afa4afc [usbus1] 100056 D - 0x1afa4b2c [usbus1] 100057 D - 0x1afa4b5c [usbus1] 100058 D - 0x1afa4b8c [usbus1] 100060 D - 0x1afb0acc [usbus2] 100061 D - 0x1afb0afc [usbus2] 100062 D - 0x1afb0b2c [usbus2] 100063 D - 0x1afb0b5c [usbus2] 100064 D - 0x1afb0b8c [usbus2] 100065 D - 0x1afb4acc [usbus3] 100066 D - 0x1afb4afc [usbus3] 100067 D - 0x1afb4b2c [usbus3] 100068 D - 0x1afb4b5c [usbus3] 100069 D - 0x1afb4b8c [usbus3] 100070 D - 0x1afbdacc [usbus4] 100071 D - 0x1afbdafc [usbus4] 100072 D - 0x1afbdb2c [usbus4] 100073 D - 0x1afbdb5c [usbus4] 100074 D - 0x1afbdb8c [usbus4] 100076 D - 0x1acc1ca4 [usbus5] 100077 D - 0x1acc1cd4 [usbus5] 100078 D - 0x1acc1d04 [usbus5] 100079 D - 0x1acc1d34 [usbus5] 100080 D - 0x1acc1d64 [usbus5] 7 0 0 0 DL (threaded) [cam] 100043 D - 0x1e39ec0 [doneq0] 100096 D - 0x1e39dec [scanner] 14 0 0 0 DL seqstat 0x9f8984c [sequencer 00] 6 0 0 0 DL crypto_ 0x9f790c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x9f7908c [crypto returns 2] 4 0 0 0 DL crypto_ 0x9f79054 [crypto returns 1] 3 0 0 0 DL crypto_ 0x9f7901c [crypto returns 0] 2 0 0 0 DL crypto_ 0x2115ed8 [crypto] 13 0 0 0 DL (threaded) [geom] 100029 D - 0x21393fc [g_event] 100030 D - 0x2139404 [g_up] 100031 D - 0x21393f8 [g_down] 12 0 0 0 RL (threaded) [intr] 100009 I [swi6: task queue] 100010 I [swi6: Giant taskq] 100012 I [swi5: fast taskq] 100023 I [swi4: clock (0)] 100024 I [swi4: clock (1)] 100025 I [swi4: clock (2)] 100026 I [swi4: clock (3)] 100027 I [swi3: vm] 100028 I [swi1: netisr 0] 100044 I [irq256: hdac0] 100045 I [irq257: re0] 100046 RunQ [irq22: ahci0] 100047 I [irq16: hdac1 ohci0] 100053 I [irq17: ohci1 ohci3] 100059 I [irq18: ohci2 ohci4] 100075 I [irq19: ehci0] 100081 I [irq14: ata0] 100082 I [swi0: uart] 100084 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xbfe09fc [init] 10 0 0 0 DL audit_w 0x213d81c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x213941c [swapper] 100007 D - 0x9e18680 [kqueue_ctx taskq] 100008 D - 0x9e18600 [aiod_kick taskq] 100011 D - 0x9e18480 [thread taskq] 100013 D - 0x9e83000 [config_0] 100014 D - 0x9e82f00 [if_config_tqg_0] 100015 D - 0x9e82e80 [softirq_0] 100016 D - 0x9e82e00 [softirq_1] 100017 D - 0x9e82d80 [softirq_2] 100018 D - 0x9e82d00 [softirq_3] 100019 D - 0x9e82c80 [if_io_tqg_0] 100020 D - 0x9e82c00 [if_io_tqg_1] 100021 D - 0x9e82b80 [if_io_tqg_2] 100022 D - 0x9e82b00 [if_io_tqg_3] 100032 D - 0x9e18300 [firmware taskq] 100033 D - 0x9ea1f00 [crypto_0] 100034 D - 0x9ea1f00 [crypto_1] 100035 D - 0x9ea1f00 [crypto_2] 100036 D - 0x9ea1f00 [crypto_3] 100085 D - 0x1b294100 [mca taskq] 100092 D - 0x1b58f100 [acpi_task_0] 100093 D - 0x1b58f100 [acpi_task_1] 100094 D - 0x1b58f100 [acpi_task_2] 100095 D - 0x9ea1c00 [CAM taskq] db:0:pho> allt Tracing command ping pid 9320 tid 100294 td 0x1fbf1380 sched_switch(1fbf1380,0,104,18a,0,...) at sched_switch+0x671/frame 0x153c78c0 mi_switch(104,0,1817fc1,22b,1fbf1380,...) at mi_switch+0x20e/frame 0x153c7900 sleepq_switch(1fbf1380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x153c7934 sleepq_catch_signals(153c799c,10c7a20,28587e60,0,28587e64,...) at sleepq_catch_signals+0x47a/frame 0x153c7978 sleepq_wait_sig(28587e64,0,153c79bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x153c799c _cv_wait_sig(28587e64,28587e50,1733ba7,6fa,256ab208,...) at _cv_wait_sig+0x1d7/frame 0x153c79e0 seltdwait(ffffffff,0,0,1fbf1380,0,...) at seltdwait+0xbb/frame 0x153c7a10 kern_select(1fbf1380,9,ffbee1a8,0,0,...) at kern_select+0x894/frame 0x153c7ba0 sys_select(1fbf1380,1fbf1604,18047f7,4,285a40b8,...) at sys_select+0x68/frame 0x153c7be8 syscall(153c7ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x153c7cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x153c7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20622fdf, esp = 0xffbee164, ebp = 0xffbee188 --- amdsmn_devclass() at 0x20622fdf Tracing command ping pid 9317 tid 100206 td 0x246bc380 sched_switch(246bc380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dba78a0 mi_switch(104,0,1817fc1,22b,246bc380,...) at mi_switch+0x20e/frame 0x1dba78e0 sleepq_switch(246bc380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1dba7914 sleepq_catch_signals(8,17cad96,cb,0,246bc380,...) at sleepq_catch_signals+0x47a/frame 0x1dba7958 sleepq_timedwait_sig(1b03aba4,0,1dba79bc,ffff8a8,0,...) at sleepq_timedwait_sig+0x17/frame 0x1dba7998 _cv_timedwait_sig_sbt(1b03aba4,1b03ab90,a003f5c2,9f4a,ffff8a8,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1dba79e0 seltdwait(9f4a,ffff8a8,0,246bc380,0,...) at seltdwait+0x82/frame 0x1dba7a10 kern_select(246bc380,5,ffbee580,0,0,...) at kern_select+0x894/frame 0x1dba7ba0 sys_select(246bc380,246bc604,18047f7,4,be82d90,...) at sys_select+0x68/frame 0x1dba7be8 syscall(1dba7ce8,3b,3b,3b,ffbee468,...) at syscall+0x363/frame 0x1dba7cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dba7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20622fdf, esp = 0xffbee3b4, ebp = 0xffbee3d8 --- amdsmn_devclass() at 0x20622fdf Tracing command sh pid 9307 tid 100321 td 0x1fb9ba80 sched_switch(1fb9ba80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0xb8f4948 mi_switch(104,0,1817fc1,22b,1fb9ba80,...) at mi_switch+0x20e/frame 0xb8f4984 sleepq_switch(1fb9ba80,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0xb8f49b8 sleepq_catch_signals(2095a90,0,1817fc1,178,1fb9ba80,...) at sleepq_catch_signals+0x47a/frame 0xb8f49fc sleepq_wait_sig(2604b000,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0xb8f4a20 _sleep(2604b000,2604b0a8,36c,1833c92,0,...) at _sleep+0x35a/frame 0xb8f4a70 kern_wait6(1fb9ba80,7,0,0,b8f4af4,...) at kern_wait6+0x551/frame 0xb8f4ac8 sys_wait4(1fb9ba80,1fb9bd04,18047f7,4,0,...) at sys_wait4+0x87/frame 0xb8f4be8 syscall(b8f4ce8,3b,3b,3b,ffbfdc24,...) at syscall+0x363/frame 0xb8f4cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb8f4cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x205b7383, esp = 0xffbfdbd8, ebp = 0xffbfdbf0 --- amdsmn_devclass() at 0x205b7383 Tracing command find pid 9305 tid 100305 td 0x27a41a80 sched_switch(27a41a80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x153ff7c0 mi_switch(104,0,1817fc1,22b,27a41a80,...) at mi_switch+0x20e/frame 0x153ff7fc sleepq_switch(27a41a80,0,1817fc1,2b3,27a41a80,...) at sleepq_switch+0x110/frame 0x153ff830 sleepq_wait(bd0efa8,5c,1762eeb,0,0,...) at sleepq_wait+0x48/frame 0x153ff854 _sleep(bd0efa8,9dd84c0,5c,1762eeb,0,...) at _sleep+0x36a/frame 0x153ff8a4 bufwait(bd0efa8,0,0,0,0,...) at bufwait+0xa4/frame 0x153ff8e0 breadn_flags(26a4e444,0,0,800,0,...) at breadn_flags+0x1e0/frame 0x153ff920 ffs_blkatoff(26a4e444,0,0,0,153ff9a8,...) at ffs_blkatoff+0xda/frame 0x153ff968 ufs_readdir(153ffb40,17f558d,17b8473,f81,17d4638,...) at ufs_readdir+0x163/frame 0x153ffae4 VOP_READDIR_APV(1dfa6f0,153ffb40,17b8473,f81,0,...) at VOP_READDIR_APV+0xac/frame 0x153ffb10 kern_getdirentries(27a41a80,6,2062d000,1000,153ffbc8,...) at kern_getdirentries+0x207/frame 0x153ffba0 sys_getdirentries(27a41a80,27a41d04,18047f7,4,a06d4c0,...) at sys_getdirentries+0x46/frame 0x153ffbe8 syscall(153ffce8,3b,3b,3b,20616114,...) at syscall+0x363/frame 0x153ffcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x153ffcdc --- syscall (554, FreeBSD ELF32, sys_getdirentries), eip = 0x205ceb4f, esp = 0xffbfe6bc, ebp = 0xffbfe6fc --- amdsmn_devclass() at 0x205ceb4f Tracing command find pid 9304 tid 100316 td 0x2162a700 sched_switch(2162a700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0xb8cd6a8 mi_switch(104,0,1817fc1,22b,2162a700,...) at mi_switch+0x20e/frame 0xb8cd6e4 sleepq_switch(2162a700,0,1817fc1,2b3,bd0f004,...) at sleepq_switch+0x110/frame 0xb8cd718 sleepq_wait(bd0f004,60,1852017,4,0,...) at sleepq_wait+0x48/frame 0xb8cd73c sleeplk(26a4e504,1852017,60,0,0,...) at sleeplk+0x1a0/frame 0xb8cd780 lockmgr_xlock_hard(26a4e504,1733fd9,f4c,b8cd820,26a4e478,...) at lockmgr_xlock_hard+0x21a/frame 0xb8cd7d8 __lockmgr_args(bd0f004,81900,26a4e504,1852017,60,...) at __lockmgr_args+0x37f/frame 0xb8cd850 getblkx(26a4e444,0,0,800,0,...) at getblkx+0x132/frame 0xb8cd8e0 breadn_flags(26a4e444,0,0,800,0,...) at breadn_flags+0x4e/frame 0xb8cd920 ffs_blkatoff(26a4e444,0,0,0,b8cd9a8,...) at ffs_blkatoff+0xda/frame 0xb8cd968 ufs_readdir(b8cdb40,17f558d,17b8473,f81,0,...) at ufs_readdir+0x163/frame 0xb8cdae4 VOP_READDIR_APV(1dfa6f0,b8cdb40,17b8473,f81,0,...) at VOP_READDIR_APV+0xac/frame 0xb8cdb10 kern_getdirentries(2162a700,6,2062d000,1000,b8cdbc8,...) at kern_getdirentries+0x207/frame 0xb8cdba0 sys_getdirentries(2162a700,2162a984,18047f7,4,a0634c0,...) at sys_getdirentries+0x46/frame 0xb8cdbe8 syscall(b8cdce8,3b,3b,3b,20616114,...) at syscall+0x363/frame 0xb8cdcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb8cdcdc --- syscall (554, FreeBSD ELF32, sys_getdirentries), eip = 0x205ceb4f, esp = 0xffbfe6bc, ebp = 0xffbfe6fc --- amdsmn_devclass() at 0x205ceb4f Tracing command hwpmc: proc(9300) pid 9303 tid 100299 td 0x1fb9ca80 sched_switch(1fb9ca80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x153e1b10 mi_switch(104,0,1817fc1,22b,1fb9ca80,...) at mi_switch+0x20e/frame 0x153e1b4c sleepq_switch(1fb9ca80,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x153e1b80 sleepq_timedwait(28d4c800,6c,0,0,0,...) at sleepq_timedwait+0x52/frame 0x153e1bc0 _sleep(28d4c800,2302011c,6c,23002d9e,3fffffb6,...) at _sleep+0x304/frame 0x153e1c10 pmclog_loop(1b7e2380,153e1ce8,17d387d,419,0,...) at pmclog_loop+0x471/frame 0x153e1cac fork_exit(23015630,1b7e2380,153e1ce8,0,0,...) at fork_exit+0x7e/frame 0x153e1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x153e1cd4 --- trap 0, eip = 0, esp = 0x153e1d20, ebp = 0 --- (null)() at 0 Tracing command hwpmc: proc(9299) pid 9302 tid 100244 td 0x246e6a80 sched_switch(246e6a80,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x151f4b10 mi_switch(104,0,1817fc1,22b,246e6a80,...) at mi_switch+0x20e/frame 0x151f4b4c sleepq_switch(246e6a80,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x151f4b80 sleepq_timedwait(28d4c100,6c,0,0,0,...) at sleepq_timedwait+0x52/frame 0x151f4bc0 _sleep(28d4c100,2302011c,6c,23002d9e,3fffffb6,...) at _sleep+0x304/frame 0x151f4c10 pmclog_loop(1ac10410,151f4ce8,17d387d,419,0,...) at pmclog_loop+0x471/frame 0x151f4cac fork_exit(23015630,1ac10410,151f4ce8,0,0,...) at fork_exit+0x7e/frame 0x151f4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x151f4cd4 --- trap 0, eip = 0, esp = 0x151f4d20, ebp = 0 --- (null)() at 0 Tracing command pmcstat pid 9300 tid 100122 td 0x1b402700 sched_switch(1b402700,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1ef11768 mi_switch(104,0,1817fc1,22b,1b402700,...) at mi_switch+0x20e/frame 0x1ef117a4 sleepq_switch(1b402700,0,1817fc1,1ca,68,...) at sleepq_switch+0x110/frame 0x1ef117d8 sleepq_catch_signals(2094650,0,1817fc1,178,1b402700,...) at sleepq_catch_signals+0x47a/frame 0x1ef1181c sleepq_wait_sig(25036a00,68,17e78ed,100,0,...) at sleepq_wait_sig+0x14/frame 0x1ef11840 _sleep(25036a00,25036a00,168,17e78ed,0,...) at _sleep+0x35a/frame 0x1ef11890 kqueue_kevent(0,1,1ef11bc0,0,0,...) at kqueue_kevent+0x3aa/frame 0x1ef11af4 kern_kevent_fp(1b402700,28092620,0,1,1ef11bc0,...) at kern_kevent_fp+0x9d/frame 0x1ef11b18 kern_kevent(1b402700,3,0,1,1ef11bc0,...) at kern_kevent+0xe8/frame 0x1ef11b68 kern_kevent_generic(1ef11bc0,1833c25,1b402984,1099f10,1099f60,...) at kern_kevent_generic+0x7c/frame 0x1ef11bb0 sys_kevent(1b402700,1b402984,18047f7,4,1,...) at sys_kevent+0x6c/frame 0x1ef11be8 syscall(1ef11ce8,3b,3b,3b,1,...) at syscall+0x363/frame 0x1ef11cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef11cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0x20961743, esp = 0xffbfe160, ebp = 0xffbfe188 --- amdsmn_devclass() at 0x20961743 Tracing command pmcstat pid 9299 tid 100313 td 0x27ff8a80 sched_switch(27ff8a80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0xb881768 mi_switch(104,0,1817fc1,22b,27ff8a80,...) at mi_switch+0x20e/frame 0xb8817a4 sleepq_switch(27ff8a80,0,1817fc1,1ca,68,...) at sleepq_switch+0x110/frame 0xb8817d8 sleepq_catch_signals(2095a90,0,1817fc1,178,27ff8a80,...) at sleepq_catch_signals+0x47a/frame 0xb88181c sleepq_wait_sig(23f99d00,68,17e78ed,100,0,...) at sleepq_wait_sig+0x14/frame 0xb881840 _sleep(23f99d00,23f99d00,168,17e78ed,0,...) at _sleep+0x35a/frame 0xb881890 kqueue_kevent(0,1,b881bc0,0,0,...) at kqueue_kevent+0x3aa/frame 0xb881af4 kern_kevent_fp(27ff8a80,1f4a2268,0,1,b881bc0,...) at kern_kevent_fp+0x9d/frame 0xb881b18 kern_kevent(27ff8a80,3,0,1,b881bc0,...) at kern_kevent+0xe8/frame 0xb881b68 kern_kevent_generic(b881bc0,1833c25,27ff8d04,1099f10,1099f60,...) at kern_kevent_generic+0x7c/frame 0xb881bb0 sys_kevent(27ff8a80,27ff8d04,18047f7,4,be82d90,...) at sys_kevent+0x6c/frame 0xb881be8 syscall(b881ce8,3b,3b,3b,1,...) at syscall+0x363/frame 0xb881cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb881cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0x20961743, esp = 0xffbfe160, ebp = 0xffbfe188 --- amdsmn_devclass() at 0x20961743 Tracing command tee pid 9275 tid 101127 td 0x263aba80 sched_switch(263aba80,0,104,18a,0,...) at sched_switch+0x671/frame 0xb96c9d0 mi_switch(104,0,1817fc1,22b,263aba80,...) at mi_switch+0x20e/frame 0xb96ca10 sleepq_switch(263aba80,0,1817fc1,1ca,5c,...) at sleepq_switch+0x110/frame 0xb96ca44 sleepq_catch_signals(2094650,0,1817fc1,178,263aba80,...) at sleepq_catch_signals+0x47a/frame 0xb96ca88 sleepq_wait_sig(24804000,5c,181817a,100,0,...) at sleepq_wait_sig+0x14/frame 0xb96caac _sleep(24804000,24804190,15c,181817a,0,...) at _sleep+0x35a/frame 0xb96cafc pipe_read(1f4a2d90,b96cbb8,21595b00,0,263aba80,...) at pipe_read+0x47b/frame 0xb96cb3c dofileread(1f4a2d90,b96cbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0xb96cb6c kern_readv(263aba80,0,b96cbb8,246d10a8,2060d000,...) at kern_readv+0x5e/frame 0xb96cb98 sys_read(263aba80,263abd04,18047f7,4,246d10b8,...) at sys_read+0x54/frame 0xb96cbe8 syscall(b96cce8,3b,3b,3b,ffbfe88c,...) at syscall+0x363/frame 0xb96ccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb96ccdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205d6fef, esp = 0xffbfe7c4, ebp = 0xffbfe7d8 --- amdsmn_devclass() at 0x205d6fef Tracing command sh pid 9274 tid 100251 td 0x25685380 sched_switch(25685380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1521d948 mi_switch(104,0,1817fc1,22b,25685380,...) at mi_switch+0x20e/frame 0x1521d984 sleepq_switch(25685380,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1521d9b8 sleepq_catch_signals(2094d10,0,1817fc1,178,25685380,...) at sleepq_catch_signals+0x47a/frame 0x1521d9fc sleepq_wait_sig(256709fc,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0x1521da20 _sleep(256709fc,25670aa4,36c,1833c92,0,...) at _sleep+0x35a/frame 0x1521da70 kern_wait6(25685380,7,0,0,1521daf4,...) at kern_wait6+0x551/frame 0x1521dac8 sys_wait4(25685380,25685604,18047f7,4,0,...) at sys_wait4+0x87/frame 0x1521dbe8 syscall(1521dce8,3b,3b,3b,ffbfe6b4,...) at syscall+0x363/frame 0x1521dcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1521dcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x205b7383, esp = 0xffbfe668, ebp = 0xffbfe680 --- amdsmn_devclass() at 0x205b7383 Tracing command sh pid 9273 tid 100254 td 0x25684a80 sched_switch(25684a80,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1522d948 mi_switch(104,0,1817fc1,22b,25684a80,...) at mi_switch+0x20e/frame 0x1522d984 sleepq_switch(25684a80,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1522d9b8 sleepq_catch_signals(2094d10,0,1817fc1,178,25684a80,...) at sleepq_catch_signals+0x47a/frame 0x1522d9fc sleepq_wait_sig(25670354,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0x1522da20 _sleep(25670354,256703fc,36c,1833c92,0,...) at _sleep+0x35a/frame 0x1522da70 kern_wait6(25684a80,7,0,0,1522daf4,...) at kern_wait6+0x551/frame 0x1522dac8 sys_wait4(25684a80,25684d04,18047f7,4,2567040c,...) at sys_wait4+0x87/frame 0x1522dbe8 syscall(1522dce8,3b,3b,3b,ffbfe57c,...) at syscall+0x363/frame 0x1522dcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1522dcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x205b7383, esp = 0xffbfe530, ebp = 0xffbfe548 --- amdsmn_devclass() at 0x205b7383 Tracing command sleep pid 9163 tid 100208 td 0x246c6700 sched_switch(246c6700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dbb69e8 mi_switch(104,0,1817fc1,22b,246c6700,...) at mi_switch+0x20e/frame 0x1dbb6a28 sleepq_switch(246c6700,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1dbb6a5c sleepq_catch_signals(246c6700,0,601,0,9f4e,...) at sleepq_catch_signals+0x47a/frame 0x1dbb6aa0 sleepq_timedwait_sig(208740c,6c,9f4e,a0000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x1dbb6ae0 _sleep(208740c,0,16c,182aa35,8cd19bf4,...) at _sleep+0x2ed/frame 0x1dbb6b30 kern_clock_nanosleep(246c6700,0,0,1dbb6bc8,1dbb6bc0,...) at kern_clock_nanosleep+0x1f4/frame 0x1dbb6ba0 sys_nanosleep(246c6700,246c6984,18047f7,4,2428d90,...) at sys_nanosleep+0x7f/frame 0x1dbb6be8 syscall(1dbb6ce8,3b,3b,3b,ffbfe8a0,...) at syscall+0x363/frame 0x1dbb6cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dbb6cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x20568443, esp = 0xffbfe860, ebp = 0xffbfe870 --- amdsmn_devclass() at 0x20568443 Tracing command sh pid 7573 tid 100202 td 0x1fbee380 sched_switch(1fbee380,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1db91948 mi_switch(104,0,1817fc1,22b,1fbee380,...) at mi_switch+0x20e/frame 0x1db91984 sleepq_switch(1fbee380,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1db919b8 sleepq_catch_signals(2095a90,0,1817fc1,178,1fbee380,...) at sleepq_catch_signals+0x47a/frame 0x1db919fc sleepq_wait_sig(20d1d354,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0x1db91a20 _sleep(20d1d354,20d1d3fc,36c,1833c92,0,...) at _sleep+0x35a/frame 0x1db91a70 kern_wait6(1fbee380,7,0,0,1db91af4,...) at kern_wait6+0x551/frame 0x1db91ac8 sys_wait4(1fbee380,1fbee604,18047f7,4,0,...) at sys_wait4+0x87/frame 0x1db91be8 syscall(1db91ce8,3b,3b,3b,ffbfe7dc,...) at syscall+0x363/frame 0x1db91cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db91cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x205b7383, esp = 0xffbfe790, ebp = 0xffbfe7a8 --- amdsmn_devclass() at 0x205b7383 Tracing command bash pid 41759 tid 100288 td 0x26c1d700 sched_switch(26c1d700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1539e7b0 mi_switch(104,0,1817fc1,22b,26c1d700,...) at mi_switch+0x20e/frame 0x1539e7ec sleepq_switch(26c1d700,0,1817fc1,2b3,26c1d700,...) at sleepq_switch+0x110/frame 0x1539e820 sleepq_wait(bd4f40c,5c,1762eeb,0,0,...) at sleepq_wait+0x48/frame 0x1539e844 _sleep(bd4f40c,9dd8074,5c,1762eeb,0,...) at _sleep+0x36a/frame 0x1539e894 bufwait(bd4f40c,0,21e2bb00,8,0,...) at bufwait+0xa4/frame 0x1539e8d0 breadn_flags(20c64b60,3,0,4000,0,...) at breadn_flags+0x1e0/frame 0x1539e910 vfs_bio_getpages(20c64b60,1539eaa8,1,1539eb20,1539eb24,142eb40,142eb80) at vfs_bio_getpages+0x455/frame 0x1539e9a8 ffs_getpages(1539ea18,17b228a,208daaf8,8,17b9c18,...) at ffs_getpages+0x99/frame 0x1539e9d4 VOP_GETPAGES_APV(1dfa6f0,1539ea18,2bc,127b,1e103b4,...) at VOP_GETPAGES_APV+0xb2/frame 0x1539ea00 vnode_pager_getpages(208daaf8,1539eaa8,1,1539eb20,1539eb24,...) at vnode_pager_getpages+0x68/frame 0x1539ea44 vm_pager_get_pages(208daaf8,1539eaa8,1,1539eb20,1539eb24,...) at vm_pager_get_pages+0x53/frame 0x1539ea78 vm_fault_hold(1bb2a2b8,28176000,1,0,0,...) at vm_fault_hold+0x1a2a/frame 0x1539eb68 vm_fault(1bb2a2b8,28176000,1,0,1,...) at vm_fault+0x5e/frame 0x1539eb90 trap_pfault(28176538,1539ece8,1539ecdc,16e9164,5,...) at trap_pfault+0xfe/frame 0x1539ebd8 trap(1539ece8,3b,3b,3b,2814a400,...) at trap+0x524/frame 0x1539ecdc calltrap() at PTDpde+0x4165/frame 0x1539ecdc --- trap 0xc, eip = 0x28130aaa, esp = 0xffbfdc00, ebp = 0xffbfdc88 --- exit_tag() at 0x28130aaa Tracing command sshd pid 41758 tid 100265 td 0x26488000 sched_switch(26488000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1527a8c0 mi_switch(104,0,1817fc1,22b,26488000,...) at mi_switch+0x20e/frame 0x1527a900 sleepq_switch(26488000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1527a934 sleepq_catch_signals(1527a99c,10c7a20,214d90a0,0,214d90a4,...) at sleepq_catch_signals+0x47a/frame 0x1527a978 sleepq_wait_sig(214d90a4,0,1527a9bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1527a99c _cv_wait_sig(214d90a4,214d9090,1733ba7,6fa,1b1,...) at _cv_wait_sig+0x1d7/frame 0x1527a9e0 seltdwait(ffffffff,0,0,26488000,0,...) at seltdwait+0xbb/frame 0x1527aa10 kern_select(26488000,e,204820b8,204821d8,0,...) at kern_select+0x894/frame 0x1527aba0 sys_select(26488000,26488284,18047f7,4,be82d90,...) at sys_select+0x68/frame 0x1527abe8 syscall(1527ace8,3b,3b,3b,20b65000,...) at syscall+0x363/frame 0x1527acdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1527acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20949fdf, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x20949fdf Tracing command sshd pid 41756 tid 100310 td 0x26d04700 sched_switch(26d04700,0,104,18a,0,...) at sched_switch+0x671/frame 0xb8728f8 mi_switch(104,0,1817fc1,22b,26d04700,...) at mi_switch+0x20e/frame 0xb872938 sleepq_switch(26d04700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0xb87296c sleepq_catch_signals(b8729d4,10c7a20,1b80cd20,0,1b80cd24,...) at sleepq_catch_signals+0x47a/frame 0xb8729b0 sleepq_wait_sig(1b80cd24,0,b8729f4,101,0,...) at sleepq_wait_sig+0x14/frame 0xb8729d4 _cv_wait_sig(1b80cd24,1b80cd10,1733ba7,6fa,0,...) at _cv_wait_sig+0x1d7/frame 0xb872a18 seltdwait(ffffffff,0,0,26d04700,10000,...) at seltdwait+0xbb/frame 0xb872a48 kern_poll(26d04700,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0xb872bb0 sys_poll(26d04700,26d04984,18047f7,4,be7dd90,...) at sys_poll+0x67/frame 0xb872be8 syscall(b872ce8,3b,3b,3b,20b65000,...) at syscall+0x363/frame 0xb872cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb872cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x208ed73b, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x208ed73b Tracing command sh pid 1441 tid 100257 td 0x25684000 sched_switch(25684000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa289d0 mi_switch(104,0,1817fc1,22b,25684000,...) at mi_switch+0x20e/frame 0x1aa28a10 sleepq_switch(25684000,0,1817fc1,1ca,5c,...) at sleepq_switch+0x110/frame 0x1aa28a44 sleepq_catch_signals(2094d10,0,1817fc1,178,25684000,...) at sleepq_catch_signals+0x47a/frame 0x1aa28a88 sleepq_wait_sig(220bcb98,5c,181817a,100,0,...) at sleepq_wait_sig+0x14/frame 0x1aa28aac _sleep(220bcb98,220bcd28,15c,181817a,0,...) at _sleep+0x35a/frame 0x1aa28afc pipe_read(1bac0c08,1aa28bb8,1b578000,0,25684000,...) at pipe_read+0x47b/frame 0x1aa28b3c dofileread(1bac0c08,1aa28bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1aa28b6c kern_readv(25684000,0,1aa28bb8,1bb49570,ffbfe55f,...) at kern_readv+0x5e/frame 0x1aa28b98 sys_read(25684000,25684284,18047f7,4,256700b8,...) at sys_read+0x54/frame 0x1aa28be8 syscall(1aa28ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1aa28cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1aa28cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x20625fef, esp = 0xffbfe4fc, ebp = 0xffbfe510 --- amdsmn_devclass() at 0x20625fef Tracing command awk pid 1439 tid 100201 td 0x1fbee700 sched_switch(1fbee700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db8b9d0 mi_switch(104,0,1817fc1,22b,1fbee700,...) at mi_switch+0x20e/frame 0x1db8ba10 sleepq_switch(1fbee700,0,1817fc1,1ca,5c,...) at sleepq_switch+0x110/frame 0x1db8ba44 sleepq_catch_signals(2095a90,0,1817fc1,178,1fbee700,...) at sleepq_catch_signals+0x47a/frame 0x1db8ba88 sleepq_wait_sig(1bb33000,5c,181817a,100,0,...) at sleepq_wait_sig+0x14/frame 0x1db8baac _sleep(1bb33000,1bb33190,15c,181817a,0,...) at _sleep+0x35a/frame 0x1db8bafc pipe_read(1f4a1e70,1db8bbb8,1b578000,0,1fbee700,...) at pipe_read+0x47b/frame 0x1db8bb3c dofileread(1f4a1e70,1db8bbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db8bb6c kern_readv(1fbee700,0,1db8bbb8,be6b000,206d2000,...) at kern_readv+0x5e/frame 0x1db8bb98 sys_read(1fbee700,1fbee984,18047f7,4,be78d90,...) at sys_read+0x54/frame 0x1db8bbe8 syscall(1db8bce8,3b,3b,3b,2063a518,...) at syscall+0x363/frame 0x1db8bcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db8bcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x20631fef, esp = 0xffbfe570, ebp = 0xffbfe584 --- amdsmn_devclass() at 0x20631fef Tracing command sh pid 1438 tid 100252 td 0x25685000 sched_switch(25685000,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x15223948 mi_switch(104,0,1817fc1,22b,25685000,...) at mi_switch+0x20e/frame 0x15223984 sleepq_switch(25685000,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x152239b8 sleepq_catch_signals(20953d0,0,1817fc1,178,25685000,...) at sleepq_catch_signals+0x47a/frame 0x152239fc sleepq_wait_sig(256706a8,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0x15223a20 _sleep(256706a8,25670750,36c,1833c92,0,...) at _sleep+0x35a/frame 0x15223a70 kern_wait6(25685000,7,0,0,15223af4,...) at kern_wait6+0x551/frame 0x15223ac8 sys_wait4(25685000,25685284,18047f7,4,0,...) at sys_wait4+0x87/frame 0x15223be8 syscall(15223ce8,3b,3b,3b,ffbfe56c,...) at syscall+0x363/frame 0x15223cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15223cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x205b7383, esp = 0xffbfe520, ebp = 0xffbfe538 --- amdsmn_devclass() at 0x205b7383 Tracing command sh pid 1417 tid 100223 td 0x1f75a380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1f75a380,ffffffff,589,1877f,1f75a380,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command csh pid 1003 tid 100205 td 0x1f75aa80 sched_switch(1f75aa80,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1dba2a50 mi_switch(104,0,1817fc1,22b,1f75aa80,...) at mi_switch+0x20e/frame 0x1dba2a8c sleepq_switch(1f75aa80,0,1817fc1,1ca,74,...) at sleepq_switch+0x110/frame 0x1dba2ac0 sleepq_catch_signals(2094650,0,1817fc1,178,1f75aa80,...) at sleepq_catch_signals+0x47a/frame 0x1dba2b04 sleepq_wait_sig(bfe170c,74,177a4f4,100,0,...) at sleepq_wait_sig+0x14/frame 0x1dba2b28 _sleep(bfe170c,bfe1750,174,177a4f4,0,...) at _sleep+0x35a/frame 0x1dba2b78 kern_sigsuspend(1f75aa80,2,0,0,0,...) at kern_sigsuspend+0xd4/frame 0x1dba2bb8 sys_sigsuspend(1f75aa80,1f75ad04,18047f7,4,1,...) at sys_sigsuspend+0x55/frame 0x1dba2be8 syscall(1dba2ce8,3b,3b,3b,ffbfe118,...) at syscall+0x363/frame 0x1dba2cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dba2cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2062e3c3, esp = 0xffbfe0e4, ebp = 0xffbfe0f0 --- amdsmn_devclass() at 0x2062e3c3 Tracing command su pid 1002 tid 100210 td 0x246c6000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(246c6000,ffffffff,3ea,18772,246c6000,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command bash pid 994 tid 100233 td 0x246bba80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(246bba80,ffffffff,3e2,18789,246bba80,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command sshd pid 993 tid 100195 td 0x1fbefa80 sched_switch(1fbefa80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db688c0 mi_switch(104,0,1817fc1,22b,1fbefa80,...) at mi_switch+0x20e/frame 0x1db68900 sleepq_switch(1fbefa80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db68934 sleepq_catch_signals(1db6899c,10c7a20,be6f4e0,0,be6f4e4,...) at sleepq_catch_signals+0x47a/frame 0x1db68978 sleepq_wait_sig(be6f4e4,0,1db689bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db6899c _cv_wait_sig(be6f4e4,be6f4d0,1733ba7,6fa,1b1,...) at _cv_wait_sig+0x1d7/frame 0x1db689e0 seltdwait(ffffffff,0,0,1fbefa80,0,...) at seltdwait+0xbb/frame 0x1db68a10 kern_select(1fbefa80,e,204820b8,204821d8,0,...) at kern_select+0x894/frame 0x1db68ba0 sys_select(1fbefa80,1fbefd04,18047f7,4,be78d90,...) at sys_select+0x68/frame 0x1db68be8 syscall(1db68ce8,3b,3b,3b,20b65000,...) at syscall+0x363/frame 0x1db68cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db68cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20949fdf, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x20949fdf Tracing command sshd pid 991 tid 100232 td 0x246bc000 sched_switch(246bc000,0,104,18a,0,...) at sched_switch+0x671/frame 0x151b68f8 mi_switch(104,0,1817fc1,22b,246bc000,...) at mi_switch+0x20e/frame 0x151b6938 sleepq_switch(246bc000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x151b696c sleepq_catch_signals(151b69d4,10c7a20,be6f4a0,0,be6f4a4,...) at sleepq_catch_signals+0x47a/frame 0x151b69b0 sleepq_wait_sig(be6f4a4,0,151b69f4,101,0,...) at sleepq_wait_sig+0x14/frame 0x151b69d4 _cv_wait_sig(be6f4a4,be6f490,1733ba7,6fa,0,...) at _cv_wait_sig+0x1d7/frame 0x151b6a18 seltdwait(ffffffff,0,0,246bc000,10000,...) at seltdwait+0xbb/frame 0x151b6a48 kern_poll(246bc000,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0x151b6bb0 sys_poll(246bc000,246bc284,18047f7,4,2428d90,...) at sys_poll+0x67/frame 0x151b6be8 syscall(151b6ce8,3b,3b,3b,20b65000,...) at syscall+0x363/frame 0x151b6cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x151b6cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x208ed73b, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x208ed73b Tracing command bash pid 797 tid 100197 td 0x1fbef380 sched_switch(1fbef380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db72898 mi_switch(104,0,1817fc1,22b,1fbef380,...) at mi_switch+0x20e/frame 0x1db728d8 sleepq_switch(1fbef380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db7290c sleepq_catch_signals(1db72974,10c7a20,1f33b320,0,1f33b324,...) at sleepq_catch_signals+0x47a/frame 0x1db72950 sleepq_wait_sig(1f33b324,0,1db72994,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db72974 _cv_wait_sig(1f33b324,1f33b310,1733ba7,6fa,10000000,...) at _cv_wait_sig+0x1d7/frame 0x1db729b8 seltdwait(ffffffff,0,0,1fbef380,0,...) at seltdwait+0xbb/frame 0x1db729e8 kern_select(1fbef380,1,ffbfe008,0,0,...) at kern_select+0x894/frame 0x1db72b78 sys_pselect(1fbef380,1fbef604,18047f7,4,be82d90,...) at sys_pselect+0x166/frame 0x1db72be8 syscall(1db72ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db72cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db72cdc --- syscall (522, FreeBSD ELF32, sys_pselect), eip = 0x282dc413, esp = 0xffbfdfa8, ebp = 0xffbfdfd0 --- exit_tag() at 0x282dc413 Tracing command csh pid 790 tid 100174 td 0x1f509700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1f509700,ffffffff,316,1874e,1f509700,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command login pid 775 tid 100192 td 0x1fbf0000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1fbf0000,ffffffff,307,18760,1fbf0000,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command getty pid 774 tid 100191 td 0x1fbf0380 sched_switch(1fbf0380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1db53968 mi_switch(104,0,1817fc1,22b,1fbf0380,...) at mi_switch+0x20e/frame 0x1db539a4 sleepq_switch(1fbf0380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db539d8 sleepq_catch_signals(1db53a40,10c7a20,1b56d814,0,1b56d874,...) at sleepq_catch_signals+0x47a/frame 0x1db53a1c sleepq_wait_sig(1b56d874,0,1db53a60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db53a40 _cv_wait_sig(1b56d874,1b56d804,183e628,5a9,1db53bb8,...) at _cv_wait_sig+0x1d7/frame 0x1db53a84 tty_wait(1b56d800,1b56d874,1db53bb8,1,0,...) at tty_wait+0x4a/frame 0x1db53aa4 ttydisc_read(1b56d800,1db53bb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db53ae4 ttydev_read(1b486200,1db53bb8,0,1db53b40,1fbf0380,...) at ttydev_read+0x55/frame 0x1db53b04 devfs_read_f(1bab21c0,1db53bb8,1b9c0100,0,1fbf0380,...) at devfs_read_f+0xc5/frame 0x1db53b3c dofileread(1bab21c0,1db53bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db53b6c kern_readv(1fbf0380,0,1db53bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db53b98 sys_read(1fbf0380,1fbf0604,18047f7,4,be7dd90,...) at sys_read+0x54/frame 0x1db53be8 syscall(1db53ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db53cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db53cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 773 tid 100190 td 0x1fbf0700 sched_switch(1fbf0700,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1db4e968 mi_switch(104,0,1817fc1,22b,1fbf0700,...) at mi_switch+0x20e/frame 0x1db4e9a4 sleepq_switch(1fbf0700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db4e9d8 sleepq_catch_signals(1db4ea40,10c7a20,1b56da14,0,1b56da74,...) at sleepq_catch_signals+0x47a/frame 0x1db4ea1c sleepq_wait_sig(1b56da74,0,1db4ea60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db4ea40 _cv_wait_sig(1b56da74,1b56da04,183e628,5a9,1db4ebb8,...) at _cv_wait_sig+0x1d7/frame 0x1db4ea84 tty_wait(1b56da00,1b56da74,1db4ebb8,1,0,...) at tty_wait+0x4a/frame 0x1db4eaa4 ttydisc_read(1b56da00,1db4ebb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db4eae4 ttydev_read(1b486300,1db4ebb8,0,1db4eb40,1fbf0700,...) at ttydev_read+0x55/frame 0x1db4eb04 devfs_read_f(1ba9a118,1db4ebb8,1bb5ab00,0,1fbf0700,...) at devfs_read_f+0xc5/frame 0x1db4eb3c dofileread(1ba9a118,1db4ebb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db4eb6c kern_readv(1fbf0700,0,1db4ebb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db4eb98 sys_read(1fbf0700,1fbf0984,18047f7,4,be82d90,...) at sys_read+0x54/frame 0x1db4ebe8 syscall(1db4ece8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db4ecdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db4ecdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 772 tid 100189 td 0x1fbf0a80 sched_switch(1fbf0a80,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1db49968 mi_switch(104,0,1817fc1,22b,1fbf0a80,...) at mi_switch+0x20e/frame 0x1db499a4 sleepq_switch(1fbf0a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db499d8 sleepq_catch_signals(1db49a40,10c7a20,1b56dc14,0,1b56dc74,...) at sleepq_catch_signals+0x47a/frame 0x1db49a1c sleepq_wait_sig(1b56dc74,0,1db49a60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db49a40 _cv_wait_sig(1b56dc74,1b56dc04,183e628,5a9,1db49bb8,...) at _cv_wait_sig+0x1d7/frame 0x1db49a84 tty_wait(1b56dc00,1b56dc74,1db49bb8,1,0,...) at tty_wait+0x4a/frame 0x1db49aa4 ttydisc_read(1b56dc00,1db49bb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db49ae4 ttydev_read(1b486400,1db49bb8,0,1db49b40,1fbf0a80,...) at ttydev_read+0x55/frame 0x1db49b04 devfs_read_f(1ba9a150,1db49bb8,1b578100,0,1fbf0a80,...) at devfs_read_f+0xc5/frame 0x1db49b3c dofileread(1ba9a150,1db49bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db49b6c kern_readv(1fbf0a80,0,1db49bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db49b98 sys_read(1fbf0a80,1fbf0d04,18047f7,4,be7dd90,...) at sys_read+0x54/frame 0x1db49be8 syscall(1db49ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db49cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db49cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 771 tid 100188 td 0x1fbf1000 sched_switch(1fbf1000,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1db44968 mi_switch(104,0,1817fc1,22b,1fbf1000,...) at mi_switch+0x20e/frame 0x1db449a4 sleepq_switch(1fbf1000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db449d8 sleepq_catch_signals(1db44a40,10c7a20,1b56de14,0,1b56de74,...) at sleepq_catch_signals+0x47a/frame 0x1db44a1c sleepq_wait_sig(1b56de74,0,1db44a60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db44a40 _cv_wait_sig(1b56de74,1b56de04,183e628,5a9,1db44bb8,...) at _cv_wait_sig+0x1d7/frame 0x1db44a84 tty_wait(1b56de00,1b56de74,1db44bb8,1,0,...) at tty_wait+0x4a/frame 0x1db44aa4 ttydisc_read(1b56de00,1db44bb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db44ae4 ttydev_read(1b486500,1db44bb8,0,1db44b40,1fbf1000,...) at ttydev_read+0x55/frame 0x1db44b04 devfs_read_f(1bab17a8,1db44bb8,1b9c0300,0,1fbf1000,...) at devfs_read_f+0xc5/frame 0x1db44b3c dofileread(1bab17a8,1db44bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db44b6c kern_readv(1fbf1000,0,1db44bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db44b98 sys_read(1fbf1000,1fbf1284,18047f7,4,be7dd90,...) at sys_read+0x54/frame 0x1db44be8 syscall(1db44ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db44cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db44cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 770 tid 100187 td 0x1b560700 sched_switch(1b560700,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1db3f968 mi_switch(104,0,1817fc1,22b,1b560700,...) at mi_switch+0x20e/frame 0x1db3f9a4 sleepq_switch(1b560700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db3f9d8 sleepq_catch_signals(1db3fa40,10c7a20,1b56e014,0,1b56e074,...) at sleepq_catch_signals+0x47a/frame 0x1db3fa1c sleepq_wait_sig(1b56e074,0,1db3fa60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db3fa40 _cv_wait_sig(1b56e074,1b56e004,183e628,5a9,1db3fbb8,...) at _cv_wait_sig+0x1d7/frame 0x1db3fa84 tty_wait(1b56e000,1b56e074,1db3fbb8,1,0,...) at tty_wait+0x4a/frame 0x1db3faa4 ttydisc_read(1b56e000,1db3fbb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db3fae4 ttydev_read(1b486600,1db3fbb8,0,1db3fb40,1b560700,...) at ttydev_read+0x55/frame 0x1db3fb04 devfs_read_f(1bab1620,1db3fbb8,bfb4900,0,1b560700,...) at devfs_read_f+0xc5/frame 0x1db3fb3c dofileread(1bab1620,1db3fbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db3fb6c kern_readv(1b560700,0,1db3fbb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db3fb98 sys_read(1b560700,1b560984,18047f7,4,be82d90,...) at sys_read+0x54/frame 0x1db3fbe8 syscall(1db3fce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db3fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db3fcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 769 tid 100186 td 0x1b560a80 sched_switch(1b560a80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1db3a968 mi_switch(104,0,1817fc1,22b,1b560a80,...) at mi_switch+0x20e/frame 0x1db3a9a4 sleepq_switch(1b560a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db3a9d8 sleepq_catch_signals(1db3aa40,10c7a20,1b56e214,0,1b56e274,...) at sleepq_catch_signals+0x47a/frame 0x1db3aa1c sleepq_wait_sig(1b56e274,0,1db3aa60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db3aa40 _cv_wait_sig(1b56e274,1b56e204,183e628,5a9,1db3abb8,...) at _cv_wait_sig+0x1d7/frame 0x1db3aa84 tty_wait(1b56e200,1b56e274,1db3abb8,1,0,...) at tty_wait+0x4a/frame 0x1db3aaa4 ttydisc_read(1b56e200,1db3abb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db3aae4 ttydev_read(1b486700,1db3abb8,0,1db3ab40,1b560a80,...) at ttydev_read+0x55/frame 0x1db3ab04 devfs_read_f(1bad0c40,1db3abb8,1bb5a700,0,1b560a80,...) at devfs_read_f+0xc5/frame 0x1db3ab3c dofileread(1bad0c40,1db3abb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db3ab6c kern_readv(1b560a80,0,1db3abb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db3ab98 sys_read(1b560a80,1b560d04,18047f7,4,be82d90,...) at sys_read+0x54/frame 0x1db3abe8 syscall(1db3ace8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db3acdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db3acdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 768 tid 100182 td 0x1f70c380 sched_switch(1f70c380,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1db22968 mi_switch(104,0,1817fc1,22b,1f70c380,...) at mi_switch+0x20e/frame 0x1db229a4 sleepq_switch(1f70c380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db229d8 sleepq_catch_signals(1db22a40,10c7a20,1b56e414,0,1b56e474,...) at sleepq_catch_signals+0x47a/frame 0x1db22a1c sleepq_wait_sig(1b56e474,0,1db22a60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db22a40 _cv_wait_sig(1b56e474,1b56e404,183e628,5a9,1db22bb8,...) at _cv_wait_sig+0x1d7/frame 0x1db22a84 tty_wait(1b56e400,1b56e474,1db22bb8,1,0,...) at tty_wait+0x4a/frame 0x1db22aa4 ttydisc_read(1b56e400,1db22bb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db22ae4 ttydev_read(1b486800,1db22bb8,0,1db22b40,1f70c380,...) at ttydev_read+0x55/frame 0x1db22b04 devfs_read_f(1bab1930,1db22bb8,1bb5ad00,0,1f70c380,...) at devfs_read_f+0xc5/frame 0x1db22b3c dofileread(1bab1930,1db22bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db22b6c kern_readv(1f70c380,0,1db22bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db22b98 sys_read(1f70c380,1f70c604,18047f7,4,1fbbf760,...) at sys_read+0x54/frame 0x1db22be8 syscall(1db22ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db22cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db22cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command getty pid 767 tid 100183 td 0x1f70c000 sched_switch(1f70c000,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1db28968 mi_switch(104,0,1817fc1,22b,1f70c000,...) at mi_switch+0x20e/frame 0x1db289a4 sleepq_switch(1f70c000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db289d8 sleepq_catch_signals(1db28a40,10c7a20,1b56e614,0,1b56e674,...) at sleepq_catch_signals+0x47a/frame 0x1db28a1c sleepq_wait_sig(1b56e674,0,1db28a60,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db28a40 _cv_wait_sig(1b56e674,1b56e604,183e628,5a9,1db28bb8,...) at _cv_wait_sig+0x1d7/frame 0x1db28a84 tty_wait(1b56e600,1b56e674,1db28bb8,1,0,...) at tty_wait+0x4a/frame 0x1db28aa4 ttydisc_read(1b56e600,1db28bb8,0,d1,1dcbaa0,...) at ttydisc_read+0x238/frame 0x1db28ae4 ttydev_read(1b486900,1db28bb8,0,1db28b40,1f70c000,...) at ttydev_read+0x55/frame 0x1db28b04 devfs_read_f(1bab1738,1db28bb8,1b9c0500,0,1f70c000,...) at devfs_read_f+0xc5/frame 0x1db28b3c dofileread(1bab1738,1db28bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db28b6c kern_readv(1f70c000,0,1db28bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db28b98 sys_read(1f70c000,1f70c284,18047f7,4,17ddfd6,...) at sys_read+0x54/frame 0x1db28be8 syscall(1db28ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1db28cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db28cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205effef, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x205effef Tracing command inetd pid 750 tid 100121 td 0x1b400380 sched_switch(1b400380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eef78c0 mi_switch(104,0,1817fc1,22b,1b400380,...) at mi_switch+0x20e/frame 0x1eef7900 sleepq_switch(1b400380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eef7934 sleepq_catch_signals(1eef799c,10c7a20,1f33b3a0,0,1f33b3a4,...) at sleepq_catch_signals+0x47a/frame 0x1eef7978 sleepq_wait_sig(1f33b3a4,0,1eef79bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eef799c _cv_wait_sig(1f33b3a4,1f33b390,1733ba7,6fa,1f78a208,...) at _cv_wait_sig+0x1d7/frame 0x1eef79e0 seltdwait(ffffffff,0,0,1b400380,0,...) at seltdwait+0xbb/frame 0x1eef7a10 kern_select(1b400380,9,ffbfe5c0,0,0,...) at kern_select+0x894/frame 0x1eef7ba0 sys_select(1b400380,1b400604,18047f7,4,be78d90,...) at sys_select+0x68/frame 0x1eef7be8 syscall(1eef7ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1eef7cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eef7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2060afdf, esp = 0xffbfdfec, ebp = 0xffbfe010 --- amdsmn_devclass() at 0x2060afdf Tracing command nfscbd pid 716 tid 100118 td 0x1b400a80 sched_switch(1b400a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eee87e8 mi_switch(104,0,1817fc1,22b,1b400a80,...) at mi_switch+0x20e/frame 0x1eee8828 sleepq_switch(1b400a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eee885c sleepq_catch_signals(8,17cad96,cb,0,1b400a80,...) at sleepq_catch_signals+0x47a/frame 0x1eee88a0 sleepq_timedwait_sig(1b597850,0,1eee8904,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1eee88e0 _cv_timedwait_sig_sbt(1b597850,1af900c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eee8928 svc_run_internal(13c9720,1af900c0,1b3e8354,1eee8a30,0,...) at svc_run_internal+0xbef/frame 0x1eee8a04 svc_run(1af90000,0,1829ed4,f5,1eee8ad0,...) at svc_run+0x1ed/frame 0x1eee8a58 nfscbd_nfsd(1b400a80,1eee8b30,4,10f9e20,2060d000,...) at nfscbd_nfsd+0x105/frame 0x1eee8af8 nfssvc_nfscl(1b400a80,1b400d04,1b400a80,1b3e8354,1eee8cdc,...) at nfssvc_nfscl+0x23c/frame 0x1eee8bd0 sys_nfssvc(1b400a80,1b400d04,18047f7,4,1b3e840c,...) at sys_nfssvc+0xa6/frame 0x1eee8be8 syscall(1eee8ce8,3b,3b,3b,1,...) at syscall+0x363/frame 0x1eee8cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eee8cdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x20490a6f, esp = 0xffbfeb0c, ebp = 0xffbfee28 --- amdsmn_devclass() at 0x20490a6f Tracing command nfscbd pid 716 tid 100177 td 0x1fb9da80 sched_switch(1fb9da80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db08aa8 mi_switch(104,0,1817fc1,22b,1fb9da80,...) at mi_switch+0x20e/frame 0x1db08ae8 sleepq_switch(1fb9da80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db08b1c sleepq_catch_signals(1db08b84,10c7a20,1af900d0,0,1f687c50,...) at sleepq_catch_signals+0x47a/frame 0x1db08b60 sleepq_wait_sig(1f687c50,0,1db08ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db08b84 _cv_wait_sig(1f687c50,1af900c0,179d187,439,a5a5a5a5,...) at _cv_wait_sig+0x1d7/frame 0x1db08bc8 svc_run_internal(1db08cd4,10a779e,1af900c0,1db08ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1db08ca4 svc_thread_start(1af900c0,1db08ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1db08cac fork_exit(13c9720,1af900c0,1db08ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db08cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db08cd4 --- trap 0, eip = 0, esp = 0x1db08d20, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 716 tid 100178 td 0x1fb9d700 sched_switch(1fb9d700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db0daa8 mi_switch(104,0,1817fc1,22b,1fb9d700,...) at mi_switch+0x20e/frame 0x1db0dae8 sleepq_switch(1fb9d700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db0db1c sleepq_catch_signals(1db0db84,10c7a20,1af900d0,0,1f687bd0,...) at sleepq_catch_signals+0x47a/frame 0x1db0db60 sleepq_wait_sig(1f687bd0,0,1db0dba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db0db84 _cv_wait_sig(1f687bd0,1af900c0,179d187,481,a5a5a5a5,...) at _cv_wait_sig+0x1d7/frame 0x1db0dbc8 svc_run_internal(1db0dcd4,10a779e,1af900c0,1db0dce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1db0dca4 svc_thread_start(1af900c0,1db0dce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1db0dcac fork_exit(13c9720,1af900c0,1db0dce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db0dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db0dcd4 --- trap 0, eip = 0, esp = 0x1db0dd20, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 716 tid 100179 td 0x1fb9d380 sched_switch(1fb9d380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1db12aa8 mi_switch(104,0,1817fc1,22b,1fb9d380,...) at mi_switch+0x20e/frame 0x1db12ae8 sleepq_switch(1fb9d380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1db12b1c sleepq_catch_signals(1db12b84,10c7a20,1af900d0,0,1f687b50,...) at sleepq_catch_signals+0x47a/frame 0x1db12b60 sleepq_wait_sig(1f687b50,0,1db12ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db12b84 _cv_wait_sig(1f687b50,1af900c0,179d187,481,1ab33d40,...) at _cv_wait_sig+0x1d7/frame 0x1db12bc8 svc_run_internal(1db12cd4,10a779e,1af900c0,1db12ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1db12ca4 svc_thread_start(1af900c0,1db12ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1db12cac fork_exit(13c9720,1af900c0,1db12ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db12cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db12cd4 --- trap 0, eip = 0, esp = 0x1db12d20, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 715 tid 100110 td 0x9f7ca80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(9f7ca80,ffffffff,2cb,1870e,9f7ca80,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command cron pid 682 tid 100171 td 0x1b3eea80 sched_switch(1b3eea80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dae69e8 mi_switch(104,0,1817fc1,22b,1b3eea80,...) at mi_switch+0x20e/frame 0x1dae6a28 sleepq_switch(1b3eea80,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1dae6a5c sleepq_catch_signals(1b3eea80,0,601,0,9f4e,...) at sleepq_catch_signals+0x47a/frame 0x1dae6aa0 sleepq_timedwait_sig(208740c,6c,9f4e,beefce4e,3,...) at sleepq_timedwait_sig+0x17/frame 0x1dae6ae0 _sleep(208740c,0,16c,182aa35,fe6f12bc,...) at _sleep+0x2ed/frame 0x1dae6b30 kern_clock_nanosleep(1b3eea80,0,0,1dae6bc8,1dae6bc0,...) at kern_clock_nanosleep+0x1f4/frame 0x1dae6ba0 sys_nanosleep(1b3eea80,1b3eed04,18047f7,4,1,...) at sys_nanosleep+0x7f/frame 0x1dae6be8 syscall(1dae6ce8,3b,3b,3b,ffbfed48,...) at syscall+0x363/frame 0x1dae6cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dae6cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x20593443, esp = 0xffbfec98, ebp = 0xffbfeca8 --- amdsmn_devclass() at 0x20593443 Tracing command sendmail pid 678 tid 100172 td 0x1f509a80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1f509a80,ffffffff,2a6,1874c,1f509a80,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command sendmail pid 675 tid 100167 td 0x1f50a380 sched_switch(1f50a380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1daca8a0 mi_switch(104,0,1817fc1,22b,1f50a380,...) at mi_switch+0x20e/frame 0x1daca8e0 sleepq_switch(1f50a380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1daca914 sleepq_catch_signals(8,17cad96,cb,0,1f50a380,...) at sleepq_catch_signals+0x47a/frame 0x1daca958 sleepq_timedwait_sig(1b43dba4,0,1daca9bc,50000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x1daca998 _cv_timedwait_sig_sbt(1b43dba4,1b43db90,c0989023,9f4e,50000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1daca9e0 seltdwait(9f4e,50000000,0,1f50a380,0,...) at seltdwait+0x82/frame 0x1dacaa10 kern_select(1f50a380,5,ffbfd168,0,0,...) at kern_select+0x894/frame 0x1dacaba0 sys_select(1f50a380,1f50a604,18047f7,4,be82d90,...) at sys_select+0x68/frame 0x1dacabe8 syscall(1dacace8,3b,3b,3b,ffbfd0d8,...) at syscall+0x363/frame 0x1dacacdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dacacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x208b0fdf, esp = 0xffbfd02c, ebp = 0xffbfd050 --- amdsmn_devclass() at 0x208b0fdf Tracing command sshd pid 672 tid 100170 td 0x1b400000 sched_switch(1b400000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dae18c0 mi_switch(104,0,1817fc1,22b,1b400000,...) at mi_switch+0x20e/frame 0x1dae1900 sleepq_switch(1b400000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1dae1934 sleepq_catch_signals(1dae199c,10c7a20,1f33b420,0,1f33b424,...) at sleepq_catch_signals+0x47a/frame 0x1dae1978 sleepq_wait_sig(1f33b424,0,1dae19bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dae199c _cv_wait_sig(1f33b424,1f33b410,1733ba7,6fa,1f78a820,...) at _cv_wait_sig+0x1d7/frame 0x1dae19e0 seltdwait(ffffffff,0,0,1b400000,0,...) at seltdwait+0xbb/frame 0x1dae1a10 kern_select(1b400000,7,20482120,0,0,...) at kern_select+0x894/frame 0x1dae1ba0 sys_select(1b400000,1b400284,18047f7,4,1,...) at sys_select+0x68/frame 0x1dae1be8 syscall(1dae1ce8,3b,3b,3b,20b65000,...) at syscall+0x363/frame 0x1dae1cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dae1cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20949fdf, esp = 0xffbfe7f4, ebp = 0xffbfe818 --- amdsmn_devclass() at 0x20949fdf Tracing command smartd pid 634 tid 100120 td 0x1b400700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1b400700,ffffffff,27a,18718,1b400700,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command powerd pid 604 tid 100165 td 0x1f50a700 sched_switch(1f50a700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dabd8a0 mi_switch(104,0,1817fc1,22b,1f50a700,...) at mi_switch+0x20e/frame 0x1dabd8e0 sleepq_switch(1f50a700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1dabd914 sleepq_catch_signals(8,17cad96,cb,0,1f50a700,...) at sleepq_catch_signals+0x47a/frame 0x1dabd958 sleepq_timedwait_sig(1b03ac64,0,1dabd9bc,3ffffff,0,...) at sleepq_timedwait_sig+0x17/frame 0x1dabd998 _cv_timedwait_sig_sbt(1b03ac64,1b03ac50,cdf9fece,9f49,3ffffff,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1dabd9e0 seltdwait(9f49,3ffffff,0,2086358,0,...) at seltdwait+0x82/frame 0x1dabda10 kern_select(1f50a700,0,ffbfed90,0,ffbfed90,...) at kern_select+0x894/frame 0x1dabdba0 sys_select(1f50a700,1f50a984,18047f7,4,be7dd90,...) at sys_select+0x68/frame 0x1dabdbe8 syscall(1dabdce8,3b,3b,3b,ffbfed80,...) at syscall+0x363/frame 0x1dabdcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dabdcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205edfdf, esp = 0xffbfecf4, ebp = 0xffbfed18 --- amdsmn_devclass() at 0x205edfdf Tracing command ntpd pid 601 tid 100164 td 0x1f50aa80 sched_switch(1f50aa80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dab68c0 mi_switch(104,0,1817fc1,22b,1f50aa80,...) at mi_switch+0x20e/frame 0x1dab6900 sleepq_switch(1f50aa80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1dab6934 sleepq_catch_signals(1dab699c,10c7a20,be6f5e0,0,be6f5e4,...) at sleepq_catch_signals+0x47a/frame 0x1dab6978 sleepq_wait_sig(be6f5e4,0,1dab69bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dab699c _cv_wait_sig(be6f5e4,be6f5d0,1733ba7,6fa,1f4e2410,...) at _cv_wait_sig+0x1d7/frame 0x1dab69e0 seltdwait(ffffffff,0,0,1f50aa80,0,...) at seltdwait+0xbb/frame 0x1dab6a10 kern_select(1f50aa80,1b,ffbfe2c0,0,0,...) at kern_select+0x894/frame 0x1dab6ba0 sys_select(1f50aa80,1f50ad04,18047f7,4,1,...) at sys_select+0x68/frame 0x1dab6be8 syscall(1dab6ce8,3b,3b,3b,208d6000,...) at syscall+0x363/frame 0x1dab6cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dab6cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x20895fdf, esp = 0xffbfe1e4, ebp = 0xffbfe208 --- amdsmn_devclass() at 0x20895fdf Tracing command rpc.lockd pid 563 tid 100163 td 0x1f50b000 sched_switch(1f50b000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1dab1838 mi_switch(104,0,1817fc1,22b,1f50b000,...) at mi_switch+0x20e/frame 0x1dab1878 sleepq_switch(1f50b000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1dab18ac sleepq_catch_signals(8,17cad96,cb,0,1f50b000,...) at sleepq_catch_signals+0x47a/frame 0x1dab18f0 sleepq_timedwait_sig(1acb2d50,0,1dab1954,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1dab1930 _cv_timedwait_sig_sbt(1acb2d50,1f2fc0c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1dab1978 svc_run_internal(1f50b1c4,14,1751e7b,1849351,19,...) at svc_run_internal+0xbef/frame 0x1dab1a54 svc_run(1f2fc000,0,4,13af990,1dab1b40,...) at svc_run+0x1ed/frame 0x1dab1aa8 sys_nlm_syscall(1f50b000,1f50b284,18047f7,4,1b55040c,...) at sys_nlm_syscall+0x92e/frame 0x1dab1be8 syscall(1dab1ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1dab1cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dab1cdc --- syscall (154, FreeBSD ELF32, sys_nlm_syscall), eip = 0x204a8a7f, esp = 0xffbfed74, ebp = 0xffbfee28 --- amdsmn_devclass() at 0x204a8a7f Tracing command rpc.statd pid 560 tid 100106 td 0x9f7d380 sched_switch(9f7d380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ee928a0 mi_switch(104,0,1817fc1,22b,9f7d380,...) at mi_switch+0x20e/frame 0x1ee928e0 sleepq_switch(9f7d380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ee92914 sleepq_catch_signals(8,17cad96,cb,0,9f7d380,...) at sleepq_catch_signals+0x47a/frame 0x1ee92958 sleepq_timedwait_sig(1b43dbe4,0,1ee929bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1ee92998 _cv_timedwait_sig_sbt(1b43dbe4,1b43dbd0,4b48bbe8,9f4b,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1ee929e0 seltdwait(9f4b,e0000000,1,9f7d380,0,...) at seltdwait+0x82/frame 0x1ee92a10 kern_select(9f7d380,8,ffbfece8,0,0,...) at kern_select+0x894/frame 0x1ee92ba0 sys_select(9f7d380,9f7d604,18047f7,4,2428d90,...) at sys_select+0x68/frame 0x1ee92be8 syscall(1ee92ce8,3b,3b,3b,ffbfece8,...) at syscall+0x363/frame 0x1ee92cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ee92cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205e4fdf, esp = 0xffbfec34, ebp = 0xffbfed80 --- amdsmn_devclass() at 0x205e4fdf Tracing command nfsd pid 559 tid 100131 td 0x1f70ca80 sched_switch(1f70ca80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef7f370 mi_switch(104,0,1817fc1,22b,1f70ca80,...) at mi_switch+0x20e/frame 0x1ef7f3b0 sleepq_switch(1f70ca80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef7f3e4 sleepq_catch_signals(8,17cad96,cb,0,1f70ca80,...) at sleepq_catch_signals+0x47a/frame 0x1ef7f428 sleepq_timedwait_sig(1f687ed0,0,1ef7f48c,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1ef7f468 _cv_timedwait_sig_sbt(1f687ed0,1b3f50c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1ef7f4b0 svc_run_internal(13c9720,1b3f50c0,1babb354,1ef7f5b8,0,...) at svc_run_internal+0xbef/frame 0x1ef7f58c svc_run(1b3f5000,1f70ca80,1829feb,1fe,0,...) at svc_run+0x1ed/frame 0x1ef7f5e0 nfsrvd_nfsd(1f70ca80,1ef7f7b0,34,0,0,...) at nfsrvd_nfsd+0x3ad/frame 0x1ef7f720 nfssvc_nfsd(1f70ca80,1f70cd04,1f70ca80,1babb354,1ef7fcdc,...) at nfssvc_nfsd+0x60a/frame 0x1ef7fbd0 sys_nfssvc(1f70ca80,1f70cd04,18047f7,4,1babb40c,...) at sys_nfssvc+0xa6/frame 0x1ef7fbe8 syscall(1ef7fce8,3b,3b,3b,ffbfeb08,...) at syscall+0x363/frame 0x1ef7fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef7fcdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x20494a6f, esp = 0xffbfe77c, ebp = 0xffbfe9e8 --- amdsmn_devclass() at 0x20494a6f Tracing command nfsd pid 559 tid 100132 td 0x1afc3a80 sched_switch(1afc3a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef84aa8 mi_switch(104,0,1817fc1,22b,1afc3a80,...) at mi_switch+0x20e/frame 0x1ef84ae8 sleepq_switch(1afc3a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef84b1c sleepq_catch_signals(1ef84b84,10c7a20,1b3f50d0,0,1acb2250,...) at sleepq_catch_signals+0x47a/frame 0x1ef84b60 sleepq_wait_sig(1acb2250,0,1ef84ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef84b84 _cv_wait_sig(1acb2250,1b3f50c0,179d187,439,0,...) at _cv_wait_sig+0x1d7/frame 0x1ef84bc8 svc_run_internal(1ef84cd4,10a779e,1b3f50c0,1ef84ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef84ca4 svc_thread_start(1b3f50c0,1ef84ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef84cac fork_exit(13c9720,1b3f50c0,1ef84ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef84cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef84cd4 --- trap 0x20, eip = 0, esp = 0x1ef84d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100133 td 0x1f752000 sched_switch(1f752000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef89aa8 mi_switch(104,0,1817fc1,22b,1f752000,...) at mi_switch+0x20e/frame 0x1ef89ae8 sleepq_switch(1f752000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef89b1c sleepq_catch_signals(1ef89b84,10c7a20,1b3f50d0,0,1acb22d0,...) at sleepq_catch_signals+0x47a/frame 0x1ef89b60 sleepq_wait_sig(1acb22d0,0,1ef89ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef89b84 _cv_wait_sig(1acb22d0,1b3f50c0,179d187,481,a5a5a5a5,...) at _cv_wait_sig+0x1d7/frame 0x1ef89bc8 svc_run_internal(1ef89cd4,10a779e,1b3f50c0,1ef89ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef89ca4 svc_thread_start(1b3f50c0,1ef89ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef89cac fork_exit(13c9720,1b3f50c0,1ef89ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef89cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef89cd4 --- trap 0x20, eip = 0, esp = 0x1ef89d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100134 td 0x1f751a80 sched_switch(1f751a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef8eaa8 mi_switch(104,0,1817fc1,22b,1f751a80,...) at mi_switch+0x20e/frame 0x1ef8eae8 sleepq_switch(1f751a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef8eb1c sleepq_catch_signals(1ef8eb84,10c7a20,1b3f50d0,0,1acb21d0,...) at sleepq_catch_signals+0x47a/frame 0x1ef8eb60 sleepq_wait_sig(1acb21d0,0,1ef8eba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef8eb84 _cv_wait_sig(1acb21d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1ef8ebc8 svc_run_internal(1ef8ecd4,10a779e,1b3f50c0,1ef8ece8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef8eca4 svc_thread_start(1b3f50c0,1ef8ece8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef8ecac fork_exit(13c9720,1b3f50c0,1ef8ece8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef8ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef8ecd4 --- trap 0x20, eip = 0, esp = 0x1ef8ed20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100135 td 0x1f751700 sched_switch(1f751700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef93aa8 mi_switch(104,0,1817fc1,22b,1f751700,...) at mi_switch+0x20e/frame 0x1ef93ae8 sleepq_switch(1f751700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef93b1c sleepq_catch_signals(1ef93b84,10c7a20,1b3f50d0,0,1acb2350,...) at sleepq_catch_signals+0x47a/frame 0x1ef93b60 sleepq_wait_sig(1acb2350,0,1ef93ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef93b84 _cv_wait_sig(1acb2350,1b3f50c0,179d187,481,5a5a5a5a,...) at _cv_wait_sig+0x1d7/frame 0x1ef93bc8 svc_run_internal(1ef93cd4,10a779e,1b3f50c0,1ef93ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef93ca4 svc_thread_start(1b3f50c0,1ef93ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef93cac fork_exit(13c9720,1b3f50c0,1ef93ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef93cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef93cd4 --- trap 0x20, eip = 0, esp = 0x1ef93d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100136 td 0x1f751380 sched_switch(1f751380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef98aa8 mi_switch(104,0,1817fc1,22b,1f751380,...) at mi_switch+0x20e/frame 0x1ef98ae8 sleepq_switch(1f751380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef98b1c sleepq_catch_signals(1ef98b84,10c7a20,1b3f50d0,0,1b597dd0,...) at sleepq_catch_signals+0x47a/frame 0x1ef98b60 sleepq_wait_sig(1b597dd0,0,1ef98ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef98b84 _cv_wait_sig(1b597dd0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1ef98bc8 svc_run_internal(1ef98cd4,10a779e,1b3f50c0,1ef98ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef98ca4 svc_thread_start(1b3f50c0,1ef98ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef98cac fork_exit(13c9720,1b3f50c0,1ef98ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef98cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef98cd4 --- trap 0x20, eip = 0, esp = 0x1ef98d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100137 td 0x1f751000 sched_switch(1f751000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef9daa8 mi_switch(104,0,1817fc1,22b,1f751000,...) at mi_switch+0x20e/frame 0x1ef9dae8 sleepq_switch(1f751000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef9db1c sleepq_catch_signals(1ef9db84,10c7a20,1b3f50d0,0,1acb23d0,...) at sleepq_catch_signals+0x47a/frame 0x1ef9db60 sleepq_wait_sig(1acb23d0,0,1ef9dba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef9db84 _cv_wait_sig(1acb23d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1ef9dbc8 svc_run_internal(1ef9dcd4,10a779e,1b3f50c0,1ef9dce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1ef9dca4 svc_thread_start(1b3f50c0,1ef9dce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1ef9dcac fork_exit(13c9720,1b3f50c0,1ef9dce8,3b,3b,...) at fork_exit+0x7e/frame 0x1ef9dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ef9dcd4 --- trap 0x20, eip = 0, esp = 0x1ef9dd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100138 td 0x1f750a80 sched_switch(1f750a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efa2aa8 mi_switch(104,0,1817fc1,22b,1f750a80,...) at mi_switch+0x20e/frame 0x1efa2ae8 sleepq_switch(1f750a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efa2b1c sleepq_catch_signals(1efa2b84,10c7a20,1b3f50d0,0,1acb2450,...) at sleepq_catch_signals+0x47a/frame 0x1efa2b60 sleepq_wait_sig(1acb2450,0,1efa2ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efa2b84 _cv_wait_sig(1acb2450,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efa2bc8 svc_run_internal(1efa2cd4,10a779e,1b3f50c0,1efa2ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efa2ca4 svc_thread_start(1b3f50c0,1efa2ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efa2cac fork_exit(13c9720,1b3f50c0,1efa2ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efa2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efa2cd4 --- trap 0x20, eip = 0, esp = 0x1efa2d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100139 td 0x1f750700 sched_switch(1f750700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efa7aa8 mi_switch(104,0,1817fc1,22b,1f750700,...) at mi_switch+0x20e/frame 0x1efa7ae8 sleepq_switch(1f750700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efa7b1c sleepq_catch_signals(1efa7b84,10c7a20,1b3f50d0,0,1acb24d0,...) at sleepq_catch_signals+0x47a/frame 0x1efa7b60 sleepq_wait_sig(1acb24d0,0,1efa7ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efa7b84 _cv_wait_sig(1acb24d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efa7bc8 svc_run_internal(1efa7cd4,10a779e,1b3f50c0,1efa7ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efa7ca4 svc_thread_start(1b3f50c0,1efa7ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efa7cac fork_exit(13c9720,1b3f50c0,1efa7ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efa7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efa7cd4 --- trap 0x20, eip = 0, esp = 0x1efa7d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100140 td 0x1f750380 sched_switch(1f750380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efacaa8 mi_switch(104,0,1817fc1,22b,1f750380,...) at mi_switch+0x20e/frame 0x1efacae8 sleepq_switch(1f750380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efacb1c sleepq_catch_signals(1efacb84,10c7a20,1b3f50d0,0,1b597d50,...) at sleepq_catch_signals+0x47a/frame 0x1efacb60 sleepq_wait_sig(1b597d50,0,1efacba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efacb84 _cv_wait_sig(1b597d50,1b3f50c0,179d187,481,1,...) at _cv_wait_sig+0x1d7/frame 0x1efacbc8 svc_run_internal(1efaccd4,10a779e,1b3f50c0,1efacce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efacca4 svc_thread_start(1b3f50c0,1efacce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efaccac fork_exit(13c9720,1b3f50c0,1efacce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efaccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efaccd4 --- trap 0x20, eip = 0, esp = 0x1efacd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100141 td 0x1f750000 sched_switch(1f750000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efb1aa8 mi_switch(104,0,1817fc1,22b,1f750000,...) at mi_switch+0x20e/frame 0x1efb1ae8 sleepq_switch(1f750000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efb1b1c sleepq_catch_signals(1efb1b84,10c7a20,1b3f50d0,0,1acb2550,...) at sleepq_catch_signals+0x47a/frame 0x1efb1b60 sleepq_wait_sig(1acb2550,0,1efb1ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efb1b84 _cv_wait_sig(1acb2550,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efb1bc8 svc_run_internal(1efb1cd4,10a779e,1b3f50c0,1efb1ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efb1ca4 svc_thread_start(1b3f50c0,1efb1ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efb1cac fork_exit(13c9720,1b3f50c0,1efb1ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efb1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efb1cd4 --- trap 0x20, eip = 0, esp = 0x1efb1d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100142 td 0x1f74fa80 sched_switch(1f74fa80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efb6aa8 mi_switch(104,0,1817fc1,22b,1f74fa80,...) at mi_switch+0x20e/frame 0x1efb6ae8 sleepq_switch(1f74fa80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efb6b1c sleepq_catch_signals(1efb6b84,10c7a20,1b3f50d0,0,1b597cd0,...) at sleepq_catch_signals+0x47a/frame 0x1efb6b60 sleepq_wait_sig(1b597cd0,0,1efb6ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efb6b84 _cv_wait_sig(1b597cd0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efb6bc8 svc_run_internal(1efb6cd4,10a779e,1b3f50c0,1efb6ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efb6ca4 svc_thread_start(1b3f50c0,1efb6ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efb6cac fork_exit(13c9720,1b3f50c0,1efb6ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efb6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efb6cd4 --- trap 0x20, eip = 0, esp = 0x1efb6d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100143 td 0x1f74f700 sched_switch(1f74f700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efbbaa8 mi_switch(104,0,1817fc1,22b,1f74f700,...) at mi_switch+0x20e/frame 0x1efbbae8 sleepq_switch(1f74f700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efbbb1c sleepq_catch_signals(1efbbb84,10c7a20,1b3f50d0,0,1acb25d0,...) at sleepq_catch_signals+0x47a/frame 0x1efbbb60 sleepq_wait_sig(1acb25d0,0,1efbbba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efbbb84 _cv_wait_sig(1acb25d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efbbbc8 svc_run_internal(1efbbcd4,10a779e,1b3f50c0,1efbbce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efbbca4 svc_thread_start(1b3f50c0,1efbbce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efbbcac fork_exit(13c9720,1b3f50c0,1efbbce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efbbcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efbbcd4 --- trap 0x20, eip = 0, esp = 0x1efbbd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100144 td 0x1f74f380 sched_switch(1f74f380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efc0aa8 mi_switch(104,0,1817fc1,22b,1f74f380,...) at mi_switch+0x20e/frame 0x1efc0ae8 sleepq_switch(1f74f380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efc0b1c sleepq_catch_signals(1efc0b84,10c7a20,1b3f50d0,0,1b597c50,...) at sleepq_catch_signals+0x47a/frame 0x1efc0b60 sleepq_wait_sig(1b597c50,0,1efc0ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efc0b84 _cv_wait_sig(1b597c50,1b3f50c0,179d187,481,1,...) at _cv_wait_sig+0x1d7/frame 0x1efc0bc8 svc_run_internal(1efc0cd4,10a779e,1b3f50c0,1efc0ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efc0ca4 svc_thread_start(1b3f50c0,1efc0ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efc0cac fork_exit(13c9720,1b3f50c0,1efc0ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efc0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efc0cd4 --- trap 0x20, eip = 0, esp = 0x1efc0d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100145 td 0x1f74f000 sched_switch(1f74f000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efc5aa8 mi_switch(104,0,1817fc1,22b,1f74f000,...) at mi_switch+0x20e/frame 0x1efc5ae8 sleepq_switch(1f74f000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efc5b1c sleepq_catch_signals(1efc5b84,10c7a20,1b3f50d0,0,1acb2650,...) at sleepq_catch_signals+0x47a/frame 0x1efc5b60 sleepq_wait_sig(1acb2650,0,1efc5ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efc5b84 _cv_wait_sig(1acb2650,1b3f50c0,179d187,481,2046458b,...) at _cv_wait_sig+0x1d7/frame 0x1efc5bc8 svc_run_internal(1efc5cd4,10a779e,1b3f50c0,1efc5ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efc5ca4 svc_thread_start(1b3f50c0,1efc5ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efc5cac fork_exit(13c9720,1b3f50c0,1efc5ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efc5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efc5cd4 --- trap 0x20, eip = 0, esp = 0x1efc5d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100146 td 0x1f759380 sched_switch(1f759380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efcaaa8 mi_switch(104,0,1817fc1,22b,1f759380,...) at mi_switch+0x20e/frame 0x1efcaae8 sleepq_switch(1f759380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efcab1c sleepq_catch_signals(1efcab84,10c7a20,1b3f50d0,0,1acb26d0,...) at sleepq_catch_signals+0x47a/frame 0x1efcab60 sleepq_wait_sig(1acb26d0,0,1efcaba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efcab84 _cv_wait_sig(1acb26d0,1b3f50c0,179d187,481,205f09b6,...) at _cv_wait_sig+0x1d7/frame 0x1efcabc8 svc_run_internal(1efcacd4,10a779e,1b3f50c0,1efcace8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efcaca4 svc_thread_start(1b3f50c0,1efcace8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efcacac fork_exit(13c9720,1b3f50c0,1efcace8,3b,3b,...) at fork_exit+0x7e/frame 0x1efcacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efcacd4 --- trap 0x20, eip = 0, esp = 0x1efcad20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100147 td 0x1f759000 sched_switch(1f759000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efcfaa8 mi_switch(104,0,1817fc1,22b,1f759000,...) at mi_switch+0x20e/frame 0x1efcfae8 sleepq_switch(1f759000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efcfb1c sleepq_catch_signals(1efcfb84,10c7a20,1b3f50d0,0,1acb2750,...) at sleepq_catch_signals+0x47a/frame 0x1efcfb60 sleepq_wait_sig(1acb2750,0,1efcfba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efcfb84 _cv_wait_sig(1acb2750,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efcfbc8 svc_run_internal(1efcfcd4,10a779e,1b3f50c0,1efcfce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efcfca4 svc_thread_start(1b3f50c0,1efcfce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efcfcac fork_exit(13c9720,1b3f50c0,1efcfce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efcfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efcfcd4 --- trap 0x20, eip = 0, esp = 0x1efcfd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100148 td 0x1f758a80 sched_switch(1f758a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efd4aa8 mi_switch(104,0,1817fc1,22b,1f758a80,...) at mi_switch+0x20e/frame 0x1efd4ae8 sleepq_switch(1f758a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efd4b1c sleepq_catch_signals(1efd4b84,10c7a20,1b3f50d0,0,1b597bd0,...) at sleepq_catch_signals+0x47a/frame 0x1efd4b60 sleepq_wait_sig(1b597bd0,0,1efd4ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efd4b84 _cv_wait_sig(1b597bd0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efd4bc8 svc_run_internal(1efd4cd4,10a779e,1b3f50c0,1efd4ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efd4ca4 svc_thread_start(1b3f50c0,1efd4ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efd4cac fork_exit(13c9720,1b3f50c0,1efd4ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efd4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efd4cd4 --- trap 0x20, eip = 0, esp = 0x1efd4d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100149 td 0x1f758700 sched_switch(1f758700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efd9aa8 mi_switch(104,0,1817fc1,22b,1f758700,...) at mi_switch+0x20e/frame 0x1efd9ae8 sleepq_switch(1f758700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efd9b1c sleepq_catch_signals(1efd9b84,10c7a20,1b3f50d0,0,1acb27d0,...) at sleepq_catch_signals+0x47a/frame 0x1efd9b60 sleepq_wait_sig(1acb27d0,0,1efd9ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efd9b84 _cv_wait_sig(1acb27d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efd9bc8 svc_run_internal(1efd9cd4,10a779e,1b3f50c0,1efd9ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efd9ca4 svc_thread_start(1b3f50c0,1efd9ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efd9cac fork_exit(13c9720,1b3f50c0,1efd9ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efd9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efd9cd4 --- trap 0x20, eip = 0, esp = 0x1efd9d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100150 td 0x1f758380 sched_switch(1f758380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efdeaa8 mi_switch(104,0,1817fc1,22b,1f758380,...) at mi_switch+0x20e/frame 0x1efdeae8 sleepq_switch(1f758380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efdeb1c sleepq_catch_signals(1efdeb84,10c7a20,1b3f50d0,0,1acb2850,...) at sleepq_catch_signals+0x47a/frame 0x1efdeb60 sleepq_wait_sig(1acb2850,0,1efdeba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efdeb84 _cv_wait_sig(1acb2850,1b3f50c0,179d187,481,2053a000,...) at _cv_wait_sig+0x1d7/frame 0x1efdebc8 svc_run_internal(1efdecd4,10a779e,1b3f50c0,1efdece8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efdeca4 svc_thread_start(1b3f50c0,1efdece8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efdecac fork_exit(13c9720,1b3f50c0,1efdece8,3b,3b,...) at fork_exit+0x7e/frame 0x1efdecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efdecd4 --- trap 0x20, eip = 0, esp = 0x1efded20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100151 td 0x1f758000 sched_switch(1f758000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efe3aa8 mi_switch(104,0,1817fc1,22b,1f758000,...) at mi_switch+0x20e/frame 0x1efe3ae8 sleepq_switch(1f758000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efe3b1c sleepq_catch_signals(1efe3b84,10c7a20,1b3f50d0,0,1acb28d0,...) at sleepq_catch_signals+0x47a/frame 0x1efe3b60 sleepq_wait_sig(1acb28d0,0,1efe3ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efe3b84 _cv_wait_sig(1acb28d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efe3bc8 svc_run_internal(1efe3cd4,10a779e,1b3f50c0,1efe3ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efe3ca4 svc_thread_start(1b3f50c0,1efe3ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efe3cac fork_exit(13c9720,1b3f50c0,1efe3ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efe3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efe3cd4 --- trap 0x20, eip = 0, esp = 0x1efe3d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100152 td 0x1f757a80 sched_switch(1f757a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efe8aa8 mi_switch(104,0,1817fc1,22b,1f757a80,...) at mi_switch+0x20e/frame 0x1efe8ae8 sleepq_switch(1f757a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efe8b1c sleepq_catch_signals(1efe8b84,10c7a20,1b3f50d0,0,1b597b50,...) at sleepq_catch_signals+0x47a/frame 0x1efe8b60 sleepq_wait_sig(1b597b50,0,1efe8ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efe8b84 _cv_wait_sig(1b597b50,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efe8bc8 svc_run_internal(1efe8cd4,10a779e,1b3f50c0,1efe8ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efe8ca4 svc_thread_start(1b3f50c0,1efe8ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efe8cac fork_exit(13c9720,1b3f50c0,1efe8ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efe8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efe8cd4 --- trap 0x20, eip = 0, esp = 0x1efe8d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100153 td 0x1f757700 sched_switch(1f757700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1efedaa8 mi_switch(104,0,1817fc1,22b,1f757700,...) at mi_switch+0x20e/frame 0x1efedae8 sleepq_switch(1f757700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1efedb1c sleepq_catch_signals(1efedb84,10c7a20,1b3f50d0,0,1acb2950,...) at sleepq_catch_signals+0x47a/frame 0x1efedb60 sleepq_wait_sig(1acb2950,0,1efedba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efedb84 _cv_wait_sig(1acb2950,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1efedbc8 svc_run_internal(1efedcd4,10a779e,1b3f50c0,1efedce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1efedca4 svc_thread_start(1b3f50c0,1efedce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1efedcac fork_exit(13c9720,1b3f50c0,1efedce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efedcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efedcd4 --- trap 0x20, eip = 0, esp = 0x1efedd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100154 td 0x1f757380 sched_switch(1f757380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eff2aa8 mi_switch(104,0,1817fc1,22b,1f757380,...) at mi_switch+0x20e/frame 0x1eff2ae8 sleepq_switch(1f757380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eff2b1c sleepq_catch_signals(1eff2b84,10c7a20,1b3f50d0,0,1acb29d0,...) at sleepq_catch_signals+0x47a/frame 0x1eff2b60 sleepq_wait_sig(1acb29d0,0,1eff2ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eff2b84 _cv_wait_sig(1acb29d0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1eff2bc8 svc_run_internal(1eff2cd4,10a779e,1b3f50c0,1eff2ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1eff2ca4 svc_thread_start(1b3f50c0,1eff2ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1eff2cac fork_exit(13c9720,1b3f50c0,1eff2ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1eff2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eff2cd4 --- trap 0x20, eip = 0, esp = 0x1eff2d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100155 td 0x1f757000 sched_switch(1f757000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eff7aa8 mi_switch(104,0,1817fc1,22b,1f757000,...) at mi_switch+0x20e/frame 0x1eff7ae8 sleepq_switch(1f757000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eff7b1c sleepq_catch_signals(1eff7b84,10c7a20,1b3f50d0,0,1acb2a50,...) at sleepq_catch_signals+0x47a/frame 0x1eff7b60 sleepq_wait_sig(1acb2a50,0,1eff7ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eff7b84 _cv_wait_sig(1acb2a50,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1eff7bc8 svc_run_internal(1eff7cd4,10a779e,1b3f50c0,1eff7ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1eff7ca4 svc_thread_start(1b3f50c0,1eff7ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1eff7cac fork_exit(13c9720,1b3f50c0,1eff7ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1eff7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eff7cd4 --- trap 0x20, eip = 0, esp = 0x1eff7d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100156 td 0x1f752a80 sched_switch(1f752a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1effcaa8 mi_switch(104,0,1817fc1,22b,1f752a80,...) at mi_switch+0x20e/frame 0x1effcae8 sleepq_switch(1f752a80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1effcb1c sleepq_catch_signals(1effcb84,10c7a20,1b3f50d0,0,1acb2ad0,...) at sleepq_catch_signals+0x47a/frame 0x1effcb60 sleepq_wait_sig(1acb2ad0,0,1effcba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1effcb84 _cv_wait_sig(1acb2ad0,1b3f50c0,179d187,481,2044a0e1,...) at _cv_wait_sig+0x1d7/frame 0x1effcbc8 svc_run_internal(1effccd4,10a779e,1b3f50c0,1effcce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1effcca4 svc_thread_start(1b3f50c0,1effcce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1effccac fork_exit(13c9720,1b3f50c0,1effcce8,3b,3b,...) at fork_exit+0x7e/frame 0x1effccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1effccd4 --- trap 0x20, eip = 0, esp = 0x1effcd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100157 td 0x1f752700 sched_switch(1f752700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1da93aa8 mi_switch(104,0,1817fc1,22b,1f752700,...) at mi_switch+0x20e/frame 0x1da93ae8 sleepq_switch(1f752700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1da93b1c sleepq_catch_signals(1da93b84,10c7a20,1b3f50d0,0,1acb2b50,...) at sleepq_catch_signals+0x47a/frame 0x1da93b60 sleepq_wait_sig(1acb2b50,0,1da93ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1da93b84 _cv_wait_sig(1acb2b50,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1da93bc8 svc_run_internal(1da93cd4,10a779e,1b3f50c0,1da93ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1da93ca4 svc_thread_start(1b3f50c0,1da93ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1da93cac fork_exit(13c9720,1b3f50c0,1da93ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1da93cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1da93cd4 --- trap 0x20, eip = 0, esp = 0x1da93d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100158 td 0x1f752380 sched_switch(1f752380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1da98aa8 mi_switch(104,0,1817fc1,22b,1f752380,...) at mi_switch+0x20e/frame 0x1da98ae8 sleepq_switch(1f752380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1da98b1c sleepq_catch_signals(1da98b84,10c7a20,1b3f50d0,0,1acb2bd0,...) at sleepq_catch_signals+0x47a/frame 0x1da98b60 sleepq_wait_sig(1acb2bd0,0,1da98ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1da98b84 _cv_wait_sig(1acb2bd0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1da98bc8 svc_run_internal(1da98cd4,10a779e,1b3f50c0,1da98ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1da98ca4 svc_thread_start(1b3f50c0,1da98ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1da98cac fork_exit(13c9720,1b3f50c0,1da98ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1da98cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1da98cd4 --- trap 0x20, eip = 0, esp = 0x1da98d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100159 td 0x1f75c700 sched_switch(1f75c700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1da9daa8 mi_switch(104,0,1817fc1,22b,1f75c700,...) at mi_switch+0x20e/frame 0x1da9dae8 sleepq_switch(1f75c700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1da9db1c sleepq_catch_signals(1da9db84,10c7a20,1b3f50d0,0,1b597ad0,...) at sleepq_catch_signals+0x47a/frame 0x1da9db60 sleepq_wait_sig(1b597ad0,0,1da9dba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1da9db84 _cv_wait_sig(1b597ad0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1da9dbc8 svc_run_internal(1da9dcd4,10a779e,1b3f50c0,1da9dce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1da9dca4 svc_thread_start(1b3f50c0,1da9dce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1da9dcac fork_exit(13c9720,1b3f50c0,1da9dce8,3b,3b,...) at fork_exit+0x7e/frame 0x1da9dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1da9dcd4 --- trap 0x20, eip = 0, esp = 0x1da9dd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100160 td 0x1f75c380 sched_switch(1f75c380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1daa2aa8 mi_switch(104,0,1817fc1,22b,1f75c380,...) at mi_switch+0x20e/frame 0x1daa2ae8 sleepq_switch(1f75c380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1daa2b1c sleepq_catch_signals(1daa2b84,10c7a20,1b3f50d0,0,1b597a50,...) at sleepq_catch_signals+0x47a/frame 0x1daa2b60 sleepq_wait_sig(1b597a50,0,1daa2ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daa2b84 _cv_wait_sig(1b597a50,1b3f50c0,179d187,481,2053a000,...) at _cv_wait_sig+0x1d7/frame 0x1daa2bc8 svc_run_internal(1daa2cd4,10a779e,1b3f50c0,1daa2ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1daa2ca4 svc_thread_start(1b3f50c0,1daa2ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1daa2cac fork_exit(13c9720,1b3f50c0,1daa2ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daa2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daa2cd4 --- trap 0x20, eip = 0, esp = 0x1daa2d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100161 td 0x1f75c000 sched_switch(1f75c000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1daa7aa8 mi_switch(104,0,1817fc1,22b,1f75c000,...) at mi_switch+0x20e/frame 0x1daa7ae8 sleepq_switch(1f75c000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1daa7b1c sleepq_catch_signals(1daa7b84,10c7a20,1b3f50d0,0,1acb2c50,...) at sleepq_catch_signals+0x47a/frame 0x1daa7b60 sleepq_wait_sig(1acb2c50,0,1daa7ba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daa7b84 _cv_wait_sig(1acb2c50,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1daa7bc8 svc_run_internal(1daa7cd4,10a779e,1b3f50c0,1daa7ce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1daa7ca4 svc_thread_start(1b3f50c0,1daa7ce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1daa7cac fork_exit(13c9720,1b3f50c0,1daa7ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daa7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daa7cd4 --- trap 0x20, eip = 0, esp = 0x1daa7d20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 559 tid 100162 td 0x1f75ba80 sched_switch(1f75ba80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1daacaa8 mi_switch(104,0,1817fc1,22b,1f75ba80,...) at mi_switch+0x20e/frame 0x1daacae8 sleepq_switch(1f75ba80,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1daacb1c sleepq_catch_signals(1daacb84,10c7a20,1b3f50d0,0,1acb2cd0,...) at sleepq_catch_signals+0x47a/frame 0x1daacb60 sleepq_wait_sig(1acb2cd0,0,1daacba4,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daacb84 _cv_wait_sig(1acb2cd0,1b3f50c0,179d187,481,0,...) at _cv_wait_sig+0x1d7/frame 0x1daacbc8 svc_run_internal(1daaccd4,10a779e,1b3f50c0,1daacce8,17d387d,...) at svc_run_internal+0xd49/frame 0x1daacca4 svc_thread_start(1b3f50c0,1daacce8,17d387d,419,0,...) at svc_thread_start+0xd/frame 0x1daaccac fork_exit(13c9720,1b3f50c0,1daacce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daaccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daaccd4 --- trap 0x20, eip = 0, esp = 0x1daacd20, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 556 tid 100130 td 0x1f70d000 sched_switch(1f70d000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef7a8c0 mi_switch(104,0,1817fc1,22b,1f70d000,...) at mi_switch+0x20e/frame 0x1ef7a900 sleepq_switch(1f70d000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef7a934 sleepq_catch_signals(1ef7a99c,10c7a20,be6f6a0,0,be6f6a4,...) at sleepq_catch_signals+0x47a/frame 0x1ef7a978 sleepq_wait_sig(be6f6a4,0,1ef7a9bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef7a99c _cv_wait_sig(be6f6a4,be6f690,1733ba7,6fa,1f340c30,...) at _cv_wait_sig+0x1d7/frame 0x1ef7a9e0 seltdwait(ffffffff,0,0,1f70d000,0,...) at seltdwait+0xbb/frame 0x1ef7aa10 kern_select(1f70d000,7,ffbfed40,0,0,...) at kern_select+0x894/frame 0x1ef7aba0 sys_select(1f70d000,1f70d284,18047f7,4,17ddfd6,...) at sys_select+0x68/frame 0x1ef7abe8 syscall(1ef7ace8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1ef7acdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef7acdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205dafdf, esp = 0xffbfe9c4, ebp = 0xffbfe9e8 --- amdsmn_devclass() at 0x205dafdf Tracing command mountd pid 550 tid 100128 td 0x1f70d700 sched_switch(1f70d700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef6f8c0 mi_switch(104,0,1817fc1,22b,1f70d700,...) at mi_switch+0x20e/frame 0x1ef6f900 sleepq_switch(1f70d700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef6f934 sleepq_catch_signals(1ef6f99c,10c7a20,1f33b5e0,0,1f33b5e4,...) at sleepq_catch_signals+0x47a/frame 0x1ef6f978 sleepq_wait_sig(1f33b5e4,0,1ef6f9bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef6f99c _cv_wait_sig(1f33b5e4,1f33b5d0,1733ba7,6fa,1f4e2618,...) at _cv_wait_sig+0x1d7/frame 0x1ef6f9e0 seltdwait(ffffffff,0,0,1f70d700,0,...) at seltdwait+0xbb/frame 0x1ef6fa10 kern_select(1f70d700,a,ffbfe8e0,0,0,...) at kern_select+0x894/frame 0x1ef6fba0 sys_select(1f70d700,1f70d984,18047f7,4,1,...) at sys_select+0x68/frame 0x1ef6fbe8 syscall(1ef6fce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1ef6fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef6fcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205f3fdf, esp = 0xffbfe84c, ebp = 0xffbfe870 --- amdsmn_devclass() at 0x205f3fdf Tracing command nfsuserd pid 544 tid 100109 td 0x9f7d000 sched_switch(9f7d000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eea78a0 mi_switch(104,0,1817fc1,22b,9f7d000,...) at mi_switch+0x20e/frame 0x1eea78e0 sleepq_switch(9f7d000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eea7914 sleepq_catch_signals(8,17cad96,cb,0,9f7d000,...) at sleepq_catch_signals+0x47a/frame 0x1eea7958 sleepq_timedwait_sig(1b43dc24,0,1eea79bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eea7998 _cv_timedwait_sig_sbt(1b43dc24,1b43dc10,7fc35d0e,9f61,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eea79e0 seltdwait(9f61,e0000000,1,9f7d000,0,...) at seltdwait+0x82/frame 0x1eea7a10 kern_select(9f7d000,4,ffbfdbe8,0,0,...) at kern_select+0x894/frame 0x1eea7ba0 sys_select(9f7d000,9f7d284,18047f7,4,be82d90,...) at sys_select+0x68/frame 0x1eea7be8 syscall(1eea7ce8,3b,3b,3b,ffbfdbe8,...) at syscall+0x363/frame 0x1eea7cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eea7cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205d8fdf, esp = 0xffbfdb34, ebp = 0xffbfdc80 --- amdsmn_devclass() at 0x205d8fdf Tracing command nfsuserd pid 542 tid 100117 td 0x9f7c380 sched_switch(9f7c380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eee18a0 mi_switch(104,0,1817fc1,22b,9f7c380,...) at mi_switch+0x20e/frame 0x1eee18e0 sleepq_switch(9f7c380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eee1914 sleepq_catch_signals(8,17cad96,cb,0,9f7c380,...) at sleepq_catch_signals+0x47a/frame 0x1eee1958 sleepq_timedwait_sig(1f33b624,0,1eee19bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eee1998 _cv_timedwait_sig_sbt(1f33b624,1f33b610,c9a4db,9f65,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eee19e0 seltdwait(9f65,e0000000,1,9f7c380,0,...) at seltdwait+0x82/frame 0x1eee1a10 kern_select(9f7c380,4,ffbfdbe8,0,0,...) at kern_select+0x894/frame 0x1eee1ba0 sys_select(9f7c380,9f7c604,18047f7,4,be82d90,...) at sys_select+0x68/frame 0x1eee1be8 syscall(1eee1ce8,3b,3b,3b,ffbfdbe8,...) at syscall+0x363/frame 0x1eee1cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eee1cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205d8fdf, esp = 0xffbfdb34, ebp = 0xffbfdc80 --- amdsmn_devclass() at 0x205d8fdf Tracing command nfsuserd pid 541 tid 100112 td 0x1b036000 sched_switch(1b036000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eec28a0 mi_switch(104,0,1817fc1,22b,1b036000,...) at mi_switch+0x20e/frame 0x1eec28e0 sleepq_switch(1b036000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eec2914 sleepq_catch_signals(8,17cad96,cb,0,1b036000,...) at sleepq_catch_signals+0x47a/frame 0x1eec2958 sleepq_timedwait_sig(1b43dc64,0,1eec29bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eec2998 _cv_timedwait_sig_sbt(1b43dc64,1b43dc50,6b5b094d,9f64,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eec29e0 seltdwait(9f64,e0000000,1,1b036000,0,...) at seltdwait+0x82/frame 0x1eec2a10 kern_select(1b036000,4,ffbfdbe8,0,0,...) at kern_select+0x894/frame 0x1eec2ba0 sys_select(1b036000,1b036284,18047f7,4,2095a90,...) at sys_select+0x68/frame 0x1eec2be8 syscall(1eec2ce8,3b,3b,3b,ffbfdbe8,...) at syscall+0x363/frame 0x1eec2cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eec2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205d8fdf, esp = 0xffbfdb34, ebp = 0xffbfdc80 --- amdsmn_devclass() at 0x205d8fdf Tracing command nfsuserd pid 540 tid 100119 td 0x9f7c000 sched_switch(9f7c000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eeed8a0 mi_switch(104,0,1817fc1,22b,9f7c000,...) at mi_switch+0x20e/frame 0x1eeed8e0 sleepq_switch(9f7c000,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eeed914 sleepq_catch_signals(8,17cad96,cb,0,9f7c000,...) at sleepq_catch_signals+0x47a/frame 0x1eeed958 sleepq_timedwait_sig(1f33b664,0,1eeed9bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eeed998 _cv_timedwait_sig_sbt(1f33b664,1f33b650,8f608aa1,9f5f,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eeed9e0 seltdwait(9f5f,e0000000,1,9f7c000,0,...) at seltdwait+0x82/frame 0x1eeeda10 kern_select(9f7c000,4,ffbfdbe8,0,0,...) at kern_select+0x894/frame 0x1eeedba0 sys_select(9f7c000,9f7c284,18047f7,4,be7dd90,...) at sys_select+0x68/frame 0x1eeedbe8 syscall(1eeedce8,3b,3b,3b,ffbfdbe8,...) at syscall+0x363/frame 0x1eeedcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eeedcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205d8fdf, esp = 0xffbfdb34, ebp = 0xffbfdc80 --- amdsmn_devclass() at 0x205d8fdf Tracing command nfsuserd pid 539 tid 100127 td 0x1f70da80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1847866,0,183cc4c,10b,1511f4b0,...) at db_trace_self_wrapper+0x2a/frame 0x1511f430 kdb_backtrace(1746d2a,16d615f,1511f5b0,16e79fe,1511f4e4,...) at kdb_backtrace+0x2d/frame 0x1511f498 kdb_reenter(1511f4e4,114b51f,1dac874,0,17504fd,...) at kdb_reenter+0x2a/frame 0x1511f4a8 trap(1511f5bc,8,28,28,ffffffff,...) at trap+0xde/frame 0x1511f5b0 calltrap() at PTDpde+0x4165/frame 0x1511f5b0 --- trap 0xc, eip = 0x16cf489, esp = 0x1511f5fc, ebp = 0x1511f618 --- db_trace_thread(1f70da80,ffffffff,21b,1871f,1f70da80,...) at db_trace_thread+0x29/frame 0x1511f618 _db_stack_trace_all(1135fba,0,ffffffff,1511f6b0,1e3bad0,...) at _db_stack_trace_all+0xe9/frame 0x1511f684 db_command(1e3c8fa,0,1e3c4ec,1e3c8f9,1511f768,...) at db_command+0x297/frame 0x1511f738 db_script_exec(21188f8,0,1511f7e0,1511f7d4,1e3bad0,...) at db_script_exec+0x168/frame 0x1511f7a8 db_command(17f635e,1511f8c0,9f92cd,1812f68,0,...) at db_command+0x297/frame 0x1511f85c db_command_loop(1812f68,0,1511f887,1511f886,1dac7a0,...) at db_command_loop+0x50/frame 0x1511f868 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x1511f8c0 kdb_trap(3,0,1511fa0c,1520fa01,1511f964,...) at kdb_trap+0x11e/frame 0x1511f8f8 trap(1511fa0c,8,28,28,100,...) at trap+0x3c2/frame 0x1511fa00 calltrap() at PTDpde+0x4165/frame 0x1511fa00 --- trap 0x3, eip = 0x1135fba, esp = 0x1511fa4c, ebp = 0x1511fa58 --- kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command watchdogd pid 502 tid 100125 td 0x1f50b380 sched_switch(1f50b380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef3d9e8 mi_switch(104,0,1817fc1,22b,1f50b380,...) at mi_switch+0x20e/frame 0x1ef3da28 sleepq_switch(1f50b380,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x1ef3da5c sleepq_catch_signals(1f50b380,0,601,0,9f74,...) at sleepq_catch_signals+0x47a/frame 0x1ef3daa0 sleepq_timedwait_sig(208740c,6c,9f74,c0000000,3,...) at sleepq_timedwait_sig+0x17/frame 0x1ef3dae0 _sleep(208740c,0,16c,182aa35,e045f3d0,...) at _sleep+0x2ed/frame 0x1ef3db30 kern_clock_nanosleep(1f50b380,0,0,1ef3dbc8,1ef3dbc0,...) at kern_clock_nanosleep+0x1f4/frame 0x1ef3dba0 sys_nanosleep(1f50b380,1f50b604,18047f7,4,0,...) at sys_nanosleep+0x7f/frame 0x1ef3dbe8 syscall(1ef3dce8,3b,3b,3b,3c,...) at syscall+0x363/frame 0x1ef3dcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef3dcdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2057f443, esp = 0xffbfec64, ebp = 0xffbfec98 --- amdsmn_devclass() at 0x2057f443 Tracing command rpcbind pid 418 tid 100111 td 0x9f7c700 sched_switch(9f7c700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1eebc8d8 mi_switch(104,0,1817fc1,22b,9f7c700,...) at mi_switch+0x20e/frame 0x1eebc918 sleepq_switch(9f7c700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1eebc94c sleepq_catch_signals(8,17cad96,cb,0,9f7c700,...) at sleepq_catch_signals+0x47a/frame 0x1eebc990 sleepq_timedwait_sig(1b43dca4,0,1eebc9f4,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eebc9d0 _cv_timedwait_sig_sbt(1b43dca4,1b43dc90,9d032bb1,9f4f,e0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1eebca18 seltdwait(9f4f,e0000000,1,9f7c700,20863a0,...) at seltdwait+0x82/frame 0x1eebca48 kern_poll(9f7c700,ffbfcdb0,8,1eebcbd0,0,...) at kern_poll+0x4d4/frame 0x1eebcbb0 sys_poll(9f7c700,9f7c984,18047f7,4,2094d10,...) at sys_poll+0x67/frame 0x1eebcbe8 syscall(1eebcce8,3b,3b,3b,40,...) at syscall+0x363/frame 0x1eebccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eebccdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2058d73b, esp = 0xffbfcbe4, ebp = 0xffbfcbf8 --- amdsmn_devclass() at 0x2058d73b Tracing command syslogd pid 409 tid 100123 td 0x1b402380 sched_switch(1b402380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef198c0 mi_switch(104,0,1817fc1,22b,1b402380,...) at mi_switch+0x20e/frame 0x1ef19900 sleepq_switch(1b402380,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef19934 sleepq_catch_signals(1ef1999c,10c7a20,1b03ae20,0,1b03ae24,...) at sleepq_catch_signals+0x47a/frame 0x1ef19978 sleepq_wait_sig(1b03ae24,0,1ef199bc,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef1999c _cv_wait_sig(1b03ae24,1b03ae10,1733ba7,6fa,1f37a208,...) at _cv_wait_sig+0x1d7/frame 0x1ef199e0 seltdwait(ffffffff,0,0,1b402380,0,...) at seltdwait+0xbb/frame 0x1ef19a10 kern_select(1b402380,a,20660000,0,0,...) at kern_select+0x894/frame 0x1ef19ba0 sys_select(1b402380,1b402604,18047f7,4,1,...) at sys_select+0x68/frame 0x1ef19be8 syscall(1ef19ce8,3b,3b,3b,0,...) at syscall+0x363/frame 0x1ef19cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef19cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x205f4fdf, esp = 0xffbfe87c, ebp = 0xffbfe8a0 --- amdsmn_devclass() at 0x205f4fdf Tracing command devd pid 337 tid 100124 td 0x1f50b700 sched_switch(1f50b700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ef338a0 mi_switch(104,0,1817fc1,22b,1f50b700,...) at mi_switch+0x20e/frame 0x1ef338e0 sleepq_switch(1f50b700,0,1817fc1,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef33914 sleepq_catch_signals(8,17cad96,cb,0,1f50b700,...) at sleepq_catch_signals+0x47a/frame 0x1ef33958 sleepq_timedwait_sig(be6f8a4,0,1ef339bc,c0000000,3,...) at sleepq_timedwait_sig+0x17/frame 0x1ef33998 _cv_timedwait_sig_sbt(be6f8a4,be6f890,8e2a1bc7,9f85,c0000000,...) at _cv_timedwait_sig_sbt+0x205/frame 0x1ef339e0 seltdwait(9f85,c0000000,3,1f50b700,0,...) at seltdwait+0x82/frame 0x1ef33a10 kern_select(1f50b700,6,ffbfcda0,0,0,...) at kern_select+0x894/frame 0x1ef33ba0 sys_select(1f50b700,1f50b984,18047f7,4,173708f,...) at sys_select+0x68/frame 0x1ef33be8 syscall(1ef33ce8,3b,3b,3b,3c,...) at syscall+0x363/frame 0x1ef33cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef33cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x4d892b, esp = 0xffbfcd0c, ebp = 0xffbfee38 --- (null)() at 0x4d892b Tracing command syncer pid 25 tid 100101 td 0x1b037000 sched_switch(1b037000,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x15178b48 mi_switch(104,0,1817fc1,22b,1b037000,...) at mi_switch+0x20e/frame 0x15178b84 sleepq_switch(1b037000,0,1817fc1,2d6,1b037000,...) at sleepq_switch+0x110/frame 0x15178bb8 sleepq_timedwait(2101ba0,0,15178c1c,0,0,...) at sleepq_timedwait+0x52/frame 0x15178bf8 _cv_timedwait_sbt(2101ba0,2101b80,fffffed8,0,0,...) at _cv_timedwait_sbt+0x205/frame 0x15178c40 sched_sync(0,15178ce8,17d387d,419,0,...) at sched_sync+0x6e0/frame 0x15178cac fork_exit(11c9170,0,15178ce8,0,0,...) at fork_exit+0x7e/frame 0x15178cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15178cd4 --- trap 0, eip = 0, esp = 0x15178d20, ebp = 0 --- (null)() at 0 Tracing command vnlru pid 24 tid 100100 td 0x1b037380 sched_switch(1b037380,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x15173b30 mi_switch(104,0,1817fc1,22b,1b037380,...) at mi_switch+0x20e/frame 0x15173b6c sleepq_switch(1b037380,0,1817fc1,2d6,9f4a,...) at sleepq_switch+0x110/frame 0x15173ba0 sleepq_timedwait(1af9c000,60,0,0,0,...) at sleepq_timedwait+0x52/frame 0x15173be0 _sleep(1af9c000,2101b4c,260,183ea5c,fffffed8,...) at _sleep+0x304/frame 0x15173c30 vnlru_proc(0,15173ce8,17d387d,419,0,...) at vnlru_proc+0xbc/frame 0x15173cac fork_exit(11c8790,0,15173ce8,0,0,...) at fork_exit+0x7e/frame 0x15173cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15173cd4 --- trap 0, eip = 0, esp = 0x15173d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100099 td 0x1b037700 sched_switch(1b037700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1516eb68 mi_switch(104,0,1817fc1,22b,1b037700,...) at mi_switch+0x20e/frame 0x1516eba4 sleepq_switch(1b037700,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x1516ebd8 sleepq_timedwait(210152c,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1516ec18 _sleep(210152c,1dce080,54,1762f58,1999997c,...) at _sleep+0x304/frame 0x1516ec68 buf_daemon(0,1516ece8,17d387d,419,0,...) at buf_daemon+0xfc/frame 0x1516ecac fork_exit(11a5c10,0,1516ece8,0,0,...) at fork_exit+0x7e/frame 0x1516ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1516ecd4 --- trap 0, eip = 0, esp = 0x1516ed20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100103 td 0x9f7da80 sched_switch(9f7da80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1ee82b60 mi_switch(104,0,1817fc1,22b,9f7da80,...) at mi_switch+0x20e/frame 0x1ee82b9c sleepq_switch(9f7da80,0,1817fc1,2d6,9f4a,...) at sleepq_switch+0x110/frame 0x1ee82bd0 sleepq_timedwait(1dcf380,5c,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1ee82c10 _sleep(1dcf380,1dcf2c0,25c,1766623,fffffed8,...) at _sleep+0x304/frame 0x1ee82c60 bufspace_daemon(1dce180,1ee82ce8,17d387d,419,0,...) at bufspace_daemon+0x369/frame 0x1ee82cac fork_exit(11a6000,1dce180,1ee82ce8,0,0,...) at fork_exit+0x7e/frame 0x1ee82cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee82cd4 --- trap 0, eip = 0, esp = 0x1ee82d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100113 td 0x1b552700 sched_switch(1b552700,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1eec8b70 mi_switch(104,0,1817fc1,22b,1b552700,...) at mi_switch+0x20e/frame 0x1eec8bac sleepq_switch(1b552700,0,1817fc1,2d6,9f4a,...) at sleepq_switch+0x110/frame 0x1eec8be0 sleepq_timedwait(1b572084,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1eec8c20 _sleep(1b572084,1b572000,54,179409e,7fffff6c,...) at _sleep+0x304/frame 0x1eec8c70 softdep_flush(1bb6f000,1eec8ce8,17d387d,419,0,...) at softdep_flush+0x199/frame 0x1eec8cac fork_exit(140d800,1bb6f000,1eec8ce8,0,0,...) at fork_exit+0x7e/frame 0x1eec8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eec8cd4 --- trap 0, eip = 0, esp = 0x1eec8d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100114 td 0x1b552380 sched_switch(1b552380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1eecdb70 mi_switch(104,0,1817fc1,22b,1b552380,...) at mi_switch+0x20e/frame 0x1eecdbac sleepq_switch(1b552380,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x1eecdbe0 sleepq_timedwait(1b571e84,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1eecdc20 _sleep(1b571e84,1b571e00,54,179409e,7fffff6c,...) at _sleep+0x304/frame 0x1eecdc70 softdep_flush(1bb6e000,1eecdce8,17d387d,419,0,...) at softdep_flush+0x199/frame 0x1eecdcac fork_exit(140d800,1bb6e000,1eecdce8,0,0,...) at fork_exit+0x7e/frame 0x1eecdcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eecdcd4 --- trap 0, eip = 0, esp = 0x1eecdd20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100115 td 0x1b552000 sched_switch(1b552000,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1eed2b70 mi_switch(104,0,1817fc1,22b,1b552000,...) at mi_switch+0x20e/frame 0x1eed2bac sleepq_switch(1b552000,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x1eed2be0 sleepq_timedwait(1b571c84,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1eed2c20 _sleep(1b571c84,1b571c00,54,179409e,7fffff6c,...) at _sleep+0x304/frame 0x1eed2c70 softdep_flush(1bb6d000,1eed2ce8,17d387d,419,0,...) at softdep_flush+0x199/frame 0x1eed2cac fork_exit(140d800,1bb6d000,1eed2ce8,0,0,...) at fork_exit+0x7e/frame 0x1eed2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eed2cd4 --- trap 0, eip = 0, esp = 0x1eed2d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100116 td 0x1b402a80 sched_switch(1b402a80,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1eed7b70 mi_switch(104,0,1817fc1,22b,1b402a80,...) at mi_switch+0x20e/frame 0x1eed7bac sleepq_switch(1b402a80,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x1eed7be0 sleepq_timedwait(1b571a84,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x1eed7c20 _sleep(1b571a84,1b571a00,54,179409e,7fffff6c,...) at _sleep+0x304/frame 0x1eed7c70 softdep_flush(1bbff000,1eed7ce8,17d387d,419,0,...) at softdep_flush+0x199/frame 0x1eed7cac fork_exit(140d800,1bbff000,1eed7ce8,0,0,...) at fork_exit+0x7e/frame 0x1eed7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eed7cd4 --- trap 0, eip = 0, esp = 0x1eed7d20, ebp = 0 --- (null)() at 0 Tracing command vmdaemon pid 22 tid 100098 td 0x1b037a80 sched_switch(1b037a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x15169b50 mi_switch(104,0,1817fc1,22b,1b037a80,...) at mi_switch+0x20e/frame 0x15169b90 sleepq_switch(1b037a80,0,1817fc1,2b3,1b037a80,...) at sleepq_switch+0x110/frame 0x15169bc4 sleepq_wait(21170b8,74,1762f51,0,0,...) at sleepq_wait+0x48/frame 0x15169be8 _sleep(21170b8,21170a4,74,1762f51,0,...) at _sleep+0x36a/frame 0x15169c38 vm_daemon(0,15169ce8,17d387d,419,0,...) at vm_daemon+0xc76/frame 0x15169cac fork_exit(147c640,0,15169ce8,0,0,...) at fork_exit+0x7e/frame 0x15169cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15169cd4 --- trap 0, eip = 0, esp = 0x15169d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100097 td 0x1b038000 sched_switch(1b038000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x15164978 mi_switch(104,0,1817fc1,22b,1b038000,...) at mi_switch+0x20e/frame 0x151649b4 sleepq_switch(1b038000,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x151649e8 sleepq_timedwait(213df5c,54,0,0,0,...) at sleepq_timedwait+0x52/frame 0x15164a28 _sleep(213df5c,213dc80,254,1762f51,1999997c,...) at _sleep+0x304/frame 0x15164a78 vm_pageout_worker(0,0,1af9c9fc,0,0,...) at vm_pageout_worker+0x2b7/frame 0x15164c7c vm_pageout(0,15164ce8,17d387d,419,0,...) at vm_pageout+0x16a/frame 0x15164cac fork_exit(1471090,0,15164ce8,0,0,...) at fork_exit+0x7e/frame 0x15164cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15164cd4 --- trap 0, eip = 0, esp = 0x15164d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100102 td 0x1b036a80 sched_switch(1b036a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1ee7da10 mi_switch(104,0,1817fc1,22b,1b036a80,...) at mi_switch+0x20e/frame 0x1ee7da50 sleepq_switch(1b036a80,0,1817fc1,2b3,1b036a80,...) at sleepq_switch+0x110/frame 0x1ee7da84 sleepq_wait(213df68,54,1853c46,0,0,...) at sleepq_wait+0x48/frame 0x1ee7daa8 _sleep(213df68,213dbc0,54,1853c46,0,...) at _sleep+0x36a/frame 0x1ee7daf8 vm_pageout_laundry_worker(0,1ee7dce8,17d387d,419,0,...) at vm_pageout_laundry_worker+0xfda/frame 0x1ee7dcac fork_exit(1471260,0,1ee7dce8,0,0,...) at fork_exit+0x7e/frame 0x1ee7dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee7dcd4 --- trap 0, eip = 0, esp = 0x1ee7dd20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100104 td 0x1b036700 sched_switch(1b036700,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1ee87b78 mi_switch(104,0,1817fc1,22b,1b036700,...) at mi_switch+0x20e/frame 0x1ee87bb4 sleepq_switch(1b036700,0,1817fc1,2d6,1b036700,...) at sleepq_switch+0x110/frame 0x1ee87be8 sleepq_timedwait(144d400,54,1ee87c50,0,0,...) at sleepq_timedwait+0x52/frame 0x1ee87c24 _sleep(144d400,2116d2c,54,17c3407,fffffed8,...) at _sleep+0x304/frame 0x1ee87c74 uma_reclaim_worker(0,1ee87ce8,17d387d,419,0,...) at uma_reclaim_worker+0x147/frame 0x1ee87cac fork_exit(144da80,0,1ee87ce8,0,0,...) at fork_exit+0x7e/frame 0x1ee87cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee87cd4 --- trap 0, eip = 0, esp = 0x1ee87d20, ebp = 0 --- (null)() at 0 Tracing command rand_harvestq pid 20 tid 100091 td 0x1b560000 sched_switch(1b560000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x15146b48 mi_switch(104,0,1817fc1,22b,1b560000,...) at mi_switch+0x20e/frame 0x15146b84 sleepq_switch(1b560000,0,1817fc1,2d6,9f49,...) at sleepq_switch+0x110/frame 0x15146bb8 sleepq_timedwait(1eecb14,0,0,0,0,...) at sleepq_timedwait+0x52/frame 0x15146bf8 _sleep(1eecb14,0,0,1766623,19999999,...) at _sleep+0x304/frame 0x15146c48 random_kthread(0,15146ce8,17d387d,419,0,...) at random_kthread+0x7c/frame 0x15146cac fork_exit(de1c90,0,15146ce8,0,0,...) at fork_exit+0x7e/frame 0x15146cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15146cd4 --- trap 0, eip = 0, esp = 0x15146d20, ebp = 0 --- (null)() at 0 Tracing command sctp_iterator pid 19 tid 100090 td 0x1b560380 sched_switch(1b560380,0,104,18a,0,...) at sched_switch+0x671/frame 0x15141b98 mi_switch(104,0,1817fc1,22b,1b560380,...) at mi_switch+0x20e/frame 0x15141bd8 sleepq_switch(1b560380,0,1817fc1,2b3,1b560380,...) at sleepq_switch+0x110/frame 0x15141c0c sleepq_wait(213d044,0,175aee2,0,0,...) at sleepq_wait+0x48/frame 0x15141c30 _sleep(213d044,213d00c,0,175aee2,0,...) at _sleep+0x36a/frame 0x15141c80 sctp_iterator_thread(0,15141ce8,17d387d,419,0,...) at sctp_iterator_thread+0x7c/frame 0x15141cac fork_exit(129bee0,0,15141ce8,0,0,...) at fork_exit+0x7e/frame 0x15141cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15141cd4 --- trap 0, eip = 0, esp = 0x15141d20, ebp = 0 --- (null)() at 0 Tracing command soaiod4 pid 18 tid 100089 td 0x1b401380 sched_switch(1b401380,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x15133b70 mi_switch(104,0,1817fc1,22b,1b401380,...) at mi_switch+0x20e/frame 0x15133bac sleepq_switch(1b401380,0,1817fc1,2d6,9f4f,...) at sleepq_switch+0x110/frame 0x15133be0 sleepq_timedwait(2101158,0,1d,0,0,...) at sleepq_timedwait+0x52/frame 0x15133c20 _sleep(2101158,2101138,0,1766623,ffffdd50,...) at _sleep+0x304/frame 0x15133c70 soaio_kproc_loop(4,15133ce8,17d387d,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x15133cac fork_exit(11656b0,4,15133ce8,0,0,...) at fork_exit+0x7e/frame 0x15133cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15133cd4 --- trap 0, eip = 0, esp = 0x15133d20, ebp = 0 --- (null)() at 0 Tracing command soaiod3 pid 17 tid 100088 td 0x1b401700 sched_switch(1b401700,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1512eb70 mi_switch(104,0,1817fc1,22b,1b401700,...) at mi_switch+0x20e/frame 0x1512ebac sleepq_switch(1b401700,0,1817fc1,2d6,9f4f,...) at sleepq_switch+0x110/frame 0x1512ebe0 sleepq_timedwait(2101158,0,1d,0,0,...) at sleepq_timedwait+0x52/frame 0x1512ec20 _sleep(2101158,2101138,0,1766623,ffffdd50,...) at _sleep+0x304/frame 0x1512ec70 soaio_kproc_loop(3,1512ece8,17d387d,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1512ecac fork_exit(11656b0,3,1512ece8,0,0,...) at fork_exit+0x7e/frame 0x1512ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1512ecd4 --- trap 0, eip = 0, esp = 0x1512ed20, ebp = 0 --- (null)() at 0 Tracing command soaiod2 pid 16 tid 100087 td 0x1b401a80 sched_switch(1b401a80,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1abffb70 mi_switch(104,0,1817fc1,22b,1b401a80,...) at mi_switch+0x20e/frame 0x1abffbac sleepq_switch(1b401a80,0,1817fc1,2d6,9f4d,...) at sleepq_switch+0x110/frame 0x1abffbe0 sleepq_timedwait(2101158,0,1d,0,0,...) at sleepq_timedwait+0x52/frame 0x1abffc20 _sleep(2101158,2101138,0,1766623,ffffdd50,...) at _sleep+0x304/frame 0x1abffc70 soaio_kproc_loop(2,1abffce8,17d387d,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abffcac fork_exit(11656b0,2,1abffce8,0,0,...) at fork_exit+0x7e/frame 0x1abffcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abffcd4 --- trap 0, eip = 0, esp = 0x1abffd20, ebp = 0 --- (null)() at 0 Tracing command soaiod1 pid 9 tid 100086 td 0x1b402000 sched_switch(1b402000,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1abfab70 mi_switch(104,0,1817fc1,22b,1b402000,...) at mi_switch+0x20e/frame 0x1abfabac sleepq_switch(1b402000,0,1817fc1,2d6,9f4d,...) at sleepq_switch+0x110/frame 0x1abfabe0 sleepq_timedwait(2101158,0,1d,0,0,...) at sleepq_timedwait+0x52/frame 0x1abfac20 _sleep(2101158,2101138,0,1766623,ffffdd50,...) at _sleep+0x304/frame 0x1abfac70 soaio_kproc_loop(1,1abface8,17d387d,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abfacac fork_exit(11656b0,1,1abface8,0,0,...) at fork_exit+0x7e/frame 0x1abfacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abfacd4 --- trap 0, eip = 0, esp = 0x1abfad20, ebp = 0 --- (null)() at 0 Tracing command fdc0 pid 8 tid 100083 td 0x1b038a80 sched_switch(1b038a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1abebb40 mi_switch(104,0,1817fc1,22b,1b038a80,...) at mi_switch+0x20e/frame 0x1abebb80 sleepq_switch(1b038a80,0,1817fc1,2b3,1b038a80,...) at sleepq_switch+0x110/frame 0x1abebbb4 sleepq_wait(1ac9d03c,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1abebbd8 _sleep(1ac9d03c,1ac9d0f8,5c,1766623,0,...) at _sleep+0x36a/frame 0x1abebc28 fdc_thread(1ac9d000,1abebce8,17d387d,419,0,...) at fdc_thread+0x336/frame 0x1abebcac fork_exit(1651cf0,1ac9d000,1abebce8,0,0,...) at fork_exit+0x7e/frame 0x1abebcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abebcd4 --- trap 0, eip = 0, esp = 0x1abebd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100048 td 0x1acfd000 sched_switch(1acfd000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab33bb0 mi_switch(104,0,1817fc1,22b,1acfd000,...) at mi_switch+0x20e/frame 0x1ab33bec sleepq_switch(1acfd000,0,1817fc1,2b3,1acd6acc,...) at sleepq_switch+0x110/frame 0x1ab33c20 sleepq_wait(1acd6acc,0,1ab33c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab33c44 _cv_wait(1acd6acc,1acd6cf8,17f9eb5,78,1acd6ad4,...) at _cv_wait+0x1d7/frame 0x1ab33c84 usb_process(1acd6ac4,1ab33ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab33cac fork_exit(f12960,1acd6ac4,1ab33ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab33cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab33cd4 --- trap 0, eip = 0, esp = 0x1ab33d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100049 td 0x1acfca80 sched_switch(1acfca80,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab38bb0 mi_switch(104,0,1817fc1,22b,1acfca80,...) at mi_switch+0x20e/frame 0x1ab38bec sleepq_switch(1acfca80,0,1817fc1,2b3,1acd6afc,...) at sleepq_switch+0x110/frame 0x1ab38c20 sleepq_wait(1acd6afc,0,1ab38c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab38c44 _cv_wait(1acd6afc,1acd6cf8,17f9eb5,78,1acd6b04,...) at _cv_wait+0x1d7/frame 0x1ab38c84 usb_process(1acd6af4,1ab38ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab38cac fork_exit(f12960,1acd6af4,1ab38ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab38cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab38cd4 --- trap 0, eip = 0, esp = 0x1ab38d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100050 td 0x1acfc700 sched_switch(1acfc700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab3dbb0 mi_switch(104,0,1817fc1,22b,1acfc700,...) at mi_switch+0x20e/frame 0x1ab3dbec sleepq_switch(1acfc700,0,1817fc1,2b3,1acd6b2c,...) at sleepq_switch+0x110/frame 0x1ab3dc20 sleepq_wait(1acd6b2c,0,1ab3dc64,1,0,...) at sleepq_wait+0x48/frame 0x1ab3dc44 _cv_wait(1acd6b2c,1acd6cf8,17f9eb5,78,1acd6b34,...) at _cv_wait+0x1d7/frame 0x1ab3dc84 usb_process(1acd6b24,1ab3dce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab3dcac fork_exit(f12960,1acd6b24,1ab3dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab3dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab3dcd4 --- trap 0, eip = 0, esp = 0x1ab3dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100051 td 0x1acfc380 sched_switch(1acfc380,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1ab42bb0 mi_switch(104,0,1817fc1,22b,1acfc380,...) at mi_switch+0x20e/frame 0x1ab42bec sleepq_switch(1acfc380,0,1817fc1,2b3,1acd6b5c,...) at sleepq_switch+0x110/frame 0x1ab42c20 sleepq_wait(1acd6b5c,0,1ab42c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab42c44 _cv_wait(1acd6b5c,1acd6cf8,17f9eb5,78,1acd6b64,...) at _cv_wait+0x1d7/frame 0x1ab42c84 usb_process(1acd6b54,1ab42ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab42cac fork_exit(f12960,1acd6b54,1ab42ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab42cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab42cd4 --- trap 0, eip = 0, esp = 0x1ab42d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100052 td 0x1acfc000 sched_switch(1acfc000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab47bb0 mi_switch(104,0,1817fc1,22b,1acfc000,...) at mi_switch+0x20e/frame 0x1ab47bec sleepq_switch(1acfc000,0,1817fc1,2b3,1acd6b8c,...) at sleepq_switch+0x110/frame 0x1ab47c20 sleepq_wait(1acd6b8c,0,1ab47c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab47c44 _cv_wait(1acd6b8c,1acd6cf8,17f9eb5,78,1acd6b94,...) at _cv_wait+0x1d7/frame 0x1ab47c84 usb_process(1acd6b84,1ab47ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab47cac fork_exit(f12960,1acd6b84,1ab47ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab47cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab47cd4 --- trap 0, eip = 0, esp = 0x1ab47d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100054 td 0x1aceb700 sched_switch(1aceb700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab52bb0 mi_switch(104,0,1817fc1,22b,1aceb700,...) at mi_switch+0x20e/frame 0x1ab52bec sleepq_switch(1aceb700,0,1817fc1,2b3,1afa4acc,...) at sleepq_switch+0x110/frame 0x1ab52c20 sleepq_wait(1afa4acc,0,1ab52c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab52c44 _cv_wait(1afa4acc,1afa4cf8,17f9eb5,78,1afa4ad4,...) at _cv_wait+0x1d7/frame 0x1ab52c84 usb_process(1afa4ac4,1ab52ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab52cac fork_exit(f12960,1afa4ac4,1ab52ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab52cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab52cd4 --- trap 0, eip = 0, esp = 0x1ab52d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100055 td 0x1aceb380 sched_switch(1aceb380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab57bb0 mi_switch(104,0,1817fc1,22b,1aceb380,...) at mi_switch+0x20e/frame 0x1ab57bec sleepq_switch(1aceb380,0,1817fc1,2b3,1afa4afc,...) at sleepq_switch+0x110/frame 0x1ab57c20 sleepq_wait(1afa4afc,0,1ab57c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab57c44 _cv_wait(1afa4afc,1afa4cf8,17f9eb5,78,1afa4b04,...) at _cv_wait+0x1d7/frame 0x1ab57c84 usb_process(1afa4af4,1ab57ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab57cac fork_exit(f12960,1afa4af4,1ab57ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab57cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab57cd4 --- trap 0, eip = 0, esp = 0x1ab57d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100056 td 0x1aceb000 sched_switch(1aceb000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab5cbb0 mi_switch(104,0,1817fc1,22b,1aceb000,...) at mi_switch+0x20e/frame 0x1ab5cbec sleepq_switch(1aceb000,0,1817fc1,2b3,1afa4b2c,...) at sleepq_switch+0x110/frame 0x1ab5cc20 sleepq_wait(1afa4b2c,0,1ab5cc64,1,0,...) at sleepq_wait+0x48/frame 0x1ab5cc44 _cv_wait(1afa4b2c,1afa4cf8,17f9eb5,78,1afa4b34,...) at _cv_wait+0x1d7/frame 0x1ab5cc84 usb_process(1afa4b24,1ab5cce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab5ccac fork_exit(f12960,1afa4b24,1ab5cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab5ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab5ccd4 --- trap 0, eip = 0, esp = 0x1ab5cd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100057 td 0x1afaf380 sched_switch(1afaf380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1ab61bb0 mi_switch(104,0,1817fc1,22b,1afaf380,...) at mi_switch+0x20e/frame 0x1ab61bec sleepq_switch(1afaf380,0,1817fc1,2b3,1afa4b5c,...) at sleepq_switch+0x110/frame 0x1ab61c20 sleepq_wait(1afa4b5c,0,1ab61c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab61c44 _cv_wait(1afa4b5c,1afa4cf8,17f9eb5,78,1afa4b64,...) at _cv_wait+0x1d7/frame 0x1ab61c84 usb_process(1afa4b54,1ab61ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab61cac fork_exit(f12960,1afa4b54,1ab61ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab61cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab61cd4 --- trap 0, eip = 0, esp = 0x1ab61d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100058 td 0x1afaf000 sched_switch(1afaf000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab66bb0 mi_switch(104,0,1817fc1,22b,1afaf000,...) at mi_switch+0x20e/frame 0x1ab66bec sleepq_switch(1afaf000,0,1817fc1,2b3,1afa4b8c,...) at sleepq_switch+0x110/frame 0x1ab66c20 sleepq_wait(1afa4b8c,0,1ab66c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab66c44 _cv_wait(1afa4b8c,1afa4cf8,17f9eb5,78,1afa4b94,...) at _cv_wait+0x1d7/frame 0x1ab66c84 usb_process(1afa4b84,1ab66ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab66cac fork_exit(f12960,1afa4b84,1ab66ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab66cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab66cd4 --- trap 0, eip = 0, esp = 0x1ab66d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100060 td 0x1afae700 sched_switch(1afae700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab71bb0 mi_switch(104,0,1817fc1,22b,1afae700,...) at mi_switch+0x20e/frame 0x1ab71bec sleepq_switch(1afae700,0,1817fc1,2b3,1afb0acc,...) at sleepq_switch+0x110/frame 0x1ab71c20 sleepq_wait(1afb0acc,0,1ab71c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab71c44 _cv_wait(1afb0acc,1afb0cf8,17f9eb5,78,1afb0ad4,...) at _cv_wait+0x1d7/frame 0x1ab71c84 usb_process(1afb0ac4,1ab71ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab71cac fork_exit(f12960,1afb0ac4,1ab71ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab71cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab71cd4 --- trap 0, eip = 0, esp = 0x1ab71d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100061 td 0x1afae380 sched_switch(1afae380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab76bb0 mi_switch(104,0,1817fc1,22b,1afae380,...) at mi_switch+0x20e/frame 0x1ab76bec sleepq_switch(1afae380,0,1817fc1,2b3,1afb0afc,...) at sleepq_switch+0x110/frame 0x1ab76c20 sleepq_wait(1afb0afc,0,1ab76c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab76c44 _cv_wait(1afb0afc,1afb0cf8,17f9eb5,78,1afb0b04,...) at _cv_wait+0x1d7/frame 0x1ab76c84 usb_process(1afb0af4,1ab76ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab76cac fork_exit(f12960,1afb0af4,1ab76ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab76cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab76cd4 --- trap 0, eip = 0, esp = 0x1ab76d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100062 td 0x1afae000 sched_switch(1afae000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab7bbb0 mi_switch(104,0,1817fc1,22b,1afae000,...) at mi_switch+0x20e/frame 0x1ab7bbec sleepq_switch(1afae000,0,1817fc1,2b3,1afb0b2c,...) at sleepq_switch+0x110/frame 0x1ab7bc20 sleepq_wait(1afb0b2c,0,1ab7bc64,1,0,...) at sleepq_wait+0x48/frame 0x1ab7bc44 _cv_wait(1afb0b2c,1afb0cf8,17f9eb5,78,1afb0b34,...) at _cv_wait+0x1d7/frame 0x1ab7bc84 usb_process(1afb0b24,1ab7bce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab7bcac fork_exit(f12960,1afb0b24,1ab7bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab7bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab7bcd4 --- trap 0, eip = 0, esp = 0x1ab7bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100063 td 0x1afada80 sched_switch(1afada80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1ab80bb0 mi_switch(104,0,1817fc1,22b,1afada80,...) at mi_switch+0x20e/frame 0x1ab80bec sleepq_switch(1afada80,0,1817fc1,2b3,1afb0b5c,...) at sleepq_switch+0x110/frame 0x1ab80c20 sleepq_wait(1afb0b5c,0,1ab80c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab80c44 _cv_wait(1afb0b5c,1afb0cf8,17f9eb5,78,1afb0b64,...) at _cv_wait+0x1d7/frame 0x1ab80c84 usb_process(1afb0b54,1ab80ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab80cac fork_exit(f12960,1afb0b54,1ab80ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab80cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab80cd4 --- trap 0, eip = 0, esp = 0x1ab80d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100064 td 0x1afad700 sched_switch(1afad700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab85bb0 mi_switch(104,0,1817fc1,22b,1afad700,...) at mi_switch+0x20e/frame 0x1ab85bec sleepq_switch(1afad700,0,1817fc1,2b3,1afb0b8c,...) at sleepq_switch+0x110/frame 0x1ab85c20 sleepq_wait(1afb0b8c,0,1ab85c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab85c44 _cv_wait(1afb0b8c,1afb0cf8,17f9eb5,78,1afb0b94,...) at _cv_wait+0x1d7/frame 0x1ab85c84 usb_process(1afb0b84,1ab85ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab85cac fork_exit(f12960,1afb0b84,1ab85ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab85cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab85cd4 --- trap 0, eip = 0, esp = 0x1ab85d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100065 td 0x1afad380 sched_switch(1afad380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab8bbb0 mi_switch(104,0,1817fc1,22b,1afad380,...) at mi_switch+0x20e/frame 0x1ab8bbec sleepq_switch(1afad380,0,1817fc1,2b3,1afb4acc,...) at sleepq_switch+0x110/frame 0x1ab8bc20 sleepq_wait(1afb4acc,0,1ab8bc64,1,0,...) at sleepq_wait+0x48/frame 0x1ab8bc44 _cv_wait(1afb4acc,1afb4cf8,17f9eb5,78,1afb4ad4,...) at _cv_wait+0x1d7/frame 0x1ab8bc84 usb_process(1afb4ac4,1ab8bce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab8bcac fork_exit(f12960,1afb4ac4,1ab8bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab8bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab8bcd4 --- trap 0, eip = 0, esp = 0x1ab8bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100066 td 0x1afad000 sched_switch(1afad000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab90bb0 mi_switch(104,0,1817fc1,22b,1afad000,...) at mi_switch+0x20e/frame 0x1ab90bec sleepq_switch(1afad000,0,1817fc1,2b3,1afb4afc,...) at sleepq_switch+0x110/frame 0x1ab90c20 sleepq_wait(1afb4afc,0,1ab90c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab90c44 _cv_wait(1afb4afc,1afb4cf8,17f9eb5,78,1afb4b04,...) at _cv_wait+0x1d7/frame 0x1ab90c84 usb_process(1afb4af4,1ab90ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab90cac fork_exit(f12960,1afb4af4,1ab90ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab90cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab90cd4 --- trap 0, eip = 0, esp = 0x1ab90d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100067 td 0x1acfea80 sched_switch(1acfea80,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab95bb0 mi_switch(104,0,1817fc1,22b,1acfea80,...) at mi_switch+0x20e/frame 0x1ab95bec sleepq_switch(1acfea80,0,1817fc1,2b3,1afb4b2c,...) at sleepq_switch+0x110/frame 0x1ab95c20 sleepq_wait(1afb4b2c,0,1ab95c64,1,0,...) at sleepq_wait+0x48/frame 0x1ab95c44 _cv_wait(1afb4b2c,1afb4cf8,17f9eb5,78,1afb4b34,...) at _cv_wait+0x1d7/frame 0x1ab95c84 usb_process(1afb4b24,1ab95ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab95cac fork_exit(f12960,1afb4b24,1ab95ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab95cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab95cd4 --- trap 0, eip = 0, esp = 0x1ab95d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100068 td 0x1acfe700 sched_switch(1acfe700,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1ab9abb0 mi_switch(104,0,1817fc1,22b,1acfe700,...) at mi_switch+0x20e/frame 0x1ab9abec sleepq_switch(1acfe700,0,1817fc1,2b3,1afb4b5c,...) at sleepq_switch+0x110/frame 0x1ab9ac20 sleepq_wait(1afb4b5c,0,1ab9ac64,1,0,...) at sleepq_wait+0x48/frame 0x1ab9ac44 _cv_wait(1afb4b5c,1afb4cf8,17f9eb5,78,1afb4b64,...) at _cv_wait+0x1d7/frame 0x1ab9ac84 usb_process(1afb4b54,1ab9ace8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab9acac fork_exit(f12960,1afb4b54,1ab9ace8,0,0,...) at fork_exit+0x7e/frame 0x1ab9acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab9acd4 --- trap 0, eip = 0, esp = 0x1ab9ad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100069 td 0x1acfe380 sched_switch(1acfe380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1ab9fbb0 mi_switch(104,0,1817fc1,22b,1acfe380,...) at mi_switch+0x20e/frame 0x1ab9fbec sleepq_switch(1acfe380,0,1817fc1,2b3,1afb4b8c,...) at sleepq_switch+0x110/frame 0x1ab9fc20 sleepq_wait(1afb4b8c,0,1ab9fc64,1,0,...) at sleepq_wait+0x48/frame 0x1ab9fc44 _cv_wait(1afb4b8c,1afb4cf8,17f9eb5,78,1afb4b94,...) at _cv_wait+0x1d7/frame 0x1ab9fc84 usb_process(1afb4b84,1ab9fce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1ab9fcac fork_exit(f12960,1afb4b84,1ab9fce8,0,0,...) at fork_exit+0x7e/frame 0x1ab9fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab9fcd4 --- trap 0, eip = 0, esp = 0x1ab9fd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100070 td 0x1afc3700 sched_switch(1afc3700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1aba5bb0 mi_switch(104,0,1817fc1,22b,1afc3700,...) at mi_switch+0x20e/frame 0x1aba5bec sleepq_switch(1afc3700,0,1817fc1,2b3,1afbdacc,...) at sleepq_switch+0x110/frame 0x1aba5c20 sleepq_wait(1afbdacc,0,1aba5c64,1,0,...) at sleepq_wait+0x48/frame 0x1aba5c44 _cv_wait(1afbdacc,1afbdcf8,17f9eb5,78,1afbdad4,...) at _cv_wait+0x1d7/frame 0x1aba5c84 usb_process(1afbdac4,1aba5ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1aba5cac fork_exit(f12960,1afbdac4,1aba5ce8,0,0,...) at fork_exit+0x7e/frame 0x1aba5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aba5cd4 --- trap 0, eip = 0, esp = 0x1aba5d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100071 td 0x1afc3380 sched_switch(1afc3380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abaabb0 mi_switch(104,0,1817fc1,22b,1afc3380,...) at mi_switch+0x20e/frame 0x1abaabec sleepq_switch(1afc3380,0,1817fc1,2b3,1afbdafc,...) at sleepq_switch+0x110/frame 0x1abaac20 sleepq_wait(1afbdafc,0,1abaac64,1,0,...) at sleepq_wait+0x48/frame 0x1abaac44 _cv_wait(1afbdafc,1afbdcf8,17f9eb5,78,1afbdb04,...) at _cv_wait+0x1d7/frame 0x1abaac84 usb_process(1afbdaf4,1abaace8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abaacac fork_exit(f12960,1afbdaf4,1abaace8,0,0,...) at fork_exit+0x7e/frame 0x1abaacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abaacd4 --- trap 0, eip = 0, esp = 0x1abaad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100072 td 0x1afc3000 sched_switch(1afc3000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abafbb0 mi_switch(104,0,1817fc1,22b,1afc3000,...) at mi_switch+0x20e/frame 0x1abafbec sleepq_switch(1afc3000,0,1817fc1,2b3,1afbdb2c,...) at sleepq_switch+0x110/frame 0x1abafc20 sleepq_wait(1afbdb2c,0,1abafc64,1,0,...) at sleepq_wait+0x48/frame 0x1abafc44 _cv_wait(1afbdb2c,1afbdcf8,17f9eb5,78,1afbdb34,...) at _cv_wait+0x1d7/frame 0x1abafc84 usb_process(1afbdb24,1abafce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abafcac fork_exit(f12960,1afbdb24,1abafce8,0,0,...) at fork_exit+0x7e/frame 0x1abafcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abafcd4 --- trap 0, eip = 0, esp = 0x1abafd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100073 td 0x1afc2a80 sched_switch(1afc2a80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1abb4bb0 mi_switch(104,0,1817fc1,22b,1afc2a80,...) at mi_switch+0x20e/frame 0x1abb4bec sleepq_switch(1afc2a80,0,1817fc1,2b3,1afbdb5c,...) at sleepq_switch+0x110/frame 0x1abb4c20 sleepq_wait(1afbdb5c,0,1abb4c64,1,0,...) at sleepq_wait+0x48/frame 0x1abb4c44 _cv_wait(1afbdb5c,1afbdcf8,17f9eb5,78,1afbdb64,...) at _cv_wait+0x1d7/frame 0x1abb4c84 usb_process(1afbdb54,1abb4ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abb4cac fork_exit(f12960,1afbdb54,1abb4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abb4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abb4cd4 --- trap 0, eip = 0, esp = 0x1abb4d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100074 td 0x1afc2700 sched_switch(1afc2700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abb9bb0 mi_switch(104,0,1817fc1,22b,1afc2700,...) at mi_switch+0x20e/frame 0x1abb9bec sleepq_switch(1afc2700,0,1817fc1,2b3,1afbdb8c,...) at sleepq_switch+0x110/frame 0x1abb9c20 sleepq_wait(1afbdb8c,0,1abb9c64,1,0,...) at sleepq_wait+0x48/frame 0x1abb9c44 _cv_wait(1afbdb8c,1afbdcf8,17f9eb5,78,1afbdb94,...) at _cv_wait+0x1d7/frame 0x1abb9c84 usb_process(1afbdb84,1abb9ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abb9cac fork_exit(f12960,1afbdb84,1abb9ce8,0,0,...) at fork_exit+0x7e/frame 0x1abb9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abb9cd4 --- trap 0, eip = 0, esp = 0x1abb9d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100076 td 0x1afc2000 sched_switch(1afc2000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abc4bb0 mi_switch(104,0,1817fc1,22b,1afc2000,...) at mi_switch+0x20e/frame 0x1abc4bec sleepq_switch(1afc2000,0,1817fc1,2b3,1acc1ca4,...) at sleepq_switch+0x110/frame 0x1abc4c20 sleepq_wait(1acc1ca4,0,1abc4c64,1,0,...) at sleepq_wait+0x48/frame 0x1abc4c44 _cv_wait(1acc1ca4,1acc1ed0,17f9eb5,78,1acc1cac,...) at _cv_wait+0x1d7/frame 0x1abc4c84 usb_process(1acc1c9c,1abc4ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abc4cac fork_exit(f12960,1acc1c9c,1abc4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc4cd4 --- trap 0, eip = 0, esp = 0x1abc4d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100077 td 0x1afc1a80 sched_switch(1afc1a80,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abc9bb0 mi_switch(104,0,1817fc1,22b,1afc1a80,...) at mi_switch+0x20e/frame 0x1abc9bec sleepq_switch(1afc1a80,0,1817fc1,2b3,1acc1cd4,...) at sleepq_switch+0x110/frame 0x1abc9c20 sleepq_wait(1acc1cd4,0,1abc9c64,1,0,...) at sleepq_wait+0x48/frame 0x1abc9c44 _cv_wait(1acc1cd4,1acc1ed0,17f9eb5,78,1acc1cdc,...) at _cv_wait+0x1d7/frame 0x1abc9c84 usb_process(1acc1ccc,1abc9ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abc9cac fork_exit(f12960,1acc1ccc,1abc9ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc9cd4 --- trap 0, eip = 0, esp = 0x1abc9d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100078 td 0x1afc1700 sched_switch(1afc1700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abcebb0 mi_switch(104,0,1817fc1,22b,1afc1700,...) at mi_switch+0x20e/frame 0x1abcebec sleepq_switch(1afc1700,0,1817fc1,2b3,1acc1d04,...) at sleepq_switch+0x110/frame 0x1abcec20 sleepq_wait(1acc1d04,0,1abcec64,1,0,...) at sleepq_wait+0x48/frame 0x1abcec44 _cv_wait(1acc1d04,1acc1ed0,17f9eb5,78,1acc1d0c,...) at _cv_wait+0x1d7/frame 0x1abcec84 usb_process(1acc1cfc,1abcece8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abcecac fork_exit(f12960,1acc1cfc,1abcece8,0,0,...) at fork_exit+0x7e/frame 0x1abcecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abcecd4 --- trap 0, eip = 0, esp = 0x1abced20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100079 td 0x1afc1380 sched_switch(1afc1380,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abd3bb0 mi_switch(104,0,1817fc1,22b,1afc1380,...) at mi_switch+0x20e/frame 0x1abd3bec sleepq_switch(1afc1380,0,1817fc1,2b3,1acc1d34,...) at sleepq_switch+0x110/frame 0x1abd3c20 sleepq_wait(1acc1d34,0,1abd3c64,1,0,...) at sleepq_wait+0x48/frame 0x1abd3c44 _cv_wait(1acc1d34,1acc1ed0,17f9eb5,78,1acc1d3c,...) at _cv_wait+0x1d7/frame 0x1abd3c84 usb_process(1acc1d2c,1abd3ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abd3cac fork_exit(f12960,1acc1d2c,1abd3ce8,0,0,...) at fork_exit+0x7e/frame 0x1abd3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abd3cd4 --- trap 0, eip = 0, esp = 0x1abd3d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100080 td 0x1afc1000 sched_switch(1afc1000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1abd8bb0 mi_switch(104,0,1817fc1,22b,1afc1000,...) at mi_switch+0x20e/frame 0x1abd8bec sleepq_switch(1afc1000,0,1817fc1,2b3,1acc1d64,...) at sleepq_switch+0x110/frame 0x1abd8c20 sleepq_wait(1acc1d64,0,1abd8c64,1,0,...) at sleepq_wait+0x48/frame 0x1abd8c44 _cv_wait(1acc1d64,1acc1ed0,17f9eb5,78,1acc1d6c,...) at _cv_wait+0x1d7/frame 0x1abd8c84 usb_process(1acc1d5c,1abd8ce8,17d387d,419,0,...) at usb_process+0x140/frame 0x1abd8cac fork_exit(f12960,1acc1d5c,1abd8ce8,0,0,...) at fork_exit+0x7e/frame 0x1abd8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abd8cd4 --- trap 0, eip = 0, esp = 0x1abd8d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100043 td 0x9f7e000 sched_switch(9f7e000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aae5b78 mi_switch(104,0,1817fc1,22b,9f7e000,...) at mi_switch+0x20e/frame 0x1aae5bb8 sleepq_switch(9f7e000,0,1817fc1,2b3,9f7e000,...) at sleepq_switch+0x110/frame 0x1aae5bec sleepq_wait(1e39ec0,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aae5c10 _sleep(1e39ec0,1e39e80,5c,1766623,0,...) at _sleep+0x36a/frame 0x1aae5c60 xpt_done_td(1e39e80,1aae5ce8,17d387d,419,0,...) at xpt_done_td+0x9b/frame 0x1aae5cac fork_exit(90bbd0,1e39e80,1aae5ce8,0,0,...) at fork_exit+0x7e/frame 0x1aae5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aae5cd4 --- trap 0, eip = 0, esp = 0x1aae5d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100096 td 0x1b552a80 sched_switch(1b552a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1515fb70 mi_switch(104,0,1817fc1,22b,1b552a80,...) at mi_switch+0x20e/frame 0x1515fbb0 sleepq_switch(1b552a80,0,1817fc1,2b3,1b552a80,...) at sleepq_switch+0x110/frame 0x1515fbe4 sleepq_wait(1e39dec,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1515fc08 _sleep(1e39dec,1e39e40,5c,1766623,0,...) at _sleep+0x36a/frame 0x1515fc58 xpt_scanner_thread(0,1515fce8,17d387d,419,0,...) at xpt_scanner_thread+0x97/frame 0x1515fcac fork_exit(90bf50,0,1515fce8,0,0,...) at fork_exit+0x7e/frame 0x1515fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1515fcd4 --- trap 0, eip = 0, esp = 0x1515fd20, ebp = 0 --- (null)() at 0 Tracing command sequencer 00 pid 14 tid 100042 td 0x9f7e380 sched_switch(9f7e380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1aae0b88 mi_switch(104,0,1817fc1,22b,9f7e380,...) at mi_switch+0x20e/frame 0x1aae0bc4 sleepq_switch(9f7e380,0,1817fc1,2b3,9f8984c,...) at sleepq_switch+0x110/frame 0x1aae0bf8 sleepq_wait(9f8984c,0,1aae0c3c,1,0,...) at sleepq_wait+0x48/frame 0x1aae0c1c _cv_wait(9f8984c,9f89804,17f968c,1b7,2095a90,...) at _cv_wait+0x1d7/frame 0x1aae0c5c seq_eventthread(9f89800,1aae0ce8,17d387d,419,0,...) at seq_eventthread+0xd2/frame 0x1aae0cac fork_exit(e7f080,9f89800,1aae0ce8,0,0,...) at fork_exit+0x7e/frame 0x1aae0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aae0cd4 --- trap 0, eip = 0, esp = 0x1aae0d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 3 pid 6 tid 100041 td 0x9f7e700 sched_switch(9f7e700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aadbb70 mi_switch(104,0,1817fc1,22b,9f7e700,...) at mi_switch+0x20e/frame 0x1aadbbb0 sleepq_switch(9f7e700,0,1817fc1,2b3,9f7e700,...) at sleepq_switch+0x110/frame 0x1aadbbe4 sleepq_wait(9f790c4,6c,18493a5,0,0,...) at sleepq_wait+0x48/frame 0x1aadbc08 _sleep(9f790c4,9f790a8,6c,18493a5,0,...) at _sleep+0x36a/frame 0x1aadbc58 crypto_ret_proc(9f790a8,1aadbce8,17d387d,419,0,...) at crypto_ret_proc+0x241/frame 0x1aadbcac fork_exit(13b3f70,9f790a8,1aadbce8,0,0,...) at fork_exit+0x7e/frame 0x1aadbcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aadbcd4 --- trap 0, eip = 0, esp = 0x1aadbd20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 2 pid 5 tid 100040 td 0x9f7ea80 sched_switch(9f7ea80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aad6b70 mi_switch(104,0,1817fc1,22b,9f7ea80,...) at mi_switch+0x20e/frame 0x1aad6bb0 sleepq_switch(9f7ea80,0,1817fc1,2b3,9f7ea80,...) at sleepq_switch+0x110/frame 0x1aad6be4 sleepq_wait(9f7908c,6c,18493a5,0,0,...) at sleepq_wait+0x48/frame 0x1aad6c08 _sleep(9f7908c,9f79070,6c,18493a5,0,...) at _sleep+0x36a/frame 0x1aad6c58 crypto_ret_proc(9f79070,1aad6ce8,17d387d,419,0,...) at crypto_ret_proc+0x241/frame 0x1aad6cac fork_exit(13b3f70,9f79070,1aad6ce8,0,0,...) at fork_exit+0x7e/frame 0x1aad6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aad6cd4 --- trap 0, eip = 0, esp = 0x1aad6d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 1 pid 4 tid 100039 td 0x9e86700 sched_switch(9e86700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aad1b70 mi_switch(104,0,1817fc1,22b,9e86700,...) at mi_switch+0x20e/frame 0x1aad1bb0 sleepq_switch(9e86700,0,1817fc1,2b3,9e86700,...) at sleepq_switch+0x110/frame 0x1aad1be4 sleepq_wait(9f79054,6c,18493a5,0,0,...) at sleepq_wait+0x48/frame 0x1aad1c08 _sleep(9f79054,9f79038,6c,18493a5,0,...) at _sleep+0x36a/frame 0x1aad1c58 crypto_ret_proc(9f79038,1aad1ce8,17d387d,419,0,...) at crypto_ret_proc+0x241/frame 0x1aad1cac fork_exit(13b3f70,9f79038,1aad1ce8,0,0,...) at fork_exit+0x7e/frame 0x1aad1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aad1cd4 --- trap 0, eip = 0, esp = 0x1aad1d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 0 pid 3 tid 100038 td 0x9e86a80 sched_switch(9e86a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aaccb70 mi_switch(104,0,1817fc1,22b,9e86a80,...) at mi_switch+0x20e/frame 0x1aaccbb0 sleepq_switch(9e86a80,0,1817fc1,2b3,9e86a80,...) at sleepq_switch+0x110/frame 0x1aaccbe4 sleepq_wait(9f7901c,6c,18493a5,0,0,...) at sleepq_wait+0x48/frame 0x1aaccc08 _sleep(9f7901c,9f79000,6c,18493a5,0,...) at _sleep+0x36a/frame 0x1aaccc58 crypto_ret_proc(9f79000,1aaccce8,17d387d,419,0,...) at crypto_ret_proc+0x241/frame 0x1aacccac fork_exit(13b3f70,9f79000,1aaccce8,0,0,...) at fork_exit+0x7e/frame 0x1aacccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aacccd4 --- trap 0, eip = 0, esp = 0x1aaccd20, ebp = 0 --- (null)() at 0 Tracing command crypto pid 2 tid 100037 td 0x9e8b000 sched_switch(9e8b000,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1aac7b88 mi_switch(104,0,1817fc1,22b,9e8b000,...) at mi_switch+0x20e/frame 0x1aac7bc4 sleepq_switch(9e8b000,0,1817fc1,2b3,9e8b000,...) at sleepq_switch+0x110/frame 0x1aac7bf8 sleepq_wait(2115ed8,6c,17fcf4c,0,0,...) at sleepq_wait+0x48/frame 0x1aac7c1c _sleep(2115ed8,2115ec0,6c,17fcf4c,0,...) at _sleep+0x36a/frame 0x1aac7c6c crypto_proc(0,1aac7ce8,17d387d,419,0,...) at crypto_proc+0x28d/frame 0x1aac7cac fork_exit(13b3b70,0,1aac7ce8,0,0,...) at fork_exit+0x7e/frame 0x1aac7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aac7cd4 --- trap 0, eip = 0, esp = 0x1aac7d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100029 td 0x9e8d000 sched_switch(9e8d000,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1aa9fb90 mi_switch(104,0,1817fc1,22b,9e8d000,...) at mi_switch+0x20e/frame 0x1aa9fbcc sleepq_switch(9e8d000,0,1817fc1,2b3,9e8d000,...) at sleepq_switch+0x110/frame 0x1aa9fc00 sleepq_wait(21393fc,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aa9fc24 _sleep(21393fc,2081df4,25c,1766623,0,...) at _sleep+0x36a/frame 0x1aa9fc74 g_run_events(0,1aa9fce8,17d387d,419,0,...) at g_run_events+0x67/frame 0x1aa9fcac fork_exit(102bb40,0,1aa9fce8,0,0,...) at fork_exit+0x7e/frame 0x1aa9fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa9fcd4 --- trap 0, eip = 0, esp = 0x1aa9fd20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100030 td 0x9e8ca80 sched_switch(9e8ca80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aaa4b78 mi_switch(104,0,1817fc1,22b,9e8ca80,...) at mi_switch+0x20e/frame 0x1aaa4bb8 sleepq_switch(9e8ca80,0,1817fc1,2b3,9e8ca80,...) at sleepq_switch+0x110/frame 0x1aaa4bec sleepq_wait(2139404,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aaa4c10 _sleep(2139404,2081e3c,25c,1766623,0,...) at _sleep+0x36a/frame 0x1aaa4c60 g_io_schedule_up(9e8ca80,0,176bee4,61,1aaa4cd4,...) at g_io_schedule_up+0xcf/frame 0x1aaa4c94 g_up_procbody(0,1aaa4ce8,17d387d,419,0,...) at g_up_procbody+0x7d/frame 0x1aaa4cac fork_exit(102bbc0,0,1aaa4ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa4cd4 --- trap 0, eip = 0, esp = 0x1aaa4d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100031 td 0x9e8c700 sched_switch(9e8c700,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1aaa9b78 mi_switch(104,0,1817fc1,22b,9e8c700,...) at mi_switch+0x20e/frame 0x1aaa9bb4 sleepq_switch(9e8c700,0,1817fc1,2b3,9e8c700,...) at sleepq_switch+0x110/frame 0x1aaa9be8 sleepq_wait(21393f8,5c,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aaa9c0c _sleep(21393f8,2081e1c,25c,1766623,0,...) at _sleep+0x36a/frame 0x1aaa9c5c g_io_schedule_down(9e8c700,0,176bee4,6d,1aaa9cd4,...) at g_io_schedule_down+0xec/frame 0x1aaa9c94 g_down_procbody(0,1aaa9ce8,17d387d,419,0,...) at g_down_procbody+0x7d/frame 0x1aaa9cac fork_exit(102bc40,0,1aaa9ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa9cd4 --- trap 0, eip = 0, esp = 0x1aaa9d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100009 td 0xbfe8000 sched_switch(bfe8000,0,109,18a,8,...) at sched_switch+0x671/frame 0x1aa0fc28 mi_switch(109,0,1820dd4,479,1aa0fcac,...) at mi_switch+0x20e/frame 0x1aa0fc68 ithread_loop(9e0df40,1aa0fce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1aa0fcac fork_exit(10aa7a0,9e0df40,1aa0fce8,0,0,...) at fork_exit+0x7e/frame 0x1aa0fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa0fcd4 --- trap 0, eip = 0, esp = 0x1aa0fd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100010 td 0xbfe2a80 sched_switch(bfe2a80,0,109,18a,8,...) at sched_switch+0x671/frame 0x1aa14c28 mi_switch(109,0,1820dd4,479,1aa14cac,...) at mi_switch+0x20e/frame 0x1aa14c68 ithread_loop(9e0df30,1aa14ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1aa14cac fork_exit(10aa7a0,9e0df30,1aa14ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa14cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa14cd4 --- trap 0, eip = 0, esp = 0x1aa14d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100012 td 0xbfe2380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100023 td 0x9e84000 sched_switch(9e84000,0,109,18a,8,...) at sched_switch+0x671/frame 0x1aa61c28 mi_switch(109,0,1820dd4,479,1aa61cac,...) at mi_switch+0x20e/frame 0x1aa61c68 ithread_loop(9e0df10,1aa61ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1aa61cac fork_exit(10aa7a0,9e0df10,1aa61ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa61cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa61cd4 --- trap 0, eip = 0, esp = 0x1aa61d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100024 td 0xbfeaa80 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100025 td 0xbfea700 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100026 td 0xbfea380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100027 td 0x9e8d700 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100028 td 0x9e8d380 sched_switch(9e8d380,0,109,18a,8,...) at sched_switch+0x671/frame 0x1aa7ac28 mi_switch(109,0,1820dd4,479,1aa7acac,...) at mi_switch+0x20e/frame 0x1aa7ac68 ithread_loop(9e0dec0,1aa7ace8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1aa7acac fork_exit(10aa7a0,9e0dec0,1aa7ace8,0,0,...) at fork_exit+0x7e/frame 0x1aa7acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa7acd4 --- trap 0, eip = 0, esp = 0x1aa7ad20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100044 td 0x1acfe000 sched_switch(1acfe000,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab18c28 mi_switch(109,0,1820dd4,479,1ab18cac,...) at mi_switch+0x20e/frame 0x1ab18c68 ithread_loop(1acea050,1ab18ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab18cac fork_exit(10aa7a0,1acea050,1ab18ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab18cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab18cd4 --- trap 0, eip = 0, esp = 0x1ab18d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100045 td 0x1acfda80 sched_switch(1acfda80,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab22c28 mi_switch(109,0,1820dd4,479,1ab22cac,...) at mi_switch+0x20e/frame 0x1ab22c68 ithread_loop(1acea030,1ab22ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab22cac fork_exit(10aa7a0,1acea030,1ab22ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab22cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab22cd4 --- trap 0, eip = 0, esp = 0x1ab22d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100046 td 0x1acfd700 sched_switch(1acfd700,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab28c28 mi_switch(109,0,1820dd4,479,1ab28cac,...) at mi_switch+0x20e/frame 0x1ab28c68 ithread_loop(1acea020,1ab28ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab28cac fork_exit(10aa7a0,1acea020,1ab28ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab28cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab28cd4 --- trap 0, eip = 0, esp = 0x1ab28d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100047 td 0x1acfd380 sched_switch(1acfd380,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab2ec28 mi_switch(109,0,1820dd4,479,1ab2ecac,...) at mi_switch+0x20e/frame 0x1ab2ec68 ithread_loop(1acea010,1ab2ece8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab2ecac fork_exit(10aa7a0,1acea010,1ab2ece8,0,0,...) at fork_exit+0x7e/frame 0x1ab2ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab2ecd4 --- trap 0, eip = 0, esp = 0x1ab2ed20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100053 td 0x1aceba80 sched_switch(1aceba80,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab4dc28 mi_switch(109,0,1820dd4,479,1ab4dcac,...) at mi_switch+0x20e/frame 0x1ab4dc68 ithread_loop(1acea000,1ab4dce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab4dcac fork_exit(10aa7a0,1acea000,1ab4dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab4dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab4dcd4 --- trap 0, eip = 0, esp = 0x1ab4dd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100059 td 0x1afaea80 sched_switch(1afaea80,0,109,18a,8,...) at sched_switch+0x671/frame 0x1ab6cc28 mi_switch(109,0,1820dd4,479,1ab6ccac,...) at mi_switch+0x20e/frame 0x1ab6cc68 ithread_loop(9e0ef90,1ab6cce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1ab6ccac fork_exit(10aa7a0,9e0ef90,1ab6cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab6ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab6ccd4 --- trap 0, eip = 0, esp = 0x1ab6cd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100075 td 0x1afc2380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100081 td 0x1afafa80 sched_switch(1afafa80,0,109,18a,8,...) at sched_switch+0x671/frame 0x1abddc28 mi_switch(109,0,1820dd4,479,1abddcac,...) at mi_switch+0x20e/frame 0x1abddc68 ithread_loop(9e0ef70,1abddce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1abddcac fork_exit(10aa7a0,9e0ef70,1abddce8,0,0,...) at fork_exit+0x7e/frame 0x1abddcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abddcd4 --- trap 0, eip = 0, esp = 0x1abddd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100082 td 0x1afaf700 sched_switch(1afaf700,0,109,18a,8,...) at sched_switch+0x671/frame 0x1abe6c28 mi_switch(109,0,1820dd4,479,1abe6cac,...) at mi_switch+0x20e/frame 0x1abe6c68 ithread_loop(9e0ef60,1abe6ce8,17d387d,419,0,...) at ithread_loop+0x333/frame 0x1abe6cac fork_exit(10aa7a0,9e0ef60,1abe6ce8,0,0,...) at fork_exit+0x7e/frame 0x1abe6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abe6cd4 --- trap 0, eip = 0, esp = 0x1abe6d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100084 td 0x1b038700 PTDpde() at PTDpde+0x43b8 Tracing command idle pid 11 tid 100003 td 0xbfe9700 (CPU 0) cpustop_handler(0,1511aba0,1511ab94,16e79ec,2094640,...) at cpustop_handler+0x25/frame 0x1511aa80 ipi_nmi_handler(2094640,1acfda80,2094650,2094640,0,...) at ipi_nmi_handler+0x36/frame 0x1511aa90 trap(1511aba0,8,28,28,1ac50800,...) at trap+0xcc/frame 0x1511ab94 calltrap() at PTDpde+0x4165/frame 0x1511ab94 --- trap 0x13, eip = 0x1700155, esp = 0x1511abe0, ebp = 0x1511abe0 --- acpi_cpu_c1(1511ac34,16f0412,2428d90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x1511abe0 acpi_cpu_idle(125b84bf,0,125b84bf,1511ac50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x1511ac20 cpu_idle_acpi(125b84bf,0,2094640,2,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511ac34 cpu_idle(0,1511ac80,17ddfd6,adb,2,...) at cpu_idle+0xab/frame 0x1511ac50 sched_idletd(0,1511ace8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511acac fork_exit(111c240,0,1511ace8,0,0,...) at fork_exit+0x7e/frame 0x1511acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511acd4 --- trap 0, eip = 0, esp = 0x1511ad20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfe9380 (CPU 1) kdb_enter(1812f68,1812f68,0,1511faac,d,...) at kdb_enter+0x3a/frame 0x1511fa58 vpanic(2300148e,1511faac,1511faac,1511fae8,23011b92,...) at vpanic+0x164/frame 0x1511fa8c panic(2300148e,1374,9e81380,0,8,...) at panic+0x1b/frame 0x1511faa0 pmc_process_samples(bfe9380,bfc6200,1,bfe9380,1511fb10,...) at pmc_process_samples+0x422/frame 0x1511fae8 pmc_hook_handler(bfe9380,4,0,fba0,0,...) at pmc_hook_handler+0xb6c/frame 0x1511fb70 hardclock_cnt(b,0,6b4a1c44,0,0,...) at hardclock_cnt+0x22b/frame 0x1511fbc0 handleevents(1,213f980,9f49,60bcacf3,ca913d4b,...) at handleevents+0xfe/frame 0x1511fc08 cpu_activeclock(adc9387,0,2094d00,5,ffffffff,...) at cpu_activeclock+0x75/frame 0x1511fc34 cpu_idle(0,1511fc80,17ddfd6,adb,5,...) at cpu_idle+0xb4/frame 0x1511fc50 sched_idletd(0,1511fce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(111c240,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfe9000 (CPU 2) cpustop_handler(2,15124ba0,15124b94,16e79ec,213aa50,...) at cpustop_handler+0x25/frame 0x15124a80 ipi_nmi_handler(213aa50,bfe9000,15124ad0,10c85c0,213aa40,...) at ipi_nmi_handler+0x36/frame 0x15124a90 trap(15124ba0,8,28,28,1ac50400,...) at trap+0xcc/frame 0x15124b94 calltrap() at PTDpde+0x4165/frame 0x15124b94 --- trap 0x13, eip = 0x1700155, esp = 0x15124be0, ebp = 0x15124be0 --- acpi_cpu_c1(15124c34,16f0412,be7dd90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x15124be0 acpi_cpu_idle(a1727d4,0,a1727d4,15124c50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x15124c20 cpu_idle_acpi(a1727d4,0,20953c0,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15124c34 cpu_idle(0,15124c80,17ddfd6,adb,3,...) at cpu_idle+0xab/frame 0x15124c50 sched_idletd(0,15124ce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x15124cac fork_exit(111c240,0,15124ce8,0,0,...) at fork_exit+0x7e/frame 0x15124cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15124cd4 --- trap 0, eip = 0, esp = 0x15124d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfe8a80 (CPU 3) cpustop_handler(3,15129ba0,15129b94,16e79ec,214aa0bc,...) at cpustop_handler+0x25/frame 0x15129a80 ipi_nmi_handler(214aa0bc,415,15129ae8,23011830,1aa97f20,...) at ipi_nmi_handler+0x36/frame 0x15129a90 trap(15129ba0,8,28,28,1ac50200,...) at trap+0xcc/frame 0x15129b94 calltrap() at PTDpde+0x4165/frame 0x15129b94 --- trap 0x13, eip = 0x1700155, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,16f0412,be82d90,0,173708f,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(361a971c,0,361a971c,15129c50,17007ab,...) at acpi_cpu_idle+0x249/frame 0x15129c20 cpu_idle_acpi(361a971c,0,2095a80,4,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17ddfd6,adb,4,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17d387d,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(111c240,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command init pid 1 tid 100002 td 0xbfe9a80 sched_switch(bfe9a80,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x15114948 mi_switch(104,0,1817fc1,22b,bfe9a80,...) at mi_switch+0x20e/frame 0x15114984 sleepq_switch(bfe9a80,0,1817fc1,1ca,6c,...) at sleepq_switch+0x110/frame 0x151149b8 sleepq_catch_signals(20953d0,0,1817fc1,178,bfe9a80,...) at sleepq_catch_signals+0x47a/frame 0x151149fc sleepq_wait_sig(bfe09fc,6c,1833c92,100,0,...) at sleepq_wait_sig+0x14/frame 0x15114a20 _sleep(bfe09fc,bfe0aa4,36c,1833c92,0,...) at _sleep+0x35a/frame 0x15114a70 kern_wait6(bfe9a80,7,0,0,15114af4,...) at kern_wait6+0x551/frame 0x15114ac8 sys_wait4(bfe9a80,bfe9d04,18047f7,4,1,...) at sys_wait4+0x87/frame 0x15114be8 syscall(15114ce8,3b,3b,3b,ffbfe8d0,...) at syscall+0x363/frame 0x15114cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15114cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x4895cb, esp = 0xffbfe884, ebp = 0xffbfe898 --- (null)() at 0x4895cb Tracing command audit pid 10 tid 100001 td 0xbfea000 sched_switch(bfea000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1510fb80 mi_switch(104,0,1817fc1,22b,bfea000,...) at mi_switch+0x20e/frame 0x1510fbc0 sleepq_switch(bfea000,0,1817fc1,2b3,213d81c,...) at sleepq_switch+0x110/frame 0x1510fbf4 sleepq_wait(213d81c,0,1510fc38,1,0,...) at sleepq_wait+0x48/frame 0x1510fc18 _cv_wait(213d81c,213d7cc,17643a3,1be,0,...) at _cv_wait+0x1d7/frame 0x1510fc58 audit_worker(0,1510fce8,17d387d,419,0,...) at audit_worker+0x94/frame 0x1510fcac fork_exit(13d68f0,0,1510fce8,0,0,...) at fork_exit+0x7e/frame 0x1510fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1510fcd4 --- trap 0, eip = 0, esp = 0x1510fd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100000 td 0x2139780 sched_switch(2139780,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x2423bf0 mi_switch(104,0,1817fc1,22b,2139780,...) at mi_switch+0x20e/frame 0x2423c2c sleepq_switch(2139780,0,1817fc1,2d6,9f51,...) at sleepq_switch+0x110/frame 0x2423c60 sleepq_timedwait(213941c,54,9,0,0,...) at sleepq_timedwait+0x52/frame 0x2423c9c _sleep(213941c,0,54,1836904,fffff470,...) at _sleep+0x304/frame 0x2423cec swapper() at swapper+0x98/frame 0x2423d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100007 td 0xbfe8700 sched_switch(bfe8700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1aa05b90 mi_switch(104,0,1817fc1,22b,bfe8700,...) at mi_switch+0x20e/frame 0x1aa05bcc sleepq_switch(bfe8700,0,1817fc1,2b3,bfe8700,...) at sleepq_switch+0x110/frame 0x1aa05c00 sleepq_wait(9e18680,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aa05c24 _sleep(9e18680,9e1869c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aa05c74 taskqueue_thread_loop(2139d1c,1aa05ce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa05cac fork_exit(114a650,2139d1c,1aa05ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa05cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa05cd4 --- trap 0, eip = 0, esp = 0x1aa05d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100008 td 0xbfe8380 sched_switch(bfe8380,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1aa0ab90 mi_switch(104,0,1817fc1,22b,bfe8380,...) at mi_switch+0x20e/frame 0x1aa0abcc sleepq_switch(bfe8380,0,1817fc1,2b3,bfe8380,...) at sleepq_switch+0x110/frame 0x1aa0ac00 sleepq_wait(9e18600,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aa0ac24 _sleep(9e18600,9e1861c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aa0ac74 taskqueue_thread_loop(213cbb8,1aa0ace8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa0acac fork_exit(114a650,213cbb8,1aa0ace8,0,0,...) at fork_exit+0x7e/frame 0x1aa0acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa0acd4 --- trap 0, eip = 0, esp = 0x1aa0ad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100011 td 0xbfe2700 sched_switch(bfe2700,0,104,18a,2094640,...) at sched_switch+0x671/frame 0x1aa19b90 mi_switch(104,0,1817fc1,22b,bfe2700,...) at mi_switch+0x20e/frame 0x1aa19bcc sleepq_switch(bfe2700,0,1817fc1,2b3,bfe2700,...) at sleepq_switch+0x110/frame 0x1aa19c00 sleepq_wait(9e18480,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aa19c24 _sleep(9e18480,9e1849c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aa19c74 taskqueue_thread_loop(213cb68,1aa19ce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa19cac fork_exit(114a650,213cb68,1aa19ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa19cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa19cd4 --- trap 0, eip = 0, esp = 0x1aa19d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100013 td 0xbfe2000 sched_switch(bfe2000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa23b98 mi_switch(104,0,1817fc1,22b,bfe2000,...) at mi_switch+0x20e/frame 0x1aa23bd8 sleepq_switch(bfe2000,0,1817fc1,2b3,9e83000,...) at sleepq_switch+0x110/frame 0x1aa23c0c sleepq_wait(9e83000,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa23c30 msleep_spin_sbt(9e83000,9e8301c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa23c74 gtaskqueue_thread_loop(9dee804,1aa23ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa23cac fork_exit(1134710,9dee804,1aa23ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa23cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa23cd4 --- trap 0, eip = 0, esp = 0x1aa23d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100014 td 0x9e86380 sched_switch(9e86380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa34b98 mi_switch(104,0,1817fc1,22b,9e86380,...) at mi_switch+0x20e/frame 0x1aa34bd8 sleepq_switch(9e86380,0,1817fc1,2b3,9e82f00,...) at sleepq_switch+0x110/frame 0x1aa34c0c sleepq_wait(9e82f00,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa34c30 msleep_spin_sbt(9e82f00,9e82f1c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa34c74 gtaskqueue_thread_loop(9dee004,1aa34ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa34cac fork_exit(1134710,9dee004,1aa34ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa34cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa34cd4 --- trap 0, eip = 0, esp = 0x1aa34d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100015 td 0x9e86000 sched_switch(9e86000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa39b98 mi_switch(104,0,1817fc1,22b,9e86000,...) at mi_switch+0x20e/frame 0x1aa39bd8 sleepq_switch(9e86000,0,1817fc1,2b3,9e82e80,...) at sleepq_switch+0x110/frame 0x1aa39c0c sleepq_wait(9e82e80,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa39c30 msleep_spin_sbt(9e82e80,9e82e9c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa39c74 gtaskqueue_thread_loop(9deec04,1aa39ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa39cac fork_exit(1134710,9deec04,1aa39ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa39cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa39cd4 --- trap 0, eip = 0, esp = 0x1aa39d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100016 td 0x9e85a80 sched_switch(9e85a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa3eb98 mi_switch(104,0,1817fc1,22b,9e85a80,...) at mi_switch+0x20e/frame 0x1aa3ebd8 sleepq_switch(9e85a80,0,1817fc1,2b3,9e82e00,...) at sleepq_switch+0x110/frame 0x1aa3ec0c sleepq_wait(9e82e00,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa3ec30 msleep_spin_sbt(9e82e00,9e82e1c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa3ec74 gtaskqueue_thread_loop(9deec14,1aa3ece8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa3ecac fork_exit(1134710,9deec14,1aa3ece8,0,0,...) at fork_exit+0x7e/frame 0x1aa3ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa3ecd4 --- trap 0, eip = 0, esp = 0x1aa3ed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100017 td 0x9e85700 sched_switch(9e85700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa43b98 mi_switch(104,0,1817fc1,22b,9e85700,...) at mi_switch+0x20e/frame 0x1aa43bd8 sleepq_switch(9e85700,0,1817fc1,2b3,9e82d80,...) at sleepq_switch+0x110/frame 0x1aa43c0c sleepq_wait(9e82d80,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa43c30 msleep_spin_sbt(9e82d80,9e82d9c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa43c74 gtaskqueue_thread_loop(9deec24,1aa43ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa43cac fork_exit(1134710,9deec24,1aa43ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa43cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa43cd4 --- trap 0, eip = 0, esp = 0x1aa43d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100018 td 0x9e85380 sched_switch(9e85380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa48b98 mi_switch(104,0,1817fc1,22b,9e85380,...) at mi_switch+0x20e/frame 0x1aa48bd8 sleepq_switch(9e85380,0,1817fc1,2b3,9e82d00,...) at sleepq_switch+0x110/frame 0x1aa48c0c sleepq_wait(9e82d00,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa48c30 msleep_spin_sbt(9e82d00,9e82d1c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa48c74 gtaskqueue_thread_loop(9deec34,1aa48ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa48cac fork_exit(1134710,9deec34,1aa48ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa48cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa48cd4 --- trap 0, eip = 0, esp = 0x1aa48d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100019 td 0x9e85000 sched_switch(9e85000,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa4db98 mi_switch(104,0,1817fc1,22b,9e85000,...) at mi_switch+0x20e/frame 0x1aa4dbd8 sleepq_switch(9e85000,0,1817fc1,2b3,9e82c80,...) at sleepq_switch+0x110/frame 0x1aa4dc0c sleepq_wait(9e82c80,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa4dc30 msleep_spin_sbt(9e82c80,9e82c9c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa4dc74 gtaskqueue_thread_loop(9dee404,1aa4dce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa4dcac fork_exit(1134710,9dee404,1aa4dce8,0,0,...) at fork_exit+0x7e/frame 0x1aa4dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa4dcd4 --- trap 0, eip = 0, esp = 0x1aa4dd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100020 td 0x9e84a80 sched_switch(9e84a80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa52b98 mi_switch(104,0,1817fc1,22b,9e84a80,...) at mi_switch+0x20e/frame 0x1aa52bd8 sleepq_switch(9e84a80,0,1817fc1,2b3,9e82c00,...) at sleepq_switch+0x110/frame 0x1aa52c0c sleepq_wait(9e82c00,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa52c30 msleep_spin_sbt(9e82c00,9e82c1c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa52c74 gtaskqueue_thread_loop(9dee414,1aa52ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa52cac fork_exit(1134710,9dee414,1aa52ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa52cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa52cd4 --- trap 0, eip = 0, esp = 0x1aa52d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100021 td 0x9e84700 sched_switch(9e84700,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa57b98 mi_switch(104,0,1817fc1,22b,9e84700,...) at mi_switch+0x20e/frame 0x1aa57bd8 sleepq_switch(9e84700,0,1817fc1,2b3,9e82b80,...) at sleepq_switch+0x110/frame 0x1aa57c0c sleepq_wait(9e82b80,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa57c30 msleep_spin_sbt(9e82b80,9e82b9c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa57c74 gtaskqueue_thread_loop(9dee424,1aa57ce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa57cac fork_exit(1134710,9dee424,1aa57ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa57cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa57cd4 --- trap 0, eip = 0, esp = 0x1aa57d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100022 td 0x9e84380 sched_switch(9e84380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1aa5cb98 mi_switch(104,0,1817fc1,22b,9e84380,...) at mi_switch+0x20e/frame 0x1aa5cbd8 sleepq_switch(9e84380,0,1817fc1,2b3,9e82b00,...) at sleepq_switch+0x110/frame 0x1aa5cc0c sleepq_wait(9e82b00,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1aa5cc30 msleep_spin_sbt(9e82b00,9e82b1c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa5cc74 gtaskqueue_thread_loop(9dee434,1aa5cce8,17d387d,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa5ccac fork_exit(1134710,9dee434,1aa5cce8,0,0,...) at fork_exit+0x7e/frame 0x1aa5ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa5ccd4 --- trap 0, eip = 0, esp = 0x1aa5cd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100032 td 0x9e8c380 sched_switch(9e8c380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1aaaeb90 mi_switch(104,0,1817fc1,22b,9e8c380,...) at mi_switch+0x20e/frame 0x1aaaebcc sleepq_switch(9e8c380,0,1817fc1,2b3,9e8c380,...) at sleepq_switch+0x110/frame 0x1aaaec00 sleepq_wait(9e18300,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aaaec24 _sleep(9e18300,9e1831c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aaaec74 taskqueue_thread_loop(20a1f8c,1aaaece8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aaaecac fork_exit(114a650,20a1f8c,1aaaece8,0,0,...) at fork_exit+0x7e/frame 0x1aaaecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaaecd4 --- trap 0, eip = 0, esp = 0x1aaaed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100033 td 0x9e8c000 sched_switch(9e8c000,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1aab3b90 mi_switch(104,0,1817fc1,22b,9e8c000,...) at mi_switch+0x20e/frame 0x1aab3bcc sleepq_switch(9e8c000,0,1817fc1,2b3,9e8c000,...) at sleepq_switch+0x110/frame 0x1aab3c00 sleepq_wait(9ea1f00,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aab3c24 _sleep(9ea1f00,9ea1f1c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aab3c74 taskqueue_thread_loop(2115f7c,1aab3ce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aab3cac fork_exit(114a650,2115f7c,1aab3ce8,0,0,...) at fork_exit+0x7e/frame 0x1aab3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aab3cd4 --- trap 0, eip = 0, esp = 0x1aab3d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100034 td 0x9e8ba80 sched_switch(9e8ba80,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1aab8b90 mi_switch(104,0,1817fc1,22b,9e8ba80,...) at mi_switch+0x20e/frame 0x1aab8bcc sleepq_switch(9e8ba80,0,1817fc1,2b3,9e8ba80,...) at sleepq_switch+0x110/frame 0x1aab8c00 sleepq_wait(9ea1f00,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aab8c24 _sleep(9ea1f00,9ea1f1c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aab8c74 taskqueue_thread_loop(2115f7c,1aab8ce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aab8cac fork_exit(114a650,2115f7c,1aab8ce8,0,0,...) at fork_exit+0x7e/frame 0x1aab8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aab8cd4 --- trap 0, eip = 0, esp = 0x1aab8d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100035 td 0x9e8b700 sched_switch(9e8b700,0,104,18a,2094d00,...) at sched_switch+0x671/frame 0x1aabdb90 mi_switch(104,0,1817fc1,22b,9e8b700,...) at mi_switch+0x20e/frame 0x1aabdbcc sleepq_switch(9e8b700,0,1817fc1,2b3,9e8b700,...) at sleepq_switch+0x110/frame 0x1aabdc00 sleepq_wait(9ea1f00,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aabdc24 _sleep(9ea1f00,9ea1f1c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aabdc74 taskqueue_thread_loop(2115f7c,1aabdce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aabdcac fork_exit(114a650,2115f7c,1aabdce8,0,0,...) at fork_exit+0x7e/frame 0x1aabdcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aabdcd4 --- trap 0, eip = 0, esp = 0x1aabdd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100036 td 0x9e8b380 sched_switch(9e8b380,0,104,18a,20953c0,...) at sched_switch+0x671/frame 0x1aac2b90 mi_switch(104,0,1817fc1,22b,9e8b380,...) at mi_switch+0x20e/frame 0x1aac2bcc sleepq_switch(9e8b380,0,1817fc1,2b3,9e8b380,...) at sleepq_switch+0x110/frame 0x1aac2c00 sleepq_wait(9ea1f00,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1aac2c24 _sleep(9ea1f00,9ea1f1c,0,1766623,0,...) at _sleep+0x36a/frame 0x1aac2c74 taskqueue_thread_loop(2115f7c,1aac2ce8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aac2cac fork_exit(114a650,2115f7c,1aac2ce8,0,0,...) at fork_exit+0x7e/frame 0x1aac2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aac2cd4 --- trap 0, eip = 0, esp = 0x1aac2d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100085 td 0x1b038380 sched_switch(1b038380,0,104,18a,0,...) at sched_switch+0x671/frame 0x1abf5b98 mi_switch(104,0,1817fc1,22b,1b038380,...) at mi_switch+0x20e/frame 0x1abf5bd8 sleepq_switch(1b038380,0,1817fc1,2b3,1b294100,...) at sleepq_switch+0x110/frame 0x1abf5c0c sleepq_wait(1b294100,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1abf5c30 msleep_spin_sbt(1b294100,1b29411c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abf5c74 taskqueue_thread_loop(212d81c,1abf5ce8,17d387d,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abf5cac fork_exit(114a650,212d81c,1abf5ce8,0,0,...) at fork_exit+0x7e/frame 0x1abf5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abf5cd4 --- trap 0, eip = 0, esp = 0x1abf5d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100092 td 0x1b55fa80 sched_switch(1b55fa80,0,104,18a,0,...) at sched_switch+0x671/frame 0x1514bb98 mi_switch(104,0,1817fc1,22b,1b55fa80,...) at mi_switch+0x20e/frame 0x1514bbd8 sleepq_switch(1b55fa80,0,1817fc1,2b3,1b58f100,...) at sleepq_switch+0x110/frame 0x1514bc0c sleepq_wait(1b58f100,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x1514bc30 msleep_spin_sbt(1b58f100,1b58f11c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1514bc74 taskqueue_thread_loop(212f494,1514bce8,17d387d,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x1514bcac fork_exit(114a650,212f494,1514bce8,0,0,...) at fork_exit+0x7e/frame 0x1514bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1514bcd4 --- trap 0, eip = 0, esp = 0x1514bd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100093 td 0x1b55f700 sched_switch(1b55f700,0,104,18a,0,...) at sched_switch+0x671/frame 0x15150b98 mi_switch(104,0,1817fc1,22b,1b55f700,...) at mi_switch+0x20e/frame 0x15150bd8 sleepq_switch(1b55f700,0,1817fc1,2b3,1b58f100,...) at sleepq_switch+0x110/frame 0x15150c0c sleepq_wait(1b58f100,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x15150c30 msleep_spin_sbt(1b58f100,1b58f11c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x15150c74 taskqueue_thread_loop(212f494,15150ce8,17d387d,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x15150cac fork_exit(114a650,212f494,15150ce8,0,0,...) at fork_exit+0x7e/frame 0x15150cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15150cd4 --- trap 0, eip = 0, esp = 0x15150d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100094 td 0x1b55f380 sched_switch(1b55f380,0,104,18a,0,...) at sched_switch+0x671/frame 0x15155b98 mi_switch(104,0,1817fc1,22b,1b55f380,...) at mi_switch+0x20e/frame 0x15155bd8 sleepq_switch(1b55f380,0,1817fc1,2b3,1b58f100,...) at sleepq_switch+0x110/frame 0x15155c0c sleepq_wait(1b58f100,0,17501ea,1766623,0,...) at sleepq_wait+0x48/frame 0x15155c30 msleep_spin_sbt(1b58f100,1b58f11c,1766623,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x15155c74 taskqueue_thread_loop(212f494,15155ce8,17d387d,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x15155cac fork_exit(114a650,212f494,15155ce8,0,0,...) at fork_exit+0x7e/frame 0x15155cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15155cd4 --- trap 0, eip = 0, esp = 0x15155d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100095 td 0x1b55f000 sched_switch(1b55f000,0,104,18a,2095a80,...) at sched_switch+0x671/frame 0x1515ab90 mi_switch(104,0,1817fc1,22b,1b55f000,...) at mi_switch+0x20e/frame 0x1515abcc sleepq_switch(1b55f000,0,1817fc1,2b3,1b55f000,...) at sleepq_switch+0x110/frame 0x1515ac00 sleepq_wait(9ea1c00,0,1766623,0,0,...) at sleepq_wait+0x48/frame 0x1515ac24 _sleep(9ea1c00,9ea1c1c,0,1766623,0,...) at _sleep+0x36a/frame 0x1515ac74 taskqueue_thread_loop(1e39e68,1515ace8,17d387d,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1515acac fork_exit(114a650,1e39e68,1515ace8,0,0,...) at fork_exit+0x7e/frame 0x1515acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1515acd4 --- trap 0, eip = 0, esp = 0x1515ad20, ebp = 0 --- (null)() at 0 db:0:pho> dump Physical memory: 3427 MB Dumping 404 MB: 389 373 357 341 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump (textdump=) at pcpu.h:226 #1 0x009f656e in db_dump (dummy=0x1135fba, dummy2=0x0, dummy3=0xffffffff, dummy4=0x1511f6b0 "") at ../../../ddb/db_command.c:574 #2 0x009f6347 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #3 0x009fb018 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #4 0x009f6347 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #5 0x009f60a0 in db_command_loop () at ../../../ddb/db_command.c:534 #6 0x009f92cd in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:252 #7 0x0113676e in kdb_trap (type=0x3, code=0x0, tf=0x1511fa0c) at ../../../kern/subr_kdb.c:693 #8 0x016e7ce2 in trap (frame=0x1511fa0c) at ../../../i386/i386/trap.c:712 #9 0xffc0315d in ?? () #10 0x01135fba in kdb_enter (why=, msg=) at cpufunc.h:62 #11 0x010eae14 in vpanic (fmt=0x2300148e "[pmc,%d] pm=%p runcount %ld", ap=0x1511faac "t\023") at ../../../kern/kern_shutdown.c:861 #12 0x010eac2b in panic (fmt=0x2300148e "[pmc,%d] pm=%p runcount %ld") at ../../../kern/kern_shutdown.c:799 #13 0x23011b92 in pmc_process_samples (cpu=0x1, ring=) at counter.h:59 #14 0x2300efec in pmc_hook_handler (td=0xbfe9380, function=0x4, arg=0x0) at /usr/src/sys/dev/hwpmc/hwpmc_mod.c:2248 #15 0x0107eddb in hardclock_cnt (cnt=0xb, usermode=0x0) at ../../../kern/kern_clock.c:550 #16 0x016eeebe in handleevents (now=0x9f49ca913d4b, fake=0x1) at ../../../kern/kern_clocksource.c:186 #17 0x016f0625 in cpu_activeclock () at ../../../kern/kern_clocksource.c:812 #18 0x017007b4 in cpu_idle (busy=0x0) at ../../../x86/x86/cpu_machdep.c:585 #19 0x0111c476 in sched_idletd (dummy=0x0) at ../../../kern/sched_ule.c:2829 #20 0x010a779e in fork_exit (callout=0x111c240 , arg=0x0, frame=0x1511fce8) at ../../../kern/kern_fork.c:1057 #21 0xffc033ba in ?? () #22 0x00000000 in ?? () Current language: auto; currently minimal (kgdb) f 13 #13 0x23011b92 in pmc_process_samples (cpu=0x1, ring=) at counter.h:59 59 __asm __volatile( (kgdb) l 54 55 static inline void 56 counter_64_inc_8b(uint64_t *p, int64_t inc) 57 { 58 59 __asm __volatile( 60 "movl %%fs:(%%esi),%%eax\n\t" 61 "movl %%fs:4(%%esi),%%edx\n" 62 "1:\n\t" 63 "movl %%eax,%%ebx\n\t" (kgdb) f 14 #14 0x2300efec in pmc_hook_handler (td=0xbfe9380, function=0x4, arg=0x0) at /usr/src/sys/dev/hwpmc/hwpmc_mod.c:2248 2248 pmc_process_samples(cpu, PMC_HR); (kgdb) l 2243 * had already processed the interrupt). We don't 2244 * lose the interrupt sample. 2245 */ 2246 DPCPU_SET(pmc_sampled, 0); 2247 cpu = PCPU_GET(cpuid); 2248 pmc_process_samples(cpu, PMC_HR); 2249 pmc_process_samples(cpu, PMC_SR); 2250 pmc_process_samples(cpu, PMC_UR); 2251 break; 2252 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/dev/hwpmc/hwpmc_mod.c =================================================================== --- /usr/src/sys/dev/hwpmc/hwpmc_mod.c (revision 338461) +++ /usr/src/sys/dev/hwpmc/hwpmc_mod.c (working copy) @@ -61,6 +61,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -210,7 +211,7 @@ static int pmc_debugflags_parse(char *newstr, char #endif static int load(struct module *module, int cmd, void *arg); -static int pmc_add_sample(int ring, struct pmc *pm, struct trapframe *tf); +static int pmc_add_sample(ring_type_t ring, struct pmc *pm, struct trapframe *tf); static void pmc_add_thread_descriptors_from_proc(struct proc *p, struct pmc_process *pp); static int pmc_attach_process(struct proc *p, struct pmc *pm); @@ -248,7 +249,7 @@ static void pmc_process_csw_out(struct thread *td) static void pmc_process_exit(void *arg, struct proc *p); static void pmc_process_fork(void *arg, struct proc *p1, struct proc *p2, int n); -static void pmc_process_samples(int cpu, int soft); +static void pmc_process_samples(int cpu, ring_type_t soft); static void pmc_release_pmc_descriptor(struct pmc *pmc); static void pmc_process_thread_add(struct thread *td); static void pmc_process_thread_delete(struct thread *td); @@ -341,6 +342,7 @@ static int pmc_nsamples = PMC_NSAMPLES; SYSCTL_INT(_kern_hwpmc, OID_AUTO, nsamples, CTLFLAG_RDTUN, &pmc_nsamples, 0, "number of PC samples per CPU"); +static uint64_t pmc_sample_mask = PMC_NSAMPLES-1; /* * kern.hwpmc.mtxpoolsize -- number of mutexes in the mutex pool. @@ -1401,6 +1403,10 @@ pmc_process_csw_in(struct thread *td) if (pm->pm_state != PMC_STATE_RUNNING) continue; + KASSERT(counter_u64_fetch(pm->pm_runcount) >= 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); + /* increment PMC runcount */ counter_u64_add(pm->pm_runcount, 1); @@ -1595,6 +1601,10 @@ pmc_process_csw_out(struct thread *td) if (pm->pm_pcpu_state[cpu].pps_stalled == 0) pcd->pcd_stop_pmc(cpu, adjri); + KASSERT(counter_u64_fetch(pm->pm_runcount) > 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); + /* reduce this PMC's runcount */ counter_u64_add(pm->pm_runcount, -1); @@ -2724,7 +2734,7 @@ pmc_destroy_pmc_descriptor(struct pmc *pm) static void pmc_wait_for_pmc_idle(struct pmc *pm) { -#ifdef HWPMC_DEBUG +#ifdef INVARIANTS volatile int maxloop; maxloop = 100 * pmc_cpu_max(); @@ -2736,7 +2746,7 @@ pmc_wait_for_pmc_idle(struct pmc *pm) pmclog_flush(pm->pm_owner, 1); while (counter_u64_fetch(pm->pm_runcount) > 0) { pmclog_flush(pm->pm_owner, 1); -#ifdef HWPMC_DEBUG +#ifdef INVARIANTS maxloop--; KASSERT(maxloop > 0, ("[pmc,%d] (ri%d, rc%ld) waiting too long for " @@ -4642,7 +4652,7 @@ pmc_post_callchain_callback(void) */ static int -pmc_add_sample(int ring, struct pmc *pm, struct trapframe *tf) +pmc_add_sample(ring_type_t ring, struct pmc *pm, struct trapframe *tf) { int error, cpu, callchaindepth, inuserspace; struct thread *td; @@ -4657,19 +4667,20 @@ static int cpu = curcpu; psb = pmc_pcpu[cpu]->pc_sb[ring]; inuserspace = TRAPF_USERMODE(tf); - ps = psb->ps_write; - if (ps->ps_nsamples == PMC_SAMPLE_INUSE) { - counter_u64_add(ps->ps_pmc->pm_runcount, -1); - counter_u64_add(pmc_stats.pm_overwrites, 1); - ps->ps_nsamples = 0; - } else if (ps->ps_nsamples) { /* in use, reader hasn't caught up */ + ps = PMC_PROD_SAMPLE(psb); + if (psb->ps_considx != psb->ps_prodidx && + ps->ps_nsamples) { /* in use, reader hasn't caught up */ pm->pm_pcpu_state[cpu].pps_stalled = 1; counter_u64_add(pmc_stats.pm_intr_bufferfull, 1); PMCDBG6(SAM,INT,1,"(spc) cpu=%d pm=%p tf=%p um=%d wr=%d rd=%d", cpu, pm, (void *) tf, inuserspace, - (int) (psb->ps_write - psb->ps_samples), - (int) (psb->ps_read - psb->ps_samples)); + (int) (psb->ps_prodidx & pmc_sample_mask), + (int) (psb->ps_considx & pmc_sample_mask)); callchaindepth = 1; + KASSERT(counter_u64_fetch(pm->pm_runcount) > 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); + error = ENOMEM; goto done; } @@ -4677,15 +4688,9 @@ static int /* Fill in entry. */ PMCDBG6(SAM,INT,1,"cpu=%d pm=%p tf=%p um=%d wr=%d rd=%d", cpu, pm, (void *) tf, inuserspace, - (int) (psb->ps_write - psb->ps_samples), - (int) (psb->ps_read - psb->ps_samples)); + (int) (psb->ps_prodidx & pmc_sample_mask), + (int) (psb->ps_considx & pmc_sample_mask)); - KASSERT(counter_u64_fetch(pm->pm_runcount) >= 0, - ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, - (unsigned long)counter_u64_fetch(pm->pm_runcount))); - - counter_u64_add(pm->pm_runcount, 1); /* hold onto PMC */ - td = curthread; ps->ps_pmc = pm; ps->ps_td = td; @@ -4692,7 +4697,7 @@ static int ps->ps_pid = td->td_proc->p_pid; ps->ps_tid = td->td_tid; ps->ps_tsc = pmc_rdtsc(); - + ps->ps_ticks = ticks; ps->ps_cpu = cpu; ps->ps_flags = inuserspace ? PMC_CC_F_USERSPACE : 0; @@ -4699,6 +4704,7 @@ static int callchaindepth = (pm->pm_flags & PMC_F_CALLCHAIN) ? pmc_callchaindepth : 1; + MPASS(ps->ps_pc != NULL); if (callchaindepth == 1) ps->ps_pc[0] = PMC_TRAPFRAME_TO_PC(tf); else { @@ -4722,13 +4728,14 @@ static int ps->ps_nsamples = PMC_SAMPLE_INUSE; } else ps->ps_nsamples = callchaindepth; /* mark entry as in use */ - /* increment write pointer, modulo ring buffer size */ - ps++; - if (ps == psb->ps_fence) - psb->ps_write = psb->ps_samples; - else - psb->ps_write = ps; + KASSERT(counter_u64_fetch(pm->pm_runcount) >= 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); + + counter_u64_add(pm->pm_runcount, 1); /* hold onto PMC */ + /* increment write pointer */ + psb->ps_prodidx++; done: /* mark CPU as needing processing */ if (callchaindepth != PMC_SAMPLE_INUSE) @@ -4770,14 +4777,15 @@ pmc_capture_user_callchain(int cpu, int ring, stru { struct pmc *pm; struct thread *td; - struct pmc_sample *ps, *ps_end; + struct pmc_sample *ps; struct pmc_samplebuffer *psb; - int nsamples, nrecords, pass; + uint64_t considx, prodidx; + int nsamples, nrecords, pass, iter; #ifdef INVARIANTS int ncallchains; int nfree; + int start_ticks = ticks; #endif - psb = pmc_pcpu[cpu]->pc_sb[ring]; td = curthread; @@ -4795,6 +4803,10 @@ pmc_capture_user_callchain(int cpu, int ring, stru if (ring == PMC_UR) nrecords = atomic_readandclear_32(&td->td_pmcpend); + for (iter = 0, considx = psb->ps_considx, prodidx = psb->ps_prodidx; + considx < prodidx && iter < pmc_nsamples; considx++, iter++) { + ps = PMC_CONS_SAMPLE_OFF(psb, considx); + /* * Iterate through all deferred callchain requests. * Walk from the current read pointer to the current @@ -4801,13 +4813,10 @@ pmc_capture_user_callchain(int cpu, int ring, stru * write pointer. */ - ps = psb->ps_read; - ps_end = psb->ps_write; - do { #ifdef INVARIANTS if (ps->ps_nsamples == PMC_SAMPLE_FREE) { nfree++; - goto next; + continue; } if ((ps->ps_pmc == NULL) || @@ -4814,10 +4823,10 @@ pmc_capture_user_callchain(int cpu, int ring, stru (ps->ps_pmc->pm_state != PMC_STATE_RUNNING)) nfree++; #endif - if (ps->ps_nsamples != PMC_SAMPLE_INUSE) - goto next; - if (ps->ps_td != td) - goto next; + if (ps->ps_td != td || + ps->ps_nsamples == PMC_SAMPLE_INUSE || + ps->ps_pmc->pm_state != PMC_STATE_RUNNING) + continue; KASSERT(ps->ps_cpu == cpu, ("[pmc,%d] cpu mismatch ps_cpu=%d pcpu=%d", __LINE__, @@ -4850,15 +4859,28 @@ pmc_capture_user_callchain(int cpu, int ring, stru if (__predict_true(nsamples < pmc_callchaindepth - 1)) nsamples += pmc_save_user_callchain(ps->ps_pc + nsamples, pmc_callchaindepth - nsamples - 1, tf); - wmb(); - ps->ps_nsamples = nsamples; + + /* + * We have to prevent hardclock from potentially overwriting + * this sample between when we read the value and when we set + * it + */ + spinlock_enter(); + /* + * Verify that the sample hasn't been dropped in the meantime + */ + if (ps->ps_nsamples == PMC_SAMPLE_INUSE) { + ps->ps_nsamples = nsamples; + /* + * If we couldn't get a sample, simply drop the reference + */ + if (nsamples == 0) + counter_u64_add(pm->pm_runcount, -1); + } + spinlock_exit(); if (nrecords-- == 1) break; -next: - /* increment the pointer, modulo sample ring size */ - if (++ps == psb->ps_fence) - ps = psb->ps_samples; - } while (ps != ps_end); + } if (__predict_false(ring == PMC_UR && td->td_pmcpend)) { if (pass == 0) { pass = 1; @@ -4870,9 +4892,12 @@ pmc_capture_user_callchain(int cpu, int ring, stru #ifdef INVARIANTS if (ring == PMC_HR) - KASSERT(ncallchains > 0 || nfree > 0, + KASSERT(iter > 0, ("[pmc,%d] cpu %d didn't find a sample to collect", __LINE__, cpu)); + if ((ticks - start_ticks) > hz) + log(LOG_ERR, "%s took %d ticks\n", __func__, (ticks - start_ticks)); + MPASS((ticks - start_ticks) < (10*hz)); #endif /* mark CPU as needing processing */ @@ -4881,29 +4906,27 @@ pmc_capture_user_callchain(int cpu, int ring, stru static void -pmc_flush_ring(int cpu, int ring) +pmc_flush_ring(int cpu, ring_type_t ring) { struct pmc *pm; struct pmc_sample *ps; struct pmc_samplebuffer *psb; - int n; + uint64_t delta; psb = pmc_pcpu[cpu]->pc_sb[ring]; - - for (n = 0; n < pmc_nsamples; n++) { /* bound on #iterations */ - - ps = psb->ps_read; + delta = psb->ps_prodidx - psb->ps_considx; + MPASS(delta <= pmc_nsamples); + MPASS(psb->ps_considx <= psb->ps_prodidx); + for (;psb->ps_considx < psb->ps_prodidx; psb->ps_considx++) { + ps = PMC_CONS_SAMPLE(psb); + pm = ps->ps_pmc; if (ps->ps_nsamples == PMC_SAMPLE_FREE) - goto next; - pm = ps->ps_pmc; + continue; + ps->ps_nsamples = PMC_SAMPLE_FREE; + KASSERT(counter_u64_fetch(pm->pm_runcount) > 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); counter_u64_add(pm->pm_runcount, -1); - ps->ps_nsamples = PMC_SAMPLE_FREE; - /* increment read pointer, modulo sample size */ - next: - if (++ps == psb->ps_fence) - psb->ps_read = psb->ps_samples; - else - psb->ps_read = ps; } } @@ -4912,8 +4935,10 @@ pmc_flush_samples(int cpu) { int n; + spinlock_enter(); for (n = 0; n < PMC_NUM_SR; n++) pmc_flush_ring(cpu, n); + spinlock_exit(); } @@ -4922,7 +4947,7 @@ pmc_flush_samples(int cpu) */ static void -pmc_process_samples(int cpu, int ring) +pmc_process_samples(int cpu, ring_type_t ring) { struct pmc *pm; int adjri, n; @@ -4931,6 +4956,7 @@ static void struct pmc_sample *ps; struct pmc_classdep *pcd; struct pmc_samplebuffer *psb; + uint64_t delta; KASSERT(PCPU_GET(cpuid) == cpu, ("[pmc,%d] not on the correct CPU pcpu=%d cpu=%d", __LINE__, @@ -4937,14 +4963,18 @@ static void PCPU_GET(cpuid), cpu)); psb = pmc_pcpu[cpu]->pc_sb[ring]; + delta = psb->ps_prodidx - psb->ps_considx; + MPASS(delta <= pmc_nsamples); + MPASS(psb->ps_considx <= psb->ps_prodidx); + for (n = 0; psb->ps_considx < psb->ps_prodidx; psb->ps_considx++, n++) { + ps = PMC_CONS_SAMPLE(psb); - for (n = 0; n < pmc_nsamples; n++) { /* bound on #iterations */ - - ps = psb->ps_read; - if (ps->ps_nsamples == PMC_SAMPLE_FREE) - break; - + if (__predict_false(ps->ps_nsamples == PMC_SAMPLE_FREE)) + continue; pm = ps->ps_pmc; + /* skip non-running samples */ + if (pm->pm_state != PMC_STATE_RUNNING) + goto entrydone; KASSERT(counter_u64_fetch(pm->pm_runcount) > 0, ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, @@ -4956,12 +4986,16 @@ static void ("[pmc,%d] pmc=%p non-sampling mode=%d", __LINE__, pm, PMC_TO_MODE(pm))); - /* Ignore PMCs that have been switched off */ - if (pm->pm_state != PMC_STATE_RUNNING) - goto entrydone; /* If there is a pending AST wait for completion */ if (ps->ps_nsamples == PMC_SAMPLE_INUSE) { + /* + * XXX track how often we hit this as it will preferentially lose user samples + * for long running system calls + */ + /* if sample is more than 65 ms old, drop it */ + if (ticks - ps->ps_ticks > (hz >> 4)) + goto entrydone; /* Need a rescan at a later time. */ DPCPU_SET(pmc_sampled, 1); break; @@ -4969,8 +5003,8 @@ static void PMCDBG6(SAM,OPS,1,"cpu=%d pm=%p n=%d fl=%x wr=%d rd=%d", cpu, pm, ps->ps_nsamples, ps->ps_flags, - (int) (psb->ps_write - psb->ps_samples), - (int) (psb->ps_read - psb->ps_samples)); + (int) (psb->ps_prodidx & pmc_sample_mask), + (int) (psb->ps_considx & pmc_sample_mask)); /* * If this is a process-mode PMC that is attached to @@ -4993,13 +5027,11 @@ static void entrydone: ps->ps_nsamples = 0; /* mark entry as free */ + KASSERT(counter_u64_fetch(pm->pm_runcount) > 0, + ("[pmc,%d] pm=%p runcount %ld", __LINE__, (void *) pm, + (unsigned long)counter_u64_fetch(pm->pm_runcount))); + counter_u64_add(pm->pm_runcount, -1); - - /* increment read pointer, modulo sample size */ - if (++ps == psb->ps_fence) - psb->ps_read = psb->ps_samples; - else - psb->ps_read = ps; } counter_u64_add(pmc_stats.pm_log_sweeps, 1); @@ -5182,11 +5214,11 @@ pmc_process_exit(void *arg __unused, struct proc * } } + KASSERT((int) counter_u64_fetch(pm->pm_runcount) > 0, + ("[pmc,%d] runcount is %d", __LINE__, ri)); + counter_u64_add(pm->pm_runcount, -1); - KASSERT((int) counter_u64_fetch(pm->pm_runcount) >= 0, - ("[pmc,%d] runcount is %d", __LINE__, ri)); - (void) pcd->pcd_config_pmc(cpu, adjri, NULL); } @@ -5567,7 +5599,9 @@ pmc_initialize(void) (void) printf("hwpmc: tunable \"nsamples\"=%d out of " "range.\n", pmc_nsamples); pmc_nsamples = PMC_NSAMPLES; - } + pmc_sample_mask = PMC_NSAMPLES-1; + } else + pmc_sample_mask = pmc_nsamples-1; if (pmc_callchaindepth <= 0 || pmc_callchaindepth > PMC_CALLCHAIN_DEPTH_MAX) { @@ -5643,8 +5677,6 @@ pmc_initialize(void) sb = malloc_domain(sizeof(struct pmc_samplebuffer) + pmc_nsamples * sizeof(struct pmc_sample), M_PMC, domain, M_WAITOK|M_ZERO); - sb->ps_read = sb->ps_write = sb->ps_samples; - sb->ps_fence = sb->ps_samples + pmc_nsamples; KASSERT(pmc_pcpu[cpu] != NULL, ("[pmc,%d] cpu=%d Null per-cpu data", __LINE__, cpu)); @@ -5661,8 +5693,6 @@ pmc_initialize(void) sb = malloc_domain(sizeof(struct pmc_samplebuffer) + pmc_nsamples * sizeof(struct pmc_sample), M_PMC, domain, M_WAITOK|M_ZERO); - sb->ps_read = sb->ps_write = sb->ps_samples; - sb->ps_fence = sb->ps_samples + pmc_nsamples; KASSERT(pmc_pcpu[cpu] != NULL, ("[pmc,%d] cpu=%d Null per-cpu data", __LINE__, cpu)); @@ -5679,8 +5709,6 @@ pmc_initialize(void) sb = malloc_domain(sizeof(struct pmc_samplebuffer) + pmc_nsamples * sizeof(struct pmc_sample), M_PMC, domain, M_WAITOK|M_ZERO); - sb->ps_read = sb->ps_write = sb->ps_samples; - sb->ps_fence = sb->ps_samples + pmc_nsamples; KASSERT(pmc_pcpu[cpu] != NULL, ("[pmc,%d] cpu=%d Null per-cpu data", __LINE__, cpu)); Index: /usr/src/sys/sys/pmc.h =================================================================== --- /usr/src/sys/sys/pmc.h (revision 338461) +++ /usr/src/sys/sys/pmc.h (working copy) @@ -936,6 +936,8 @@ struct pmc_sample { uint16_t ps_flags; /* other flags */ lwpid_t ps_tid; /* thread id */ pid_t ps_pid; /* process PID or -1 */ + int ps_ticks; /* ticks at sample time */ + /* pad */ struct thread *ps_td; /* which thread */ struct pmc *ps_pmc; /* interrupting PMC */ uintptr_t *ps_pc; /* (const) callchain start */ @@ -946,14 +948,21 @@ struct pmc_sample { #define PMC_SAMPLE_INUSE ((uint16_t) 0xFFFF) struct pmc_samplebuffer { - struct pmc_sample * volatile ps_read; /* read pointer */ - struct pmc_sample * volatile ps_write; /* write pointer */ + volatile uint64_t ps_prodidx; /* producer index */ + volatile uint64_t ps_considx; /* consumer index */ uintptr_t *ps_callchains; /* all saved call chains */ - struct pmc_sample *ps_fence; /* one beyond ps_samples[] */ struct pmc_sample ps_samples[]; /* array of sample entries */ }; +#define PMC_CONS_SAMPLE(psb) \ + (&(psb)->ps_samples[(psb)->ps_considx & pmc_sample_mask]) +#define PMC_CONS_SAMPLE_OFF(psb, off) \ + (&(psb)->ps_samples[(off) & pmc_sample_mask]) + +#define PMC_PROD_SAMPLE(psb) \ + (&(psb)->ps_samples[(psb)->ps_prodidx & pmc_sample_mask]) + /* * struct pmc_cpustate * Index: /usr/src/sys/sys/pmckern.h =================================================================== --- /usr/src/sys/sys/pmckern.h (revision 338461) +++ /usr/src/sys/sys/pmckern.h (working copy) @@ -67,11 +67,13 @@ #define PMC_FN_THR_EXIT_LOG 16 #define PMC_FN_PROC_CREATE_LOG 17 -#define PMC_HR 0 /* Hardware ring buffer */ -#define PMC_SR 1 /* Software ring buffer */ -#define PMC_UR 2 /* userret ring buffer */ -#define PMC_NUM_SR (PMC_UR+1) - +typedef enum ring_type { + PMC_HR = 0, /* Hardware ring buffer */ + PMC_SR = 1, /* Software ring buffer */ + PMC_UR = 2, /* userret ring buffer */ + PMC_NUM_SR = PMC_UR+1 +} ring_type_t; + struct pmckern_procexec { int pm_credentialschanged; uintfptr_t pm_entryaddr; $ strings -a /var/crash/vmcore.1106 | grep /boot | grep hwpmc /boot/kernel/hwpmc.ko $ -- Test scenario: misc/pmc.sh