GENERIC HEAD from 2018-05-22 15:13:25 UTC, r334048M, vmcore.1095 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb MP Table Default Configuration 133 is unsupported Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r334048M: Tue May 22 17:35:41 CEST 2018 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 10 boot pages configured startup_alloc from "UMA Kegs", 9 boot pages left startup_alloc from "UMA Zones", 8 boot pages left startup_alloc from "UMA Hash", 7 boot pages left startup_alloc from "UMA Zones", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "MAP", 4 boot pages left startup_alloc from "UMA Kegs", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3526578176 (3363 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. eval: limits: not found /etc/rc: WARNING: failed to start ddb warning: total configured swap (4194304 pages) exceeds maximum recommended amount (1724580 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 641732 free (2172 frags, 79945 blocks, 0.2% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 816792 free (11328 frags, 100683 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 19294644 free (44300 frags, 2406293 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1183233 free (289537 frags, 111712 blocks, 2.9% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 4546834 free (114242 frags, 554074 blocks, 0.4% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc49 /usr/local/lib/libav /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: uhub3: 10 ports with 10 removable, self powered . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Additional ABI support: linux. lock order reversal: 1st 0xbc776b0 bufwait (bufwait) @ kern/vfs_bio.c:3916 2nd 0x9fa6400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10dafb4 at _sx_xlock+0x64 #3 0x1405fe3 at ufsdirhash_add+0x53 #4 0x140ab4f at ufs_direnter+0x4df #5 0x1413c96 at ufs_mkdir+0x846 #6 0x16ec4a6 at VOP_MKDIR_APV+0xf6 #7 0x11bc68b at kern_mkdirat+0x1eb #8 0x11bc491 at sys_mkdir+0x31 #9 0x16b7be2 at syscall+0x342 #10 0xffc033a7 at PTDpde+0x43af Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. May 22 22:07:00 x4 smartd[636]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, May 22 22:07:00 x4 smartd[636]: see the following Seagate web pages: May 22 22:07:00 x4 smartd[636]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en May 22 22:07:00 x4 smartd[636]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Tue May 22 22:07 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: May 22 22:07:17 x4 su[2245]: pho to root on /dev/pts/1 20180522 22:07:31 all (1/1): kevent7.sh lock order reversal: 1st 0x1f248e6c ufs (ufs) @ kern/vfs_mount.c:1343 2nd 0x21dc3d00 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1371 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3605 at lockmgr_lock_fast_path+0x1a5 #3 0x119951b at vop_stdlock+0x3b #4 0x16ed2aa at VOP_LOCK1_APV+0xda #5 0x11bfc11 at _vn_lock+0x61 #6 0x13fb01d at ffs_flushfiles+0xcd #7 0x13dd6d7 at softdep_flushfiles+0x67 #8 0x13fdd58 at ffs_unmount+0xe8 #9 0x11a3b13 at dounmount+0x583 #10 0x11a3485 at sys_unmount+0x315 #11 0x16b7be2 at syscall+0x342 #12 0xffc033a7 at PTDpde+0x43af 20180522 22:12:47 all (1/1): kevent7.sh 20180522 22:17:56 all (1/1): kevent7.sh pts leak: 1/3 20180522 22:23:06 all (1/1): kevent7.sh 20180522 22:28:16 all (1/1): kevent7.sh 20180522 22:33:26 all (1/1): kevent7.sh 20180522 22:38:36 all (1/1): kevent7.sh 20180522 22:43:45 all (1/1): kevent7.sh 20180522 22:48:54 all (1/1): kevent7.sh 20180522 22:54:04 all (1/1): kevent7.sh 20180522 22:59:12 all: done 20180522 22:59:44 all (1/562): pathconf2.sh 20180522 22:59:47 all (2/562): nullfs18.sh mount leak: 1/107 20180522 23:01:55 all (3/562): link.sh lock order reversal: 1st 0x2203bb94 ufs (ufs) @ kern/vfs_subr.c:2572 2nd 0xbd04d78 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:282 3rd 0x225ca5e4 ufs (ufs) @ kern/vfs_subr.c:2572 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3a8b at lockmgr_xlock_hard+0x5b #3 0x10a42d2 at __lockmgr_args+0x3f2 #4 0x14005a7 at ffs_lock+0xa7 #5 0x16ed2aa at VOP_LOCK1_APV+0xda #6 0x11bfc11 at _vn_lock+0x61 #7 0x11abe51 at vget+0xc1 #8 0x119d3f2 at vfs_hash_get+0xd2 #9 0x13fb1b4 at ffs_vgetf+0x44 #10 0x13f06ef at softdep_sync_buf+0xa7f #11 0x140144f at ffs_syncvnode+0x2df #12 0x13fe648 at ffs_sync+0x248 #13 0x11c1394 at vfs_write_suspend+0x1c4 #14 0x11c15c5 at vfs_write_suspend_umnt+0x45 #15 0x13fdd24 at ffs_unmount+0xb4 #16 0x11a3b13 at dounmount+0x583 #17 0x11a3485 at sys_unmount+0x315 lock order reversal: 1st 0x207cd00 cpslk (cpslk) @ i386/i386/copyout.c:128 2nd 0xbc18770 bufwait (bufwait) @ vm/vm_pager.c:374 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3a8b at lockmgr_xlock_hard+0x5b #3 0x10a42d2 at __lockmgr_args+0x3f2 #4 0x144768c at getpbuf+0x22c #5 0x1417119 at swap_pager_getpages+0x39 #6 0x1447183 at vm_pager_get_pages+0x53 #7 0x1424af6 at vm_fault_hold+0xe26 #8 0x1423c7e at vm_fault+0x5e #9 0x16b753e at trap_pfault+0xfe #10 0x16b6aa3 at trap+0x3f3 #11 0xffc0315d at PTDpde+0x4165 #12 0x169d3f1 at copyout_slow0+0x21 #13 0x169ceaa at cp_slow0+0x15a #14 0x169d39a at copyout+0xca #15 0x1136653 at uiomove_faultflag+0x173 #16 0x11364cf at uiomove+0x1f #17 0x114557f at pipe_read+0x26f mount leak: 202/309 20180522 23:06:31 all (4/562): pcatch2.sh 20180522 23:06:33 all (5/562): datamove.sh freework leak: 53/59. newblk leak: 3/16. 20180522 23:06:41 all (6/562): mmap7.sh 20180522 23:08:49 all (7/562): gnop.sh GEOM_NOP: Device md10.nop created. May 22 23:09:58 x4 kernel: pid 43972 (symlink), uid 1004 inumber 131377 on /mnt: out of inodes GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. swap_pager: I/O error - pageout failed; blkno 2528358,size 32768, error 12 swap_pager: I/O error - pageout failed; blkno 2528366,size 4096, error 12 swap_pager: I/O error - pageout failed; blkno 2528367,size 36864, error 12 swap_pager: I/O error - pageout failed; blkno 2528376,size 4096, error 12 swap_pager: I/O error - pageout failed; blkno 2528377,size 4096, error 12 swap_pager: I/O error - pageout failed; blkno 2528378,size 32768, error 12 swap_pager: I/O error - pageout failed; blkno 2528386,size 36864, error 12 GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. mount leak: 4/313 20180522 23:26:39 all (8/562): crossmp2.sh mount leak: 1/314 20180522 23:30:05 all (9/562): vfork.sh 20180522 23:30:14 all (10/562): ptrace9.sh 20180522 23:30:24 all (11/562): fork.sh 20180522 23:30:33 all (12/562): nfs3.sh 20180522 23:31:35 all (13/562): linger3.sh 20180522 23:32:03 all (14/562): sendfile.sh lock order reversal: 1st 0x20a42964 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x23d1c034 ufs (ufs) @ kern/kern_sendfile.c:698 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3756 at lockmgr_slock_hard+0x76 #3 0x10a4307 at __lockmgr_args+0x427 #4 0x14005a7 at ffs_lock+0xa7 #5 0x16ed2aa at VOP_LOCK1_APV+0xda #6 0x11bfc11 at _vn_lock+0x61 #7 0x10ce808 at vn_sendfile+0x7f8 #8 0x10d0641 at sendfile+0x1a1 #9 0x10d0496 at sys_sendfile+0x16 #10 0x16b7be2 at syscall+0x342 #11 0xffc033a7 at PTDpde+0x43af 20180522 23:32:08 all (15/562): umount3.sh May 22 23:32:11 x4 mountd[556]: bad exports list line '/mnt': symbolic link in export path or statfs failed May 22 23:33:57 x4 syslogd: last message repeated 1 times May 22 23:34:53 x4 syslogd: last message repeated 1 times mount leak: 771/1085 GEOM leak: 2/255 20180522 23:37:11 all (16/562): unlink.sh 20180522 23:37:16 all (17/562): snap11.sh lock order reversal: 1st 0xbc0f4fc bufwait (bufwait) @ vm/vm_pager.c:374 2nd 0x2244601c snaplk (snaplk) @ dev/md/md.c:987 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3a8b at lockmgr_xlock_hard+0x5b #3 0x10a42d2 at __lockmgr_args+0x3f2 #4 0x14005a7 at ffs_lock+0xa7 #5 0x16ed2aa at VOP_LOCK1_APV+0xda #6 0x11bfc11 at _vn_lock+0x61 #7 0xc43fa1 at mdstart_vnode+0x391 #8 0xc41fb0 at md_kthread+0x240 #9 0x108e12e at fork_exit+0x7e #10 0xffc033ba at PTDpde+0x43c2 lock order reversal: 1st 0x2244601c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:940 2nd 0x24a00478 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1636 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3a8b at lockmgr_xlock_hard+0x5b #3 0x10a42d2 at __lockmgr_args+0x3f2 #4 0x13da023 at ffs_snapremove+0x123 #5 0x13d311a at ffs_truncate+0x4fa #6 0x1409050 at ufs_inactive+0x250 #7 0x16ecef6 at VOP_INACTIVE_APV+0xf6 #8 0x11ac28e at vinactive+0x11e #9 0x11ac955 at vputx+0x365 #10 0x11b85ce at kern_unlinkat+0x2ce #11 0x11b82fa at sys_unlink+0x3a #12 0x16b7be2 at syscall+0x342 #13 0xffc033a7 at PTDpde+0x43af 20180522 23:39:24 all (18/562): graid1_5.sh 20180522 23:39:28 all (19/562): fullpath2.sh mount leak: 91/1176 20180522 23:44:56 all (20/562): pfl4.sh 20180522 23:44:59 all (21/562): symlink.sh mount leak: 2/1178 20180523 00:05:16 all (22/562): pause.sh 20180523 00:10:38 all (23/562): md5.sh newblk leak: 311/327. 20180523 00:10:45 all (24/562): ptrace4.sh mount leak: 1/1179 20180523 00:14:58 all (25/562): quota10.sh lock order reversal: 1st 0x25afe048 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:730 2nd 0x21853000 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:736 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10ae8ee at __mtx_lock_flags+0x8e #3 0x140e3dc at quotaoff_inchange+0x7c #4 0x140e4ef at quotaoff+0xbf #5 0x140fcc0 at ufs_quotactl+0xb0 #6 0x11b4544 at sys_quotactl+0x144 #7 0x16b7be2 at syscall+0x342 #8 0xffc033a7 at PTDpde+0x43af lock order reversal: 1st 0x1f7c1034 syncer (syncer) @ kern/vfs_subr.c:2153 2nd 0x26266a28 ufs (ufs) @ kern/vfs_subr.c:2572 stack backtrace: #0 0x113e131 at witness_debugger+0x81 #1 0x113dff8 at witness_checkorder+0xcd8 #2 0x10a3a8b at lockmgr_xlock_hard+0x5b #3 0x10a42d2 at __lockmgr_args+0x3f2 #4 0x14005a7 at ffs_lock+0xa7 #5 0x16ed2aa at VOP_LOCK1_APV+0xda #6 0x11bfc11 at _vn_lock+0x61 #7 0x11abe51 at vget+0xc1 #8 0x140f016 at qsync+0x96 #9 0x13fea6a at ffs_sync+0x66a #10 0x11b3bcc at sync_fsync+0x11c #11 0x16ebca6 at VOP_FSYNC_APV+0xf6 #12 0x11b1676 at sched_sync+0x2b6 #13 0x108e12e at fork_exit+0x7e #14 0xffc033ba at PTDpde+0x43c2 freework leak: 141/200. mount leak: 1/1180 20180523 00:20:52 all (26/562): zfs5.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (11 allocations, 2368 bytes leaked). 20180523 00:31:58 all (27/562): pthread9.sh 20180523 00:35:30 all (28/562): md3.sh 20180523 00:46:38 all (29/562): tar.sh freework leak: 79588/79788. 20180523 00:55:35 all (30/562): mmap6.sh 20180523 01:00:43 all: done 20180523 06:59:36 all (1/562): cmp.sh May 23 07:03:05 x4 kernel: pid 80469 (swap), uid 0, was killed: out of swap space mount leak: 300/1480 20180523 07:04:18 all (2/562): rename4.sh 20180523 07:04:23 all (3/562): pathconf2.sh 20180523 07:04:26 all (4/562): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff May 23 07:36:36 x4 su[80547]: pho to root on /dev/pts/2 Stopping inetd. Waiting for PIDS: 753. Stopping nfscbd. Waiting for PIDS: 718 719. Shutting down local packages:. Stopping cron. Waiting for PIDS: 685. Stopping sshd. Waiting for PIDS: 675. Stopping smartd. Waiting for PIDS: 638. Stopping powerd. Waiting for PIDS: 608. Stopping ntpd. Waiting for PIDS: 605, 605. Stopping lockd. Waiting for PIDS: 569. Stopping statd. Waiting for PIDS: 566. Stopping nfsd. Waiting for PIDS: 562 565. Stopping mountd. Waiting for PIDS: 556. Stopping nfsuserd. Waiting for PIDS: 548 549. Stopping watchdogd. Waiting for PIDS: 507. Stopping rpcbind. Waiting for PIDS: 422if_delmulti_locked: detaching ifnet instance 0x1b240400 . Stopping devd. Waiting for PIDS: 341. Writing entropy file:. Writing early boot entropy file:. . Terminated May 23 07:36:48 x4 syslogd: exiting on signal 15 2018-05-23T07:37:08.024915+02:00 x4.osted.lan init 1 - - some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: 78# ps auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 11 400.0 0.0 0 64 - RNL 22:06 1524:20.66 [idle] 0 0 0 155 0 - root 0 0.0 0.0 0 384 - DLs 22:06 2:24.92 [kernel] 0 0 0 -16 0 swapin root 1 0.0 0.0 3672 800 - ILs 22:06 0:00.10 /sbin/init -- 0 0 0 20 0 wait root 2 0.0 0.0 0 16 - DL 22:06 0:00.00 [crypto] 0 0 0 -16 0 crypto_w root 3 0.0 0.0 0 16 - DL 22:06 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 4 0.0 0.0 0 16 - DL 22:06 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 5 0.0 0.0 0 16 - DL 22:06 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 6 0.0 0.0 0 16 - DL 22:06 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 7 0.0 0.0 0 32 - DL 22:06 4:48.66 [cam] 0 0 0 -16 0 - root 8 0.0 0.0 0 16 - DL 22:06 0:00.00 [fdc0] 0 0 0 -16 0 - root 9 0.0 0.0 0 16 - DL 22:06 0:00.01 [soaiod1] 0 0 0 -16 0 - root 10 0.0 0.0 0 16 - DL 22:06 0:00.00 [audit] 0 0 0 -16 0 audit_wo root 12 0.0 0.0 0 304 - WL 22:06 4:38.07 [intr] 0 0 0 -52 0 - root 13 0.0 0.0 0 48 - DL 22:06 0:35.28 [geom] 0 0 0 -8 0 - root 14 0.0 0.0 0 480 - DL 22:06 0:01.56 [usb] 0 0 0 -68 0 - root 15 0.0 0.0 0 16 - DL 22:06 0:00.01 [soaiod2] 0 0 0 -16 0 - root 16 0.0 0.0 0 16 - DL 22:06 0:00.01 [soaiod3] 0 0 0 -16 0 - root 17 0.0 0.0 0 16 - DL 22:06 0:00.01 [soaiod4] 0 0 0 -16 0 - root 18 0.0 0.0 0 16 - DL 22:06 0:00.00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 19 0.0 0.0 0 16 - DL 22:06 0:08.68 [rand_harvestq] 0 0 0 -16 0 - root 20 0.0 0.0 0 48 - DL 22:06 12:10.15 [pagedaemon] 0 0 0 -16 0 psleep root 21 0.0 0.0 0 16 - DL 22:06 0:01.96 [vmdaemon] 0 0 0 -16 0 psleep root 22 0.0 0.0 0 96 - DL 22:06 5:08.32 [bufdaemon] 0 0 0 20 0 qsleep root 23 0.0 0.0 0 16 - DL 22:06 3:14.14 [syncer] 0 0 0 16 0 syncer root 24 0.0 0.0 0 16 - DL 22:06 0:08.47 [vnlru] 0 0 0 20 0 vlruwt root 82333 0.0 0.1 5432 2792 u0 Ss 07:37 0:00.01 -sh (sh) 0 1 0 20 0 wait root 82335 0.0 0.1 5044 2556 u0 R+ 07:37 0:00.00 ps auxwwl 0 82333 0 20 0 - root 93442 0.0 0.0 6108 416 1- D 07:04 0:00.01 pmcstat -P instr 0 1 0 20 0 pmc-sx root 93443 0.0 0.0 6108 416 1- D 07:04 0:10.67 pmcstat -P instr 0 1 0 20 0 pmcctx root 93449 0.0 0.0 0 0 1- Z 07:04 0:12.81 0 93442 0 22 0 - root 93450 0.0 0.0 0 0 1- Z 07:04 0:12.81 0 93443 0 22 0 - # # KDB: enter: Break to debugger [ thread pid 11 tid 100006 ] Stopped at kdb_alt_break_internal+0xf0: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 11 tid 100006 td 0xbfc8a80 kdb_alt_break_internal(0,15125acc,ecaef7,2,1b037888,...) at kdb_alt_break_internal+0xf0/frame 0x15125aa0 kdb_alt_break(2,1b037888,1807360,0,40000,...) at kdb_alt_break+0x16/frame 0x15125aac uart_intr_rxready(1b037800,16f78b5,2f1,bfc8a80,1fcfa40,...) at uart_intr_rxready+0x87/frame 0x15125acc uart_intr(1b037800,bfc8a80,15125b28,20781d0,bfc8a80,...) at uart_intr+0x130/frame 0x15125b0c intr_event_handle(9e17580,15125ba0,1fcfa9c,bff8380,1,...) at intr_event_handle+0xa5/frame 0x15125b38 intr_execute_handlers(bfed130,15125ba0,3) at intr_execute_handlers+0xa5/frame 0x15125b70 lapic_handle_intr(32,15125ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15125b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15125b90 --- interrupt, eip = 0x16ced35, esp = 0x15125be0, ebp = 0x15125be0 --- acpi_cpu_c1(15125c34,16bef52,be80d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15125be0 acpi_cpu_idle(269d86d3,0,269d86d3,15125c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15125c20 cpu_idle_acpi(269d86d3,0,1fcfa40,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15125c34 cpu_idle(0,15125c80,17a9b8a,ade,3,...) at cpu_idle+0xa0/frame 0x15125c50 sched_idletd(0,15125ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15125cac fork_exit(1102b90,0,15125ce8,0,0,...) at fork_exit+0x7e/frame 0x15125cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15125cd4 --- trap 0, eip = 0, esp = 0x15125d20, ebp = 0 --- (null)() at 0 db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x64a4c0 curthread = 0xbfc9700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x15116d40 fpcurthread = none idlethread = 0xbfc9700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x15116d30 common_tssp = 0xffc01260 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xa0984c0 curthread = 0xbfc9380: pid 11 tid 100004 "idle: cpu1" curpcb = 0x1511bd40 fpcurthread = none idlethread = 0xbfc9380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x1511bd30 common_tssp = 0xffc012c8 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xa09d4c0 curthread = 0xbfc9000: pid 11 tid 100005 "idle: cpu2" curpcb = 0x15120d40 fpcurthread = none idlethread = 0xbfc9000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x15120d30 common_tssp = 0xffc01330 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xa0a24c0 curthread = 0xbfc8a80: pid 11 tid 100006 "idle: cpu3" curpcb = 0x15125d40 fpcurthread = none idlethread = 0xbfc8a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x15125d30 common_tssp = 0xffc01398 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 93443 (pmcstat) thread 0x213f0a80 (101428) exclusive sx pmc-sx (pmc-sx) r = 0 (0x2077454) locked @ /usr/src/sys/dev/hwpmc/hwpmc_mod.c:3309 db:1:pho1> show lockedvnods Locked vnodes db:1:pho1> show allchains chain 1: thread 102033 (pid 82333, sh) sleeping on 0x1b037674 "ttyin" chain 2: thread 101428 (pid 93443, pmcstat) sleeping on 0x1fc02be "pmcctx" chain 3: thread 100163 (pid 93442, pmcstat) blocked on sx "pmc-sx" XLOCK thread 101428 (pid 93443, pmcstat) sleeping on 0x1fc02be "pmcctx" chain 4: thread 100100 (pid 24, vnlru) sleeping on 0xbfbc6b8 "vlruwt" chain 5: thread 100099 (pid 23, syncer) sleeping on 0x203bb20 "syncer" chain 6: thread 100098 (pid 22, bufdaemon) sleeping on 0x203b4ac "qsleep" chain 7: thread 100103 (pid 22, bufspacedaemon-0) sleeping on 0x1d95b80 "-" chain 8: thread 100116 (pid 22, /home worker) sleeping on 0x1b2e3884 "sdflush" chain 9: thread 100117 (pid 22, /tmp worker) sleeping on 0x1b2e3684 "sdflush" chain 10: thread 100118 (pid 22, /usr worker) sleeping on 0x1b2e3484 "sdflush" chain 11: thread 100119 (pid 22, /var worker) sleeping on 0x1b2e3284 "sdflush" chain 12: thread 100097 (pid 21, vmdaemon) sleeping on 0x205102c "psleep" chain 13: thread 100096 (pid 20, dom0) sleeping on 0x207b5dc "psleep" chain 14: thread 100101 (pid 20, laundry: dom0) sleeping on 0x207b5e8 "launds" chain 15: thread 100102 (pid 20, uma) sleeping on 0x1420810 "umarcl" chain 16: thread 100090 (pid 19, rand_harvestq) sleeping on 0x1e9720c "-" chain 17: thread 100089 (pid 18, sctp_iterator) sleeping on 0x207a6c0 "waiting_for_work" chain 18: thread 100088 (pid 17, soaiod4) sleeping on 0x203b0d8 "-" chain 19: thread 100087 (pid 16, soaiod3) sleeping on 0x203b0d8 "-" chain 20: thread 100086 (pid 15, soaiod2) sleeping on 0x203b0d8 "-" chain 21: thread 100085 (pid 9, soaiod1) sleeping on 0x203b0d8 "-" chain 22: thread 100082 (pid 8, fdc0) sleeping on 0x1acd323c "-" chain 23: thread 100047 (pid 14, usbus0) sleeping on 0x1ad0facc "-" chain 24: thread 100048 (pid 14, usbus0) sleeping on 0x1ad0fafc "-" chain 25: thread 100049 (pid 14, usbus0) sleeping on 0x1ad0fb2c "-" chain 26: thread 100050 (pid 14, usbus0) sleeping on 0x1ad0fb5c "-" chain 27: thread 100051 (pid 14, usbus0) sleeping on 0x1ad0fb8c "-" chain 28: thread 100053 (pid 14, usbus1) sleeping on 0x1afdaacc "-" chain 29: thread 100054 (pid 14, usbus1) sleeping on 0x1afdaafc "-" chain 30: thread 100055 (pid 14, usbus1) sleeping on 0x1afdab2c "-" chain 31: thread 100056 (pid 14, usbus1) sleeping on 0x1afdab5c "-" chain 32: thread 100057 (pid 14, usbus1) sleeping on 0x1afdab8c "-" chain 33: thread 100059 (pid 14, usbus2) sleeping on 0x1afe4acc "-" chain 34: thread 100060 (pid 14, usbus2) sleeping on 0x1afe4afc "-" chain 35: thread 100061 (pid 14, usbus2) sleeping on 0x1afe4b2c "-" chain 36: thread 100062 (pid 14, usbus2) sleeping on 0x1afe4b5c "-" chain 37: thread 100063 (pid 14, usbus2) sleeping on 0x1afe4b8c "-" chain 38: thread 100064 (pid 14, usbus3) sleeping on 0x1afe8acc "-" chain 39: thread 100065 (pid 14, usbus3) sleeping on 0x1afe8afc "-" chain 40: thread 100066 (pid 14, usbus3) sleeping on 0x1afe8b2c "-" chain 41: thread 100067 (pid 14, usbus3) sleeping on 0x1afe8b5c "-" chain 42: thread 100068 (pid 14, usbus3) sleeping on 0x1afe8b8c "-" chain 43: thread 100069 (pid 14, usbus4) sleeping on 0x1afeeacc "-" chain 44: thread 100070 (pid 14, usbus4) sleeping on 0x1afeeafc "-" chain 45: thread 100071 (pid 14, usbus4) sleeping on 0x1afeeb2c "-" chain 46: thread 100072 (pid 14, usbus4) sleeping on 0x1afeeb5c "-" chain 47: thread 100073 (pid 14, usbus4) sleeping on 0x1afeeb8c "-" chain 48: thread 100075 (pid 14, usbus5) sleeping on 0x1acfaca4 "-" chain 49: thread 100076 (pid 14, usbus5) sleeping on 0x1acfacd4 "-" chain 50: thread 100077 (pid 14, usbus5) sleeping on 0x1acfad04 "-" chain 51: thread 100078 (pid 14, usbus5) sleeping on 0x1acfad34 "-" chain 52: thread 100079 (pid 14, usbus5) sleeping on 0x1acfad64 "-" chain 53: thread 100042 (pid 7, doneq0) sleeping on 0x1de45c0 "-" chain 54: thread 100095 (pid 7, scanner) sleeping on 0x1de44ec "-" chain 55: thread 100041 (pid 6, crypto returns 3) sleeping on 0x9fb10c4 "crypto_ret_wait" chain 56: thread 100040 (pid 5, crypto returns 2) sleeping on 0x9fb108c "crypto_ret_wait" chain 57: thread 100039 (pid 4, crypto returns 1) sleeping on 0x9fb1054 "crypto_ret_wait" chain 58: thread 100038 (pid 3, crypto returns 0) sleeping on 0x9fb101c "crypto_ret_wait" chain 59: thread 100037 (pid 2, crypto) sleeping on 0x204fe4c "crypto_wait" chain 60: thread 100029 (pid 13, g_event) sleeping on 0x2076a54 "-" chain 61: thread 100030 (pid 13, g_up) sleeping on 0x2076a5c "-" chain 62: thread 100031 (pid 13, g_down) sleeping on 0x2076a50 "-" chain 63: thread 100002 (pid 1, init) sleeping on 0xbfbea14 "wait" chain 64: thread 100001 (pid 10, audit) sleeping on 0x207ae94 "audit_worker_cv" chain 65: thread 100000 (pid 0, swapper) sleeping on 0x2076a74 "swapin" chain 66: thread 100007 (pid 0, aiod_kick taskq) sleeping on 0xbfe0a00 "-" chain 67: thread 100008 (pid 0, kqueue_ctx taskq) sleeping on 0xbfe0980 "-" chain 68: thread 100011 (pid 0, thread taskq) sleeping on 0xbfe0800 "-" chain 69: thread 100013 (pid 0, if_io_tqg_0) sleeping on 0xbfe0700 "-" chain 70: thread 100014 (pid 0, if_io_tqg_1) sleeping on 0xbfe0680 "-" chain 71: thread 100015 (pid 0, if_io_tqg_2) sleeping on 0xbfe0600 "-" chain 72: thread 100016 (pid 0, if_io_tqg_3) sleeping on 0xbfe0580 "-" chain 73: thread 100017 (pid 0, softirq_0) sleeping on 0xbfe0500 "-" chain 74: thread 100018 (pid 0, softirq_1) sleeping on 0xbfe0480 "-" chain 75: thread 100019 (pid 0, softirq_2) sleeping on 0xbfe0400 "-" chain 76: thread 100020 (pid 0, softirq_3) sleeping on 0xbfe0380 "-" chain 77: thread 100021 (pid 0, if_config_tqg_0) sleeping on 0x9e5e080 "-" chain 78: thread 100022 (pid 0, config_0) sleeping on 0x9e5e000 "-" chain 79: thread 100032 (pid 0, firmware taskq) sleeping on 0x9e5df00 "-" chain 80: thread 100033 (pid 0, crypto_0) sleeping on 0x9e5de80 "-" chain 81: thread 100034 (pid 0, crypto_1) sleeping on 0x9e5de80 "-" chain 82: thread 100035 (pid 0, crypto_2) sleeping on 0x9e5de80 "-" chain 83: thread 100036 (pid 0, crypto_3) sleeping on 0x9e5de80 "-" chain 84: thread 100084 (pid 0, mca taskq) sleeping on 0x1acdb900 "-" chain 85: thread 100091 (pid 0, acpi_task_0) sleeping on 0xbfe0300 "-" chain 86: thread 100092 (pid 0, acpi_task_1) sleeping on 0xbfe0300 "-" chain 87: thread 100093 (pid 0, acpi_task_2) sleeping on 0xbfe0300 "-" chain 88: thread 100094 (pid 0, CAM taskq) sleeping on 0x9e5de00 "-" db:1:pho1> show mount 0x1b7d6000 /dev/ada0s1a on / (ufs) 0x1b718000 devfs on /dev (devfs) 0x1b7d5000 /dev/ada0s1f on /home (ufs) 0x1b7d4000 /dev/label/tmp on /tmp (ufs) 0x1eefd000 /dev/ada0s1d on /usr (ufs) 0x1eefc000 /dev/ada0s1e on /var (ufs) 0x1eefb000 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 31 Buf domain 0 freebufs 466 lofreebufs 311 hifreebufs 466 bufspace 48472064 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 11 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5239 (47392768) dirtyq count 11 (180224) wakeup 0 lim 28 CPU 10, 25, 13, 15, Locked buffers: 0 space 0 Total buffers: 5779 space 48472064 db:1:pho1> show page vm_cnt.v_free_count: 798588 vm_cnt.v_inactive_count: 1352 vm_cnt.v_active_count: 1931 vm_cnt.v_laundry_count: 53 vm_cnt.v_wire_count: 58914 vm_cnt.v_free_reserved: 1188 vm_cnt.v_free_min: 5498 vm_cnt.v_free_target: 18428 vm_cnt.v_inactive_target: 27642 db:1:pho1> show pageq pq_free 798588 dom 0 page_cnt 862225 free 798588 pq_act 1931 pq_inact 1352 pq_laund 53 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000001 08 (001024K) | 000105 07 (000512K) | 001064 06 (000256K) | 002633 05 (000128K) | 005402 04 (000064K) | 008039 03 (000032K) | 009746 02 (000016K) | 010876 01 (000008K) | 014046 00 (000004K) | 013505 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000001 08 (001024K) | 000003 07 (000512K) | 000002 06 (000256K) | 000003 05 (000128K) | 000003 04 (000064K) | 000003 03 (000032K) | 000004 02 (000016K) | 000004 01 (000008K) | 000006 00 (000004K) | 000003 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0xbfc9700 (CPU 0) cpustop_handler(0,7b8e458,15116b94,16b675c,0,...) at cpustop_handler+0x25/frame 0x15116a80 ipi_nmi_handler(0,0,0,2428dac,0,...) at ipi_nmi_handler+0x36/frame 0x15116a90 trap(15116ba0,8,28,28,1ac89800,...) at trap+0xac/frame 0x15116b94 calltrap() at PTDpde+0x4165/frame 0x15116b94 --- trap 0x13, eip = 0x16ced35, esp = 0x15116be0, ebp = 0x15116be0 --- acpi_cpu_c1(15116c34,16bef52,2428d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15116be0 acpi_cpu_idle(ac280c6,0,ac280c6,15116c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15116c20 cpu_idle_acpi(ac280c6,0,1fce600,2,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15116c34 cpu_idle(0,15116c80,17a9b8a,ade,2,...) at cpu_idle+0xa0/frame 0x15116c50 sched_idletd(0,15116ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15116cac fork_exit(1102b90,0,15116ce8,0,0,...) at fork_exit+0x7e/frame 0x15116cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15116cd4 --- trap 0, eip = 0, esp = 0x15116d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfc9380 (CPU 1) cpustop_handler(1,7b8e458,1511bb94,16b675c,0,...) at cpustop_handler+0x25/frame 0x1511ba80 ipi_nmi_handler(0,be76dac,0,85df,1511baf0,...) at ipi_nmi_handler+0x36/frame 0x1511ba90 trap(1511bba0,8,28,28,1ac89600,...) at trap+0xac/frame 0x1511bb94 calltrap() at PTDpde+0x4165/frame 0x1511bb94 --- trap 0x13, eip = 0x16ced35, esp = 0x1511bbe0, ebp = 0x1511bbe0 --- acpi_cpu_c1(1511bc34,16bef52,be76d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x1511bbe0 acpi_cpu_idle(112cd029,0,112cd029,1511bc50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x1511bc20 cpu_idle_acpi(112cd029,0,1fcecc0,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511bc34 cpu_idle(0,1511bc80,17a9b8a,ade,3,...) at cpu_idle+0xa0/frame 0x1511bc50 sched_idletd(0,1511bce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x1511bcac fork_exit(1102b90,0,1511bce8,0,0,...) at fork_exit+0x7e/frame 0x1511bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511bcd4 --- trap 0, eip = 0, esp = 0x1511bd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfc9000 (CPU 2) cpustop_handler(2,7b8e458,15120b94,16b675c,0,...) at cpustop_handler+0x25/frame 0x15120a80 ipi_nmi_handler(0,be7bdac,0,85df,15120af0,...) at ipi_nmi_handler+0x36/frame 0x15120a90 trap(15120ba0,8,28,28,1ac89400,...) at trap+0xac/frame 0x15120b94 calltrap() at PTDpde+0x4165/frame 0x15120b94 --- trap 0x13, eip = 0x16ced35, esp = 0x15120be0, ebp = 0x15120be0 --- acpi_cpu_c1(15120c34,16bef52,be7bd90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15120be0 acpi_cpu_idle(3e2ac6,0,3e2ac6,15120c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15120c20 cpu_idle_acpi(3e2ac6,0,1fcf380,b,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15120c34 cpu_idle(0,15120c80,17a9b8a,ade,b,...) at cpu_idle+0xa0/frame 0x15120c50 sched_idletd(0,15120ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15120cac fork_exit(1102b90,0,15120ce8,0,0,...) at fork_exit+0x7e/frame 0x15120cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15120cd4 --- trap 0, eip = 0, esp = 0x15120d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfc8a80 (CPU 3) kdb_alt_break_internal(0,15125acc,ecaef7,2,1b037888,...) at kdb_alt_break_internal+0xf0/frame 0x15125aa0 kdb_alt_break(2,1b037888,1807360,0,40000,...) at kdb_alt_break+0x16/frame 0x15125aac uart_intr_rxready(1b037800,16f78b5,2f1,bfc8a80,1fcfa40,...) at uart_intr_rxready+0x87/frame 0x15125acc uart_intr(1b037800,bfc8a80,15125b28,20781d0,bfc8a80,...) at uart_intr+0x130/frame 0x15125b0c intr_event_handle(9e17580,15125ba0,1fcfa9c,bff8380,1,...) at intr_event_handle+0xa5/frame 0x15125b38 intr_execute_handlers(bfed130,15125ba0,3) at intr_execute_handlers+0xa5/frame 0x15125b70 lapic_handle_intr(32,15125ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15125b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15125b90 --- interrupt, eip = 0x16ced35, esp = 0x15125be0, ebp = 0x15125be0 --- acpi_cpu_c1(15125c34,16bef52,be80d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15125be0 acpi_cpu_idle(269d86d3,0,269d86d3,15125c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15125c20 cpu_idle_acpi(269d86d3,0,1fcfa40,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15125c34 cpu_idle(0,15125c80,17a9b8a,ade,3,...) at cpu_idle+0xa0/frame 0x15125c50 sched_idletd(0,15125ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15125cac fork_exit(1102b90,0,15125ce8,0,0,...) at fork_exit+0x7e/frame 0x15125cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15125cd4 --- trap 0, eip = 0, esp = 0x15125d20, ebp = 0 --- (null)() at 0 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 82333 1 82333 0 Ss+ ttyin 0x1b037674 sh 93443 1 76117 0 D+ pmcctx 0x1fc02be pmcstat 93442 1 76117 0 D+ pmc-sx 0x2077454 pmcstat 24 0 0 0 DL vlruwt 0xbfbc6b8 [vnlru] 23 0 0 0 DL syncer 0x203bb20 [syncer] 22 0 0 0 DL (threaded) [bufdaemon] 100098 D qsleep 0x203b4ac [bufdaemon] 100103 D - 0x1d95b80 [bufspacedaemon-0] 100116 D sdflush 0x1b2e3884 [/home worker] 100117 D sdflush 0x1b2e3684 [/tmp worker] 100118 D sdflush 0x1b2e3484 [/usr worker] 100119 D sdflush 0x1b2e3284 [/var worker] 21 0 0 0 DL psleep 0x205102c [vmdaemon] 20 0 0 0 DL (threaded) [pagedaemon] 100096 D psleep 0x207b5dc [dom0] 100101 D launds 0x207b5e8 [laundry: dom0] 100102 D umarcl 0x1420810 [uma] 19 0 0 0 DL - 0x1e9720c [rand_harvestq] 18 0 0 0 DL waiting 0x207a6c0 [sctp_iterator] 17 0 0 0 DL - 0x203b0d8 [soaiod4] 16 0 0 0 DL - 0x203b0d8 [soaiod3] 15 0 0 0 DL - 0x203b0d8 [soaiod2] 9 0 0 0 DL - 0x203b0d8 [soaiod1] 8 0 0 0 DL - 0x1acd323c [fdc0] 14 0 0 0 DL (threaded) [usb] 100047 D - 0x1ad0facc [usbus0] 100048 D - 0x1ad0fafc [usbus0] 100049 D - 0x1ad0fb2c [usbus0] 100050 D - 0x1ad0fb5c [usbus0] 100051 D - 0x1ad0fb8c [usbus0] 100053 D - 0x1afdaacc [usbus1] 100054 D - 0x1afdaafc [usbus1] 100055 D - 0x1afdab2c [usbus1] 100056 D - 0x1afdab5c [usbus1] 100057 D - 0x1afdab8c [usbus1] 100059 D - 0x1afe4acc [usbus2] 100060 D - 0x1afe4afc [usbus2] 100061 D - 0x1afe4b2c [usbus2] 100062 D - 0x1afe4b5c [usbus2] 100063 D - 0x1afe4b8c [usbus2] 100064 D - 0x1afe8acc [usbus3] 100065 D - 0x1afe8afc [usbus3] 100066 D - 0x1afe8b2c [usbus3] 100067 D - 0x1afe8b5c [usbus3] 100068 D - 0x1afe8b8c [usbus3] 100069 D - 0x1afeeacc [usbus4] 100070 D - 0x1afeeafc [usbus4] 100071 D - 0x1afeeb2c [usbus4] 100072 D - 0x1afeeb5c [usbus4] 100073 D - 0x1afeeb8c [usbus4] 100075 D - 0x1acfaca4 [usbus5] 100076 D - 0x1acfacd4 [usbus5] 100077 D - 0x1acfad04 [usbus5] 100078 D - 0x1acfad34 [usbus5] 100079 D - 0x1acfad64 [usbus5] 7 0 0 0 DL (threaded) [cam] 100042 D - 0x1de45c0 [doneq0] 100095 D - 0x1de44ec [scanner] 6 0 0 0 DL crypto_ 0x9fb10c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x9fb108c [crypto returns 2] 4 0 0 0 DL crypto_ 0x9fb1054 [crypto returns 1] 3 0 0 0 DL crypto_ 0x9fb101c [crypto returns 0] 2 0 0 0 DL crypto_ 0x204fe4c [crypto] 13 0 0 0 DL (threaded) [geom] 100029 D - 0x2076a54 [g_event] 100030 D - 0x2076a5c [g_up] 100031 D - 0x2076a50 [g_down] 12 0 0 0 WL (threaded) [intr] 100009 I [swi6: task queue] 100010 I [swi6: Giant taskq] 100012 I [swi5: fast taskq] 100023 I [swi1: netisr 0] 100024 I [swi3: vm] 100025 I [swi4: clock (0)] 100026 I [swi4: clock (1)] 100027 I [swi4: clock (2)] 100028 I [swi4: clock (3)] 100043 I [irq256: hdac0] 100044 I [irq257: re0] 100045 I [irq22: ahci0] 100046 I [irq16: hdac1 ohci0] 100052 I [irq17: ohci1 ohci3] 100058 I [irq18: ohci2 ohci4] 100074 I [irq19: ehci0] 100080 I [irq14: ata0] 100081 I [swi0: uart] 100083 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xbfbea14 [init] 10 0 0 0 DL audit_w 0x207ae94 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x2076a74 [swapper] 100007 D - 0xbfe0a00 [aiod_kick taskq] 100008 D - 0xbfe0980 [kqueue_ctx taskq] 100011 D - 0xbfe0800 [thread taskq] 100013 D - 0xbfe0700 [if_io_tqg_0] 100014 D - 0xbfe0680 [if_io_tqg_1] 100015 D - 0xbfe0600 [if_io_tqg_2] 100016 D - 0xbfe0580 [if_io_tqg_3] 100017 D - 0xbfe0500 [softirq_0] 100018 D - 0xbfe0480 [softirq_1] 100019 D - 0xbfe0400 [softirq_2] 100020 D - 0xbfe0380 [softirq_3] 100021 D - 0x9e5e080 [if_config_tqg_0] 100022 D - 0x9e5e000 [config_0] 100032 D - 0x9e5df00 [firmware taskq] 100033 D - 0x9e5de80 [crypto_0] 100034 D - 0x9e5de80 [crypto_1] 100035 D - 0x9e5de80 [crypto_2] 100036 D - 0x9e5de80 [crypto_3] 100084 D - 0x1acdb900 [mca taskq] 100091 D - 0xbfe0300 [acpi_task_0] 100092 D - 0xbfe0300 [acpi_task_1] 100093 D - 0xbfe0300 [acpi_task_2] 100094 D - 0x9e5de00 [CAM taskq] 93450 93443 76117 0 Z+ find 93449 93442 76117 0 Z+ find db:0:pho> allt Tracing command sh pid 82333 tid 102033 td 0x1f494700 sched_switch(1f494700,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x2a890960 mi_switch(104,0,17e33fd,22b,1f494700,...) at mi_switch+0x1df/frame 0x2a89099c sleepq_switch(1f494700,0,17e33fd,1ca,0,...) at sleepq_switch+0x110/frame 0x2a8909d0 sleepq_catch_signals(2a890a38,10ae7c0,1b037614,0,1b037604,...) at sleepq_catch_signals+0x360/frame 0x2a890a14 sleepq_wait_sig(1b037674,0,2a890a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x2a890a38 _cv_wait_sig(1b037674,1b037604,1809318,5a9,2a890bb8,...) at _cv_wait_sig+0x1e7/frame 0x2a890a80 tty_wait(1b037600,1b037674,2a890bb8,1,0,...) at tty_wait+0x4b/frame 0x2a890aa4 ttydisc_read(1b037600,2a890bb8,0,d1,1d923f4,...) at ttydisc_read+0x238/frame 0x2a890ae4 ttydev_read(9f0ad00,2a890bb8,0,1f494700,2a890b88,...) at ttydev_read+0x55/frame 0x2a890b04 devfs_read_f(1b55bc40,2a890bb8,bf8ce00,0,1f494700,...) at devfs_read_f+0xc5/frame 0x2a890b3c dofileread(1b55bc40,2a890bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x2a890b6c kern_readv(1f494700,0,2a890bb8,2a890be8,ffbfec6a,...) at kern_readv+0x5e/frame 0x2a890b98 sys_read(1f494700,1f494984,17cfeb8,4,1704a74,...) at sys_read+0x54/frame 0x2a890be8 syscall(2a890ce8,3b,3b,3b,8068584,...) at syscall+0x342/frame 0x2a890cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x2a890cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28252c5f, esp = 0xffbfec24, ebp = 0xffbfec38 --- exit_tag() at 0x28252c5f Tracing command pmcstat pid 93443 tid 101428 td 0x213f0a80 sched_switch(213f0a80,0,104,18a,1fcf380,...) at sched_switch+0x80c/frame 0xba54950 mi_switch(104,0,17e33fd,22b,213f0a80,...) at mi_switch+0x1df/frame 0xba5498c sleepq_switch(213f0a80,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0xba549c0 sleepq_timedwait(1fc02be,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xba54a00 _sleep(1fc02be,0,0,283b12ef,418937,...) at _sleep+0x342/frame 0xba54a54 pause_sbt(283b12ef,418937,0,0,0,...) at pause_sbt+0x12c/frame 0xba54a90 pmc_release_pmc_descriptor(ffbfe128,ba54b08,4,ced,58,...) at pmc_release_pmc_descriptor+0x257/frame 0xba54ad0 pmc_syscall_handler(213f0a80,213f0d04,17cfeb8,4,e7c60b54,...) at pmc_syscall_handler+0x42d/frame 0xba54be8 syscall(ba54ce8,3b,3b,3b,0,...) at syscall+0x342/frame 0xba54cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xba54cdc --- syscall (210, FreeBSD ELF32, pmc_syscall_handler), eip = 0x28333cbd, esp = 0xffbfe118, ebp = 0xffbfe13c --- exit_tag() at 0x28333cbd Tracing command pmcstat pid 93442 tid 100163 td 0x1f0d6a80 sched_switch(1f0d6a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1d6da978 mi_switch(104,0,17e33fd,22b,1f0d6a80,...) at mi_switch+0x1df/frame 0x1d6da9b8 sleepq_switch(1f0d6a80,0,17e33fd,2b3,2077454,...) at sleepq_switch+0x110/frame 0x1d6da9ec sleepq_wait(2077454,0,172f6f1,3,0,...) at sleepq_wait+0x48/frame 0x1d6daa10 _sx_xlock_hard(2077454,213f0a80,0,283af7e7,ced,...) at _sx_xlock_hard+0x45b/frame 0x1d6daaa4 _sx_xlock(2077454,0,283af7e7,ced,58,...) at _sx_xlock+0xcf/frame 0x1d6daad0 pmc_syscall_handler(1f0d6a80,1f0d6d04,17cfeb8,4,e7c60b54,...) at pmc_syscall_handler+0x5b/frame 0x1d6dabe8 syscall(1d6dace8,3b,3b,3b,0,...) at syscall+0x342/frame 0x1d6dacdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1d6dacdc --- syscall (210, FreeBSD ELF32, pmc_syscall_handler), eip = 0x28333cbd, esp = 0xffbfe118, ebp = 0xffbfe13c --- exit_tag() at 0x28333cbd Tracing command vnlru pid 24 tid 100100 td 0x1b2fd000 sched_switch(1b2fd000,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1d6a3b30 mi_switch(104,0,17e33fd,22b,1b2fd000,...) at mi_switch+0x1df/frame 0x1d6a3b6c sleepq_switch(1b2fd000,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1d6a3ba0 sleepq_timedwait(bfbc6b8,60,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1d6a3bdc _sleep(bfbc6b8,203bacc,260,180975e,fffffed8,...) at _sleep+0x342/frame 0x1d6a3c30 vnlru_proc(0,1d6a3ce8,179f662,407,0,...) at vnlru_proc+0xbc/frame 0x1d6a3cac fork_exit(11b0a40,0,1d6a3ce8,0,0,...) at fork_exit+0x7e/frame 0x1d6a3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1d6a3cd4 --- trap 0, eip = 0, esp = 0x1d6a3d20, ebp = 0 --- (null)() at 0 Tracing command syncer pid 23 tid 100099 td 0x1b2fd380 sched_switch(1b2fd380,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x15134b48 mi_switch(104,0,17e33fd,22b,1b2fd380,...) at mi_switch+0x1df/frame 0x15134b84 sleepq_switch(1b2fd380,0,17e33fd,2d6,1b2fd380,...) at sleepq_switch+0x110/frame 0x15134bb8 sleepq_timedwait(203bb20,0,15134c1c,0,0,...) at sleepq_timedwait+0x53/frame 0x15134bf8 _cv_timedwait_sbt(203bb20,203bb00,fffffed8,0,0,...) at _cv_timedwait_sbt+0x208/frame 0x15134c40 sched_sync(0,15134ce8,179f662,407,0,...) at sched_sync+0x6e0/frame 0x15134cac fork_exit(11b13c0,0,15134ce8,0,0,...) at fork_exit+0x7e/frame 0x15134cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15134cd4 --- trap 0, eip = 0, esp = 0x15134d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100098 td 0x1b2fd700 sched_switch(1b2fd700,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1512fb68 mi_switch(104,0,17e33fd,22b,1b2fd700,...) at mi_switch+0x1df/frame 0x1512fba4 sleepq_switch(1b2fd700,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1512fbd8 sleepq_timedwait(203b4ac,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1512fc14 _sleep(203b4ac,1d94880,54,1730031,1999997c,...) at _sleep+0x342/frame 0x1512fc68 buf_daemon(0,1512fce8,179f662,407,0,...) at buf_daemon+0xfc/frame 0x1512fcac fork_exit(118df20,0,1512fce8,0,0,...) at fork_exit+0x7e/frame 0x1512fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1512fcd4 --- trap 0, eip = 0, esp = 0x1512fd20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100103 td 0x1b066700 sched_switch(1b066700,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1ea90b60 mi_switch(104,0,17e33fd,22b,1b066700,...) at mi_switch+0x1df/frame 0x1ea90b9c sleepq_switch(1b066700,0,17e33fd,2d6,85e0,...) at sleepq_switch+0x110/frame 0x1ea90bd0 sleepq_timedwait(1d95b80,5c,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1ea90c0c _sleep(1d95b80,1d95ac0,25c,1733677,fffffed8,...) at _sleep+0x342/frame 0x1ea90c60 bufspace_daemon(1d94980,1ea90ce8,179f662,407,0,...) at bufspace_daemon+0x369/frame 0x1ea90cac fork_exit(118e310,1d94980,1ea90ce8,0,0,...) at fork_exit+0x7e/frame 0x1ea90cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ea90cd4 --- trap 0, eip = 0, esp = 0x1ea90d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100116 td 0x1b1fb700 sched_switch(1b1fb700,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1eae3b70 mi_switch(104,0,17e33fd,22b,1b1fb700,...) at mi_switch+0x1df/frame 0x1eae3bac sleepq_switch(1b1fb700,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1eae3be0 sleepq_timedwait(1b2e3884,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eae3c1c _sleep(1b2e3884,1b2e3800,54,17608e6,7fffff6c,...) at _sleep+0x342/frame 0x1eae3c70 softdep_flush(1b7d5000,1eae3ce8,179f662,407,0,...) at softdep_flush+0x199/frame 0x1eae3cac fork_exit(13e0b10,1b7d5000,1eae3ce8,0,0,...) at fork_exit+0x7e/frame 0x1eae3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eae3cd4 --- trap 0, eip = 0, esp = 0x1eae3d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100117 td 0x1b1fb380 sched_switch(1b1fb380,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1eae8b70 mi_switch(104,0,17e33fd,22b,1b1fb380,...) at mi_switch+0x1df/frame 0x1eae8bac sleepq_switch(1b1fb380,0,17e33fd,2d6,85e0,...) at sleepq_switch+0x110/frame 0x1eae8be0 sleepq_timedwait(1b2e3684,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eae8c1c _sleep(1b2e3684,1b2e3600,54,17608e6,7fffff6c,...) at _sleep+0x342/frame 0x1eae8c70 softdep_flush(1b7d4000,1eae8ce8,179f662,407,0,...) at softdep_flush+0x199/frame 0x1eae8cac fork_exit(13e0b10,1b7d4000,1eae8ce8,0,0,...) at fork_exit+0x7e/frame 0x1eae8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eae8cd4 --- trap 0, eip = 0, esp = 0x1eae8d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100118 td 0x1ef92700 sched_switch(1ef92700,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1eaedb70 mi_switch(104,0,17e33fd,22b,1ef92700,...) at mi_switch+0x1df/frame 0x1eaedbac sleepq_switch(1ef92700,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1eaedbe0 sleepq_timedwait(1b2e3484,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eaedc1c _sleep(1b2e3484,1b2e3400,54,17608e6,7fffff6c,...) at _sleep+0x342/frame 0x1eaedc70 softdep_flush(1eefd000,1eaedce8,179f662,407,0,...) at softdep_flush+0x199/frame 0x1eaedcac fork_exit(13e0b10,1eefd000,1eaedce8,0,0,...) at fork_exit+0x7e/frame 0x1eaedcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eaedcd4 --- trap 0, eip = 0, esp = 0x1eaedd20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100119 td 0x1ef92380 sched_switch(1ef92380,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1eaf2b70 mi_switch(104,0,17e33fd,22b,1ef92380,...) at mi_switch+0x1df/frame 0x1eaf2bac sleepq_switch(1ef92380,0,17e33fd,2d6,85e0,...) at sleepq_switch+0x110/frame 0x1eaf2be0 sleepq_timedwait(1b2e3284,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eaf2c1c _sleep(1b2e3284,1b2e3200,54,17608e6,7fffff6c,...) at _sleep+0x342/frame 0x1eaf2c70 softdep_flush(1eefc000,1eaf2ce8,179f662,407,0,...) at softdep_flush+0x199/frame 0x1eaf2cac fork_exit(13e0b10,1eefc000,1eaf2ce8,0,0,...) at fork_exit+0x7e/frame 0x1eaf2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eaf2cd4 --- trap 0, eip = 0, esp = 0x1eaf2d20, ebp = 0 --- (null)() at 0 Tracing command vmdaemon pid 21 tid 100097 td 0x1b2fda80 sched_switch(1b2fda80,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1512ab50 mi_switch(104,0,17e33fd,22b,1b2fda80,...) at mi_switch+0x1df/frame 0x1512ab8c sleepq_switch(1b2fda80,0,17e33fd,2b3,1b2fda80,...) at sleepq_switch+0x110/frame 0x1512abc0 sleepq_wait(205102c,74,173002a,0,0,...) at sleepq_wait+0x48/frame 0x1512abe4 _sleep(205102c,2051018,74,173002a,0,...) at _sleep+0x3a6/frame 0x1512ac38 vm_daemon(0,1512ace8,179f662,407,0,...) at vm_daemon+0xa6/frame 0x1512acac fork_exit(144ecb0,0,1512ace8,0,0,...) at fork_exit+0x7e/frame 0x1512acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1512acd4 --- trap 0, eip = 0, esp = 0x1512ad20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100096 td 0x1b2fe000 sched_switch(1b2fe000,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1abfd978 mi_switch(104,0,17e33fd,22b,1b2fe000,...) at mi_switch+0x1df/frame 0x1abfd9b4 sleepq_switch(1b2fe000,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1abfd9e8 sleepq_timedwait(207b5dc,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1abfda24 _sleep(207b5dc,207b300,254,173002a,1999997c,...) at _sleep+0x342/frame 0x1abfda78 vm_pageout_worker(0,0,bfbd6b8,0,0,...) at vm_pageout_worker+0x293/frame 0x1abfdc7c vm_pageout(0,1abfdce8,179f662,407,0,...) at vm_pageout+0x16a/frame 0x1abfdcac fork_exit(1443820,0,1abfdce8,0,0,...) at fork_exit+0x7e/frame 0x1abfdcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abfdcd4 --- trap 0, eip = 0, esp = 0x1abfdd20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100101 td 0x1b067000 sched_switch(1b067000,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1ea86a10 mi_switch(104,0,17e33fd,22b,1b067000,...) at mi_switch+0x1df/frame 0x1ea86a4c sleepq_switch(1b067000,0,17e33fd,2b3,1b067000,...) at sleepq_switch+0x110/frame 0x1ea86a80 sleepq_wait(207b5e8,54,181e418,0,0,...) at sleepq_wait+0x48/frame 0x1ea86aa4 _sleep(207b5e8,207b240,54,181e418,0,...) at _sleep+0x3a6/frame 0x1ea86af8 vm_pageout_laundry_worker(0,1ea86ce8,179f662,407,0,...) at vm_pageout_laundry_worker+0xfeb/frame 0x1ea86cac fork_exit(14439f0,0,1ea86ce8,0,0,...) at fork_exit+0x7e/frame 0x1ea86cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ea86cd4 --- trap 0, eip = 0, esp = 0x1ea86d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100102 td 0x1b066a80 sched_switch(1b066a80,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1ea8bb70 mi_switch(104,0,17e33fd,22b,1b066a80,...) at mi_switch+0x1df/frame 0x1ea8bbac sleepq_switch(1b066a80,0,17e33fd,2d6,1fdcec4,...) at sleepq_switch+0x110/frame 0x1ea8bbe0 sleepq_timedwait(1420810,54,1ea8bc50,0,0,...) at sleepq_timedwait+0x53/frame 0x1ea8bc20 _sleep(1420810,2050cb4,54,178f606,fffffed8,...) at _sleep+0x342/frame 0x1ea8bc74 uma_reclaim_worker(0,1ea8bce8,179f662,407,0,...) at uma_reclaim_worker+0x147/frame 0x1ea8bcac fork_exit(1420ea0,0,1ea8bce8,0,0,...) at fork_exit+0x7e/frame 0x1ea8bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ea8bcd4 --- trap 0, eip = 0, esp = 0x1ea8bd20, ebp = 0 --- (null)() at 0 Tracing command rand_harvestq pid 19 tid 100090 td 0xbff8380 sched_switch(bff8380,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1abdfb48 mi_switch(104,0,17e33fd,22b,bff8380,...) at mi_switch+0x1df/frame 0x1abdfb84 sleepq_switch(bff8380,0,17e33fd,2d6,85df,...) at sleepq_switch+0x110/frame 0x1abdfbb8 sleepq_timedwait(1e9720c,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1abdfbf4 _sleep(1e9720c,0,0,1733677,19999999,...) at _sleep+0x342/frame 0x1abdfc48 random_kthread(0,1abdfce8,179f662,407,0,...) at random_kthread+0x7c/frame 0x1abdfcac fork_exit(dd8150,0,1abdfce8,0,0,...) at fork_exit+0x7e/frame 0x1abdfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abdfcd4 --- trap 0, eip = 0, esp = 0x1abdfd20, ebp = 0 --- (null)() at 0 Tracing command sctp_iterator pid 18 tid 100089 td 0xbff8700 sched_switch(bff8700,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1abdab98 mi_switch(104,0,17e33fd,22b,bff8700,...) at mi_switch+0x1df/frame 0x1abdabd4 sleepq_switch(bff8700,0,17e33fd,2b3,bff8700,...) at sleepq_switch+0x110/frame 0x1abdac08 sleepq_wait(207a6c0,0,172805a,0,0,...) at sleepq_wait+0x48/frame 0x1abdac2c _sleep(207a6c0,207a688,0,172805a,0,...) at _sleep+0x3a6/frame 0x1abdac80 sctp_iterator_thread(0,1abdace8,179f662,407,0,...) at sctp_iterator_thread+0x7c/frame 0x1abdacac fork_exit(1280900,0,1abdace8,0,0,...) at fork_exit+0x7e/frame 0x1abdacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abdacd4 --- trap 0, eip = 0, esp = 0x1abdad20, ebp = 0 --- (null)() at 0 Tracing command soaiod4 pid 17 tid 100088 td 0x1b1fa380 sched_switch(1b1fa380,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1abd4b70 mi_switch(104,0,17e33fd,22b,1b1fa380,...) at mi_switch+0x1df/frame 0x1abd4bac sleepq_switch(1b1fa380,0,17e33fd,2d6,85f3,...) at sleepq_switch+0x110/frame 0x1abd4be0 sleepq_timedwait(203b0d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abd4c1c _sleep(203b0d8,203b0b8,0,1733677,ffffdd50,...) at _sleep+0x342/frame 0x1abd4c70 soaio_kproc_loop(4,1abd4ce8,179f662,407,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abd4cac fork_exit(114ce20,4,1abd4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abd4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abd4cd4 --- trap 0, eip = 0, esp = 0x1abd4d20, ebp = 0 --- (null)() at 0 Tracing command soaiod3 pid 16 tid 100087 td 0x1b1fa700 sched_switch(1b1fa700,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1abcfb70 mi_switch(104,0,17e33fd,22b,1b1fa700,...) at mi_switch+0x1df/frame 0x1abcfbac sleepq_switch(1b1fa700,0,17e33fd,2d6,85f7,...) at sleepq_switch+0x110/frame 0x1abcfbe0 sleepq_timedwait(203b0d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abcfc1c _sleep(203b0d8,203b0b8,0,1733677,ffffdd50,...) at _sleep+0x342/frame 0x1abcfc70 soaio_kproc_loop(3,1abcfce8,179f662,407,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abcfcac fork_exit(114ce20,3,1abcfce8,0,0,...) at fork_exit+0x7e/frame 0x1abcfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abcfcd4 --- trap 0, eip = 0, esp = 0x1abcfd20, ebp = 0 --- (null)() at 0 Tracing command soaiod2 pid 15 tid 100086 td 0x1b1faa80 sched_switch(1b1faa80,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1abcab70 mi_switch(104,0,17e33fd,22b,1b1faa80,...) at mi_switch+0x1df/frame 0x1abcabac sleepq_switch(1b1faa80,0,17e33fd,2d6,85f4,...) at sleepq_switch+0x110/frame 0x1abcabe0 sleepq_timedwait(203b0d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abcac1c _sleep(203b0d8,203b0b8,0,1733677,ffffdd50,...) at _sleep+0x342/frame 0x1abcac70 soaio_kproc_loop(2,1abcace8,179f662,407,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abcacac fork_exit(114ce20,2,1abcace8,0,0,...) at fork_exit+0x7e/frame 0x1abcacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abcacd4 --- trap 0, eip = 0, esp = 0x1abcad20, ebp = 0 --- (null)() at 0 Tracing command soaiod1 pid 9 tid 100085 td 0x1b1fb000 sched_switch(1b1fb000,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1abc5b70 mi_switch(104,0,17e33fd,22b,1b1fb000,...) at mi_switch+0x1df/frame 0x1abc5bac sleepq_switch(1b1fb000,0,17e33fd,2d6,85f6,...) at sleepq_switch+0x110/frame 0x1abc5be0 sleepq_timedwait(203b0d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abc5c1c _sleep(203b0d8,203b0b8,0,1733677,ffffdd50,...) at _sleep+0x342/frame 0x1abc5c70 soaio_kproc_loop(1,1abc5ce8,179f662,407,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abc5cac fork_exit(114ce20,1,1abc5ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc5cd4 --- trap 0, eip = 0, esp = 0x1abc5d20, ebp = 0 --- (null)() at 0 Tracing command fdc0 pid 8 tid 100082 td 0x1b067a80 sched_switch(1b067a80,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1abb6b40 mi_switch(104,0,17e33fd,22b,1b067a80,...) at mi_switch+0x1df/frame 0x1abb6b7c sleepq_switch(1b067a80,0,17e33fd,2b3,1b067a80,...) at sleepq_switch+0x110/frame 0x1abb6bb0 sleepq_wait(1acd323c,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1abb6bd4 _sleep(1acd323c,1acd32f0,5c,1733677,0,...) at _sleep+0x3a6/frame 0x1abb6c28 fdc_thread(1acd3200,1abb6ce8,179f662,407,0,...) at fdc_thread+0x336/frame 0x1abb6cac fork_exit(16223b0,1acd3200,1abb6ce8,0,0,...) at fork_exit+0x7e/frame 0x1abb6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abb6cd4 --- trap 0, eip = 0, esp = 0x1abb6d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100047 td 0x1ad33000 sched_switch(1ad33000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aafeba8 mi_switch(104,0,17e33fd,22b,1ad33000,...) at mi_switch+0x1df/frame 0x1aafebe8 sleepq_switch(1ad33000,0,17e33fd,2b3,1ad0fcf8,...) at sleepq_switch+0x110/frame 0x1aafec1c sleepq_wait(1ad0facc,0,1aafec60,1,0,...) at sleepq_wait+0x48/frame 0x1aafec40 _cv_wait(1ad0facc,1ad0fcf8,17c56c6,78,1ad0fad4,...) at _cv_wait+0x1d7/frame 0x1aafec84 usb_process(1ad0fac4,1aafece8,179f662,407,0,...) at usb_process+0x140/frame 0x1aafecac fork_exit(f08bd0,1ad0fac4,1aafece8,0,0,...) at fork_exit+0x7e/frame 0x1aafecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aafecd4 --- trap 0, eip = 0, esp = 0x1aafed20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100048 td 0x1ad32a80 sched_switch(1ad32a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab03ba8 mi_switch(104,0,17e33fd,22b,1ad32a80,...) at mi_switch+0x1df/frame 0x1ab03be8 sleepq_switch(1ad32a80,0,17e33fd,2b3,1ad0fcf8,...) at sleepq_switch+0x110/frame 0x1ab03c1c sleepq_wait(1ad0fafc,0,1ab03c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab03c40 _cv_wait(1ad0fafc,1ad0fcf8,17c56c6,78,1ad0fb04,...) at _cv_wait+0x1d7/frame 0x1ab03c84 usb_process(1ad0faf4,1ab03ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab03cac fork_exit(f08bd0,1ad0faf4,1ab03ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab03cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab03cd4 --- trap 0, eip = 0, esp = 0x1ab03d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100049 td 0x1ad32700 sched_switch(1ad32700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab08ba8 mi_switch(104,0,17e33fd,22b,1ad32700,...) at mi_switch+0x1df/frame 0x1ab08be8 sleepq_switch(1ad32700,0,17e33fd,2b3,1ad0fcf8,...) at sleepq_switch+0x110/frame 0x1ab08c1c sleepq_wait(1ad0fb2c,0,1ab08c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab08c40 _cv_wait(1ad0fb2c,1ad0fcf8,17c56c6,78,1ad0fb34,...) at _cv_wait+0x1d7/frame 0x1ab08c84 usb_process(1ad0fb24,1ab08ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab08cac fork_exit(f08bd0,1ad0fb24,1ab08ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab08cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab08cd4 --- trap 0, eip = 0, esp = 0x1ab08d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100050 td 0x1ad32380 sched_switch(1ad32380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab0dba8 mi_switch(104,0,17e33fd,22b,1ad32380,...) at mi_switch+0x1df/frame 0x1ab0dbe8 sleepq_switch(1ad32380,0,17e33fd,2b3,1ad0fcf8,...) at sleepq_switch+0x110/frame 0x1ab0dc1c sleepq_wait(1ad0fb5c,0,1ab0dc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab0dc40 _cv_wait(1ad0fb5c,1ad0fcf8,17c56c6,78,1ad0fb64,...) at _cv_wait+0x1d7/frame 0x1ab0dc84 usb_process(1ad0fb54,1ab0dce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab0dcac fork_exit(f08bd0,1ad0fb54,1ab0dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab0dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab0dcd4 --- trap 0, eip = 0, esp = 0x1ab0dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100051 td 0x1ad32000 sched_switch(1ad32000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab12ba8 mi_switch(104,0,17e33fd,22b,1ad32000,...) at mi_switch+0x1df/frame 0x1ab12be8 sleepq_switch(1ad32000,0,17e33fd,2b3,1ad0fcf8,...) at sleepq_switch+0x110/frame 0x1ab12c1c sleepq_wait(1ad0fb8c,0,1ab12c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab12c40 _cv_wait(1ad0fb8c,1ad0fcf8,17c56c6,78,1ad0fb94,...) at _cv_wait+0x1d7/frame 0x1ab12c84 usb_process(1ad0fb84,1ab12ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab12cac fork_exit(f08bd0,1ad0fb84,1ab12ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab12cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab12cd4 --- trap 0, eip = 0, esp = 0x1ab12d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100053 td 0x1ad21700 sched_switch(1ad21700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab1dba8 mi_switch(104,0,17e33fd,22b,1ad21700,...) at mi_switch+0x1df/frame 0x1ab1dbe8 sleepq_switch(1ad21700,0,17e33fd,2b3,1afdacf8,...) at sleepq_switch+0x110/frame 0x1ab1dc1c sleepq_wait(1afdaacc,0,1ab1dc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab1dc40 _cv_wait(1afdaacc,1afdacf8,17c56c6,78,1afdaad4,...) at _cv_wait+0x1d7/frame 0x1ab1dc84 usb_process(1afdaac4,1ab1dce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab1dcac fork_exit(f08bd0,1afdaac4,1ab1dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab1dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab1dcd4 --- trap 0, eip = 0, esp = 0x1ab1dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100054 td 0x1ad21380 sched_switch(1ad21380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab22ba8 mi_switch(104,0,17e33fd,22b,1ad21380,...) at mi_switch+0x1df/frame 0x1ab22be8 sleepq_switch(1ad21380,0,17e33fd,2b3,1afdacf8,...) at sleepq_switch+0x110/frame 0x1ab22c1c sleepq_wait(1afdaafc,0,1ab22c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab22c40 _cv_wait(1afdaafc,1afdacf8,17c56c6,78,1afdab04,...) at _cv_wait+0x1d7/frame 0x1ab22c84 usb_process(1afdaaf4,1ab22ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab22cac fork_exit(f08bd0,1afdaaf4,1ab22ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab22cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab22cd4 --- trap 0, eip = 0, esp = 0x1ab22d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100055 td 0x1ad21000 sched_switch(1ad21000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab27ba8 mi_switch(104,0,17e33fd,22b,1ad21000,...) at mi_switch+0x1df/frame 0x1ab27be8 sleepq_switch(1ad21000,0,17e33fd,2b3,1afdacf8,...) at sleepq_switch+0x110/frame 0x1ab27c1c sleepq_wait(1afdab2c,0,1ab27c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab27c40 _cv_wait(1afdab2c,1afdacf8,17c56c6,78,1afdab34,...) at _cv_wait+0x1d7/frame 0x1ab27c84 usb_process(1afdab24,1ab27ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab27cac fork_exit(f08bd0,1afdab24,1ab27ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab27cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab27cd4 --- trap 0, eip = 0, esp = 0x1ab27d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100056 td 0x1afe3380 sched_switch(1afe3380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab2cba8 mi_switch(104,0,17e33fd,22b,1afe3380,...) at mi_switch+0x1df/frame 0x1ab2cbe8 sleepq_switch(1afe3380,0,17e33fd,2b3,1afdacf8,...) at sleepq_switch+0x110/frame 0x1ab2cc1c sleepq_wait(1afdab5c,0,1ab2cc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab2cc40 _cv_wait(1afdab5c,1afdacf8,17c56c6,78,1afdab64,...) at _cv_wait+0x1d7/frame 0x1ab2cc84 usb_process(1afdab54,1ab2cce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab2ccac fork_exit(f08bd0,1afdab54,1ab2cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab2ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab2ccd4 --- trap 0, eip = 0, esp = 0x1ab2cd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100057 td 0x1afe3000 sched_switch(1afe3000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab31ba8 mi_switch(104,0,17e33fd,22b,1afe3000,...) at mi_switch+0x1df/frame 0x1ab31be8 sleepq_switch(1afe3000,0,17e33fd,2b3,1afdacf8,...) at sleepq_switch+0x110/frame 0x1ab31c1c sleepq_wait(1afdab8c,0,1ab31c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab31c40 _cv_wait(1afdab8c,1afdacf8,17c56c6,78,1afdab94,...) at _cv_wait+0x1d7/frame 0x1ab31c84 usb_process(1afdab84,1ab31ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab31cac fork_exit(f08bd0,1afdab84,1ab31ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab31cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab31cd4 --- trap 0, eip = 0, esp = 0x1ab31d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100059 td 0x1afe2700 sched_switch(1afe2700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab3cba8 mi_switch(104,0,17e33fd,22b,1afe2700,...) at mi_switch+0x1df/frame 0x1ab3cbe8 sleepq_switch(1afe2700,0,17e33fd,2b3,1afe4cf8,...) at sleepq_switch+0x110/frame 0x1ab3cc1c sleepq_wait(1afe4acc,0,1ab3cc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab3cc40 _cv_wait(1afe4acc,1afe4cf8,17c56c6,78,1afe4ad4,...) at _cv_wait+0x1d7/frame 0x1ab3cc84 usb_process(1afe4ac4,1ab3cce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab3ccac fork_exit(f08bd0,1afe4ac4,1ab3cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab3ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab3ccd4 --- trap 0, eip = 0, esp = 0x1ab3cd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100060 td 0x1afe2380 sched_switch(1afe2380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab41ba8 mi_switch(104,0,17e33fd,22b,1afe2380,...) at mi_switch+0x1df/frame 0x1ab41be8 sleepq_switch(1afe2380,0,17e33fd,2b3,1afe4cf8,...) at sleepq_switch+0x110/frame 0x1ab41c1c sleepq_wait(1afe4afc,0,1ab41c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab41c40 _cv_wait(1afe4afc,1afe4cf8,17c56c6,78,1afe4b04,...) at _cv_wait+0x1d7/frame 0x1ab41c84 usb_process(1afe4af4,1ab41ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab41cac fork_exit(f08bd0,1afe4af4,1ab41ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab41cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab41cd4 --- trap 0, eip = 0, esp = 0x1ab41d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100061 td 0x1afe2000 sched_switch(1afe2000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab46ba8 mi_switch(104,0,17e33fd,22b,1afe2000,...) at mi_switch+0x1df/frame 0x1ab46be8 sleepq_switch(1afe2000,0,17e33fd,2b3,1afe4cf8,...) at sleepq_switch+0x110/frame 0x1ab46c1c sleepq_wait(1afe4b2c,0,1ab46c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab46c40 _cv_wait(1afe4b2c,1afe4cf8,17c56c6,78,1afe4b34,...) at _cv_wait+0x1d7/frame 0x1ab46c84 usb_process(1afe4b24,1ab46ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab46cac fork_exit(f08bd0,1afe4b24,1ab46ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab46cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab46cd4 --- trap 0, eip = 0, esp = 0x1ab46d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100062 td 0x1afe1a80 sched_switch(1afe1a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab4bba8 mi_switch(104,0,17e33fd,22b,1afe1a80,...) at mi_switch+0x1df/frame 0x1ab4bbe8 sleepq_switch(1afe1a80,0,17e33fd,2b3,1afe4cf8,...) at sleepq_switch+0x110/frame 0x1ab4bc1c sleepq_wait(1afe4b5c,0,1ab4bc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab4bc40 _cv_wait(1afe4b5c,1afe4cf8,17c56c6,78,1afe4b64,...) at _cv_wait+0x1d7/frame 0x1ab4bc84 usb_process(1afe4b54,1ab4bce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab4bcac fork_exit(f08bd0,1afe4b54,1ab4bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab4bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab4bcd4 --- trap 0, eip = 0, esp = 0x1ab4bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100063 td 0x1afe1700 sched_switch(1afe1700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab50ba8 mi_switch(104,0,17e33fd,22b,1afe1700,...) at mi_switch+0x1df/frame 0x1ab50be8 sleepq_switch(1afe1700,0,17e33fd,2b3,1afe4cf8,...) at sleepq_switch+0x110/frame 0x1ab50c1c sleepq_wait(1afe4b8c,0,1ab50c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab50c40 _cv_wait(1afe4b8c,1afe4cf8,17c56c6,78,1afe4b94,...) at _cv_wait+0x1d7/frame 0x1ab50c84 usb_process(1afe4b84,1ab50ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab50cac fork_exit(f08bd0,1afe4b84,1ab50ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab50cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab50cd4 --- trap 0, eip = 0, esp = 0x1ab50d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100064 td 0x1afe1380 sched_switch(1afe1380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab56ba8 mi_switch(104,0,17e33fd,22b,1afe1380,...) at mi_switch+0x1df/frame 0x1ab56be8 sleepq_switch(1afe1380,0,17e33fd,2b3,1afe8cf8,...) at sleepq_switch+0x110/frame 0x1ab56c1c sleepq_wait(1afe8acc,0,1ab56c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab56c40 _cv_wait(1afe8acc,1afe8cf8,17c56c6,78,1afe8ad4,...) at _cv_wait+0x1d7/frame 0x1ab56c84 usb_process(1afe8ac4,1ab56ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab56cac fork_exit(f08bd0,1afe8ac4,1ab56ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab56cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab56cd4 --- trap 0, eip = 0, esp = 0x1ab56d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100065 td 0x1afe1000 sched_switch(1afe1000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab5bba8 mi_switch(104,0,17e33fd,22b,1afe1000,...) at mi_switch+0x1df/frame 0x1ab5bbe8 sleepq_switch(1afe1000,0,17e33fd,2b3,1afe8cf8,...) at sleepq_switch+0x110/frame 0x1ab5bc1c sleepq_wait(1afe8afc,0,1ab5bc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab5bc40 _cv_wait(1afe8afc,1afe8cf8,17c56c6,78,1afe8b04,...) at _cv_wait+0x1d7/frame 0x1ab5bc84 usb_process(1afe8af4,1ab5bce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab5bcac fork_exit(f08bd0,1afe8af4,1ab5bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab5bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab5bcd4 --- trap 0, eip = 0, esp = 0x1ab5bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100066 td 0x1ad34a80 sched_switch(1ad34a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab60ba8 mi_switch(104,0,17e33fd,22b,1ad34a80,...) at mi_switch+0x1df/frame 0x1ab60be8 sleepq_switch(1ad34a80,0,17e33fd,2b3,1afe8cf8,...) at sleepq_switch+0x110/frame 0x1ab60c1c sleepq_wait(1afe8b2c,0,1ab60c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab60c40 _cv_wait(1afe8b2c,1afe8cf8,17c56c6,78,1afe8b34,...) at _cv_wait+0x1d7/frame 0x1ab60c84 usb_process(1afe8b24,1ab60ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab60cac fork_exit(f08bd0,1afe8b24,1ab60ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab60cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab60cd4 --- trap 0, eip = 0, esp = 0x1ab60d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100067 td 0x1ad34700 sched_switch(1ad34700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab65ba8 mi_switch(104,0,17e33fd,22b,1ad34700,...) at mi_switch+0x1df/frame 0x1ab65be8 sleepq_switch(1ad34700,0,17e33fd,2b3,1afe8cf8,...) at sleepq_switch+0x110/frame 0x1ab65c1c sleepq_wait(1afe8b5c,0,1ab65c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab65c40 _cv_wait(1afe8b5c,1afe8cf8,17c56c6,78,1afe8b64,...) at _cv_wait+0x1d7/frame 0x1ab65c84 usb_process(1afe8b54,1ab65ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab65cac fork_exit(f08bd0,1afe8b54,1ab65ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab65cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab65cd4 --- trap 0, eip = 0, esp = 0x1ab65d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100068 td 0x1ad34380 sched_switch(1ad34380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab6aba8 mi_switch(104,0,17e33fd,22b,1ad34380,...) at mi_switch+0x1df/frame 0x1ab6abe8 sleepq_switch(1ad34380,0,17e33fd,2b3,1afe8cf8,...) at sleepq_switch+0x110/frame 0x1ab6ac1c sleepq_wait(1afe8b8c,0,1ab6ac60,1,0,...) at sleepq_wait+0x48/frame 0x1ab6ac40 _cv_wait(1afe8b8c,1afe8cf8,17c56c6,78,1afe8b94,...) at _cv_wait+0x1d7/frame 0x1ab6ac84 usb_process(1afe8b84,1ab6ace8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab6acac fork_exit(f08bd0,1afe8b84,1ab6ace8,0,0,...) at fork_exit+0x7e/frame 0x1ab6acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab6acd4 --- trap 0, eip = 0, esp = 0x1ab6ad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100069 td 0x1aff4700 sched_switch(1aff4700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab70ba8 mi_switch(104,0,17e33fd,22b,1aff4700,...) at mi_switch+0x1df/frame 0x1ab70be8 sleepq_switch(1aff4700,0,17e33fd,2b3,1afeecf8,...) at sleepq_switch+0x110/frame 0x1ab70c1c sleepq_wait(1afeeacc,0,1ab70c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab70c40 _cv_wait(1afeeacc,1afeecf8,17c56c6,78,1afeead4,...) at _cv_wait+0x1d7/frame 0x1ab70c84 usb_process(1afeeac4,1ab70ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab70cac fork_exit(f08bd0,1afeeac4,1ab70ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab70cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab70cd4 --- trap 0, eip = 0, esp = 0x1ab70d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100070 td 0x1aff4380 sched_switch(1aff4380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab75ba8 mi_switch(104,0,17e33fd,22b,1aff4380,...) at mi_switch+0x1df/frame 0x1ab75be8 sleepq_switch(1aff4380,0,17e33fd,2b3,1afeecf8,...) at sleepq_switch+0x110/frame 0x1ab75c1c sleepq_wait(1afeeafc,0,1ab75c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab75c40 _cv_wait(1afeeafc,1afeecf8,17c56c6,78,1afeeb04,...) at _cv_wait+0x1d7/frame 0x1ab75c84 usb_process(1afeeaf4,1ab75ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab75cac fork_exit(f08bd0,1afeeaf4,1ab75ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab75cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab75cd4 --- trap 0, eip = 0, esp = 0x1ab75d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100071 td 0x1aff4000 sched_switch(1aff4000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab7aba8 mi_switch(104,0,17e33fd,22b,1aff4000,...) at mi_switch+0x1df/frame 0x1ab7abe8 sleepq_switch(1aff4000,0,17e33fd,2b3,1afeecf8,...) at sleepq_switch+0x110/frame 0x1ab7ac1c sleepq_wait(1afeeb2c,0,1ab7ac60,1,0,...) at sleepq_wait+0x48/frame 0x1ab7ac40 _cv_wait(1afeeb2c,1afeecf8,17c56c6,78,1afeeb34,...) at _cv_wait+0x1d7/frame 0x1ab7ac84 usb_process(1afeeb24,1ab7ace8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab7acac fork_exit(f08bd0,1afeeb24,1ab7ace8,0,0,...) at fork_exit+0x7e/frame 0x1ab7acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab7acd4 --- trap 0, eip = 0, esp = 0x1ab7ad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100072 td 0x1aff3a80 sched_switch(1aff3a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab7fba8 mi_switch(104,0,17e33fd,22b,1aff3a80,...) at mi_switch+0x1df/frame 0x1ab7fbe8 sleepq_switch(1aff3a80,0,17e33fd,2b3,1afeecf8,...) at sleepq_switch+0x110/frame 0x1ab7fc1c sleepq_wait(1afeeb5c,0,1ab7fc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab7fc40 _cv_wait(1afeeb5c,1afeecf8,17c56c6,78,1afeeb64,...) at _cv_wait+0x1d7/frame 0x1ab7fc84 usb_process(1afeeb54,1ab7fce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab7fcac fork_exit(f08bd0,1afeeb54,1ab7fce8,0,0,...) at fork_exit+0x7e/frame 0x1ab7fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab7fcd4 --- trap 0, eip = 0, esp = 0x1ab7fd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100073 td 0x1aff3700 sched_switch(1aff3700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab84ba8 mi_switch(104,0,17e33fd,22b,1aff3700,...) at mi_switch+0x1df/frame 0x1ab84be8 sleepq_switch(1aff3700,0,17e33fd,2b3,1afeecf8,...) at sleepq_switch+0x110/frame 0x1ab84c1c sleepq_wait(1afeeb8c,0,1ab84c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab84c40 _cv_wait(1afeeb8c,1afeecf8,17c56c6,78,1afeeb94,...) at _cv_wait+0x1d7/frame 0x1ab84c84 usb_process(1afeeb84,1ab84ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab84cac fork_exit(f08bd0,1afeeb84,1ab84ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab84cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab84cd4 --- trap 0, eip = 0, esp = 0x1ab84d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100075 td 0x1aff3000 sched_switch(1aff3000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab8fba8 mi_switch(104,0,17e33fd,22b,1aff3000,...) at mi_switch+0x1df/frame 0x1ab8fbe8 sleepq_switch(1aff3000,0,17e33fd,2b3,1acfaed0,...) at sleepq_switch+0x110/frame 0x1ab8fc1c sleepq_wait(1acfaca4,0,1ab8fc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab8fc40 _cv_wait(1acfaca4,1acfaed0,17c56c6,78,1acfacac,...) at _cv_wait+0x1d7/frame 0x1ab8fc84 usb_process(1acfac9c,1ab8fce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab8fcac fork_exit(f08bd0,1acfac9c,1ab8fce8,0,0,...) at fork_exit+0x7e/frame 0x1ab8fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab8fcd4 --- trap 0, eip = 0, esp = 0x1ab8fd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100076 td 0x1aff2a80 sched_switch(1aff2a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab94ba8 mi_switch(104,0,17e33fd,22b,1aff2a80,...) at mi_switch+0x1df/frame 0x1ab94be8 sleepq_switch(1aff2a80,0,17e33fd,2b3,1acfaed0,...) at sleepq_switch+0x110/frame 0x1ab94c1c sleepq_wait(1acfacd4,0,1ab94c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab94c40 _cv_wait(1acfacd4,1acfaed0,17c56c6,78,1acfacdc,...) at _cv_wait+0x1d7/frame 0x1ab94c84 usb_process(1acfaccc,1ab94ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab94cac fork_exit(f08bd0,1acfaccc,1ab94ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab94cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab94cd4 --- trap 0, eip = 0, esp = 0x1ab94d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100077 td 0x1aff2700 sched_switch(1aff2700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab99ba8 mi_switch(104,0,17e33fd,22b,1aff2700,...) at mi_switch+0x1df/frame 0x1ab99be8 sleepq_switch(1aff2700,0,17e33fd,2b3,1acfaed0,...) at sleepq_switch+0x110/frame 0x1ab99c1c sleepq_wait(1acfad04,0,1ab99c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab99c40 _cv_wait(1acfad04,1acfaed0,17c56c6,78,1acfad0c,...) at _cv_wait+0x1d7/frame 0x1ab99c84 usb_process(1acfacfc,1ab99ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab99cac fork_exit(f08bd0,1acfacfc,1ab99ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab99cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab99cd4 --- trap 0, eip = 0, esp = 0x1ab99d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100078 td 0x1aff2380 sched_switch(1aff2380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1ab9eba8 mi_switch(104,0,17e33fd,22b,1aff2380,...) at mi_switch+0x1df/frame 0x1ab9ebe8 sleepq_switch(1aff2380,0,17e33fd,2b3,1acfaed0,...) at sleepq_switch+0x110/frame 0x1ab9ec1c sleepq_wait(1acfad34,0,1ab9ec60,1,0,...) at sleepq_wait+0x48/frame 0x1ab9ec40 _cv_wait(1acfad34,1acfaed0,17c56c6,78,1acfad3c,...) at _cv_wait+0x1d7/frame 0x1ab9ec84 usb_process(1acfad2c,1ab9ece8,179f662,407,0,...) at usb_process+0x140/frame 0x1ab9ecac fork_exit(f08bd0,1acfad2c,1ab9ece8,0,0,...) at fork_exit+0x7e/frame 0x1ab9ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab9ecd4 --- trap 0, eip = 0, esp = 0x1ab9ed20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100079 td 0x1aff2000 sched_switch(1aff2000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aba3ba8 mi_switch(104,0,17e33fd,22b,1aff2000,...) at mi_switch+0x1df/frame 0x1aba3be8 sleepq_switch(1aff2000,0,17e33fd,2b3,1acfaed0,...) at sleepq_switch+0x110/frame 0x1aba3c1c sleepq_wait(1acfad64,0,1aba3c60,1,0,...) at sleepq_wait+0x48/frame 0x1aba3c40 _cv_wait(1acfad64,1acfaed0,17c56c6,78,1acfad6c,...) at _cv_wait+0x1d7/frame 0x1aba3c84 usb_process(1acfad5c,1aba3ce8,179f662,407,0,...) at usb_process+0x140/frame 0x1aba3cac fork_exit(f08bd0,1acfad5c,1aba3ce8,0,0,...) at fork_exit+0x7e/frame 0x1aba3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aba3cd4 --- trap 0, eip = 0, esp = 0x1aba3d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100042 td 0x9fa9a80 sched_switch(9fa9a80,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1aab4b78 mi_switch(104,0,17e33fd,22b,9fa9a80,...) at mi_switch+0x1df/frame 0x1aab4bb4 sleepq_switch(9fa9a80,0,17e33fd,2b3,9fa9a80,...) at sleepq_switch+0x110/frame 0x1aab4be8 sleepq_wait(1de45c0,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aab4c0c _sleep(1de45c0,1de4580,5c,1733677,0,...) at _sleep+0x3a6/frame 0x1aab4c60 xpt_done_td(1de4580,1aab4ce8,179f662,407,0,...) at xpt_done_td+0x9b/frame 0x1aab4cac fork_exit(90aa50,1de4580,1aab4ce8,0,0,...) at fork_exit+0x7e/frame 0x1aab4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aab4cd4 --- trap 0, eip = 0, esp = 0x1aab4d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100095 td 0x1b2fe380 sched_switch(1b2fe380,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1abf8b70 mi_switch(104,0,17e33fd,22b,1b2fe380,...) at mi_switch+0x1df/frame 0x1abf8bac sleepq_switch(1b2fe380,0,17e33fd,2b3,1b2fe380,...) at sleepq_switch+0x110/frame 0x1abf8be0 sleepq_wait(1de44ec,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1abf8c04 _sleep(1de44ec,1de4540,5c,1733677,0,...) at _sleep+0x3a6/frame 0x1abf8c58 xpt_scanner_thread(0,1abf8ce8,179f662,407,0,...) at xpt_scanner_thread+0x97/frame 0x1abf8cac fork_exit(90add0,0,1abf8ce8,0,0,...) at fork_exit+0x7e/frame 0x1abf8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abf8cd4 --- trap 0, eip = 0, esp = 0x1abf8d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 3 pid 6 tid 100041 td 0x9faa000 sched_switch(9faa000,0,104,18a,1fcf380,...) at sched_switch+0x80c/frame 0x1aaafb70 mi_switch(104,0,17e33fd,22b,9faa000,...) at mi_switch+0x1df/frame 0x1aaafbac sleepq_switch(9faa000,0,17e33fd,2b3,9faa000,...) at sleepq_switch+0x110/frame 0x1aaafbe0 sleepq_wait(9fb10c4,6c,1813ef8,0,0,...) at sleepq_wait+0x48/frame 0x1aaafc04 _sleep(9fb10c4,9fb10a8,6c,1813ef8,0,...) at _sleep+0x3a6/frame 0x1aaafc58 crypto_ret_proc(9fb10a8,1aaafce8,179f662,407,0,...) at crypto_ret_proc+0x241/frame 0x1aaafcac fork_exit(1387d20,9fb10a8,1aaafce8,0,0,...) at fork_exit+0x7e/frame 0x1aaafcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaafcd4 --- trap 0, eip = 0, esp = 0x1aaafd20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 2 pid 5 tid 100040 td 0x9faa380 sched_switch(9faa380,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x1aaaab70 mi_switch(104,0,17e33fd,22b,9faa380,...) at mi_switch+0x1df/frame 0x1aaaabac sleepq_switch(9faa380,0,17e33fd,2b3,9faa380,...) at sleepq_switch+0x110/frame 0x1aaaabe0 sleepq_wait(9fb108c,6c,1813ef8,0,0,...) at sleepq_wait+0x48/frame 0x1aaaac04 _sleep(9fb108c,9fb1070,6c,1813ef8,0,...) at _sleep+0x3a6/frame 0x1aaaac58 crypto_ret_proc(9fb1070,1aaaace8,179f662,407,0,...) at crypto_ret_proc+0x241/frame 0x1aaaacac fork_exit(1387d20,9fb1070,1aaaace8,0,0,...) at fork_exit+0x7e/frame 0x1aaaacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaaacd4 --- trap 0, eip = 0, esp = 0x1aaaad20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 1 pid 4 tid 100039 td 0x9faa700 sched_switch(9faa700,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1aaa5b70 mi_switch(104,0,17e33fd,22b,9faa700,...) at mi_switch+0x1df/frame 0x1aaa5bac sleepq_switch(9faa700,0,17e33fd,2b3,9faa700,...) at sleepq_switch+0x110/frame 0x1aaa5be0 sleepq_wait(9fb1054,6c,1813ef8,0,0,...) at sleepq_wait+0x48/frame 0x1aaa5c04 _sleep(9fb1054,9fb1038,6c,1813ef8,0,...) at _sleep+0x3a6/frame 0x1aaa5c58 crypto_ret_proc(9fb1038,1aaa5ce8,179f662,407,0,...) at crypto_ret_proc+0x241/frame 0x1aaa5cac fork_exit(1387d20,9fb1038,1aaa5ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa5cd4 --- trap 0, eip = 0, esp = 0x1aaa5d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 0 pid 3 tid 100038 td 0x9faaa80 sched_switch(9faaa80,0,104,18a,1fcf380,...) at sched_switch+0x80c/frame 0x1aaa0b70 mi_switch(104,0,17e33fd,22b,9faaa80,...) at mi_switch+0x1df/frame 0x1aaa0bac sleepq_switch(9faaa80,0,17e33fd,2b3,9faaa80,...) at sleepq_switch+0x110/frame 0x1aaa0be0 sleepq_wait(9fb101c,6c,1813ef8,0,0,...) at sleepq_wait+0x48/frame 0x1aaa0c04 _sleep(9fb101c,9fb1000,6c,1813ef8,0,...) at _sleep+0x3a6/frame 0x1aaa0c58 crypto_ret_proc(9fb1000,1aaa0ce8,179f662,407,0,...) at crypto_ret_proc+0x241/frame 0x1aaa0cac fork_exit(1387d20,9fb1000,1aaa0ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa0cd4 --- trap 0, eip = 0, esp = 0x1aaa0d20, ebp = 0 --- (null)() at 0 Tracing command crypto pid 2 tid 100037 td 0x9fab000 sched_switch(9fab000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa9bb80 mi_switch(104,0,17e33fd,22b,9fab000,...) at mi_switch+0x1df/frame 0x1aa9bbc0 sleepq_switch(9fab000,0,17e33fd,2b3,9fab000,...) at sleepq_switch+0x110/frame 0x1aa9bbf4 sleepq_wait(204fe4c,6c,17c8710,0,0,...) at sleepq_wait+0x48/frame 0x1aa9bc18 _sleep(204fe4c,204fe34,6c,17c8710,0,...) at _sleep+0x3a6/frame 0x1aa9bc6c crypto_proc(0,1aa9bce8,179f662,407,0,...) at crypto_proc+0x2a9/frame 0x1aa9bcac fork_exit(1387920,0,1aa9bce8,0,0,...) at fork_exit+0x7e/frame 0x1aa9bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa9bcd4 --- trap 0, eip = 0, esp = 0x1aa9bd20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100029 td 0x9e64700 sched_switch(9e64700,0,104,18a,1fcf380,...) at sched_switch+0x80c/frame 0x1aa73b88 mi_switch(104,0,17e33fd,22b,9e64700,...) at mi_switch+0x1df/frame 0x1aa73bc4 sleepq_switch(9e64700,0,17e33fd,2b3,9e64700,...) at sleepq_switch+0x110/frame 0x1aa73bf8 sleepq_wait(2076a54,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa73c1c _sleep(2076a54,1fbbc98,25c,1733677,0,...) at _sleep+0x3a6/frame 0x1aa73c70 g_run_events(0,1aa73ce8,179f662,407,0,...) at g_run_events+0x67/frame 0x1aa73cac fork_exit(1012180,0,1aa73ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa73cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa73cd4 --- trap 0, eip = 0, esp = 0x1aa73d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100030 td 0x9e64380 sched_switch(9e64380,0,104,18a,1fce600,...) at sched_switch+0x80c/frame 0x1aa78b78 mi_switch(104,0,17e33fd,22b,9e64380,...) at mi_switch+0x1df/frame 0x1aa78bb4 sleepq_switch(9e64380,0,17e33fd,2b3,9e64380,...) at sleepq_switch+0x110/frame 0x1aa78be8 sleepq_wait(2076a5c,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa78c0c _sleep(2076a5c,1fbbce0,25c,1733677,0,...) at _sleep+0x3a6/frame 0x1aa78c60 g_io_schedule_up(9e64380,0,1738dab,61,1aa78cd4,...) at g_io_schedule_up+0xcf/frame 0x1aa78c94 g_up_procbody(0,1aa78ce8,179f662,407,0,...) at g_up_procbody+0x7d/frame 0x1aa78cac fork_exit(1012200,0,1aa78ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa78cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa78cd4 --- trap 0, eip = 0, esp = 0x1aa78d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100031 td 0x9e64000 sched_switch(9e64000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa7db70 mi_switch(104,0,17e33fd,22b,9e64000,...) at mi_switch+0x1df/frame 0x1aa7dbb0 sleepq_switch(9e64000,0,17e33fd,2b3,9e64000,...) at sleepq_switch+0x110/frame 0x1aa7dbe4 sleepq_wait(2076a50,5c,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa7dc08 _sleep(2076a50,1fbbcc0,25c,1733677,0,...) at _sleep+0x3a6/frame 0x1aa7dc5c g_io_schedule_down(9e64000,0,1738dab,6d,1aa7dcd4,...) at g_io_schedule_down+0xec/frame 0x1aa7dc94 g_down_procbody(0,1aa7dce8,179f662,407,0,...) at g_down_procbody+0x7d/frame 0x1aa7dcac fork_exit(1012280,0,1aa7dce8,0,0,...) at fork_exit+0x7e/frame 0x1aa7dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa7dcd4 --- trap 0, eip = 0, esp = 0x1aa7dd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100009 td 0xbfc8000 sched_switch(bfc8000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa0fc30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aa0fc70 ithread_loop(be82010,1aa0fce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aa0fcac fork_exit(10913c0,be82010,1aa0fce8,0,0,...) at fork_exit+0x7e/frame 0x1aa0fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa0fcd4 --- trap 0, eip = 0, esp = 0x1aa0fd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100010 td 0xbfc0a80 sched_switch(bfc0a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa14c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aa14c70 ithread_loop(be82000,1aa14ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aa14cac fork_exit(10913c0,be82000,1aa14ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa14cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa14cd4 --- trap 0, eip = 0, esp = 0x1aa14d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100012 td 0xbfc0380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100023 td 0x9e8f000 sched_switch(9e8f000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa55c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aa55c70 ithread_loop(9ea2050,1aa55ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aa55cac fork_exit(10913c0,9ea2050,1aa55ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa55cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa55cd4 --- trap 0, eip = 0, esp = 0x1aa55d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100024 td 0x9e8ea80 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100025 td 0x9e8e700 sched_switch(9e8e700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa5fc30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aa5fc70 ithread_loop(9ea2030,1aa5fce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aa5fcac fork_exit(10913c0,9ea2030,1aa5fce8,0,0,...) at fork_exit+0x7e/frame 0x1aa5fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa5fcd4 --- trap 0, eip = 0, esp = 0x1aa5fd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100026 td 0x9e8e380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100027 td 0x9e8e000 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100028 td 0x9e64a80 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100043 td 0x1ad34000 sched_switch(1ad34000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aae7c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aae7c70 ithread_loop(9ea2f70,1aae7ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aae7cac fork_exit(10913c0,9ea2f70,1aae7ce8,0,0,...) at fork_exit+0x7e/frame 0x1aae7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aae7cd4 --- trap 0, eip = 0, esp = 0x1aae7d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100044 td 0x1ad33a80 sched_switch(1ad33a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aaedc30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aaedc70 ithread_loop(9ea2f50,1aaedce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aaedcac fork_exit(10913c0,9ea2f50,1aaedce8,0,0,...) at fork_exit+0x7e/frame 0x1aaedcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaedcd4 --- trap 0, eip = 0, esp = 0x1aaedd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100045 td 0x1ad33700 sched_switch(1ad33700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aaf3c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aaf3c70 ithread_loop(9ea2f40,1aaf3ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aaf3cac fork_exit(10913c0,9ea2f40,1aaf3ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaf3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaf3cd4 --- trap 0, eip = 0, esp = 0x1aaf3d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100046 td 0x1ad33380 sched_switch(1ad33380,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aaf9c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aaf9c70 ithread_loop(9ea2f30,1aaf9ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aaf9cac fork_exit(10913c0,9ea2f30,1aaf9ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaf9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaf9cd4 --- trap 0, eip = 0, esp = 0x1aaf9d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100052 td 0x1ad21a80 sched_switch(1ad21a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab18c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1ab18c70 ithread_loop(9ea2f20,1ab18ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1ab18cac fork_exit(10913c0,9ea2f20,1ab18ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab18cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab18cd4 --- trap 0, eip = 0, esp = 0x1ab18d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100058 td 0x1afe2a80 sched_switch(1afe2a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab37c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1ab37c70 ithread_loop(9ea2f10,1ab37ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1ab37cac fork_exit(10913c0,9ea2f10,1ab37ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab37cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab37cd4 --- trap 0, eip = 0, esp = 0x1ab37d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100074 td 0x1aff3380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100080 td 0x1afe3a80 sched_switch(1afe3a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aba8c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1aba8c70 ithread_loop(9ea2ef0,1aba8ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1aba8cac fork_exit(10913c0,9ea2ef0,1aba8ce8,0,0,...) at fork_exit+0x7e/frame 0x1aba8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aba8cd4 --- trap 0, eip = 0, esp = 0x1aba8d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100081 td 0x1afe3700 sched_switch(1afe3700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1abb1c30 mi_switch(109,0,17ec153,59f,179f662,...) at mi_switch+0x1df/frame 0x1abb1c70 ithread_loop(9ea2e40,1abb1ce8,179f662,407,0,...) at ithread_loop+0x200/frame 0x1abb1cac fork_exit(10913c0,9ea2e40,1abb1ce8,0,0,...) at fork_exit+0x7e/frame 0x1abb1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abb1cd4 --- trap 0, eip = 0, esp = 0x1abb1d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100083 td 0x1b067700 PTDpde() at PTDpde+0x43b8 Tracing command idle pid 11 tid 100003 td 0xbfc9700 (CPU 0) cpustop_handler(0,7b8e458,15116b94,16b675c,0,...) at cpustop_handler+0x25/frame 0x15116a80 ipi_nmi_handler(0,0,0,2428dac,0,...) at ipi_nmi_handler+0x36/frame 0x15116a90 trap(15116ba0,8,28,28,1ac89800,...) at trap+0xac/frame 0x15116b94 calltrap() at PTDpde+0x4165/frame 0x15116b94 --- trap 0x13, eip = 0x16ced35, esp = 0x15116be0, ebp = 0x15116be0 --- acpi_cpu_c1(15116c34,16bef52,2428d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15116be0 acpi_cpu_idle(ac280c6,0,ac280c6,15116c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15116c20 cpu_idle_acpi(ac280c6,0,1fce600,2,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15116c34 cpu_idle(0,15116c80,17a9b8a,ade,2,...) at cpu_idle+0xa0/frame 0x15116c50 sched_idletd(0,15116ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15116cac fork_exit(1102b90,0,15116ce8,0,0,...) at fork_exit+0x7e/frame 0x15116cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15116cd4 --- trap 0, eip = 0, esp = 0x15116d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfc9380 (CPU 1) cpustop_handler(1,7b8e458,1511bb94,16b675c,0,...) at cpustop_handler+0x25/frame 0x1511ba80 ipi_nmi_handler(0,be76dac,0,85df,1511baf0,...) at ipi_nmi_handler+0x36/frame 0x1511ba90 trap(1511bba0,8,28,28,1ac89600,...) at trap+0xac/frame 0x1511bb94 calltrap() at PTDpde+0x4165/frame 0x1511bb94 --- trap 0x13, eip = 0x16ced35, esp = 0x1511bbe0, ebp = 0x1511bbe0 --- acpi_cpu_c1(1511bc34,16bef52,be76d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x1511bbe0 acpi_cpu_idle(112cd029,0,112cd029,1511bc50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x1511bc20 cpu_idle_acpi(112cd029,0,1fcecc0,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511bc34 cpu_idle(0,1511bc80,17a9b8a,ade,3,...) at cpu_idle+0xa0/frame 0x1511bc50 sched_idletd(0,1511bce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x1511bcac fork_exit(1102b90,0,1511bce8,0,0,...) at fork_exit+0x7e/frame 0x1511bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511bcd4 --- trap 0, eip = 0, esp = 0x1511bd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfc9000 (CPU 2) cpustop_handler(2,7b8e458,15120b94,16b675c,0,...) at cpustop_handler+0x25/frame 0x15120a80 ipi_nmi_handler(0,be7bdac,0,85df,15120af0,...) at ipi_nmi_handler+0x36/frame 0x15120a90 trap(15120ba0,8,28,28,1ac89400,...) at trap+0xac/frame 0x15120b94 calltrap() at PTDpde+0x4165/frame 0x15120b94 --- trap 0x13, eip = 0x16ced35, esp = 0x15120be0, ebp = 0x15120be0 --- acpi_cpu_c1(15120c34,16bef52,be7bd90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15120be0 acpi_cpu_idle(3e2ac6,0,3e2ac6,15120c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15120c20 cpu_idle_acpi(3e2ac6,0,1fcf380,b,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15120c34 cpu_idle(0,15120c80,17a9b8a,ade,b,...) at cpu_idle+0xa0/frame 0x15120c50 sched_idletd(0,15120ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15120cac fork_exit(1102b90,0,15120ce8,0,0,...) at fork_exit+0x7e/frame 0x15120cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15120cd4 --- trap 0, eip = 0, esp = 0x15120d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfc8a80 (CPU 3) kdb_alt_break_internal(0,15125acc,ecaef7,2,1b037888,...) at kdb_alt_break_internal+0xf0/frame 0x15125aa0 kdb_alt_break(2,1b037888,1807360,0,40000,...) at kdb_alt_break+0x16/frame 0x15125aac uart_intr_rxready(1b037800,16f78b5,2f1,bfc8a80,1fcfa40,...) at uart_intr_rxready+0x87/frame 0x15125acc uart_intr(1b037800,bfc8a80,15125b28,20781d0,bfc8a80,...) at uart_intr+0x130/frame 0x15125b0c intr_event_handle(9e17580,15125ba0,1fcfa9c,bff8380,1,...) at intr_event_handle+0xa5/frame 0x15125b38 intr_execute_handlers(bfed130,15125ba0,3) at intr_execute_handlers+0xa5/frame 0x15125b70 lapic_handle_intr(32,15125ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15125b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15125b90 --- interrupt, eip = 0x16ced35, esp = 0x15125be0, ebp = 0x15125be0 --- acpi_cpu_c1(15125c34,16bef52,be80d90,0,1704a74,...) at acpi_cpu_c1+0x5/frame 0x15125be0 acpi_cpu_idle(269d86d3,0,269d86d3,15125c50,16cf380,...) at acpi_cpu_idle+0x30b/frame 0x15125c20 cpu_idle_acpi(269d86d3,0,1fcfa40,3,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15125c34 cpu_idle(0,15125c80,17a9b8a,ade,3,...) at cpu_idle+0xa0/frame 0x15125c50 sched_idletd(0,15125ce8,179f662,407,0,...) at sched_idletd+0x236/frame 0x15125cac fork_exit(1102b90,0,15125ce8,0,0,...) at fork_exit+0x7e/frame 0x15125cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15125cd4 --- trap 0, eip = 0, esp = 0x15125d20, ebp = 0 --- (null)() at 0 Tracing command init pid 1 tid 100002 td 0xbfc9a80 sched_switch(bfc9a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x15110940 mi_switch(104,0,17e33fd,22b,bfc9a80,...) at mi_switch+0x1df/frame 0x15110980 sleepq_switch(bfc9a80,0,17e33fd,1ca,6c,...) at sleepq_switch+0x110/frame 0x151109b4 sleepq_catch_signals(0,17e33fd,178,1fe0854,bfc9a80,...) at sleepq_catch_signals+0x360/frame 0x151109f8 sleepq_wait_sig(bfbea14,6c,17fecc7,100,0,...) at sleepq_wait_sig+0x14/frame 0x15110a1c _sleep(bfbea14,bfbeabc,36c,17fecc7,0,...) at _sleep+0x396/frame 0x15110a70 kern_wait6(bfc9a80,7,0,0,15110af4,...) at kern_wait6+0x567/frame 0x15110ac8 sys_wait4(bfc9a80,bfc9d04,17cfeb8,4,1704a74,...) at sys_wait4+0x87/frame 0x15110be8 syscall(15110ce8,3b,3b,3b,813a8eb,...) at syscall+0x342/frame 0x15110cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15110cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x806a577, esp = 0xffbfe80c, ebp = 0xffbfe820 --- amdsmn_devclass() at 0x806a577 Tracing command audit pid 10 tid 100001 td 0xbfca000 sched_switch(bfca000,0,104,18a,1fcecc0,...) at sched_switch+0x80c/frame 0x1510bb80 mi_switch(104,0,17e33fd,22b,bfca000,...) at mi_switch+0x1df/frame 0x1510bbbc sleepq_switch(bfca000,0,17e33fd,2b3,207ae44,...) at sleepq_switch+0x110/frame 0x1510bbf0 sleepq_wait(207ae94,0,1510bc34,1,0,...) at sleepq_wait+0x48/frame 0x1510bc14 _cv_wait(207ae94,207ae44,173149a,1be,28,...) at _cv_wait+0x1d7/frame 0x1510bc58 audit_worker(0,1510bce8,179f662,407,0,...) at audit_worker+0x94/frame 0x1510bcac fork_exit(13aa360,0,1510bce8,0,0,...) at fork_exit+0x7e/frame 0x1510bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1510bcd4 --- trap 0, eip = 0, esp = 0x1510bd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100000 td 0x2076de0 sched_switch(2076de0,0,104,18a,1fcfa40,...) at sched_switch+0x80c/frame 0x2423bf0 mi_switch(104,0,17e33fd,22b,2076de0,...) at mi_switch+0x1df/frame 0x2423c2c sleepq_switch(2076de0,0,17e33fd,2d6,85e4,...) at sleepq_switch+0x110/frame 0x2423c60 sleepq_timedwait(2076a74,54,9,0,0,...) at sleepq_timedwait+0x53/frame 0x2423c9c _sleep(2076a74,0,54,18016d7,fffff470,...) at _sleep+0x342/frame 0x2423cf0 swapper() at swapper+0x2c8/frame 0x2423d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100007 td 0xbfc8700 sched_switch(bfc8700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa05b88 mi_switch(104,0,17e33fd,22b,bfc8700,...) at mi_switch+0x1df/frame 0x1aa05bc8 sleepq_switch(bfc8700,0,17e33fd,2b3,bfc8700,...) at sleepq_switch+0x110/frame 0x1aa05bfc sleepq_wait(bfe0a00,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa05c20 _sleep(bfe0a00,bfe0a1c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa05c74 taskqueue_thread_loop(207a238,1aa05ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa05cac fork_exit(1130fb0,207a238,1aa05ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa05cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa05cd4 --- trap 0, eip = 0, esp = 0x1aa05d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100008 td 0xbfc8380 sched_switch(bfc8380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa0ab88 mi_switch(104,0,17e33fd,22b,bfc8380,...) at mi_switch+0x1df/frame 0x1aa0abc8 sleepq_switch(bfc8380,0,17e33fd,2b3,bfc8380,...) at sleepq_switch+0x110/frame 0x1aa0abfc sleepq_wait(bfe0980,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa0ac20 _sleep(bfe0980,bfe099c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa0ac74 taskqueue_thread_loop(207737c,1aa0ace8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa0acac fork_exit(1130fb0,207737c,1aa0ace8,0,0,...) at fork_exit+0x7e/frame 0x1aa0acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa0acd4 --- trap 0, eip = 0, esp = 0x1aa0ad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100011 td 0xbfc0700 sched_switch(bfc0700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa19b88 mi_switch(104,0,17e33fd,22b,bfc0700,...) at mi_switch+0x1df/frame 0x1aa19bc8 sleepq_switch(bfc0700,0,17e33fd,2b3,bfc0700,...) at sleepq_switch+0x110/frame 0x1aa19bfc sleepq_wait(bfe0800,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa19c20 _sleep(bfe0800,bfe081c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa19c74 taskqueue_thread_loop(207a1e8,1aa19ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa19cac fork_exit(1130fb0,207a1e8,1aa19ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa19cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa19cd4 --- trap 0, eip = 0, esp = 0x1aa19d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100013 td 0xbfc0000 sched_switch(bfc0000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa23b98 mi_switch(104,0,17e33fd,22b,bfc0000,...) at mi_switch+0x1df/frame 0x1aa23bd8 sleepq_switch(bfc0000,0,17e33fd,2b3,bfe0700,...) at sleepq_switch+0x110/frame 0x1aa23c0c sleepq_wait(bfe0700,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa23c30 msleep_spin_sbt(bfe0700,bfe071c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa23c74 gtaskqueue_thread_loop(bff0c04,1aa23ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa23cac fork_exit(111adf0,bff0c04,1aa23ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa23cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa23cd4 --- trap 0, eip = 0, esp = 0x1aa23d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100014 td 0xbffa380 sched_switch(bffa380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa28b98 mi_switch(104,0,17e33fd,22b,bffa380,...) at mi_switch+0x1df/frame 0x1aa28bd8 sleepq_switch(bffa380,0,17e33fd,2b3,bfe0680,...) at sleepq_switch+0x110/frame 0x1aa28c0c sleepq_wait(bfe0680,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa28c30 msleep_spin_sbt(bfe0680,bfe069c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa28c74 gtaskqueue_thread_loop(bff0c14,1aa28ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa28cac fork_exit(111adf0,bff0c14,1aa28ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa28cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa28cd4 --- trap 0, eip = 0, esp = 0x1aa28d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100015 td 0xbffa000 sched_switch(bffa000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa2db98 mi_switch(104,0,17e33fd,22b,bffa000,...) at mi_switch+0x1df/frame 0x1aa2dbd8 sleepq_switch(bffa000,0,17e33fd,2b3,bfe0600,...) at sleepq_switch+0x110/frame 0x1aa2dc0c sleepq_wait(bfe0600,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa2dc30 msleep_spin_sbt(bfe0600,bfe061c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa2dc74 gtaskqueue_thread_loop(bff0c24,1aa2dce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa2dcac fork_exit(111adf0,bff0c24,1aa2dce8,0,0,...) at fork_exit+0x7e/frame 0x1aa2dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa2dcd4 --- trap 0, eip = 0, esp = 0x1aa2dd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100016 td 0xbff9a80 sched_switch(bff9a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa32b98 mi_switch(104,0,17e33fd,22b,bff9a80,...) at mi_switch+0x1df/frame 0x1aa32bd8 sleepq_switch(bff9a80,0,17e33fd,2b3,bfe0580,...) at sleepq_switch+0x110/frame 0x1aa32c0c sleepq_wait(bfe0580,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa32c30 msleep_spin_sbt(bfe0580,bfe059c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa32c74 gtaskqueue_thread_loop(bff0c34,1aa32ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa32cac fork_exit(111adf0,bff0c34,1aa32ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa32cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa32cd4 --- trap 0, eip = 0, esp = 0x1aa32d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100017 td 0xbff9700 sched_switch(bff9700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa37b98 mi_switch(104,0,17e33fd,22b,bff9700,...) at mi_switch+0x1df/frame 0x1aa37bd8 sleepq_switch(bff9700,0,17e33fd,2b3,bfe0500,...) at sleepq_switch+0x110/frame 0x1aa37c0c sleepq_wait(bfe0500,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa37c30 msleep_spin_sbt(bfe0500,bfe051c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa37c74 gtaskqueue_thread_loop(bff0404,1aa37ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa37cac fork_exit(111adf0,bff0404,1aa37ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa37cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa37cd4 --- trap 0, eip = 0, esp = 0x1aa37d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100018 td 0xbff9380 sched_switch(bff9380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa3cb98 mi_switch(104,0,17e33fd,22b,bff9380,...) at mi_switch+0x1df/frame 0x1aa3cbd8 sleepq_switch(bff9380,0,17e33fd,2b3,bfe0480,...) at sleepq_switch+0x110/frame 0x1aa3cc0c sleepq_wait(bfe0480,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa3cc30 msleep_spin_sbt(bfe0480,bfe049c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa3cc74 gtaskqueue_thread_loop(bff0414,1aa3cce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa3ccac fork_exit(111adf0,bff0414,1aa3cce8,0,0,...) at fork_exit+0x7e/frame 0x1aa3ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa3ccd4 --- trap 0, eip = 0, esp = 0x1aa3cd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100019 td 0xbff9000 sched_switch(bff9000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa41b98 mi_switch(104,0,17e33fd,22b,bff9000,...) at mi_switch+0x1df/frame 0x1aa41bd8 sleepq_switch(bff9000,0,17e33fd,2b3,bfe0400,...) at sleepq_switch+0x110/frame 0x1aa41c0c sleepq_wait(bfe0400,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa41c30 msleep_spin_sbt(bfe0400,bfe041c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa41c74 gtaskqueue_thread_loop(bff0424,1aa41ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa41cac fork_exit(111adf0,bff0424,1aa41ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa41cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa41cd4 --- trap 0, eip = 0, esp = 0x1aa41d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100020 td 0xbff8a80 sched_switch(bff8a80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa46b98 mi_switch(104,0,17e33fd,22b,bff8a80,...) at mi_switch+0x1df/frame 0x1aa46bd8 sleepq_switch(bff8a80,0,17e33fd,2b3,bfe0380,...) at sleepq_switch+0x110/frame 0x1aa46c0c sleepq_wait(bfe0380,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa46c30 msleep_spin_sbt(bfe0380,bfe039c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa46c74 gtaskqueue_thread_loop(bff0434,1aa46ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa46cac fork_exit(111adf0,bff0434,1aa46ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa46cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa46cd4 --- trap 0, eip = 0, esp = 0x1aa46d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100021 td 0x9e8f700 sched_switch(9e8f700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa4bb98 mi_switch(104,0,17e33fd,22b,9e8f700,...) at mi_switch+0x1df/frame 0x1aa4bbd8 sleepq_switch(9e8f700,0,17e33fd,2b3,9e5e080,...) at sleepq_switch+0x110/frame 0x1aa4bc0c sleepq_wait(9e5e080,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa4bc30 msleep_spin_sbt(9e5e080,9e5e09c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa4bc74 gtaskqueue_thread_loop(bff0804,1aa4bce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa4bcac fork_exit(111adf0,bff0804,1aa4bce8,0,0,...) at fork_exit+0x7e/frame 0x1aa4bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa4bcd4 --- trap 0, eip = 0, esp = 0x1aa4bd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100022 td 0x9e8f380 sched_switch(9e8f380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa50b98 mi_switch(104,0,17e33fd,22b,9e8f380,...) at mi_switch+0x1df/frame 0x1aa50bd8 sleepq_switch(9e8f380,0,17e33fd,2b3,9e5e000,...) at sleepq_switch+0x110/frame 0x1aa50c0c sleepq_wait(9e5e000,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1aa50c30 msleep_spin_sbt(9e5e000,9e5e01c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa50c74 gtaskqueue_thread_loop(bff1004,1aa50ce8,179f662,407,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa50cac fork_exit(111adf0,bff1004,1aa50ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa50cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa50cd4 --- trap 0, eip = 0, esp = 0x1aa50d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100032 td 0xbffaa80 sched_switch(bffaa80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa82b88 mi_switch(104,0,17e33fd,22b,bffaa80,...) at mi_switch+0x1df/frame 0x1aa82bc8 sleepq_switch(bffaa80,0,17e33fd,2b3,bffaa80,...) at sleepq_switch+0x110/frame 0x1aa82bfc sleepq_wait(9e5df00,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa82c20 _sleep(9e5df00,9e5df1c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa82c74 taskqueue_thread_loop(1fdbf40,1aa82ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa82cac fork_exit(1130fb0,1fdbf40,1aa82ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa82cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa82cd4 --- trap 0, eip = 0, esp = 0x1aa82d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100033 td 0xbffa700 sched_switch(bffa700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa87b88 mi_switch(104,0,17e33fd,22b,bffa700,...) at mi_switch+0x1df/frame 0x1aa87bc8 sleepq_switch(bffa700,0,17e33fd,2b3,bffa700,...) at sleepq_switch+0x110/frame 0x1aa87bfc sleepq_wait(9e5de80,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa87c20 _sleep(9e5de80,9e5de9c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa87c74 taskqueue_thread_loop(204fef0,1aa87ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa87cac fork_exit(1130fb0,204fef0,1aa87ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa87cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa87cd4 --- trap 0, eip = 0, esp = 0x1aa87d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100034 td 0x9faba80 sched_switch(9faba80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa8cb88 mi_switch(104,0,17e33fd,22b,9faba80,...) at mi_switch+0x1df/frame 0x1aa8cbc8 sleepq_switch(9faba80,0,17e33fd,2b3,9faba80,...) at sleepq_switch+0x110/frame 0x1aa8cbfc sleepq_wait(9e5de80,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa8cc20 _sleep(9e5de80,9e5de9c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa8cc74 taskqueue_thread_loop(204fef0,1aa8cce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa8ccac fork_exit(1130fb0,204fef0,1aa8cce8,0,0,...) at fork_exit+0x7e/frame 0x1aa8ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa8ccd4 --- trap 0, eip = 0, esp = 0x1aa8cd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100035 td 0x9fab700 sched_switch(9fab700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa91b88 mi_switch(104,0,17e33fd,22b,9fab700,...) at mi_switch+0x1df/frame 0x1aa91bc8 sleepq_switch(9fab700,0,17e33fd,2b3,9fab700,...) at sleepq_switch+0x110/frame 0x1aa91bfc sleepq_wait(9e5de80,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa91c20 _sleep(9e5de80,9e5de9c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa91c74 taskqueue_thread_loop(204fef0,1aa91ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa91cac fork_exit(1130fb0,204fef0,1aa91ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa91cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa91cd4 --- trap 0, eip = 0, esp = 0x1aa91d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100036 td 0x9fab380 sched_switch(9fab380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1aa96b88 mi_switch(104,0,17e33fd,22b,9fab380,...) at mi_switch+0x1df/frame 0x1aa96bc8 sleepq_switch(9fab380,0,17e33fd,2b3,9fab380,...) at sleepq_switch+0x110/frame 0x1aa96bfc sleepq_wait(9e5de80,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1aa96c20 _sleep(9e5de80,9e5de9c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1aa96c74 taskqueue_thread_loop(204fef0,1aa96ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa96cac fork_exit(1130fb0,204fef0,1aa96ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa96cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa96cd4 --- trap 0, eip = 0, esp = 0x1aa96d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100084 td 0x1b067380 sched_switch(1b067380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1abc0b98 mi_switch(104,0,17e33fd,22b,1b067380,...) at mi_switch+0x1df/frame 0x1abc0bd8 sleepq_switch(1b067380,0,17e33fd,2b3,1acdb900,...) at sleepq_switch+0x110/frame 0x1abc0c0c sleepq_wait(1acdb900,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1abc0c30 msleep_spin_sbt(1acdb900,1acdb91c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abc0c74 taskqueue_thread_loop(206af4c,1abc0ce8,179f662,407,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abc0cac fork_exit(1130fb0,206af4c,1abc0ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc0cd4 --- trap 0, eip = 0, esp = 0x1abc0d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100091 td 0xbff8000 sched_switch(bff8000,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1abe4b98 mi_switch(104,0,17e33fd,22b,bff8000,...) at mi_switch+0x1df/frame 0x1abe4bd8 sleepq_switch(bff8000,0,17e33fd,2b3,bfe0300,...) at sleepq_switch+0x110/frame 0x1abe4c0c sleepq_wait(bfe0300,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1abe4c30 msleep_spin_sbt(bfe0300,bfe031c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abe4c74 taskqueue_thread_loop(206cb38,1abe4ce8,179f662,407,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abe4cac fork_exit(1130fb0,206cb38,1abe4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abe4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abe4cd4 --- trap 0, eip = 0, esp = 0x1abe4d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100092 td 0xbfcaa80 sched_switch(bfcaa80,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1abe9b98 mi_switch(104,0,17e33fd,22b,bfcaa80,...) at mi_switch+0x1df/frame 0x1abe9bd8 sleepq_switch(bfcaa80,0,17e33fd,2b3,bfe0300,...) at sleepq_switch+0x110/frame 0x1abe9c0c sleepq_wait(bfe0300,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1abe9c30 msleep_spin_sbt(bfe0300,bfe031c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abe9c74 taskqueue_thread_loop(206cb38,1abe9ce8,179f662,407,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abe9cac fork_exit(1130fb0,206cb38,1abe9ce8,0,0,...) at fork_exit+0x7e/frame 0x1abe9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abe9cd4 --- trap 0, eip = 0, esp = 0x1abe9d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100093 td 0xbfca700 sched_switch(bfca700,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1abeeb98 mi_switch(104,0,17e33fd,22b,bfca700,...) at mi_switch+0x1df/frame 0x1abeebd8 sleepq_switch(bfca700,0,17e33fd,2b3,bfe0300,...) at sleepq_switch+0x110/frame 0x1abeec0c sleepq_wait(bfe0300,0,171d774,1733677,0,...) at sleepq_wait+0x48/frame 0x1abeec30 msleep_spin_sbt(bfe0300,bfe031c,1733677,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abeec74 taskqueue_thread_loop(206cb38,1abeece8,179f662,407,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abeecac fork_exit(1130fb0,206cb38,1abeece8,0,0,...) at fork_exit+0x7e/frame 0x1abeecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abeecd4 --- trap 0, eip = 0, esp = 0x1abeed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100094 td 0xbfca380 sched_switch(bfca380,0,104,18a,16a60e0,...) at sched_switch+0x80c/frame 0x1abf3b88 mi_switch(104,0,17e33fd,22b,bfca380,...) at mi_switch+0x1df/frame 0x1abf3bc8 sleepq_switch(bfca380,0,17e33fd,2b3,bfca380,...) at sleepq_switch+0x110/frame 0x1abf3bfc sleepq_wait(9e5de00,0,1733677,0,0,...) at sleepq_wait+0x48/frame 0x1abf3c20 _sleep(9e5de00,9e5de1c,0,1733677,0,...) at _sleep+0x3a6/frame 0x1abf3c74 taskqueue_thread_loop(1de4568,1abf3ce8,179f662,407,0,...) at taskqueue_thread_loop+0x144/frame 0x1abf3cac fork_exit(1130fb0,1de4568,1abf3ce8,0,0,...) at fork_exit+0x7e/frame 0x1abf3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abf3cd4 --- trap 0, eip = 0, esp = 0x1abf3d20, ebp = 0 --- (null)() at 0 db:0:pho> dump Physical memory: 3427 MB Dumping 334 MB: 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/i386/i386/exception.s =================================================================== --- /usr/src/sys/i386/i386/exception.s (revision 334048) +++ /usr/src/sys/i386/i386/exception.s (working copy) @@ -310,6 +310,8 @@ IDTVEC(dbg) jz calltrap dbg_user: NMOVE_STACKS + movl $handle_ibrs_entry,%eax + call *%eax pushl %esp movl $trap,%eax call *%eax @@ -337,6 +339,8 @@ nmi_mchk_common: * Do not switch to the thread kernel stack, otherwise we might * obliterate the previous context partially copied from the * trampoline stack. + * Do not re-enable IBRS, there is no good place to store + * previous state if we come from the kernel. */ movl %cr3, %eax movl %eax, TF_ERR(%esp) @@ -364,6 +368,8 @@ IDTVEC(int0x80_syscall) SET_KERNEL_SREGS cld MOVE_STACKS + movl $handle_ibrs_entry,%eax + call *%eax sti FAKE_MCOUNT(TF_EIP(%esp)) pushl %esp @@ -509,7 +515,9 @@ doreti_exit: jmp 2f 1: testl $SEL_RPL_MASK, TF_CS(%esp) jz doreti_popl_fs -2: movl %esp, %esi +2: movl $handle_ibrs_exit,%eax + call *%eax + movl %esp, %esi movl PCPU(TRAMPSTK), %edx subl %ecx, %edx movl %edx, %edi Index: /usr/src/sys/i386/i386/genassym.c =================================================================== --- /usr/src/sys/i386/i386/genassym.c (revision 334048) +++ /usr/src/sys/i386/i386/genassym.c (working copy) @@ -221,6 +221,7 @@ ASSYM(PC_PRIVATE_TSS, offsetof(struct pcpu, pc_pri ASSYM(PC_KESP0, offsetof(struct pcpu, pc_kesp0)); ASSYM(PC_TRAMPSTK, offsetof(struct pcpu, pc_trampstk)); ASSYM(PC_COPYOUT_BUF, offsetof(struct pcpu, pc_copyout_buf)); +ASSYM(PC_IBPB_SET, offsetof(struct pcpu, pc_ibpb_set)); #ifdef DEV_APIC ASSYM(LA_EOI, LAPIC_EOI * LAPIC_MEM_MUL); Index: /usr/src/sys/i386/i386/support.s =================================================================== --- /usr/src/sys/i386/i386/support.s (revision 334048) +++ /usr/src/sys/i386/i386/support.s (working copy) @@ -433,9 +433,31 @@ msr_onfault: ret ENTRY(handle_ibrs_entry) - ret + cmpb $0,hw_ibrs_active + je 1f + movl $MSR_IA32_SPEC_CTRL,%ecx + rdmsr + orl $(IA32_SPEC_CTRL_IBRS|IA32_SPEC_CTRL_STIBP),%eax + orl $(IA32_SPEC_CTRL_IBRS|IA32_SPEC_CTRL_STIBP)>>32,%edx + wrmsr + movb $1,PCPU(IBPB_SET) + /* + * i386 does not implement SMEP, but the 4/4 split makes this not + * that important. + */ +1: ret END(handle_ibrs_entry) ENTRY(handle_ibrs_exit) - ret + cmpb $0,PCPU(IBPB_SET) + je 1f + pushl %ecx + movl $MSR_IA32_SPEC_CTRL,%ecx + rdmsr + andl $~(IA32_SPEC_CTRL_IBRS|IA32_SPEC_CTRL_STIBP),%eax + andl $~((IA32_SPEC_CTRL_IBRS|IA32_SPEC_CTRL_STIBP)>>32),%edx + wrmsr + popl %ecx + movb $0,PCPU(IBPB_SET) +1: ret END(handle_ibrs_exit) Index: /usr/src/sys/i386/include/asmacros.h =================================================================== --- /usr/src/sys/i386/include/asmacros.h (revision 334048) +++ /usr/src/sys/i386/include/asmacros.h (working copy) @@ -218,10 +218,14 @@ testl $PCB_VM86CALL, PCB_FLAGS(%eax) jnz .L\@.3 NMOVE_STACKS + movl $handle_ibrs_entry,%edx + call *%edx jmp .L\@.3 .L\@.1: testb $SEL_RPL_MASK, TF_CS(%esp) jz .L\@.3 .L\@.2: MOVE_STACKS + movl $handle_ibrs_entry,%edx + call *%edx .L\@.3: .endm -- Test scenario: misc/pmc.sh