GENERIC HEAD from 2018-08-21 03:45:09 UTC, r338128M, vmcore.1103 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-ALPHA2 #0 r338128M: Tue Aug 21 07:15:06 CEST 2018 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3526610944 (3363 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. eval: limits: not found /etc/rc: WARNING: failed to start ddb warning: total configured swap (4194304 pages) exceeds maximum recommended amount (1724580 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 625902 free (2294 frags, 77951 blocks, 0.2% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 809982 free (11118 frags, 99858 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 34228391 free (43655 frags, 4273092 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1952761 free (369857 frags, 197863 blocks, 3.6% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2513666 free (136930 frags, 297092 blocks, 0.5% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc49 /usr/local/lib/libav /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . uhub5: 10 ports with 10 removable, self powered lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0xbc7e8f4 bufwait (bufwait) @ kern/vfs_bio.c:3916 2nd 0x1ba69000 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10fb3a4 at _sx_xlock+0x64 #3 0x14499e9 at ufsdirhash_move+0x59 #4 0x144dd63 at ufs_direnter+0x353 #5 0x1457096 at ufs_mkdir+0x846 #6 0x1737656 at VOP_MKDIR_APV+0xf6 #7 0x11dfe0b at kern_mkdirat+0x1eb #8 0x11dfc11 at sys_mkdir+0x31 #9 0x1701de4 at syscall+0x364 #10 0xffc033a7 at PTDpde+0x43af Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Aug 21 07:33:02 x4 smartd[621]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Aug 21 07:33:02 x4 smartd[621]: see the following Seagate web pages: Aug 21 07:33:02 x4 smartd[621]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Aug 21 07:33:02 x4 smartd[621]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Tue Aug 21 07:33 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Aug 21 07:44:48 x4 su[16581]: pho to root on /dev/pts/0 20180821 07:45:21 all (1/1): sort.sh lock order reversal: 1st 0x20ec900 cpslk (cpslk) @ i386/i386/copyout.c:128 2nd 0xbc192b4 bufwait (bufwait) @ vm/vm_pager.c:374 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x148cdec at getpbuf+0x22c #5 0x145a7f3 at swap_pager_getpages+0x313 #6 0x148c8e3 at vm_pager_get_pages+0x53 #7 0x146928a at vm_fault_hold+0x110a #8 0x146812e at vm_fault+0x5e #9 0x170171e at trap_pfault+0xfe #10 0x1700bee at trap+0x40e #11 0xffc0315d at PTDpde+0x4165 #12 0x16e5e81 at copyin_slow0+0x21 #13 0x16e5af2 at cp_slow0+0x152 #14 0x16e5e2a at copyin+0xba #15 0x11587b6 at uiomove_faultflag+0x146 #16 0x115865f at uiomove+0x1f #17 0x116871c at pipe_write+0x8cc 20180821 07:47:05 all: done 20180821 09:25:55 all (1/580): spin.sh lock order reversal: 1st 0x1f5c71a0 ufs (ufs) @ kern/vfs_mount.c:1343 2nd 0x21aa58bc devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1376 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c2c89 at lockmgr_lock_fast_path+0x189 #3 0x11bc9bb at vop_stdlock+0x3b #4 0x173845a at VOP_LOCK1_APV+0xda #5 0x11e3301 at _vn_lock+0x61 #6 0x143e1bd at ffs_flushfiles+0xcd #7 0x14205b7 at softdep_flushfiles+0x67 #8 0x1440f68 at ffs_unmount+0xe8 #9 0x11c7033 at dounmount+0x583 #10 0x11c69a5 at sys_unmount+0x315 #11 0x1701de4 at syscall+0x364 #12 0xffc033a7 at PTDpde+0x43af 20180821 09:26:08 all (2/580): msdos9.sh mount leak: 1/98 20180821 09:26:21 all (3/580): fork.sh 20180821 09:26:33 all (4/580): mount2.sh lock order reversal: 1st 0xbc192b4 bufwait (bufwait) @ vm/vm_pager.c:374 2nd 0x21aa5750 ufs (ufs) @ dev/md/md.c:991 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x14437d7 at ffs_lock+0xa7 #5 0x173845a at VOP_LOCK1_APV+0xda #6 0x11e3301 at _vn_lock+0x61 #7 0xc46555 at mdstart_vnode+0x3a5 #8 0xc445e0 at md_kthread+0x240 #9 0x10ad5de at fork_exit+0x7e #10 0xffc033ba at PTDpde+0x43c2 freework leak: 189/199. mount leak: 1/99 20180821 09:31:43 all (5/580): unix_socket_detach2.sh pts leak: 1/3 20180821 09:36:49 all (6/580): nfs15.sh 20180821 09:36:55 all (7/580): extattr_set_fd.sh 20180821 09:37:02 all (8/580): snap11.sh lock order reversal: 1st 0xbc18aa8 bufwait (bufwait) @ vm/vm_pager.c:374 2nd 0x253c261c snaplk (snaplk) @ dev/md/md.c:991 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x14437d7 at ffs_lock+0xa7 #5 0x173845a at VOP_LOCK1_APV+0xda #6 0x11e3301 at _vn_lock+0x61 #7 0xc46555 at mdstart_vnode+0x3a5 #8 0xc445e0 at md_kthread+0x240 #9 0x10ad5de at fork_exit+0x7e #10 0xffc033ba at PTDpde+0x43c2 lock order reversal: 1st 0x253c261c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:939 2nd 0x21e9d034 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1636 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x141cf03 at ffs_snapremove+0x123 #5 0x1415f32 at ffs_truncate+0x502 #6 0x144c3f0 at ufs_inactive+0x250 #7 0x17380a6 at VOP_INACTIVE_APV+0xf6 #8 0x11cf8ae at vinactive+0x11e #9 0x11cff75 at vputx+0x365 #10 0x11dbcbe at kern_unlinkat+0x2ce #11 0x11db9ea at sys_unlink+0x3a #12 0x1701de4 at syscall+0x364 #13 0xffc033a7 at PTDpde+0x43af mount leak: 1/100 20180821 09:39:12 all (9/580): dev2.sh 20180821 09:44:19 all (10/580): readdir.sh mount leak: 4/104 20180821 09:56:43 all (11/580): nullfs7.sh mount leak: 1/105 20180821 09:56:50 all (12/580): marcus5.sh mount leak: 1/106 20180821 10:12:28 all (13/580): symlink3.sh mount leak: 10/116 20180821 10:32:55 all (14/580): nullfs19.sh mount leak: 1/117 20180821 10:35:09 all (15/580): temp.sh 20180821 10:35:21 all (16/580): rwlock_ronly.sh 20180821 10:35:27 all (17/580): thr2.sh 20180821 10:56:24 all (18/580): suj2.sh mount leak: 1/118 20180821 10:56:29 all (19/580): tmpfs10.sh 20180821 10:57:24 all (20/580): split.sh 20180821 11:25:37 all (21/580): msdos7.sh lock order reversal: 1st 0x21e74034 msdosfs (msdosfs) @ kern/vfs_syscalls.c:3366 2nd 0x25cae5e4 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:854 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c2c89 at lockmgr_lock_fast_path+0x189 #3 0x11bc9bb at vop_stdlock+0x3b #4 0x173845a at VOP_LOCK1_APV+0xda #5 0x11e3301 at _vn_lock+0x61 #6 0xfa69a1 at msdosfs_fsync+0x51 #7 0x1736e56 at VOP_FSYNC_APV+0xf6 #8 0x11df521 at kern_fsync+0x1e1 #9 0x11df5f2 at sys_fsync+0x22 #10 0x1701de4 at syscall+0x364 #11 0xffc033a7 at PTDpde+0x43af mount leak: 1/119 20180821 11:25:49 all (22/580): lockf3.sh mount leak: 1/120 GEOM leak: 29/282 20180821 11:45:46 all (23/580): kevent9.sh 20180821 11:50:51 all (24/580): tmpfs16.sh 20180821 11:53:55 all (25/580): signal0.sh 20180821 12:01:17 all (26/580): sendfile.sh lock order reversal: 1st 0x2526516c so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x21572034 ufs (ufs) @ kern/kern_sendfile.c:697 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c2ddb at lockmgr_slock_hard+0x7b #3 0x10c3977 at __lockmgr_args+0x427 #4 0x14437d7 at ffs_lock+0xa7 #5 0x173845a at VOP_LOCK1_APV+0xda #6 0x11e3301 at _vn_lock+0x61 #7 0x10ee4c1 at vn_sendfile+0x881 #8 0x10f0591 at sendfile+0x1a1 #9 0x10f03e6 at sys_sendfile+0x16 #10 0x1701de4 at syscall+0x364 #11 0xffc033a7 at PTDpde+0x43af 20180821 12:01:22 all (27/580): dev3.sh pts leak: 1/4 20180821 12:24:26 all (28/580): newfs.sh lock order reversal: 1st 0x20ae011c ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:118 2nd 0x28788750 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1682 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x143e465 at ffs_vgetf+0x155 #5 0x143e307 at ffs_vget+0x37 #6 0x1452fa4 at ufs_root+0x54 #7 0x144a4c4 at ufs_extattr_autostart+0x84 #8 0x1440d40 at ffs_mount+0x2070 #9 0x11c5960 at vfs_domount+0x800 #10 0x11c4b45 at vfs_donmount+0x7e5 #11 0x11c433a at sys_nmount+0x7a #12 0x1701de4 at syscall+0x364 #13 0xffc033a7 at PTDpde+0x43af Aug 21 12:34:26 x4 kernel: pid 72138 (mkdir), uid 1004 inumber 4094 on /mnt: out of inodes Aug 21 12:34:28 x4 kernel: pid 72137 (mkdir), uid 1004 inumber 4091 on /mnt: out of inodes Aug 21 12:34:57 x4 kernel: pid 77428 (creat), uid 1004 inumber 8198 on /mnt: out of inodes Aug 21 12:35:12 x4 kernel: pid 80877 (mkdir), uid 1004 inumber 3845 on /mnt: out of inodes Aug 21 12:35:13 x4 kernel: pid 80896 (creat), uid 1004 inumber 2073 on /mnt: out of inodes Aug 21 12:38:09 x4 kernel: pid 13189 (creat), uid 1004 inumber 4107 on /mnt: out of inodes Aug 21 12:38:11 x4 kernel: pid 13199 (creat), uid 1004 inumber 12292 on /mnt: out of inodes mount leak: 80/200 20180821 12:39:02 all (29/580): select.sh 20180821 12:49:14 all (30/580): tar.sh freework leak: 34329/34528. 20180821 12:54:34 all (31/580): sendfile10.sh mount leak: 1/201 20180821 12:56:46 all (32/580): truncate7.sh mount leak: 1/202 20180821 12:56:55 all (33/580): nullfs14.sh mount leak: 1/203 20180821 12:56:59 all (34/580): buildworld.sh mount leak: 1/204 20180821 13:17:38 all (35/580): zfs2.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Expensive timeout(9) function: 0x1465420(0) 0.010443179 s Warning: memory type solaris leaked memory on destroy (13 allocations, 2880 bytes leaked). 20180821 13:30:11 all (36/580): pthread8.sh 20180821 13:33:22 all (37/580): execi386.sh 20180821 13:33:26 all (38/580): ffs_syncvnode.sh Aug 21 13:43:43 x4 kernel: pid 5071 (rw), uid 1004 inumber 5 on /mnt: filesystem full mount leak: 1/205 20180821 14:04:07 all (39/580): parallelmount.sh Aug 21 14:06:29 x4 mountd[539]: bad exports list line '/mnt': symbolic link in export path or statfs failed Aug 21 14:07:17 x4 syslogd: last message repeated 1 times Aug 21 14:09:01 x4 syslogd: last message repeated 1 times Aug 21 14:10:12 x4 mountd[539]: can't delete exports for /mnt: Device busy Aug 21 14:10:36 x4 mountd[539]: bad exports list line '/mnt': symbolic link in export path or statfs failed Aug 21 14:12:55 x4 syslogd: last message repeated 1 times mount leak: 8294/8499 GEOM leak: 1102/1384 20180821 14:14:16 all (40/580): mlockall6.sh 20180821 14:24:31 all (41/580): fullpath2.sh mount leak: 91/8590 20180821 14:30:18 all (42/580): quota10.sh lock order reversal: 1st 0x1b419148 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:730 2nd 0x26ffe000 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:736 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10cddfe at __mtx_lock_flags+0x8e #3 0x145179c at quotaoff_inchange+0x7c #4 0x14518af at quotaoff+0xbf #5 0x1453080 at ufs_quotactl+0xb0 #6 0x11d7bf4 at sys_quotactl+0x144 #7 0x1701de4 at syscall+0x364 #8 0xffc033a7 at PTDpde+0x43af lock order reversal: 1st 0x409f8478 syncer (syncer) @ kern/vfs_subr.c:2170 2nd 0x50fefa28 ufs (ufs) @ kern/vfs_subr.c:2590 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x14437d7 at ffs_lock+0xa7 #5 0x173845a at VOP_LOCK1_APV+0xda #6 0x11e3301 at _vn_lock+0x61 #7 0x11cf471 at vget+0xc1 #8 0x14523d6 at qsync+0x96 #9 0x1441c9a at ffs_sync+0x66a #10 0x11d727c at sync_fsync+0x11c #11 0x1736e56 at VOP_FSYNC_APV+0xf6 #12 0x11d4d26 at sched_sync+0x2b6 #13 0x10ad5de at fork_exit+0x7e #14 0xffc033ba at PTDpde+0x43c2 mount leak: 1/8591 20180821 14:35:26 all (43/580): zfs6.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (13 allocations, 2880 bytes leaked). 20180821 14:41:37 all (44/580): mmap14.sh 20180821 14:45:20 all (45/580): nullfs6.sh 20180821 14:45:26 all (46/580): linger4.sh 20180821 15:05:39 all (47/580): dd.sh newblk leak: 25742/25752. 20180821 15:09:36 all (48/580): datamove5.sh mount leak: 1/8592 20180821 15:10:06 all (49/580): suj5.sh mount leak: 1/8593 20180821 15:11:55 all (50/580): mlockall2.sh 20180821 15:23:07 all (51/580): mountu.sh mount leak: 1/8594 20180821 15:23:29 all (52/580): msdos4.sh mount leak: 1/8595 20180821 15:34:21 all (53/580): kevent11.sh lock order reversal: 1st 0x2adc3a90 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1239 2nd 0x2d445e6c ufs (ufs) @ kern/vfs_subr.c:5045 stack backtrace: #0 0x1160421 at witness_debugger+0x81 #1 0x11602e8 at witness_checkorder+0xcd8 #2 0x10c30fb at lockmgr_xlock_hard+0x5b #3 0x10c3942 at __lockmgr_args+0x3f2 #4 0x14437d7 at ffs_lock+0xa7 #5 0x173845a at VOP_LOCK1_APV+0xda #6 0x11e3301 at _vn_lock+0x61 #7 0x11d1e29 at vfs_knllock+0x29 #8 0x10a1cb9 at knlist_remove_kq+0x99 #9 0x10a1c0f at knlist_remove+0x1f #10 0x11d764f at filt_vfsdetach+0x2f #11 0x10a2803 at knote_fdclose+0x123 #12 0x1094443 at closefp+0x63 #13 0x1094e4c at kern_close+0x16c #14 0x1094cda at sys_close+0x1a #15 0x1701de4 at syscall+0x364 #16 0xffc033a7 at PTDpde+0x43af mount leak: 1/8596 20180821 15:37:36 all (54/580): dup2.sh 20180821 15:37:42 all (55/580): quota9.sh mount leak: 1/8597 20180821 15:37:53 all (56/580): oom.sh Aug 21 15:38:12 x4 kernel: pid 80610 (sort), uid 0, was killed: out of swap space Aug 21 15:38:15 x4 kernel: pid 80608 (sort), uid 0, was killed: out of swap space Aug 21 15:38:17 x4 kernel: pid 80609 (sort), uid 0, was killed: out of swap space warning: total configured swap (4194304 pages) exceeds maximum recommended amount (1724580 pages). warning: increase kern.maxswzone or reduce amount of swap. 20180821 15:38:38 all (57/580): pcatch.sh 20180821 15:38:42 all (58/580): fsgs.sh 20180821 15:39:50 all (59/580): snap2.sh mount leak: 1/8598 20180821 15:41:51 all (60/580): quota8.sh Aug 21 15:42:47 x4 kernel: pid 97874 (rw), uid 1004 inumber 100099 on /mnt: filesystem full mount leak: 1/8599 20180821 15:44:02 all (61/580): pathconf2.sh 20180821 15:44:06 all (62/580): extattr.sh mount leak: 2/8601 20180821 15:44:19 all (63/580): sigstop2.sh 20180821 15:45:24 all (64/580): msdos10.sh mount leak: 1/8602 20180821 15:51:06 all (65/580): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff FreeBSD/i386 (x4.osted.lan) (ttyu0) login: root Password: Aug 21 19:47:09 x4 login[764]: ROOT LOGIN (root) ON ttyu0 Last login: Fri Aug 3 13:09:05 on ttyu0 FreeBSD 12.0-ALPHA2 (PHO) #0 r338128M: Tue Aug 21 07:15:06 CEST 2018 You have new mail. root@x4:~ # pgrep pmcstat | xargs ps -lp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 92970 91032 0 20 0 10248 348 - R+ 0 7:25.69 pmcstat -P instructions -O /tmp/sample.out.1 find -x /var -name not.there 0 92971 91032 0 20 0 10248 352 pmc-sx D+ 0 0:00.02 pmcstat -P instructions -O /tmp/sample.out.2 find -x /var -name not.there root@x4:~ # pgrep pmcstat | xargs ps -lp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 92970 91032 0 20 0 10248 348 - R+ 0 7:25.88 pmcstat -P instructions -O /tmp/sample.out.1 find -x /var -name not.there 0 92971 91032 0 20 0 10248 352 pmc-sx D+ 0 0:00.02 pmcstat -P instructions -O /tmp/sample.out.2 find -x /var -name not.there root@x4:~ # kill -9 92971 root@x4:~ # pgrep pmcstat | xargs ps -lp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 92970 91032 0 20 0 10248 348 - R+ 0 7:26.40 pmcstat -P instructions -O /tmp/sample.out.1 find -x /var -name not.there 0 92971 91032 0 20 0 10248 352 pmc-sx D+ 0 0:00.02 pmcstat -P instructions -O /tmp/sample.out.2 find -x /var -name not.there root@x4:~ # ~KDB: enter: Break to debugger [ thread pid 11 tid 100006 ] Stopped at kdb_alt_break_internal+0xf0: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 11 tid 100006 td 0xbfe8a80 kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x5fa740 curthread = 0xbfe9700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x1511ad40 fpcurthread = none idlethread = 0xbfe9700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x1511ad30 common_tssp = 0xffc01260 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xa04a740 curthread = 0xbfe9380: pid 11 tid 100004 "idle: cpu1" curpcb = 0x1511fd40 fpcurthread = none idlethread = 0xbfe9380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x1511fd30 common_tssp = 0xffc012c8 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xa04f740 curthread = 0xbfe9000: pid 11 tid 100005 "idle: cpu2" curpcb = 0x15124d40 fpcurthread = none idlethread = 0xbfe9000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x15124d30 common_tssp = 0xffc01330 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xa054740 curthread = 0xbfe8a80: pid 11 tid 100006 "idle: cpu3" curpcb = 0x15129d40 fpcurthread = none idlethread = 0xbfe8a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x15129d30 common_tssp = 0xffc01398 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 92970 (pmcstat) thread 0x255c0000 (100687) exclusive sx pmc-sx (pmc-sx) r = 0 (0x20e5c74) locked @ /usr/src/sys/dev/hwpmc/hwpmc_mod.c:3398 db:1:pho1> show lockedvnods Locked vnodes db:1:pho1> show allchains chain 1: thread 103302 (pid 27714, sleep) sleeping on 0x202fa9c "nanslp" chain 2: thread 103323 (pid 26244, sleep) sleeping on 0x202fa9c "nanslp" chain 3: thread 100349 (pid 18899, csh) sleeping on 0x1af00c74 "ttyin" chain 4: thread 103292 (pid 92971, pmcstat) blocked on sx "pmc-sx" XLOCK thread 100687 (pid 92970, pmcstat) sleeping on 0x202e997 "pmcctx" chain 5: thread 100687 (pid 92970, pmcstat) sleeping on 0x202e997 "pmcctx" chain 6: thread 103189 (pid 91032, sh) sleeping on 0x26667a60 "pause" chain 7: thread 100985 (pid 91031, tee) sleeping on 0x2bf89b98 "piperd" chain 8: thread 103175 (pid 91030, sh) sleeping on 0x24a57354 "wait" chain 9: thread 100348 (pid 61932, bash) sleeping on 0x1b9ee824 "select" chain 10: thread 100577 (pid 61851, sshd) sleeping on 0x1f583764 "select" chain 11: thread 100377 (pid 61379, sshd) sleeping on 0x1b9f0d24 "select" chain 12: thread 100251 (pid 96183, sh) sleeping on 0x221089fc "wait" chain 13: thread 100226 (pid 16819, bash) sleeping on 0x220dd354 "wait" chain 14: thread 100237 (pid 16581, csh) sleeping on 0x220a33b8 "pause" chain 15: thread 100231 (pid 16404, su) sleeping on 0x22051354 "wait" chain 16: thread 100120 (pid 8138, sh) sleeping on 0x1badd4f8 "piperd" chain 17: thread 100126 (pid 8136, awk) sleeping on 0x220956a0 "piperd" chain 18: thread 100215 (pid 8135, sh) sleeping on 0x1faf3354 "wait" chain 19: thread 100206 (pid 8131, sh) sleeping on 0x220529fc "wait" chain 20: thread 100216 (pid 8129, tail) sleeping on 0x1b72bc00 "kqread" chain 21: thread 100207 (pid 8127, top) sleeping on 0x1b3934e4 "select" chain 22: thread 100222 (pid 8126, sshd) sleeping on 0x1b392f64 "select" chain 23: thread 100131 (pid 8125, sshd) sleeping on 0x1b393024 "select" chain 24: thread 100219 (pid 8124, sshd) sleeping on 0x1b392ca4 "select" chain 25: thread 100214 (pid 8122, sshd) sleeping on 0x1b392c24 "select" chain 26: thread 100220 (pid 8119, sshd) sleeping on 0x1b392ce4 "select" chain 27: thread 100121 (pid 8118, sshd) sleeping on 0x1b6f96e4 "select" chain 28: thread 100128 (pid 6660, bash) sleeping on 0x1af56354 "wait" chain 29: thread 100119 (pid 6659, sshd) sleeping on 0x1b6f9364 "select" chain 30: thread 100123 (pid 6657, sshd) sleeping on 0x1b392f24 "select" chain 31: thread 100197 (pid 870, sh) sleeping on 0x1ba946a0 "piperd" chain 32: thread 100196 (pid 869, awk) sleeping on 0x1bae59f0 "piperd" chain 33: thread 100195 (pid 868, sh) sleeping on 0x1bb089fc "wait" chain 34: thread 100186 (pid 860, sh) sleeping on 0x1bb0b9fc "wait" chain 35: thread 100192 (pid 764, login) sleeping on 0x1faf39fc "wait" chain 36: thread 100191 (pid 763, getty) sleeping on 0x1b77d474 "ttyin" chain 37: thread 100190 (pid 762, getty) sleeping on 0x1b77d674 "ttyin" chain 38: thread 100189 (pid 761, getty) sleeping on 0x1b77d874 "ttyin" chain 39: thread 100109 (pid 760, getty) sleeping on 0x1b77da74 "ttyin" chain 40: thread 100179 (pid 759, getty) sleeping on 0x1b77dc74 "ttyin" chain 41: thread 100187 (pid 758, getty) sleeping on 0x1b77de74 "ttyin" chain 42: thread 100188 (pid 757, getty) sleeping on 0x1b780074 "ttyin" chain 43: thread 100104 (pid 756, getty) sleeping on 0x1b780274 "ttyin" chain 44: thread 100129 (pid 739, inetd) sleeping on 0xbf79ee4 "select" chain 45: thread 100124 (pid 705, nfscbd: master) sleeping on 0x1b3a2b50 "rpcsvc" chain 46: thread 100183 (pid 705, nfscbd: service) sleeping on 0x1b3a2ad0 "rpcsvc" chain 47: thread 100184 (pid 705, nfscbd: service) sleeping on 0x1b3a2a50 "rpcsvc" chain 48: thread 100185 (pid 705, nfscbd: service) sleeping on 0x1b72ecd0 "rpcsvc" chain 49: thread 100125 (pid 704, nfscbd) sleeping on 0x1faa08e8 "accept" chain 50: thread 100181 (pid 671, cron) sleeping on 0x202fa9e "nanslp" chain 51: thread 100130 (pid 667, sendmail) sleeping on 0x1bae8a60 "pause" chain 52: thread 100135 (pid 664, sendmail) sleeping on 0x1b6f89e4 "select" chain 53: thread 100171 (pid 661, sshd) sleeping on 0x1b3931a4 "select" chain 54: thread 100172 (pid 623, smartd) sleeping on 0x202fa9c "nanslp" chain 55: thread 100174 (pid 593, powerd) sleeping on 0x1b6f95e4 "select" chain 56: thread 100173 (pid 590, ntpd) sleeping on 0x1ac686e4 "select" chain 57: thread 100170 (pid 552, NLM: master) sleeping on 0x1acadb50 "rpcsvc" chain 58: thread 100169 (pid 549, rpc.statd) sleeping on 0x1b393324 "select" chain 59: thread 100137 (pid 548, nfsd: master) sleeping on 0xbfdcc50 "rpcsvc" chain 60: thread 100138 (pid 548, nfsd: service) sleeping on 0x1acae550 "rpcsvc" chain 61: thread 100139 (pid 548, nfsd: service) sleeping on 0x1b72f3d0 "rpcsvc" chain 62: thread 100140 (pid 548, nfsd: service) sleeping on 0x1acae4d0 "rpcsvc" chain 63: thread 100141 (pid 548, nfsd: service) sleeping on 0x1acae450 "rpcsvc" chain 64: thread 100142 (pid 548, nfsd: service) sleeping on 0x1b72f350 "rpcsvc" chain 65: thread 100143 (pid 548, nfsd: service) sleeping on 0x1acae3d0 "rpcsvc" chain 66: thread 100144 (pid 548, nfsd: service) sleeping on 0x1b72f2d0 "rpcsvc" chain 67: thread 100145 (pid 548, nfsd: service) sleeping on 0x1b72f250 "rpcsvc" chain 68: thread 100146 (pid 548, nfsd: service) sleeping on 0x1b72f1d0 "rpcsvc" chain 69: thread 100147 (pid 548, nfsd: service) sleeping on 0x1acae350 "rpcsvc" chain 70: thread 100148 (pid 548, nfsd: service) sleeping on 0x1acae2d0 "rpcsvc" chain 71: thread 100149 (pid 548, nfsd: service) sleeping on 0x1acae250 "rpcsvc" chain 72: thread 100150 (pid 548, nfsd: service) sleeping on 0x1acae1d0 "rpcsvc" chain 73: thread 100151 (pid 548, nfsd: service) sleeping on 0x1acae150 "rpcsvc" chain 74: thread 100152 (pid 548, nfsd: service) sleeping on 0x1acae0d0 "rpcsvc" chain 75: thread 100153 (pid 548, nfsd: service) sleeping on 0x1b72f150 "rpcsvc" chain 76: thread 100154 (pid 548, nfsd: service) sleeping on 0x1acae050 "rpcsvc" chain 77: thread 100155 (pid 548, nfsd: service) sleeping on 0x1b72f0d0 "rpcsvc" chain 78: thread 100156 (pid 548, nfsd: service) sleeping on 0x1acadf50 "rpcsvc" chain 79: thread 100157 (pid 548, nfsd: service) sleeping on 0x1b72f050 "rpcsvc" chain 80: thread 100158 (pid 548, nfsd: service) sleeping on 0x1b72ef50 "rpcsvc" chain 81: thread 100159 (pid 548, nfsd: service) sleeping on 0x1b72eed0 "rpcsvc" chain 82: thread 100160 (pid 548, nfsd: service) sleeping on 0x1acaded0 "rpcsvc" chain 83: thread 100161 (pid 548, nfsd: service) sleeping on 0x1b72ee50 "rpcsvc" chain 84: thread 100162 (pid 548, nfsd: service) sleeping on 0x1acade50 "rpcsvc" chain 85: thread 100163 (pid 548, nfsd: service) sleeping on 0x1acaddd0 "rpcsvc" chain 86: thread 100164 (pid 548, nfsd: service) sleeping on 0x1acadd50 "rpcsvc" chain 87: thread 100165 (pid 548, nfsd: service) sleeping on 0x1acadcd0 "rpcsvc" chain 88: thread 100166 (pid 548, nfsd: service) sleeping on 0x1acadc50 "rpcsvc" chain 89: thread 100167 (pid 548, nfsd: service) sleeping on 0x1acadbd0 "rpcsvc" chain 90: thread 100168 (pid 548, nfsd: service) sleeping on 0x1b72edd0 "rpcsvc" chain 91: thread 100132 (pid 545, nfsd) sleeping on 0x1b393264 "select" chain 92: thread 100134 (pid 539, mountd) sleeping on 0x1b6f9524 "select" chain 93: thread 100133 (pid 532, nfsuserd) sleeping on 0x1b6f9664 "select" chain 94: thread 100114 (pid 531, nfsuserd) sleeping on 0x1af5ca60 "pause" chain 95: thread 100106 (pid 494, watchdogd) sleeping on 0x202fa9e "nanslp" chain 96: thread 100107 (pid 410, rpcbind) sleeping on 0x1b393364 "select" chain 97: thread 100127 (pid 401, syslogd) sleeping on 0x1ac68564 "select" chain 98: thread 100113 (pid 325, devd) sleeping on 0x1ac685a4 "select" chain 99: thread 100100 (pid 24, syncer) sleeping on 0x20aa220 "syncer" chain 100: thread 100099 (pid 23, vnlru) sleeping on 0x1b38a9fc "vlruwt" chain 101: thread 100098 (pid 22, bufdaemon) sleeping on 0x20a9bac "qsleep" chain 102: thread 100103 (pid 22, bufspacedaemon-0) sleeping on 0x1de8380 "-" chain 103: thread 100115 (pid 22, /home worker) sleeping on 0x1bafc684 "sdflush" chain 104: thread 100116 (pid 22, /tmp worker) sleeping on 0x1bafc484 "sdflush" chain 105: thread 100117 (pid 22, /usr worker) sleeping on 0x1bafc284 "sdflush" chain 106: thread 100118 (pid 22, /var worker) sleeping on 0x1bafc084 "sdflush" chain 107: thread 100097 (pid 21, vmdaemon) sleeping on 0x20bf738 "psleep" chain 108: thread 100096 (pid 20, dom0) sleeping on 0x20e9ddc "psleep" chain 109: thread 100101 (pid 20, laundry: dom0) sleeping on 0x20e9de8 "launds" chain 110: thread 100102 (pid 20, uma) sleeping on 0x14642a0 "umarcl" chain 111: thread 100090 (pid 19, rand_harvestq) sleeping on 0x1f05894 "-" chain 112: thread 100089 (pid 18, sctp_iterator) sleeping on 0x20e8ec4 "waiting_for_work" chain 113: thread 100088 (pid 17, soaiod4) sleeping on 0x20a97d8 "-" chain 114: thread 100087 (pid 16, soaiod3) sleeping on 0x20a97d8 "-" chain 115: thread 100086 (pid 15, soaiod2) sleeping on 0x20a97d8 "-" chain 116: thread 100085 (pid 9, soaiod1) sleeping on 0x20a97d8 "-" chain 117: thread 100082 (pid 8, fdc0) sleeping on 0x1ac5d23c "-" chain 118: thread 100047 (pid 14, usbus0) sleeping on 0x1ac8dacc "-" chain 119: thread 100048 (pid 14, usbus0) sleeping on 0x1ac8dafc "-" chain 120: thread 100049 (pid 14, usbus0) sleeping on 0x1ac8db2c "-" chain 121: thread 100050 (pid 14, usbus0) sleeping on 0x1ac8db5c "-" chain 122: thread 100051 (pid 14, usbus0) sleeping on 0x1ac8db8c "-" chain 123: thread 100053 (pid 14, usbus1) sleeping on 0x1af63acc "-" chain 124: thread 100054 (pid 14, usbus1) sleeping on 0x1af63afc "-" chain 125: thread 100055 (pid 14, usbus1) sleeping on 0x1af63b2c "-" chain 126: thread 100056 (pid 14, usbus1) sleeping on 0x1af63b5c "-" chain 127: thread 100057 (pid 14, usbus1) sleeping on 0x1af63b8c "-" chain 128: thread 100059 (pid 14, usbus2) sleeping on 0x1af6eacc "-" chain 129: thread 100060 (pid 14, usbus2) sleeping on 0x1af6eafc "-" chain 130: thread 100061 (pid 14, usbus2) sleeping on 0x1af6eb2c "-" chain 131: thread 100062 (pid 14, usbus2) sleeping on 0x1af6eb5c "-" chain 132: thread 100063 (pid 14, usbus2) sleeping on 0x1af6eb8c "-" chain 133: thread 100064 (pid 14, usbus3) sleeping on 0x1af81acc "-" chain 134: thread 100065 (pid 14, usbus3) sleeping on 0x1af81afc "-" chain 135: thread 100066 (pid 14, usbus3) sleeping on 0x1af81b2c "-" chain 136: thread 100067 (pid 14, usbus3) sleeping on 0x1af81b5c "-" chain 137: thread 100068 (pid 14, usbus3) sleeping on 0x1af81b8c "-" chain 138: thread 100069 (pid 14, usbus4) sleeping on 0x1af88acc "-" chain 139: thread 100070 (pid 14, usbus4) sleeping on 0x1af88afc "-" chain 140: thread 100071 (pid 14, usbus4) sleeping on 0x1af88b2c "-" chain 141: thread 100072 (pid 14, usbus4) sleeping on 0x1af88b5c "-" chain 142: thread 100073 (pid 14, usbus4) sleeping on 0x1af88b8c "-" chain 143: thread 100075 (pid 14, usbus5) sleeping on 0x1ac78ca4 "-" chain 144: thread 100076 (pid 14, usbus5) sleeping on 0x1ac78cd4 "-" chain 145: thread 100077 (pid 14, usbus5) sleeping on 0x1ac78d04 "-" chain 146: thread 100078 (pid 14, usbus5) sleeping on 0x1ac78d34 "-" chain 147: thread 100079 (pid 14, usbus5) sleeping on 0x1ac78d64 "-" chain 148: thread 100042 (pid 7, doneq0) sleeping on 0x1e52c40 "-" chain 149: thread 100095 (pid 7, scanner) sleeping on 0x1e52b6c "-" chain 150: thread 100041 (pid 6, crypto returns 3) sleeping on 0x9f500c4 "crypto_ret_wait" chain 151: thread 100040 (pid 5, crypto returns 2) sleeping on 0x9f5008c "crypto_ret_wait" chain 152: thread 100039 (pid 4, crypto returns 1) sleeping on 0x9f50054 "crypto_ret_wait" chain 153: thread 100038 (pid 3, crypto returns 0) sleeping on 0x9f5001c "crypto_ret_wait" chain 154: thread 100037 (pid 2, crypto) sleeping on 0x20be558 "crypto_wait" chain 155: thread 100029 (pid 13, g_event) sleeping on 0x20e5270 "-" chain 156: thread 100030 (pid 13, g_up) sleeping on 0x20e5278 "-" chain 157: thread 100031 (pid 13, g_down) sleeping on 0x20e526c "-" chain 158: thread 100002 (pid 1, init) sleeping on 0xbfd99fc "wait" chain 159: thread 100001 (pid 10, audit) sleeping on 0x20e969c "audit_worker_cv" chain 160: thread 100000 (pid 0, swapper) sleeping on 0x20e5290 "swapin" chain 161: thread 100008 (pid 0, thread taskq) sleeping on 0x9e56f00 "-" chain 162: thread 100010 (pid 0, config_0) sleeping on 0xbffda00 "-" chain 163: thread 100011 (pid 0, kqueue_ctx taskq) sleeping on 0x9e56e00 "-" chain 164: thread 100012 (pid 0, aiod_kick taskq) sleeping on 0x9e56d80 "-" chain 165: thread 100014 (pid 0, softirq_0) sleeping on 0xbffd980 "-" chain 166: thread 100015 (pid 0, softirq_1) sleeping on 0xbffd900 "-" chain 167: thread 100016 (pid 0, softirq_2) sleeping on 0xbffd880 "-" chain 168: thread 100017 (pid 0, softirq_3) sleeping on 0xbffd800 "-" chain 169: thread 100018 (pid 0, if_io_tqg_0) sleeping on 0xbffd780 "-" chain 170: thread 100019 (pid 0, if_io_tqg_1) sleeping on 0xbffd700 "-" chain 171: thread 100020 (pid 0, if_io_tqg_2) sleeping on 0xbffd680 "-" chain 172: thread 100021 (pid 0, if_io_tqg_3) sleeping on 0xbffd600 "-" chain 173: thread 100022 (pid 0, if_config_tqg_0) sleeping on 0xbffd580 "-" chain 174: thread 100032 (pid 0, firmware taskq) sleeping on 0x9e56c80 "-" chain 175: thread 100033 (pid 0, crypto_0) sleeping on 0x9e56c00 "-" chain 176: thread 100034 (pid 0, crypto_1) sleeping on 0x9e56c00 "-" chain 177: thread 100035 (pid 0, crypto_2) sleeping on 0x9e56c00 "-" chain 178: thread 100036 (pid 0, crypto_3) sleeping on 0x9e56c00 "-" chain 179: thread 100084 (pid 0, mca taskq) sleeping on 0x1afcc000 "-" chain 180: thread 100091 (pid 0, acpi_task_0) sleeping on 0x9e56a80 "-" chain 181: thread 100092 (pid 0, acpi_task_1) sleeping on 0x9e56a80 "-" chain 182: thread 100093 (pid 0, acpi_task_2) sleeping on 0x9e56a80 "-" chain 183: thread 100094 (pid 0, CAM taskq) sleeping on 0x9e56b80 "-" db:1:pho1> show mount 0x1b9fd000 /dev/ada0s1a on / (ufs) 0x1b91c000 devfs on /dev (devfs) 0x1b9fc000 /dev/ada0s1f on /home (ufs) 0x1b9fb000 /dev/label/tmp on /tmp (ufs) 0x1bbf7000 /dev/ada0s1d on /usr (ufs) 0x1bbf6000 /dev/ada0s1e on /var (ufs) 0x1bbf5000 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 0 Buf domain 0 freebufs 466 lofreebufs 311 hifreebufs 466 bufspace 40554496 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 17 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5227 (39426048) dirtyq count 17 (278528) wakeup 0 lim 28 CPU 27, 8, 19, 15, Locked buffers: 0 space 0 Total buffers: 5779 space 40554496 db:1:pho1> show page vm_cnt.v_free_count: 787872 vm_cnt.v_inactive_count: 1003 vm_cnt.v_active_count: 1293 vm_cnt.v_laundry_count: 47 vm_cnt.v_wire_count: 70996 vm_cnt.v_free_reserved: 1188 vm_cnt.v_free_min: 5498 vm_cnt.v_free_target: 18428 vm_cnt.v_inactive_target: 27642 db:1:pho1> show pageq pq_free 787872 dom 0 page_cnt 862226 free 787872 pq_act 1293 pq_inact 1003 pq_laund 47 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000000 08 (001024K) | 000016 07 (000512K) | 000356 06 (000256K) | 002127 05 (000128K) | 006309 04 (000064K) | 009948 03 (000032K) | 014433 02 (000016K) | 017692 01 (000008K) | 017138 00 (000004K) | 018742 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 10 (004096K) | 000000 09 (002048K) | 000001 08 (001024K) | 000000 07 (000512K) | 000002 06 (000256K) | 000006 05 (000128K) | 000001 04 (000064K) | 000002 03 (000032K) | 000029 02 (000016K) | 000036 01 (000008K) | 000051 00 (000004K) | 000080 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0xbfe9700 (CPU 0) cpustop_handler(0,9772830,1511ab94,17008b4,20d960c,...) at cpustop_handler+0x25/frame 0x1511aa80 ipi_nmi_handler(20d960c,4a2bb,0,0,0,...) at ipi_nmi_handler+0x36/frame 0x1511aa90 trap(1511aba0,8,28,28,9fe1e00,...) at trap+0xd4/frame 0x1511ab94 calltrap() at PTDpde+0x4165/frame 0x1511ab94 --- trap 0x13, eip = 0x17191c5, esp = 0x1511abe0, ebp = 0x1511abe0 --- acpi_cpu_c1(1511ac34,17093b2,2428d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x1511abe0 acpi_cpu_idle(aa43247,0,aa43247,1511ac50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x1511ac20 cpu_idle_acpi(aa43247,0,203ccc0,12,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511ac34 cpu_idle(0,1511ac80,17f6cfa,adb,12,...) at cpu_idle+0xab/frame 0x1511ac50 sched_idletd(0,1511ace8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x1511acac fork_exit(1123830,0,1511ace8,0,0,...) at fork_exit+0x7e/frame 0x1511acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511acd4 --- trap 0, eip = 0, esp = 0x1511ad20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfe9380 (CPU 1) cpustop_handler(1,9772830,1511fb94,17008b4,1511fac8,...) at cpustop_handler+0x25/frame 0x1511fa80 ipi_nmi_handler(1511fac8,10ce8c4,20e67c0,8,1742ab1,...) at ipi_nmi_handler+0x36/frame 0x1511fa90 trap(1511fba0,8,28,28,9fe1c00,...) at trap+0xd4/frame 0x1511fb94 calltrap() at PTDpde+0x4165/frame 0x1511fb94 --- trap 0x13, eip = 0x17191c5, esp = 0x1511fbe0, ebp = 0x1511fbe0 --- acpi_cpu_c1(1511fc34,17093b2,be78d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x1511fbe0 acpi_cpu_idle(462ed30,0,462ed30,1511fc50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x1511fc20 cpu_idle_acpi(462ed30,0,203d380,14,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511fc34 cpu_idle(0,1511fc80,17f6cfa,395,14,...) at cpu_idle+0xab/frame 0x1511fc50 sched_idletd(0,1511fce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(1123830,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfe9000 (CPU 2) cpustop_handler(2,9772830,15124b94,17008b4,15124ac8,...) at cpustop_handler+0x25/frame 0x15124a80 ipi_nmi_handler(15124ac8,10ce8c4,20e68c0,8,1742ab1,...) at ipi_nmi_handler+0x36/frame 0x15124a90 trap(15124ba0,8,28,28,9fe1a00,...) at trap+0xd4/frame 0x15124b94 calltrap() at PTDpde+0x4165/frame 0x15124b94 --- trap 0x13, eip = 0x17191c5, esp = 0x15124be0, ebp = 0x15124be0 --- acpi_cpu_c1(15124c34,17093b2,be7dd90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15124be0 acpi_cpu_idle(47f7c07f,0,47f7c07f,15124c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15124c20 cpu_idle_acpi(47f7c07f,0,203da40,11,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15124c34 cpu_idle(0,15124c80,17f6cfa,adb,11,...) at cpu_idle+0xab/frame 0x15124c50 sched_idletd(0,15124ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15124cac fork_exit(1123830,0,15124ce8,0,0,...) at fork_exit+0x7e/frame 0x15124cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15124cd4 --- trap 0, eip = 0, esp = 0x15124d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfe8a80 (CPU 3) kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 27714 868 860 1001 S nanslp 0x202fa9c sleep 26244 8135 8131 1001 S nanslp 0x202fa9c sleep 18899 764 18899 0 S+ ttyin 0x1af00c74 csh 92971 91032 96183 0 D+ pmc-sx 0x20e5c74 pmcstat 92970 91032 96183 0 D+ pmcctx 0x202e997 pmcstat 91032 91030 96183 0 S+ pause 0x26667a60 sh 91031 96183 96183 0 S+ piperd 0x2bf89b98 tee 91030 96183 96183 0 SW+ wait 0x24a57354 sh 61932 61851 61932 1001 Ss+ select 0x1b9ee824 bash 61851 61379 61379 1001 S select 0x1f583764 sshd 61379 661 61379 0 Ss select 0x1b9f0d24 sshd 96183 16819 96183 0 SW+ wait 0x221089fc sh 16819 16581 16819 0 SW+ wait 0x220dd354 bash 16581 16404 16581 0 SW+ pause 0x220a33b8 csh 16404 6660 16404 1001 SW+ wait 0x22051354 su 8138 8131 8131 1001 S piperd 0x1badd4f8 sh 8136 8131 8131 1001 S piperd 0x220956a0 awk 8135 8131 8131 1001 S wait 0x1faf3354 sh 8131 8126 8131 1001 SWs wait 0x220529fc sh 8129 8125 8129 1001 Ss kqread 0x1b72bc00 tail 8127 8124 8127 1001 Ss+ select 0x1b3934e4 top 8126 8122 8122 1001 S select 0x1b392f64 sshd 8125 8119 8119 1001 S select 0x1b393024 sshd 8124 8118 8118 1001 S select 0x1b392ca4 sshd 8122 661 8122 0 Ss select 0x1b392c24 sshd 8119 661 8119 0 Ss select 0x1b392ce4 sshd 8118 661 8118 0 Ss select 0x1b6f96e4 sshd 6660 6659 6660 1001 SWs+ wait 0x1af56354 bash 6659 6657 6657 1001 S select 0x1b6f9364 sshd 6657 661 6657 0 Ss select 0x1b392f24 sshd 870 860 860 1001 S piperd 0x1ba946a0 sh 869 860 860 1001 S piperd 0x1bae59f0 awk 868 860 860 1001 S wait 0x1bb089fc sh 860 1 860 1001 SWs wait 0x1bb0b9fc sh 764 1 764 0 Ss+ wait 0x1faf39fc login 763 1 763 0 Ss+ ttyin 0x1b77d474 getty 762 1 762 0 Ss+ ttyin 0x1b77d674 getty 761 1 761 0 Ss+ ttyin 0x1b77d874 getty 760 1 760 0 Ss+ ttyin 0x1b77da74 getty 759 1 759 0 Ss+ ttyin 0x1b77dc74 getty 758 1 758 0 Ss+ ttyin 0x1b77de74 getty 757 1 757 0 Ss+ ttyin 0x1b780074 getty 756 1 756 0 Ss+ ttyin 0x1b780274 getty 739 1 739 0 Ss select 0xbf79ee4 inetd 705 704 704 0 S (threaded) nfscbd 100124 S rpcsvc 0x1b3a2b50 nfscbd: master 100183 S rpcsvc 0x1b3a2ad0 nfscbd: service 100184 S rpcsvc 0x1b3a2a50 nfscbd: service 100185 S rpcsvc 0x1b72ecd0 nfscbd: service 704 1 704 0 SWs accept 0x1faa08e8 nfscbd 671 1 671 0 Ss nanslp 0x202fa9e cron 667 1 667 25 Ss pause 0x1bae8a60 sendmail 664 1 664 0 Ss select 0x1b6f89e4 sendmail 661 1 661 0 Ss select 0x1b3931a4 sshd 623 1 622 0 S nanslp 0x202fa9c smartd 593 1 593 0 Ss select 0x1b6f95e4 powerd 590 1 590 0 Ss (threaded) ntpd 100173 S select 0x1ac686e4 ntpd 552 1 552 0 Ss rpcsvc 0x1acadb50 NLM: master 549 1 549 0 Ss select 0x1b393324 rpc.statd 548 545 545 0 S (threaded) nfsd 100137 S rpcsvc 0xbfdcc50 nfsd: master 100138 S rpcsvc 0x1acae550 nfsd: service 100139 S rpcsvc 0x1b72f3d0 nfsd: service 100140 S rpcsvc 0x1acae4d0 nfsd: service 100141 S rpcsvc 0x1acae450 nfsd: service 100142 S rpcsvc 0x1b72f350 nfsd: service 100143 S rpcsvc 0x1acae3d0 nfsd: service 100144 S rpcsvc 0x1b72f2d0 nfsd: service 100145 S rpcsvc 0x1b72f250 nfsd: service 100146 S rpcsvc 0x1b72f1d0 nfsd: service 100147 S rpcsvc 0x1acae350 nfsd: service 100148 S rpcsvc 0x1acae2d0 nfsd: service 100149 S rpcsvc 0x1acae250 nfsd: service 100150 S rpcsvc 0x1acae1d0 nfsd: service 100151 S rpcsvc 0x1acae150 nfsd: service 100152 S rpcsvc 0x1acae0d0 nfsd: service 100153 S rpcsvc 0x1b72f150 nfsd: service 100154 S rpcsvc 0x1acae050 nfsd: service 100155 S rpcsvc 0x1b72f0d0 nfsd: service 100156 S rpcsvc 0x1acadf50 nfsd: service 100157 S rpcsvc 0x1b72f050 nfsd: service 100158 S rpcsvc 0x1b72ef50 nfsd: service 100159 S rpcsvc 0x1b72eed0 nfsd: service 100160 S rpcsvc 0x1acaded0 nfsd: service 100161 S rpcsvc 0x1b72ee50 nfsd: service 100162 S rpcsvc 0x1acade50 nfsd: service 100163 S rpcsvc 0x1acaddd0 nfsd: service 100164 S rpcsvc 0x1acadd50 nfsd: service 100165 S rpcsvc 0x1acadcd0 nfsd: service 100166 S rpcsvc 0x1acadc50 nfsd: service 100167 S rpcsvc 0x1acadbd0 nfsd: service 100168 S rpcsvc 0x1b72edd0 nfsd: service 545 1 545 0 Ss select 0x1b393264 nfsd 539 1 539 0 Ss select 0x1b6f9524 mountd 532 531 531 0 S select 0x1b6f9664 nfsuserd 531 1 531 0 SWs pause 0x1af5ca60 nfsuserd 494 1 494 0 Ss nanslp 0x202fa9e watchdogd 410 1 410 0 Ss select 0x1b393364 rpcbind 401 1 401 0 Ss select 0x1ac68564 syslogd 325 1 325 0 Ss select 0x1ac685a4 devd 24 0 0 0 DL syncer 0x20aa220 [syncer] 23 0 0 0 DL vlruwt 0x1b38a9fc [vnlru] 22 0 0 0 DL (threaded) [bufdaemon] 100098 D qsleep 0x20a9bac [bufdaemon] 100103 D - 0x1de8380 [bufspacedaemon-0] 100115 D sdflush 0x1bafc684 [/home worker] 100116 D sdflush 0x1bafc484 [/tmp worker] 100117 D sdflush 0x1bafc284 [/usr worker] 100118 D sdflush 0x1bafc084 [/var worker] 21 0 0 0 DL psleep 0x20bf738 [vmdaemon] 20 0 0 0 DL (threaded) [pagedaemon] 100096 D psleep 0x20e9ddc [dom0] 100101 D launds 0x20e9de8 [laundry: dom0] 100102 D umarcl 0x14642a0 [uma] 19 0 0 0 DL - 0x1f05894 [rand_harvestq] 18 0 0 0 DL waiting 0x20e8ec4 [sctp_iterator] 17 0 0 0 DL - 0x20a97d8 [soaiod4] 16 0 0 0 DL - 0x20a97d8 [soaiod3] 15 0 0 0 DL - 0x20a97d8 [soaiod2] 9 0 0 0 DL - 0x20a97d8 [soaiod1] 8 0 0 0 DL - 0x1ac5d23c [fdc0] 14 0 0 0 DL (threaded) [usb] 100047 D - 0x1ac8dacc [usbus0] 100048 D - 0x1ac8dafc [usbus0] 100049 D - 0x1ac8db2c [usbus0] 100050 D - 0x1ac8db5c [usbus0] 100051 D - 0x1ac8db8c [usbus0] 100053 D - 0x1af63acc [usbus1] 100054 D - 0x1af63afc [usbus1] 100055 D - 0x1af63b2c [usbus1] 100056 D - 0x1af63b5c [usbus1] 100057 D - 0x1af63b8c [usbus1] 100059 D - 0x1af6eacc [usbus2] 100060 D - 0x1af6eafc [usbus2] 100061 D - 0x1af6eb2c [usbus2] 100062 D - 0x1af6eb5c [usbus2] 100063 D - 0x1af6eb8c [usbus2] 100064 D - 0x1af81acc [usbus3] 100065 D - 0x1af81afc [usbus3] 100066 D - 0x1af81b2c [usbus3] 100067 D - 0x1af81b5c [usbus3] 100068 D - 0x1af81b8c [usbus3] 100069 D - 0x1af88acc [usbus4] 100070 D - 0x1af88afc [usbus4] 100071 D - 0x1af88b2c [usbus4] 100072 D - 0x1af88b5c [usbus4] 100073 D - 0x1af88b8c [usbus4] 100075 D - 0x1ac78ca4 [usbus5] 100076 D - 0x1ac78cd4 [usbus5] 100077 D - 0x1ac78d04 [usbus5] 100078 D - 0x1ac78d34 [usbus5] 100079 D - 0x1ac78d64 [usbus5] 7 0 0 0 DL (threaded) [cam] 100042 D - 0x1e52c40 [doneq0] 100095 D - 0x1e52b6c [scanner] 6 0 0 0 DL crypto_ 0x9f500c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x9f5008c [crypto returns 2] 4 0 0 0 DL crypto_ 0x9f50054 [crypto returns 1] 3 0 0 0 DL crypto_ 0x9f5001c [crypto returns 0] 2 0 0 0 DL crypto_ 0x20be558 [crypto] 13 0 0 0 DL (threaded) [geom] 100029 D - 0x20e5270 [g_event] 100030 D - 0x20e5278 [g_up] 100031 D - 0x20e526c [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi6: Giant taskq] 100009 I [swi5: fast taskq] 100013 I [swi6: task queue] 100023 I [swi1: netisr 0] 100024 I [swi4: clock (0)] 100025 I [swi4: clock (1)] 100026 I [swi4: clock (2)] 100027 I [swi4: clock (3)] 100028 I [swi3: vm] 100043 I [irq256: hdac0] 100044 I [irq257: re0] 100045 I [irq22: ahci0] 100046 I [irq16: hdac1 ohci0] 100052 I [irq17: ohci1 ohci3] 100058 I [irq18: ohci2 ohci4] 100074 I [irq19: ehci0] 100080 I [irq14: ata0] 100081 I [swi0: uart] 100083 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xbfd99fc [init] 10 0 0 0 DL audit_w 0x20e969c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x20e5290 [swapper] 100008 D - 0x9e56f00 [thread taskq] 100010 D - 0xbffda00 [config_0] 100011 D - 0x9e56e00 [kqueue_ctx taskq] 100012 D - 0x9e56d80 [aiod_kick taskq] 100014 D - 0xbffd980 [softirq_0] 100015 D - 0xbffd900 [softirq_1] 100016 D - 0xbffd880 [softirq_2] 100017 D - 0xbffd800 [softirq_3] 100018 D - 0xbffd780 [if_io_tqg_0] 100019 D - 0xbffd700 [if_io_tqg_1] 100020 D - 0xbffd680 [if_io_tqg_2] 100021 D - 0xbffd600 [if_io_tqg_3] 100022 D - 0xbffd580 [if_config_tqg_0] 100032 D - 0x9e56c80 [firmware taskq] 100033 D - 0x9e56c00 [crypto_0] 100034 D - 0x9e56c00 [crypto_1] 100035 D - 0x9e56c00 [crypto_2] 100036 D - 0x9e56c00 [crypto_3] 100084 D - 0x1afcc000 [mca taskq] 100091 D - 0x9e56a80 [acpi_task_0] 100092 D - 0x9e56a80 [acpi_task_1] 100093 D - 0x9e56a80 [acpi_task_2] 100094 D - 0x9e56b80 [CAM taskq] 93076 92971 96183 0 Z+ find 93077 92970 96183 0 Z+ find db:0:pho> allt Tracing command sleep pid 27714 tid 103302 td 0x2588b000 sched_switch(2588b000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x544369f0 mi_switch(104,0,1830cc7,22b,2588b000,...) at mi_switch+0x209/frame 0x54436a30 sleepq_switch(2588b000,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x54436a64 sleepq_catch_signals(2588b000,0,601,54436ae4,ac4f,...) at sleepq_catch_signals+0x360/frame 0x54436aa8 sleepq_timedwait_sig(202fa9c,6c,ac4f,a0000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x54436ae4 _sleep(202fa9c,0,16c,1843745,3740a07a,...) at _sleep+0x32a/frame 0x54436b38 kern_clock_nanosleep(2588b000,0,0,54436bc8,54436bc0,...) at kern_clock_nanosleep+0x1e2/frame 0x54436ba0 sys_nanosleep(2588b000,2588b284,181d4b1,4,54436c28,...) at sys_nanosleep+0x7f/frame 0x54436be8 syscall(54436ce8,3b,3b,3b,ffbfec90,...) at syscall+0x364/frame 0x54436cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x54436cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x201784f3, esp = 0xffbfec50, ebp = 0xffbfec60 --- amdsmn_devclass() at 0x201784f3 Tracing command sleep pid 26244 tid 103323 td 0x2f9d8380 sched_switch(2f9d8380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x5449f9f0 mi_switch(104,0,1830cc7,22b,2f9d8380,...) at mi_switch+0x209/frame 0x5449fa30 sleepq_switch(2f9d8380,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x5449fa64 sleepq_catch_signals(2f9d8380,0,601,5449fae4,ac4a,...) at sleepq_catch_signals+0x360/frame 0x5449faa8 sleepq_timedwait_sig(202fa9c,6c,ac4a,a0000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x5449fae4 _sleep(202fa9c,0,16c,1843745,d4b1445b,...) at _sleep+0x32a/frame 0x5449fb38 kern_clock_nanosleep(2f9d8380,0,0,5449fbc8,5449fbc0,...) at kern_clock_nanosleep+0x1e2/frame 0x5449fba0 sys_nanosleep(2f9d8380,2f9d8604,181d4b1,4,203ccd0,...) at sys_nanosleep+0x7f/frame 0x5449fbe8 syscall(5449fce8,3b,3b,3b,ffbfec90,...) at syscall+0x364/frame 0x5449fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x5449fcdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x201784f3, esp = 0xffbfec50, ebp = 0xffbfec60 --- amdsmn_devclass() at 0x201784f3 Tracing command csh pid 18899 tid 100349 td 0x25504700 sched_switch(25504700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0xb940960 mi_switch(104,0,1830cc7,22b,25504700,...) at mi_switch+0x209/frame 0xb94099c sleepq_switch(25504700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0xb9409d0 sleepq_catch_signals(b940a38,10cdcd0,1af00c14,0,1af00c04,...) at sleepq_catch_signals+0x360/frame 0xb940a14 sleepq_wait_sig(1af00c74,0,b940a58,101,0,...) at sleepq_wait_sig+0x14/frame 0xb940a38 _cv_wait_sig(1af00c74,1af00c04,1857381,5a9,b940bb8,...) at _cv_wait_sig+0x1e7/frame 0xb940a80 tty_wait(1af00c00,1af00c74,b940bb8,1,0,...) at tty_wait+0x4b/frame 0xb940aa4 ttydisc_read(1af00c00,b940bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0xb940ae4 ttydev_read(1ac55700,b940bb8,0,25504700,b940b88,...) at ttydev_read+0x55/frame 0xb940b04 devfs_read_f(1ba8d1c0,b940bb8,26e60200,0,25504700,...) at devfs_read_f+0xc5/frame 0xb940b3c dofileread(1ba8d1c0,b940bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0xb940b6c kern_readv(25504700,10,b940bb8,256f0910,ffbfe4ca,...) at kern_readv+0x5e/frame 0xb940b98 sys_read(25504700,25504984,181d4b1,4,20737d8,...) at sys_read+0x54/frame 0xb940be8 syscall(b940ce8,3b,3b,3b,10,...) at syscall+0x364/frame 0xb940cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb940cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x202ad22f, esp = 0xffbfe478, ebp = 0xffbfe48c --- amdsmn_devclass() at 0x202ad22f Tracing command pmcstat pid 92971 tid 103292 td 0x2f9c9000 sched_switch(2f9c9000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x54404978 mi_switch(104,0,1830cc7,22b,2f9c9000,...) at mi_switch+0x209/frame 0x544049b8 sleepq_switch(2f9c9000,0,1830cc7,2b3,20e5c74,...) at sleepq_switch+0x110/frame 0x544049ec sleepq_wait(20e5c74,0,177b3c8,3,0,...) at sleepq_wait+0x48/frame 0x54404a10 _sx_xlock_hard(20e5c74,255c0000,0,2880525c,d46,...) at _sx_xlock_hard+0x50e/frame 0x54404aa4 _sx_xlock(20e5c74,0,2880525c,d46,1,...) at _sx_xlock+0xcf/frame 0x54404ad0 pmc_syscall_handler(2f9c9000,2f9c9284,181d4b1,4,20681a8,...) at pmc_syscall_handler+0x5b/frame 0x54404be8 syscall(54404ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x54404cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x54404cdc --- syscall (210, FreeBSD ELF32, pmc_syscall_handler), eip = 0x2058028d, esp = 0xffbfe0c8, ebp = 0xffbfe0ec --- amdsmn_devclass() at 0x2058028d Tracing command pmcstat pid 92970 tid 100687 td 0x255c0000 sched_switch(255c0000,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x153c3950 mi_switch(104,0,1830cc7,22b,255c0000,...) at mi_switch+0x209/frame 0x153c398c sleepq_switch(255c0000,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x153c39c0 sleepq_timedwait(202e997,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0x153c3a00 _sleep(202e997,0,0,288068bb,418937,...) at _sleep+0x342/frame 0x153c3a54 pause_sbt(288068bb,418937,0,0,0,...) at pause_sbt+0x12c/frame 0x153c3a90 pmc_release_pmc_descriptor(ffbfe0d8,153c3b08,4,d46,1,...) at pmc_release_pmc_descriptor+0x26a/frame 0x153c3ad0 pmc_syscall_handler(255c0000,255c0284,181d4b1,4,2060380,...) at pmc_syscall_handler+0x543/frame 0x153c3be8 syscall(153c3ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x153c3cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x153c3cdc --- syscall (210, FreeBSD ELF32, pmc_syscall_handler), eip = 0x2058028d, esp = 0xffbfe0c8, ebp = 0xffbfe0ec --- amdsmn_devclass() at 0x2058028d Tracing command sh pid 91032 tid 103189 td 0x20424380 sched_switch(20424380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x541fda48 mi_switch(104,0,1830cc7,22b,20424380,...) at mi_switch+0x209/frame 0x541fda84 sleepq_switch(20424380,0,1830cc7,1ca,74,...) at sleepq_switch+0x110/frame 0x541fdab8 sleepq_catch_signals(0,1830cc7,178,204b614,20424380,...) at sleepq_catch_signals+0x360/frame 0x541fdafc sleepq_wait_sig(26667a60,74,17932c6,100,0,...) at sleepq_wait_sig+0x14/frame 0x541fdb20 _sleep(26667a60,26667aa4,174,17932c6,0,...) at _sleep+0x396/frame 0x541fdb74 kern_sigsuspend(20424380,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0x541fdbb8 sys_sigsuspend(20424380,20424604,181d4b1,4,0,...) at sys_sigsuspend+0x55/frame 0x541fdbe8 syscall(541fdce8,3b,3b,3b,ffbfe564,...) at syscall+0x364/frame 0x541fdcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x541fdcdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x201c6473, esp = 0xffbfe524, ebp = 0xffbfe530 --- amdsmn_devclass() at 0x201c6473 Tracing command tee pid 91031 tid 100985 td 0x22437a80 sched_switch(22437a80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x299569d0 mi_switch(104,0,1830cc7,22b,22437a80,...) at mi_switch+0x209/frame 0x29956a0c sleepq_switch(22437a80,0,1830cc7,1ca,5c,...) at sleepq_switch+0x110/frame 0x29956a40 sleepq_catch_signals(0,1830cc7,178,204b054,22437a80,...) at sleepq_catch_signals+0x360/frame 0x29956a84 sleepq_wait_sig(2bf89b98,5c,1830e80,100,0,...) at sleepq_wait_sig+0x14/frame 0x29956aa8 _sleep(2bf89b98,2bf89d28,15c,1830e80,0,...) at _sleep+0x396/frame 0x29956afc pipe_read(22098d90,29956bb8,221e1400,0,22437a80,...) at pipe_read+0x49b/frame 0x29956b3c dofileread(22098d90,29956bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x29956b6c kern_readv(22437a80,0,29956bb8,20a372b8,2021d000,...) at kern_readv+0x5e/frame 0x29956b98 sys_read(22437a80,22437d04,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x29956be8 syscall(29956ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x29956cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x29956cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x201e722f, esp = 0xffbfe744, ebp = 0xffbfe758 --- amdsmn_devclass() at 0x201e722f Tracing command sh pid 91030 tid 103175 td 0x2fa12380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(2fa12380,ffffffff,16396,19307,2fa12380,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command bash pid 61932 tid 100348 td 0x25504a80 sched_switch(25504a80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0xb93b898 mi_switch(104,0,1830cc7,22b,25504a80,...) at mi_switch+0x209/frame 0xb93b8d4 sleepq_switch(25504a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0xb93b908 sleepq_catch_signals(b93b970,10cdcd0,1b9ee820,0,1b9ee810,...) at sleepq_catch_signals+0x360/frame 0xb93b94c sleepq_wait_sig(1b9ee824,0,b93b990,101,0,...) at sleepq_wait_sig+0x14/frame 0xb93b970 _cv_wait_sig(1b9ee824,1b9ee810,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0xb93b9b8 seltdwait(ffffffff,0,0,25504a80,0,...) at seltdwait+0xbb/frame 0xb93b9e8 kern_select(25504a80,1,ffbfdfd8,0,0,...) at kern_select+0x92a/frame 0xb93bb78 sys_pselect(25504a80,25504d04,181d4b1,4,10cedc6,...) at sys_pselect+0x166/frame 0xb93bbe8 syscall(b93bce8,3b,3b,3b,0,...) at syscall+0x364/frame 0xb93bcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb93bcdc --- syscall (522, FreeBSD ELF32, sys_pselect), eip = 0x282dc4c3, esp = 0xffbfdf78, ebp = 0xffbfdfa0 --- amdsmn_devclass() at 0x282dc4c3 Tracing command sshd pid 61851 tid 100577 td 0x254b6380 sched_switch(254b6380,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x98ea8c0 mi_switch(104,0,1830cc7,22b,254b6380,...) at mi_switch+0x209/frame 0x98ea8fc sleepq_switch(254b6380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x98ea930 sleepq_catch_signals(98ea998,10cdcd0,1f583760,0,1f583750,...) at sleepq_catch_signals+0x360/frame 0x98ea974 sleepq_wait_sig(1f583764,0,98ea9b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x98ea998 _cv_wait_sig(1f583764,1f583750,174c903,6fa,1b1,...) at _cv_wait_sig+0x1e7/frame 0x98ea9e0 seltdwait(ffffffff,0,0,254b6380,0,...) at seltdwait+0xbb/frame 0x98eaa10 kern_select(254b6380,e,200920b8,200921d8,0,...) at kern_select+0x92a/frame 0x98eaba0 sys_select(254b6380,254b6604,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x98eabe8 syscall(98eace8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x98eacdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x98eacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x2055a21f Tracing command sshd pid 61379 tid 100377 td 0x25542000 sched_switch(25542000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0xb9cc8f8 mi_switch(104,0,1830cc7,22b,25542000,...) at mi_switch+0x209/frame 0xb9cc934 sleepq_switch(25542000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0xb9cc968 sleepq_catch_signals(b9cc9d0,10cdcd0,1b9f0d20,0,1b9f0d10,...) at sleepq_catch_signals+0x360/frame 0xb9cc9ac sleepq_wait_sig(1b9f0d24,0,b9cc9f0,101,0,...) at sleepq_wait_sig+0x14/frame 0xb9cc9d0 _cv_wait_sig(1b9f0d24,1b9f0d10,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0xb9cca18 seltdwait(ffffffff,0,0,25542000,0,...) at seltdwait+0xbb/frame 0xb9cca48 kern_poll(25542000,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0xb9ccbb0 sys_poll(25542000,25542284,181d4b1,4,2073518,...) at sys_poll+0x67/frame 0xb9ccbe8 syscall(b9ccce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0xb9cccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0xb9cccdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x204fd7eb, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x204fd7eb Tracing command sh pid 96183 tid 100251 td 0x220f5000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(220f5000,ffffffff,177b7,1879b,220f5000,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command bash pid 16819 tid 100226 td 0x220e8000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(220e8000,ffffffff,41b3,18782,220e8000,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command csh pid 16581 tid 100237 td 0x220a6700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(220a6700,ffffffff,40c5,1878d,220a6700,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command su pid 16404 tid 100231 td 0x22049380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(22049380,ffffffff,4014,18787,22049380,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command sh pid 8138 tid 100120 td 0x1bbf4000 sched_switch(1bbf4000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1ef079d0 mi_switch(104,0,1830cc7,22b,1bbf4000,...) at mi_switch+0x209/frame 0x1ef07a0c sleepq_switch(1bbf4000,0,1830cc7,1ca,5c,...) at sleepq_switch+0x110/frame 0x1ef07a40 sleepq_catch_signals(0,1830cc7,178,204ba94,1bbf4000,...) at sleepq_catch_signals+0x360/frame 0x1ef07a84 sleepq_wait_sig(1badd4f8,5c,1830e80,100,0,...) at sleepq_wait_sig+0x14/frame 0x1ef07aa8 _sleep(1badd4f8,1badd688,15c,1830e80,0,...) at _sleep+0x396/frame 0x1ef07afc pipe_read(1ba4d700,1ef07bb8,1b772400,0,1bbf4000,...) at pipe_read+0x49b/frame 0x1ef07b3c dofileread(1ba4d700,1ef07bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1ef07b6c kern_readv(1bbf4000,0,1ef07bb8,1ba972b8,ffbfe95f,...) at kern_readv+0x5e/frame 0x1ef07b98 sys_read(1bbf4000,1bbf4284,181d4b1,4,1ef07c28,...) at sys_read+0x54/frame 0x1ef07be8 syscall(1ef07ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1ef07cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef07cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2023522f, esp = 0xffbfe8fc, ebp = 0xffbfe910 --- amdsmn_devclass() at 0x2023522f Tracing command awk pid 8136 tid 100126 td 0x1afdda80 sched_switch(1afdda80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1ef679d0 mi_switch(104,0,1830cc7,22b,1afdda80,...) at mi_switch+0x209/frame 0x1ef67a0c sleepq_switch(1afdda80,0,1830cc7,1ca,5c,...) at sleepq_switch+0x110/frame 0x1ef67a40 sleepq_catch_signals(0,1830cc7,178,204ed14,1afdda80,...) at sleepq_catch_signals+0x360/frame 0x1ef67a84 sleepq_wait_sig(220956a0,5c,1830e80,100,0,...) at sleepq_wait_sig+0x14/frame 0x1ef67aa8 _sleep(220956a0,22095830,15c,1830e80,0,...) at _sleep+0x396/frame 0x1ef67afc pipe_read(22098818,1ef67bb8,1b772400,0,1afdda80,...) at pipe_read+0x49b/frame 0x1ef67b3c dofileread(22098818,1ef67bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1ef67b6c kern_readv(1afdda80,0,1ef67bb8,1856e31,202e2000,...) at kern_readv+0x5e/frame 0x1ef67b98 sys_read(1afdda80,1afddd04,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1ef67be8 syscall(1ef67ce8,3b,3b,3b,2024a518,...) at syscall+0x364/frame 0x1ef67cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef67cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2024222f, esp = 0xffbfe950, ebp = 0xffbfe964 --- amdsmn_devclass() at 0x2024222f Tracing command sh pid 8135 tid 100215 td 0x1f6bb000 sched_switch(1f6bb000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15275940 mi_switch(104,0,1830cc7,22b,1f6bb000,...) at mi_switch+0x209/frame 0x15275980 sleepq_switch(1f6bb000,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x152759b4 sleepq_catch_signals(0,1830cc7,178,204c954,1f6bb000,...) at sleepq_catch_signals+0x360/frame 0x152759f8 sleepq_wait_sig(1faf3354,6c,184c984,100,0,...) at sleepq_wait_sig+0x14/frame 0x15275a1c _sleep(1faf3354,1faf33fc,36c,184c984,0,...) at _sleep+0x396/frame 0x15275a70 kern_wait6(1f6bb000,7,0,0,15275af4,...) at kern_wait6+0x567/frame 0x15275ac8 sys_wait4(1f6bb000,1f6bb284,181d4b1,4,0,...) at sys_wait4+0x87/frame 0x15275be8 syscall(15275ce8,3b,3b,3b,ffbfe96c,...) at syscall+0x364/frame 0x15275cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15275cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x201c6433, esp = 0xffbfe920, ebp = 0xffbfe938 --- amdsmn_devclass() at 0x201c6433 Tracing command sh pid 8131 tid 100206 td 0x2204aa80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(2204aa80,ffffffff,1fc3,1876e,2204aa80,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command tail pid 8129 tid 100216 td 0x220a7700 sched_switch(220a7700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15281748 mi_switch(104,0,1830cc7,22b,220a7700,...) at mi_switch+0x209/frame 0x15281788 sleepq_switch(220a7700,0,1830cc7,1ca,68,...) at sleepq_switch+0x110/frame 0x152817bc sleepq_catch_signals(220a7700,0,601,1528183c,ac4b,...) at sleepq_catch_signals+0x360/frame 0x15281800 sleepq_timedwait_sig(1b72bc00,68,ac4b,10000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x1528183c _sleep(1b72bc00,1b72bc00,168,180060d,563f1c4b,...) at _sleep+0x32a/frame 0x15281890 kqueue_kevent(0,1,15281bc0,15281b90,0,...) at kqueue_kevent+0x3aa/frame 0x15281af4 kern_kevent_fp(220a7700,220990a8,0,1,15281bc0,...) at kern_kevent_fp+0x9d/frame 0x15281b18 kern_kevent(220a7700,4,0,1,15281bc0,...) at kern_kevent+0xe8/frame 0x15281b68 kern_kevent_generic(15281bc0,184c917,220a7984,109fbe0,109fc30,...) at kern_kevent_generic+0x7c/frame 0x15281bb0 sys_kevent(220a7700,220a7984,181d4b1,4,174fdeb,...) at sys_kevent+0x6c/frame 0x15281be8 syscall(15281ce8,3b,3b,3b,1,...) at syscall+0x364/frame 0x15281cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15281cdc --- syscall (560, FreeBSD ELF32, sys_kevent), eip = 0x201d1813, esp = 0xffbfea58, ebp = 0xffbfea80 --- amdsmn_devclass() at 0x201d1813 Tracing command top pid 8127 tid 100207 td 0x2204a700 sched_switch(2204a700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1dbf18a0 mi_switch(104,0,1830cc7,22b,2204a700,...) at mi_switch+0x209/frame 0x1dbf18e0 sleepq_switch(2204a700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dbf1914 sleepq_catch_signals(8,17e3b45,cb,2204a700,2204a700,...) at sleepq_catch_signals+0x360/frame 0x1dbf1958 sleepq_timedwait_sig(1b3934e4,0,1dbf19bc,10000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x1dbf1998 _cv_timedwait_sig_sbt(1b3934e4,1b3934d0,95b75d55,ac4a,10000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1dbf19e0 seltdwait(ac4a,10000000,0,2204a700,0,...) at seltdwait+0x82/frame 0x1dbf1a10 kern_select(2204a700,2,ffbfec20,0,0,...) at kern_select+0x92a/frame 0x1dbf1ba0 sys_select(2204a700,2204a984,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1dbf1be8 syscall(1dbf1ce8,3b,3b,3b,ffbfeb80,...) at syscall+0x364/frame 0x1dbf1cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dbf1cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x202a821f, esp = 0xffbfeae4, ebp = 0xffbfeb08 --- amdsmn_devclass() at 0x202a821f Tracing command sshd pid 8126 tid 100222 td 0x1f976000 sched_switch(1f976000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1529f8c0 mi_switch(104,0,1830cc7,22b,1f976000,...) at mi_switch+0x209/frame 0x1529f8fc sleepq_switch(1f976000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1529f930 sleepq_catch_signals(1529f998,10cdcd0,1b392f60,0,1b392f50,...) at sleepq_catch_signals+0x360/frame 0x1529f974 sleepq_wait_sig(1b392f64,0,1529f9b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1529f998 _cv_wait_sig(1b392f64,1b392f50,174c903,6fa,5ab,...) at _cv_wait_sig+0x1e7/frame 0x1529f9e0 seltdwait(ffffffff,0,0,1f976000,0,...) at seltdwait+0xbb/frame 0x1529fa10 kern_select(1f976000,10,200920b8,200921d8,0,...) at kern_select+0x92a/frame 0x1529fba0 sys_select(1f976000,1f976284,181d4b1,4,17f6cfa,...) at sys_select+0x68/frame 0x1529fbe8 syscall(1529fce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1529fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1529fcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x2055a21f Tracing command sshd pid 8125 tid 100131 td 0x1baeb000 sched_switch(1baeb000,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1ef9b8c0 mi_switch(104,0,1830cc7,22b,1baeb000,...) at mi_switch+0x209/frame 0x1ef9b8fc sleepq_switch(1baeb000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef9b930 sleepq_catch_signals(1ef9b998,10cdcd0,1b393020,0,1b393010,...) at sleepq_catch_signals+0x360/frame 0x1ef9b974 sleepq_wait_sig(1b393024,0,1ef9b9b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef9b998 _cv_wait_sig(1b393024,1b393010,174c903,6fa,5ab,...) at _cv_wait_sig+0x1e7/frame 0x1ef9b9e0 seltdwait(ffffffff,0,0,1baeb000,0,...) at seltdwait+0xbb/frame 0x1ef9ba10 kern_select(1baeb000,10,200920b8,200921d8,0,...) at kern_select+0x92a/frame 0x1ef9bba0 sys_select(1baeb000,1baeb284,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1ef9bbe8 syscall(1ef9bce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1ef9bcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef9bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x2055a21f Tracing command sshd pid 8124 tid 100219 td 0x1f976a80 sched_switch(1f976a80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x152908c0 mi_switch(104,0,1830cc7,22b,1f976a80,...) at mi_switch+0x209/frame 0x152908fc sleepq_switch(1f976a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x15290930 sleepq_catch_signals(15290998,10cdcd0,1b392ca0,0,1b392c90,...) at sleepq_catch_signals+0x360/frame 0x15290974 sleepq_wait_sig(1b392ca4,0,152909b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x15290998 _cv_wait_sig(1b392ca4,1b392c90,174c903,6fa,1b1,...) at _cv_wait_sig+0x1e7/frame 0x152909e0 seltdwait(ffffffff,0,0,1f976a80,0,...) at seltdwait+0xbb/frame 0x15290a10 kern_select(1f976a80,e,200920b8,200921d8,0,...) at kern_select+0x92a/frame 0x15290ba0 sys_select(1f976a80,1f976d04,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x15290be8 syscall(15290ce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x15290cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15290cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x2055a21f Tracing command sshd pid 8122 tid 100214 td 0x22049700 sched_switch(22049700,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x152668f8 mi_switch(104,0,1830cc7,22b,22049700,...) at mi_switch+0x209/frame 0x15266934 sleepq_switch(22049700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x15266968 sleepq_catch_signals(152669d0,10cdcd0,1b392c20,0,1b392c10,...) at sleepq_catch_signals+0x360/frame 0x152669ac sleepq_wait_sig(1b392c24,0,152669f0,101,0,...) at sleepq_wait_sig+0x14/frame 0x152669d0 _cv_wait_sig(1b392c24,1b392c10,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0x15266a18 seltdwait(ffffffff,0,0,22049700,1860aae,...) at seltdwait+0xbb/frame 0x15266a48 kern_poll(22049700,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0x15266bb0 sys_poll(22049700,22049984,181d4b1,4,174fdeb,...) at sys_poll+0x67/frame 0x15266be8 syscall(15266ce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x15266cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15266cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x204fd7eb, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x204fd7eb Tracing command sshd pid 8119 tid 100220 td 0x1f976700 sched_switch(1f976700,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x152958f8 mi_switch(104,0,1830cc7,22b,1f976700,...) at mi_switch+0x209/frame 0x15295934 sleepq_switch(1f976700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x15295968 sleepq_catch_signals(152959d0,10cdcd0,1b392ce0,0,1b392cd0,...) at sleepq_catch_signals+0x360/frame 0x152959ac sleepq_wait_sig(1b392ce4,0,152959f0,101,0,...) at sleepq_wait_sig+0x14/frame 0x152959d0 _cv_wait_sig(1b392ce4,1b392cd0,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0x15295a18 seltdwait(ffffffff,0,0,1f976700,1860aae,...) at seltdwait+0xbb/frame 0x15295a48 kern_poll(1f976700,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0x15295bb0 sys_poll(1f976700,1f976984,181d4b1,4,174fdeb,...) at sys_poll+0x67/frame 0x15295be8 syscall(15295ce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x15295cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15295cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x204fd7eb, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x204fd7eb Tracing command sshd pid 8118 tid 100121 td 0x1bbf3a80 sched_switch(1bbf3a80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1ef0c8f8 mi_switch(104,0,1830cc7,22b,1bbf3a80,...) at mi_switch+0x209/frame 0x1ef0c934 sleepq_switch(1bbf3a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef0c968 sleepq_catch_signals(1ef0c9d0,10cdcd0,1b6f96e0,0,1b6f96d0,...) at sleepq_catch_signals+0x360/frame 0x1ef0c9ac sleepq_wait_sig(1b6f96e4,0,1ef0c9f0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef0c9d0 _cv_wait_sig(1b6f96e4,1b6f96d0,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0x1ef0ca18 seltdwait(ffffffff,0,0,1bbf3a80,2,...) at seltdwait+0xbb/frame 0x1ef0ca48 kern_poll(1bbf3a80,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0x1ef0cbb0 sys_poll(1bbf3a80,1bbf3d04,181d4b1,4,174fdeb,...) at sys_poll+0x67/frame 0x1ef0cbe8 syscall(1ef0cce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1ef0ccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef0ccdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x204fd7eb, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x204fd7eb Tracing command bash pid 6660 tid 100128 td 0x1afdd700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(1afdd700,ffffffff,1a04,18720,1afdd700,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command sshd pid 6659 tid 100119 td 0x1baeba80 sched_switch(1baeba80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1eefb8c0 mi_switch(104,0,1830cc7,22b,1baeba80,...) at mi_switch+0x209/frame 0x1eefb8fc sleepq_switch(1baeba80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eefb930 sleepq_catch_signals(1eefb998,10cdcd0,1b6f9360,0,1b6f9350,...) at sleepq_catch_signals+0x360/frame 0x1eefb974 sleepq_wait_sig(1b6f9364,0,1eefb9b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eefb998 _cv_wait_sig(1b6f9364,1b6f9350,174c903,6fa,1b1,...) at _cv_wait_sig+0x1e7/frame 0x1eefb9e0 seltdwait(ffffffff,0,0,1baeba80,0,...) at seltdwait+0xbb/frame 0x1eefba10 kern_select(1baeba80,e,200920b8,200921d8,0,...) at kern_select+0x92a/frame 0x1eefbba0 sys_select(1baeba80,1baebd04,181d4b1,4,1839ab1,...) at sys_select+0x68/frame 0x1eefbbe8 syscall(1eefbce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1eefbcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eefbcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfa754, ebp = 0xffbfa778 --- amdsmn_devclass() at 0x2055a21f Tracing command sshd pid 6657 tid 100123 td 0x1bbf3380 sched_switch(1bbf3380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1ef268f8 mi_switch(104,0,1830cc7,22b,1bbf3380,...) at mi_switch+0x209/frame 0x1ef26934 sleepq_switch(1bbf3380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef26968 sleepq_catch_signals(1ef269d0,10cdcd0,1b392f20,0,1b392f10,...) at sleepq_catch_signals+0x360/frame 0x1ef269ac sleepq_wait_sig(1b392f24,0,1ef269f0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef269d0 _cv_wait_sig(1b392f24,1b392f10,174c903,6fa,0,...) at _cv_wait_sig+0x1e7/frame 0x1ef26a18 seltdwait(ffffffff,0,0,1bbf3380,0,...) at seltdwait+0xbb/frame 0x1ef26a48 kern_poll(1bbf3380,ffbfe7f0,1,0,0,...) at kern_poll+0x4d4/frame 0x1ef26bb0 sys_poll(1bbf3380,1bbf3604,181d4b1,4,174fdeb,...) at sys_poll+0x67/frame 0x1ef26be8 syscall(1ef26ce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1ef26cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef26cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x204fd7eb, esp = 0xffbfe77c, ebp = 0xffbfe798 --- amdsmn_devclass() at 0x204fd7eb Tracing command sh pid 870 tid 100197 td 0x1baec700 sched_switch(1baec700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dbad9d0 mi_switch(104,0,1830cc7,22b,1baec700,...) at mi_switch+0x209/frame 0x1dbada0c sleepq_switch(1baec700,0,1830cc7,1ca,5c,...) at sleepq_switch+0x110/frame 0x1dbada40 sleepq_catch_signals(0,1830cc7,178,204e914,1baec700,...) at sleepq_catch_signals+0x360/frame 0x1dbada84 sleepq_wait_sig(1ba946a0,5c,1830e80,100,0,...) at sleepq_wait_sig+0x14/frame 0x1dbadaa8 _sleep(1ba946a0,1ba94830,15c,1830e80,0,...) at _sleep+0x396/frame 0x1dbadafc pipe_read(1ba4e268,1dbadbb8,1b770700,0,1baec700,...) at pipe_read+0x49b/frame 0x1dbadb3c dofileread(1ba4e268,1dbadbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1dbadb6c kern_readv(1baec700,0,1dbadbb8,bff21d0,ffbfe95f,...) at kern_readv+0x5e/frame 0x1dbadb98 sys_read(1baec700,1baec984,181d4b1,4,1dbadc28,...) at sys_read+0x54/frame 0x1dbadbe8 syscall(1dbadce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1dbadcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dbadcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2023522f, esp = 0xffbfe8fc, ebp = 0xffbfe910 --- amdsmn_devclass() at 0x2023522f Tracing command awk pid 869 tid 100196 td 0x1baeca80 sched_switch(1baeca80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1dba89d0 mi_switch(104,0,1830cc7,22b,1baeca80,...) at mi_switch+0x209/frame 0x1dba8a0c sleepq_switch(1baeca80,0,1830cc7,1ca,5c,...) at sleepq_switch+0x110/frame 0x1dba8a40 sleepq_catch_signals(0,1830cc7,178,204d9d4,1baeca80,...) at sleepq_catch_signals+0x360/frame 0x1dba8a84 sleepq_wait_sig(1bae59f0,5c,1830e80,100,0,...) at sleepq_wait_sig+0x14/frame 0x1dba8aa8 _sleep(1bae59f0,1bae5b80,15c,1830e80,0,...) at _sleep+0x396/frame 0x1dba8afc pipe_read(1ba4e348,1dba8bb8,1b770700,0,1baeca80,...) at pipe_read+0x49b/frame 0x1dba8b3c dofileread(1ba4e348,1dba8bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1dba8b6c kern_readv(1baeca80,0,1dba8bb8,1856e31,202e2000,...) at kern_readv+0x5e/frame 0x1dba8b98 sys_read(1baeca80,1baecd04,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1dba8be8 syscall(1dba8ce8,3b,3b,3b,2024a518,...) at syscall+0x364/frame 0x1dba8cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dba8cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2024222f, esp = 0xffbfe960, ebp = 0xffbfe974 --- amdsmn_devclass() at 0x2024222f Tracing command sh pid 868 tid 100195 td 0x1bb0c000 sched_switch(1bb0c000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1dba3940 mi_switch(104,0,1830cc7,22b,1bb0c000,...) at mi_switch+0x209/frame 0x1dba3980 sleepq_switch(1bb0c000,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x1dba39b4 sleepq_catch_signals(0,1830cc7,178,204ccd4,1bb0c000,...) at sleepq_catch_signals+0x360/frame 0x1dba39f8 sleepq_wait_sig(1bb089fc,6c,184c984,100,0,...) at sleepq_wait_sig+0x14/frame 0x1dba3a1c _sleep(1bb089fc,1bb08aa4,36c,184c984,0,...) at _sleep+0x396/frame 0x1dba3a70 kern_wait6(1bb0c000,7,0,0,1dba3af4,...) at kern_wait6+0x567/frame 0x1dba3ac8 sys_wait4(1bb0c000,1bb0c284,181d4b1,4,0,...) at sys_wait4+0x87/frame 0x1dba3be8 syscall(1dba3ce8,3b,3b,3b,ffbfe96c,...) at syscall+0x364/frame 0x1dba3cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1dba3cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x201c6433, esp = 0xffbfe920, ebp = 0xffbfe938 --- amdsmn_devclass() at 0x201c6433 Tracing command sh pid 860 tid 100186 td 0x1f977a80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(1f977a80,ffffffff,35c,1875a,1f977a80,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command login pid 764 tid 100192 td 0x1f6bb700 sched_switch(1f6bb700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db91940 mi_switch(104,0,1830cc7,22b,1f6bb700,...) at mi_switch+0x209/frame 0x1db91980 sleepq_switch(1f6bb700,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x1db919b4 sleepq_catch_signals(0,1830cc7,178,204e0d4,1f6bb700,...) at sleepq_catch_signals+0x360/frame 0x1db919f8 sleepq_wait_sig(1faf39fc,6c,184c984,100,0,...) at sleepq_wait_sig+0x14/frame 0x1db91a1c _sleep(1faf39fc,1faf3aa4,36c,184c984,0,...) at _sleep+0x396/frame 0x1db91a70 kern_wait6(1f6bb700,0,49d3,0,1db91af4,...) at kern_wait6+0x567/frame 0x1db91ac8 sys_wait4(1f6bb700,1f6bb984,181d4b1,4,174fdeb,...) at sys_wait4+0x87/frame 0x1db91be8 syscall(1db91ce8,3b,3b,3b,49d3,...) at syscall+0x364/frame 0x1db91cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db91cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x201ba433, esp = 0xffbfecb8, ebp = 0xffbfecd0 --- amdsmn_devclass() at 0x201ba433 Tracing command getty pid 763 tid 100191 td 0x1f6bba80 sched_switch(1f6bba80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1db8c960 mi_switch(104,0,1830cc7,22b,1f6bba80,...) at mi_switch+0x209/frame 0x1db8c99c sleepq_switch(1f6bba80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db8c9d0 sleepq_catch_signals(1db8ca38,10cdcd0,1b77d414,0,1b77d404,...) at sleepq_catch_signals+0x360/frame 0x1db8ca14 sleepq_wait_sig(1b77d474,0,1db8ca58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db8ca38 _cv_wait_sig(1b77d474,1b77d404,1857381,5a9,1db8cbb8,...) at _cv_wait_sig+0x1e7/frame 0x1db8ca80 tty_wait(1b77d400,1b77d474,1db8cbb8,1,0,...) at tty_wait+0x4b/frame 0x1db8caa4 ttydisc_read(1b77d400,1db8cbb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db8cae4 ttydev_read(1b691500,1db8cbb8,0,1f6bba80,1db8cb88,...) at ttydev_read+0x55/frame 0x1db8cb04 devfs_read_f(1ba8d0a8,1db8cbb8,1b93b600,0,1f6bba80,...) at devfs_read_f+0xc5/frame 0x1db8cb3c dofileread(1ba8d0a8,1db8cbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db8cb6c kern_readv(1f6bba80,0,1db8cbb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db8cb98 sys_read(1f6bba80,1f6bbd04,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1db8cbe8 syscall(1db8cce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db8ccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db8ccdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 762 tid 100190 td 0x1f6bc000 sched_switch(1f6bc000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1db87960 mi_switch(104,0,1830cc7,22b,1f6bc000,...) at mi_switch+0x209/frame 0x1db8799c sleepq_switch(1f6bc000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db879d0 sleepq_catch_signals(1db87a38,10cdcd0,1b77d614,0,1b77d604,...) at sleepq_catch_signals+0x360/frame 0x1db87a14 sleepq_wait_sig(1b77d674,0,1db87a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db87a38 _cv_wait_sig(1b77d674,1b77d604,1857381,5a9,1db87bb8,...) at _cv_wait_sig+0x1e7/frame 0x1db87a80 tty_wait(1b77d600,1b77d674,1db87bb8,1,0,...) at tty_wait+0x4b/frame 0x1db87aa4 ttydisc_read(1b77d600,1db87bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db87ae4 ttydev_read(1b691600,1db87bb8,0,1f6bc000,1db87b88,...) at ttydev_read+0x55/frame 0x1db87b04 devfs_read_f(1ba8cf50,1db87bb8,1ac9d500,0,1f6bc000,...) at devfs_read_f+0xc5/frame 0x1db87b3c dofileread(1ba8cf50,1db87bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db87b6c kern_readv(1f6bc000,0,1db87bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db87b98 sys_read(1f6bc000,1f6bc284,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1db87be8 syscall(1db87ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db87cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db87cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 761 tid 100189 td 0x1f6bc380 sched_switch(1f6bc380,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1db82960 mi_switch(104,0,1830cc7,22b,1f6bc380,...) at mi_switch+0x209/frame 0x1db8299c sleepq_switch(1f6bc380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db829d0 sleepq_catch_signals(1db82a38,10cdcd0,1b77d814,0,1b77d804,...) at sleepq_catch_signals+0x360/frame 0x1db82a14 sleepq_wait_sig(1b77d874,0,1db82a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db82a38 _cv_wait_sig(1b77d874,1b77d804,1857381,5a9,1db82bb8,...) at _cv_wait_sig+0x1e7/frame 0x1db82a80 tty_wait(1b77d800,1b77d874,1db82bb8,1,0,...) at tty_wait+0x4b/frame 0x1db82aa4 ttydisc_read(1b77d800,1db82bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db82ae4 ttydev_read(1b691700,1db82bb8,0,1f6bc380,1db82b88,...) at ttydev_read+0x55/frame 0x1db82b04 devfs_read_f(1ba8d0e0,1db82bb8,1b9aa900,0,1f6bc380,...) at devfs_read_f+0xc5/frame 0x1db82b3c dofileread(1ba8d0e0,1db82bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db82b6c kern_readv(1f6bc380,0,1db82bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db82b98 sys_read(1f6bc380,1f6bc604,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1db82be8 syscall(1db82ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db82cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db82cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 760 tid 100109 td 0x1baec000 sched_switch(1baec000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1eeb5960 mi_switch(104,0,1830cc7,22b,1baec000,...) at mi_switch+0x209/frame 0x1eeb599c sleepq_switch(1baec000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eeb59d0 sleepq_catch_signals(1eeb5a38,10cdcd0,1b77da14,0,1b77da04,...) at sleepq_catch_signals+0x360/frame 0x1eeb5a14 sleepq_wait_sig(1b77da74,0,1eeb5a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eeb5a38 _cv_wait_sig(1b77da74,1b77da04,1857381,5a9,1eeb5bb8,...) at _cv_wait_sig+0x1e7/frame 0x1eeb5a80 tty_wait(1b77da00,1b77da74,1eeb5bb8,1,0,...) at tty_wait+0x4b/frame 0x1eeb5aa4 ttydisc_read(1b77da00,1eeb5bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1eeb5ae4 ttydev_read(1b691800,1eeb5bb8,0,1baec000,1eeb5b88,...) at ttydev_read+0x55/frame 0x1eeb5b04 devfs_read_f(1ba8d000,1eeb5bb8,1ac9d300,0,1baec000,...) at devfs_read_f+0xc5/frame 0x1eeb5b3c dofileread(1ba8d000,1eeb5bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1eeb5b6c kern_readv(1baec000,0,1eeb5bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1eeb5b98 sys_read(1baec000,1baec284,181d4b1,4,2079870,...) at sys_read+0x54/frame 0x1eeb5be8 syscall(1eeb5ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1eeb5cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eeb5cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 759 tid 100179 td 0x1f978a80 sched_switch(1f978a80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1db3f960 mi_switch(104,0,1830cc7,22b,1f978a80,...) at mi_switch+0x209/frame 0x1db3f99c sleepq_switch(1f978a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db3f9d0 sleepq_catch_signals(1db3fa38,10cdcd0,1b77dc14,0,1b77dc04,...) at sleepq_catch_signals+0x360/frame 0x1db3fa14 sleepq_wait_sig(1b77dc74,0,1db3fa58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db3fa38 _cv_wait_sig(1b77dc74,1b77dc04,1857381,5a9,1db3fbb8,...) at _cv_wait_sig+0x1e7/frame 0x1db3fa80 tty_wait(1b77dc00,1b77dc74,1db3fbb8,1,0,...) at tty_wait+0x4b/frame 0x1db3faa4 ttydisc_read(1b77dc00,1db3fbb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db3fae4 ttydev_read(1b691900,1db3fbb8,0,1f978a80,1db3fb88,...) at ttydev_read+0x55/frame 0x1db3fb04 devfs_read_f(1ba8cee0,1db3fbb8,1b9aa700,0,1f978a80,...) at devfs_read_f+0xc5/frame 0x1db3fb3c dofileread(1ba8cee0,1db3fbb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db3fb6c kern_readv(1f978a80,0,1db3fbb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db3fb98 sys_read(1f978a80,1f978d04,181d4b1,4,20799d0,...) at sys_read+0x54/frame 0x1db3fbe8 syscall(1db3fce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db3fcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db3fcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 758 tid 100187 td 0x1f977700 sched_switch(1f977700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1db74960 mi_switch(104,0,1830cc7,22b,1f977700,...) at mi_switch+0x209/frame 0x1db7499c sleepq_switch(1f977700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db749d0 sleepq_catch_signals(1db74a38,10cdcd0,1b77de14,0,1b77de04,...) at sleepq_catch_signals+0x360/frame 0x1db74a14 sleepq_wait_sig(1b77de74,0,1db74a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db74a38 _cv_wait_sig(1b77de74,1b77de04,1857381,5a9,1db74bb8,...) at _cv_wait_sig+0x1e7/frame 0x1db74a80 tty_wait(1b77de00,1b77de74,1db74bb8,1,0,...) at tty_wait+0x4b/frame 0x1db74aa4 ttydisc_read(1b77de00,1db74bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db74ae4 ttydev_read(1b691a00,1db74bb8,0,1f977700,1db74b88,...) at ttydev_read+0x55/frame 0x1db74b04 devfs_read_f(1ba26850,1db74bb8,1ac9d100,0,1f977700,...) at devfs_read_f+0xc5/frame 0x1db74b3c dofileread(1ba26850,1db74bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db74b6c kern_readv(1f977700,0,1db74bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db74b98 sys_read(1f977700,1f977984,181d4b1,4,20797c0,...) at sys_read+0x54/frame 0x1db74be8 syscall(1db74ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db74cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db74cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 757 tid 100188 td 0x1f977380 sched_switch(1f977380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1db79960 mi_switch(104,0,1830cc7,22b,1f977380,...) at mi_switch+0x209/frame 0x1db7999c sleepq_switch(1f977380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db799d0 sleepq_catch_signals(1db79a38,10cdcd0,1b780014,0,1b780004,...) at sleepq_catch_signals+0x360/frame 0x1db79a14 sleepq_wait_sig(1b780074,0,1db79a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db79a38 _cv_wait_sig(1b780074,1b780004,1857381,5a9,1db79bb8,...) at _cv_wait_sig+0x1e7/frame 0x1db79a80 tty_wait(1b780000,1b780074,1db79bb8,1,0,...) at tty_wait+0x4b/frame 0x1db79aa4 ttydisc_read(1b780000,1db79bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1db79ae4 ttydev_read(1b691b00,1db79bb8,0,1f977380,1db79b88,...) at ttydev_read+0x55/frame 0x1db79b04 devfs_read_f(1ba8d070,1db79bb8,1b93b900,0,1f977380,...) at devfs_read_f+0xc5/frame 0x1db79b3c dofileread(1ba8d070,1db79bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1db79b6c kern_readv(1f977380,0,1db79bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1db79b98 sys_read(1f977380,1f977604,181d4b1,4,174fdeb,...) at sys_read+0x54/frame 0x1db79be8 syscall(1db79ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db79cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db79cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command getty pid 756 tid 100104 td 0x1b395a80 sched_switch(1b395a80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1ee96960 mi_switch(104,0,1830cc7,22b,1b395a80,...) at mi_switch+0x209/frame 0x1ee9699c sleepq_switch(1b395a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ee969d0 sleepq_catch_signals(1ee96a38,10cdcd0,1b780214,0,1b780204,...) at sleepq_catch_signals+0x360/frame 0x1ee96a14 sleepq_wait_sig(1b780274,0,1ee96a58,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ee96a38 _cv_wait_sig(1b780274,1b780204,1857381,5a9,1ee96bb8,...) at _cv_wait_sig+0x1e7/frame 0x1ee96a80 tty_wait(1b780200,1b780274,1ee96bb8,1,0,...) at tty_wait+0x4b/frame 0x1ee96aa4 ttydisc_read(1b780200,1ee96bb8,0,d1,1de4aa0,...) at ttydisc_read+0x238/frame 0x1ee96ae4 ttydev_read(1b691c00,1ee96bb8,0,1b395a80,1ee96b88,...) at ttydev_read+0x55/frame 0x1ee96b04 devfs_read_f(1ba8cea8,1ee96bb8,1b9aab00,0,1b395a80,...) at devfs_read_f+0xc5/frame 0x1ee96b3c dofileread(1ba8cea8,1ee96bb8,ffffffff,ffffffff,0,...) at dofileread+0x8a/frame 0x1ee96b6c kern_readv(1b395a80,0,1ee96bb8,0,ffbfed61,...) at kern_readv+0x5e/frame 0x1ee96b98 sys_read(1b395a80,1b395d04,181d4b1,4,207b290,...) at sys_read+0x54/frame 0x1ee96be8 syscall(1ee96ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1ee96cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ee96cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2020022f, esp = 0xffbfed34, ebp = 0xffbfed48 --- amdsmn_devclass() at 0x2020022f Tracing command inetd pid 739 tid 100129 td 0x1afdd380 sched_switch(1afdd380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1ef918c0 mi_switch(104,0,1830cc7,22b,1afdd380,...) at mi_switch+0x209/frame 0x1ef918fc sleepq_switch(1afdd380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef91930 sleepq_catch_signals(1ef91998,10cdcd0,bf79ee0,0,bf79ed0,...) at sleepq_catch_signals+0x360/frame 0x1ef91974 sleepq_wait_sig(bf79ee4,0,1ef919b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef91998 _cv_wait_sig(bf79ee4,bf79ed0,174c903,6fa,1f67c000,...) at _cv_wait_sig+0x1e7/frame 0x1ef919e0 seltdwait(ffffffff,0,0,1afdd380,0,...) at seltdwait+0xbb/frame 0x1ef91a10 kern_select(1afdd380,9,ffbfe5c0,0,0,...) at kern_select+0x92a/frame 0x1ef91ba0 sys_select(1afdd380,1afdd604,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1ef91be8 syscall(1ef91ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1ef91cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef91cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2021b21f, esp = 0xffbfdfec, ebp = 0xffbfe010 --- amdsmn_devclass() at 0x2021b21f Tracing command nfscbd pid 705 tid 100124 td 0x1afde380 sched_switch(1afde380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ef5a7e8 mi_switch(104,0,1830cc7,22b,1afde380,...) at mi_switch+0x209/frame 0x1ef5a828 sleepq_switch(1afde380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef5a85c sleepq_catch_signals(8,17e3b45,cb,1afde380,1afde380,...) at sleepq_catch_signals+0x360/frame 0x1ef5a8a0 sleepq_timedwait_sig(1b3a2b50,0,1ef5a904,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1ef5a8e0 _cv_timedwait_sig_sbt(1b3a2b50,1b2750c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1ef5a928 svc_run_internal(13de740,1b2750c0,1af5b000,1ef5aa30,0,...) at svc_run_internal+0xb5f/frame 0x1ef5aa04 svc_run(1b275000,0,1842be4,f5,1ef5aad0,...) at svc_run+0x1ed/frame 0x1ef5aa58 nfscbd_nfsd(1afde380,1ef5ab30,4,1100ef0,2021d000,...) at nfscbd_nfsd+0x105/frame 0x1ef5aaf8 nfssvc_nfscl(1afde380,1afde604,1afde380,1af5b000,1ef5acdc,...) at nfssvc_nfscl+0x279/frame 0x1ef5abd0 sys_nfssvc(1afde380,1afde604,181d4b1,4,1ef5ac28,...) at sys_nfssvc+0xa6/frame 0x1ef5abe8 syscall(1ef5ace8,3b,3b,3b,1,...) at syscall+0x364/frame 0x1ef5acdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef5acdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x200a0a7f, esp = 0xffbfeb0c, ebp = 0xffbfee28 --- amdsmn_devclass() at 0x200a0a7f Tracing command nfscbd pid 705 tid 100183 td 0x1f978700 sched_switch(1f978700,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1db5caa8 mi_switch(104,0,1830cc7,22b,1f978700,...) at mi_switch+0x209/frame 0x1db5cae4 sleepq_switch(1f978700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db5cb18 sleepq_catch_signals(1db5cb80,10cdcd0,1b2750d0,0,1b2750c0,...) at sleepq_catch_signals+0x360/frame 0x1db5cb5c sleepq_wait_sig(1b3a2ad0,0,1db5cba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db5cb80 _cv_wait_sig(1b3a2ad0,1b2750c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1db5cbc8 svc_run_internal(1db5ccd4,10ad5de,1b2750c0,1db5cce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1db5cca4 svc_thread_start(1b2750c0,1db5cce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1db5ccac fork_exit(13de740,1b2750c0,1db5cce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db5ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db5ccd4 --- trap 0, eip = 0, esp = 0x33, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 705 tid 100184 td 0x1f978380 sched_switch(1f978380,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1db61aa8 mi_switch(104,0,1830cc7,22b,1f978380,...) at mi_switch+0x209/frame 0x1db61ae4 sleepq_switch(1f978380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db61b18 sleepq_catch_signals(1db61b80,10cdcd0,1b2750d0,0,1b2750c0,...) at sleepq_catch_signals+0x360/frame 0x1db61b5c sleepq_wait_sig(1b3a2a50,0,1db61ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db61b80 _cv_wait_sig(1b3a2a50,1b2750c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1db61bc8 svc_run_internal(1db61cd4,10ad5de,1b2750c0,1db61ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1db61ca4 svc_thread_start(1b2750c0,1db61ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1db61cac fork_exit(13de740,1b2750c0,1db61ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db61cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db61cd4 --- trap 0, eip = 0, esp = 0x33, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 705 tid 100185 td 0x1f978000 sched_switch(1f978000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1db66aa8 mi_switch(104,0,1830cc7,22b,1f978000,...) at mi_switch+0x209/frame 0x1db66ae4 sleepq_switch(1f978000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db66b18 sleepq_catch_signals(1db66b80,10cdcd0,1b2750d0,0,1b2750c0,...) at sleepq_catch_signals+0x360/frame 0x1db66b5c sleepq_wait_sig(1b72ecd0,0,1db66ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db66b80 _cv_wait_sig(1b72ecd0,1b2750c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1db66bc8 svc_run_internal(1db66cd4,10ad5de,1b2750c0,1db66ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1db66ca4 svc_thread_start(1b2750c0,1db66ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1db66cac fork_exit(13de740,1b2750c0,1db66ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db66cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db66cd4 --- trap 0, eip = 0, esp = 0x33, ebp = 0x1 --- (null)() at 0 Tracing command nfscbd pid 704 tid 100125 td 0x1afde000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(1afde000,ffffffff,2c0,1871d,1afde000,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command cron pid 671 tid 100181 td 0x1f6bca80 sched_switch(1f6bca80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db4d9f0 mi_switch(104,0,1830cc7,22b,1f6bca80,...) at mi_switch+0x209/frame 0x1db4da30 sleepq_switch(1f6bca80,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x1db4da64 sleepq_catch_signals(1f6bca80,2,601,1db4dae4,ac51,...) at sleepq_catch_signals+0x360/frame 0x1db4daa8 sleepq_timedwait_sig(202fa9e,6c,ac51,bffbcbdd,3,...) at sleepq_timedwait_sig+0x17/frame 0x1db4dae4 _sleep(202fa9e,0,16c,1843745,469b56e4,...) at _sleep+0x32a/frame 0x1db4db38 kern_clock_nanosleep(1f6bca80,0,0,1db4dbc8,1db4dbc0,...) at kern_clock_nanosleep+0x1e2/frame 0x1db4dba0 sys_nanosleep(1f6bca80,1f6bcd04,181d4b1,4,2079d40,...) at sys_nanosleep+0x7f/frame 0x1db4dbe8 syscall(1db4dce8,3b,3b,3b,ffbfed48,...) at syscall+0x364/frame 0x1db4dcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db4dcdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x201a34f3, esp = 0xffbfec98, ebp = 0xffbfeca8 --- amdsmn_devclass() at 0x201a34f3 Tracing command sendmail pid 667 tid 100130 td 0x1baeb380 sched_switch(1baeb380,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1ef96a48 mi_switch(104,0,1830cc7,22b,1baeb380,...) at mi_switch+0x209/frame 0x1ef96a84 sleepq_switch(1baeb380,0,1830cc7,1ca,74,...) at sleepq_switch+0x110/frame 0x1ef96ab8 sleepq_catch_signals(0,1830cc7,178,204ea14,1baeb380,...) at sleepq_catch_signals+0x360/frame 0x1ef96afc sleepq_wait_sig(1bae8a60,74,17932c6,100,0,...) at sleepq_wait_sig+0x14/frame 0x1ef96b20 _sleep(1bae8a60,1bae8aa4,174,17932c6,0,...) at _sleep+0x396/frame 0x1ef96b74 kern_sigsuspend(1baeb380,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0x1ef96bb8 sys_sigsuspend(1baeb380,1baeb604,181d4b1,4,2079ea0,...) at sys_sigsuspend+0x55/frame 0x1ef96be8 syscall(1ef96ce8,3b,3b,3b,ffbfda80,...) at syscall+0x364/frame 0x1ef96cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef96cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x20452473, esp = 0xffbfda5c, ebp = 0xffbfda68 --- amdsmn_devclass() at 0x20452473 Tracing command sendmail pid 664 tid 100135 td 0x1bbf2700 sched_switch(1bbf2700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1efba8a0 mi_switch(104,0,1830cc7,22b,1bbf2700,...) at mi_switch+0x209/frame 0x1efba8e0 sleepq_switch(1bbf2700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efba914 sleepq_catch_signals(8,17e3b45,cb,1bbf2700,1bbf2700,...) at sleepq_catch_signals+0x360/frame 0x1efba958 sleepq_timedwait_sig(1b6f89e4,0,1efba9bc,50000000,0,...) at sleepq_timedwait_sig+0x17/frame 0x1efba998 _cv_timedwait_sig_sbt(1b6f89e4,1b6f89d0,cc01ad6a,ac4b,50000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1efba9e0 seltdwait(ac4b,50000000,0,1bbf2700,0,...) at seltdwait+0x82/frame 0x1efbaa10 kern_select(1bbf2700,5,ffbfd168,0,0,...) at kern_select+0x92a/frame 0x1efbaba0 sys_select(1bbf2700,1bbf2984,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1efbabe8 syscall(1efbace8,3b,3b,3b,ffbfd0d8,...) at syscall+0x364/frame 0x1efbacdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1efbacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x204c121f, esp = 0xffbfd02c, ebp = 0xffbfd050 --- amdsmn_devclass() at 0x204c121f Tracing command sshd pid 661 tid 100171 td 0x1bb15a80 sched_switch(1bb15a80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1db118c0 mi_switch(104,0,1830cc7,22b,1bb15a80,...) at mi_switch+0x209/frame 0x1db118fc sleepq_switch(1bb15a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db11930 sleepq_catch_signals(1db11998,10cdcd0,1b3931a0,0,1b393190,...) at sleepq_catch_signals+0x360/frame 0x1db11974 sleepq_wait_sig(1b3931a4,0,1db119b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db11998 _cv_wait_sig(1b3931a4,1b393190,174c903,6fa,1f67c410,...) at _cv_wait_sig+0x1e7/frame 0x1db119e0 seltdwait(ffffffff,0,0,1bb15a80,0,...) at seltdwait+0xbb/frame 0x1db11a10 kern_select(1bb15a80,9,20092120,0,0,...) at kern_select+0x92a/frame 0x1db11ba0 sys_select(1bb15a80,1bb15d04,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1db11be8 syscall(1db11ce8,3b,3b,3b,20775000,...) at syscall+0x364/frame 0x1db11cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db11cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2055a21f, esp = 0xffbfe7f4, ebp = 0xffbfe818 --- amdsmn_devclass() at 0x2055a21f Tracing command smartd pid 623 tid 100172 td 0x1f6bd380 sched_switch(1f6bd380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db179f0 mi_switch(104,0,1830cc7,22b,1f6bd380,...) at mi_switch+0x209/frame 0x1db17a30 sleepq_switch(1f6bd380,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x1db17a64 sleepq_catch_signals(1f6bd380,0,601,1db17ae4,afd8,...) at sleepq_catch_signals+0x360/frame 0x1db17aa8 sleepq_timedwait_sig(202fa9c,6c,afd8,80000000,70,...) at sleepq_timedwait_sig+0x17/frame 0x1db17ae4 _sleep(202fa9c,0,16c,1843745,9225d599,...) at _sleep+0x32a/frame 0x1db17b38 kern_clock_nanosleep(1f6bd380,0,0,1db17bc8,1db17bc0,...) at kern_clock_nanosleep+0x1e2/frame 0x1db17ba0 sys_nanosleep(1f6bd380,1f6bd604,181d4b1,4,2079df0,...) at sys_nanosleep+0x7f/frame 0x1db17be8 syscall(1db17ce8,3b,3b,3b,2844c000,...) at syscall+0x364/frame 0x1db17cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db17cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x283714f3, esp = 0xffbf9f50, ebp = 0xffbf9f68 --- amdsmn_devclass() at 0x283714f3 Tracing command powerd pid 593 tid 100174 td 0x1bb15380 sched_switch(1bb15380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db228a0 mi_switch(104,0,1830cc7,22b,1bb15380,...) at mi_switch+0x209/frame 0x1db228e0 sleepq_switch(1bb15380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db22914 sleepq_catch_signals(8,17e3b45,cb,1bb15380,1bb15380,...) at sleepq_catch_signals+0x360/frame 0x1db22958 sleepq_timedwait_sig(1b6f95e4,0,1db229bc,3ffffff,0,...) at sleepq_timedwait_sig+0x17/frame 0x1db22998 _cv_timedwait_sig_sbt(1b6f95e4,1b6f95d0,8f972826,ac4a,3ffffff,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1db229e0 seltdwait(ac4a,3ffffff,0,0,1bb1544c,...) at seltdwait+0x82/frame 0x1db22a10 kern_select(1bb15380,0,ffbfed90,0,ffbfed90,...) at kern_select+0x92a/frame 0x1db22ba0 sys_select(1bb15380,1bb15604,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1db22be8 syscall(1db22ce8,3b,3b,3b,ffbfed80,...) at syscall+0x364/frame 0x1db22cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db22cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x201fe21f, esp = 0xffbfecf4, ebp = 0xffbfed18 --- amdsmn_devclass() at 0x201fe21f Tracing command ntpd pid 590 tid 100173 td 0x1bb15700 sched_switch(1bb15700,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1db1d8c0 mi_switch(104,0,1830cc7,22b,1bb15700,...) at mi_switch+0x209/frame 0x1db1d8fc sleepq_switch(1bb15700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db1d930 sleepq_catch_signals(1db1d998,10cdcd0,1ac686e0,0,1ac686d0,...) at sleepq_catch_signals+0x360/frame 0x1db1d974 sleepq_wait_sig(1ac686e4,0,1db1d9b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db1d998 _cv_wait_sig(1ac686e4,1ac686d0,174c903,6fa,1f67d410,...) at _cv_wait_sig+0x1e7/frame 0x1db1d9e0 seltdwait(ffffffff,0,0,1bb15700,0,...) at seltdwait+0xbb/frame 0x1db1da10 kern_select(1bb15700,1b,ffbfe2c0,0,0,...) at kern_select+0x92a/frame 0x1db1dba0 sys_select(1bb15700,1bb15984,181d4b1,4,207ab58,...) at sys_select+0x68/frame 0x1db1dbe8 syscall(1db1dce8,3b,3b,3b,204e6000,...) at syscall+0x364/frame 0x1db1dcdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db1dcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x204a621f, esp = 0xffbfe1e4, ebp = 0xffbfe208 --- amdsmn_devclass() at 0x204a621f Tracing command rpc.lockd pid 552 tid 100170 td 0x1f6bd700 sched_switch(1f6bd700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db0c830 mi_switch(104,0,1830cc7,22b,1f6bd700,...) at mi_switch+0x209/frame 0x1db0c870 sleepq_switch(1f6bd700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db0c8a4 sleepq_catch_signals(8,17e3b45,cb,1f6bd700,1f6bd700,...) at sleepq_catch_signals+0x360/frame 0x1db0c8e8 sleepq_timedwait_sig(1acadb50,0,1db0c94c,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1db0c928 _cv_timedwait_sig_sbt(1acadb50,1f6c00c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1db0c970 svc_run_internal(1f6bd8c4,14,176aca0,18620c9,19,...) at svc_run_internal+0xb5f/frame 0x1db0ca4c svc_run(1f6c0000,0,4,13c4600,1db0cb40,...) at svc_run+0x1ed/frame 0x1db0caa0 sys_nlm_syscall(1f6bd700,1f6bd984,181d4b1,4,1db0cc28,...) at sys_nlm_syscall+0x93e/frame 0x1db0cbe8 syscall(1db0cce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1db0ccdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db0ccdc --- syscall (154, FreeBSD ELF32, sys_nlm_syscall), eip = 0x200b8a8f, esp = 0xffbfed74, ebp = 0xffbfee28 --- amdsmn_devclass() at 0x200b8a8f Tracing command rpc.statd pid 549 tid 100169 td 0x1f68f380 sched_switch(1f68f380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1db078a0 mi_switch(104,0,1830cc7,22b,1f68f380,...) at mi_switch+0x209/frame 0x1db078e0 sleepq_switch(1f68f380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db07914 sleepq_catch_signals(8,17e3b45,cb,1f68f380,1f68f380,...) at sleepq_catch_signals+0x360/frame 0x1db07958 sleepq_timedwait_sig(1b393324,0,1db079bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1db07998 _cv_timedwait_sig_sbt(1b393324,1b393310,c86ba077,ac4f,e0000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1db079e0 seltdwait(ac4f,e0000000,1,1f68f380,0,...) at seltdwait+0x82/frame 0x1db07a10 kern_select(1f68f380,8,ffbfece8,0,0,...) at kern_select+0x92a/frame 0x1db07ba0 sys_select(1f68f380,1f68f604,181d4b1,4,203da50,...) at sys_select+0x68/frame 0x1db07be8 syscall(1db07ce8,3b,3b,3b,ffbfece8,...) at syscall+0x364/frame 0x1db07cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1db07cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x201f521f, esp = 0xffbfec34, ebp = 0xffbfed80 --- amdsmn_devclass() at 0x201f521f Tracing command nfsd pid 548 tid 100137 td 0x1bbf2000 sched_switch(1bbf2000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1efc4370 mi_switch(104,0,1830cc7,22b,1bbf2000,...) at mi_switch+0x209/frame 0x1efc43b0 sleepq_switch(1bbf2000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efc43e4 sleepq_catch_signals(8,17e3b45,cb,1bbf2000,1bbf2000,...) at sleepq_catch_signals+0x360/frame 0x1efc4428 sleepq_timedwait_sig(bfdcc50,0,1efc448c,0,0,...) at sleepq_timedwait_sig+0x17/frame 0x1efc4468 _cv_timedwait_sig_sbt(bfdcc50,1b3850c0,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1efc44b0 svc_run_internal(13de740,1b3850c0,1f2ef6a8,1efc45b8,0,...) at svc_run_internal+0xb5f/frame 0x1efc458c svc_run(1b385000,1bbf2000,1842cfb,1fe,0,...) at svc_run+0x1ed/frame 0x1efc45e0 nfsrvd_nfsd(1bbf2000,1efc47b0,34,0,0,...) at nfsrvd_nfsd+0x3ad/frame 0x1efc4720 nfssvc_nfsd(1bbf2000,1bbf2284,1bbf2000,1f2ef6a8,1efc4cdc,...) at nfssvc_nfsd+0x621/frame 0x1efc4bd0 sys_nfssvc(1bbf2000,1bbf2284,181d4b1,4,174fdeb,...) at sys_nfssvc+0xa6/frame 0x1efc4be8 syscall(1efc4ce8,3b,3b,3b,ffbfeb08,...) at syscall+0x364/frame 0x1efc4cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1efc4cdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x200a4a7f, esp = 0xffbfe77c, ebp = 0xffbfe9e8 --- amdsmn_devclass() at 0x200a4a7f Tracing command nfsd pid 548 tid 100138 td 0x1baea700 sched_switch(1baea700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1efc9aa8 mi_switch(104,0,1830cc7,22b,1baea700,...) at mi_switch+0x209/frame 0x1efc9ae4 sleepq_switch(1baea700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efc9b18 sleepq_catch_signals(1efc9b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efc9b5c sleepq_wait_sig(1acae550,0,1efc9ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efc9b80 _cv_wait_sig(1acae550,1b3850c0,17b5f47,439,0,...) at _cv_wait_sig+0x1e7/frame 0x1efc9bc8 svc_run_internal(1efc9cd4,10ad5de,1b3850c0,1efc9ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efc9ca4 svc_thread_start(1b3850c0,1efc9ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efc9cac fork_exit(13de740,1b3850c0,1efc9ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efc9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efc9cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100139 td 0x1baea380 sched_switch(1baea380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1efceaa8 mi_switch(104,0,1830cc7,22b,1baea380,...) at mi_switch+0x209/frame 0x1efceae4 sleepq_switch(1baea380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efceb18 sleepq_catch_signals(1efceb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efceb5c sleepq_wait_sig(1b72f3d0,0,1efceba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efceb80 _cv_wait_sig(1b72f3d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efcebc8 svc_run_internal(1efcecd4,10ad5de,1b3850c0,1efcece8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efceca4 svc_thread_start(1b3850c0,1efcece8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efcecac fork_exit(13de740,1b3850c0,1efcece8,3b,3b,...) at fork_exit+0x7e/frame 0x1efcecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efcecd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100140 td 0x1baea000 sched_switch(1baea000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1efd3aa8 mi_switch(104,0,1830cc7,22b,1baea000,...) at mi_switch+0x209/frame 0x1efd3ae4 sleepq_switch(1baea000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efd3b18 sleepq_catch_signals(1efd3b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efd3b5c sleepq_wait_sig(1acae4d0,0,1efd3ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efd3b80 _cv_wait_sig(1acae4d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efd3bc8 svc_run_internal(1efd3cd4,10ad5de,1b3850c0,1efd3ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efd3ca4 svc_thread_start(1b3850c0,1efd3ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efd3cac fork_exit(13de740,1b3850c0,1efd3ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efd3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efd3cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100141 td 0x1b3aba80 sched_switch(1b3aba80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1efd8aa8 mi_switch(104,0,1830cc7,22b,1b3aba80,...) at mi_switch+0x209/frame 0x1efd8ae4 sleepq_switch(1b3aba80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efd8b18 sleepq_catch_signals(1efd8b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efd8b5c sleepq_wait_sig(1acae450,0,1efd8ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efd8b80 _cv_wait_sig(1acae450,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efd8bc8 svc_run_internal(1efd8cd4,10ad5de,1b3850c0,1efd8ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efd8ca4 svc_thread_start(1b3850c0,1efd8ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efd8cac fork_exit(13de740,1b3850c0,1efd8ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efd8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efd8cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100142 td 0x1b3ab700 sched_switch(1b3ab700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1efddaa8 mi_switch(104,0,1830cc7,22b,1b3ab700,...) at mi_switch+0x209/frame 0x1efddae4 sleepq_switch(1b3ab700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efddb18 sleepq_catch_signals(1efddb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efddb5c sleepq_wait_sig(1b72f350,0,1efddba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efddb80 _cv_wait_sig(1b72f350,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efddbc8 svc_run_internal(1efddcd4,10ad5de,1b3850c0,1efddce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efddca4 svc_thread_start(1b3850c0,1efddce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efddcac fork_exit(13de740,1b3850c0,1efddce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efddcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efddcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100143 td 0x1b3ab380 sched_switch(1b3ab380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1efe2aa8 mi_switch(104,0,1830cc7,22b,1b3ab380,...) at mi_switch+0x209/frame 0x1efe2ae4 sleepq_switch(1b3ab380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efe2b18 sleepq_catch_signals(1efe2b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efe2b5c sleepq_wait_sig(1acae3d0,0,1efe2ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efe2b80 _cv_wait_sig(1acae3d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efe2bc8 svc_run_internal(1efe2cd4,10ad5de,1b3850c0,1efe2ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efe2ca4 svc_thread_start(1b3850c0,1efe2ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efe2cac fork_exit(13de740,1b3850c0,1efe2ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efe2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efe2cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100144 td 0x1f68f000 sched_switch(1f68f000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1efe7aa8 mi_switch(104,0,1830cc7,22b,1f68f000,...) at mi_switch+0x209/frame 0x1efe7ae4 sleepq_switch(1f68f000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efe7b18 sleepq_catch_signals(1efe7b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efe7b5c sleepq_wait_sig(1b72f2d0,0,1efe7ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efe7b80 _cv_wait_sig(1b72f2d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efe7bc8 svc_run_internal(1efe7cd4,10ad5de,1b3850c0,1efe7ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efe7ca4 svc_thread_start(1b3850c0,1efe7ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efe7cac fork_exit(13de740,1b3850c0,1efe7ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efe7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efe7cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100145 td 0x1f68ea80 sched_switch(1f68ea80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1efecaa8 mi_switch(104,0,1830cc7,22b,1f68ea80,...) at mi_switch+0x209/frame 0x1efecae4 sleepq_switch(1f68ea80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efecb18 sleepq_catch_signals(1efecb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1efecb5c sleepq_wait_sig(1b72f250,0,1efecba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efecb80 _cv_wait_sig(1b72f250,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1efecbc8 svc_run_internal(1efeccd4,10ad5de,1b3850c0,1efecce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1efecca4 svc_thread_start(1b3850c0,1efecce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1efeccac fork_exit(13de740,1b3850c0,1efecce8,3b,3b,...) at fork_exit+0x7e/frame 0x1efeccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1efeccd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100146 td 0x1f68e700 sched_switch(1f68e700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1eff1aa8 mi_switch(104,0,1830cc7,22b,1f68e700,...) at mi_switch+0x209/frame 0x1eff1ae4 sleepq_switch(1f68e700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eff1b18 sleepq_catch_signals(1eff1b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1eff1b5c sleepq_wait_sig(1b72f1d0,0,1eff1ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eff1b80 _cv_wait_sig(1b72f1d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1eff1bc8 svc_run_internal(1eff1cd4,10ad5de,1b3850c0,1eff1ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1eff1ca4 svc_thread_start(1b3850c0,1eff1ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1eff1cac fork_exit(13de740,1b3850c0,1eff1ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1eff1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eff1cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100147 td 0x1f68e380 sched_switch(1f68e380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1eff6aa8 mi_switch(104,0,1830cc7,22b,1f68e380,...) at mi_switch+0x209/frame 0x1eff6ae4 sleepq_switch(1f68e380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eff6b18 sleepq_catch_signals(1eff6b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1eff6b5c sleepq_wait_sig(1acae350,0,1eff6ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1eff6b80 _cv_wait_sig(1acae350,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1eff6bc8 svc_run_internal(1eff6cd4,10ad5de,1b3850c0,1eff6ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1eff6ca4 svc_thread_start(1b3850c0,1eff6ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1eff6cac fork_exit(13de740,1b3850c0,1eff6ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1eff6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eff6cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100148 td 0x1f68e000 sched_switch(1f68e000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1effbaa8 mi_switch(104,0,1830cc7,22b,1f68e000,...) at mi_switch+0x209/frame 0x1effbae4 sleepq_switch(1f68e000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1effbb18 sleepq_catch_signals(1effbb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1effbb5c sleepq_wait_sig(1acae2d0,0,1effbba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1effbb80 _cv_wait_sig(1acae2d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1effbbc8 svc_run_internal(1effbcd4,10ad5de,1b3850c0,1effbce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1effbca4 svc_thread_start(1b3850c0,1effbce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1effbcac fork_exit(13de740,1b3850c0,1effbce8,3b,3b,...) at fork_exit+0x7e/frame 0x1effbcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1effbcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100149 td 0x1f68da80 sched_switch(1f68da80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daa3aa8 mi_switch(104,0,1830cc7,22b,1f68da80,...) at mi_switch+0x209/frame 0x1daa3ae4 sleepq_switch(1f68da80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daa3b18 sleepq_catch_signals(1daa3b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daa3b5c sleepq_wait_sig(1acae250,0,1daa3ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daa3b80 _cv_wait_sig(1acae250,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daa3bc8 svc_run_internal(1daa3cd4,10ad5de,1b3850c0,1daa3ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daa3ca4 svc_thread_start(1b3850c0,1daa3ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daa3cac fork_exit(13de740,1b3850c0,1daa3ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daa3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daa3cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100150 td 0x1f68d700 sched_switch(1f68d700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daa8aa8 mi_switch(104,0,1830cc7,22b,1f68d700,...) at mi_switch+0x209/frame 0x1daa8ae4 sleepq_switch(1f68d700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daa8b18 sleepq_catch_signals(1daa8b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daa8b5c sleepq_wait_sig(1acae1d0,0,1daa8ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daa8b80 _cv_wait_sig(1acae1d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daa8bc8 svc_run_internal(1daa8cd4,10ad5de,1b3850c0,1daa8ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daa8ca4 svc_thread_start(1b3850c0,1daa8ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daa8cac fork_exit(13de740,1b3850c0,1daa8ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daa8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daa8cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100151 td 0x1f68d380 sched_switch(1f68d380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daadaa8 mi_switch(104,0,1830cc7,22b,1f68d380,...) at mi_switch+0x209/frame 0x1daadae4 sleepq_switch(1f68d380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daadb18 sleepq_catch_signals(1daadb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daadb5c sleepq_wait_sig(1acae150,0,1daadba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daadb80 _cv_wait_sig(1acae150,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daadbc8 svc_run_internal(1daadcd4,10ad5de,1b3850c0,1daadce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daadca4 svc_thread_start(1b3850c0,1daadce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daadcac fork_exit(13de740,1b3850c0,1daadce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daadcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daadcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100152 td 0x1f68d000 sched_switch(1f68d000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dab2aa8 mi_switch(104,0,1830cc7,22b,1f68d000,...) at mi_switch+0x209/frame 0x1dab2ae4 sleepq_switch(1f68d000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dab2b18 sleepq_catch_signals(1dab2b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dab2b5c sleepq_wait_sig(1acae0d0,0,1dab2ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dab2b80 _cv_wait_sig(1acae0d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dab2bc8 svc_run_internal(1dab2cd4,10ad5de,1b3850c0,1dab2ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dab2ca4 svc_thread_start(1b3850c0,1dab2ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dab2cac fork_exit(13de740,1b3850c0,1dab2ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dab2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dab2cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100153 td 0x1f68ca80 sched_switch(1f68ca80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dab7aa8 mi_switch(104,0,1830cc7,22b,1f68ca80,...) at mi_switch+0x209/frame 0x1dab7ae4 sleepq_switch(1f68ca80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dab7b18 sleepq_catch_signals(1dab7b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dab7b5c sleepq_wait_sig(1b72f150,0,1dab7ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dab7b80 _cv_wait_sig(1b72f150,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dab7bc8 svc_run_internal(1dab7cd4,10ad5de,1b3850c0,1dab7ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dab7ca4 svc_thread_start(1b3850c0,1dab7ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dab7cac fork_exit(13de740,1b3850c0,1dab7ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dab7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dab7cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100154 td 0x1f68c700 sched_switch(1f68c700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dabcaa8 mi_switch(104,0,1830cc7,22b,1f68c700,...) at mi_switch+0x209/frame 0x1dabcae4 sleepq_switch(1f68c700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dabcb18 sleepq_catch_signals(1dabcb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dabcb5c sleepq_wait_sig(1acae050,0,1dabcba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dabcb80 _cv_wait_sig(1acae050,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dabcbc8 svc_run_internal(1dabccd4,10ad5de,1b3850c0,1dabcce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dabcca4 svc_thread_start(1b3850c0,1dabcce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dabccac fork_exit(13de740,1b3850c0,1dabcce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dabccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dabccd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100155 td 0x1f68c380 sched_switch(1f68c380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dac1aa8 mi_switch(104,0,1830cc7,22b,1f68c380,...) at mi_switch+0x209/frame 0x1dac1ae4 sleepq_switch(1f68c380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dac1b18 sleepq_catch_signals(1dac1b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dac1b5c sleepq_wait_sig(1b72f0d0,0,1dac1ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dac1b80 _cv_wait_sig(1b72f0d0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dac1bc8 svc_run_internal(1dac1cd4,10ad5de,1b3850c0,1dac1ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dac1ca4 svc_thread_start(1b3850c0,1dac1ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dac1cac fork_exit(13de740,1b3850c0,1dac1ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dac1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dac1cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100156 td 0x1f68c000 sched_switch(1f68c000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dac6aa8 mi_switch(104,0,1830cc7,22b,1f68c000,...) at mi_switch+0x209/frame 0x1dac6ae4 sleepq_switch(1f68c000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dac6b18 sleepq_catch_signals(1dac6b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dac6b5c sleepq_wait_sig(1acadf50,0,1dac6ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dac6b80 _cv_wait_sig(1acadf50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dac6bc8 svc_run_internal(1dac6cd4,10ad5de,1b3850c0,1dac6ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dac6ca4 svc_thread_start(1b3850c0,1dac6ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dac6cac fork_exit(13de740,1b3850c0,1dac6ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dac6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dac6cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100157 td 0x1f694380 sched_switch(1f694380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dacbaa8 mi_switch(104,0,1830cc7,22b,1f694380,...) at mi_switch+0x209/frame 0x1dacbae4 sleepq_switch(1f694380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dacbb18 sleepq_catch_signals(1dacbb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dacbb5c sleepq_wait_sig(1b72f050,0,1dacbba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dacbb80 _cv_wait_sig(1b72f050,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dacbbc8 svc_run_internal(1dacbcd4,10ad5de,1b3850c0,1dacbce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dacbca4 svc_thread_start(1b3850c0,1dacbce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dacbcac fork_exit(13de740,1b3850c0,1dacbce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dacbcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dacbcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100158 td 0x1f694000 sched_switch(1f694000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dad0aa8 mi_switch(104,0,1830cc7,22b,1f694000,...) at mi_switch+0x209/frame 0x1dad0ae4 sleepq_switch(1f694000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dad0b18 sleepq_catch_signals(1dad0b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dad0b5c sleepq_wait_sig(1b72ef50,0,1dad0ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dad0b80 _cv_wait_sig(1b72ef50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dad0bc8 svc_run_internal(1dad0cd4,10ad5de,1b3850c0,1dad0ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dad0ca4 svc_thread_start(1b3850c0,1dad0ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dad0cac fork_exit(13de740,1b3850c0,1dad0ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dad0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dad0cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100159 td 0x1f693a80 sched_switch(1f693a80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dad5aa8 mi_switch(104,0,1830cc7,22b,1f693a80,...) at mi_switch+0x209/frame 0x1dad5ae4 sleepq_switch(1f693a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dad5b18 sleepq_catch_signals(1dad5b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dad5b5c sleepq_wait_sig(1b72eed0,0,1dad5ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dad5b80 _cv_wait_sig(1b72eed0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dad5bc8 svc_run_internal(1dad5cd4,10ad5de,1b3850c0,1dad5ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dad5ca4 svc_thread_start(1b3850c0,1dad5ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dad5cac fork_exit(13de740,1b3850c0,1dad5ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dad5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dad5cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100160 td 0x1f693700 sched_switch(1f693700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dadaaa8 mi_switch(104,0,1830cc7,22b,1f693700,...) at mi_switch+0x209/frame 0x1dadaae4 sleepq_switch(1f693700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dadab18 sleepq_catch_signals(1dadab80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dadab5c sleepq_wait_sig(1acaded0,0,1dadaba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dadab80 _cv_wait_sig(1acaded0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dadabc8 svc_run_internal(1dadacd4,10ad5de,1b3850c0,1dadace8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dadaca4 svc_thread_start(1b3850c0,1dadace8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dadacac fork_exit(13de740,1b3850c0,1dadace8,3b,3b,...) at fork_exit+0x7e/frame 0x1dadacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dadacd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100161 td 0x1f693380 sched_switch(1f693380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1dadfaa8 mi_switch(104,0,1830cc7,22b,1f693380,...) at mi_switch+0x209/frame 0x1dadfae4 sleepq_switch(1f693380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dadfb18 sleepq_catch_signals(1dadfb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dadfb5c sleepq_wait_sig(1b72ee50,0,1dadfba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dadfb80 _cv_wait_sig(1b72ee50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dadfbc8 svc_run_internal(1dadfcd4,10ad5de,1b3850c0,1dadfce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dadfca4 svc_thread_start(1b3850c0,1dadfce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dadfcac fork_exit(13de740,1b3850c0,1dadfce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dadfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dadfcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100162 td 0x1f693000 sched_switch(1f693000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dae4aa8 mi_switch(104,0,1830cc7,22b,1f693000,...) at mi_switch+0x209/frame 0x1dae4ae4 sleepq_switch(1f693000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dae4b18 sleepq_catch_signals(1dae4b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dae4b5c sleepq_wait_sig(1acade50,0,1dae4ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dae4b80 _cv_wait_sig(1acade50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dae4bc8 svc_run_internal(1dae4cd4,10ad5de,1b3850c0,1dae4ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dae4ca4 svc_thread_start(1b3850c0,1dae4ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dae4cac fork_exit(13de740,1b3850c0,1dae4ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dae4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dae4cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100163 td 0x1f692a80 sched_switch(1f692a80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dae9aa8 mi_switch(104,0,1830cc7,22b,1f692a80,...) at mi_switch+0x209/frame 0x1dae9ae4 sleepq_switch(1f692a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dae9b18 sleepq_catch_signals(1dae9b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dae9b5c sleepq_wait_sig(1acaddd0,0,1dae9ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dae9b80 _cv_wait_sig(1acaddd0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dae9bc8 svc_run_internal(1dae9cd4,10ad5de,1b3850c0,1dae9ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dae9ca4 svc_thread_start(1b3850c0,1dae9ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dae9cac fork_exit(13de740,1b3850c0,1dae9ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dae9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dae9cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100164 td 0x1f692700 sched_switch(1f692700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daeeaa8 mi_switch(104,0,1830cc7,22b,1f692700,...) at mi_switch+0x209/frame 0x1daeeae4 sleepq_switch(1f692700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daeeb18 sleepq_catch_signals(1daeeb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daeeb5c sleepq_wait_sig(1acadd50,0,1daeeba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daeeb80 _cv_wait_sig(1acadd50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daeebc8 svc_run_internal(1daeecd4,10ad5de,1b3850c0,1daeece8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daeeca4 svc_thread_start(1b3850c0,1daeece8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daeecac fork_exit(13de740,1b3850c0,1daeece8,3b,3b,...) at fork_exit+0x7e/frame 0x1daeecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daeecd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100165 td 0x1f692380 sched_switch(1f692380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daf3aa8 mi_switch(104,0,1830cc7,22b,1f692380,...) at mi_switch+0x209/frame 0x1daf3ae4 sleepq_switch(1f692380,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daf3b18 sleepq_catch_signals(1daf3b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daf3b5c sleepq_wait_sig(1acadcd0,0,1daf3ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daf3b80 _cv_wait_sig(1acadcd0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daf3bc8 svc_run_internal(1daf3cd4,10ad5de,1b3850c0,1daf3ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daf3ca4 svc_thread_start(1b3850c0,1daf3ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daf3cac fork_exit(13de740,1b3850c0,1daf3ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daf3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daf3cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100166 td 0x1f692000 sched_switch(1f692000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1daf8aa8 mi_switch(104,0,1830cc7,22b,1f692000,...) at mi_switch+0x209/frame 0x1daf8ae4 sleepq_switch(1f692000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1daf8b18 sleepq_catch_signals(1daf8b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1daf8b5c sleepq_wait_sig(1acadc50,0,1daf8ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1daf8b80 _cv_wait_sig(1acadc50,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1daf8bc8 svc_run_internal(1daf8cd4,10ad5de,1b3850c0,1daf8ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1daf8ca4 svc_thread_start(1b3850c0,1daf8ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1daf8cac fork_exit(13de740,1b3850c0,1daf8ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1daf8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1daf8cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100167 td 0x1f68fa80 sched_switch(1f68fa80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1dafdaa8 mi_switch(104,0,1830cc7,22b,1f68fa80,...) at mi_switch+0x209/frame 0x1dafdae4 sleepq_switch(1f68fa80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1dafdb18 sleepq_catch_signals(1dafdb80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1dafdb5c sleepq_wait_sig(1acadbd0,0,1dafdba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1dafdb80 _cv_wait_sig(1acadbd0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1dafdbc8 svc_run_internal(1dafdcd4,10ad5de,1b3850c0,1dafdce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1dafdca4 svc_thread_start(1b3850c0,1dafdce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1dafdcac fork_exit(13de740,1b3850c0,1dafdce8,3b,3b,...) at fork_exit+0x7e/frame 0x1dafdcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1dafdcd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 548 tid 100168 td 0x1f68f700 sched_switch(1f68f700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1db02aa8 mi_switch(104,0,1830cc7,22b,1f68f700,...) at mi_switch+0x209/frame 0x1db02ae4 sleepq_switch(1f68f700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1db02b18 sleepq_catch_signals(1db02b80,10cdcd0,1b3850d0,0,1b3850c0,...) at sleepq_catch_signals+0x360/frame 0x1db02b5c sleepq_wait_sig(1b72edd0,0,1db02ba0,101,0,...) at sleepq_wait_sig+0x14/frame 0x1db02b80 _cv_wait_sig(1b72edd0,1b3850c0,17b5f47,481,0,...) at _cv_wait_sig+0x1e7/frame 0x1db02bc8 svc_run_internal(1db02cd4,10ad5de,1b3850c0,1db02ce8,17ec59f,...) at svc_run_internal+0xcbe/frame 0x1db02ca4 svc_thread_start(1b3850c0,1db02ce8,17ec59f,419,0,...) at svc_thread_start+0xd/frame 0x1db02cac fork_exit(13de740,1b3850c0,1db02ce8,3b,3b,...) at fork_exit+0x7e/frame 0x1db02cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1db02cd4 --- trap 0x20, eip = 0, esp = 0x33, ebp = 0xffbfeb08 --- (null)() at 0 Tracing command nfsd pid 545 tid 100132 td 0x1baeaa80 sched_switch(1baeaa80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1efa08c0 mi_switch(104,0,1830cc7,22b,1baeaa80,...) at mi_switch+0x209/frame 0x1efa08fc sleepq_switch(1baeaa80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efa0930 sleepq_catch_signals(1efa0998,10cdcd0,1b393260,0,1b393250,...) at sleepq_catch_signals+0x360/frame 0x1efa0974 sleepq_wait_sig(1b393264,0,1efa09b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efa0998 _cv_wait_sig(1b393264,1b393250,174c903,6fa,1f2f2a28,...) at _cv_wait_sig+0x1e7/frame 0x1efa09e0 seltdwait(ffffffff,0,0,1baeaa80,0,...) at seltdwait+0xbb/frame 0x1efa0a10 kern_select(1baeaa80,7,ffbfed40,0,0,...) at kern_select+0x92a/frame 0x1efa0ba0 sys_select(1baeaa80,1baead04,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1efa0be8 syscall(1efa0ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1efa0cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1efa0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x201eb21f, esp = 0xffbfe9c4, ebp = 0xffbfe9e8 --- amdsmn_devclass() at 0x201eb21f Tracing command mountd pid 539 tid 100134 td 0x1bbf2a80 sched_switch(1bbf2a80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1efb58c0 mi_switch(104,0,1830cc7,22b,1bbf2a80,...) at mi_switch+0x209/frame 0x1efb58fc sleepq_switch(1bbf2a80,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efb5930 sleepq_catch_signals(1efb5998,10cdcd0,1b6f9520,0,1b6f9510,...) at sleepq_catch_signals+0x360/frame 0x1efb5974 sleepq_wait_sig(1b6f9524,0,1efb59b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1efb5998 _cv_wait_sig(1b6f9524,1b6f9510,174c903,6fa,1f67d208,...) at _cv_wait_sig+0x1e7/frame 0x1efb59e0 seltdwait(ffffffff,0,0,1bbf2a80,0,...) at seltdwait+0xbb/frame 0x1efb5a10 kern_select(1bbf2a80,a,ffbfe8e0,0,0,...) at kern_select+0x92a/frame 0x1efb5ba0 sys_select(1bbf2a80,1bbf2d04,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1efb5be8 syscall(1efb5ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1efb5cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1efb5cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2020421f, esp = 0xffbfe84c, ebp = 0xffbfe870 --- amdsmn_devclass() at 0x2020421f Tracing command nfsuserd pid 532 tid 100133 td 0x1bbf3000 sched_switch(1bbf3000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1efb08a0 mi_switch(104,0,1830cc7,22b,1bbf3000,...) at mi_switch+0x209/frame 0x1efb08e0 sleepq_switch(1bbf3000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1efb0914 sleepq_catch_signals(8,17e3b45,cb,1bbf3000,1bbf3000,...) at sleepq_catch_signals+0x360/frame 0x1efb0958 sleepq_timedwait_sig(1b6f9664,0,1efb09bc,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1efb0998 _cv_timedwait_sig_sbt(1b6f9664,1b6f9650,d26830aa,ac4c,e0000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1efb09e0 seltdwait(ac4c,e0000000,1,1bbf3000,0,...) at seltdwait+0x82/frame 0x1efb0a10 kern_select(1bbf3000,4,ffbfdb80,0,0,...) at kern_select+0x92a/frame 0x1efb0ba0 sys_select(1bbf3000,1bbf3284,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1efb0be8 syscall(1efb0ce8,3b,3b,3b,ffbfdb80,...) at syscall+0x364/frame 0x1efb0cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1efb0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x201e921f, esp = 0xffbfdacc, ebp = 0xffbfdc18 --- amdsmn_devclass() at 0x201e921f Tracing command nfsuserd pid 531 tid 100114 td 0x1b395380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(18605de,0,18559a5,10b,151294f0,...) at db_trace_self_wrapper+0x2a/frame 0x15129470 kdb_backtrace(175fac9,46,151295f0,17008c6,15129524,...) at kdb_backtrace+0x2d/frame 0x151294d8 kdb_reenter(15129524,1153b8f,1dc584c,0,176929c,...) at kdb_reenter+0x2a/frame 0x151294e8 trap(151295fc,8,28,28,ffffffff,...) at trap+0xe6/frame 0x151295f0 calltrap() at PTDpde+0x4165/frame 0x151295f0 --- trap 0xc, eip = 0x16e80f9, esp = 0x1512963c, ebp = 0x15129658 --- db_trace_thread(1b395380,ffffffff,213,18712,1b395380,...) at db_trace_thread+0x29/frame 0x15129658 _db_stack_trace_all(113df70,0,ffffffff,151296f0,1e54850,...) at _db_stack_trace_all+0xe9/frame 0x151296c4 db_command(1e5567a,0,1e5526c,1e55679,151297a8,...) at db_command+0x297/frame 0x15129778 db_script_exec(20c0f78,0,15129820,15129814,1e54850,...) at db_script_exec+0x168/frame 0x151297e8 db_command(180effd,15129900,9faf5d,18223a8,0,...) at db_command+0x297/frame 0x1512989c db_command_loop(18223a8,0,151298c7,151298c6,9f4fe00,...) at db_command_loop+0x50/frame 0x151298a8 db_trap(3,0,7,0,1,...) at db_trap+0x10d/frame 0x15129900 kdb_trap(3,0,15129a48,25200000,0,...) at kdb_trap+0x11e/frame 0x15129938 trap(15129a48,8,28,28,49,...) at trap+0x3f3/frame 0x15129a3c calltrap() at PTDpde+0x4165/frame 0x15129a3c --- trap 0x3, eip = 0x113df70, esp = 0x15129a88, ebp = 0x15129a98 --- kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command watchdogd pid 494 tid 100106 td 0x1afdea80 sched_switch(1afdea80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1eea59f0 mi_switch(104,0,1830cc7,22b,1afdea80,...) at mi_switch+0x209/frame 0x1eea5a30 sleepq_switch(1afdea80,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x1eea5a64 sleepq_catch_signals(1afdea80,2,601,1eea5ae4,ac60,...) at sleepq_catch_signals+0x360/frame 0x1eea5aa8 sleepq_timedwait_sig(202fa9e,6c,ac60,c0000000,3,...) at sleepq_timedwait_sig+0x17/frame 0x1eea5ae4 _sleep(202fa9e,0,16c,1843745,54f2eaa0,...) at _sleep+0x32a/frame 0x1eea5b38 kern_clock_nanosleep(1afdea80,0,0,1eea5bc8,1eea5bc0,...) at kern_clock_nanosleep+0x1e2/frame 0x1eea5ba0 sys_nanosleep(1afdea80,1afded04,181d4b1,4,0,...) at sys_nanosleep+0x7f/frame 0x1eea5be8 syscall(1eea5ce8,3b,3b,3b,3c,...) at syscall+0x364/frame 0x1eea5cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eea5cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2018f4f3, esp = 0xffbfec64, ebp = 0xffbfec98 --- amdsmn_devclass() at 0x2018f4f3 Tracing command rpcbind pid 410 tid 100107 td 0x1b395700 sched_switch(1b395700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1eeaa8d8 mi_switch(104,0,1830cc7,22b,1b395700,...) at mi_switch+0x209/frame 0x1eeaa918 sleepq_switch(1b395700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eeaa94c sleepq_catch_signals(8,17e3b45,cb,1b395700,1b395700,...) at sleepq_catch_signals+0x360/frame 0x1eeaa990 sleepq_timedwait_sig(1b393364,0,1eeaa9f4,e0000000,1,...) at sleepq_timedwait_sig+0x17/frame 0x1eeaa9d0 _cv_timedwait_sig_sbt(1b393364,1b393350,9b9f61b2,ac66,e0000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1eeaaa18 seltdwait(ac66,e0000000,1,1b395700,174fdeb,...) at seltdwait+0x82/frame 0x1eeaaa48 kern_poll(1b395700,ffbfcdb0,8,1eeaabd0,0,...) at kern_poll+0x4d4/frame 0x1eeaabb0 sys_poll(1b395700,1b395984,181d4b1,4,1839ab1,...) at sys_poll+0x67/frame 0x1eeaabe8 syscall(1eeaace8,3b,3b,3b,40,...) at syscall+0x364/frame 0x1eeaacdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eeaacdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2019d7eb, esp = 0xffbfcbe4, ebp = 0xffbfcbf8 --- amdsmn_devclass() at 0x2019d7eb Tracing command syslogd pid 401 tid 100127 td 0x1baeb700 sched_switch(1baeb700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1ef868c0 mi_switch(104,0,1830cc7,22b,1baeb700,...) at mi_switch+0x209/frame 0x1ef868fc sleepq_switch(1baeb700,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1ef86930 sleepq_catch_signals(1ef86998,10cdcd0,1ac68560,0,1ac68550,...) at sleepq_catch_signals+0x360/frame 0x1ef86974 sleepq_wait_sig(1ac68564,0,1ef869b8,101,0,...) at sleepq_wait_sig+0x14/frame 0x1ef86998 _cv_wait_sig(1ac68564,1ac68550,174c903,6fa,1f2fcc30,...) at _cv_wait_sig+0x1e7/frame 0x1ef869e0 seltdwait(ffffffff,0,0,1baeb700,0,...) at seltdwait+0xbb/frame 0x1ef86a10 kern_select(1baeb700,a,20270000,0,0,...) at kern_select+0x92a/frame 0x1ef86ba0 sys_select(1baeb700,1baeb984,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1ef86be8 syscall(1ef86ce8,3b,3b,3b,0,...) at syscall+0x364/frame 0x1ef86cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1ef86cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2020521f, esp = 0xffbfe87c, ebp = 0xffbfe8a0 --- amdsmn_devclass() at 0x2020521f Tracing command devd pid 325 tid 100113 td 0x1bb16000 sched_switch(1bb16000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1eed48a0 mi_switch(104,0,1830cc7,22b,1bb16000,...) at mi_switch+0x209/frame 0x1eed48e0 sleepq_switch(1bb16000,0,1830cc7,1ca,0,...) at sleepq_switch+0x110/frame 0x1eed4914 sleepq_catch_signals(8,17e3b45,cb,1bb16000,1bb16000,...) at sleepq_catch_signals+0x360/frame 0x1eed4958 sleepq_timedwait_sig(1ac685a4,0,1eed49bc,c0000000,3,...) at sleepq_timedwait_sig+0x17/frame 0x1eed4998 _cv_timedwait_sig_sbt(1ac685a4,1ac68590,c28428d6,ac83,c0000000,...) at _cv_timedwait_sig_sbt+0x208/frame 0x1eed49e0 seltdwait(ac83,c0000000,3,1bb16000,0,...) at seltdwait+0x82/frame 0x1eed4a10 kern_select(1bb16000,6,ffbfcda0,0,0,...) at kern_select+0x92a/frame 0x1eed4ba0 sys_select(1bb16000,1bb16284,181d4b1,4,174fdeb,...) at sys_select+0x68/frame 0x1eed4be8 syscall(1eed4ce8,3b,3b,3b,3c,...) at syscall+0x364/frame 0x1eed4cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x1eed4cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0xe892b, esp = 0xffbfcd0c, ebp = 0xffbfee38 --- (null)() at 0xe892b Tracing command syncer pid 24 tid 100100 td 0x1b3a9000 sched_switch(1b3a9000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1516fb48 mi_switch(104,0,1830cc7,22b,1b3a9000,...) at mi_switch+0x209/frame 0x1516fb84 sleepq_switch(1b3a9000,0,1830cc7,2d6,1b3a9000,...) at sleepq_switch+0x110/frame 0x1516fbb8 sleepq_timedwait(20aa220,0,1516fc1c,0,0,...) at sleepq_timedwait+0x53/frame 0x1516fbf8 _cv_timedwait_sbt(20aa220,20aa200,fffffed8,0,0,...) at _cv_timedwait_sbt+0x208/frame 0x1516fc40 sched_sync(0,1516fce8,17ec59f,419,0,...) at sched_sync+0x6e0/frame 0x1516fcac fork_exit(11d4a70,0,1516fce8,0,0,...) at fork_exit+0x7e/frame 0x1516fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1516fcd4 --- trap 0, eip = 0, esp = 0x1516fd20, ebp = 0 --- (null)() at 0 Tracing command vnlru pid 23 tid 100099 td 0x1b3a9380 sched_switch(1b3a9380,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1516ab30 mi_switch(104,0,1830cc7,22b,1b3a9380,...) at mi_switch+0x209/frame 0x1516ab6c sleepq_switch(1b3a9380,0,1830cc7,2d6,ac4b,...) at sleepq_switch+0x110/frame 0x1516aba0 sleepq_timedwait(1b38a9fc,60,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1516abdc _sleep(1b38a9fc,20aa1cc,260,18577b5,fffffed8,...) at _sleep+0x342/frame 0x1516ac30 vnlru_proc(0,1516ace8,17ec59f,419,0,...) at vnlru_proc+0xbc/frame 0x1516acac fork_exit(11d40b0,0,1516ace8,0,0,...) at fork_exit+0x7e/frame 0x1516acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1516acd4 --- trap 0, eip = 0, esp = 0x1516ad20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100098 td 0x1b3a9700 sched_switch(1b3a9700,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x15165b68 mi_switch(104,0,1830cc7,22b,1b3a9700,...) at mi_switch+0x209/frame 0x15165ba4 sleepq_switch(1b3a9700,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x15165bd8 sleepq_timedwait(20a9bac,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x15165c14 _sleep(20a9bac,1de7080,54,177bd08,1999997c,...) at _sleep+0x342/frame 0x15165c68 buf_daemon(0,15165ce8,17ec59f,419,0,...) at buf_daemon+0xfc/frame 0x15165cac fork_exit(11b0d20,0,15165ce8,0,0,...) at fork_exit+0x7e/frame 0x15165cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15165cd4 --- trap 0, eip = 0, esp = 0x15165d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100103 td 0x9ee1a80 sched_switch(9ee1a80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1ee90b60 mi_switch(104,0,1830cc7,22b,9ee1a80,...) at mi_switch+0x209/frame 0x1ee90b9c sleepq_switch(9ee1a80,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1ee90bd0 sleepq_timedwait(1de8380,5c,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1ee90c0c _sleep(1de8380,1de82c0,25c,177f3f9,fffffed8,...) at _sleep+0x342/frame 0x1ee90c60 bufspace_daemon(1de7180,1ee90ce8,17ec59f,419,0,...) at bufspace_daemon+0x369/frame 0x1ee90cac fork_exit(11b1110,1de7180,1ee90ce8,0,0,...) at fork_exit+0x7e/frame 0x1ee90cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee90cd4 --- trap 0, eip = 0, esp = 0x1ee90d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100115 td 0x1b395000 sched_switch(1b395000,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1eedfb70 mi_switch(104,0,1830cc7,22b,1b395000,...) at mi_switch+0x209/frame 0x1eedfbac sleepq_switch(1b395000,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1eedfbe0 sleepq_timedwait(1bafc684,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eedfc1c _sleep(1bafc684,1bafc600,54,17ace9f,7fffff6c,...) at _sleep+0x342/frame 0x1eedfc70 softdep_flush(1b9fc000,1eedfce8,17ec59f,419,0,...) at softdep_flush+0x199/frame 0x1eedfcac fork_exit(1423c80,1b9fc000,1eedfce8,0,0,...) at fork_exit+0x7e/frame 0x1eedfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eedfcd4 --- trap 0, eip = 0, esp = 0x1eedfd20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100116 td 0x1bbf4a80 sched_switch(1bbf4a80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1eee4b70 mi_switch(104,0,1830cc7,22b,1bbf4a80,...) at mi_switch+0x209/frame 0x1eee4bac sleepq_switch(1bbf4a80,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1eee4be0 sleepq_timedwait(1bafc484,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eee4c1c _sleep(1bafc484,1bafc400,54,17ace9f,7fffff6c,...) at _sleep+0x342/frame 0x1eee4c70 softdep_flush(1b9fb000,1eee4ce8,17ec59f,419,0,...) at softdep_flush+0x199/frame 0x1eee4cac fork_exit(1423c80,1b9fb000,1eee4ce8,0,0,...) at fork_exit+0x7e/frame 0x1eee4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eee4cd4 --- trap 0, eip = 0, esp = 0x1eee4d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100117 td 0x1bbf4700 sched_switch(1bbf4700,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1eee9b70 mi_switch(104,0,1830cc7,22b,1bbf4700,...) at mi_switch+0x209/frame 0x1eee9bac sleepq_switch(1bbf4700,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1eee9be0 sleepq_timedwait(1bafc284,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eee9c1c _sleep(1bafc284,1bafc200,54,17ace9f,7fffff6c,...) at _sleep+0x342/frame 0x1eee9c70 softdep_flush(1bbf7000,1eee9ce8,17ec59f,419,0,...) at softdep_flush+0x199/frame 0x1eee9cac fork_exit(1423c80,1bbf7000,1eee9ce8,0,0,...) at fork_exit+0x7e/frame 0x1eee9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eee9cd4 --- trap 0, eip = 0, esp = 0x1eee9d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 22 tid 100118 td 0x1bbf4380 sched_switch(1bbf4380,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1eeeeb70 mi_switch(104,0,1830cc7,22b,0,...) at mi_switch+0x209/frame 0x1eeeebac sleepq_switch(1bbf4380,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1eeeebe0 sleepq_timedwait(1bafc084,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1eeeec1c _sleep(1bafc084,1bafc000,54,17ace9f,7fffff6c,...) at _sleep+0x342/frame 0x1eeeec70 softdep_flush(1bbf6000,1eeeece8,17ec59f,419,0,...) at softdep_flush+0x199/frame 0x1eeeecac fork_exit(1423c80,1bbf6000,1eeeece8,0,0,...) at fork_exit+0x7e/frame 0x1eeeecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1eeeecd4 --- trap 0, eip = 0, esp = 0x1eeeed20, ebp = 0 --- (null)() at 0 Tracing command vmdaemon pid 21 tid 100097 td 0x1b3a9a80 sched_switch(1b3a9a80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x15160b50 mi_switch(104,0,1830cc7,22b,1b3a9a80,...) at mi_switch+0x209/frame 0x15160b8c sleepq_switch(1b3a9a80,0,1830cc7,2b3,1b3a9a80,...) at sleepq_switch+0x110/frame 0x15160bc0 sleepq_wait(20bf738,74,177bd01,0,0,...) at sleepq_wait+0x48/frame 0x15160be4 _sleep(20bf738,20bf724,74,177bd01,0,...) at _sleep+0x3a6/frame 0x15160c38 vm_daemon(0,15160ce8,17ec59f,419,0,...) at vm_daemon+0xa6/frame 0x15160cac fork_exit(1494910,0,15160ce8,0,0,...) at fork_exit+0x7e/frame 0x15160cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15160cd4 --- trap 0, eip = 0, esp = 0x15160d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100096 td 0x1b3aa000 sched_switch(1b3aa000,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1515b978 mi_switch(104,0,1830cc7,22b,1b3aa000,...) at mi_switch+0x209/frame 0x1515b9b4 sleepq_switch(1b3aa000,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1515b9e8 sleepq_timedwait(20e9ddc,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1515ba24 _sleep(20e9ddc,20e9b00,254,177bd01,1999997c,...) at _sleep+0x342/frame 0x1515ba78 vm_pageout_worker(0,0,1b38f6a8,0,0,...) at vm_pageout_worker+0x2c7/frame 0x1515bc7c vm_pageout(0,1515bce8,17ec59f,419,0,...) at vm_pageout+0x16a/frame 0x1515bcac fork_exit(1488e20,0,1515bce8,0,0,...) at fork_exit+0x7e/frame 0x1515bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1515bcd4 --- trap 0, eip = 0, esp = 0x1515bd20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100101 td 0x9f53380 sched_switch(9f53380,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1ee86a10 mi_switch(104,0,1830cc7,22b,9f53380,...) at mi_switch+0x209/frame 0x1ee86a4c sleepq_switch(9f53380,0,1830cc7,2b3,9f53380,...) at sleepq_switch+0x110/frame 0x1ee86a80 sleepq_wait(20e9de8,54,186c946,0,0,...) at sleepq_wait+0x48/frame 0x1ee86aa4 _sleep(20e9de8,20e9a40,54,186c946,0,...) at _sleep+0x3a6/frame 0x1ee86af8 vm_pageout_laundry_worker(0,1ee86ce8,17ec59f,419,0,...) at vm_pageout_laundry_worker+0x107b/frame 0x1ee86cac fork_exit(1488ff0,0,1ee86ce8,0,0,...) at fork_exit+0x7e/frame 0x1ee86cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee86cd4 --- trap 0, eip = 0, esp = 0x1ee86d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 20 tid 100102 td 0x9f53000 sched_switch(9f53000,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1ee8bb70 mi_switch(104,0,1830cc7,22b,9f53000,...) at mi_switch+0x209/frame 0x1ee8bbac sleepq_switch(9f53000,0,1830cc7,2d6,204e804,...) at sleepq_switch+0x110/frame 0x1ee8bbe0 sleepq_timedwait(14642a0,54,1ee8bc50,0,0,...) at sleepq_timedwait+0x53/frame 0x1ee8bc20 _sleep(14642a0,20bf3ac,54,17dc1c6,fffffed8,...) at _sleep+0x342/frame 0x1ee8bc74 uma_reclaim_worker(0,1ee8bce8,17ec59f,419,0,...) at uma_reclaim_worker+0x117/frame 0x1ee8bcac fork_exit(14647d0,0,1ee8bce8,0,0,...) at fork_exit+0x7e/frame 0x1ee8bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ee8bcd4 --- trap 0, eip = 0, esp = 0x1ee8bd20, ebp = 0 --- (null)() at 0 Tracing command rand_harvestq pid 19 tid 100090 td 0x9f54a80 sched_switch(9f54a80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1513db48 mi_switch(104,0,1830cc7,22b,9f54a80,...) at mi_switch+0x209/frame 0x1513db84 sleepq_switch(9f54a80,0,1830cc7,2d6,ac4a,...) at sleepq_switch+0x110/frame 0x1513dbb8 sleepq_timedwait(1f05894,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0x1513dbf4 _sleep(1f05894,0,0,177f3f9,19999999,...) at _sleep+0x342/frame 0x1513dc48 random_kthread(0,1513dce8,17ec59f,419,0,...) at random_kthread+0x7c/frame 0x1513dcac fork_exit(de4790,0,1513dce8,0,0,...) at fork_exit+0x7e/frame 0x1513dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1513dcd4 --- trap 0, eip = 0, esp = 0x1513dd20, ebp = 0 --- (null)() at 0 Tracing command sctp_iterator pid 18 tid 100089 td 0x9f55000 sched_switch(9f55000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x15134b98 mi_switch(104,0,1830cc7,22b,9f55000,...) at mi_switch+0x209/frame 0x15134bd4 sleepq_switch(9f55000,0,1830cc7,2b3,9f55000,...) at sleepq_switch+0x110/frame 0x15134c08 sleepq_wait(20e8ec4,0,1773ca8,0,0,...) at sleepq_wait+0x48/frame 0x15134c2c _sleep(20e8ec4,20e8e8c,0,1773ca8,0,...) at _sleep+0x3a6/frame 0x15134c80 sctp_iterator_thread(0,15134ce8,17ec59f,419,0,...) at sctp_iterator_thread+0x7c/frame 0x15134cac fork_exit(12aac10,0,15134ce8,0,0,...) at fork_exit+0x7e/frame 0x15134cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15134cd4 --- trap 0, eip = 0, esp = 0x15134d20, ebp = 0 --- (null)() at 0 Tracing command soaiod4 pid 17 tid 100088 td 0x1b3aa380 sched_switch(1b3aa380,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1512eb70 mi_switch(104,0,1830cc7,22b,1b3aa380,...) at mi_switch+0x209/frame 0x1512ebac sleepq_switch(1b3aa380,0,1830cc7,2d6,ac5e,...) at sleepq_switch+0x110/frame 0x1512ebe0 sleepq_timedwait(20a97d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1512ec1c _sleep(20a97d8,20a97b8,0,177f3f9,ffffdd50,...) at _sleep+0x342/frame 0x1512ec70 soaio_kproc_loop(4,1512ece8,17ec59f,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1512ecac fork_exit(116f390,4,1512ece8,0,0,...) at fork_exit+0x7e/frame 0x1512ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1512ecd4 --- trap 0, eip = 0, esp = 0x1512ed20, ebp = 0 --- (null)() at 0 Tracing command soaiod3 pid 16 tid 100087 td 0x1b3aa700 sched_switch(1b3aa700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1abffb70 mi_switch(104,0,1830cc7,22b,1b3aa700,...) at mi_switch+0x209/frame 0x1abffbac sleepq_switch(1b3aa700,0,1830cc7,2d6,ac5d,...) at sleepq_switch+0x110/frame 0x1abffbe0 sleepq_timedwait(20a97d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abffc1c _sleep(20a97d8,20a97b8,0,177f3f9,ffffdd50,...) at _sleep+0x342/frame 0x1abffc70 soaio_kproc_loop(3,1abffce8,17ec59f,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abffcac fork_exit(116f390,3,1abffce8,0,0,...) at fork_exit+0x7e/frame 0x1abffcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abffcd4 --- trap 0, eip = 0, esp = 0x1abffd20, ebp = 0 --- (null)() at 0 Tracing command soaiod2 pid 15 tid 100086 td 0x1b3aaa80 sched_switch(1b3aaa80,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1abfab70 mi_switch(104,0,1830cc7,22b,1b3aaa80,...) at mi_switch+0x209/frame 0x1abfabac sleepq_switch(1b3aaa80,0,1830cc7,2d6,ac5f,...) at sleepq_switch+0x110/frame 0x1abfabe0 sleepq_timedwait(20a97d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abfac1c _sleep(20a97d8,20a97b8,0,177f3f9,ffffdd50,...) at _sleep+0x342/frame 0x1abfac70 soaio_kproc_loop(2,1abface8,17ec59f,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abfacac fork_exit(116f390,2,1abface8,0,0,...) at fork_exit+0x7e/frame 0x1abfacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abfacd4 --- trap 0, eip = 0, esp = 0x1abfad20, ebp = 0 --- (null)() at 0 Tracing command soaiod1 pid 9 tid 100085 td 0x1b3ab000 sched_switch(1b3ab000,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1abf5b70 mi_switch(104,0,1830cc7,22b,1b3ab000,...) at mi_switch+0x209/frame 0x1abf5bac sleepq_switch(1b3ab000,0,1830cc7,2d6,ac5d,...) at sleepq_switch+0x110/frame 0x1abf5be0 sleepq_timedwait(20a97d8,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0x1abf5c1c _sleep(20a97d8,20a97b8,0,177f3f9,ffffdd50,...) at _sleep+0x342/frame 0x1abf5c70 soaio_kproc_loop(1,1abf5ce8,17ec59f,419,0,...) at soaio_kproc_loop+0x1a7/frame 0x1abf5cac fork_exit(116f390,1,1abf5ce8,0,0,...) at fork_exit+0x7e/frame 0x1abf5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abf5cd4 --- trap 0, eip = 0, esp = 0x1abf5d20, ebp = 0 --- (null)() at 0 Tracing command fdc0 pid 8 tid 100082 td 0x1afdfa80 sched_switch(1afdfa80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1abe6b40 mi_switch(104,0,1830cc7,22b,1afdfa80,...) at mi_switch+0x209/frame 0x1abe6b7c sleepq_switch(1afdfa80,0,1830cc7,2b3,1afdfa80,...) at sleepq_switch+0x110/frame 0x1abe6bb0 sleepq_wait(1ac5d23c,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1abe6bd4 _sleep(1ac5d23c,1ac5d2f8,5c,177f3f9,0,...) at _sleep+0x3a6/frame 0x1abe6c28 fdc_thread(1ac5d200,1abe6ce8,17ec59f,419,0,...) at fdc_thread+0x336/frame 0x1abe6cac fork_exit(166a310,1ac5d200,1abe6ce8,0,0,...) at fork_exit+0x7e/frame 0x1abe6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abe6cd4 --- trap 0, eip = 0, esp = 0x1abe6d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100047 td 0x1acb6000 sched_switch(1acb6000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab2eba8 mi_switch(104,0,1830cc7,22b,1acb6000,...) at mi_switch+0x209/frame 0x1ab2ebe8 sleepq_switch(1acb6000,0,1830cc7,2b3,1ac8dcf8,...) at sleepq_switch+0x110/frame 0x1ab2ec1c sleepq_wait(1ac8dacc,0,1ab2ec60,1,0,...) at sleepq_wait+0x48/frame 0x1ab2ec40 _cv_wait(1ac8dacc,1ac8dcf8,1812b54,78,1ac8dad4,...) at _cv_wait+0x1d7/frame 0x1ab2ec84 usb_process(1ac8dac4,1ab2ece8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab2ecac fork_exit(f165a0,1ac8dac4,1ab2ece8,0,0,...) at fork_exit+0x7e/frame 0x1ab2ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab2ecd4 --- trap 0, eip = 0, esp = 0x1ab2ed20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100048 td 0x1acb5a80 sched_switch(1acb5a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab33ba8 mi_switch(104,0,1830cc7,22b,1acb5a80,...) at mi_switch+0x209/frame 0x1ab33be8 sleepq_switch(1acb5a80,0,1830cc7,2b3,1ac8dcf8,...) at sleepq_switch+0x110/frame 0x1ab33c1c sleepq_wait(1ac8dafc,0,1ab33c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab33c40 _cv_wait(1ac8dafc,1ac8dcf8,1812b54,78,1ac8db04,...) at _cv_wait+0x1d7/frame 0x1ab33c84 usb_process(1ac8daf4,1ab33ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab33cac fork_exit(f165a0,1ac8daf4,1ab33ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab33cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab33cd4 --- trap 0, eip = 0, esp = 0x1ab33d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100049 td 0x1acb5700 sched_switch(1acb5700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab38ba8 mi_switch(104,0,1830cc7,22b,1acb5700,...) at mi_switch+0x209/frame 0x1ab38be8 sleepq_switch(1acb5700,0,1830cc7,2b3,1ac8dcf8,...) at sleepq_switch+0x110/frame 0x1ab38c1c sleepq_wait(1ac8db2c,0,1ab38c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab38c40 _cv_wait(1ac8db2c,1ac8dcf8,1812b54,78,1ac8db34,...) at _cv_wait+0x1d7/frame 0x1ab38c84 usb_process(1ac8db24,1ab38ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab38cac fork_exit(f165a0,1ac8db24,1ab38ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab38cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab38cd4 --- trap 0, eip = 0, esp = 0x1ab38d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100050 td 0x1acb5380 sched_switch(1acb5380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab3dba8 mi_switch(104,0,1830cc7,22b,1acb5380,...) at mi_switch+0x209/frame 0x1ab3dbe8 sleepq_switch(1acb5380,0,1830cc7,2b3,1ac8dcf8,...) at sleepq_switch+0x110/frame 0x1ab3dc1c sleepq_wait(1ac8db5c,0,1ab3dc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab3dc40 _cv_wait(1ac8db5c,1ac8dcf8,1812b54,78,1ac8db64,...) at _cv_wait+0x1d7/frame 0x1ab3dc84 usb_process(1ac8db54,1ab3dce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab3dcac fork_exit(f165a0,1ac8db54,1ab3dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab3dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab3dcd4 --- trap 0, eip = 0, esp = 0x1ab3dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100051 td 0x1acb5000 sched_switch(1acb5000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab42ba8 mi_switch(104,0,1830cc7,22b,1acb5000,...) at mi_switch+0x209/frame 0x1ab42be8 sleepq_switch(1acb5000,0,1830cc7,2b3,1ac8dcf8,...) at sleepq_switch+0x110/frame 0x1ab42c1c sleepq_wait(1ac8db8c,0,1ab42c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab42c40 _cv_wait(1ac8db8c,1ac8dcf8,1812b54,78,1ac8db94,...) at _cv_wait+0x1d7/frame 0x1ab42c84 usb_process(1ac8db84,1ab42ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab42cac fork_exit(f165a0,1ac8db84,1ab42ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab42cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab42cd4 --- trap 0, eip = 0, esp = 0x1ab42d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100053 td 0x1aca4700 sched_switch(1aca4700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab4dba8 mi_switch(104,0,1830cc7,22b,1aca4700,...) at mi_switch+0x209/frame 0x1ab4dbe8 sleepq_switch(1aca4700,0,1830cc7,2b3,1af63cf8,...) at sleepq_switch+0x110/frame 0x1ab4dc1c sleepq_wait(1af63acc,0,1ab4dc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab4dc40 _cv_wait(1af63acc,1af63cf8,1812b54,78,1af63ad4,...) at _cv_wait+0x1d7/frame 0x1ab4dc84 usb_process(1af63ac4,1ab4dce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab4dcac fork_exit(f165a0,1af63ac4,1ab4dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab4dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab4dcd4 --- trap 0, eip = 0, esp = 0x1ab4dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100054 td 0x1aca4380 sched_switch(1aca4380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab52ba8 mi_switch(104,0,1830cc7,22b,1aca4380,...) at mi_switch+0x209/frame 0x1ab52be8 sleepq_switch(1aca4380,0,1830cc7,2b3,1af63cf8,...) at sleepq_switch+0x110/frame 0x1ab52c1c sleepq_wait(1af63afc,0,1ab52c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab52c40 _cv_wait(1af63afc,1af63cf8,1812b54,78,1af63b04,...) at _cv_wait+0x1d7/frame 0x1ab52c84 usb_process(1af63af4,1ab52ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab52cac fork_exit(f165a0,1af63af4,1ab52ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab52cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab52cd4 --- trap 0, eip = 0, esp = 0x1ab52d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100055 td 0x1aca4000 sched_switch(1aca4000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab57ba8 mi_switch(104,0,1830cc7,22b,1aca4000,...) at mi_switch+0x209/frame 0x1ab57be8 sleepq_switch(1aca4000,0,1830cc7,2b3,1af63cf8,...) at sleepq_switch+0x110/frame 0x1ab57c1c sleepq_wait(1af63b2c,0,1ab57c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab57c40 _cv_wait(1af63b2c,1af63cf8,1812b54,78,1af63b34,...) at _cv_wait+0x1d7/frame 0x1ab57c84 usb_process(1af63b24,1ab57ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab57cac fork_exit(f165a0,1af63b24,1ab57ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab57cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab57cd4 --- trap 0, eip = 0, esp = 0x1ab57d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100056 td 0x1af6d380 sched_switch(1af6d380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab5cba8 mi_switch(104,0,1830cc7,22b,1af6d380,...) at mi_switch+0x209/frame 0x1ab5cbe8 sleepq_switch(1af6d380,0,1830cc7,2b3,1af63cf8,...) at sleepq_switch+0x110/frame 0x1ab5cc1c sleepq_wait(1af63b5c,0,1ab5cc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab5cc40 _cv_wait(1af63b5c,1af63cf8,1812b54,78,1af63b64,...) at _cv_wait+0x1d7/frame 0x1ab5cc84 usb_process(1af63b54,1ab5cce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab5ccac fork_exit(f165a0,1af63b54,1ab5cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab5ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab5ccd4 --- trap 0, eip = 0, esp = 0x1ab5cd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100057 td 0x1af6d000 sched_switch(1af6d000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab61ba8 mi_switch(104,0,1830cc7,22b,1af6d000,...) at mi_switch+0x209/frame 0x1ab61be8 sleepq_switch(1af6d000,0,1830cc7,2b3,1af63cf8,...) at sleepq_switch+0x110/frame 0x1ab61c1c sleepq_wait(1af63b8c,0,1ab61c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab61c40 _cv_wait(1af63b8c,1af63cf8,1812b54,78,1af63b94,...) at _cv_wait+0x1d7/frame 0x1ab61c84 usb_process(1af63b84,1ab61ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab61cac fork_exit(f165a0,1af63b84,1ab61ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab61cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab61cd4 --- trap 0, eip = 0, esp = 0x1ab61d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100059 td 0x1af6c700 sched_switch(1af6c700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab6cba8 mi_switch(104,0,1830cc7,22b,1af6c700,...) at mi_switch+0x209/frame 0x1ab6cbe8 sleepq_switch(1af6c700,0,1830cc7,2b3,1af6ecf8,...) at sleepq_switch+0x110/frame 0x1ab6cc1c sleepq_wait(1af6eacc,0,1ab6cc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab6cc40 _cv_wait(1af6eacc,1af6ecf8,1812b54,78,1af6ead4,...) at _cv_wait+0x1d7/frame 0x1ab6cc84 usb_process(1af6eac4,1ab6cce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab6ccac fork_exit(f165a0,1af6eac4,1ab6cce8,0,0,...) at fork_exit+0x7e/frame 0x1ab6ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab6ccd4 --- trap 0, eip = 0, esp = 0x1ab6cd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100060 td 0x1af6c380 sched_switch(1af6c380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab71ba8 mi_switch(104,0,1830cc7,22b,1af6c380,...) at mi_switch+0x209/frame 0x1ab71be8 sleepq_switch(1af6c380,0,1830cc7,2b3,1af6ecf8,...) at sleepq_switch+0x110/frame 0x1ab71c1c sleepq_wait(1af6eafc,0,1ab71c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab71c40 _cv_wait(1af6eafc,1af6ecf8,1812b54,78,1af6eb04,...) at _cv_wait+0x1d7/frame 0x1ab71c84 usb_process(1af6eaf4,1ab71ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab71cac fork_exit(f165a0,1af6eaf4,1ab71ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab71cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab71cd4 --- trap 0, eip = 0, esp = 0x1ab71d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100061 td 0x1af6c000 sched_switch(1af6c000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab76ba8 mi_switch(104,0,1830cc7,22b,1af6c000,...) at mi_switch+0x209/frame 0x1ab76be8 sleepq_switch(1af6c000,0,1830cc7,2b3,1af6ecf8,...) at sleepq_switch+0x110/frame 0x1ab76c1c sleepq_wait(1af6eb2c,0,1ab76c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab76c40 _cv_wait(1af6eb2c,1af6ecf8,1812b54,78,1af6eb34,...) at _cv_wait+0x1d7/frame 0x1ab76c84 usb_process(1af6eb24,1ab76ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab76cac fork_exit(f165a0,1af6eb24,1ab76ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab76cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab76cd4 --- trap 0, eip = 0, esp = 0x1ab76d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100062 td 0x1af6ba80 sched_switch(1af6ba80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab7bba8 mi_switch(104,0,1830cc7,22b,1af6ba80,...) at mi_switch+0x209/frame 0x1ab7bbe8 sleepq_switch(1af6ba80,0,1830cc7,2b3,1af6ecf8,...) at sleepq_switch+0x110/frame 0x1ab7bc1c sleepq_wait(1af6eb5c,0,1ab7bc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab7bc40 _cv_wait(1af6eb5c,1af6ecf8,1812b54,78,1af6eb64,...) at _cv_wait+0x1d7/frame 0x1ab7bc84 usb_process(1af6eb54,1ab7bce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab7bcac fork_exit(f165a0,1af6eb54,1ab7bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab7bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab7bcd4 --- trap 0, eip = 0, esp = 0x1ab7bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100063 td 0x1af6b700 sched_switch(1af6b700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab80ba8 mi_switch(104,0,1830cc7,22b,1af6b700,...) at mi_switch+0x209/frame 0x1ab80be8 sleepq_switch(1af6b700,0,1830cc7,2b3,1af6ecf8,...) at sleepq_switch+0x110/frame 0x1ab80c1c sleepq_wait(1af6eb8c,0,1ab80c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab80c40 _cv_wait(1af6eb8c,1af6ecf8,1812b54,78,1af6eb94,...) at _cv_wait+0x1d7/frame 0x1ab80c84 usb_process(1af6eb84,1ab80ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab80cac fork_exit(f165a0,1af6eb84,1ab80ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab80cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab80cd4 --- trap 0, eip = 0, esp = 0x1ab80d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100064 td 0x1af6b380 sched_switch(1af6b380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab86ba8 mi_switch(104,0,1830cc7,22b,1af6b380,...) at mi_switch+0x209/frame 0x1ab86be8 sleepq_switch(1af6b380,0,1830cc7,2b3,1af81cf8,...) at sleepq_switch+0x110/frame 0x1ab86c1c sleepq_wait(1af81acc,0,1ab86c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab86c40 _cv_wait(1af81acc,1af81cf8,1812b54,78,1af81ad4,...) at _cv_wait+0x1d7/frame 0x1ab86c84 usb_process(1af81ac4,1ab86ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab86cac fork_exit(f165a0,1af81ac4,1ab86ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab86cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab86cd4 --- trap 0, eip = 0, esp = 0x1ab86d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100065 td 0x1af6b000 sched_switch(1af6b000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab8bba8 mi_switch(104,0,1830cc7,22b,1af6b000,...) at mi_switch+0x209/frame 0x1ab8bbe8 sleepq_switch(1af6b000,0,1830cc7,2b3,1af81cf8,...) at sleepq_switch+0x110/frame 0x1ab8bc1c sleepq_wait(1af81afc,0,1ab8bc60,1,0,...) at sleepq_wait+0x48/frame 0x1ab8bc40 _cv_wait(1af81afc,1af81cf8,1812b54,78,1af81b04,...) at _cv_wait+0x1d7/frame 0x1ab8bc84 usb_process(1af81af4,1ab8bce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab8bcac fork_exit(f165a0,1af81af4,1ab8bce8,0,0,...) at fork_exit+0x7e/frame 0x1ab8bcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab8bcd4 --- trap 0, eip = 0, esp = 0x1ab8bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100066 td 0x1acb7a80 sched_switch(1acb7a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab90ba8 mi_switch(104,0,1830cc7,22b,1acb7a80,...) at mi_switch+0x209/frame 0x1ab90be8 sleepq_switch(1acb7a80,0,1830cc7,2b3,1af81cf8,...) at sleepq_switch+0x110/frame 0x1ab90c1c sleepq_wait(1af81b2c,0,1ab90c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab90c40 _cv_wait(1af81b2c,1af81cf8,1812b54,78,1af81b34,...) at _cv_wait+0x1d7/frame 0x1ab90c84 usb_process(1af81b24,1ab90ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab90cac fork_exit(f165a0,1af81b24,1ab90ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab90cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab90cd4 --- trap 0, eip = 0, esp = 0x1ab90d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100067 td 0x1acb7700 sched_switch(1acb7700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab95ba8 mi_switch(104,0,1830cc7,22b,1acb7700,...) at mi_switch+0x209/frame 0x1ab95be8 sleepq_switch(1acb7700,0,1830cc7,2b3,1af81cf8,...) at sleepq_switch+0x110/frame 0x1ab95c1c sleepq_wait(1af81b5c,0,1ab95c60,1,0,...) at sleepq_wait+0x48/frame 0x1ab95c40 _cv_wait(1af81b5c,1af81cf8,1812b54,78,1af81b64,...) at _cv_wait+0x1d7/frame 0x1ab95c84 usb_process(1af81b54,1ab95ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab95cac fork_exit(f165a0,1af81b54,1ab95ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab95cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab95cd4 --- trap 0, eip = 0, esp = 0x1ab95d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100068 td 0x1acb7380 sched_switch(1acb7380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1ab9aba8 mi_switch(104,0,1830cc7,22b,1acb7380,...) at mi_switch+0x209/frame 0x1ab9abe8 sleepq_switch(1acb7380,0,1830cc7,2b3,1af81cf8,...) at sleepq_switch+0x110/frame 0x1ab9ac1c sleepq_wait(1af81b8c,0,1ab9ac60,1,0,...) at sleepq_wait+0x48/frame 0x1ab9ac40 _cv_wait(1af81b8c,1af81cf8,1812b54,78,1af81b94,...) at _cv_wait+0x1d7/frame 0x1ab9ac84 usb_process(1af81b84,1ab9ace8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1ab9acac fork_exit(f165a0,1af81b84,1ab9ace8,0,0,...) at fork_exit+0x7e/frame 0x1ab9acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab9acd4 --- trap 0, eip = 0, esp = 0x1ab9ad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100069 td 0x1af8e700 sched_switch(1af8e700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aba0ba8 mi_switch(104,0,1830cc7,22b,1af8e700,...) at mi_switch+0x209/frame 0x1aba0be8 sleepq_switch(1af8e700,0,1830cc7,2b3,1af88cf8,...) at sleepq_switch+0x110/frame 0x1aba0c1c sleepq_wait(1af88acc,0,1aba0c60,1,0,...) at sleepq_wait+0x48/frame 0x1aba0c40 _cv_wait(1af88acc,1af88cf8,1812b54,78,1af88ad4,...) at _cv_wait+0x1d7/frame 0x1aba0c84 usb_process(1af88ac4,1aba0ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1aba0cac fork_exit(f165a0,1af88ac4,1aba0ce8,0,0,...) at fork_exit+0x7e/frame 0x1aba0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aba0cd4 --- trap 0, eip = 0, esp = 0x1aba0d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100070 td 0x1af8e380 sched_switch(1af8e380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aba5ba8 mi_switch(104,0,1830cc7,22b,1af8e380,...) at mi_switch+0x209/frame 0x1aba5be8 sleepq_switch(1af8e380,0,1830cc7,2b3,1af88cf8,...) at sleepq_switch+0x110/frame 0x1aba5c1c sleepq_wait(1af88afc,0,1aba5c60,1,0,...) at sleepq_wait+0x48/frame 0x1aba5c40 _cv_wait(1af88afc,1af88cf8,1812b54,78,1af88b04,...) at _cv_wait+0x1d7/frame 0x1aba5c84 usb_process(1af88af4,1aba5ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1aba5cac fork_exit(f165a0,1af88af4,1aba5ce8,0,0,...) at fork_exit+0x7e/frame 0x1aba5cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aba5cd4 --- trap 0, eip = 0, esp = 0x1aba5d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100071 td 0x1af8e000 sched_switch(1af8e000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abaaba8 mi_switch(104,0,1830cc7,22b,1af8e000,...) at mi_switch+0x209/frame 0x1abaabe8 sleepq_switch(1af8e000,0,1830cc7,2b3,1af88cf8,...) at sleepq_switch+0x110/frame 0x1abaac1c sleepq_wait(1af88b2c,0,1abaac60,1,0,...) at sleepq_wait+0x48/frame 0x1abaac40 _cv_wait(1af88b2c,1af88cf8,1812b54,78,1af88b34,...) at _cv_wait+0x1d7/frame 0x1abaac84 usb_process(1af88b24,1abaace8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abaacac fork_exit(f165a0,1af88b24,1abaace8,0,0,...) at fork_exit+0x7e/frame 0x1abaacd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abaacd4 --- trap 0, eip = 0, esp = 0x1abaad20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100072 td 0x1af8da80 sched_switch(1af8da80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abafba8 mi_switch(104,0,1830cc7,22b,1af8da80,...) at mi_switch+0x209/frame 0x1abafbe8 sleepq_switch(1af8da80,0,1830cc7,2b3,1af88cf8,...) at sleepq_switch+0x110/frame 0x1abafc1c sleepq_wait(1af88b5c,0,1abafc60,1,0,...) at sleepq_wait+0x48/frame 0x1abafc40 _cv_wait(1af88b5c,1af88cf8,1812b54,78,1af88b64,...) at _cv_wait+0x1d7/frame 0x1abafc84 usb_process(1af88b54,1abafce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abafcac fork_exit(f165a0,1af88b54,1abafce8,0,0,...) at fork_exit+0x7e/frame 0x1abafcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abafcd4 --- trap 0, eip = 0, esp = 0x1abafd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100073 td 0x1af8d700 sched_switch(1af8d700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abb4ba8 mi_switch(104,0,1830cc7,22b,1af8d700,...) at mi_switch+0x209/frame 0x1abb4be8 sleepq_switch(1af8d700,0,1830cc7,2b3,1af88cf8,...) at sleepq_switch+0x110/frame 0x1abb4c1c sleepq_wait(1af88b8c,0,1abb4c60,1,0,...) at sleepq_wait+0x48/frame 0x1abb4c40 _cv_wait(1af88b8c,1af88cf8,1812b54,78,1af88b94,...) at _cv_wait+0x1d7/frame 0x1abb4c84 usb_process(1af88b84,1abb4ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abb4cac fork_exit(f165a0,1af88b84,1abb4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abb4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abb4cd4 --- trap 0, eip = 0, esp = 0x1abb4d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100075 td 0x1af8d000 sched_switch(1af8d000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abbfba8 mi_switch(104,0,1830cc7,22b,1af8d000,...) at mi_switch+0x209/frame 0x1abbfbe8 sleepq_switch(1af8d000,0,1830cc7,2b3,1ac78ed0,...) at sleepq_switch+0x110/frame 0x1abbfc1c sleepq_wait(1ac78ca4,0,1abbfc60,1,0,...) at sleepq_wait+0x48/frame 0x1abbfc40 _cv_wait(1ac78ca4,1ac78ed0,1812b54,78,1ac78cac,...) at _cv_wait+0x1d7/frame 0x1abbfc84 usb_process(1ac78c9c,1abbfce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abbfcac fork_exit(f165a0,1ac78c9c,1abbfce8,0,0,...) at fork_exit+0x7e/frame 0x1abbfcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abbfcd4 --- trap 0, eip = 0, esp = 0x1abbfd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100076 td 0x1af8ca80 sched_switch(1af8ca80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abc4ba8 mi_switch(104,0,1830cc7,22b,1af8ca80,...) at mi_switch+0x209/frame 0x1abc4be8 sleepq_switch(1af8ca80,0,1830cc7,2b3,1ac78ed0,...) at sleepq_switch+0x110/frame 0x1abc4c1c sleepq_wait(1ac78cd4,0,1abc4c60,1,0,...) at sleepq_wait+0x48/frame 0x1abc4c40 _cv_wait(1ac78cd4,1ac78ed0,1812b54,78,1ac78cdc,...) at _cv_wait+0x1d7/frame 0x1abc4c84 usb_process(1ac78ccc,1abc4ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abc4cac fork_exit(f165a0,1ac78ccc,1abc4ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc4cd4 --- trap 0, eip = 0, esp = 0x1abc4d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100077 td 0x1af8c700 sched_switch(1af8c700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abc9ba8 mi_switch(104,0,1830cc7,22b,1af8c700,...) at mi_switch+0x209/frame 0x1abc9be8 sleepq_switch(1af8c700,0,1830cc7,2b3,1ac78ed0,...) at sleepq_switch+0x110/frame 0x1abc9c1c sleepq_wait(1ac78d04,0,1abc9c60,1,0,...) at sleepq_wait+0x48/frame 0x1abc9c40 _cv_wait(1ac78d04,1ac78ed0,1812b54,78,1ac78d0c,...) at _cv_wait+0x1d7/frame 0x1abc9c84 usb_process(1ac78cfc,1abc9ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abc9cac fork_exit(f165a0,1ac78cfc,1abc9ce8,0,0,...) at fork_exit+0x7e/frame 0x1abc9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abc9cd4 --- trap 0, eip = 0, esp = 0x1abc9d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100078 td 0x1af8c380 sched_switch(1af8c380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abceba8 mi_switch(104,0,1830cc7,22b,1af8c380,...) at mi_switch+0x209/frame 0x1abcebe8 sleepq_switch(1af8c380,0,1830cc7,2b3,1ac78ed0,...) at sleepq_switch+0x110/frame 0x1abcec1c sleepq_wait(1ac78d34,0,1abcec60,1,0,...) at sleepq_wait+0x48/frame 0x1abcec40 _cv_wait(1ac78d34,1ac78ed0,1812b54,78,1ac78d3c,...) at _cv_wait+0x1d7/frame 0x1abcec84 usb_process(1ac78d2c,1abcece8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abcecac fork_exit(f165a0,1ac78d2c,1abcece8,0,0,...) at fork_exit+0x7e/frame 0x1abcecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abcecd4 --- trap 0, eip = 0, esp = 0x1abced20, ebp = 0 --- (null)() at 0 Tracing command usb pid 14 tid 100079 td 0x1af8c000 sched_switch(1af8c000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abd3ba8 mi_switch(104,0,1830cc7,22b,1af8c000,...) at mi_switch+0x209/frame 0x1abd3be8 sleepq_switch(1af8c000,0,1830cc7,2b3,1ac78ed0,...) at sleepq_switch+0x110/frame 0x1abd3c1c sleepq_wait(1ac78d64,0,1abd3c60,1,0,...) at sleepq_wait+0x48/frame 0x1abd3c40 _cv_wait(1ac78d64,1ac78ed0,1812b54,78,1ac78d6c,...) at _cv_wait+0x1d7/frame 0x1abd3c84 usb_process(1ac78d5c,1abd3ce8,17ec59f,419,0,...) at usb_process+0x140/frame 0x1abd3cac fork_exit(f165a0,1ac78d5c,1abd3ce8,0,0,...) at fork_exit+0x7e/frame 0x1abd3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abd3cd4 --- trap 0, eip = 0, esp = 0x1abd3d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100042 td 0x9f55380 sched_switch(9f55380,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1aae0b78 mi_switch(104,0,1830cc7,22b,9f55380,...) at mi_switch+0x209/frame 0x1aae0bb4 sleepq_switch(9f55380,0,1830cc7,2b3,9f55380,...) at sleepq_switch+0x110/frame 0x1aae0be8 sleepq_wait(1e52c40,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aae0c0c _sleep(1e52c40,1e52c00,5c,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aae0c60 xpt_done_td(1e52c00,1aae0ce8,17ec59f,419,0,...) at xpt_done_td+0x9b/frame 0x1aae0cac fork_exit(90bc20,1e52c00,1aae0ce8,0,0,...) at fork_exit+0x7e/frame 0x1aae0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aae0cd4 --- trap 0, eip = 0, esp = 0x1aae0d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100095 td 0x9f53700 sched_switch(9f53700,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x15156b70 mi_switch(104,0,1830cc7,22b,9f53700,...) at mi_switch+0x209/frame 0x15156bac sleepq_switch(9f53700,0,1830cc7,2b3,9f53700,...) at sleepq_switch+0x110/frame 0x15156be0 sleepq_wait(1e52b6c,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x15156c04 _sleep(1e52b6c,1e52bc0,5c,177f3f9,0,...) at _sleep+0x3a6/frame 0x15156c58 xpt_scanner_thread(0,15156ce8,17ec59f,419,0,...) at xpt_scanner_thread+0x97/frame 0x15156cac fork_exit(90bfa0,0,15156ce8,0,0,...) at fork_exit+0x7e/frame 0x15156cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15156cd4 --- trap 0, eip = 0, esp = 0x15156d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 3 pid 6 tid 100041 td 0x9f55700 sched_switch(9f55700,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1aadbb70 mi_switch(104,0,1830cc7,22b,9f55700,...) at mi_switch+0x209/frame 0x1aadbbac sleepq_switch(9f55700,0,1830cc7,2b3,9f55700,...) at sleepq_switch+0x110/frame 0x1aadbbe0 sleepq_wait(9f500c4,6c,186211d,0,0,...) at sleepq_wait+0x48/frame 0x1aadbc04 _sleep(9f500c4,9f500a8,6c,186211d,0,...) at _sleep+0x3a6/frame 0x1aadbc58 crypto_ret_proc(9f500a8,1aadbce8,17ec59f,419,0,...) at crypto_ret_proc+0x241/frame 0x1aadbcac fork_exit(13c8d70,9f500a8,1aadbce8,0,0,...) at fork_exit+0x7e/frame 0x1aadbcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aadbcd4 --- trap 0, eip = 0, esp = 0x1aadbd20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 2 pid 5 tid 100040 td 0x9f55a80 sched_switch(9f55a80,0,104,18a,203d380,...) at sched_switch+0x80c/frame 0x1aad6b70 mi_switch(104,0,1830cc7,22b,9f55a80,...) at mi_switch+0x209/frame 0x1aad6bac sleepq_switch(9f55a80,0,1830cc7,2b3,9f55a80,...) at sleepq_switch+0x110/frame 0x1aad6be0 sleepq_wait(9f5008c,6c,186211d,0,0,...) at sleepq_wait+0x48/frame 0x1aad6c04 _sleep(9f5008c,9f50070,6c,186211d,0,...) at _sleep+0x3a6/frame 0x1aad6c58 crypto_ret_proc(9f50070,1aad6ce8,17ec59f,419,0,...) at crypto_ret_proc+0x241/frame 0x1aad6cac fork_exit(13c8d70,9f50070,1aad6ce8,0,0,...) at fork_exit+0x7e/frame 0x1aad6cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aad6cd4 --- trap 0, eip = 0, esp = 0x1aad6d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 1 pid 4 tid 100039 td 0x9e65700 sched_switch(9e65700,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1aad1b70 mi_switch(104,0,1830cc7,22b,9e65700,...) at mi_switch+0x209/frame 0x1aad1bac sleepq_switch(9e65700,0,1830cc7,2b3,9e65700,...) at sleepq_switch+0x110/frame 0x1aad1be0 sleepq_wait(9f50054,6c,186211d,0,0,...) at sleepq_wait+0x48/frame 0x1aad1c04 _sleep(9f50054,9f50038,6c,186211d,0,...) at _sleep+0x3a6/frame 0x1aad1c58 crypto_ret_proc(9f50038,1aad1ce8,17ec59f,419,0,...) at crypto_ret_proc+0x241/frame 0x1aad1cac fork_exit(13c8d70,9f50038,1aad1ce8,0,0,...) at fork_exit+0x7e/frame 0x1aad1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aad1cd4 --- trap 0, eip = 0, esp = 0x1aad1d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 0 pid 3 tid 100038 td 0x9e65a80 sched_switch(9e65a80,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1aaccb70 mi_switch(104,0,1830cc7,22b,9e65a80,...) at mi_switch+0x209/frame 0x1aaccbac sleepq_switch(9e65a80,0,1830cc7,2b3,9e65a80,...) at sleepq_switch+0x110/frame 0x1aaccbe0 sleepq_wait(9f5001c,6c,186211d,0,0,...) at sleepq_wait+0x48/frame 0x1aaccc04 _sleep(9f5001c,9f50000,6c,186211d,0,...) at _sleep+0x3a6/frame 0x1aaccc58 crypto_ret_proc(9f50000,1aaccce8,17ec59f,419,0,...) at crypto_ret_proc+0x241/frame 0x1aacccac fork_exit(13c8d70,9f50000,1aaccce8,0,0,...) at fork_exit+0x7e/frame 0x1aacccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aacccd4 --- trap 0, eip = 0, esp = 0x1aaccd20, ebp = 0 --- (null)() at 0 Tracing command crypto pid 2 tid 100037 td 0x9edf000 sched_switch(9edf000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aac7b80 mi_switch(104,0,1830cc7,22b,9edf000,...) at mi_switch+0x209/frame 0x1aac7bc0 sleepq_switch(9edf000,0,1830cc7,2b3,9edf000,...) at sleepq_switch+0x110/frame 0x1aac7bf4 sleepq_wait(20be558,6c,1815c0a,0,0,...) at sleepq_wait+0x48/frame 0x1aac7c18 _sleep(20be558,20be540,6c,1815c0a,0,...) at _sleep+0x3a6/frame 0x1aac7c6c crypto_proc(0,1aac7ce8,17ec59f,419,0,...) at crypto_proc+0x2ad/frame 0x1aac7cac fork_exit(13c8980,0,1aac7ce8,0,0,...) at fork_exit+0x7e/frame 0x1aac7cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aac7cd4 --- trap 0, eip = 0, esp = 0x1aac7d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100029 td 0x9ee1000 sched_switch(9ee1000,0,104,18a,203e100,...) at sched_switch+0x80c/frame 0x1aa9fb88 mi_switch(104,0,1830cc7,22b,9ee1000,...) at mi_switch+0x209/frame 0x1aa9fbc4 sleepq_switch(9ee1000,0,1830cc7,2b3,9ee1000,...) at sleepq_switch+0x110/frame 0x1aa9fbf8 sleepq_wait(20e5270,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aa9fc1c _sleep(20e5270,202a374,25c,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aa9fc70 g_run_events(0,1aa9fce8,17ec59f,419,0,...) at g_run_events+0x67/frame 0x1aa9fcac fork_exit(102ff40,0,1aa9fce8,0,0,...) at fork_exit+0x7e/frame 0x1aa9fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa9fcd4 --- trap 0, eip = 0, esp = 0x1aa9fd20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100030 td 0x9ee0a80 sched_switch(9ee0a80,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x1aaa4b78 mi_switch(104,0,1830cc7,22b,9ee0a80,...) at mi_switch+0x209/frame 0x1aaa4bb4 sleepq_switch(9ee0a80,0,1830cc7,2b3,9ee0a80,...) at sleepq_switch+0x110/frame 0x1aaa4be8 sleepq_wait(20e5278,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aaa4c0c _sleep(20e5278,202a3bc,25c,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aaa4c60 g_io_schedule_up(9ee0a80,0,1784cba,61,1aaa4cd4,...) at g_io_schedule_up+0xcf/frame 0x1aaa4c94 g_up_procbody(0,1aaa4ce8,17ec59f,419,0,...) at g_up_procbody+0x7d/frame 0x1aaa4cac fork_exit(102ffc0,0,1aaa4ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa4cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa4cd4 --- trap 0, eip = 0, esp = 0x1aaa4d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100031 td 0x9ee0700 sched_switch(9ee0700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aaa9b70 mi_switch(104,0,1830cc7,22b,9ee0700,...) at mi_switch+0x209/frame 0x1aaa9bb0 sleepq_switch(9ee0700,0,1830cc7,2b3,9ee0700,...) at sleepq_switch+0x110/frame 0x1aaa9be4 sleepq_wait(20e526c,5c,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aaa9c08 _sleep(20e526c,202a39c,25c,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aaa9c5c g_io_schedule_down(9ee0700,0,1784cba,6d,1aaa9cd4,...) at g_io_schedule_down+0xec/frame 0x1aaa9c94 g_down_procbody(0,1aaa9ce8,17ec59f,419,0,...) at g_down_procbody+0x7d/frame 0x1aaa9cac fork_exit(1030040,0,1aaa9ce8,0,0,...) at fork_exit+0x7e/frame 0x1aaa9cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaa9cd4 --- trap 0, eip = 0, esp = 0x1aaa9d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100007 td 0xbfe8700 sched_switch(bfe8700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa05c28 mi_switch(109,0,1839ab1,479,203e100,...) at mi_switch+0x209/frame 0x1aa05c68 ithread_loop(9e61020,1aa05ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1aa05cac fork_exit(10b0660,9e61020,1aa05ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa05cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa05cd4 --- trap 0, eip = 0, esp = 0x1aa05d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100009 td 0xbfe8000 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100013 td 0xbfdb000 sched_switch(bfdb000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa23c28 mi_switch(109,0,1839ab1,479,203ccc0,...) at mi_switch+0x209/frame 0x1aa23c68 ithread_loop(9e61000,1aa23ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1aa23cac fork_exit(10b0660,9e61000,1aa23ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa23cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa23cd4 --- trap 0, eip = 0, esp = 0x1aa23d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100023 td 0x9e63000 sched_switch(9e63000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa61c28 mi_switch(109,0,1839ab1,479,203ccc0,...) at mi_switch+0x209/frame 0x1aa61c68 ithread_loop(9e60f90,1aa61ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1aa61cac fork_exit(10b0660,9e60f90,1aa61ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa61cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa61cd4 --- trap 0, eip = 0, esp = 0x1aa61d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100024 td 0xbfeaa80 sched_switch(bfeaa80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1aa66c28 mi_switch(109,0,1839ab1,479,203ccc0,...) at mi_switch+0x209/frame 0x1aa66c68 ithread_loop(9e60f80,1aa66ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1aa66cac fork_exit(10b0660,9e60f80,1aa66ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa66cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa66cd4 --- trap 0, eip = 0, esp = 0x1aa66d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100025 td 0xbfea700 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100026 td 0xbfea380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100027 td 0x9ee1700 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100028 td 0x9ee1380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100043 td 0x1acb7000 sched_switch(1acb7000,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab13c28 mi_switch(109,0,1839ab1,479,203e100,...) at mi_switch+0x209/frame 0x1ab13c68 ithread_loop(1aca3050,1ab13ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab13cac fork_exit(10b0660,1aca3050,1ab13ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab13cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab13cd4 --- trap 0, eip = 0, esp = 0x1ab13d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100044 td 0x1acb6a80 sched_switch(1acb6a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab1dc28 mi_switch(109,0,1839ab1,479,203ccc0,...) at mi_switch+0x209/frame 0x1ab1dc68 ithread_loop(1aca3040,1ab1dce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab1dcac fork_exit(10b0660,1aca3040,1ab1dce8,0,0,...) at fork_exit+0x7e/frame 0x1ab1dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab1dcd4 --- trap 0, eip = 0, esp = 0x1ab1dd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100045 td 0x1acb6700 sched_switch(1acb6700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab23c28 mi_switch(109,0,1839ab1,479,203d380,...) at mi_switch+0x209/frame 0x1ab23c68 ithread_loop(1aca3030,1ab23ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab23cac fork_exit(10b0660,1aca3030,1ab23ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab23cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab23cd4 --- trap 0, eip = 0, esp = 0x1ab23d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100046 td 0x1acb6380 sched_switch(1acb6380,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab29c28 mi_switch(109,0,1839ab1,479,203da40,...) at mi_switch+0x209/frame 0x1ab29c68 ithread_loop(1aca3020,1ab29ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab29cac fork_exit(10b0660,1aca3020,1ab29ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab29cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab29cd4 --- trap 0, eip = 0, esp = 0x1ab29d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100052 td 0x1aca4a80 sched_switch(1aca4a80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab48c28 mi_switch(109,0,1839ab1,479,203e100,...) at mi_switch+0x209/frame 0x1ab48c68 ithread_loop(1aca3010,1ab48ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab48cac fork_exit(10b0660,1aca3010,1ab48ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab48cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab48cd4 --- trap 0, eip = 0, esp = 0x1ab48d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100058 td 0x1af6ca80 sched_switch(1af6ca80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1ab67c28 mi_switch(109,0,1839ab1,479,203ccc0,...) at mi_switch+0x209/frame 0x1ab67c68 ithread_loop(1aca3000,1ab67ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1ab67cac fork_exit(10b0660,1aca3000,1ab67ce8,0,0,...) at fork_exit+0x7e/frame 0x1ab67cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1ab67cd4 --- trap 0, eip = 0, esp = 0x1ab67d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100074 td 0x1af8d380 PTDpde() at PTDpde+0x43b8 Tracing command intr pid 12 tid 100080 td 0x1af6da80 sched_switch(1af6da80,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1abd8c28 mi_switch(109,0,1839ab1,479,203da40,...) at mi_switch+0x209/frame 0x1abd8c68 ithread_loop(9e61f80,1abd8ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1abd8cac fork_exit(10b0660,9e61f80,1abd8ce8,0,0,...) at fork_exit+0x7e/frame 0x1abd8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abd8cd4 --- trap 0, eip = 0, esp = 0x1abd8d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100081 td 0x1af6d700 sched_switch(1af6d700,0,109,18a,8,...) at sched_switch+0x80c/frame 0x1abe1c28 mi_switch(109,0,1839ab1,479,203e100,...) at mi_switch+0x209/frame 0x1abe1c68 ithread_loop(9e61f60,1abe1ce8,17ec59f,419,0,...) at ithread_loop+0x32a/frame 0x1abe1cac fork_exit(10b0660,9e61f60,1abe1ce8,0,0,...) at fork_exit+0x7e/frame 0x1abe1cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abe1cd4 --- trap 0, eip = 0, esp = 0x1abe1d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100083 td 0x1afdf700 PTDpde() at PTDpde+0x43b8 Tracing command idle pid 11 tid 100003 td 0xbfe9700 (CPU 0) cpustop_handler(0,9772830,1511ab94,17008b4,20d960c,...) at cpustop_handler+0x25/frame 0x1511aa80 ipi_nmi_handler(20d960c,4a2bb,0,0,0,...) at ipi_nmi_handler+0x36/frame 0x1511aa90 trap(1511aba0,8,28,28,9fe1e00,...) at trap+0xd4/frame 0x1511ab94 calltrap() at PTDpde+0x4165/frame 0x1511ab94 --- trap 0x13, eip = 0x17191c5, esp = 0x1511abe0, ebp = 0x1511abe0 --- acpi_cpu_c1(1511ac34,17093b2,2428d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x1511abe0 acpi_cpu_idle(aa43247,0,aa43247,1511ac50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x1511ac20 cpu_idle_acpi(aa43247,0,203ccc0,12,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511ac34 cpu_idle(0,1511ac80,17f6cfa,adb,12,...) at cpu_idle+0xab/frame 0x1511ac50 sched_idletd(0,1511ace8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x1511acac fork_exit(1123830,0,1511ace8,0,0,...) at fork_exit+0x7e/frame 0x1511acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511acd4 --- trap 0, eip = 0, esp = 0x1511ad20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0xbfe9380 (CPU 1) cpustop_handler(1,9772830,1511fb94,17008b4,1511fac8,...) at cpustop_handler+0x25/frame 0x1511fa80 ipi_nmi_handler(1511fac8,10ce8c4,20e67c0,8,1742ab1,...) at ipi_nmi_handler+0x36/frame 0x1511fa90 trap(1511fba0,8,28,28,9fe1c00,...) at trap+0xd4/frame 0x1511fb94 calltrap() at PTDpde+0x4165/frame 0x1511fb94 --- trap 0x13, eip = 0x17191c5, esp = 0x1511fbe0, ebp = 0x1511fbe0 --- acpi_cpu_c1(1511fc34,17093b2,be78d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x1511fbe0 acpi_cpu_idle(462ed30,0,462ed30,1511fc50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x1511fc20 cpu_idle_acpi(462ed30,0,203d380,14,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x1511fc34 cpu_idle(0,1511fc80,17f6cfa,395,14,...) at cpu_idle+0xab/frame 0x1511fc50 sched_idletd(0,1511fce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x1511fcac fork_exit(1123830,0,1511fce8,0,0,...) at fork_exit+0x7e/frame 0x1511fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1511fcd4 --- trap 0, eip = 0, esp = 0x1511fd20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0xbfe9000 (CPU 2) cpustop_handler(2,9772830,15124b94,17008b4,15124ac8,...) at cpustop_handler+0x25/frame 0x15124a80 ipi_nmi_handler(15124ac8,10ce8c4,20e68c0,8,1742ab1,...) at ipi_nmi_handler+0x36/frame 0x15124a90 trap(15124ba0,8,28,28,9fe1a00,...) at trap+0xd4/frame 0x15124b94 calltrap() at PTDpde+0x4165/frame 0x15124b94 --- trap 0x13, eip = 0x17191c5, esp = 0x15124be0, ebp = 0x15124be0 --- acpi_cpu_c1(15124c34,17093b2,be7dd90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15124be0 acpi_cpu_idle(47f7c07f,0,47f7c07f,15124c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15124c20 cpu_idle_acpi(47f7c07f,0,203da40,11,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15124c34 cpu_idle(0,15124c80,17f6cfa,adb,11,...) at cpu_idle+0xab/frame 0x15124c50 sched_idletd(0,15124ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15124cac fork_exit(1123830,0,15124ce8,0,0,...) at fork_exit+0x7e/frame 0x15124cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15124cd4 --- trap 0, eip = 0, esp = 0x15124d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0xbfe8a80 (CPU 3) kdb_alt_break_internal(0,15129ac4,ed8457,2,1af00e8c,...) at kdb_alt_break_internal+0xf0/frame 0x15129a98 kdb_alt_break(2,1af00e8c,1855350,0,40000,...) at kdb_alt_break+0x16/frame 0x15129aa4 uart_intr_rxready(1af00e00,e0,d9,ac4a,bfe8a80,...) at uart_intr_rxready+0x87/frame 0x15129ac4 uart_intr(1af00e00,203e110,8,17f6cfa,bfe8a80,...) at uart_intr+0x130/frame 0x15129b04 intr_event_handle(bfb0900,15129ba0,203e15c,255c0000,1,...) at intr_event_handle+0xd5/frame 0x15129b38 intr_execute_handlers(9e52130,15129ba0,3) at intr_execute_handlers+0xcf/frame 0x15129b70 lapic_handle_intr(32,15129ba0,8,28,28,...) at lapic_handle_intr+0x5e/frame 0x15129b90 Xapic_isr1() at PTDpde+0x5200/frame 0x15129b90 --- interrupt, eip = 0x17191c5, esp = 0x15129be0, ebp = 0x15129be0 --- acpi_cpu_c1(15129c34,17093b2,be82d90,0,174fdeb,...) at acpi_cpu_c1+0x5/frame 0x15129be0 acpi_cpu_idle(3cb533,0,3cb533,15129c50,171981b,...) at acpi_cpu_idle+0x30b/frame 0x15129c20 cpu_idle_acpi(3cb533,0,203e100,20,ffffffff,...) at cpu_idle_acpi+0x3f/frame 0x15129c34 cpu_idle(0,15129c80,17f6cfa,adb,20,...) at cpu_idle+0xab/frame 0x15129c50 sched_idletd(0,15129ce8,17ec59f,419,0,...) at sched_idletd+0x236/frame 0x15129cac fork_exit(1123830,0,15129ce8,0,0,...) at fork_exit+0x7e/frame 0x15129cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15129cd4 --- trap 0, eip = 0, esp = 0x15129d20, ebp = 0 --- (null)() at 0 Tracing command init pid 1 tid 100002 td 0xbfe9a80 sched_switch(bfe9a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15114940 mi_switch(104,0,1830cc7,22b,bfe9a80,...) at mi_switch+0x209/frame 0x15114980 sleepq_switch(bfe9a80,0,1830cc7,1ca,6c,...) at sleepq_switch+0x110/frame 0x151149b4 sleepq_catch_signals(0,1830cc7,178,204c8d4,bfe9a80,...) at sleepq_catch_signals+0x360/frame 0x151149f8 sleepq_wait_sig(bfd99fc,6c,184c984,100,0,...) at sleepq_wait_sig+0x14/frame 0x15114a1c _sleep(bfd99fc,bfd9aa4,36c,184c984,0,...) at _sleep+0x396/frame 0x15114a70 kern_wait6(bfe9a80,7,0,0,15114af4,...) at kern_wait6+0x567/frame 0x15114ac8 sys_wait4(bfe9a80,bfe9d04,181d4b1,4,17f6cfa,...) at sys_wait4+0x87/frame 0x15114be8 syscall(15114ce8,3b,3b,3b,ffbfe8d0,...) at syscall+0x364/frame 0x15114cdc Xint0x80_syscall() at PTDpde+0x43af/frame 0x15114cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x995cb, esp = 0xffbfe884, ebp = 0xffbfe898 --- (null)() at 0x995cb Tracing command audit pid 10 tid 100001 td 0xbfea000 sched_switch(bfea000,0,104,18a,203da40,...) at sched_switch+0x80c/frame 0x1510fb80 mi_switch(104,0,1830cc7,22b,bfea000,...) at mi_switch+0x209/frame 0x1510fbbc sleepq_switch(bfea000,0,1830cc7,2b3,20e964c,...) at sleepq_switch+0x110/frame 0x1510fbf0 sleepq_wait(20e969c,0,1510fc34,1,0,...) at sleepq_wait+0x48/frame 0x1510fc14 _cv_wait(20e969c,20e964c,177d153,1be,8,...) at _cv_wait+0x1d7/frame 0x1510fc58 audit_worker(0,1510fce8,17ec59f,419,0,...) at audit_worker+0x94/frame 0x1510fcac fork_exit(13ec330,0,1510fce8,0,0,...) at fork_exit+0x7e/frame 0x1510fcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1510fcd4 --- trap 0, eip = 0, esp = 0x1510fd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100000 td 0x20e5600 sched_switch(20e5600,0,104,18a,203ccc0,...) at sched_switch+0x80c/frame 0x2423be8 mi_switch(104,0,1830cc7,22b,20e5600,...) at mi_switch+0x209/frame 0x2423c24 sleepq_switch(20e5600,0,1830cc7,2d6,ac4b,...) at sleepq_switch+0x110/frame 0x2423c58 sleepq_timedwait(20e5290,54,9,0,0,...) at sleepq_timedwait+0x53/frame 0x2423c98 _sleep(20e5290,0,54,184f65d,fffff470,...) at _sleep+0x342/frame 0x2423cec swapper() at swapper+0x98/frame 0x2423d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100008 td 0xbfe8380 sched_switch(bfe8380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa0ab88 mi_switch(104,0,1830cc7,22b,bfe8380,...) at mi_switch+0x209/frame 0x1aa0abc8 sleepq_switch(bfe8380,0,1830cc7,2b3,bfe8380,...) at sleepq_switch+0x110/frame 0x1aa0abfc sleepq_wait(9e56f00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aa0ac20 _sleep(9e56f00,9e56f1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aa0ac74 taskqueue_thread_loop(20e89e8,1aa0ace8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa0acac fork_exit(1152cc0,20e89e8,1aa0ace8,0,0,...) at fork_exit+0x7e/frame 0x1aa0acd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa0acd4 --- trap 0, eip = 0, esp = 0x1aa0ad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100010 td 0xbfdba80 sched_switch(bfdba80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa14b98 mi_switch(104,0,1830cc7,22b,bfdba80,...) at mi_switch+0x209/frame 0x1aa14bd8 sleepq_switch(bfdba80,0,1830cc7,2b3,bffda00,...) at sleepq_switch+0x110/frame 0x1aa14c0c sleepq_wait(bffda00,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa14c30 msleep_spin_sbt(bffda00,bffda1c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa14c74 gtaskqueue_thread_loop(9de6804,1aa14ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa14cac fork_exit(113c540,9de6804,1aa14ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa14cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa14cd4 --- trap 0, eip = 0, esp = 0x1aa14d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100011 td 0xbfdb700 sched_switch(bfdb700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa19b88 mi_switch(104,0,1830cc7,22b,bfdb700,...) at mi_switch+0x209/frame 0x1aa19bc8 sleepq_switch(bfdb700,0,1830cc7,2b3,bfdb700,...) at sleepq_switch+0x110/frame 0x1aa19bfc sleepq_wait(9e56e00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aa19c20 _sleep(9e56e00,9e56e1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aa19c74 taskqueue_thread_loop(20e5b9c,1aa19ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa19cac fork_exit(1152cc0,20e5b9c,1aa19ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa19cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa19cd4 --- trap 0, eip = 0, esp = 0x1aa19d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100012 td 0xbfdb380 sched_switch(bfdb380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa1eb88 mi_switch(104,0,1830cc7,22b,bfdb380,...) at mi_switch+0x209/frame 0x1aa1ebc8 sleepq_switch(bfdb380,0,1830cc7,2b3,bfdb380,...) at sleepq_switch+0x110/frame 0x1aa1ebfc sleepq_wait(9e56d80,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aa1ec20 _sleep(9e56d80,9e56d9c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aa1ec74 taskqueue_thread_loop(20e8a38,1aa1ece8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aa1ecac fork_exit(1152cc0,20e8a38,1aa1ece8,0,0,...) at fork_exit+0x7e/frame 0x1aa1ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa1ecd4 --- trap 0, eip = 0, esp = 0x1aa1ed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100014 td 0x9e65380 sched_switch(9e65380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa34b98 mi_switch(104,0,1830cc7,22b,9e65380,...) at mi_switch+0x209/frame 0x1aa34bd8 sleepq_switch(9e65380,0,1830cc7,2b3,bffd980,...) at sleepq_switch+0x110/frame 0x1aa34c0c sleepq_wait(bffd980,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa34c30 msleep_spin_sbt(bffd980,bffd99c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa34c74 gtaskqueue_thread_loop(9de6c04,1aa34ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa34cac fork_exit(113c540,9de6c04,1aa34ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa34cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa34cd4 --- trap 0, eip = 0, esp = 0x1aa34d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100015 td 0x9e65000 sched_switch(9e65000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa39b98 mi_switch(104,0,1830cc7,22b,9e65000,...) at mi_switch+0x209/frame 0x1aa39bd8 sleepq_switch(9e65000,0,1830cc7,2b3,bffd900,...) at sleepq_switch+0x110/frame 0x1aa39c0c sleepq_wait(bffd900,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa39c30 msleep_spin_sbt(bffd900,bffd91c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa39c74 gtaskqueue_thread_loop(9de6c14,1aa39ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa39cac fork_exit(113c540,9de6c14,1aa39ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa39cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa39cd4 --- trap 0, eip = 0, esp = 0x1aa39d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100016 td 0x9e64a80 sched_switch(9e64a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa3eb98 mi_switch(104,0,1830cc7,22b,9e64a80,...) at mi_switch+0x209/frame 0x1aa3ebd8 sleepq_switch(9e64a80,0,1830cc7,2b3,bffd880,...) at sleepq_switch+0x110/frame 0x1aa3ec0c sleepq_wait(bffd880,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa3ec30 msleep_spin_sbt(bffd880,bffd89c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa3ec74 gtaskqueue_thread_loop(9de6c24,1aa3ece8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa3ecac fork_exit(113c540,9de6c24,1aa3ece8,0,0,...) at fork_exit+0x7e/frame 0x1aa3ecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa3ecd4 --- trap 0, eip = 0, esp = 0x1aa3ed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100017 td 0x9e64700 sched_switch(9e64700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa43b98 mi_switch(104,0,1830cc7,22b,9e64700,...) at mi_switch+0x209/frame 0x1aa43bd8 sleepq_switch(9e64700,0,1830cc7,2b3,bffd800,...) at sleepq_switch+0x110/frame 0x1aa43c0c sleepq_wait(bffd800,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa43c30 msleep_spin_sbt(bffd800,bffd81c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa43c74 gtaskqueue_thread_loop(9de6c34,1aa43ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa43cac fork_exit(113c540,9de6c34,1aa43ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa43cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa43cd4 --- trap 0, eip = 0, esp = 0x1aa43d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100018 td 0x9e64380 sched_switch(9e64380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa48b98 mi_switch(104,0,1830cc7,22b,9e64380,...) at mi_switch+0x209/frame 0x1aa48bd8 sleepq_switch(9e64380,0,1830cc7,2b3,bffd780,...) at sleepq_switch+0x110/frame 0x1aa48c0c sleepq_wait(bffd780,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa48c30 msleep_spin_sbt(bffd780,bffd79c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa48c74 gtaskqueue_thread_loop(9de6404,1aa48ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa48cac fork_exit(113c540,9de6404,1aa48ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa48cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa48cd4 --- trap 0, eip = 0, esp = 0x1aa48d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100019 td 0x9e64000 sched_switch(9e64000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa4db98 mi_switch(104,0,1830cc7,22b,9e64000,...) at mi_switch+0x209/frame 0x1aa4dbd8 sleepq_switch(9e64000,0,1830cc7,2b3,bffd700,...) at sleepq_switch+0x110/frame 0x1aa4dc0c sleepq_wait(bffd700,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa4dc30 msleep_spin_sbt(bffd700,bffd71c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa4dc74 gtaskqueue_thread_loop(9de6414,1aa4dce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa4dcac fork_exit(113c540,9de6414,1aa4dce8,0,0,...) at fork_exit+0x7e/frame 0x1aa4dcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa4dcd4 --- trap 0, eip = 0, esp = 0x1aa4dd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100020 td 0x9e63a80 sched_switch(9e63a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa52b98 mi_switch(104,0,1830cc7,22b,9e63a80,...) at mi_switch+0x209/frame 0x1aa52bd8 sleepq_switch(9e63a80,0,1830cc7,2b3,bffd680,...) at sleepq_switch+0x110/frame 0x1aa52c0c sleepq_wait(bffd680,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa52c30 msleep_spin_sbt(bffd680,bffd69c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa52c74 gtaskqueue_thread_loop(9de6424,1aa52ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa52cac fork_exit(113c540,9de6424,1aa52ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa52cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa52cd4 --- trap 0, eip = 0, esp = 0x1aa52d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100021 td 0x9e63700 sched_switch(9e63700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa57b98 mi_switch(104,0,1830cc7,22b,9e63700,...) at mi_switch+0x209/frame 0x1aa57bd8 sleepq_switch(9e63700,0,1830cc7,2b3,bffd600,...) at sleepq_switch+0x110/frame 0x1aa57c0c sleepq_wait(bffd600,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa57c30 msleep_spin_sbt(bffd600,bffd61c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa57c74 gtaskqueue_thread_loop(9de6434,1aa57ce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa57cac fork_exit(113c540,9de6434,1aa57ce8,0,0,...) at fork_exit+0x7e/frame 0x1aa57cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa57cd4 --- trap 0, eip = 0, esp = 0x1aa57d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100022 td 0x9e63380 sched_switch(9e63380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aa5cb98 mi_switch(104,0,1830cc7,22b,9e63380,...) at mi_switch+0x209/frame 0x1aa5cbd8 sleepq_switch(9e63380,0,1830cc7,2b3,bffd580,...) at sleepq_switch+0x110/frame 0x1aa5cc0c sleepq_wait(bffd580,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1aa5cc30 msleep_spin_sbt(bffd580,bffd59c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1aa5cc74 gtaskqueue_thread_loop(9de6004,1aa5cce8,17ec59f,419,0,...) at gtaskqueue_thread_loop+0xf3/frame 0x1aa5ccac fork_exit(113c540,9de6004,1aa5cce8,0,0,...) at fork_exit+0x7e/frame 0x1aa5ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aa5ccd4 --- trap 0, eip = 0, esp = 0x1aa5cd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100032 td 0x9ee0380 sched_switch(9ee0380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aaaeb88 mi_switch(104,0,1830cc7,22b,9ee0380,...) at mi_switch+0x209/frame 0x1aaaebc8 sleepq_switch(9ee0380,0,1830cc7,2b3,9ee0380,...) at sleepq_switch+0x110/frame 0x1aaaebfc sleepq_wait(9e56c80,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aaaec20 _sleep(9e56c80,9e56c9c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aaaec74 taskqueue_thread_loop(204a608,1aaaece8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aaaecac fork_exit(1152cc0,204a608,1aaaece8,0,0,...) at fork_exit+0x7e/frame 0x1aaaecd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aaaecd4 --- trap 0, eip = 0, esp = 0x1aaaed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100033 td 0x9ee0000 sched_switch(9ee0000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aab3b88 mi_switch(104,0,1830cc7,22b,9ee0000,...) at mi_switch+0x209/frame 0x1aab3bc8 sleepq_switch(9ee0000,0,1830cc7,2b3,9ee0000,...) at sleepq_switch+0x110/frame 0x1aab3bfc sleepq_wait(9e56c00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aab3c20 _sleep(9e56c00,9e56c1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aab3c74 taskqueue_thread_loop(20be5fc,1aab3ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aab3cac fork_exit(1152cc0,20be5fc,1aab3ce8,0,0,...) at fork_exit+0x7e/frame 0x1aab3cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aab3cd4 --- trap 0, eip = 0, esp = 0x1aab3d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100034 td 0x9edfa80 sched_switch(9edfa80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aab8b88 mi_switch(104,0,1830cc7,22b,9edfa80,...) at mi_switch+0x209/frame 0x1aab8bc8 sleepq_switch(9edfa80,0,1830cc7,2b3,9edfa80,...) at sleepq_switch+0x110/frame 0x1aab8bfc sleepq_wait(9e56c00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aab8c20 _sleep(9e56c00,9e56c1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aab8c74 taskqueue_thread_loop(20be5fc,1aab8ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aab8cac fork_exit(1152cc0,20be5fc,1aab8ce8,0,0,...) at fork_exit+0x7e/frame 0x1aab8cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aab8cd4 --- trap 0, eip = 0, esp = 0x1aab8d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100035 td 0x9edf700 sched_switch(9edf700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aabdb88 mi_switch(104,0,1830cc7,22b,9edf700,...) at mi_switch+0x209/frame 0x1aabdbc8 sleepq_switch(9edf700,0,1830cc7,2b3,9edf700,...) at sleepq_switch+0x110/frame 0x1aabdbfc sleepq_wait(9e56c00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aabdc20 _sleep(9e56c00,9e56c1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aabdc74 taskqueue_thread_loop(20be5fc,1aabdce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aabdcac fork_exit(1152cc0,20be5fc,1aabdce8,0,0,...) at fork_exit+0x7e/frame 0x1aabdcd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aabdcd4 --- trap 0, eip = 0, esp = 0x1aabdd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100036 td 0x9edf380 sched_switch(9edf380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1aac2b88 mi_switch(104,0,1830cc7,22b,9edf380,...) at mi_switch+0x209/frame 0x1aac2bc8 sleepq_switch(9edf380,0,1830cc7,2b3,9edf380,...) at sleepq_switch+0x110/frame 0x1aac2bfc sleepq_wait(9e56c00,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x1aac2c20 _sleep(9e56c00,9e56c1c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x1aac2c74 taskqueue_thread_loop(20be5fc,1aac2ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x1aac2cac fork_exit(1152cc0,20be5fc,1aac2ce8,0,0,...) at fork_exit+0x7e/frame 0x1aac2cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1aac2cd4 --- trap 0, eip = 0, esp = 0x1aac2d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100084 td 0x1afdf380 sched_switch(1afdf380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1abf0b98 mi_switch(104,0,1830cc7,22b,1afdf380,...) at mi_switch+0x209/frame 0x1abf0bd8 sleepq_switch(1afdf380,0,1830cc7,2b3,1afcc000,...) at sleepq_switch+0x110/frame 0x1abf0c0c sleepq_wait(1afcc000,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1abf0c30 msleep_spin_sbt(1afcc000,1afcc01c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1abf0c74 taskqueue_thread_loop(20d9694,1abf0ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x1abf0cac fork_exit(1152cc0,20d9694,1abf0ce8,0,0,...) at fork_exit+0x7e/frame 0x1abf0cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1abf0cd4 --- trap 0, eip = 0, esp = 0x1abf0d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100091 td 0x9f54700 sched_switch(9f54700,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15142b98 mi_switch(104,0,1830cc7,22b,9f54700,...) at mi_switch+0x209/frame 0x15142bd8 sleepq_switch(9f54700,0,1830cc7,2b3,9e56a80,...) at sleepq_switch+0x110/frame 0x15142c0c sleepq_wait(9e56a80,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x15142c30 msleep_spin_sbt(9e56a80,9e56a9c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x15142c74 taskqueue_thread_loop(20db308,15142ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x15142cac fork_exit(1152cc0,20db308,15142ce8,0,0,...) at fork_exit+0x7e/frame 0x15142cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15142cd4 --- trap 0, eip = 0, esp = 0x15142d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100092 td 0x9f54380 sched_switch(9f54380,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15147b98 mi_switch(104,0,1830cc7,22b,9f54380,...) at mi_switch+0x209/frame 0x15147bd8 sleepq_switch(9f54380,0,1830cc7,2b3,9e56a80,...) at sleepq_switch+0x110/frame 0x15147c0c sleepq_wait(9e56a80,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x15147c30 msleep_spin_sbt(9e56a80,9e56a9c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x15147c74 taskqueue_thread_loop(20db308,15147ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x15147cac fork_exit(1152cc0,20db308,15147ce8,0,0,...) at fork_exit+0x7e/frame 0x15147cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15147cd4 --- trap 0, eip = 0, esp = 0x15147d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100093 td 0x9f54000 sched_switch(9f54000,0,104,18a,0,...) at sched_switch+0x80c/frame 0x1514cb98 mi_switch(104,0,1830cc7,22b,9f54000,...) at mi_switch+0x209/frame 0x1514cbd8 sleepq_switch(9f54000,0,1830cc7,2b3,9e56a80,...) at sleepq_switch+0x110/frame 0x1514cc0c sleepq_wait(9e56a80,0,1768f89,177f3f9,0,...) at sleepq_wait+0x48/frame 0x1514cc30 msleep_spin_sbt(9e56a80,9e56a9c,177f3f9,0,0,...) at msleep_spin_sbt+0x1f6/frame 0x1514cc74 taskqueue_thread_loop(20db308,1514cce8,17ec59f,419,0,...) at taskqueue_thread_loop+0xf3/frame 0x1514ccac fork_exit(1152cc0,20db308,1514cce8,0,0,...) at fork_exit+0x7e/frame 0x1514ccd4 fork_trampoline() at PTDpde+0x43c2/frame 0x1514ccd4 --- trap 0, eip = 0, esp = 0x1514cd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100094 td 0x9f53a80 sched_switch(9f53a80,0,104,18a,0,...) at sched_switch+0x80c/frame 0x15151b88 mi_switch(104,0,1830cc7,22b,9f53a80,...) at mi_switch+0x209/frame 0x15151bc8 sleepq_switch(9f53a80,0,1830cc7,2b3,9f53a80,...) at sleepq_switch+0x110/frame 0x15151bfc sleepq_wait(9e56b80,0,177f3f9,0,0,...) at sleepq_wait+0x48/frame 0x15151c20 _sleep(9e56b80,9e56b9c,0,177f3f9,0,...) at _sleep+0x3a6/frame 0x15151c74 taskqueue_thread_loop(1e52be8,15151ce8,17ec59f,419,0,...) at taskqueue_thread_loop+0x144/frame 0x15151cac fork_exit(1152cc0,1e52be8,15151ce8,0,0,...) at fork_exit+0x7e/frame 0x15151cd4 fork_trampoline() at PTDpde+0x43c2/frame 0x15151cd4 --- trap 0, eip = 0, esp = 0x15151d20, ebp = 0 --- (null)() at 0 db:0:pho> dump Physical memory: 3427 MB Dumping 370 MB: 355 339 323 307 291 275 259 243 227 211 195 179 163 147 131 115 99 83 67 51 35 19 3 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) proc 92970 [Switching to thread 204 (Thread 100687)]#0 sched_switch (td=0x255c0000, newtd=, flags=0x104) at ../../../kern/sched_ule.c:2112 2112 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=0x255c0000, newtd=, flags=0x104) at ../../../kern/sched_ule.c:2112 #1 0x010fdf99 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:439 #2 0x0114d4a0 in sleepq_switch (wchan=, pri=0x0) at ../../../kern/subr_sleepqueue.c:613 #3 0x0114dc73 in sleepq_timedwait (wchan=0x202e997, pri=0x0) at ../../../kern/subr_sleepqueue.c:727 #4 0x010fd672 in _sleep (ident=0x202e997, lock=0x0, priority=0x0, wmesg=0x288068bb "pmcctx", sbt=0x418937, pr=0x0, flags=0x100) at ../../../kern/kern_synch.c:213 #5 0x010fdcac in pause_sbt (wmesg=0x288068bb "pmcctx", sbt=0x418937, pr=0x0, flags=0x100) at ../../../kern/kern_synch.c:332 #6 0x2881527a in pmc_release_pmc_descriptor (pm=0x2911f980) at /usr/src/sys/dev/hwpmc/hwpmc_mod.c:811 #7 0x288167a3 in pmc_syscall_handler (td=0x255c0000, syscall_args=0x255c0284) at /usr/src/sys/dev/hwpmc/hwpmc_mod.c:4298 #8 0x01701de4 in syscall (frame=0x153c3ce8) at subr_syscall.c:135 #9 0xffc033a7 in ?? () #10 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c =================================================================== --- /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c (revision 338128) +++ /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c (working copy) @@ -238,7 +238,7 @@ void kmem_cache_reap_soon(kmem_cache_t *cache) { #ifndef KMEM_DEBUG - zone_drain(cache->kc_zone); + uma_zreclaim(cache->kc_zone, UMA_RECLAIM_DRAIN); #endif } @@ -245,7 +245,7 @@ kmem_cache_reap_soon(kmem_cache_t *cache) void kmem_reap(void) { - uma_reclaim(); + uma_reclaim(UMA_RECLAIM_TRIM); } #else void Index: /usr/src/sys/kern/kern_mbuf.c =================================================================== --- /usr/src/sys/kern/kern_mbuf.c (revision 338128) +++ /usr/src/sys/kern/kern_mbuf.c (working copy) @@ -673,14 +673,14 @@ mb_dtor_pack(void *mem, int size, void *arg) #endif /* * If there are processes blocked on zone_clust, waiting for pages - * to be freed up, * cause them to be woken up by draining the - * packet zone. We are exposed to a race here * (in the check for + * to be freed up, cause them to be woken up by draining the + * packet zone. We are exposed to a race here (in the check for * the UMA_ZFLAG_FULL) where we might miss the flag set, but that * is deliberate. We don't want to acquire the zone lock for every * mbuf free. */ if (uma_zone_exhausted_nolock(zone_clust)) - zone_drain(zone_pack); + uma_zreclaim(zone_pack, UMA_RECLAIM_DRAIN); } /* @@ -924,7 +924,7 @@ m_clget(struct mbuf *m, int how) * we might be able to loosen a few clusters up on the drain. */ if ((how & M_NOWAIT) && (m->m_ext.ext_buf == NULL)) { - zone_drain(zone_pack); + uma_zreclaim(zone_pack, UMA_RECLAIM_DRAIN); uma_zalloc_arg(zone_clust, m, how); } MBUF_PROBE2(m__clget, m, how); Index: /usr/src/sys/kern/subr_vmem.c =================================================================== --- /usr/src/sys/kern/subr_vmem.c (revision 338128) +++ /usr/src/sys/kern/subr_vmem.c (working copy) @@ -589,7 +589,7 @@ qc_drain(vmem_t *vm) qcache_idx_max = vm->vm_qcache_max >> vm->vm_quantum_shift; for (i = 0; i < qcache_idx_max; i++) - zone_drain(vm->vm_qcache[i].qc_cache); + uma_zreclaim(vm->vm_qcache[i].qc_cache, UMA_RECLAIM_DRAIN); } #ifndef UMA_MD_SMALL_ALLOC Index: /usr/src/sys/kern/vfs_subr.c =================================================================== --- /usr/src/sys/kern/vfs_subr.c (revision 338128) +++ /usr/src/sys/kern/vfs_subr.c (working copy) @@ -1222,7 +1222,7 @@ vnlru_proc(void) } mtx_unlock(&mountlist_mtx); if (onumvnodes > desiredvnodes && numvnodes <= desiredvnodes) - uma_reclaim(); + uma_reclaim(UMA_RECLAIM_TRIM); if (done == 0) { if (force == 0 || force == 1) { force = 2; Index: /usr/src/sys/vm/uma.h =================================================================== --- /usr/src/sys/vm/uma.h (revision 338128) +++ /usr/src/sys/vm/uma.h (working copy) @@ -50,8 +50,6 @@ struct uma_zone; /* Opaque type used as a handle to the zone */ typedef struct uma_zone * uma_zone_t; -void zone_drain(uma_zone_t); - /* * Item constructor * @@ -450,18 +448,19 @@ typedef void *(*uma_alloc)(uma_zone_t zone, vm_siz typedef void (*uma_free)(void *item, vm_size_t size, uint8_t pflag); /* - * Reclaims unused memory for all zones + * Reclaims unused memory * * Arguments: - * None + * req Reclamation request type. * Returns: * None - * - * This should only be called by the page out daemon. */ +#define UMA_RECLAIM_DRAIN 1 /* release bucket cache */ +#define UMA_RECLAIM_DRAIN_CPU 2 /* release bucket and per-CPU caches */ +#define UMA_RECLAIM_TRIM 3 /* trim bucket cache to WSS */ +void uma_reclaim(int req); +void uma_zreclaim(uma_zone_t, int req); -void uma_reclaim(void); - /* * Sets the alignment mask to be used for all zones requesting cache * alignment. Should be called by MD boot code prior to starting VM/UMA. Index: /usr/src/sys/vm/uma_core.c =================================================================== --- /usr/src/sys/vm/uma_core.c (revision 338128) +++ /usr/src/sys/vm/uma_core.c (working copy) @@ -236,7 +236,7 @@ static void pcpu_page_free(void *, vm_size_t, uint static uma_slab_t keg_alloc_slab(uma_keg_t, uma_zone_t, int, int); static void cache_drain(uma_zone_t); static void bucket_drain(uma_zone_t, uma_bucket_t); -static void bucket_cache_drain(uma_zone_t zone); +static void bucket_cache_reclaim(uma_zone_t zone, bool); static int keg_ctor(void *, int, void *, int); static void keg_dtor(void *, int, void *); static int zone_ctor(void *, int, void *, int); @@ -453,10 +453,40 @@ bucket_zone_drain(void) struct uma_bucket_zone *ubz; for (ubz = &bucket_zones[0]; ubz->ubz_entries != 0; ubz++) - zone_drain(ubz->ubz_zone); + uma_zreclaim(ubz->ubz_zone, UMA_RECLAIM_DRAIN); } +static uma_bucket_t +zone_try_fetch_bucket(uma_zone_t zone, uma_zone_domain_t zdom, const bool ws) +{ + uma_bucket_t bucket; + + ZONE_LOCK_ASSERT(zone); + + if ((bucket = LIST_FIRST(&zdom->uzd_buckets)) != NULL) { + MPASS(zdom->uzd_nitems >= bucket->ub_cnt); + LIST_REMOVE(bucket, ub_link); + zdom->uzd_nitems -= bucket->ub_cnt; + if (ws && zdom->uzd_imin > zdom->uzd_nitems) + zdom->uzd_imin = zdom->uzd_nitems; + } + return (bucket); +} + static void +zone_put_bucket(uma_zone_t zone, uma_zone_domain_t zdom, uma_bucket_t bucket, + const bool ws) +{ + + ZONE_LOCK_ASSERT(zone); + + LIST_INSERT_HEAD(&zdom->uzd_buckets, bucket, ub_link); + zdom->uzd_nitems += bucket->ub_cnt; + if (ws && zdom->uzd_imax < zdom->uzd_nitems) + zdom->uzd_imax = zdom->uzd_nitems; +} + +static void zone_log_warning(uma_zone_t zone) { static const struct timeval warninterval = { 300, 0 }; @@ -505,6 +535,18 @@ uma_timeout(void *unused) callout_reset(&uma_callout, UMA_TIMEOUT * hz, uma_timeout, NULL); } +/* XXX */ +static void +zone_domain_update_wss(uma_zone_domain_t zdom) +{ + long wss; + + MPASS(zdom->uzd_imax >= zdom->uzd_imin); + wss = zdom->uzd_imax - zdom->uzd_imin; + zdom->uzd_imax = zdom->uzd_imin = zdom->uzd_nitems; + zdom->uzd_wss = (3 * wss + 2 * zdom->uzd_wss) / 5; +} + /* * Routine to perform timeout driven calculations. This expands the * hashes and does per cpu statistics aggregation. @@ -557,8 +599,14 @@ keg_timeout(uma_keg_t keg) static void zone_timeout(uma_zone_t zone) { + int i; zone_foreach_keg(zone, &keg_timeout); + + ZONE_LOCK(zone); + for (i = 0; i < vm_ndomains; i++) + zone_domain_update_wss(&zone->uz_domain[i]); + ZONE_UNLOCK(zone); } /* @@ -718,7 +766,7 @@ cache_drain(uma_zone_t zone) * XXX: It would good to be able to assert that the zone is being * torn down to prevent improper use of cache_drain(). * - * XXX: We lock the zone before passing into bucket_cache_drain() as + * XXX: We lock the zone before passing into bucket_cache_reclaim() as * it is used elsewhere. Should the tear-down path be made special * there in some form? */ @@ -733,7 +781,7 @@ cache_drain(uma_zone_t zone) cache->uc_allocbucket = cache->uc_freebucket = NULL; } ZONE_LOCK(zone); - bucket_cache_drain(zone); + bucket_cache_reclaim(zone, true); ZONE_UNLOCK(zone); } @@ -769,8 +817,8 @@ cache_drain_safe_cpu(uma_zone_t zone) cache = &zone->uz_cpu[curcpu]; if (cache->uc_allocbucket) { if (cache->uc_allocbucket->ub_cnt != 0) - LIST_INSERT_HEAD(&zone->uz_domain[domain].uzd_buckets, - cache->uc_allocbucket, ub_link); + zone_put_bucket(zone, &zone->uz_domain[domain], + cache->uc_allocbucket, false); else b1 = cache->uc_allocbucket; cache->uc_allocbucket = NULL; @@ -777,8 +825,8 @@ cache_drain_safe_cpu(uma_zone_t zone) } if (cache->uc_freebucket) { if (cache->uc_freebucket->ub_cnt != 0) - LIST_INSERT_HEAD(&zone->uz_domain[domain].uzd_buckets, - cache->uc_freebucket, ub_link); + zone_put_bucket(zone, &zone->uz_domain[domain], + cache->uc_freebucket, false); else b2 = cache->uc_freebucket; cache->uc_freebucket = NULL; @@ -799,7 +847,7 @@ cache_drain_safe_cpu(uma_zone_t zone) * Zone lock must not be held on call this function. */ static void -cache_drain_safe(uma_zone_t zone) +pcpu_cache_drain_safe(uma_zone_t zone) { int cpu; @@ -827,22 +875,24 @@ static void } /* - * Drain the cached buckets from a zone. Expects a locked zone on entry. + * Reclaim cached buckets from a zone. All buckets are reclaimed if the caller + * requested a drain, otherwise the per-domain caches are trimmed to either + * estimated working set size. */ static void -bucket_cache_drain(uma_zone_t zone) +bucket_cache_reclaim(uma_zone_t zone, bool drain) { uma_zone_domain_t zdom; uma_bucket_t bucket; - int i; + int i, target; - /* - * Drain the bucket queues and free the buckets. - */ for (i = 0; i < vm_ndomains; i++) { zdom = &zone->uz_domain[i]; - while ((bucket = LIST_FIRST(&zdom->uzd_buckets)) != NULL) { - LIST_REMOVE(bucket, ub_link); + target = drain ? 0 : zdom->uzd_wss; + while (zdom->uzd_nitems > target) { + bucket = zone_try_fetch_bucket(zone, zdom, false); + if (bucket == NULL) + break; ZONE_UNLOCK(zone); bucket_drain(zone, bucket); bucket_free(zone, bucket, NULL); @@ -851,8 +901,8 @@ static void } /* - * Shrink further bucket sizes. Price of single zone lock collision - * is probably lower then price of global cache drain. + * Shrink the zone bucket size to ensure that the per-CPU caches + * don't grow too large. */ if (zone->uz_count > zone->uz_count_min) zone->uz_count--; @@ -950,7 +1000,7 @@ finished: } static void -zone_drain_wait(uma_zone_t zone, int waitok) +zone_reclaim(uma_zone_t zone, int waitok, bool drain) { /* @@ -960,13 +1010,13 @@ static void * when it wakes up. */ ZONE_LOCK(zone); - while (zone->uz_flags & UMA_ZFLAG_DRAINING) { + while (zone->uz_flags & UMA_ZFLAG_RECLAIMING) { if (waitok == M_NOWAIT) goto out; msleep(zone, zone->uz_lockptr, PVM, "zonedrain", 1); } - zone->uz_flags |= UMA_ZFLAG_DRAINING; - bucket_cache_drain(zone); + zone->uz_flags |= UMA_ZFLAG_RECLAIMING; + bucket_cache_reclaim(zone, drain); ZONE_UNLOCK(zone); /* * The DRAINING flag protects us from being freed while @@ -975,19 +1025,26 @@ static void */ zone_foreach_keg(zone, &keg_drain); ZONE_LOCK(zone); - zone->uz_flags &= ~UMA_ZFLAG_DRAINING; + zone->uz_flags &= ~UMA_ZFLAG_RECLAIMING; wakeup(zone); out: ZONE_UNLOCK(zone); } -void +static void zone_drain(uma_zone_t zone) { - zone_drain_wait(zone, M_NOWAIT); + zone_reclaim(zone, M_NOWAIT, true); } +static void +zone_trim(uma_zone_t zone) +{ + + zone_reclaim(zone, M_NOWAIT, false); +} + /* * Allocate a new slab for a keg. This does not insert the slab onto a list. * @@ -1867,7 +1924,7 @@ zone_dtor(void *arg, int size, void *udata) * released and then refilled before we * remove it... we dont care for now */ - zone_drain_wait(zone, M_WAITOK); + zone_reclaim(zone, M_WAITOK, true); /* * Unlink all of our kegs. */ @@ -2513,11 +2570,9 @@ zalloc_start: zdom = &zone->uz_domain[0]; else zdom = &zone->uz_domain[domain]; - if ((bucket = LIST_FIRST(&zdom->uzd_buckets)) != NULL) { + if ((bucket = zone_try_fetch_bucket(zone, zdom, true)) != NULL) { KASSERT(bucket->ub_cnt != 0, ("uma_zalloc_arg: Returning an empty bucket.")); - - LIST_REMOVE(bucket, ub_link); cache->uc_allocbucket = bucket; ZONE_UNLOCK(zone); goto zalloc_start; @@ -2554,9 +2609,11 @@ zalloc_start: if (cache->uc_allocbucket != NULL || (zone->uz_flags & UMA_ZONE_NUMA && domain != PCPU_GET(domain))) - LIST_INSERT_HEAD(&zdom->uzd_buckets, bucket, ub_link); - else + zone_put_bucket(zone, zdom, bucket, false); + else { cache->uc_allocbucket = bucket; + zdom->uzd_imax += bucket->ub_cnt; + } ZONE_UNLOCK(zone); goto zalloc_start; } @@ -3164,7 +3221,7 @@ zfree_start: bucket_free(zone, bucket, udata); goto zfree_restart; } else - LIST_INSERT_HEAD(&zdom->uzd_buckets, bucket, ub_link); + zone_put_bucket(zone, zdom, bucket, true); } /* @@ -3599,18 +3656,30 @@ uma_prealloc(uma_zone_t zone, int items) } /* See uma.h */ -static void -uma_reclaim_locked(bool kmem_danger) +void +uma_reclaim(int req) { CTR0(KTR_UMA, "UMA: vm asked us to release pages!"); - sx_assert(&uma_drain_lock, SA_XLOCKED); + sx_xlock(&uma_drain_lock); bucket_enable(); - zone_foreach(zone_drain); - if (vm_page_count_min() || kmem_danger) { - cache_drain_safe(NULL); + + switch (req) { + case UMA_RECLAIM_TRIM: + zone_foreach(zone_trim); + break; + case UMA_RECLAIM_DRAIN: + case UMA_RECLAIM_DRAIN_CPU: zone_foreach(zone_drain); + if (req == UMA_RECLAIM_DRAIN_CPU) { + pcpu_cache_drain_safe(NULL); + zone_foreach(zone_drain); + } + break; + default: + panic("unhandled reclamation request %d", req); } + /* * Some slabs may have been freed but this zone will be visited early * we visit again so that we can free pages that are empty once other @@ -3618,14 +3687,6 @@ uma_prealloc(uma_zone_t zone, int items) */ zone_drain(slabzone); bucket_zone_drain(); -} - -void -uma_reclaim(void) -{ - - sx_xlock(&uma_drain_lock); - uma_reclaim_locked(false); sx_xunlock(&uma_drain_lock); } @@ -3650,10 +3711,8 @@ uma_reclaim_worker(void *arg __unused) hz); sx_xunlock(&uma_drain_lock); EVENTHANDLER_INVOKE(vm_lowmem, VM_LOW_KMEM); - sx_xlock(&uma_drain_lock); - uma_reclaim_locked(true); + uma_reclaim(UMA_RECLAIM_DRAIN_CPU); atomic_store_int(&uma_reclaim_needed, 0); - sx_xunlock(&uma_drain_lock); /* Don't fire more than once per-second. */ pause("umarclslp", hz); } @@ -3660,6 +3719,27 @@ uma_reclaim_worker(void *arg __unused) } /* See uma.h */ +void +uma_zreclaim(uma_zone_t zone, int req) +{ + + switch (req) { + case UMA_RECLAIM_TRIM: + zone_trim(zone); + break; + case UMA_RECLAIM_DRAIN: + zone_drain(zone); + break; + case UMA_RECLAIM_DRAIN_CPU: + pcpu_cache_drain_safe(zone); + zone_drain(zone); + break; + default: + panic("unhandled reclamation request %d", req); + } +} + +/* See uma.h */ int uma_zone_exhausted(uma_zone_t zone) { @@ -3852,7 +3932,7 @@ uma_print_zone(uma_zone_t zone) * directly so that we don't have to. */ static void -uma_zone_sumstat(uma_zone_t z, int *cachefreep, uint64_t *allocsp, +uma_zone_sumstat(uma_zone_t z, long *cachefreep, uint64_t *allocsp, uint64_t *freesp, uint64_t *sleepsp) { uma_cache_t cache; @@ -3907,7 +3987,6 @@ sysctl_vm_zone_stats(SYSCTL_HANDLER_ARGS) struct uma_stream_header ush; struct uma_type_header uth; struct uma_percpu_stat *ups; - uma_bucket_t bucket; uma_zone_domain_t zdom; struct sbuf sbuf; uma_cache_t cache; @@ -3967,9 +4046,7 @@ sysctl_vm_zone_stats(SYSCTL_HANDLER_ARGS) for (i = 0; i < vm_ndomains; i++) { zdom = &z->uz_domain[i]; - LIST_FOREACH(bucket, &zdom->uzd_buckets, - ub_link) - uth.uth_zone_free += bucket->ub_cnt; + uth.uth_zone_free += zdom->uzd_nitems; } uth.uth_allocs = z->uz_allocs; uth.uth_frees = z->uz_frees; @@ -4169,12 +4246,11 @@ uma_dbg_free(uma_zone_t zone, uma_slab_t slab, voi #ifdef DDB DB_SHOW_COMMAND(uma, db_show_uma) { - uma_bucket_t bucket; uma_keg_t kz; uma_zone_t z; - uma_zone_domain_t zdom; uint64_t allocs, frees, sleeps; - int cachefree, i; + long cachefree; + int i; db_printf("%18s %8s %8s %8s %12s %8s %8s\n", "Zone", "Size", "Used", "Free", "Requests", "Sleeps", "Bucket"); @@ -4191,13 +4267,10 @@ DB_SHOW_COMMAND(uma, db_show_uma) if (!((z->uz_flags & UMA_ZONE_SECONDARY) && (LIST_FIRST(&kz->uk_zones) != z))) cachefree += kz->uk_free; - for (i = 0; i < vm_ndomains; i++) { - zdom = &z->uz_domain[i]; - LIST_FOREACH(bucket, &zdom->uzd_buckets, - ub_link) - cachefree += bucket->ub_cnt; - } - db_printf("%18s %8ju %8jd %8d %12ju %8ju %8u\n", + for (i = 0; i < vm_ndomains; i++) + cachefree += z->uz_domain[i].uzd_nitems; + + db_printf("%18s %8ju %8jd %8ld %12ju %8ju %8u\n", z->uz_name, (uintmax_t)kz->uk_size, (intmax_t)(allocs - frees), cachefree, (uintmax_t)allocs, sleeps, z->uz_count); @@ -4209,22 +4282,18 @@ DB_SHOW_COMMAND(uma, db_show_uma) DB_SHOW_COMMAND(umacache, db_show_umacache) { - uma_bucket_t bucket; uma_zone_t z; - uma_zone_domain_t zdom; uint64_t allocs, frees; - int cachefree, i; + long cachefree; + int i; db_printf("%18s %8s %8s %8s %12s %8s\n", "Zone", "Size", "Used", "Free", "Requests", "Bucket"); LIST_FOREACH(z, &uma_cachezones, uz_link) { uma_zone_sumstat(z, &cachefree, &allocs, &frees, NULL); - for (i = 0; i < vm_ndomains; i++) { - zdom = &z->uz_domain[i]; - LIST_FOREACH(bucket, &zdom->uzd_buckets, ub_link) - cachefree += bucket->ub_cnt; - } - db_printf("%18s %8ju %8jd %8d %12ju %8u\n", + for (i = 0; i < vm_ndomains; i++) + cachefree += z->uz_domain[i].uzd_nitems; + db_printf("%18s %8ju %8jd %8ld %12ju %8u\n", z->uz_name, (uintmax_t)z->uz_size, (intmax_t)(allocs - frees), cachefree, (uintmax_t)allocs, z->uz_count); Index: /usr/src/sys/vm/uma_int.h =================================================================== --- /usr/src/sys/vm/uma_int.h (revision 338128) +++ /usr/src/sys/vm/uma_int.h (working copy) @@ -303,6 +303,10 @@ typedef struct uma_klink *uma_klink_t; struct uma_zone_domain { LIST_HEAD(,uma_bucket) uzd_buckets; /* full buckets */ + long uzd_nitems; /* total item count */ + long uzd_imax; /* maximum item count this period */ + long uzd_imin; /* minimum item count this period */ + long uzd_wss; /* working set size estimate */ }; typedef struct uma_zone_domain * uma_zone_domain_t; @@ -370,7 +374,7 @@ struct uma_zone { * These flags must not overlap with the UMA_ZONE flags specified in uma.h. */ #define UMA_ZFLAG_MULTI 0x04000000 /* Multiple kegs in the zone. */ -#define UMA_ZFLAG_DRAINING 0x08000000 /* Running zone_drain. */ +#define UMA_ZFLAG_RECLAIMING 0x08000000 /* Running zone_reclaim(). */ #define UMA_ZFLAG_BUCKET 0x10000000 /* Bucket zone. */ #define UMA_ZFLAG_INTERNAL 0x20000000 /* No offpage no PCPU. */ #define UMA_ZFLAG_FULL 0x40000000 /* Reached uz_maxpages */ @@ -422,11 +426,12 @@ void uma_large_free(uma_slab_t slab); mtx_init(&(z)->uz_lock, (z)->uz_name, \ "UMA zone", MTX_DEF | MTX_DUPOK); \ } while (0) - + #define ZONE_LOCK(z) mtx_lock((z)->uz_lockptr) #define ZONE_TRYLOCK(z) mtx_trylock((z)->uz_lockptr) #define ZONE_UNLOCK(z) mtx_unlock((z)->uz_lockptr) #define ZONE_LOCK_FINI(z) mtx_destroy(&(z)->uz_lock) +#define ZONE_LOCK_ASSERT(z) mtx_assert((z)->uz_lockptr, MA_OWNED) /* * Find a slab within a hash table. This is used for OFFPAGE zones to lookup Index: /usr/src/sys/vm/vm_pageout.c =================================================================== --- /usr/src/sys/vm/vm_pageout.c (revision 338128) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -1874,9 +1874,12 @@ vm_pageout_lowmem(void) /* * We do this explicitly after the caches have been - * drained above. + * drained above. If we have a severe page shortage on + * our hands, completely drain all UMA zones. Otherwise, + * just prune the caches. */ - uma_reclaim(); + uma_reclaim(vm_page_count_min() ? UMA_RECLAIM_DRAIN_CPU : + UMA_RECLAIM_TRIM); return (true); } return (false); -- Test scenario: misc/pmc.sh