GENERIC HEAD from 2015-04-25 22:42:03 UTC, r281995M, vmcore.803 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb 4718592K of memory above 4GB ignored Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r281995M: Sun Apr 26 13:58:29 CEST 2015 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 3.6.0 (tags/RELEASE_360/final 230434) 20150225 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT: running with driver "vga". CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3528839168 (3365 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... WARNING: / was not properly dismounted Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Starting file system checks: /dev/ad4s1a: 6843 files, 988533 used, 24482 free (1178 frags, 2913 blocks, 0.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 53012664 free (19576 frags, 6624136 blocks, 0.0% fragmentation) /dev/ad4s1f: 21343 files, 281947 used, 731068 free (8924 frags, 90268 blocks, 0.9% fragmentation) /dev/ad4s1d: 437624 files, 5951926 used, 4202273 free (302233 frags, 487505 blocks, 3.0% fragmentation) ** SU+J Recovering /dev/ad4s1e ** Reading 33554432 byte journal from inode 6. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 10 journal records in 512 bytes for 62.50% utilization ** Freed 0 inodes (0 dirs) 0 blocks, and 0 frags. ***** FILE SYSTEM MARKED CLEAN ***** Mounting local file systems:. Setting hostname: x4.osted.lan. Feeding entropy:. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=23 media: Ethernet autoselect (none) status: no carrier Starting devd. Starting pflogd: add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc46 /usr/local/lib/gcc48 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/pth /usr/local/lib/qt4 a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. Starting watchdogd. savecore: reboot after panic: 1 vncache entries remaining Apr 26 14:31:17 x4 savecore: reboot after panic: 1 vncache entries remaining lock order reversal: 1st 0xe44f66e8 bufwait (bufwait) @ kern/vfs_bio.c:3126 2nd 0xc9e4fe00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c9e15a98,e44f6698,edbab6b8,c0bfbb7a,...) at db_trace_self_wrapper+0x2a/frame 0xedbab688 kdb_backtrace(c11e6b25,c9e4fe00,c121a50d,c858b8c0,c121a152,...) at kdb_backtrace+0x2d/frame 0xedbab6f0 witness_checkorder(c9e4fe00,9,c121a149,11d,0,...) at witness_checkorder+0xd2e/frame 0xedbab744 _sx_xlock(c9e4fe00,0,c121a149,11d,0,...) at _sx_xlock+0x74/frame 0xedbab774 ufsdirhash_add(c93291e0,edbab890,5a8,edbab800,edbab804,...) at ufsdirhash_add+0x4a/frame 0xedbab7a8 ufs_direnter(c9e159d8,c9ab4b40,edbab890,edbabb9c,0,...) at ufs_direnter+0x654/frame 0xedbab830 ufs_makeinode(edbabb88,edbabb9c,c9e159d8,c148e0e4,c9e159d8,...) at ufs_makeinode+0x533/frame 0xedbab9ac ufs_create(edbabaa8,c13aa47b,c1213266,c9e159d8,edbabaa4,...) at ufs_create+0x30/frame 0xedbab9d0 VOP_CREATE_APV(c147ad9c,edbabaa8,edbabb9c,edbaba38,c1f9b900,...) at VOP_CREATE_APV+0x12f/frame 0xedbaba00 vn_open_cred(edbabb40,edbabbcc,180,0,c93a4300,c93f3968) at vn_open_cred+0x282/frame 0xedbabad0 vn_open(edbabb40,edbabbcc,180,c93f3968,804cad5,...) at vn_open+0x3d/frame 0xedbabaf8 kern_openat(c935e930,ffffff9c,804cad5,0,601,180) at kern_openat+0x2ec/frame 0xedbabbf0 sys_openat(c935e930,edbabca8,c11d731d,4,c10646c7,...) at sys_openat+0x3a/frame 0xedbabc18 syscall(edbabce8) at syscall+0x34a/frame 0xedbabcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbabcdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2815b83f, esp = 0xbfbfe664, ebp = 0xbfbfe680 --- savecore: writing core to /var/crash/vmcore.802 Writing crash summary to /var/crash/core.txt.802. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Starting casperd. Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Starting smartd. Apr 26 14:31:54 x4 smartd[665]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Apr 26 14:31:54 x4 smartd[665]: see the following Seagate web pages: Apr 26 14:31:54 x4 smartd[665]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Apr 26 14:31:54 x4 smartd[665]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Starting default moused. Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc93695d4 ufs (ufs) @ kern/vfs_subr.c:2176 2nd 0xe451dfa0 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:263 3rd 0xca19f034 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,f88e5630,c0ab278f,4000,0,...) at db_trace_self_wrapper+0x2a/frame 0xf88e55d8 kdb_backtrace(c11e6b3e,ca19f034,c11cce89,c858b858,c11f0185,...) at kdb_backtrace+0x2d/frame 0xf88e563c witness_checkorder(ca19f034,9,c11f017c,880,ca19f0a0,...) at witness_checkorder+0xd2e/frame 0xf88e5690 __lockmgr_args(ca19f034,80100,ca19f0a0,0,0,...) at __lockmgr_args+0x935/frame 0xf88e576c ffs_lock(f88e57e8,c8589a48,c15a89e0,c8589a48,c8583178,...) at ffs_lock+0x98/frame 0xf88e57a4 VOP_LOCK1_APV(c147ad9c,f88e57e8,0,c9353620,c148ffec,...) at VOP_LOCK1_APV+0x10a/frame 0xf88e57d0 _vn_lock(ca19f000,80100,c11f017c,880,c11ef39c,...) at _vn_lock+0xba/frame 0xf88e5810 vget(ca19f000,80100,c9353620,57,80100,...) at vget+0x74/frame 0xf88e5848 vfs_hash_get(c9725bb0,627800,80000,c9353620,f88e5940,...) at vfs_hash_get+0xe7/frame 0xf88e5874 ffs_vgetf(c9725bb0,627800,80000,f88e5940,1,...) at ffs_vgetf+0x44/frame 0xf88e58d0 softdep_sync_buf(c93695a0,e451df50,1,0,0,...) at softdep_sync_buf+0x36b/frame 0xf88e5958 ffs_syncvnode(c93695a0,1,0,5fa,c8583588,...) at ffs_syncvnode+0x2aa/frame 0xf88e59b0 ffs_sync(c9725bb0,4,c11f1c40,733,c9353620,...) at ffs_sync+0x1e9/frame 0xf88e5a38 vfs_write_suspend(c9725bb0,0,c121a6fa,79,72,...) at vfs_write_suspend+0x1bf/frame 0xf88e5a74 vfs_write_suspend_umnt(c9725bb0,c9353620,f88e5ac0,c11f017c,8ce,...) at vfs_write_suspend_umnt+0x42/frame 0xf88e5a9c ffs_unmount(c9725bb0,8080000,f88e5b20,510,c15a89e0,...) at ffs_unmount+0xa7/frame 0xf88e5ae8 dounmount(c9725bb0,8080000,c9353620,492,14f,...) at dounmount+0x4c9/frame 0xf88e5b48 sys_unmount(c9353620,f88e5ca8,c11d731d,4,c10646c7,...) at sys_unmount+0x2d4/frame 0xf88e5c18 syscall(f88e5ce8) at syscall+0x34a/frame 0xf88e5cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88e5cdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe69c, ebp = 0xbfbfe768 --- lock order reversal: 1st 0xc93af034 ufs (ufs) @ kern/vfs_mount.c:1229 2nd 0xc942c46c devfs (devfs) @ ufs/ffs/ffs_softdep.c:1904 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c15a89e0,8,10,c1453ae8,...) at db_trace_self_wrapper+0x2a/frame 0xf88e5740 kdb_backtrace(c11e6b25,c942c46c,c11d91bc,c858b6b8,c1215d24,...) at kdb_backtrace+0x2d/frame 0xf88e57a4 witness_checkorder(c942c46c,9,c1215d1b,770,0,...) at witness_checkorder+0xd2e/frame 0xf88e57f8 __lockmgr_args(c942c46c,80400,c942c4d8,0,0,0,c1215d1b,770) at __lockmgr_args+0x935/frame 0xf88e58d4 vop_stdlock(f88e5948,c972d000,f88e5940,c0b5d569,c972d000,...) at vop_stdlock+0x53/frame 0xf88e5904 VOP_LOCK1_APV(c14432b0,f88e5948,f88e5970,c0dd6145,c148ffec,...) at VOP_LOCK1_APV+0x10a/frame 0xf88e5930 _vn_lock(c942c438,80400,c1215d1b,770,f88e5a0c,...) at _vn_lock+0xba/frame 0xf88e5970 softdep_flushworklist(c9725bb0,f88e5a08,c9353620,60e,c8583588,...) at softdep_flushworklist+0x68/frame 0xf88e59b0 ffs_sync(c9725bb0,4,c11f1c40,733,c9353620,...) at ffs_sync+0x2ae/frame 0xf88e5a38 vfs_write_suspend(c9725bb0,0,c121a6fa,79,72,...) at vfs_write_suspend+0x1bf/frame 0xf88e5a74 vfs_write_suspend_umnt(c9725bb0,c9353620,f88e5ac0,c11f017c,8ce,...) at vfs_write_suspend_umnt+0x42/frame 0xf88e5a9c ffs_unmount(c9725bb0,8080000,f88e5b20,510,c15a89e0,...) at ffs_unmount+0xa7/frame 0xf88e5ae8 dounmount(c9725bb0,8080000,c9353620,492,14f,...) at dounmount+0x4c9/frame 0xf88e5b48 sys_unmount(c9353620,f88e5ca8,c11d731d,4,c10646c7,...) at sys_unmount+0x2d4/frame 0xf88e5c18 syscall(f88e5ce8) at syscall+0x34a/frame 0xf88e5cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88e5cdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe69c, ebp = 0xbfbfe768 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 391535 files, 65301317 used, 53012664 free (19568 frags, 6624137 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp. Starting inetd. Sun Apr 26 14:35:12 CEST 2015 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Apr 26 14:42:19 x4 su: pho to root on /dev/pts/0 lock order reversal: 1st 0xcd5265d4 ufs (ufs) @ kern/vfs_vnops.c:883 2nd 0xc1631920 proctree (proctree) @ kern/subr_prf.c:146 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,2b1620,32383633,c9313734,f88f657c,...) at db_trace_self_wrapper+0x2a/frame 0xf88f6538 kdb_backtrace(c11e6b25,c1631920,c11dc7cc,c8582068,c11e3b94,...) at kdb_backtrace+0x2d/frame 0xf88f65a0 witness_checkorder(c1631920,1,c11e3b8b,92,0,...) at witness_checkorder+0xd2e/frame 0xf88f65f4 _sx_slock(c1631920,0,c11e3b8b,92,10,...) at _sx_slock+0x6e/frame 0xf88f6620 uprintf(c12144ce,c97270d4,6286,c986ac28,2,...) at uprintf+0x46/frame 0xf88f6678 ffs_alloc(cd4f8348,0,0,6a4908,0,...) at ffs_alloc+0x469/frame 0xf88f66e0 ffs_balloc_ufs2(cd5265a0,0,0,1000,c93a2800,...) at ffs_balloc_ufs2+0xc97/frame 0xf88f682c ffs_write(f88f69b0,c13aa7e5,cd5265d4,cd5265a0,c148e0e4,...) at ffs_write+0x592/frame 0xf88f68b8 VOP_WRITE_APV(c147ad9c,f88f69b0,cd5265a0,373,2,...) at VOP_WRITE_APV+0x1ac/frame 0xf88f6978 vn_write(cb1b8c40,f88f6be8,c93a2800,1,ca2b1620,...) at vn_write+0x328/frame 0xf88f69e0 vn_io_fault_doio(ca2b1620,f88f6a4c,c0baf43c,c15795dc,0,...) at vn_io_fault_doio+0x34/frame 0xf88f6a28 vn_io_fault1(ca2b1620,0,0,ffffffff,7fffffff,...) at vn_io_fault1+0x6d/frame 0Apr 26 14:44:00 xfx4 kernel: pid 288f65222 (dd), uid 2af8 inumber 1742863 on /var: filesyvn_istem full o_fault(cb1b8c40,f88f6be8,c93a2800,0,ca2b1620,...) at vn_io_fault+0x1fa/frame 0xf88f6b4c dofilewrite(cb1b8c40,f88f6be8,ffffffff,ffffffff,0,...) at dofilewrite+0xa0/frame 0xf88f6b80 kern_writev(ca2b1620,4,f88f6be8,c15a8358,28806000,...) at kern_writev+0x93/frame 0xf88f6bc8 sys_write(ca2b1620,f88f6ca8,c11d731d,4,c10646c7,...) at sys_write+0x59/frame 0xf88f6c18 syscall(f88f6ce8) at syscall+0x34a/frame 0xf88f6cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88f6cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281a5eef, esp = 0xbfbfed88, ebp = 0xbfbfedac --- lock order reversal: 1st 0xc9e33034 ufs (ufs) @ kern/vfs_mount.c:848 2nd 0xcdccb46c nfs (nfs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,cdd23162,f88f6728,f88f6708,c0b45436,...) at db_trace_self_wrapper+0x2a/frame 0xf88f66d8 kdb_backtrace(c11e6b25,cdccb46c,c11c7004,c858c148,c11f0185,...) at kdb_backtrace+0x2d/frame 0xf88f673c witness_checkorder(cdccb46c,9,c11f017c,880,cdccb4d8,...) at witness_checkorder+0xd2e/frame 0xf88f6790 __lockmgr_args(cdccb46c,80100,cdccb4d8,0,0,0,c11f017c,880) at __lockmgr_args+0x935/frame 0xf88f686c nfs_lock1(f88f68e0,c8589a48,c15a8618,c8589a48,c8583178,...) at nfs_lock1+0x9c/frame 0xf88f689c VOP_LOCK1_APV(c144472c,f88f68e0,0,cdd23140,c148ffec,...) at VOP_LOCK1_APV+0x10a/frame 0xf88f68c8 _vn_lock(cdccb438,80100,c11f017c,880,c11ef39c,...) at _vn_lock+0xba/frame 0xf88f6908 vget(cdccb438,80100,ca2b1620,57,80100,...) at vget+0x74/frame 0xf88f6940 vfs_hash_get(c92f28c4,3a941847,80000,ca2b1620,f88f69ac,...) at vfs_hash_get+0xe7/frame 0xf88f696c ncl_nget(c92f28c4,cdd1d88c,1c,f88f69d8,80000,...) at ncl_nget+0xc2/frame 0xf88f69bc nfs_root(c92f28c4,80000,f88f6bd0,359,c88d1d50,...) at nfs_root+0x3c/frame 0xf88f69e8 vfs_donmount(ca2b1620,0,0,c9c6c100,c9c6c100,...) at vfs_donmount+0x15f6/frame 0xf88f6bf0 sys_nmount(ca2b1620,f88f6ca8,c11d731d,4,c10646c7,...) at sys_nmount+0x7a/frame 0xf88f6c18 syscall(f88f6ce8) at syscall+0x34a/frame 0xf88f6cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88f6cdc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dbabb, esp = 0xbfbfe4fc, ebp = 0xbfbfedd4 --- lock order reversal: 1st 0xe44e2030 bufwait (bufwait) @ kern/vfs_bio.c:2354 2nd 0xc9ca62f4 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:265 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c15a8618,c8587140,c8583930,f8a342a4,...) at db_trace_self_wrapper+0x2a/frame 0xf8a34238 kdb_backtrace(c11e6b25,c9ca62f4,c11d50e4,c858ba60,c11ebd22,...) at kdb_backtrace+0x2d/frame 0xf8a3429c witness_checkorder(c9ca62f4,9,c11ebd19,109,0,...) at witness_checkorder+0xd2e/frame 0xf8a342f0 _sx_xlock(c9ca62f4,40,c11ebd19,109,0,...) at _sx_xlock+0x74/frame 0xf8a34320 sblock(c9ca62bc,1,30,f8a34398,c0e42186,...) at sblock+0x56/frame 0xf8a3433c sosend_generic(c9ca61c8,0,0,c9d4b200,0,...) at sosend_generic+0xd3/frame 0xf8a343b0 clnt_vc_call(c9b28e80,f8a34570,10,c9df3100,f8a34758,...) at clnt_vc_call+0x4a2/frame 0xf8a34488 clnt_reconnect_call(c9b28ec0,f8a34570,10,c9df3100,f8a34758,...) at clnt_reconnect_call+0xcc/frame 0xf8a344f0 newnfs_request(f8a34758,cdd1d800,0,cdd1d910,cad5d870,ca289930,c93a4300,186a3,3,0,1,0,0) at newnfs_request+0x8e5/frame 0xf8a345b8 nfscl_request(f8a34758,cad5d870,ca289930,c93a4300,0,...) at nfscl_request+0x8c/frame 0xf8a34600 nfsrpc_readdir(cad5d870,f8a34928,f8a34848,c93a4300,ca289930,...) at nfsrpc_readdir+0x371/frame 0xf8a34810 ncl_readdirrpc(cad5d870,f8a34928,c93a4300,ca289930,8,...) at ncl_readdirrpc+0xff/frame 0xf8a348f0 ncl_doio(cad5d870,e44e1fe0,c93a4300,ca289930,0,...) at ncl_doio+0x13d/frame 0xf8a34968 ncl_bioread(cad5d870,f8a34b98,0,c93a4300,c11de267,...) at ncl_bioread+0xadc/frame 0xf8a34a68 nfs_readdir(f8a34bb8,c13aab7c,c12133d0,c11f19c8,ff3,...) at nfs_readdir+0x1a8/frame 0xf8a34b18 VOP_READDIR_APV(c144472c,f8a34bb8,c11f19c8,ff3,0,...) at VOP_READDIR_APV+0x12f/frame 0xf8a34b48 kern_getdirentries(ca289930,0,2887f000,1000,f8a34c08,...) at kern_getdirentries+0x1e3/frame 0xf8a34be4 sys_getdirentries(ca289930,f8a34ca8,c11d731d,4,c10646c7,...) at sys_getdirentries+0x42/frame 0xf8a34c18 syscall(f8a34ce8) at syscall+0x34a/frame 0xf8a34cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8a34cdc --- syscall (196, FreeBSD ELF32, sys_getdirentries), eip = 0x281549bb, esp = 0xbfbfcb28, ebp = 0xbfbfcb5c --- 20150426 16:36:40 all (1/315): core2.sh Apr 26 16:56:03 x4 su: pho to root on /dev/pts/1 20150426 18:07:52 all (1/315): tmpfs10.sh 20150426 18:11:14 all (2/315): nullfs11.sh 20150426 18:15:17 all (3/315): umountf.sh 20150426 18:15:51 all (4/315): fpclone2.sh 20150426 18:15:56 all (5/315): msync2.sh 20150426 18:17:33 all (6/315): flock_open_close.sh lock order reversal: 1st 0xe4502c88 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:263 2nd 0xcf03aa1c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2298 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c0c179f9,c147ad9c,f8b2e6a0,d0272438,...) at db_trace_self_wrapper+0x2a/frame 0xf8b2e680 kdb_backtrace(c11e6b25,cf03aa1c,c1215866,c858c558,c1215488,...) at kdb_backtrace+0x2d/frame 0xf8b2e6e4 witness_checkorder(cf03aa1c,9,c121547f,8fa,cd6b1eb0,...) at witness_checkorder+0xd2e/frame 0xf8b2e738 __lockmgr_args(cf03aa1c,80900,cd6b1eb0,0,0,...) at __lockmgr_args+0x935/frame 0xf8b2e818 ffs_copyonwrite(cd6b1e10,e4502c38,e4502c38,cac93e10,41c0,...) at ffs_copyonwrite+0x1cf/frame 0xf8b2e88c ffs_geom_strategy(cd6b1ed0,e4502c38,f8b2e938,cef24310,f8b2e8f0,...) at ffs_geom_strategy+0xd2/frame 0xf8b2e8a8 ufs_strategy(f8b2e938,c13aad9f,4,0,1000,...) at ufs_strategy+0xb1/frame 0xf8b2e8f0 VOP_STRATEGY_APV(c147ad9c,f8b2e938,272508,c4124184,c1490304,...) at VOP_STRATEGY_APV+0x13e/frame 0xf8b2e920 bufstrategy(d02724f8,e4502c38,1175,d02724f8,f8b2e994,...) at bufstrategy+0x4d/frame 0xf8b2e950 bufwrite(e4502c38,e4502c38,1,0,0,...) at bufwrite+0x160/frame 0xf8b2e994 ffs_syncvnode(d0272438,1,0,0,c147ad9c,...) at ffs_syncvnode+0x2f3/frame 0xf8b2e9ec ffs_fsync(f8b2ea60,c13aa968,0,cac93e10,d0272444,...) at ffs_fsync+0x26/frame 0xf8b2ea10 VOP_FSYNC_APV(c147ad9c,f8b2ea60,400800c,0,d0272438,...) at VOP_FSYNC_APV+0x12f/frame 0xf8b2ea40 ufs_remove(f8b2ebd8,c13aa9b6,ade,f8b2ebd4,10,...) at ufs_remove+0xdf/frame 0xf8b2ea84 VOP_REMOVE_APV(c147ad9c,f8b2ebd8,d15159d8,f8b2ebac,bfbfea96,...) at VOP_REMOVE_APV+0x148/frame 0xf8b2eab0 kern_unlinkat(cef24310,ffffff9c,bfbfea96,0,0) at kern_unlinkat+0x266/frame 0xf8b2ebfc sys_unlink(cef24310,f8b2eca8,c11d731d,4,c10646c7,...) at sys_unlink+0x32/frame 0xf8b2ec18 syscall(f8b2ece8) at syscall+0x34a/frame 0xf8b2ecdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8b2ecdc --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x2818e4b7, esp = 0xbfbfe75c, ebp = 0xbfbfe7e8 --- lock order reversal: 1st 0xcf03aa1c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:944 2nd 0xd1515a0c ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1629 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c156b29c,f8b2e650,c0b8980e,c156b29c,...) at db_trace_self_wrapper+0x2a/frame 0xf8b2e618 kdb_backtrace(c11e6b25,d1515a0c,c11cce89,c858b858,c1215488,...) at kdb_backtrace+0x2d/frame 0xf8b2e67c witness_checkorder(d1515a0c,9,c121547f,65d,0,...) at witness_checkorder+0xd2e/frame 0xf8b2e6d0 __lockmgr_args(d1515a0c,80000,0,0,0,...) at __lockmgr_args+0x935/frame 0xf8b2e7b0 ffs_snapremove(d15159d8,c1215350,fffffffa,1,c15d5b10,...) at ffs_snapremove+0x119/frame 0xf8b2e810 ffs_truncate(d15159d8,0,0,c00,0,...) at ffs_truncate+0x2a1/frame 0xf8b2e9c0 ufs_inactive(f8b2ea48,c13aac29,8,c11f017c,9db,...) at ufs_inactive+0x1ed/frame 0xf8b2e9f0 VOP_INACTIVE_APV(c147ad9c,f8b2ea48,c11f017c,9db,cf03aa1c,...) at VOP_INACTIVE_APV+0x12f/frame 0xf8b2ea20 vinactive(d15159d8,cef24310,c11f017c,8ce,cef24310,...) at vinactive+0x182/frame 0xf8b2ea68 vputx(d15159d8,ffffffdf,d15159d8,f8b2ebac,bfbfea96,...) at vputx+0x3c3/frame 0xf8b2eab0 kern_unlinkat(cef24310,ffffff9c,bfbfea96,0,0) at kern_unlinkat+0x2ae/frame 0xf8b2ebfc sys_unlink(cef24310,f8b2eca8,c11d731d,4,c10646c7,...) at sys_unlink+0x32/frame 0xf8b2ec18 syscall(f8b2ece8) at syscall+0x34a/frame 0xf8b2ecdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8b2ecdc --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x2818e4b7, esp = 0xbfbfe75c, ebp = 0xbfbfe7e8 --- 20150426 18:24:05 all (7/315): md3.sh 20150426 18:34:26 all (8/315): link.sh lock order reversal: 1st 0xcbdaa46c ufs (ufs) @ kern/vfs_subr.c:2176 2nd 0xc1631964 allproc (allproc) @ kern/kern_descrip.c:2881 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,80100,c808ca18,c0c18ce8,cbdaa4e8,...) at db_trace_self_wrapper+0x2a/frame 0xc808c9d0 kdb_backtrace(c11e6b25,c1631964,c11d5081,c85820d0,c11d648a,...) at kdb_backtrace+0x2d/frame 0xc808ca38 witness_checkorder(c1631964,1,c11d6481,b41,0,...) at witness_checkorder+0xd2e/frame 0xc808ca8c _sx_slock(c1631964,0,c11d6481,b41,80000,...) at _sx_slock+0x6e/frame 0xc808cab8 mountcheckdirs(cbdaa438,c9e34870,c808cb20,510,0,...) at mountcheckdirs+0x41/frame 0xc808cae8 dounmount(c92f2bb0,8080000,d20f3000,492,14f,...) at dounmount+0x413/frame 0xc808cb48 sys_unmount(d20f3000,c808cca8,c11d731d,4,c156a410,...) at sys_unmount+0x2d4/frame 0xc808cc18 syscall(c808cce8) at syscall+0x34a/frame 0xc808ccdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc808ccdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe0dc, ebp = 0xbfbfe1a8 --- lock order reversal: 1st 0xd66b1a0c tmpfs (tmpfs) @ kern/vfs_subr.c:2176 2nd 0xc1631964 allproc (allproc) @ kern/kern_descrip.c:2881 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c11e7384,20,246,d6073930,...) at db_trace_self_wrapper+0x2a/frame 0xc810d9d0 kdb_backtrace(c11e6b25,c1631964,c11d5081,c85820d0,c11d648a,...) at kdb_backtrace+0x2d/frame 0xc810da38 witness_checkorder(c1631964,1,c11d6481,b41,0,...) at witness_checkorder+0xd2e/frame 0xc810da8c _sx_slock(c1631964,0,c11d6481,b41,d57f4000,...) at _sx_slock+0x6e/frame 0xc810dab8 mountcheckdirs(d66b19d8,c9e34870,c810db20,510,0,...) at mountcheckdirs+0x41/frame 0xc810dae8 dounmount(c9a8f2ec,8080000,d6073930,492,14f,...) at dounmount+0x413/frame 0xc810db48 sys_unmount(d6073930,c810dca8,c11d731d,4,c10646c7,...) at sys_unmount+0x2d4/frame 0xc810dc18 syscall(c810dce8) at syscall+0x34a/frame 0xc810dcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc810dcdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe0dc, ebp = 0xbfbfe1a8 --- 20150426 18:38:39 all (9/315): mountu.sh lock order reversal: 1st 0xd3c3c73c msdosfs (msdosfs) @ kern/vfs_subr.c:2176 2nd 0xcaa985d4 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:843 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,1,c8af7c00,c1569d00,c1569d10,...) at db_trace_self_wrapper+0x2a/frame 0xf8a40578 kdb_backtrace(c11e6b25,caa985d4,c11d91bc,c858b6b8,c11c549e,...) at kdb_backtrace+0x2d/frame 0xf8a405dc witness_checkorder(caa985d4,9,c11c5495,34b,0,...) at witness_checkorder+0xd2e/frame 0xf8a40630 __lockmgr_args(caa985d4,80400,caa98640,0,0,0,c11c5495,34b) at __lockmgr_args+0x935/frame 0xf8a4070c vop_stdlock(f8a40780,c11ef1f3,2d0,100,5d,...) at vop_stdlock+0x53/frame 0xf8a4073c VOP_LOCK1_APV(c14432b0,f8a40780,0,0,c148ffec,...) at VOP_LOCK1_APV+0x10a/frame 0xf8a40768 _vn_lock(caa985a0,80400,c11c5495,34b,c1443760,...) at _vn_lock+0xba/frame 0xf8a407a8 msdosfs_fsync(f8a40860,c13aa968,c0c1b109,c92f1bc0,0,...) at msdosfs_fsync+0x51/frame 0xf8a407e8 VOP_FSYNC_APV(c1443760,f8a40860,cb008310,3c6,f8a40884,...) at VOP_FSYNC_APV+0x12f/frame 0xf8a40818 msdosfs_sync(c92f1bb0,1,0,0,0,...) at msdosfs_sync+0xb2/frame 0xf8a40888 msdosfs_mount(c92f1bb0,d56575f0,4,3ae,d5662af0,...) at msdosfs_mount+0x3dc/frame 0xf8a409e8 vfs_donmount(cb008310,0,0,c8a84700,c8a84700,...) at vfs_donmount+0xfdb/frame 0xf8a40bf0 sys_nmount(cb008310,f8a40ca8,c11d731d,4,c10646c7,...) at sys_nmount+0x7a/frame 0xf8a40c18 syscall(f8a40ce8) at syscall+0x34a/frame 0xf8a40cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8a40cdc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dbabb, esp = 0xbfbfe284, ebp = 0xbfbfe854 --- 20150426 18:39:04 all (10/315): mlockall4.sh 20150426 18:39:10 all (11/315): ldt2.sh 20150426 18:39:10 all (12/315): fpclone.sh 20150426 18:41:15 all (13/315): union.sh 20150426 18:51:37 all (14/315): quota5.sh 20150426 18:51:38 all (15/315): zfs.sh 20150426 18:51:38 all (16/315): marcus.sh *** Excessive run time: marcus.sh, 32 min 20150426 19:24:03 all (17/315): sendfile_shm.sh 20150426 19:26:18 all (18/315): zfs3.sh 20150426 19:26:19 all (19/315): mmap13.sh 20150426 19:26:28 all (20/315): isofs.sh 20150426 19:26:37 all (21/315): nullfs5.sh lock order reversal: 1st 0xd107773c pseudofs (pseudofs) @ kern/vfs_lookup.c:509 2nd 0xc1631964 allproc (allproc) @ kern/kern_proc.c:310 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,c8cba500,40,16d,c81387b0,...) at db_trace_self_wrapper+0x2a/frame 0xc81386e0 kdb_backtrace(c11e6b25,c1631964,c11d5081,c85820d0,c11dc7e6,...) at kdb_backtrace+0x2d/frame 0xc8138744 witness_checkorder(c1631964,1,c11dc7dd,136,0,...) at witness_checkorder+0xd2e/frame 0xc8138798 _sx_slock(c1631964,0,c11dc7dd,136,22e1,...) at _sx_slock+0x6e/frame 0xc81387c4 pfind(22e1,0,c11c964b,7c,c1089e8e,...) at pfind+0x2e/frame 0xc81387e0 pfs_lookup(c81388a0,c13aa426,c813886c,c0c07f18,d107773c,...) at pfs_lookup+0x4e4/frame 0xc8138850 VOP_CACHEDLOOKUP_APV(c1446604,c81388a0,c8138b9c,0,0,...) at VOP_CACHEDLOOKUP_APV+0x12f/frame 0xc8138880 vfs_cache_lookup(c8138938,c13aa3c5,c121332d,c11ef57f,c148e0e4,...) at vfs_cache_lookup+0xe9/frame 0xc81388c8 VOP_LOOKUP_APV(c1446604,c8138938,c8138b9c,1fd,2,...) at VOP_LOOKUP_APV+0x12f/frame 0xc81388f8 lookup(c8138b40,cef5c000,0,d3,c0e25b75,...) at lookup+0x551/frame 0xc8138960 namei(c8138b40,c1453ae8,d57e7620,c1f9b910,c1f9b900,...) at namei+0x50e/frame 0xc8138a00 vn_open_cred(c8138b40,c8138bcc,0,0,c8cba500,cb1b8d58) at vn_open_cred+0x1d9/frame 0xc8138ad0 vn_open(c8138b40,c8138bcc,0,cb1b8d58,8048c6c,...) at vn_open+0x3d/frame 0xc8138af8 kern_openat(d57e7620,ffffff9c,8048c6c,0,0,0) at kern_openat+0x2ec/frame 0xc8138bf0 sys_openat(d57e7620,c8138ca8,c11d731d,4,e,...) at sys_openat+0x3a/frame 0xc8138c18 syscall(c8138ce8) at syscall+0x34a/frame 0xc8138cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc8138cdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2814783f, esp = 0xbfbfe7f8, ebp = 0xbfbfe814 --- 20150426 19:37:34 all (22/315): posix_fadvise.sh 20150426 19:37:38 all (23/315): suj10.sh 20150426 19:38:35 all (24/315): tmpfs4.sh 20150426 19:38:36 all (25/315): msdos2.sh 20150426 19:38:37 all (26/315): kevent5.sh 20150426 19:42:36 all (27/315): sendfile7.sh 20150426 19:48:57 all (28/315): contigmalloc.sh 20150426 19:51:01 all (29/315): umountf8.sh 20150426 19:53:48 all (30/315): dup2.sh 20150426 19:53:51 all (31/315): fts.sh lock order reversal: 1st 0xd668b748 FFS (FFS Lock) @ ufs/ffs/ffs_alloc.c:1945 2nd 0xd3558390 bufobj interlock (bufobj interlock) @ kern/vfs_subr.c:1675 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,a3537,c1596240,c8583318,c8583860,...) at db_trace_self_wrapper+0x2a/frame 0xf8fd9448 kdb_backtrace(c11e6b25,d3558390,c11f01d4,c85899e0,c11f0185,...) at kdb_backtrace+0x2d/frame 0xf8fd94b0 witness_checkorder(d3558390,9,c11f017c,68b,0,...) at witness_checkorder+0xd2e/frame 0xf8fd9504 _rw_wlock_cookie(d35583a0,c11f017c,68b,4,d35583a0,...) at _rw_wlock_cookie+0x78/frame 0xf8fd9538 brelvp(e452e180,c11ed1a4,1e1,781,c8ce609c) at brelvp+0x3a/frame 0xf8fd9560 vfs_vmio_release(0,dac50620,d668b758,4,799,...) at vfs_vmio_release+0x23e/frame 0xf8fd9588 brelse(e452e180,0,c1214444,799,0,...) at brelse+0x58c/frame 0xf8fd95e0 ffs_clusteralloc(2ffc8,0,2,0,e87b0000,...) at ffs_clusteralloc+0x4d1/frame 0xf8fd9648 ffs_reallocblks(f8fd97f8,c13ab023,0,0,0,...) at ffs_reallocblks+0x1128/frame 0xf8fd9758 VOP_REALLOCBLKS_APV(c147ad9c,f8fd97f8,0,8000,0,...) at VOP_REALLOCBLKS_APV+0x135/frame 0xf8fd9788 cluster_write(cd1b65a0,e45c6328,a20000,0,7f,...) at cluster_write+0x4f8/frame 0xf8fd982c ffs_write(f8fd99b0,c13aa7e5,cd1b65d4,cd1b65a0,c148e0e4,...) at ffs_write+0x412/frame 0xf8fd98b8 VOP_WRITE_APV(c147ad9c,f8fd99b0,cd1b65a0,373,2,...) at VOP_WRITE_APV+0x1ac/frame 0xf8fd9978 vn_write(dac9bc08,f8fd9be8,d78c8d00,1,dac50620,...) at vn_write+0x328/frame 0xf8fd99e0 vn_io_fault_doio(dac50620,f8fd9a4c,c0baf43c,c1579edc,0,...) at vn_io_fault_doio+0x34/frame 0xf8fd9a28 vn_io_fault1(dac50620,0,0,ffffffff,7fffffff,...) at vn_io_fault1+0x6d/frame 0xf8fd9af8 vn_io_fault(dac9bc08,f8fd9be8,d78c8d00,0,dac50620,...) at vn_io_fault+0x1fa/frame 0xf8fd9b4c dofilewrite(dac9bc08,f8fd9be8,ffffffff,ffffffff,0,...) at dofilewrite+0xa0/frame 0xf8fd9b80 kern_writev(dac50620,4,f8fd9be8,c159d930,bfbfe6c8,...) at kern_writev+0x93/frame 0xf8fd9bc8 sys_write(dac50620,f8fd9ca8,c11d731d,4,df,...) at sys_write+0x59/frame 0xf8fd9c18 syscall(f8fd9ce8) at syscall+0x34a/frame 0xf8fd9cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8fd9cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281b6eef, esp = 0xbfbfd6ac, ebp = 0xbfbfe758 --- 20150426 20:24:23 all (32/315): linger.sh 20150426 20:50:37 all (33/315): mmap12.sh 20150426 20:50:37 all (34/315): trim3.sh 20150426 21:15:52 all (35/315): rename5.sh 20150426 21:16:34 all (36/315): rename8.sh 20150426 21:17:12 all (37/315): core4.sh Apr 26 21:17:13 x4 kernel: Failed to write core file for process core4 (error 5) 20150426 21:17:34 all (38/315): radix.sh Apr 26 21:17:33 x4 last message repeated 59 times 20150426 21:18:33 all (39/315): marcus2.sh Expensive timeout(9) function: 0xc099fc60(0xc8c33c78) 0.012008113 s *** Excessive run time: marcus2.sh, 32 min 20150426 21:50:58 all (40/315): zfs2.sh 20150426 21:50:59 all (41/315): jail2.sh 20150426 21:51:01 all (42/315): lockf4.sh *** Excessive run time: lockf4.sh, 46 min 20150426 22:37:23 all (43/315): rename9.sh 20150426 22:38:08 all (44/315): newfs5.sh 20150426 22:38:16 all (45/315): udp.sh 20150426 22:38:19 all (46/315): rename7.sh 20150426 22:39:29 all (47/315): fifo2.sh kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). : : kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). kern.maxfiles limit exceeded by uid 65534, please see tuning(7). Apr 27 04:01:51 x4 su: pho to root on /dev/pts/1 Stopping inetd. Waiting for PIDS: 15579. Shutting down local packages:. Stopping cron. Waiting for PIDS: 732. Stopping sshd. Waiting for PIDS: 721lock order reversal: 1st 0xc93523e8 process lock (process lock) @ kern/kern_proc.c:288 2nd 0xd2912100 kqueue (kqueue) @ kern/kern_event.c:1943 KDB: stack backtrace: db_trace_self_wrapper(c11e2a28,af361000,fb7668d8,c85832b0,c8583248,...) at db_trace_self_wrapper+0x2a/frame 0xfb766890 kdb_backtrace(c11e6b25,d2912100,c11d7216,c8583520,c11d6efd,...) at kdb_backtrace+0x2d/frame 0xfb7668f4 witness_checkorder(d2912100,9,c11d6ef4,797,0,...) at witness_checkorder+0xd2e/frame 0xfb766948 __mtx_lock_flags(d2912110,0,c11d6ef4,797,c9352340,...) at __mtx_lock_flags+0x99/frame 0xfb76697c knlist_add(c93525e0,ce56a000,1,d2912100,debe4930,...) at knlist_add+0xa8/frame 0xfb7669a0 filt_procattach(ce56a000,0,c11d6ef4,49f,1fffff,...) at filt_procattach+0x13d/frame 0xfb7669c8 kqueue_register(debe4930,1,1,50f,0,...) at kqueue_register+0x695/frame 0xfb766a48 kern_kevent(debe4930,3,1,0,fb766be8,...) at kern_kevent+0x1d2/frame 0xfb766b98 sys_kevent(debe4930,fb766ca8,c11d731d,4,c10646c7,...) at sys_kevent+0x12c/frame 0xfb766c18 syscall(fb766ce8) at syscall+0x34a/frame 0xfb766cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfb766cdc --- syscall (363, FreeBSD ELF32, sys_kevent), eip = 0x2817efbf, esp = 0xbfbfedec, ebp = 0xbfbfee34 --- . Stopping moused. Waiting for PIDS: 674. Stopping smartd. Waiting for PIDS: 667. Stopping powerd. Waiting for PIDS: 648. Stopping ntpd. Waiting for PIDS: 645. Stopping nfsd. Waiting for PIDS: 586 589, 586. Stopping mountd. Waiting for PIDS: 580. Stopping nfsuserd. Waiting for PIDS: 569 570 571 572 573. Stopping casperd. Waiting for PIDS: 552. Stopping rpcbind. Waiting for PIDS: 537. Stopping watchdogd. Waiting for PIDS: 465. Stopping rtsold. Waiting for PIDS: 337. Stopping devd. Waiting for PIDS: 301. Writing entropy file:. . Terminated Apr 27 04:02:11 x4 syslogd: exiting on signal 15 Apr 27 04:02:31 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 11 400.0 0.0 0 32 - RL 14:29 1742:09.41 [idle] 0 0 0 155 0 - root 0 0.0 0.0 0 88 - DLs 14:29 0:16.61 [kernel] 0 0 0 -16 0 swapin root 1 0.0 0.0 9240 780 - SLs 14:29 0:00.16 /sbin/init -- 0 0 0 20 0 wait root 2 0.0 0.0 0 16 - DL 14:29 2:03.02 [cam] 0 0 0 -16 0 - root 3 0.0 0.0 0 8 - DL 14:29 0:00.27 [fdc0] 0 0 0 -16 0 - root 4 0.0 0.0 0 8 - DL 14:29 0:00.00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 5 0.0 0.0 0 8 - DL 14:29 16:14.33 [pagedaemon] 0 0 0 -16 0 psleep root 6 0.0 0.0 0 8 - DL 14:29 0:12.09 [vmdaemon] 0 0 0 -16 0 psleep root 7 0.0 0.0 0 8 - DL 14:29 0:00.00 [pagezero] 0 0 0 155 0 pgzero root 8 0.0 0.0 0 48 - DL 14:29 49:12.28 [bufdaemon] 0 0 0 20 0 psleep root 9 0.0 0.0 0 8 - DL 14:29 2:50.97 [vnlru] 0 0 0 21 0 vlruwt root 10 0.0 0.0 0 8 - DL 14:29 0:00.00 [audit] 0 0 0 -16 0 audit_wo root 12 0.0 0.0 0 160 - WL 14:29 6:24.09 [intr] 0 0 0 -64 0 - root 13 0.0 0.0 0 24 - DL 14:29 1:22.49 [geom] 0 0 0 -8 0 - root 14 0.0 0.0 0 8 - DL 14:29 0:04.61 [rand_harvestq] 0 0 0 -16 0 - root 15 0.0 0.0 0 192 - DL 14:29 0:02.21 [usb] 0 0 0 -68 0 - root 16 0.0 0.0 0 8 - DL 14:29 53:28.64 [syncer] 0 0 0 16 0 syncer root 83279 0.0 0.0 0 8 - DL 22:39 0:00.00 [md5] 0 0 0 -8 0 mdwait root 86525 0.0 0.1 10536 2884 u0 Ss 04:02 0:00.01 -sh (sh) 0 1 0 20 0 wait root 86526 0.0 0.1 10180 2012 u0 R+ 04:02 0:00.00 ps auxwwl 0 86525 0 20 0 - nobody 83312 0.0 0.1 14432 1776 0- T 22:39 0:06.35 /tmp/fifo2 65534 1 0 20 0 - nobody 83436 0.0 0.1 14432 1776 0- T 22:39 0:06.23 /tmp/fifo2 65534 1 0 20 0 - # kill -9 83312 83436 # kill -9 83312 83436 # # ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_alt_break_internal+0x162: movl $0,kdb_why db> show mount 0xc92f32ec /dev/ad4s1a on / (ufs) 0xc92f35d8 devfs on /dev (devfs) 0xc9726000 /dev/ad4s1f on /home (ufs) 0xc97258c4 /dev/ad4s1d on /usr (ufs) 0xc97255d8 /dev/ad4s1e on /var (ufs) 0xc9a908c4 procfs on /proc (procfs) 0xc9725bb0 /dev/label/tmp on /tmp (ufs) 0xdb3165d8 /dev/md5a on /mnt (ufs) More info: show mount db> show mount 0xdb3165d8 0xdb3165d8 /dev/md5a on /mnt (ufs) mnt_flag = SOFTDEP, EXPORTED, DEFEXPORTED, LOCAL mnt_kern_flag = SOFTDEP, EXTENDED_SHARED, NO_IOPF, USES_BCACHE, NOASYNC, LOOKUP_SHARED, 0x20002000 mnt_opt = fstype, fspath, from, errmsg mnt_stat = { version=537068824 type=53 flags=0x0000000000201300 bsize=4096 iosize=32768 blocks=507813 bfree=507811 bavail=467186 files=262142 ffree=262139 syncwrites=2 asyncwrites=6 syncreads=6 asyncreads=0 namemax=255 owner=0 fsid=[1430080770, 447686804] } mnt_cred = { uid=0 ruid=0 } mnt_ref = 3 mnt_gen = 2 mnt_nvnodelistsize = 3 mnt_activevnodelistsize = 3 mnt_writeopcount = 0 mnt_maxsymlinklen = 120 mnt_iosize_max = 131072 mnt_hashseed = 4170248155 mnt_lockref = 0 mnt_secondary_writes = 0 mnt_secondary_accwrites = 4 mnt_gjprovider = NULL List of active vnodes vnode 0xcdcfe000: tag syncer, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type syncer: UNLOCKED #0 0xc0b3f26a at __lockmgr_args+0xbca #1 0xc0c08113 at vop_stdlock+0x53 #2 0xc108d79a at VOP_LOCK1_APV+0x10a #3 0xc0c2b83a at _vn_lock+0xba #4 0xc0c201f2 at sched_sync+0x412 #5 0xc0b299ae at fork_exit+0x7e #6 0xc1048744 at fork_trampoline+0x8 vnode 0xccdd25a0: tag ufs, type VFIFO usecount 65532, writecount 43576, refcount 65532 mountedhere 0xc856fda0 flags (VI_ACTIVE) lock type ufs: UNLOCKED #0 0xc0b3f26a at __lockmgr_args+0xbca #1 0xc0df6fe8 at ffs_lock+0x98 #2 0xc108d79a at VOP_LOCK1_APV+0x10a #3 0xc0c2b83a at _vn_lock+0xba #4 0xc0c1a318 at vflush+0x428 #5 0xc0df19ae at ffs_flushfiles+0x14e #6 0xc0dd62fd at softdep_flushfiles+0x19d #7 0xc0df4d39 at ffs_unmount+0xd9 #8 0xc0c11c59 at dounmount+0x4c9 #9 0xc0c11744 at sys_unmount+0x2d4 #10 0xc105d48a at syscall+0x34a #11 0xc1048731 at Xint0x80_syscall+0x21 ino 4, on dev md5a, fifo with 43647 readers and 43576 writers vnode 0xca821438: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) v_object 0xcb2ea4e0 ref 0 pages 1 cleanbuf 0 dirtybuf 1 lock type ufs: UNLOCKED #0 0xc0b3f26a at __lockmgr_args+0xbca #1 0xc0df6fe8 at ffs_lock+0x98 #2 0xc108d79a at VOP_LOCK1_APV+0x10a #3 0xc0c2b83a at _vn_lock+0xba #4 0xc0c1a318 at vflush+0x428 #5 0xc0df19ae at ffs_flushfiles+0x14e #6 0xc0dd62fd at softdep_flushfiles+0x19d #7 0xc0df4d39 at ffs_unmount+0xd9 #8 0xc0c11c59 at dounmount+0x4c9 #9 0xc0c11744 at sys_unmount+0x2d4 #10 0xc105d48a at syscall+0x34a #11 0xc1048731 at Xint0x80_syscall+0x21 ino 2, on dev md5a List of inactive vnodes db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xc8a3b930 kdb_alt_break_internal(0,e4741ac8,c0980037,2,c8c18884,...) at kdb_alt_break_internal+0x162/frame 0xe4741a9c kdb_alt_break(2,c8c18884,c11a6781,c8c18800,40000,...) at kdb_alt_break+0x16/frame 0xe4741aa8 uart_intr_rxready(c8c18800,e4741b04,c107ab03,0,0,...) at uart_intr_rxready+0x87/frame 0xe4741ac8 uart_intr(c8c18800,369e99,b,0,0,...) at uart_intr+0x100/frame 0xe4741b04 intr_event_handle(c8a39680,e4741b80,0,2,0,...) at intr_event_handle+0x7a/frame 0xe4741b30 intr_execute_handlers(c88b5908,e4741b80,2) at intr_execute_handlers+0x42/frame 0xe4741b50 lapic_handle_intr(30,e4741b80) at lapic_handle_intr+0x5d/frame 0xe4741b70 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xe4741b70 --- interrupt, eip = 0xc10442b5, esp = 0xe4741bc0, ebp = 0xe4741bc0 --- acpi_cpu_c1(e472ec90,0,c13a1a03,2ec,bed1,...) at acpi_cpu_c1+0x5/frame 0xe4741bc0 acpi_cpu_idle(296f86b5,0,296f86b5,e4741c30,c107547d,...) at acpi_cpu_idle+0x166/frame 0xe4741c00 cpu_idle_acpi(296f86b5,0,5,c156ab88,c156ab84,...) at cpu_idle_acpi+0x3f/frame 0xe4741c14 cpu_idle(0,e4741c80,c11e0aa2,407,c156ab88,...) at cpu_idle+0x9d/frame 0xe4741c30 sched_idletd(0,e4741ce8,c11d76fc,3dc,0,...) at sched_idletd+0x3cb/frame 0xe4741cac fork_exit(c0b8b880,0,e4741ce8) at fork_exit+0x7e/frame 0xe4741cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4741cd4 --- trap 0, eip = 0, esp = 0xe4741d20, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x792480 curthread = 0xc8a3c000: pid 11 "idle: cpu0" curpcb = 0xe473bd40 fpcurthread = none idlethread = 0xc8a3c000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x23298480 curthread = 0xc8a3bc40: pid 11 "idle: cpu1" curpcb = 0xe473ed40 fpcurthread = none idlethread = 0xc8a3bc40: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x2329b480 curthread = 0xc8a3b930: pid 11 "idle: cpu2" curpcb = 0xe4741d40 fpcurthread = none idlethread = 0xc8a3b930: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x2329e480 curthread = 0xc8a3b620: pid 11 "idle: cpu3" curpcb = 0xe4744d40 fpcurthread = none idlethread = 0xc8a3b620: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 83436 (fifo2) thread 0xd43f4930 (103297) exclusive sx filedesc structure (filedesc structure) r = 0 (0xde241a80) locked @ kern/kern_descrip.c:1769 Process 83312 (fifo2) thread 0xd572dc40 (103296) exclusive sx filedesc structure (filedesc structure) r = 0 (0xda8e9398) locked @ kern/kern_descrip.c:1769 db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc92f32ec /dev/ad4s1a on / (ufs) 0xc92f35d8 devfs on /dev (devfs) 0xc9726000 /dev/ad4s1f on /home (ufs) 0xc97258c4 /dev/ad4s1d on /usr (ufs) 0xc97255d8 /dev/ad4s1e on /var (ufs) 0xc9a908c4 procfs on /proc (procfs) 0xc9725bb0 /dev/label/tmp on /tmp (ufs) 0xdb3165d8 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 86525 1 86525 0 Ss+ ttyin 0xc8c63c70 sh 83436 1 63572 65534 T+ (threaded) fifo2 102103 s fifo2 103297 D kmem are 0xc157b300 fifo2 83312 1 63572 65534 T+ (threaded) fifo2 102424 s fifo2 103296 D kmem are 0xc157b300 fifo2 83279 0 0 0 DL mdwait 0xc9a8e800 [md5] 16 0 0 0 DL syncer 0xc15d6354 [syncer] 9 0 0 0 DL vlruwt 0xc917c000 [vnlru] 8 0 0 0 DL (threaded) [bufdaemon] 100072 D psleep 0xc15d5a84 [bufdaemon] 100085 D sdflush 0xc972d284 [/home worker] 100087 D sdflush 0xc972ce84 [/usr worker] 100088 D sdflush 0xc972cc84 [/var worker] 100190 D sdflush 0xc972ca84 [/tmp worker] 103293 D sdflush 0xd57f1884 [/mnt worker] 7 0 0 0 DL pgzero 0xc15db5c0 [pagezero] 6 0 0 0 DL psleep 0xc15db33c [vmdaemon] 5 0 0 0 DL psleep 0xc1638b08 [pagedaemon] 4 0 0 0 DL waiting_ 0xc163549c [sctp_iterator] 3 0 0 0 DL - 0xc8c18a3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100032 D - 0xc8c2eacc [usbus0] 100033 D - 0xc8c2eafc [usbus0] 100034 D - 0xc8c2eb2c [usbus0] 100035 D - 0xc8c2eb5c [usbus0] 100037 D - 0xc8c77acc [usbus1] 100038 D - 0xc8c77afc [usbus1] 100039 D - 0xc8c77b2c [usbus1] 100040 D - 0xc8c77b5c [usbus1] 100042 D - 0xc8c80acc [usbus2] 100043 D - 0xc8c80afc [usbus2] 100044 D - 0xc8c80b2c [usbus2] 100045 D - 0xc8c80b5c [usbus2] 100046 D - 0xc8c82acc [usbus3] 100047 D - 0xc8c82afc [usbus3] 100048 D - 0xc8c82b2c [usbus3] 100049 D - 0xc8c82b5c [usbus3] 100050 D - 0xc8c88acc [usbus4] 100051 D - 0xc8c88afc [usbus4] 100052 D - 0xc8c88b2c [usbus4] 100053 D - 0xc8c88b5c [usbus4] 100055 D - 0xc8c33ca4 [usbus5] 100056 D - 0xc8c33cd4 [usbus5] 100057 D - 0xc8c33d04 [usbus5] 100058 D - 0xc8c33d34 [usbus5] 2 0 0 0 DL (threaded) [cam] 100026 D - 0xc1493b80 [doneq0] 100068 D - 0xc1493a2c [scanner] 14 0 0 0 DL - 0xc14ae770 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100013 D - 0xc1630620 [g_event] 100014 D - 0xc1630624 [g_up] 100015 D - 0xc1630628 [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi3: vm] 100008 I [swi4: clock (0)] 100009 I [swi4: clock (1)] 100010 I [swi4: clock (2)] 100011 I [swi4: clock (3)] 100012 I [swi1: netisr 0] 100018 I [swi6: Giant taskq] 100021 I [swi5: fast taskq] 100027 I [swi6: task queue] 100028 I [irq256: hdac0] 100029 I [irq257: re0] 100030 I [irq22: ahci0] 100031 I [irq16: hdac1 ohci0] 100036 I [irq17: ohci1 ohci3] 100041 I [irq18: ohci2 ohci4] 100054 I [irq19: ehci0] 100059 I [irq14: ata0] 100060 I [swi0: uart] 100062 I [irq1: atkbd0] 100063 I [irq12: psm0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc8a339c0 [init] 10 0 0 0 DL audit_wo 0xc16373d0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xc1630644 [swapper] 100016 D - 0xc8a38180 [firmware taskq] 100019 D - 0xc8a38000 [ffs_trim taskq] 100020 D - 0xc8a37f00 [thread taskq] 100022 D - 0xc8a37d80 [kqueue taskq] 100023 D - 0xc8a37d00 [acpi_task_0] 100024 D - 0xc8a37d00 [acpi_task_1] 100025 D - 0xc8a37d00 [acpi_task_2] 100064 D - 0xc8cb8900 [mca taskq] 100065 D - 0xc155c9ee [deadlkres] 100067 D - 0xc8a37c80 [CAM taskq] db:0:ps> allt Tracing command sh pid 86525 tid 100174 td 0xc935f310 sched_switch(c935f310,0,104,1af,f8a199a4,...) at sched_switch+0x4e2/frame 0xf8a1996c mi_switch(104,0,c11e4a45,200,c15a8880,...) at mi_switch+0x1bf/frame 0xf8a199a4 sleepq_switch(c935f310,0,c11e4a45,1b6,f8a19a1c,...) at sleepq_switch+0x1c5/frame 0xf8a199d8 sleepq_catch_signals(f8a19a38,c0b479f9,c8c63c14,0,c1453ae8,...) at sleepq_catch_signals+0x382/frame 0xf8a19a14 sleepq_wait_sig(c8c63c70,0,f8a19a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8a19a38 _cv_wait_sig(c8c63c70,c8c63c04,c11ea061,567,1,...) at _cv_wait_sig+0x214/frame 0xf8a19a84 tty_wait(c8c63c00,c8c63c70,f8a19be8,1,0,...) at tty_wait+0x4b/frame 0xf8a19aa8 ttydisc_read(c8c63c00,f8a19be8,0,b2,f8a19be8,...) at ttydisc_read+0x3d4/frame 0xf8a19af0 ttydev_read(c88dde00,f8a19be8,0,0,f8a19b40,...) at ttydev_read+0x55/frame 0xf8a19b10 devfs_read_f(c93f2118,f8a19be8,c89ffb00,0,c935f310,...) at devfs_read_f+0xa8/frame 0xf8a19b4c dofileread(c93f2118,f8a19be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8a19b80 kern_readv(c935f310,0,f8a19be8,c15a8880,bfbfed1a,...) at kern_readv+0x93/frame 0xf8a19bc8 sys_read(c935f310,f8a19ca8,c11d731d,4,c10646c7,...) at sys_read+0x59/frame 0xf8a19c18 syscall(f8a19ce8) at syscall+0x34a/frame 0xf8a19cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8a19cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281ebf0f, esp = 0xbfbfecf4, ebp = 0xbfbfed2c --- Tracing command fifo2 pid 83436 tid 102103 td 0xde1d9620 sched_switch(de1d9620,0,10a,1af,8,...) at sched_switch+0x4e2/frame 0xf9615b04 mi_switch(10a,0,c11df6a7,3a3,de1d86a0,...) at mi_switch+0x1bf/frame 0xf9615b3c thread_suspend_switch(de1d9620,de1d8680,c11df6a7,23e,1,...) at thread_suspend_switch+0x19f/frame 0xf9615b70 thread_single(de1d8680,1,c11d7440,d2,de1d9620,...) at thread_single+0x4ec/frame 0xf9615ba8 exit1(de1d9620,100,f9615cdc,c105d48a,de1d9620,...) at exit1+0xa2/frame 0xf9615c08 sys_sys_exit(de1d9620,f9615ca8,c11d731d,4,d8aee000,...) at sys_sys_exit+0x1d/frame 0xf9615c18 syscall(f9615ce8) at syscall+0x34a/frame 0xf9615cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf9615cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x28157aa3, esp = 0xbfbfe350, ebp = 0xbfbfe358 --- Tracing command fifo2 pid 83436 tid 103297 td 0xd43f4930 sched_switch(d43f4930,0,104,1af,fb789874,...) at sched_switch+0x4e2/frame 0xfb78983c mi_switch(104,0,c11e4a45,200,c157b280,...) at mi_switch+0x1bf/frame 0xfb789874 sleepq_switch(d43f4930,0,c11e4a45,276,c157b300,...) at sleepq_switch+0x1c5/frame 0xfb7898a8 sleepq_wait(c157b300,0,fb7898fc,1,0,...) at sleepq_wait+0x48/frame 0xfb7898cc _cv_wait(c157b300,c157b280,c11e603d,4b2,c1453ae8,...) at _cv_wait+0x1fe/frame 0xfb789910 vmem_xalloc(c157b280,281000,0,0,0,...) at vmem_xalloc+0x1df/frame 0xfb789984 vmem_alloc(c157b280,281000,2102,fb7899d4,0,...) at vmem_alloc+0xc4/frame 0xfb7899bc kmem_malloc(c157b280,281000,102,2,281000,...) at kmem_malloc+0x3b/frame 0xfb7899e4 uma_large_malloc(281000,102,198e,246,d43f4930,...) at uma_large_malloc+0x40/frame 0xfb789a04 malloc(28000c,c144f430,102,5f9,de1d8728,...) at malloc+0xb7/frame 0xfb789a50 fdgrowtable(de241a80,14,c11d6481,5de,c0b679c7,...) at fdgrowtable+0x7e/frame 0xfb789a8c fdalloc(d43f4930,0,fb789b34,6e9,9cc,...) at fdalloc+0x17c/frame 0xfb789ad0 finstall(d43f4930,df7c3230,fb789b34,198e,0,...) at finstall+0x12d/frame 0xfb789b00 kern_openat(d43f4930,ffffff9c,8048f7c,0,198d,bce) at kern_openat+0x53e/frame 0xfb789bf4 sys_open(d43f4930,fb789ca8,c11d731d,4,c10646c7,...) at sys_open+0x39/frame 0xfb789c18 syscall(fb789ce8) at syscall+0x34a/frame 0xfb789cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfb789cdc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x28119775, esp = 0xbbbfdf5c, ebp = 0xbbbfdf98 --- Tracing command fifo2 pid 83312 tid 102424 td 0xde48d310 sched_switch(de48d310,0,10a,1af,8,...) at sched_switch+0x4e2/frame 0xfaac5b04 mi_switch(10a,0,c11df6a7,3a3,de4899e0,...) at mi_switch+0x1bf/frame 0xfaac5b3c thread_suspend_switch(de48d310,de4899c0,c11df6a7,23e,1,...) at thread_suspend_switch+0x19f/frame 0xfaac5b70 thread_single(de4899c0,1,c11d7440,d2,de48d310,...) at thread_single+0x4ec/frame 0xfaac5ba8 exit1(de48d310,100,faac5cdc,c105d48a,de48d310,...) at exit1+0xa2/frame 0xfaac5c08 sys_sys_exit(de48d310,faac5ca8,c11d731d,4,deb99000,...) at sys_sys_exit+0x1d/frame 0xfaac5c18 syscall(faac5ce8) at syscall+0x34a/frame 0xfaac5cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfaac5cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x28157aa3, esp = 0xbfbfe350, ebp = 0xbfbfe358 --- Tracing command fifo2 pid 83312 tid 103296 td 0xd572dc40 sched_switch(d572dc40,0,104,1af,fb786874,...) at sched_switch+0x4e2/frame 0xfb78683c mi_switch(104,0,c11e4a45,200,c157b280,...) at mi_switch+0x1bf/frame 0xfb786874 sleepq_switch(d572dc40,0,c11e4a45,276,c157b300,...) at sleepq_switch+0x1c5/frame 0xfb7868a8 sleepq_wait(c157b300,0,fb7868fc,1,0,...) at sleepq_wait+0x48/frame 0xfb7868cc _cv_wait(c157b300,c157b280,c11e603d,4b2,c1453ae8,...) at _cv_wait+0x1fe/frame 0xfb786910 vmem_xalloc(c157b280,281000,0,0,0,...) at vmem_xalloc+0x1df/frame 0xfb786984 vmem_alloc(c157b280,281000,2102,fb7869d4,0,...) at vmem_alloc+0xc4/frame 0xfb7869bc kmem_malloc(c157b280,281000,102,2,281000,...) at kmem_malloc+0x3b/frame 0xfb7869e4 uma_large_malloc(281000,102,95302009,246,d572dc40,...) at uma_large_malloc+0x40/frame 0xfb786a04 malloc(28000c,c144f430,102,5f9,de489a68,...) at malloc+0xb7/frame 0xfb786a50 fdgrowtable(da8e9398,14,c11d6481,5de,c0b679c7,...) at fdgrowtable+0x7e/frame 0xfb786a8c fdalloc(d572dc40,0,fb786b34,6e9,1ec,...) at fdalloc+0x17c/frame 0xfb786ad0 finstall(d572dc40,d501a658,fb786b34,95302009,0,...) at finstall+0x12d/frame 0xfb786b00 kern_openat(d572dc40,ffffff9c,8048f7c,0,95302208,804f1fc) at kern_openat+0x53e/frame 0xfb786bf4 sys_open(d572dc40,fb786ca8,c11d731d,4,c10646c7,...) at sys_open+0x39/frame 0xfb786c18 syscall(fb786ce8) at syscall+0x34a/frame 0xfb786cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfb786cdc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x28119775, esp = 0xbbbfdf5c, ebp = 0xbbbfdf98 --- Tracing command md5 pid 83279 tid 102724 td 0xde720310 sched_switch(de720310,0,104,1af,faf2dbb4,...) at sched_switch+0x4e2/frame 0xfaf2db7c mi_switch(104,0,c11e4a45,200,c9a8e820,...) at mi_switch+0x1bf/frame 0xfaf2dbb4 sleepq_switch(de720310,0,c11e4a45,276,de720310,...) at sleepq_switch+0x1c5/frame 0xfaf2dbe8 sleepq_wait(c9a8e800,5c,c116671a,0,0,...) at sleepq_wait+0x48/frame 0xfaf2dc0c _sleep(c9a8e800,c9a8e820,25c,c116671a,0,...) at _sleep+0x3a1/frame 0xfaf2dc68 md_kthread(c9a8e800,faf2dce8,c11d76fc,3dc,0,...) at md_kthread+0x1c7/frame 0xfaf2dcac fork_exit(c07ab0f0,c9a8e800,faf2dce8) at fork_exit+0x7e/frame 0xfaf2dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xfaf2dcd4 --- trap 0, eip = 0, esp = 0xfaf2dd20, ebp = 0 --- Tracing command syncer pid 16 tid 100074 td 0xc8efcc40 sched_switch(c8efcc40,0,104,1af,edb09b90,...) at sched_switch+0x4e2/frame 0xedb09b58 mi_switch(104,0,c11e4a45,200,c15d6334,...) at mi_switch+0x1bf/frame 0xedb09b90 sleepq_switch(c8efcc40,0,c11e4a45,299,0,...) at sleepq_switch+0x1c5/frame 0xedb09bc4 sleepq_timedwait(c15d6354,0,edb09c24,0,0,...) at sleepq_timedwait+0x48/frame 0xedb09be8 _cv_timedwait_sbt(c15d6354,c15d6334,fffffed8,0,0,...) at _cv_timedwait_sbt+0x23c/frame 0xedb09c38 sched_sync(0,edb09ce8,c11d76fc,3dc,0,...) at sched_sync+0x712/frame 0xedb09cac fork_exit(c0c1fde0,0,edb09ce8) at fork_exit+0x7e/frame 0xedb09cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb09cd4 --- trap 0, eip = 0, esp = 0xedb09d20, ebp = 0 --- Tracing command vnlru pid 9 tid 100073 td 0xc8efd000 sched_switch(c8efd000,0,104,1af,edb06b8c,...) at sched_switch+0x4e2/frame 0xedb06b54 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb06b8c sleepq_switch(c8efd000,0,c11e4a45,299,c8efd000,...) at sleepq_switch+0x1c5/frame 0xedb06bc0 sleepq_timedwait(c917c000,60,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb06be4 _sleep(c917c000,c15d630c,260,c11f15dd,fffffed8,...) at _sleep+0x33d/frame 0xedb06c40 vnlru_proc(0,edb06ce8,c11d76fc,3dc,0,...) at vnlru_proc+0xcf/frame 0xedb06cac fork_exit(c0c20610,0,edb06ce8) at fork_exit+0x7e/frame 0xedb06cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb06cd4 --- trap 0, eip = 0, esp = 0xedb06d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100072 td 0xc8efd310 sched_switch(c8efd310,0,104,1af,edb03bc4,...) at sched_switch+0x4e2/frame 0xedb03b8c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb03bc4 sleepq_switch(c8efd310,0,c11e4a45,299,c8efd310,...) at sleepq_switch+0x1c5/frame 0xedb03bf8 sleepq_timedwait(c15d5a84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb03c1c _sleep(c15d5a84,c15d5a00,54,c11edec2,fffffed8,...) at _sleep+0x33d/frame 0xedb03c78 buf_daemon(0,edb03ce8,c11d76fc,3dc,0,...) at buf_daemon+0x8c/frame 0xedb03cac fork_exit(c0c02c50,0,edb03ce8) at fork_exit+0x7e/frame 0xedb03cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb03cd4 --- trap 0, eip = 0, esp = 0xedb03d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100085 td 0xc946f000 sched_switch(c946f000,0,104,1af,edb3dbb4,...) at sched_switch+0x4e2/frame 0xedb3db7c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb3dbb4 sleepq_switch(c946f000,0,c11e4a45,299,c946f000,...) at sleepq_switch+0x1c5/frame 0xedb3dbe8 sleepq_timedwait(c972d284,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb3dc0c _sleep(c972d284,c972d200,54,c1218f58,7fffff6c,...) at _sleep+0x33d/frame 0xedb3dc68 softdep_flush(c9726000,edb3dce8,c11d76fc,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb3dcac fork_exit(c0dda3a0,c9726000,edb3dce8) at fork_exit+0x7e/frame 0xedb3dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb3dcd4 --- trap 0, eip = 0, esp = 0xedb3dd20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100087 td 0xc946e930 sched_switch(c946e930,0,104,1af,edb43bb4,...) at sched_switch+0x4e2/frame 0xedb43b7c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb43bb4 sleepq_switch(c946e930,0,c11e4a45,299,c946e930,...) at sleepq_switch+0x1c5/frame 0xedb43be8 sleepq_timedwait(c972ce84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb43c0c _sleep(c972ce84,c972ce00,54,c1218f58,7fffff6c,...) at _sleep+0x33d/frame 0xedb43c68 softdep_flush(c97258c4,edb43ce8,c11d76fc,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb43cac fork_exit(c0dda3a0,c97258c4,edb43ce8) at fork_exit+0x7e/frame 0xedb43cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb43cd4 --- trap 0, eip = 0, esp = 0xedb43d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100088 td 0xc8efc310 sched_switch(c8efc310,0,104,1af,edb46bb4,...) at sched_switch+0x4e2/frame 0xedb46b7c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb46bb4 sleepq_switch(c8efc310,0,c11e4a45,299,c8efc310,...) at sleepq_switch+0x1c5/frame 0xedb46be8 sleepq_timedwait(c972cc84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb46c0c _sleep(c972cc84,c972cc00,54,c1218f58,7fffff6c,...) at _sleep+0x33d/frame 0xedb46c68 softdep_flush(c97255d8,edb46ce8,c11d76fc,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb46cac fork_exit(c0dda3a0,c97255d8,edb46ce8) at fork_exit+0x7e/frame 0xedb46cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb46cd4 --- trap 0, eip = 0, esp = 0xedb46d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100190 td 0xcb007930 sched_switch(cb007930,0,104,1af,f88c4bb4,...) at sched_switch+0x4e2/frame 0xf88c4b7c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xf88c4bb4 sleepq_switch(cb007930,0,c11e4a45,299,cb007930,...) at sleepq_switch+0x1c5/frame 0xf88c4be8 sleepq_timedwait(c972ca84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf88c4c0c _sleep(c972ca84,c972ca00,54,c1218f58,7fffff6c,...) at _sleep+0x33d/frame 0xf88c4c68 softdep_flush(c9725bb0,f88c4ce8,c11d76fc,3dc,0,...) at softdep_flush+0x1a2/frame 0xf88c4cac fork_exit(c0dda3a0,c9725bb0,f88c4ce8) at fork_exit+0x7e/frame 0xf88c4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88c4cd4 --- trap 0, eip = 0, esp = 0xf88c4d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 103293 td 0xcdcc1310 sched_switch(cdcc1310,0,104,1af,f9430bb4,...) at sched_switch+0x4e2/frame 0xf9430b7c mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xf9430bb4 sleepq_switch(cdcc1310,0,c11e4a45,299,cdcc1310,...) at sleepq_switch+0x1c5/frame 0xf9430be8 sleepq_timedwait(d57f1884,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf9430c0c _sleep(d57f1884,d57f1800,54,c1218f58,7fffff6c,...) at _sleep+0x33d/frame 0xf9430c68 softdep_flush(db3165d8,f9430ce8,c11d76fc,3dc,0,...) at softdep_flush+0x1a2/frame 0xf9430cac fork_exit(c0dda3a0,db3165d8,f9430ce8) at fork_exit+0x7e/frame 0xf9430cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf9430cd4 --- trap 0, eip = 0, esp = 0xf9430d20, ebp = 0 --- Tracing command pagezero pid 7 tid 100071 td 0xc8efd620 sched_switch(c8efd620,0,104,1af,edb00bc0,...) at sched_switch+0x4e2/frame 0xedb00b88 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedb00bc0 sleepq_switch(c8efd620,0,c11e4a45,299,c8efd620,...) at sleepq_switch+0x1c5/frame 0xedb00bf4 sleepq_timedwait(c15db5c0,0,12b,0,0,...) at sleepq_timedwait+0x48/frame 0xedb00c18 _sleep(c15db5c0,c1638a80,0,c1220ed1,fffea520,...) at _sleep+0x33d/frame 0xedb00c74 vm_pagezero(0,edb00ce8,c11d76fc,3dc,0,...) at vm_pagezero+0xc2/frame 0xedb00cac fork_exit(c0e3bba0,0,edb00ce8) at fork_exit+0x7e/frame 0xedb00cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb00cd4 --- trap 0, eip = 0, esp = 0xedb00d20, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100070 td 0xc8efd930 sched_switch(c8efd930,0,104,1af,edafdb8c,...) at sched_switch+0x4e2/frame 0xedafdb54 mi_switch(104,0,c11e4a45,200,c15db324,...) at mi_switch+0x1bf/frame 0xedafdb8c sleepq_switch(c8efd930,0,c11e4a45,276,c8efd930,...) at sleepq_switch+0x1c5/frame 0xedafdbc0 sleepq_wait(c15db33c,74,c11edec2,0,0,...) at sleepq_wait+0x48/frame 0xedafdbe4 _sleep(c15db33c,c15db324,74,c11edec2,0,...) at _sleep+0x3a1/frame 0xedafdc40 vm_daemon(0,edafdce8,c11d76fc,3dc,0,...) at vm_daemon+0x9c/frame 0xedafdcac fork_exit(c0e33130,0,edafdce8) at fork_exit+0x7e/frame 0xedafdcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedafdcd4 --- trap 0, eip = 0, esp = 0xedafdd20, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100069 td 0xc8efdc40 sched_switch(c8efdc40,0,104,1af,edafab8c,...) at sched_switch+0x4e2/frame 0xedafab54 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xedafab8c sleepq_switch(c8efdc40,0,c11e4a45,299,c8efdc40,...) at sleepq_switch+0x1c5/frame 0xedafabc0 sleepq_timedwait(c1638b08,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedafabe4 _sleep(c1638b08,c1638a80,54,c11edec2,fffffed8,...) at _sleep+0x33d/frame 0xedafac40 vm_pageout(0,edaface8,c11d76fc,3dc,0,...) at vm_pageout+0x1f1/frame 0xedafacac fork_exit(c0e33950,0,edaface8) at fork_exit+0x7e/frame 0xedafacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedafacd4 --- trap 0, eip = 0, esp = 0xedafad20, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100066 td 0xc8efe620 sched_switch(c8efe620,0,104,1af,ed971bcc,...) at sched_switch+0x4e2/frame 0xed971b94 mi_switch(104,0,c11e4a45,200,c1635464,...) at mi_switch+0x1bf/frame 0xed971bcc sleepq_switch(c8efe620,0,c11e4a45,276,c8efe620,...) at sleepq_switch+0x1c5/frame 0xed971c00 sleepq_wait(c163549c,0,c1204ee5,0,0,...) at sleepq_wait+0x48/frame 0xed971c24 _sleep(c163549c,c1635464,0,c1204ee5,0,...) at _sleep+0x3a1/frame 0xed971c80 sctp_iterator_thread(0,ed971ce8,c11d76fc,3dc,0,...) at sctp_iterator_thread+0x7c/frame 0xed971cac fork_exit(c0cc7a30,0,ed971ce8) at fork_exit+0x7e/frame 0xed971cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed971cd4 --- trap 0, eip = 0, esp = 0xed971d20, ebp = 0 --- Tracing command fdc0 pid 3 tid 100061 td 0xc8c8a310 sched_switch(c8c8a310,0,104,1af,ed961b60,...) at sched_switch+0x4e2/frame 0xed961b28 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xed961b60 sleepq_switch(c8c8a310,0,c11e4a45,299,c8c8a310,...) at sleepq_switch+0x1c5/frame 0xed961b94 sleepq_timedwait(c8c18a3c,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xed961bb8 _sleep(c8c18a3c,c8c18af0,5c,c11c95c6,fffffed8,...) at _sleep+0x33d/frame 0xed961c14 fdc_thread(c8c18a00,ed961ce8,c11d76fc,3dc,0,...) at fdc_thread+0x95a/frame 0xed961cac fork_exit(c0fe2030,c8c18a00,ed961ce8) at fork_exit+0x7e/frame 0xed961cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed961cd4 --- trap 0, eip = 0, esp = 0xed961d20, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc8be3930 sched_switch(c8be3930,0,104,1af,ed8e8be8,...) at sched_switch+0x4e2/frame 0xed8e8bb0 mi_switch(104,0,c11e4a45,200,c8c2ecc8,...) at mi_switch+0x1bf/frame 0xed8e8be8 sleepq_switch(c8be3930,0,c11e4a45,276,c8c2eacc,...) at sleepq_switch+0x1c5/frame 0xed8e8c1c sleepq_wait(c8c2eacc,0,ed8e8c70,1,0,...) at sleepq_wait+0x48/frame 0xed8e8c40 _cv_wait(c8c2eacc,c8c2ecc8,c11af873,76,c8c2ead4,...) at _cv_wait+0x1fe/frame 0xed8e8c84 usb_process(c8c2eac4,ed8e8ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8e8cac fork_exit(c09bc0a0,c8c2eac4,ed8e8ce8) at fork_exit+0x7e/frame 0xed8e8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8e8cd4 --- trap 0, eip = 0, esp = 0xed8e8d20, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc8be3620 sched_switch(c8be3620,0,104,1af,ed8ebbe8,...) at sched_switch+0x4e2/frame 0xed8ebbb0 mi_switch(104,0,c11e4a45,200,c8c2ecc8,...) at mi_switch+0x1bf/frame 0xed8ebbe8 sleepq_switch(c8be3620,0,c11e4a45,276,c8c2eafc,...) at sleepq_switch+0x1c5/frame 0xed8ebc1c sleepq_wait(c8c2eafc,0,ed8ebc70,1,0,...) at sleepq_wait+0x48/frame 0xed8ebc40 _cv_wait(c8c2eafc,c8c2ecc8,c11af873,76,c8c2eb04,...) at _cv_wait+0x1fe/frame 0xed8ebc84 usb_process(c8c2eaf4,ed8ebce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8ebcac fork_exit(c09bc0a0,c8c2eaf4,ed8ebce8) at fork_exit+0x7e/frame 0xed8ebcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8ebcd4 --- trap 0, eip = 0, esp = 0xed8ebd20, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc8be3310 sched_switch(c8be3310,0,104,1af,ed8eebe8,...) at sched_switch+0x4e2/frame 0xed8eebb0 mi_switch(104,0,c11e4a45,200,c8c2ecc8,...) at mi_switch+0x1bf/frame 0xed8eebe8 sleepq_switch(c8be3310,0,c11e4a45,276,c8c2eb2c,...) at sleepq_switch+0x1c5/frame 0xed8eec1c sleepq_wait(c8c2eb2c,0,ed8eec70,1,0,...) at sleepq_wait+0x48/frame 0xed8eec40 _cv_wait(c8c2eb2c,c8c2ecc8,c11af873,76,c8c2eb34,...) at _cv_wait+0x1fe/frame 0xed8eec84 usb_process(c8c2eb24,ed8eece8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8eecac fork_exit(c09bc0a0,c8c2eb24,ed8eece8) at fork_exit+0x7e/frame 0xed8eecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8eecd4 --- trap 0, eip = 0, esp = 0xed8eed20, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc8be3000 sched_switch(c8be3000,0,104,1af,ed8f1be8,...) at sched_switch+0x4e2/frame 0xed8f1bb0 mi_switch(104,0,c11e4a45,200,c8c2ecc8,...) at mi_switch+0x1bf/frame 0xed8f1be8 sleepq_switch(c8be3000,0,c11e4a45,276,c8c2eb5c,...) at sleepq_switch+0x1c5/frame 0xed8f1c1c sleepq_wait(c8c2eb5c,0,ed8f1c70,1,0,...) at sleepq_wait+0x48/frame 0xed8f1c40 _cv_wait(c8c2eb5c,c8c2ecc8,c11af873,76,c8c2eb64,...) at _cv_wait+0x1fe/frame 0xed8f1c84 usb_process(c8c2eb54,ed8f1ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8f1cac fork_exit(c09bc0a0,c8c2eb54,ed8f1ce8) at fork_exit+0x7e/frame 0xed8f1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f1cd4 --- trap 0, eip = 0, esp = 0xed8f1d20, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc8aef930 sched_switch(c8aef930,0,104,1af,ed8f8be8,...) at sched_switch+0x4e2/frame 0xed8f8bb0 mi_switch(104,0,c11e4a45,200,c8c77cc8,...) at mi_switch+0x1bf/frame 0xed8f8be8 sleepq_switch(c8aef930,0,c11e4a45,276,c8c77acc,...) at sleepq_switch+0x1c5/frame 0xed8f8c1c sleepq_wait(c8c77acc,0,ed8f8c70,1,0,...) at sleepq_wait+0x48/frame 0xed8f8c40 _cv_wait(c8c77acc,c8c77cc8,c11af873,76,c8c77ad4,...) at _cv_wait+0x1fe/frame 0xed8f8c84 usb_process(c8c77ac4,ed8f8ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8f8cac fork_exit(c09bc0a0,c8c77ac4,ed8f8ce8) at fork_exit+0x7e/frame 0xed8f8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f8cd4 --- trap 0, eip = 0, esp = 0xed8f8d20, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc8aef620 sched_switch(c8aef620,0,104,1af,ed8fbbe8,...) at sched_switch+0x4e2/frame 0xed8fbbb0 mi_switch(104,0,c11e4a45,200,c8c77cc8,...) at mi_switch+0x1bf/frame 0xed8fbbe8 sleepq_switch(c8aef620,0,c11e4a45,276,c8c77afc,...) at sleepq_switch+0x1c5/frame 0xed8fbc1c sleepq_wait(c8c77afc,0,ed8fbc70,1,0,...) at sleepq_wait+0x48/frame 0xed8fbc40 _cv_wait(c8c77afc,c8c77cc8,c11af873,76,c8c77b04,...) at _cv_wait+0x1fe/frame 0xed8fbc84 usb_process(c8c77af4,ed8fbce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8fbcac fork_exit(c09bc0a0,c8c77af4,ed8fbce8) at fork_exit+0x7e/frame 0xed8fbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8fbcd4 --- trap 0, eip = 0, esp = 0xed8fbd20, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc8aef310 sched_switch(c8aef310,0,104,1af,ed8febe8,...) at sched_switch+0x4e2/frame 0xed8febb0 mi_switch(104,0,c11e4a45,200,c8c77cc8,...) at mi_switch+0x1bf/frame 0xed8febe8 sleepq_switch(c8aef310,0,c11e4a45,276,c8c77b2c,...) at sleepq_switch+0x1c5/frame 0xed8fec1c sleepq_wait(c8c77b2c,0,ed8fec70,1,0,...) at sleepq_wait+0x48/frame 0xed8fec40 _cv_wait(c8c77b2c,c8c77cc8,c11af873,76,c8c77b34,...) at _cv_wait+0x1fe/frame 0xed8fec84 usb_process(c8c77b24,ed8fece8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed8fecac fork_exit(c09bc0a0,c8c77b24,ed8fece8) at fork_exit+0x7e/frame 0xed8fecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8fecd4 --- trap 0, eip = 0, esp = 0xed8fed20, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc8c7f310 sched_switch(c8c7f310,0,104,1af,ed901be8,...) at sched_switch+0x4e2/frame 0xed901bb0 mi_switch(104,0,c11e4a45,200,c8c77cc8,...) at mi_switch+0x1bf/frame 0xed901be8 sleepq_switch(c8c7f310,0,c11e4a45,276,c8c77b5c,...) at sleepq_switch+0x1c5/frame 0xed901c1c sleepq_wait(c8c77b5c,0,ed901c70,1,0,...) at sleepq_wait+0x48/frame 0xed901c40 _cv_wait(c8c77b5c,c8c77cc8,c11af873,76,c8c77b64,...) at _cv_wait+0x1fe/frame 0xed901c84 usb_process(c8c77b54,ed901ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed901cac fork_exit(c09bc0a0,c8c77b54,ed901ce8) at fork_exit+0x7e/frame 0xed901cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed901cd4 --- trap 0, eip = 0, esp = 0xed901d20, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc8c7ec40 sched_switch(c8c7ec40,0,104,1af,ed908be8,...) at sched_switch+0x4e2/frame 0xed908bb0 mi_switch(104,0,c11e4a45,200,c8c80cc8,...) at mi_switch+0x1bf/frame 0xed908be8 sleepq_switch(c8c7ec40,0,c11e4a45,276,c8c80acc,...) at sleepq_switch+0x1c5/frame 0xed908c1c sleepq_wait(c8c80acc,0,ed908c70,1,0,...) at sleepq_wait+0x48/frame 0xed908c40 _cv_wait(c8c80acc,c8c80cc8,c11af873,76,c8c80ad4,...) at _cv_wait+0x1fe/frame 0xed908c84 usb_process(c8c80ac4,ed908ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed908cac fork_exit(c09bc0a0,c8c80ac4,ed908ce8) at fork_exit+0x7e/frame 0xed908cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed908cd4 --- trap 0, eip = 0, esp = 0xed908d20, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc8c7e930 sched_switch(c8c7e930,0,104,1af,ed90bbe8,...) at sched_switch+0x4e2/frame 0xed90bbb0 mi_switch(104,0,c11e4a45,200,c8c80cc8,...) at mi_switch+0x1bf/frame 0xed90bbe8 sleepq_switch(c8c7e930,0,c11e4a45,276,c8c80afc,...) at sleepq_switch+0x1c5/frame 0xed90bc1c sleepq_wait(c8c80afc,0,ed90bc70,1,0,...) at sleepq_wait+0x48/frame 0xed90bc40 _cv_wait(c8c80afc,c8c80cc8,c11af873,76,c8c80b04,...) at _cv_wait+0x1fe/frame 0xed90bc84 usb_process(c8c80af4,ed90bce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed90bcac fork_exit(c09bc0a0,c8c80af4,ed90bce8) at fork_exit+0x7e/frame 0xed90bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90bcd4 --- trap 0, eip = 0, esp = 0xed90bd20, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc8c7e620 sched_switch(c8c7e620,0,104,1af,ed90ebe8,...) at sched_switch+0x4e2/frame 0xed90ebb0 mi_switch(104,0,c11e4a45,200,c8c80cc8,...) at mi_switch+0x1bf/frame 0xed90ebe8 sleepq_switch(c8c7e620,0,c11e4a45,276,c8c80b2c,...) at sleepq_switch+0x1c5/frame 0xed90ec1c sleepq_wait(c8c80b2c,0,ed90ec70,1,0,...) at sleepq_wait+0x48/frame 0xed90ec40 _cv_wait(c8c80b2c,c8c80cc8,c11af873,76,c8c80b34,...) at _cv_wait+0x1fe/frame 0xed90ec84 usb_process(c8c80b24,ed90ece8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed90ecac fork_exit(c09bc0a0,c8c80b24,ed90ece8) at fork_exit+0x7e/frame 0xed90ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90ecd4 --- trap 0, eip = 0, esp = 0xed90ed20, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc8c7e310 sched_switch(c8c7e310,0,104,1af,ed911be8,...) at sched_switch+0x4e2/frame 0xed911bb0 mi_switch(104,0,c11e4a45,200,c8c80cc8,...) at mi_switch+0x1bf/frame 0xed911be8 sleepq_switch(c8c7e310,0,c11e4a45,276,c8c80b5c,...) at sleepq_switch+0x1c5/frame 0xed911c1c sleepq_wait(c8c80b5c,0,ed911c70,1,0,...) at sleepq_wait+0x48/frame 0xed911c40 _cv_wait(c8c80b5c,c8c80cc8,c11af873,76,c8c80b64,...) at _cv_wait+0x1fe/frame 0xed911c84 usb_process(c8c80b54,ed911ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed911cac fork_exit(c09bc0a0,c8c80b54,ed911ce8) at fork_exit+0x7e/frame 0xed911cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed911cd4 --- trap 0, eip = 0, esp = 0xed911d20, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc8c7e000 sched_switch(c8c7e000,0,104,1af,ed915be8,...) at sched_switch+0x4e2/frame 0xed915bb0 mi_switch(104,0,c11e4a45,200,c8c82cc8,...) at mi_switch+0x1bf/frame 0xed915be8 sleepq_switch(c8c7e000,0,c11e4a45,276,c8c82acc,...) at sleepq_switch+0x1c5/frame 0xed915c1c sleepq_wait(c8c82acc,0,ed915c70,1,0,...) at sleepq_wait+0x48/frame 0xed915c40 _cv_wait(c8c82acc,c8c82cc8,c11af873,76,c8c82ad4,...) at _cv_wait+0x1fe/frame 0xed915c84 usb_process(c8c82ac4,ed915ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed915cac fork_exit(c09bc0a0,c8c82ac4,ed915ce8) at fork_exit+0x7e/frame 0xed915cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed915cd4 --- trap 0, eip = 0, esp = 0xed915d20, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc8c7dc40 sched_switch(c8c7dc40,0,104,1af,ed918be8,...) at sched_switch+0x4e2/frame 0xed918bb0 mi_switch(104,0,c11e4a45,200,c8c82cc8,...) at mi_switch+0x1bf/frame 0xed918be8 sleepq_switch(c8c7dc40,0,c11e4a45,276,c8c82afc,...) at sleepq_switch+0x1c5/frame 0xed918c1c sleepq_wait(c8c82afc,0,ed918c70,1,0,...) at sleepq_wait+0x48/frame 0xed918c40 _cv_wait(c8c82afc,c8c82cc8,c11af873,76,c8c82b04,...) at _cv_wait+0x1fe/frame 0xed918c84 usb_process(c8c82af4,ed918ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed918cac fork_exit(c09bc0a0,c8c82af4,ed918ce8) at fork_exit+0x7e/frame 0xed918cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed918cd4 --- trap 0, eip = 0, esp = 0xed918d20, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc8c7d930 sched_switch(c8c7d930,0,104,1af,ed91bbe8,...) at sched_switch+0x4e2/frame 0xed91bbb0 mi_switch(104,0,c11e4a45,200,c8c82cc8,...) at mi_switch+0x1bf/frame 0xed91bbe8 sleepq_switch(c8c7d930,0,c11e4a45,276,c8c82b2c,...) at sleepq_switch+0x1c5/frame 0xed91bc1c sleepq_wait(c8c82b2c,0,ed91bc70,1,0,...) at sleepq_wait+0x48/frame 0xed91bc40 _cv_wait(c8c82b2c,c8c82cc8,c11af873,76,c8c82b34,...) at _cv_wait+0x1fe/frame 0xed91bc84 usb_process(c8c82b24,ed91bce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed91bcac fork_exit(c09bc0a0,c8c82b24,ed91bce8) at fork_exit+0x7e/frame 0xed91bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91bcd4 --- trap 0, eip = 0, esp = 0xed91bd20, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc8c7d620 sched_switch(c8c7d620,0,104,1af,ed91ebe8,...) at sched_switch+0x4e2/frame 0xed91ebb0 mi_switch(104,0,c11e4a45,200,c8c82cc8,...) at mi_switch+0x1bf/frame 0xed91ebe8 sleepq_switch(c8c7d620,0,c11e4a45,276,c8c82b5c,...) at sleepq_switch+0x1c5/frame 0xed91ec1c sleepq_wait(c8c82b5c,0,ed91ec70,1,0,...) at sleepq_wait+0x48/frame 0xed91ec40 _cv_wait(c8c82b5c,c8c82cc8,c11af873,76,c8c82b64,...) at _cv_wait+0x1fe/frame 0xed91ec84 usb_process(c8c82b54,ed91ece8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed91ecac fork_exit(c09bc0a0,c8c82b54,ed91ece8) at fork_exit+0x7e/frame 0xed91ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91ecd4 --- trap 0, eip = 0, esp = 0xed91ed20, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc8c7d310 sched_switch(c8c7d310,0,104,1af,ed922be8,...) at sched_switch+0x4e2/frame 0xed922bb0 mi_switch(104,0,c11e4a45,200,c8c88cc8,...) at mi_switch+0x1bf/frame 0xed922be8 sleepq_switch(c8c7d310,0,c11e4a45,276,c8c88acc,...) at sleepq_switch+0x1c5/frame 0xed922c1c sleepq_wait(c8c88acc,0,ed922c70,1,0,...) at sleepq_wait+0x48/frame 0xed922c40 _cv_wait(c8c88acc,c8c88cc8,c11af873,76,c8c88ad4,...) at _cv_wait+0x1fe/frame 0xed922c84 usb_process(c8c88ac4,ed922ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed922cac fork_exit(c09bc0a0,c8c88ac4,ed922ce8) at fork_exit+0x7e/frame 0xed922cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed922cd4 --- trap 0, eip = 0, esp = 0xed922d20, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc8c7d000 sched_switch(c8c7d000,0,104,1af,ed925be8,...) at sched_switch+0x4e2/frame 0xed925bb0 mi_switch(104,0,c11e4a45,200,c8c88cc8,...) at mi_switch+0x1bf/frame 0xed925be8 sleepq_switch(c8c7d000,0,c11e4a45,276,c8c88afc,...) at sleepq_switch+0x1c5/frame 0xed925c1c sleepq_wait(c8c88afc,0,ed925c70,1,0,...) at sleepq_wait+0x48/frame 0xed925c40 _cv_wait(c8c88afc,c8c88cc8,c11af873,76,c8c88b04,...) at _cv_wait+0x1fe/frame 0xed925c84 usb_process(c8c88af4,ed925ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed925cac fork_exit(c09bc0a0,c8c88af4,ed925ce8) at fork_exit+0x7e/frame 0xed925cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed925cd4 --- trap 0, eip = 0, esp = 0xed925d20, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc8be4c40 sched_switch(c8be4c40,0,104,1af,ed928be8,...) at sched_switch+0x4e2/frame 0xed928bb0 mi_switch(104,0,c11e4a45,200,c8c88cc8,...) at mi_switch+0x1bf/frame 0xed928be8 sleepq_switch(c8be4c40,0,c11e4a45,276,c8c88b2c,...) at sleepq_switch+0x1c5/frame 0xed928c1c sleepq_wait(c8c88b2c,0,ed928c70,1,0,...) at sleepq_wait+0x48/frame 0xed928c40 _cv_wait(c8c88b2c,c8c88cc8,c11af873,76,c8c88b34,...) at _cv_wait+0x1fe/frame 0xed928c84 usb_process(c8c88b24,ed928ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed928cac fork_exit(c09bc0a0,c8c88b24,ed928ce8) at fork_exit+0x7e/frame 0xed928cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed928cd4 --- trap 0, eip = 0, esp = 0xed928d20, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc8c8bc40 sched_switch(c8c8bc40,0,104,1af,ed92bbe8,...) at sched_switch+0x4e2/frame 0xed92bbb0 mi_switch(104,0,c11e4a45,200,c8c88cc8,...) at mi_switch+0x1bf/frame 0xed92bbe8 sleepq_switch(c8c8bc40,0,c11e4a45,276,c8c88b5c,...) at sleepq_switch+0x1c5/frame 0xed92bc1c sleepq_wait(c8c88b5c,0,ed92bc70,1,0,...) at sleepq_wait+0x48/frame 0xed92bc40 _cv_wait(c8c88b5c,c8c88cc8,c11af873,76,c8c88b64,...) at _cv_wait+0x1fe/frame 0xed92bc84 usb_process(c8c88b54,ed92bce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed92bcac fork_exit(c09bc0a0,c8c88b54,ed92bce8) at fork_exit+0x7e/frame 0xed92bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed92bcd4 --- trap 0, eip = 0, esp = 0xed92bd20, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc8c8b620 sched_switch(c8c8b620,0,104,1af,ed934be8,...) at sched_switch+0x4e2/frame 0xed934bb0 mi_switch(104,0,c11e4a45,200,c8c33ea0,...) at mi_switch+0x1bf/frame 0xed934be8 sleepq_switch(c8c8b620,0,c11e4a45,276,c8c33ca4,...) at sleepq_switch+0x1c5/frame 0xed934c1c sleepq_wait(c8c33ca4,0,ed934c70,1,0,...) at sleepq_wait+0x48/frame 0xed934c40 _cv_wait(c8c33ca4,c8c33ea0,c11af873,76,c8c33cac,...) at _cv_wait+0x1fe/frame 0xed934c84 usb_process(c8c33c9c,ed934ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed934cac fork_exit(c09bc0a0,c8c33c9c,ed934ce8) at fork_exit+0x7e/frame 0xed934cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed934cd4 --- trap 0, eip = 0, esp = 0xed934d20, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc8c8b310 sched_switch(c8c8b310,0,104,1af,ed937be8,...) at sched_switch+0x4e2/frame 0xed937bb0 mi_switch(104,0,c11e4a45,200,c8c33ea0,...) at mi_switch+0x1bf/frame 0xed937be8 sleepq_switch(c8c8b310,0,c11e4a45,276,c8c33cd4,...) at sleepq_switch+0x1c5/frame 0xed937c1c sleepq_wait(c8c33cd4,0,ed937c70,1,0,...) at sleepq_wait+0x48/frame 0xed937c40 _cv_wait(c8c33cd4,c8c33ea0,c11af873,76,c8c33cdc,...) at _cv_wait+0x1fe/frame 0xed937c84 usb_process(c8c33ccc,ed937ce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed937cac fork_exit(c09bc0a0,c8c33ccc,ed937ce8) at fork_exit+0x7e/frame 0xed937cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed937cd4 --- trap 0, eip = 0, esp = 0xed937d20, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc8c8b000 sched_switch(c8c8b000,0,104,1af,ed93abe8,...) at sched_switch+0x4e2/frame 0xed93abb0 mi_switch(104,0,c11e4a45,200,c8c33ea0,...) at mi_switch+0x1bf/frame 0xed93abe8 sleepq_switch(c8c8b000,0,c11e4a45,276,c8c33d04,...) at sleepq_switch+0x1c5/frame 0xed93ac1c sleepq_wait(c8c33d04,0,ed93ac70,1,0,...) at sleepq_wait+0x48/frame 0xed93ac40 _cv_wait(c8c33d04,c8c33ea0,c11af873,76,c8c33d0c,...) at _cv_wait+0x1fe/frame 0xed93ac84 usb_process(c8c33cfc,ed93ace8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed93acac fork_exit(c09bc0a0,c8c33cfc,ed93ace8) at fork_exit+0x7e/frame 0xed93acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93acd4 --- trap 0, eip = 0, esp = 0xed93ad20, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc8c8ac40 sched_switch(c8c8ac40,0,104,1af,ed93dbe8,...) at sched_switch+0x4e2/frame 0xed93dbb0 mi_switch(104,0,c11e4a45,200,c8c33ea0,...) at mi_switch+0x1bf/frame 0xed93dbe8 sleepq_switch(c8c8ac40,0,c11e4a45,276,c8c33d34,...) at sleepq_switch+0x1c5/frame 0xed93dc1c sleepq_wait(c8c33d34,0,ed93dc70,1,0,...) at sleepq_wait+0x48/frame 0xed93dc40 _cv_wait(c8c33d34,c8c33ea0,c11af873,76,c8c33d3c,...) at _cv_wait+0x1fe/frame 0xed93dc84 usb_process(c8c33d2c,ed93dce8,c11d76fc,3dc,0,...) at usb_process+0x140/frame 0xed93dcac fork_exit(c09bc0a0,c8c33d2c,ed93dce8) at fork_exit+0x7e/frame 0xed93dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93dcd4 --- trap 0, eip = 0, esp = 0xed93dd20, ebp = 0 --- Tracing command cam pid 2 tid 100026 td 0xc8a3c930 sched_switch(c8a3c930,0,104,1af,e4781ba4,...) at sched_switch+0x4e2/frame 0xe4781b6c mi_switch(104,0,c11e4a45,200,c1493b00,...) at mi_switch+0x1bf/frame 0xe4781ba4 sleepq_switch(c8a3c930,0,c11e4a45,276,c8a3c930,...) at sleepq_switch+0x1c5/frame 0xe4781bd8 sleepq_wait(c1493b80,5c,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe4781bfc _sleep(c1493b80,c1493b00,5c,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe4781c58 xpt_done_td(c1493b00,e4781ce8,c11d76fc,3dc,0,...) at xpt_done_td+0x17e/frame 0xe4781cac fork_exit(c04c7fa0,c1493b00,e4781ce8) at fork_exit+0x7e/frame 0xe4781cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4781cd4 --- trap 0, eip = 0, esp = 0xe4781d20, ebp = 0 --- Tracing command cam pid 2 tid 100068 td 0xc8efe000 sched_switch(c8efe000,0,104,1af,ed977ba4,...) at sched_switch+0x4e2/frame 0xed977b6c mi_switch(104,0,c11e4a45,200,c1493a7c,...) at mi_switch+0x1bf/frame 0xed977ba4 sleepq_switch(c8efe000,0,c11e4a45,276,c8efe000,...) at sleepq_switch+0x1c5/frame 0xed977bd8 sleepq_wait(c1493a2c,5c,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xed977bfc _sleep(c1493a2c,c1493a7c,5c,c11c95c6,0,...) at _sleep+0x3a1/frame 0xed977c58 xpt_scanner_thread(0,ed977ce8,c11d76fc,3dc,0,...) at xpt_scanner_thread+0xa2/frame 0xed977cac fork_exit(c04c8360,0,ed977ce8) at fork_exit+0x7e/frame 0xed977cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed977cd4 --- trap 0, eip = 0, esp = 0xed977d20, ebp = 0 --- Tracing command rand_harvestq pid 14 tid 100017 td 0xc8aee620 sched_switch(c8aee620,0,104,1af,e4766bc8,...) at sched_switch+0x4e2/frame 0xe4766b90 mi_switch(104,0,c11e4a45,200,5,...) at mi_switch+0x1bf/frame 0xe4766bc8 sleepq_switch(c8aee620,0,c11e4a45,299,c8aee620,...) at sleepq_switch+0x1c5/frame 0xe4766bfc sleepq_timedwait(c14ae770,0,0,0,0,...) at sleepq_timedwait+0x48/frame 0xe4766c20 _sleep(c14ae770,0,0,c11c95c6,19999999,...) at _sleep+0x33d/frame 0xe4766c7c random_kthread(0,e4766ce8,c11d76fc,3dc,0,...) at random_kthread+0x5c/frame 0xe4766cac fork_exit(c089ae90,0,e4766ce8) at fork_exit+0x7e/frame 0xe4766cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4766cd4 --- trap 0, eip = 0, esp = 0xe4766d20, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc8a35000 sched_switch(c8a35000,0,104,1af,e475abbc,...) at sched_switch+0x4e2/frame 0xe475ab84 mi_switch(104,0,c11e4a45,200,c155a65c,...) at mi_switch+0x1bf/frame 0xe475abbc sleepq_switch(c8a35000,0,c11e4a45,276,c8a35000,...) at sleepq_switch+0x1c5/frame 0xe475abf0 sleepq_wait(c1630620,5c,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe475ac14 _sleep(c1630620,c155a65c,25c,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe475ac70 g_run_events(0,e475ace8,c11d76fc,3dc,0,...) at g_run_events+0x67/frame 0xe475acac fork_exit(c0ab38d0,0,e475ace8) at fork_exit+0x7e/frame 0xe475acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe475acd4 --- trap 0, eip = 0, esp = 0xe475ad20, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc8aef000 sched_switch(c8aef000,0,104,1af,e475dbac,...) at sched_switch+0x4e2/frame 0xe475db74 mi_switch(104,0,c11e4a45,200,c155a6a4,...) at mi_switch+0x1bf/frame 0xe475dbac sleepq_switch(c8aef000,0,c11e4a45,276,c8aef000,...) at sleepq_switch+0x1c5/frame 0xe475dbe0 sleepq_wait(c1630624,5c,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe475dc04 _sleep(c1630624,c155a6a4,25c,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe475dc60 g_io_schedule_up(c8aef000,0,c11cb39d,60,e475dcd4,...) at g_io_schedule_up+0x12a/frame 0xe475dc94 g_up_procbody(0,e475dce8,c11d76fc,3dc,0,...) at g_up_procbody+0x9d/frame 0xe475dcac fork_exit(c0ab3970,0,e475dce8) at fork_exit+0x7e/frame 0xe475dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe475dcd4 --- trap 0, eip = 0, esp = 0xe475dd20, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc8aeec40 sched_switch(c8aeec40,0,104,1af,e4760ba8,...) at sched_switch+0x4e2/frame 0xe4760b70 mi_switch(104,0,c11e4a45,200,c155a684,...) at mi_switch+0x1bf/frame 0xe4760ba8 sleepq_switch(c8aeec40,0,c11e4a45,276,c8aeec40,...) at sleepq_switch+0x1c5/frame 0xe4760bdc sleepq_wait(c1630628,5c,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe4760c00 _sleep(c1630628,c155a684,25c,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe4760c5c g_io_schedule_down(c8aeec40,0,c11cb39d,6d,e4760cd4,...) at g_io_schedule_down+0x16c/frame 0xe4760c94 g_down_procbody(0,e4760ce8,c11d76fc,3dc,0,...) at g_down_procbody+0x9d/frame 0xe4760cac fork_exit(c0ab3a10,0,e4760ce8) at fork_exit+0x7e/frame 0xe4760cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4760cd4 --- trap 0, eip = 0, esp = 0xe4760d20, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc8a3b310 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc8a3b000 sched_switch(c8a3b000,0,109,1af,c15ab428,...) at sched_switch+0x4e2/frame 0xe474bc3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xe474bc74 ithread_loop(c88dff90,e474bce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xe474bcac fork_exit(c0b2c8c0,c88dff90,e474bce8) at fork_exit+0x7e/frame 0xe474bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe474bcd4 --- trap 0, eip = 0, esp = 0xe474bd20, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc8a35c40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc8a35930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc8a35620 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc8a35310 sched_switch(c8a35310,0,109,1af,c15a9dd0,...) at sched_switch+0x4e2/frame 0xe4757c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xe4757c74 ithread_loop(c88dff50,e4757ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4757cac fork_exit(c0b2c8c0,c88dff50,e4757ce8) at fork_exit+0x7e/frame 0xe4757cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4757cd4 --- trap 0, eip = 0, esp = 0xe4757d20, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc8aee310 sched_switch(c8aee310,0,109,1af,c15aa140,...) at sched_switch+0x4e2/frame 0xe4769c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xe4769c74 ithread_loop(c88dff10,e4769ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4769cac fork_exit(c0b2c8c0,c88dff10,e4769ce8) at fork_exit+0x7e/frame 0xe4769cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4769cd4 --- trap 0, eip = 0, esp = 0xe4769d20, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc8aed930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100027 td 0xc8be4930 sched_switch(c8be4930,0,109,1af,c15a9ed8,...) at sched_switch+0x4e2/frame 0xe4784c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xe4784c74 ithread_loop(c88dfe90,e4784ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4784cac fork_exit(c0b2c8c0,c88dfe90,e4784ce8) at fork_exit+0x7e/frame 0xe4784cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4784cd4 --- trap 0, eip = 0, esp = 0xe4784d20, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc8be4620 sched_switch(c8be4620,0,109,1af,c15ab4d8,...) at sched_switch+0x4e2/frame 0xe47b6c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xe47b6c74 ithread_loop(c88dfe40,e47b6ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xe47b6cac fork_exit(c0b2c8c0,c88dfe40,e47b6ce8) at fork_exit+0x7e/frame 0xe47b6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe47b6cd4 --- trap 0, eip = 0, esp = 0xe47b6d20, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc8be4310 sched_switch(c8be4310,0,109,1af,c15a9e80,...) at sched_switch+0x4e2/frame 0xf0ebfc3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xf0ebfc74 ithread_loop(c88dfda0,f0ebfce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xf0ebfcac fork_exit(c0b2c8c0,c88dfda0,f0ebfce8) at fork_exit+0x7e/frame 0xf0ebfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ebfcd4 --- trap 0, eip = 0, esp = 0xf0ebfd20, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc8be4000 sched_switch(c8be4000,0,109,1af,c15aa560,...) at sched_switch+0x4e2/frame 0xf0ec3c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xf0ec3c74 ithread_loop(c88dfd90,f0ec3ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xf0ec3cac fork_exit(c0b2c8c0,c88dfd90,f0ec3ce8) at fork_exit+0x7e/frame 0xf0ec3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ec3cd4 --- trap 0, eip = 0, esp = 0xf0ec3d20, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc8be3c40 sched_switch(c8be3c40,0,109,1af,c15ab480,...) at sched_switch+0x4e2/frame 0xed8e5c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xed8e5c74 ithread_loop(c88dfc20,ed8e5ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xed8e5cac fork_exit(c0b2c8c0,c88dfc20,ed8e5ce8) at fork_exit+0x7e/frame 0xed8e5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8e5cd4 --- trap 0, eip = 0, esp = 0xed8e5d20, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc8aefc40 sched_switch(c8aefc40,0,109,1af,c15aa820,...) at sched_switch+0x4e2/frame 0xed8f5c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xed8f5c74 ithread_loop(c88dfbe0,ed8f5ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xed8f5cac fork_exit(c0b2c8c0,c88dfbe0,ed8f5ce8) at fork_exit+0x7e/frame 0xed8f5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f5cd4 --- trap 0, eip = 0, esp = 0xed8f5d20, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc8c7f000 sched_switch(c8c7f000,0,109,1af,c15aa7c8,...) at sched_switch+0x4e2/frame 0xed905c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xed905c74 ithread_loop(c88dfba0,ed905ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xed905cac fork_exit(c0b2c8c0,c88dfba0,ed905ce8) at fork_exit+0x7e/frame 0xed905cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed905cd4 --- trap 0, eip = 0, esp = 0xed905d20, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc8c8b930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xc8c8a930 sched_switch(c8c8a930,0,109,1af,c15aa508,...) at sched_switch+0x4e2/frame 0xed943c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xed943c74 ithread_loop(c88dfaa0,ed943ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xed943cac fork_exit(c0b2c8c0,c88dfaa0,ed943ce8) at fork_exit+0x7e/frame 0xed943cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed943cd4 --- trap 0, eip = 0, esp = 0xed943d20, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc8c8a620 sched_switch(c8c8a620,0,109,1af,c15aa0e8,...) at sched_switch+0x4e2/frame 0xed954c3c mi_switch(109,0,c11d796d,544,8,...) at mi_switch+0x1bf/frame 0xed954c74 ithread_loop(c88df9a0,ed954ce8,c11d76fc,3dc,0,...) at ithread_loop+0x1e1/frame 0xed954cac fork_exit(c0b2c8c0,c88df9a0,ed954ce8) at fork_exit+0x7e/frame 0xed954cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed954cd4 --- trap 0, eip = 0, esp = 0xed954d20, ebp = 0 --- Tracing command intr pid 12 tid 100062 td 0xc8c8a000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc8c7fc40 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc8a3c000 cpustop_handler(0,13,e473bb74,c105bf27,369e99,...) at cpustop_handler+0x25/frame 0xe473ba28 ipi_nmi_handler(369e99,b,0,c8a3c000,0,...) at ipi_nmi_handler+0x36/frame 0xe473ba38 trap(e473bb80) at trap+0x37/frame 0xe473bb74 calltrap() at calltrap+0x6/frame 0xe473bb74 --- trap 0x13, eip = 0xc10442b5, esp = 0xe473bbc0, ebp = 0xe473bbc0 --- acpi_cpu_c1(c1c25c90,0,c13a1a03,2ec,bed1,...) at acpi_cpu_c1+0x5/frame 0xe473bbc0 acpi_cpu_idle(a5cd66d,0,a5cd66d,e473bc30,c107547d,...) at acpi_cpu_idle+0x166/frame 0xe473bc00 cpu_idle_acpi(a5cd66d,0,0,c1569d88,c1569d84,...) at cpu_idle_acpi+0x3f/frame 0xe473bc14 cpu_idle(0,e473bc80,c11e0aa2,a48,c1569d88,...) at cpu_idle+0x9d/frame 0xe473bc30 sched_idletd(0,e473bce8,c11d76fc,3dc,0,...) at sched_idletd+0x3cb/frame 0xe473bcac fork_exit(c0b8b880,0,e473bce8) at fork_exit+0x7e/frame 0xe473bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe473bcd4 --- trap 0, eip = 0, esp = 0xe473bd20, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc8a3bc40 cpustop_handler(1,13,e473eb74,c105bf27,1,...) at cpustop_handler+0x25/frame 0xe473ea28 ipi_nmi_handler(1,0,416630,0,e473ea7c,...) at ipi_nmi_handler+0x36/frame 0xe473ea38 trap(e473eb80) at trap+0x37/frame 0xe473eb74 calltrap() at calltrap+0x6/frame 0xe473eb74 --- trap 0x13, eip = 0xc10442b5, esp = 0xe473ebc0, ebp = 0xe473ebc0 --- acpi_cpu_c1(e472bc90,0,c13a1a03,2ec,bed1,...) at acpi_cpu_c1+0x5/frame 0xe473ebc0 acpi_cpu_idle(23bd138c,0,23bd138c,e473ec30,c107547d,...) at acpi_cpu_idle+0x166/frame 0xe473ec00 cpu_idle_acpi(23bd138c,0,5,c156a488,c156a484,...) at cpu_idle_acpi+0x3f/frame 0xe473ec14 cpu_idle(0,e473ec80,c11e0aa2,a48,c156a488,...) at cpu_idle+0x9d/frame 0xe473ec30 sched_idletd(0,e473ece8,c11d76fc,3dc,0,...) at sched_idletd+0x3cb/frame 0xe473ecac fork_exit(c0b8b880,0,e473ece8) at fork_exit+0x7e/frame 0xe473ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe473ecd4 --- trap 0, eip = 0, esp = 0xe473ed20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc8a3b930 kdb_alt_break_internal(0,e4741ac8,c0980037,2,c8c18884,...) at kdb_alt_break_internal+0x162/frame 0xe4741a9c kdb_alt_break(2,c8c18884,c11a6781,c8c18800,40000,...) at kdb_alt_break+0x16/frame 0xe4741aa8 uart_intr_rxready(c8c18800,e4741b04,c107ab03,0,0,...) at uart_intr_rxready+0x87/frame 0xe4741ac8 uart_intr(c8c18800,369e99,b,0,0,...) at uart_intr+0x100/frame 0xe4741b04 intr_event_handle(c8a39680,e4741b80,0,2,0,...) at intr_event_handle+0x7a/frame 0xe4741b30 intr_execute_handlers(c88b5908,e4741b80,2) at intr_execute_handlers+0x42/frame 0xe4741b50 lapic_handle_intr(30,e4741b80) at lapic_handle_intr+0x5d/frame 0xe4741b70 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xe4741b70 --- interrupt, eip = 0xc10442b5, esp = 0xe4741bc0, ebp = 0xe4741bc0 --- acpi_cpu_c1(e472ec90,0,c13a1a03,2ec,bed1,...) at acpi_cpu_c1+0x5/frame 0xe4741bc0 acpi_cpu_idle(296f86b5,0,296f86b5,e4741c30,c107547d,...) at acpi_cpu_idle+0x166/frame 0xe4741c00 cpu_idle_acpi(296f86b5,0,5,c156ab88,c156ab84,...) at cpu_idle_acpi+0x3f/frame 0xe4741c14 cpu_idle(0,e4741c80,c11e0aa2,407,c156ab88,...) at cpu_idle+0x9d/frame 0xe4741c30 sched_idletd(0,e4741ce8,c11d76fc,3dc,0,...) at sched_idletd+0x3cb/frame 0xe4741cac fork_exit(c0b8b880,0,e4741ce8) at fork_exit+0x7e/frame 0xe4741cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4741cd4 --- trap 0, eip = 0, esp = 0xe4741d20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc8a3b620 cpustop_handler(3,13,e4744b74,c105bf27,3,...) at cpustop_handler+0x25/frame 0xe4744a28 ipi_nmi_handler(3,0,202ac6,0,e4744a7c,...) at ipi_nmi_handler+0x36/frame 0xe4744a38 trap(e4744b80) at trap+0x37/frame 0xe4744b74 calltrap() at calltrap+0x6/frame 0xe4744b74 --- trap 0x13, eip = 0xc10442b5, esp = 0xe4744bc0, ebp = 0xe4744bc0 --- acpi_cpu_c1(e4731c90,0,c13a1a03,2ec,bed1,...) at acpi_cpu_c1+0x5/frame 0xe4744bc0 acpi_cpu_idle(3a8db23a,0,3a8db23a,e4744c30,c107547d,...) at acpi_cpu_idle+0x166/frame 0xe4744c00 cpu_idle_acpi(3a8db23a,0,1,c156b288,c156b284,...) at cpu_idle_acpi+0x3f/frame 0xe4744c14 cpu_idle(0,e4744c80,c11e0aa2,407,c156b288,...) at cpu_idle+0x9d/frame 0xe4744c30 sched_idletd(0,e4744ce8,c11d76fc,3dc,0,...) at sched_idletd+0x3cb/frame 0xe4744cac fork_exit(c0b8b880,0,e4744ce8) at fork_exit+0x7e/frame 0xe4744cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4744cd4 --- trap 0, eip = 0, esp = 0xe4744d20, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc8a3c310 sched_switch(c8a3c310,0,104,1af,e47379a0,...) at sched_switch+0x4e2/frame 0xe4737968 mi_switch(104,0,c11e4a45,200,c121c6fc,...) at mi_switch+0x1bf/frame 0xe47379a0 sleepq_switch(c8a3c310,0,c11e4a45,1b6,c11e4a45,...) at sleepq_switch+0x1c5/frame 0xe47379d4 sleepq_catch_signals(c11e4a45,16d,c8a33a78,1758,c8a3c310,...) at sleepq_catch_signals+0x382/frame 0xe4737a10 sleepq_wait_sig(c8a339c0,6c,c11e87dd,100,0,...) at sleepq_wait_sig+0x14/frame 0xe4737a34 _sleep(c8a339c0,c8a33a68,16c,c11e87dd,0,...) at _sleep+0x356/frame 0xe4737a90 kern_wait6(c8a3c310,7,0,0,e4737b24,...) at kern_wait6+0x45b/frame 0xe4737af8 sys_wait4(c8a3c310,e4737ca8,c11d731d,4,c8a45000,...) at sys_wait4+0x8b/frame 0xe4737c18 syscall(e4737ce8) at syscall+0x34a/frame 0xe4737cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe4737cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80653ef, esp = 0xbfbfe87c, ebp = 0xbfbfe890 --- Tracing command audit pid 10 tid 100001 td 0xc8a3c620 sched_switch(c8a3c620,0,104,1af,e4734bbc,...) at sched_switch+0x4e2/frame 0xe4734b84 mi_switch(104,0,c11e4a45,200,c163735c,...) at mi_switch+0x1bf/frame 0xe4734bbc sleepq_switch(c8a3c620,0,c11e4a45,276,c16373d0,...) at sleepq_switch+0x1c5/frame 0xe4734bf0 sleepq_wait(c16373d0,0,e4734c44,1,0,...) at sleepq_wait+0x48/frame 0xe4734c14 _cv_wait(c16373d0,c163735c,c1211a56,18d,0,...) at _cv_wait+0x1fe/frame 0xe4734c58 audit_worker(0,e4734ce8,c11d76fc,3dc,0,...) at audit_worker+0x94/frame 0xe4734cac fork_exit(c0da3c10,0,e4734ce8) at fork_exit+0x7e/frame 0xe4734cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4734cd4 --- trap 0, eip = 0, esp = 0xe4734d20, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1630990 sched_switch(c1630990,0,104,1af,c1c20c3c,...) at sched_switch+0x4e2/frame 0xc1c20c04 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xc1c20c3c sleepq_switch(c1630990,0,c11e4a45,299,c1630990,...) at sleepq_switch+0x1c5/frame 0xc1c20c70 sleepq_timedwait(c1630644,54,9,0,0,...) at sleepq_timedwait+0x48/frame 0xc1c20c94 _sleep(c1630644,0,54,c121dc1a,fffff470,...) at _sleep+0x33d/frame 0xc1c20cf0 swapper() at swapper+0x2c6/frame 0xc1c20d38 begin() at begin+0x22 Tracing command kernel pid 0 tid 100016 td 0xc8aee930 sched_switch(c8aee930,0,104,1af,e4763bc0,...) at sched_switch+0x4e2/frame 0xe4763b88 mi_switch(104,0,c11e4a45,200,c8a38198,...) at mi_switch+0x1bf/frame 0xe4763bc0 sleepq_switch(c8aee930,0,c11e4a45,276,c8aee930,...) at sleepq_switch+0x1c5/frame 0xe4763bf4 sleepq_wait(c8a38180,0,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe4763c18 _sleep(c8a38180,c8a38198,0,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe4763c74 taskqueue_thread_loop(c1577e44,e4763ce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe4763cac fork_exit(c0bb4280,c1577e44,e4763ce8) at fork_exit+0x7e/frame 0xe4763cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4763cd4 --- trap 0, eip = 0, esp = 0xe4763d20, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc8aee000 sched_switch(c8aee000,0,104,1af,e476cbc0,...) at sched_switch+0x4e2/frame 0xe476cb88 mi_switch(104,0,c11e4a45,200,c8a38018,...) at mi_switch+0x1bf/frame 0xe476cbc0 sleepq_switch(c8aee000,0,c11e4a45,276,c8aee000,...) at sleepq_switch+0x1c5/frame 0xe476cbf4 sleepq_wait(c8a38000,0,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe476cc18 _sleep(c8a38000,c8a38018,0,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe476cc74 taskqueue_thread_loop(c1637404,e476cce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe476ccac fork_exit(c0bb4280,c1637404,e476cce8) at fork_exit+0x7e/frame 0xe476ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe476ccd4 --- trap 0, eip = 0, esp = 0xe476cd20, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc8aedc40 sched_switch(c8aedc40,0,104,1af,e476fbc0,...) at sched_switch+0x4e2/frame 0xe476fb88 mi_switch(104,0,c11e4a45,200,c8a37f18,...) at mi_switch+0x1bf/frame 0xe476fbc0 sleepq_switch(c8aedc40,0,c11e4a45,276,c8aedc40,...) at sleepq_switch+0x1c5/frame 0xe476fbf4 sleepq_wait(c8a37f00,0,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe476fc18 _sleep(c8a37f00,c8a37f18,0,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe476fc74 taskqueue_thread_loop(c1634d10,e476fce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe476fcac fork_exit(c0bb4280,c1634d10,e476fce8) at fork_exit+0x7e/frame 0xe476fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe476fcd4 --- trap 0, eip = 0, esp = 0xe476fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc8aed620 sched_switch(c8aed620,0,104,1af,e4775bc0,...) at sched_switch+0x4e2/frame 0xe4775b88 mi_switch(104,0,c11e4a45,200,c8a37d98,...) at mi_switch+0x1bf/frame 0xe4775bc0 sleepq_switch(c8aed620,0,c11e4a45,276,c8aed620,...) at sleepq_switch+0x1c5/frame 0xe4775bf4 sleepq_wait(c8a37d80,0,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xe4775c18 _sleep(c8a37d80,c8a37d98,0,c11c95c6,0,...) at _sleep+0x3a1/frame 0xe4775c74 taskqueue_thread_loop(c1630e98,e4775ce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe4775cac fork_exit(c0bb4280,c1630e98,e4775ce8) at fork_exit+0x7e/frame 0xe4775cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4775cd4 --- trap 0, eip = 0, esp = 0xe4775d20, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc8aed310 sched_switch(c8aed310,0,104,1af,e4778bd4,...) at sched_switch+0x4e2/frame 0xe4778b9c mi_switch(104,0,c11e4a45,200,c1579344,...) at mi_switch+0x1bf/frame 0xe4778bd4 sleepq_switch(c8aed310,0,c11e4a45,276,c8a37d00,...) at sleepq_switch+0x1c5/frame 0xe4778c08 sleepq_wait(c8a37d00,0,c11dedfa,c11c95c6,0,...) at sleepq_wait+0x48/frame 0xe4778c2c msleep_spin_sbt(c8a37d00,c8a37d18,c11c95c6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe4778c74 taskqueue_thread_loop(c1627978,e4778ce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4778cac fork_exit(c0bb4280,c1627978,e4778ce8) at fork_exit+0x7e/frame 0xe4778cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4778cd4 --- trap 0, eip = 0, esp = 0xe4778d20, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc8aed000 sched_switch(c8aed000,0,104,1af,e477bbd4,...) at sched_switch+0x4e2/frame 0xe477bb9c mi_switch(104,0,c11e4a45,200,c1579344,...) at mi_switch+0x1bf/frame 0xe477bbd4 sleepq_switch(c8aed000,0,c11e4a45,276,c8a37d00,...) at sleepq_switch+0x1c5/frame 0xe477bc08 sleepq_wait(c8a37d00,0,c11dedfa,c11c95c6,0,...) at sleepq_wait+0x48/frame 0xe477bc2c msleep_spin_sbt(c8a37d00,c8a37d18,c11c95c6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe477bc74 taskqueue_thread_loop(c1627978,e477bce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe477bcac fork_exit(c0bb4280,c1627978,e477bce8) at fork_exit+0x7e/frame 0xe477bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe477bcd4 --- trap 0, eip = 0, esp = 0xe477bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc8a3cc40 sched_switch(c8a3cc40,0,104,1af,e477ebd4,...) at sched_switch+0x4e2/frame 0xe477eb9c mi_switch(104,0,c11e4a45,200,c1579344,...) at mi_switch+0x1bf/frame 0xe477ebd4 sleepq_switch(c8a3cc40,0,c11e4a45,276,c8a37d00,...) at sleepq_switch+0x1c5/frame 0xe477ec08 sleepq_wait(c8a37d00,0,c11dedfa,c11c95c6,0,...) at sleepq_wait+0x48/frame 0xe477ec2c msleep_spin_sbt(c8a37d00,c8a37d18,c11c95c6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe477ec74 taskqueue_thread_loop(c1627978,e477ece8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe477ecac fork_exit(c0bb4280,c1627978,e477ece8) at fork_exit+0x7e/frame 0xe477ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe477ecd4 --- trap 0, eip = 0, esp = 0xe477ed20, ebp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xc8c7f930 sched_switch(c8c7f930,0,104,1af,ed96abd4,...) at sched_switch+0x4e2/frame 0xed96ab9c mi_switch(104,0,c11e4a45,200,c1579464,...) at mi_switch+0x1bf/frame 0xed96abd4 sleepq_switch(c8c7f930,0,c11e4a45,276,c8cb8900,...) at sleepq_switch+0x1c5/frame 0xed96ac08 sleepq_wait(c8cb8900,0,c11dedfa,c11c95c6,0,...) at sleepq_wait+0x48/frame 0xed96ac2c msleep_spin_sbt(c8cb8900,c8cb8918,c11c95c6,0,0,...) at msleep_spin_sbt+0x241/frame 0xed96ac74 taskqueue_thread_loop(c1625e14,ed96ace8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xed96acac fork_exit(c0bb4280,c1625e14,ed96ace8) at fork_exit+0x7e/frame 0xed96acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed96acd4 --- trap 0, eip = 0, esp = 0xed96ad20, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc8c7f620 sched_switch(c8c7f620,0,104,1af,ed96db80,...) at sched_switch+0x4e2/frame 0xed96db48 mi_switch(104,0,c11e4a45,200,101,...) at mi_switch+0x1bf/frame 0xed96db80 sleepq_switch(c8c7f620,0,c11e4a45,299,c8c7f620,...) at sleepq_switch+0x1c5/frame 0xed96dbb4 sleepq_timedwait(c155c9ee,0,2,0,0,...) at sleepq_timedwait+0x48/frame 0xed96dbd8 _sleep(c155c9ee,0,0,c11c95c6,fffffc88,...) at _sleep+0x33d/frame 0xed96dc34 pause_sbt(c11c95c6,fffffc88,2,0,0,...) at pause_sbt+0x112/frame 0xed96dc70 deadlkres(0,ed96dce8,c11d76fc,3dc,0,...) at deadlkres+0x81/frame 0xed96dcac fork_exit(c0b07750,0,ed96dce8) at fork_exit+0x7e/frame 0xed96dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed96dcd4 --- trap 0, eip = 0, esp = 0xed96dd20, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc8efe310 sched_switch(c8efe310,0,104,1af,ed974bc0,...) at sched_switch+0x4e2/frame 0xed974b88 mi_switch(104,0,c11e4a45,200,c8a37c98,...) at mi_switch+0x1bf/frame 0xed974bc0 sleepq_switch(c8efe310,0,c11e4a45,276,c8efe310,...) at sleepq_switch+0x1c5/frame 0xed974bf4 sleepq_wait(c8a37c80,0,c11c95c6,0,0,...) at sleepq_wait+0x48/frame 0xed974c18 _sleep(c8a37c80,c8a37c98,0,c11c95c6,0,...) at _sleep+0x3a1/frame 0xed974c74 taskqueue_thread_loop(c1493aa4,ed974ce8,c11d76fc,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xed974cac fork_exit(c0bb4280,c1493aa4,ed974ce8) at fork_exit+0x7e/frame 0xed974cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed974cd4 --- trap 0, eip = 0, esp = 0xed974d20, ebp = 0 --- db:0:allt> call doadump Physical memory: 3431 MB Dumping 384 MB: 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 83312 [Switching to thread 82 (Thread 103296)]#0 sched_switch (td=0xd572dc40, newtd=, flags=) at ../../../kern/sched_ule.c:1956 1956 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=0xd572dc40, newtd=, flags=) at ../../../kern/sched_ule.c:1956 #1 0xc0b69bef in mi_switch (flags=, newtd=0xc156b210) at ../../../kern/kern_synch.c:481 #2 0xc0baff05 in sleepq_switch (wchan=, pri=0x0) at ../../../kern/subr_sleepqueue.c:552 #3 0xc0bafce8 in sleepq_wait (wchan=0xc157b300, pri=0x0) at ../../../kern/subr_sleepqueue.c:631 #4 0xc0b07ffe in _cv_wait (cvp=0xc157b300, lock=0xc157b280) at ../../../kern/kern_condvar.c:139 #5 0xc0bbbe5f in vmem_xalloc (vm=0xc157b280, size0=0x281000, align=, phase=0xc1fb7110, nocross=, minaddr=0xc1453ae8, maxaddr=0xd572dc40, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1211 #6 0xc0bbbc14 in vmem_alloc (vm=0xc157b280, size=0x281000, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1097 #7 0xc0e1dc9b in kmem_malloc (vmem=0xc157b280, size=0x281000, flags=0x102) at ../../../vm/vm_kern.c:315 #8 0xc0e15b70 in uma_large_malloc (size=, wait=0x102) at ../../../vm/uma_core.c:1105 #9 0xc0b43b57 in malloc (size=0x281000, mtp=, flags=) at ../../../kern/kern_malloc.c:511 #10 0xc0b1624e in fdgrowtable (fdp=0xda8e9374, nfd=) at ../../../kern/kern_descrip.c:1551 #11 0xc0b1571c in fdalloc (td=0xd572dc40, minfd=, result=0x0) at ../../../kern/kern_descrip.c:1507 #12 0xc0b15e1d in finstall (td=, fp=, fd=0xfb786b34, flags=0x0, fcaps=) at ../../../kern/kern_descrip.c:1770 #13 0xc0c234be in kern_openat (td=0xd572dc40, fd=, path=, pathseg=, flags=, mode=) at ../../../kern/vfs_syscalls.c:1158 #14 0xc0c22f79 in sys_open (td=0xd572dc40, uap=0xfb786ca8) at ../../../kern/vfs_syscalls.c:1022 #15 0xc105d48a in syscall (frame=) at subr_syscall.c:133 #16 0xc1048731 in Xint0x80_syscall () at ../../../i386/i386/exception.s:269 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 10 #10 0xc0b1624e in fdgrowtable (fdp=0xda8e9374, nfd=) at ../../../kern/kern_descrip.c:1551 1551 ntable = malloc(offsetof(struct fdescenttbl, fdt_ofiles) + (kgdb) l 1546 * entries, file entries themselves and the struct freetable we will use 1547 * when we decommission the table and place it on the freelist. 1548 * We place the struct freetable in the middle so we don't have 1549 * to worry about padding. 1550 */ 1551 ntable = malloc(offsetof(struct fdescenttbl, fdt_ofiles) + 1552 nnfiles * sizeof(ntable->fdt_ofiles[0]) + 1553 sizeof(struct freetable), 1554 M_FILEDESC, M_ZERO | M_WAITOK); 1555 /* copy the old data */ (kgdb) info loc nnfiles = 0x10000 onfiles = 0x8000 omap = (u_long *) 0xdebe5000 otable = (struct fdescenttbl *) 0xdaea8000 (kgdb) proc 83436 [Switching to thread 84 (Thread 103297)]#0 sched_switch (td=0xd43f4930, newtd=, flags=) at ../../../kern/sched_ule.c:1956 1956 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xd43f4930, newtd=, flags=) at ../../../kern/sched_ule.c:1956 #1 0xc0b69bef in mi_switch (flags=, newtd=0xc156ab10) at ../../../kern/kern_synch.c:481 #2 0xc0baff05 in sleepq_switch (wchan=, pri=0x0) at ../../../kern/subr_sleepqueue.c:552 #3 0xc0bafce8 in sleepq_wait (wchan=0xc157b300, pri=0x0) at ../../../kern/subr_sleepqueue.c:631 #4 0xc0b07ffe in _cv_wait (cvp=0xc157b300, lock=0xc157b280) at ../../../kern/kern_condvar.c:139 #5 0xc0bbbe5f in vmem_xalloc (vm=0xc157b280, size0=0x281000, align=, phase=0xc1fb7110, nocross=, minaddr=0xc1453ae8, maxaddr=0xd43f4930, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1211 #6 0xc0bbbc14 in vmem_alloc (vm=0xc157b280, size=0x281000, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1097 #7 0xc0e1dc9b in kmem_malloc (vmem=0xc157b280, size=0x281000, flags=0x102) at ../../../vm/vm_kern.c:315 #8 0xc0e15b70 in uma_large_malloc (size=, wait=0x102) at ../../../vm/uma_core.c:1105 #9 0xc0b43b57 in malloc (size=0x281000, mtp=, flags=) at ../../../kern/kern_malloc.c:511 #10 0xc0b1624e in fdgrowtable (fdp=0xde241a5c, nfd=) at ../../../kern/kern_descrip.c:1551 #11 0xc0b1571c in fdalloc (td=0xd43f4930, minfd=, result=0x0) at ../../../kern/kern_descrip.c:1507 #12 0xc0b15e1d in finstall (td=, fp=, fd=0xfb789b34, flags=0x0, fcaps=) at ../../../kern/kern_descrip.c:1770 #13 0xc0c234be in kern_openat (td=0xd43f4930, fd=, path=, pathseg=, flags=, mode=) at ../../../kern/vfs_syscalls.c:1158 #14 0xc0c22f79 in sys_open (td=0xd43f4930, uap=0xfb789ca8) at ../../../kern/vfs_syscalls.c:1022 #15 0xc105d48a in syscall (frame=) at subr_syscall.c:133 #16 0xc1048731 in Xint0x80_syscall () at ../../../i386/i386/exception.s:269 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 10 #10 0xc0b1624e in fdgrowtable (fdp=0xde241a5c, nfd=) at ../../../kern/kern_descrip.c:1551 1551 ntable = malloc(offsetof(struct fdescenttbl, fdt_ofiles) + (kgdb) l 1546 * entries, file entries themselves and the struct freetable we will use 1547 * when we decommission the table and place it on the freelist. 1548 * We place the struct freetable in the middle so we don't have 1549 * to worry about padding. 1550 */ 1551 ntable = malloc(offsetof(struct fdescenttbl, fdt_ofiles) + 1552 nnfiles * sizeof(ntable->fdt_ofiles[0]) + 1553 sizeof(struct freetable), 1554 M_FILEDESC, M_ZERO | M_WAITOK); 1555 /* copy the old data */ (kgdb) info loc nnfiles = 0x10000 onfiles = 0x8000 omap = (u_long *) 0xce595000 otable = (struct fdescenttbl *) 0xdc515000 (kgdb) (kgdb) f 10 #10 0xc0b1624e in fdgrowtable (fdp=0xde241a5c, nfd=) at ../../../kern/kern_descrip.c:1551 1551 ntable = malloc(offsetof(struct fdescenttbl, fdt_ofiles) + (kgdb) p *fdp $1 = {fd_files = 0xdc515000, fd_cdir = 0xca821438, fd_rdir = 0xc9300ca8, fd_jdir = 0x0, fd_map = 0xce595000, fd_lastfile = 0x7fff, fd_freefile = 0x8000, fd_cmask = 0x12, fd_refcnt = 0x1, fd_holdcnt = 0x1, fd_sx = {lock_object = { lo_name = 0xc11d662f "filedesc structure", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xc8587480}, sx_lock = 0xd43f4930}, fd_kqlist = {tqh_first = 0x0, tqh_last = 0x0}, fd_holdleaderscount = 0x0, fd_holdleaderswakeup = 0x0} (kgdb) p *fdp->fd_cdir $2 = {v_tag = 0xc11cce89 "ufs", v_op = 0xc147ad9c, v_data = 0xcd61c168, v_mount = 0xdb3165d8, v_nmntvnodes = {tqe_next = 0xccdd25a0, tqe_prev = 0xdb31660c}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xd0db2b5c}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xca821460}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xc11cce89 "ufs", lo_flags = 0x6fb0000, lo_data = 0x0, lo_witness = 0xc858b858}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xc, pcs = 0xca821490}}, v_interlock = {lock_object = {lo_name = 0xc11e77d0 "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc8583178}, mtx_lock = 0x4}, v_vnlock = 0xca82146c, v_actfreelist = {tqe_next = 0x0, tqe_prev = 0xccdd2658}, v_bufobj = {bo_lock = {lock_object = { lo_name = 0xc11f01d4 "bufobj interlock", lo_flags = 0x5230000, lo_data = 0x0, lo_witness = 0xc85899e0}, rw_lock = 0x1}, bo_ops = 0xc145c130, bo_object = 0xcb2ea4e0, bo_synclist = {le_next = 0x0, le_prev = 0xc857bb78}, bo_private = 0xca821438, __bo_vnode = 0xca821438, bo_clean = {bv_hd = {tqh_first = 0xe46ea5a0, tqh_last = 0xe46ea5d8}, bv_root = {pt_root = 0xe46ea671}, bv_cnt = 0x1}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xca821534}, bv_root = {pt_root = 0x0}, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_bsize = 0x8000}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xca82155c}, rl_currdep = 0x0}, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_holdcnt = 0x4, v_usecount = 0x2, v_iflag = 0x200, v_vflag = 0x1, v_writecount = 0x0, v_hash = 0x2, v_type = VDIR} (kgdb) p *fdp->fd_cdir->v_mount $3 = {mnt_mtx = {lock_object = {lo_name = 0xc11e782a "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc8583588}, mtx_lock = 0x4}, mnt_gen = 0x2, mnt_list = {tqe_next = 0x0, tqe_prev = 0xc9725bc8}, mnt_op = 0xc147aa04, mnt_vfc = 0xc147aa9c, mnt_vnodecovered = 0xc9e34870, mnt_syncer = 0xcdcfe000, mnt_ref = 0x3, mnt_nvnodelist = {tqh_first = 0xca821438, tqh_last = 0xcdcfe010}, mnt_nvnodelistsize = 0x3, mnt_activevnodelist = {tqh_first = 0xcdcfe000, tqh_last = 0xca8214f0}, mnt_activevnodelistsize = 0x3, mnt_writeopcount = 0x0, mnt_kern_flag = 0x60806144, mnt_flag = 0x201300, mnt_opt = 0xd5658c20, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x78, mnt_stat = {f_version = 0x20030518, f_type = 0x35, f_flags = 0x201300, f_bsize = 0x1000, f_iosize = 0x8000, f_blocks = 0x7bfa5, f_bfree = 0x7bfa3, f_bavail = 0x720f2, f_files = 0x3fffe, f_ffree = 0x3fffb, f_syncwrites = 0x2, f_asyncwrites = 0x6, f_syncreads = 0x6, f_asyncreads = 0x0, f_spare = 0xdb3166a8, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = 0xdb316700}, f_charspare = 0xdb316708 "", f_fstypename = 0xdb316758 "ufs", f_mntfromname = 0xdb316768 "/dev/md5a", f_mntonname = 0xdb3167c0 "/mnt"}, mnt_cred = 0xce601200, mnt_data = 0xd5c73300, mnt_time = 0x0, mnt_iosize_max = 0x20000, mnt_export = 0xdd1b8e80, mnt_label = 0x0, mnt_hashseed = 0xf890efdb, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x4, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xc11c8e4c "explock", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xc858a130}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x6, pcs = 0xdb31686c}}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0x0}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xdb3168bc}} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_malloc.c =================================================================== --- /usr/src/sys/kern/kern_malloc.c (revision 281995) +++ /usr/src/sys/kern/kern_malloc.c (working copy) @@ -672,6 +672,7 @@ static void kmem_reclaim(vmem_t *vm, int flags) { + uma_reclaim_needed = 1; pagedaemon_wakeup(); } Index: /usr/src/sys/kern/sysv_shm.c =================================================================== --- /usr/src/sys/kern/sysv_shm.c (revision 281995) +++ /usr/src/sys/kern/sysv_shm.c (working copy) @@ -961,6 +961,10 @@ oshmctl(struct thread *td, struct oshmctl_args *ua if (!prison_allow(td->td_ucred, PR_ALLOW_SYSVIPC)) return (ENOSYS); + if (uap->cmd != IPC_STAT) { + return (freebsd7_shmctl(td, + (struct freebsd7_shmctl_args *)uap)); + } SYSVSHM_LOCK(); shmseg = shm_find_segment(uap->shmid, true); if (shmseg == NULL) { @@ -967,33 +971,29 @@ oshmctl(struct thread *td, struct oshmctl_args *ua SYSVSHM_UNLOCK(); return (EINVAL); } - switch (uap->cmd) { - case IPC_STAT: - error = ipcperm(td, &shmseg->u.shm_perm, IPC_R); - if (error != 0) - break; + error = ipcperm(td, &shmseg->u.shm_perm, IPC_R); + if (error != 0) { + SYSVSHM_UNLOCK(); + return (error); + } #ifdef MAC - error = mac_sysvshm_check_shmctl(td->td_ucred, shmseg, - uap->cmd); - if (error != 0) - break; + error = mac_sysvshm_check_shmctl(td->td_ucred, shmseg, uap->cmd); + if (error != 0) { + SYSVSHM_UNLOCK(); + return (error); + } #endif - ipcperm_new2old(&shmseg->u.shm_perm, &outbuf.shm_perm); - outbuf.shm_segsz = shmseg->u.shm_segsz; - outbuf.shm_cpid = shmseg->u.shm_cpid; - outbuf.shm_lpid = shmseg->u.shm_lpid; - outbuf.shm_nattch = shmseg->u.shm_nattch; - outbuf.shm_atime = shmseg->u.shm_atime; - outbuf.shm_dtime = shmseg->u.shm_dtime; - outbuf.shm_ctime = shmseg->u.shm_ctime; - outbuf.shm_handle = shmseg->object; - error = copyout(&outbuf, uap->ubuf, sizeof(outbuf)); - break; - default: - error = freebsd7_shmctl(td, (struct freebsd7_shmctl_args *)uap); - break; - } + ipcperm_new2old(&shmseg->u.shm_perm, &outbuf.shm_perm); + outbuf.shm_segsz = shmseg->u.shm_segsz; + outbuf.shm_cpid = shmseg->u.shm_cpid; + outbuf.shm_lpid = shmseg->u.shm_lpid; + outbuf.shm_nattch = shmseg->u.shm_nattch; + outbuf.shm_atime = shmseg->u.shm_atime; + outbuf.shm_dtime = shmseg->u.shm_dtime; + outbuf.shm_ctime = shmseg->u.shm_ctime; + outbuf.shm_handle = shmseg->object; SYSVSHM_UNLOCK(); + error = copyout(&outbuf, uap->ubuf, sizeof(outbuf)); return (error); #else return (EINVAL); @@ -1025,9 +1025,7 @@ sys_shmsys(struct thread *td, struct shmsys_args * return (ENOSYS); if (uap->which < 0 || uap->which >= nitems(shmcalls)) return (EINVAL); - SYSVSHM_LOCK(); error = (*shmcalls[uap->which])(td, &uap->a2); - SYSVSHM_UNLOCK(); return (error); } Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 281995) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -113,8 +113,8 @@ static void vfs_setdirty_locked_object(struct buf static void vfs_vmio_release(struct buf *bp); static int vfs_bio_clcheck(struct vnode *vp, int size, daddr_t lblkno, daddr_t blkno); -static int buf_flush(int); -static int flushbufqueues(int, int); +static int buf_flush(struct vnode *vp, int); +static int flushbufqueues(struct vnode *, int, int); static void buf_daemon(void); static void bremfreel(struct buf *bp); static __inline void bd_wakeup(void); @@ -2097,7 +2097,7 @@ getnewbuf_bufd_help(struct vnode *vp, int gbflags, { struct thread *td; char *waitmsg; - int cnt, error, flags, norunbuf, wait; + int error, fl, flags, norunbuf; mtx_assert(&bqclean, MA_OWNED); @@ -2119,8 +2119,6 @@ getnewbuf_bufd_help(struct vnode *vp, int gbflags, return; td = curthread; - cnt = 0; - wait = MNT_NOWAIT; rw_wlock(&nblock); while ((needsbuffer & flags) != 0) { if (vp != NULL && vp->v_type != VCHR && @@ -2134,20 +2132,15 @@ getnewbuf_bufd_help(struct vnode *vp, int gbflags, * cannot be achieved by the buf_daemon, that * cannot lock the vnode. */ - if (cnt++ > 2) - wait = MNT_WAIT; - ASSERT_VOP_LOCKED(vp, "bufd_helper"); - error = VOP_ISLOCKED(vp) == LK_EXCLUSIVE ? 0 : - vn_lock(vp, LK_TRYUPGRADE); - if (error == 0) { - /* play bufdaemon */ - norunbuf = curthread_pflags_set(TDP_BUFNEED | - TDP_NORUNNINGBUF); - VOP_FSYNC(vp, wait, td); - atomic_add_long(¬bufdflushes, 1); - curthread_pflags_restore(norunbuf); - } + norunbuf = ~(TDP_BUFNEED | TDP_NORUNNINGBUF) | + (td->td_pflags & TDP_NORUNNINGBUF); + /* play bufdaemon */ + td->td_pflags |= TDP_BUFNEED | TDP_NORUNNINGBUF; + fl = buf_flush(vp, flushbufqtarget); + td->td_pflags &= norunbuf; rw_wlock(&nblock); + if (fl != 0) + continue; if ((needsbuffer & flags) == 0) break; } @@ -2566,11 +2559,11 @@ static struct kproc_desc buf_kp = { SYSINIT(bufdaemon, SI_SUB_KTHREAD_BUF, SI_ORDER_FIRST, kproc_start, &buf_kp); static int -buf_flush(int target) +buf_flush(struct vnode *vp, int target) { int flushed; - flushed = flushbufqueues(target, 0); + flushed = flushbufqueues(vp, target, 0); if (flushed == 0) { /* * Could not find any buffers without rollback @@ -2577,7 +2570,9 @@ static int * dependencies, so just write the first one * in the hopes of eventually making progress. */ - flushed = flushbufqueues(target, 1); + if (vp != NULL && target > 2) + target /= 2; + flushbufqueues(vp, target, 1); } return (flushed); } @@ -2614,7 +2609,7 @@ buf_daemon() * the I/O system. */ while (numdirtybuffers > lodirty) { - if (buf_flush(numdirtybuffers - lodirty) == 0) + if (buf_flush(NULL, numdirtybuffers - lodirty) == 0) break; kern_yield(PRI_USER); } @@ -2669,7 +2664,7 @@ SYSCTL_INT(_vfs, OID_AUTO, flushwithdeps, CTLFLAG_ 0, "Number of buffers flushed with dependecies that require rollbacks"); static int -flushbufqueues(int target, int flushdeps) +flushbufqueues(struct vnode *lvp, int target, int flushdeps) { struct buf *sentinel; struct vnode *vp; @@ -2679,6 +2674,7 @@ static int int flushed; int queue; int error; + bool unlock; flushed = 0; queue = QUEUE_DIRTY; @@ -2700,8 +2696,18 @@ static int mtx_unlock(&bqdirty); break; } - KASSERT(bp->b_qindex != QUEUE_SENTINEL, - ("parallel calls to flushbufqueues() bp %p", bp)); + /* + * Skip sentinels inserted by other invocations of the + * flushbufqueues(), taking care to not reorder them. + * + * Only flush the buffers that belong to the + * vnode locked by the curthread. + */ + if (bp->b_qindex == QUEUE_SENTINEL || (lvp != NULL && + bp->b_vp != lvp)) { + mtx_unlock(&bqdirty); + continue; + } error = BUF_LOCK(bp, LK_EXCLUSIVE | LK_NOWAIT, NULL); mtx_unlock(&bqdirty); if (error != 0) @@ -2749,16 +2755,37 @@ static int BUF_UNLOCK(bp); continue; } - error = vn_lock(vp, LK_EXCLUSIVE | LK_NOWAIT); + if (lvp == NULL) { + unlock = true; + error = vn_lock(vp, LK_EXCLUSIVE | LK_NOWAIT); + } else { + ASSERT_VOP_LOCKED(vp, "getbuf"); + unlock = false; + error = VOP_ISLOCKED(vp) == LK_EXCLUSIVE ? 0 : + vn_lock(vp, LK_TRYUPGRADE); + } if (error == 0) { CTR3(KTR_BUF, "flushbufqueue(%p) vp %p flags %X", bp, bp->b_vp, bp->b_flags); - vfs_bio_awrite(bp); + if (curproc == bufdaemonproc) { + vfs_bio_awrite(bp); + } else { + bremfree(bp); + bwrite(bp); + notbufdflushes++; + } vn_finished_write(mp); - VOP_UNLOCK(vp, 0); + if (unlock) + VOP_UNLOCK(vp, 0); flushwithdeps += hasdeps; flushed++; - if (runningbufspace > hirunningspace) + + /* + * Sleeping on runningbufspace while holding + * vnode lock leads to deadlock. + */ + if (curproc == bufdaemonproc && + runningbufspace > hirunningspace) waitrunningbufspace(); continue; } Index: /usr/src/sys/vm/vm_pageout.c =================================================================== --- /usr/src/sys/vm/vm_pageout.c (revision 281995) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -158,6 +158,7 @@ int vm_pages_needed; /* Event on which pageout da int vm_pageout_deficit; /* Estimated number of pages deficit */ int vm_pageout_pages_needed; /* flag saying that the pageout daemon needs pages */ int vm_pageout_wakeup_thresh; +int uma_reclaim_needed; #if !defined(NO_SWAPPING) static int vm_pageout_req_swapout; /* XXX */ @@ -687,6 +688,7 @@ vm_pageout_grow_cache(int tries, vm_paddr_t low, v * We do this explicitly after the caches have been drained * above. */ + uma_reclaim_needed = 0; uma_reclaim(); } @@ -1048,6 +1050,7 @@ vm_pageout_scan(struct vm_domain *vmd, int pass) * We do this explicitly after the caches have been * drained above. */ + uma_reclaim_needed = 0; uma_reclaim(); lowmem_ticks = ticks; } @@ -1634,6 +1637,12 @@ vm_pageout_worker(void *arg) if (vm_pages_needed && !vm_page_count_min()) { if (!vm_paging_needed()) vm_pages_needed = 0; + if (uma_reclaim_needed) { + mtx_unlock(&vm_page_queue_free_mtx); + uma_reclaim_needed = 0; + uma_reclaim(); + continue; + } wakeup(&vm_cnt.v_free_count); } if (vm_pages_needed) { Index: /usr/src/sys/vm/vm_pageout.h =================================================================== --- /usr/src/sys/vm/vm_pageout.h (revision 281995) +++ /usr/src/sys/vm/vm_pageout.h (working copy) @@ -76,6 +76,7 @@ extern int vm_pages_needed; /* should be some "eve extern int vm_pageout_pages_needed; extern int vm_pageout_deficit; extern int vm_pageout_page_count; +extern int uma_reclaim_needed; /* * Swap out requests -- Test scenario: misc/fifo2.sh