GENERIC HEAD from 2014-03-07 07:06:36 UTC, r262885M, vmcore.748 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb 4718592K of memory above 4GB ignored Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r262885M: Fri Mar 7 08:46:36 CET 2014 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 3.4 (tags/RELEASE_34/final 197956) 20140216 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3532070912 (3368 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point swi. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 441201 free (2161 frags, 54880 blocks, 0.2% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 599070 free (8910 frags, 73770 blocks, 0.9% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 3104207 free (244431 frags, 357472 blocks, 2.4% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 9134904 free (128872 frags, 1125754 blocks, 0.5% fragmentation) /dev/label/tmp: 135940 files, 20704107 used, 97609874 free (18138 frags, 12198967 blocks, 0.0% fragmentation) Mounting local file systems:Running bufdaemon at pid 8 Start thread /home worker Start thread /tmp worker Start thread /usr worker Start thread /var worker . Writing entropy file:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. Starting pflogd: add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc46 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/virtualbox a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Additional ABI support: linux. Starting casperd. lock order reversal: 1st 0xe44fa2bc bufwait (bufwait) @ kern/vfs_bio.c:3082 2nd 0xc94ce200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c115c810,4,c1166c70,409,4000,...) at db_trace_self_wrapper+0x2d/frame 0xedb1e768 kdb_backtrace(c116049d,c94ce200,c1193a65,c85ae408,c11936aa,...) at kdb_backtrace+0x30/frame 0xedb1e7cc witness_checkorder(c94ce200,9,c11936a1,11c,0,...) at witness_checkorder+0xd04/frame 0xedb1e818 _sx_xlock(c94ce200,0,c11936a1,11c,c8cfa000,...) at _sx_xlock+0x75/frame 0xedb1e848 ufsdirhash_add(c8e37740,edb1e940,33a0,edb1e8c8,edb1e8cc,...) at ufsdirhash_add+0x4a/frame 0xedb1e878 ufs_direnter(c8e395a0,c8d7eb40,edb1e940,edb1eb64,e44fee08,...) at ufs_direnter+0x604/frame 0xedb1e8f8 ufs_mkdir(edb1ec00,c1324dbb,c118c896,c8e395a0,edb1ebfc,...) at ufs_mkdir+0x8c2/frame 0xedb1ea98 VOP_MKDIR_APV(c13ea460,edb1ec00,edb1eb64,edb1eb90,28803020,...) at VOP_MKDIR_APV+0x12f/frame 0xedb1eac8 kern_mkdirat(c8e93930,ffffff9c,28803020,0,1c0) at kern_mkdirat+0x22d/frame 0xedb1ec24 sys_mkdir(c8e93930,edb1ecc8,edb1ec98,c0b09e86,c144f010,...) at sys_mkdir+0x31/frame 0xedb1ec40 syscall(edb1ed08) at syscall+0x31b/frame 0xedb1ecfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb1ecfc --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28179907, esp = 0xbfbfd924, ebp = 0xbfbfddd0 --- Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Removing stale Samba tdb files: ....... done Starting nmbd. Starting smbd. Starting winbindd. Starting smartd. /etc/rc: WARNING: failed to start smartd Starting default moused. Configuring syscons: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc8e395d4 ufs (ufs) @ kern/vfs_subr.c:2101 2nd 0xe44fa2bc bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 3rd 0xc973eb74 ufs (ufs) @ kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self_wrapper(c115c810,4000,0,200,0,...) at db_trace_self_wrapper+0x2d/frame 0xedb296b0 kdb_backtrace(c11604b6,c973eb74,c1147496,c85ae3a0,c1169c3e,...) at kdb_backtrace+0x30/frame 0xedb29714 witness_checkorder(c973eb74,9,c1169c35,835,c973ebe0,...) at witness_checkorder+0xd04/frame 0xedb29760 __lockmgr_args(c973eb74,80100,c973ebe0,0,0,...) at __lockmgr_args+0x8cd/frame 0xedb29840 ffs_lock(edb298c0,70c,c858d1e0,c8593228,c858d1e0,...) at ffs_lock+0x97/frame 0xedb2987c VOP_LOCK1_APV(c13ea460,edb298c0,e44dcc74,c8eac000,c13ff970,...) at VOP_LOCK1_APV+0x10a/frame 0xedb298a8 _vn_lock(c973eb40,80100,c1169c35,835,c1168e7d,...) at _vn_lock+0xca/frame 0xedb298e8 vget(c973eb40,80100,c8eac000,57,0,...) at vget+0x77/frame 0xedb29920 vfs_hash_get(c9219000,9b9c00,80000,c8eac000,edb29a20,...) at vfs_hash_get+0xff/frame 0xedb2994c ffs_vgetf(c9219000,9b9c00,80000,edb29a20,1,...) at ffs_vgetf+0x44/frame 0xedb299a8 softdep_sync_buf(c8e395a0,e44fa264,1,0,0,...) at softdep_sync_buf+0x35b/frame 0xedb29a38 ffs_syncvnode(c8e395a0,1,0,61b,0,...) at ffs_syncvnode+0x2dd/frame 0xedb29a90 ffs_sync(c9219000,1,edb29b48,518,c13c4154,...) at ffs_sync+0x23c/frame 0xedb29b10 dounmount(c9219000,8080000,c8eac000,494,0,...) at dounmount+0x4b8/frame 0xedb29b70 sys_unmount(c8eac000,edb29cc8,c0ff14df,e4737c90,0,...) at sys_unmount+0x3af/frame 0xedb29c40 syscall(edb29d08) at syscall+0x31b/frame 0xedb29cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb29cfc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280ca0db, esp = 0xbfbfd694, ebp = 0xbfbfd760 --- lock order reversal: 1st 0xc8d7d034 ufs (ufs) @ kern/vfs_mount.c:1237 2nd 0xc8d9f8a4 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1900 KDB: stack backtrace: db_trace_self_wrapper(c115c810,8,c1f7fc10,c1f7fc00,da,...) at db_trace_self_wrapper+0x2d/frame 0xedb29828 kdb_backtrace(c116049d,c8d9f8a4,c115332c,c85ae200,c118f2c0,...) at kdb_backtrace+0x30/frame 0xedb2988c witness_checkorder(c8d9f8a4,9,c118f2b7,76c,0,...) at witness_checkorder+0xd04/frame 0xedb298d8 __lockmgr_args(c8d9f8a4,80400,c8d9f910,0,0,0,c118f2b7,76c) at __lockmgr_args+0x8cd/frame 0xedb299b4 vop_stdlock(edb29a28,c9226200,edb29a20,c0ae05dd,c9226200,...) at vop_stdlock+0x53/frame 0xedb299e4 VOP_LOCK1_APV(c13b40c0,edb29a28,edb29a50,c0d48463,c13ff970,...) at VOP_LOCK1_APV+0x10a/frame 0xedb29a10 _vn_lock(c8d9f870,80400,c118f2b7,76c,c0b94b8e,...) at _vn_lock+0xca/frame 0xedb29a50 softdep_flushworklist(c9219000,edb29adc,c8eac000,62f,0,...) at softdep_flushworklist+0x68/frame 0xedb29a90 ffs_sync(c9219000,1,edb29b48,518,c13c4154,...) at ffs_sync+0x303/frame 0xedb29b10 dounmount(c9219000,8080000,c8eac000,494,0,...) at dounmount+0x4b8/frame 0xedb29b70 sys_unmount(c8eac000,edb29cc8,c0ff14df,e4737c90,0,...) at sys_unmount+0x3af/frame 0xedb29c40 syscall(edb29d08) at syscall+0x31b/frame 0xedb29cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb29cfc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280ca0db, esp = 0xbfbfd694, ebp = 0xbfbfd760 --- Stop thread /tmp worker: searchfailed 0, did cleanups 0 lock order reversal: 1st 0xc8e3919c syncer (syncer) @ kern/vfs_subr.c:1720 2nd 0xc9922b74 ufs (ufs) @ kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self_wrapper(c115c810,0,0,0,0,...) at db_trace_self_wrapper+0x2d/frame 0xedb0d938 kdb_backtrace(c116049d,c9922b74,c1147496,c85ae3a0,c1169c3e,...) at kdb_backtrace+0x30/frame 0xedb0d99c witness_checkorder(c9922b74,9,c1169c35,835,c9922be0,...) at witness_checkorder+0xd04/frame 0xedb0d9e8 __lockmgr_args(c9922b74,80500,c9922be0,0,0,...) at __lockmgr_args+0x8cd/frame 0xedb0dac8 ffs_lock(edb0db48,c115a873,c8593088,c85ae470,c8593088,...) at ffs_lock+0x97/frame 0xedb0db04 VOP_LOCK1_APV(c13ea460,edb0db48,c16fbf14,127e,c13ff970,...) at VOP_LOCK1_APV+0x10a/frame 0xedb0db30 _vn_lock(c9922b40,80500,c1169c35,835,c1169c35,...) at _vn_lock+0xca/frame 0xedb0db70 Expensive timeout(9) function: 0xc09004a0(0xc17147e8) 0.012306449 s vget(c9922b40,80500,c8b93000,128a,edb0dbf8,...) at vget+0x77/frame 0xedb0dba8 vfs_msync(c92fa8c4,2,2,e47,c92fa8c4,...) at vfs_msync+0xc0/frame 0xedb0dbf8 sync_fsync(edb0dca8,c1324c96,80400,c1169c35,6b8,...) at sync_fsync+0x131/frame 0xedb0dc20 VOP_FSYNC_APV(c13ce0f4,edb0dca8,c1169c35,6b8,0,...) at VOP_FSYNC_APV+0x12f/frame 0xedb0dc50 sched_sync(0,edb0dd08,c11519ec,3c9,0,...) at sched_sync+0x3b1/frame 0xedb0dccc fork_exit(c0b997e0,0,edb0dd08) at fork_exit+0x7f/frame 0xedb0dcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb0dcf4 --- trap 0, eip = 0, esp = 0xedb0dd40, ebp = 0 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 135931 files, 20704106 used, 97609875 free (18131 frags, 12198968 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Start thread /tmp worker fsck -y /tmp. Starting inetd. Fri Mar 7 09:05:52 CET 2014 FreeBSD/i386 (x4.osted.lan) (console) login: Mar 7 09:08:56 x4 su: pho to root on /dev/pts/0 20140307 09:09:04 all (1/2): pfl.sh Start thread /mnt worker Start thread /mnt2 worker Stop thread /mnt2 worker: searchfailed 0, did cleanups 0 Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 11:34:24 all (2/2): pfl2.sh Start thread /mnt worker Start thread /mnt2 worker Stop thread /mnt2 worker: searchfailed 0, did cleanups 0 Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 11:45:18 all (1/2): pfl.sh Start thread /mnt worker Start thread /mnt2 worker Mar 7 12:37:20 x4 su: pho to root on /dev/pts/1 Stop thread /mnt2 worker: searchfailed 0, did cleanups 0 Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 12:49:34 all (2/2): pfl2.sh Start thread /mnt worker Start thread /mnt2 worker Stop thread /mnt2 worker: searchfailed 0, did cleanups 0 Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 13:02:01 all (1/2): pfl.sh Start thread /mnt worker Start thread /mnt2 worker Mar 7 13:33:19 x4 su: pho to root on /dev/pts/2 kern.maxfiles limit exceeded by uid 1004, please see tuning(7). kern.maxfiles limit exceeded by uid 1004, please see tuning(7). fsync: giving up on dirty 0xcee03ca8: tag devfs, type VCHR usecount 1, writecount 0, refcount 1184 mountedhere 0xccbe4900 flags (VI_ACTIVE) v_object 0xd28aa138 ref 0 pages 14230 cleanbuf 2 dirtybuf 1180 lock type devfs: EXCL by thread 0xd35ec930 (pid 25207, umount, tid 100344) #0 0xc0ac41af at __lockmgr_args+0xb6f #1 0xc0b81fb3 at vop_stdlock+0x53 #2 0xc101374a at VOP_LOCK1_APV+0x10a #3 0xc0ba56ba at _vn_lock+0xca #4 0xc0d48298 at softdep_flushworklist+0x68 #5 0xc0d675a3 at ffs_sync+0x303 #6 0xc0b8bab8 at dounmount+0x4b8 #7 0xc0b8b5bf at sys_unmount+0x3af #8 0xc0fea76b at syscall+0x31b #9 0xc0fd2891 at Xint0x80_syscall+0x21 dev md5a Stop thread /mnt worker: searchfailed 0, did cleanups 0 Stop thread /mnt2 worker: searchfailed 0, did cleanups 0 FreeBSD/i386 (x4.osted.lan) (console) login: 20140307 14:00:13 all (1/269): procfs4.sh 20140307 14:11:43 all (2/269): rename2.sh 20140307 14:31:43 all (3/269): kinfo3.sh Expensive timeout(9) function: 0xc0953b60(0xc8940aa0) 0.014072553 s 20140307 14:47:13 all (4/269): datamove4.sh 20140307 14:47:19 all (5/269): fs.sh lock order reversal: 1st 0xd30c7e1c ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xc9c058a4 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1711 KDB: stack backtrace: db_trace_self_wrapper(c115c810,c0b0b900,c144fe10,4,c115a873,...) at db_trace_self_wrapper+0x2d/frame 0xf8b32650 kdb_backtrace(c116049d,c9c058a4,c1147496,c85ae3a0,c11929ee,...) at kdb_backtrace+0x30/frame 0xf8b326b4 witness_checkorder(c9c058a4,9,c11929e5,6af,0,...) at witness_checkorder+0xd04/frame 0xf8b32700 __lockmgr_args(c9c058a4,80000,0,0,0,...) at __lockmgr_args+0x8cd/frame 0xf8b327dc ffs_vgetf(caeefbb0,2,80000,f8b32870,0) at ffs_vgetf+0x16d/frame 0xf8b32838 ffs_vget(caeefbb0,2,80000,f8b32870,0,...) at ffs_vget+0x2f/frame 0xf8b32858 ufs_root(caeefbb0,80000,f8b328a8,72,8,...) at ufs_root+0x49/frame 0xf8b3287c ufs_extattr_autostart(caeefbb0,cb08d000,c11929e5,439,0,...) at ufs_extattr_autostart+0x7b/frame 0xf8b328b8 ffs_mount(caeefbb0,c13ea168,cfb0ba80,c8ccb600,ca7e71e0,...) at ffs_mount+0x265b/frame 0xf8b32a18 vfs_donmount(cb08d000,0,0,d0662880,d0662880,...) at vfs_donmount+0xd60/frame 0xf8b32c18 sys_nmount(cb08d000,f8b32cc8,c131d9ea,d5,0,...) at sys_nmount+0x78/frame 0xf8b32c40 syscall(f8b32d08) at syscall+0x31b/frame 0xf8b32cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8b32cfc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280dd6db, esp = 0xbfbfc8d0, ebp = 0xbfbfce28 --- Mar 7 14:53:53 x4 kernel: pid 68284 (creat), uid 1004 inumber 775 on /mnt: out of inodes Mar 7 14:53:54 x4 kernel: pid 68273 (creat), uid 1004 inumber 771 on /mnt: out of inodes Mar 7 14:54:06 x4 kernel: pid 68282 (creat), uid 1004 inumber 780 on /mnt: out of inodes Mar 7 14:54:09 x4 kernel: pid 68276 (creat), uid 1004 inumber 786 on /mnt: out of inodes Mar 7 14:54:19 x4 kernel: pid 68281 (creat), uid 1004 inumber 1538 on /mnt: out of inodes Mar 7 14:54:20 x4 kernel: pid 68273 (creat), uid 1004 inumber 771 on /mnt: out of inodes Mar 7 14:54:21 x4 kernel: pid 68287 (creat), uid 1004 inumber 773 on /mnt: out of inodes Mar 7 14:54:23 x4 kernel: pid 68282 (creat), uid 1004 inumber 780 on /mnt: out of inodes Mar 7 14:54:44 x4 kernel: pid 68276 (creat), uid 1004 inumber 786 on /mnt: out of inodes Mar 7 14:54:52 x4 kernel: pid 68287 (creat), uid 1004 inumber 773 on /mnt: out of inodes Mar 7 14:54:54 x4 kernel: pid 68287 (creat), uid 1004 inumber 773 on /mnt: out of inodes Mar 7 14:55:11 x4 kernel: pid 68278 (creat), uid 1004 inumber 774 on /mnt: out of inodes Mar 7 14:59:22 x4 kernel: pid 68566 (rw), uid 1004 inumber 641 on /mnt: filesystem full kern.maxfiles limit exceeded by uid 1004, please see tuning(7). kern.maxfiles limit exceeded by uid 1004, please see tuning(7). Mar 7 15:01:34 x4 kernel: pid 68678 (rw), uid 1004 inumber 5 on /mnt: filesystem full Mar 7 15:01:34 x4 kernel: pid 68680 (rw), uid 1004 inumber 648 on /mnt: filesystem full Mar 7 15:01:34 x4 kernel: pid 68675 (mkdir), uid 1004 inumber 671 on /mnt: filesystem full Mar 7 15:01:34 x4 kernel: pid 68662 (rw), uid 1004 inumber 1939 on /mnt: filesystem full Start thread /mnt worker Mar 7 15:03:28 x4 kernel: pid 68904 (rw), uid 1004 inumber 649 on /mnt: out of inodes Mar 7 15:03:31 x4 kernel: pid 68923 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Mar 7 15:03:32 x4 kernel: pid 68923 (creat), uid 1004 inumber 1287 on /mnt: out of inodes Mar 7 15:03:34 x4 kernel: pid 68927 (creat), uid 1004 inumber 1920 on /mnt: out of inodes Mar 7 15:03:36 x4 kernel: pid 68921 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Mar 7 15:03:39 x4 kernel: pid 68916 (creat), uid 1004 inumber 666 on /mnt: out of inodes Mar 7 15:03:45 x4 kernel: pid 68930 (creat), uid 1004 inumber 1286 on /mnt: out of inodes Mar 7 15:03:47 x4 kernel: pid 68914 (creat), uid 1004 inumber 667 on /mnt: out of inodes Mar 7 15:03:49 x4 kernel: pid 68911 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:03:52 x4 kernel: pid 68916 (creat), uid 1004 inumber 666 on /mnt: out of inodes Mar 7 15:03:53 x4 kernel: pid 68911 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:03:55 x4 kernel: pid 68924 (creat), uid 1004 inumber 656 on /mnt: out of inodes Mar 7 15:03:59 x4 kernel: pid 68902 (creat), uid 1004 inumber 657 on /mnt: out of inodes Mar 7 15:04:02 x4 kernel: pid 68924 (creat), uid 1004 inumber 656 on /mnt: out of inodes Mar 7 15:04:06 x4 kernel: pid 68924 (creat), uid 1004 inumber 656 on /mnt: out of inodes Mar 7 15:04:07 x4 kernel: pid 68926 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Mar 7 15:04:11 x4 kernel: pid 68911 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:04:13 x4 kernel: pid 68926 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Mar 7 15:04:18 x4 kernel: pid 68922 (creat), uid 1004 inumber 665 on /mnt: out of inodes Mar 7 15:04:20 x4 kernel: pid 68905 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Mar 7 15:04:22 x4 kernel: pid 68930 (creat), uid 1004 inumber 1286 on /mnt: out of inodes Mar 7 15:04:25 x4 kernel: pid 68926 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Mar 7 15:04:27 x4 kernel: pid 68929 (creat), uid 1004 inumber 1283 on /mnt: out of inodes Mar 7 15:04:32 x4 kernel: pid 68905 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Mar 7 15:04:33 x4 kernel: pid 68922 (creat), uid 1004 inumber 665 on /mnt: out of inodes Mar 7 15:04:36 x4 kernel: pid 68921 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Mar 7 15:04:38 x4 kernel: pid 68926 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Mar 7 15:04:39 x4 kernel: pid 68916 (creat), uid 1004 inumber 666 on /mnt: out of inodes Mar 7 15:04:41 x4 kernel: pid 68921 (creat), uid 1004 inumber 1285 on /mnt: out of inodes Mar 7 15:04:43 x4 kernel: pid 68908 (creat), uid 1004 inumber 668 on /mnt: out of inodes Mar 7 15:04:44 x4 kernel: pid 68908 (creat), uid 1004 inumber 668 on /mnt: out of inodes Mar 7 15:04:46 x4 kernel: pid 68929 (creat), uid 1004 inumber 1283 on /mnt: out of inodes Mar 7 15:04:52 x4 kernel: pid 68916 (creat), uid 1004 inumber 666 on /mnt: out of inodes Mar 7 15:04:55 x4 kernel: pid 68927 (creat), uid 1004 inumber 1920 on /mnt: out of inodes Mar 7 15:04:58 x4 kernel: pid 68900 (creat), uid 1004 inumber 653 on /mnt: out of inodes Mar 7 15:05:01 x4 kernel: pid 68927 (creat), uid 1004 inumber 1920 on /mnt: out of inodes Mar 7 15:05:05 x4 kernel: pid 68925 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Mar 7 15:05:09 x4 kernel: pid 68905 (creat), uid 1004 inumber 1284 on /mnt: out of inodes Mar 7 15:05:11 x4 kernel: pid 68902 (creat), uid 1004 inumber 657 on /mnt: out of inodes Mar 7 15:05:15 x4 kernel: pid 68912 (rw), uid 1004 inumber 654 on /mnt: out of inodes Mar 7 15:05:16 x4 kernel: pid 68926 (creat), uid 1004 inumber 1282 on /mnt: out of inodes Mar 7 15:05:17 x4 kernel: pid 68922 (creat), uid 1004 inumber 665 on /mnt: out of inodes Mar 7 15:05:19 x4 kernel: pid 68928 (creat), uid 1004 inumber 1922 on /mnt: out of inodes Mar 7 15:05:20 x4 kernel: pid 68927 (creat), uid 1004 inumber 1920 on /mnt: out of inodes Mar 7 15:05:22 x4 kernel: pid 68925 (creat), uid 1004 inumber 1921 on /mnt: out of inodes Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Mar 7 15:07:35 x4 kernel: pid 69183 (creat), uid 1004 inumber 643 on /mnt: out of inodes Mar 7 15:07:40 x4 kernel: pid 69208 (rw), uid 1004 inumber 1453 on /mnt: filesystem full Mar 7 15:07:42 x4 kernel: pid 69191 (rw), uid 1004 inumber 666 on /mnt: filesystem full Mar 7 15:08:22 x4 kernel: pid 69187 (rw), uid 1004 inumber 1331 on /mnt: filesystem full Mar 7 15:09:33 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:09:34 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:09:36 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:09:37 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:09:38 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:09:39 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:09:40 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:09:42 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:09:43 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:09:45 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:09:46 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:09:48 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:09:50 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:09:52 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:09:53 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:09:54 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:09:55 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:09:56 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:09:58 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:09:59 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:10:00 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:01 x4 kernel: pid 69280 (rw), uid 1004 inumber 1136 on /mnt: out of inodes Mar 7 15:10:02 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:04 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:10:05 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:10:06 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:07 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:08 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:10:09 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:10 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:10:11 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:10:12 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:14 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:10:15 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:16 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:17 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:10:18 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:10:20 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:10:21 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:22 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:23 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:24 x4 kernel: pid 69272 (rw), uid 1004 inumber 2468 on /mnt: out of inodes Mar 7 15:10:25 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:10:26 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:27 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:28 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:10:29 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:30 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:10:32 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:10:33 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:34 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:35 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:36 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:37 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:39 x4 last message repeated 2 times Mar 7 15:10:40 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:41 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:42 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:43 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:10:44 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:45 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:46 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:47 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:48 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:10:49 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:10:50 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:51 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:10:53 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:54 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:55 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:10:56 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:10:57 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:10:58 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:10:59 x4 kernel: pid 69262 (creat), uid 1004 inumber 1132 on /mnt: out of inodes Mar 7 15:11:00 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:11:01 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:11:02 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:11:03 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:11:04 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:11:06 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:11:07 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:11:08 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:11:09 x4 kernel: pid 69265 (creat), uid 1004 inumber 651 on /mnt: out of inodes Mar 7 15:11:10 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:11:11 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:11:12 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:11:13 x4 kernel: pid 69268 (creat), uid 1004 inumber 1140 on /mnt: out of inodes Mar 7 15:11:14 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:11:15 x4 kernel: pid 69257 (creat), uid 1004 inumber 652 on /mnt: out of inodes Mar 7 15:11:16 x4 kernel: pid 69271 (creat), uid 1004 inumber 1135 on /mnt: out of inodes Mar 7 15:11:17 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:11:19 x4 kernel: pid 69259 (creat), uid 1004 inumber 1705 on /mnt: out of inodes Mar 7 15:11:20 x4 kernel: pid 69254 (creat), uid 1004 inumber 1129 on /mnt: out of inodes Mar 7 15:11:21 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:11:22 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Mar 7 15:11:23 x4 kernel: pid 69275 (creat), uid 1004 inumber 2467 on /mnt: out of inodes Mar 7 15:11:24 x4 kernel: pid 69273 (creat), uid 1004 inumber 1721 on /mnt: out of inodes Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker kern.maxfiles limit exceeded by uid 1004, please see tuning(7). kern.maxfiles limit exceeded by uid 1004, please see tuning(7). kern.maxfiles limit exceeded by uid 1004, please see tuning(7). Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Mar 7 15:17:34 x4 kernel: pid 69791 (mkdir), uid 1004 inumber 1289 on /mnt: out of inodes Mar 7 15:17:35 x4 kernel: pid 69775 (rw), uid 1004 inumber 1292 on /mnt: out of inodes Mar 7 15:17:40 x4 kernel: pid 69779 (rw), uid 1004 inumber 1307 on /mnt: out of inodes Mar 7 15:17:44 x4 kernel: pid 69795 (mkdir), uid 1004 inumber 1916 on /mnt: out of inodes Mar 7 15:18:01 x4 kernel: pid 69784 (mkdir), uid 1004 inumber 2108 on /mnt: out of inodes Mar 7 15:18:05 x4 kernel: pid 69785 (rw), uid 1004 inumber 1299 on /mnt: out of inodes Mar 7 15:18:11 x4 kernel: pid 69792 (mkdir), uid 1004 inumber 2518 on /mnt: out of inodes Mar 7 15:18:22 x4 kernel: pid 69780 (mkdir), uid 1004 inumber 2557 on /mnt: out of inodes Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 15:19:26 all (6/269): rename4.sh 20140307 15:19:27 all (7/269): rename9.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 15:20:12 all (8/269): linger4.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 15:57:16 all (9/269): suj6.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 15:58:16 all (10/269): swap2.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:03:47 all (11/269): fpclone.sh 20140307 16:05:53 all (12/269): fpu.sh 20140307 16:08:33 all (13/269): mount.sh Stop thread /tmp worker: searchfailed 0, did cleanups 0 Start thread /tmp worker 20140307 16:08:34 all (14/269): kevent6.sh Start thread /mnt worker lock order reversal: 1st 0xc8cf8428 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1203 2nd 0xcea8f5d4 ufs (ufs) @ kern/vfs_subr.c:4350 KDB: stack backtrace: db_trace_self_wrapper(c115c810,32313a63,a3330,f8d43934,c0acd1d0,...) at db_trace_self_wrapper+0x2d/frame 0xf8d438f0 kdb_backtrace(c116049d,cea8f5d4,c1147496,c85ae3a0,c1169c3e,...) at kdb_backtrace+0x30/frame 0xf8d43954 witness_checkorder(cea8f5d4,9,c1169c35,10fe,0,...) at witness_checkorder+0xd04/frame 0xf8d439a0 __lockmgr_args(cea8f5d4,80400,cea8f640,0,0,...) at __lockmgr_args+0x8cd/frame 0xf8d43a80 ffs_lock(f8d43b00,f8d43ad4,c0b81db8,cea8f5d4,f8d43b00,...) at ffs_lock+0x97/frame 0xf8d43abc VOP_LOCK1_APV(c13ea460,f8d43b00,c9d83db4,c8df3180,c13ff970,...) at VOP_LOCK1_APV+0x10a/frame 0xf8d43ae8 _vn_lock(cea8f5a0,80400,c1169c35,10fe) at _vn_lock+0xca/frame 0xf8d43b28 vfs_knllock(cea8f5a0,0,c1151249,781,cea8f5a0,...) at vfs_knllock+0x29/frame 0xf8d43b40 knlist_remove_kq(0,0) at knlist_remove_kq+0x8e/frame 0xf8d43b64 knlist_remove(c9d83db4,c9f953a8,0,3,3,...) at knlist_remove+0x1f/frame 0xf8d43b74 filt_vfsdetach(c9f953a8,0,c1151249,88b,0,...) at filt_vfsdetach+0x2f/frame 0xf8d43b94 knote_fdclose(cb11cc40,3,c11509b1,46a,c8cf8428,...) at knote_fdclose+0xde/frame 0xf8d43bd0 closefp(c8e127e0,cb11cc40,1,b1,c8cf84c0,...) at closefp+0x6c/frame 0xf8d43c04 kern_close(cb11cc40,3) at kern_close+0xf5/frame 0xf8d43c30 sys_close(cb11cc40,f8d43cc8,f8d43c9c,c0ff1b87,0,...) at sys_close+0x1a/frame 0xf8d43c40 syscall(f8d43d08) at syscall+0x31b/frame 0xf8d43cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8d43cfc --- syscall (6, FreeBSD ELF32, sys_close), eip = 0x281ae1b7, esp = 0xbf5f9f30, ebp = 0xbf5f9f48 --- Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:09:07 all (15/269): sendfile_shm.sh 20140307 16:09:39 all (16/269): tcp.sh witness_lock_list_get: witness exhausted Expensive timeout(9) function: 0xc0adc9a0(0xc8dcb000) 0.015175174 s Expensive timeout(9) function: 0xc085bff0(0xc8905000) 0.015418773 s Expensive timeout(9) function: 0xc0ca9550(0xcbbf02f0) 0.027474380 s Expensive timeout(9) function: 0xc0c28080(0) 0.331560220 s [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached 20140307 16:25:12 all (17/269): nfs3.sh 20140307 16:26:15 all (18/269): trim2.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:26:26 all (19/269): sendfile7.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:32:31 all (20/269): mlockall4.sh 20140307 16:32:36 all (21/269): ptrace2.sh 20140307 16:32:43 all (22/269): thr.sh vm_thread_new: kstack allocation failed 20140307 16:32:53 all (23/269): ucom.sh 20140307 16:35:58 all (24/269): devfd.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:36:07 all (25/269): mlockall3.sh 20140307 16:39:07 all (26/269): union.sh Mar 7 16:39:07 x4 mountd[686]: can't delete exports for /home/pho/stress2/misc/mp1: Invalid argument 20140307 16:49:35 all (27/269): rdwr.sh 20140307 16:51:03 all (28/269): setuid.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 16:51:04 all (29/269): trim4.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 17:30:16 all (30/269): isofs.sh 20140307 17:30:25 all (31/269): recursiveflushes.sh Start thread /mnt worker softdep_waitidle: Failed to flush worklist for 0xcaeef000 Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 17:30:31 all (32/269): wire_no_page.sh 20140307 17:30:39 all (33/269): suj3.sh cryptosoft0: on motherboard GEOM_ELI: Device md5.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 GEOM_ELI: md5 has been killed. GEOM_ELI: Device md5.eli destroyed. 20140307 17:36:59 all (34/269): posix_fadvise.sh 20140307 17:36:59 all (35/269): namecache2.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 17:39:26 all (36/269): ffs_syncvnode2.sh Start thread /mnt worker Mar 7 17:39:31 x4 kernel: pid 17132 (rw), uid 1004 inumber 65543 on /mnt: filesystem full Mar 7 17:41:35 x4 kernel: pid 17192 (rw), uid 1004 inumber 65601 on /mnt: filesystem full Mar 7 17:41:35 x4 kernel: pid 17200 (mkdir), uid 1004 inumber 8 on /mnt: filesystem full Mar 7 17:41:35 x4 kernel: pid 17178 (creat), uid 1004 inumber 65541 on /mnt: filesystem full Mar 7 17:45:57 x4 kernel: pid 17624 (mkdir), uid 1004 inumber 91465 on /mnt: filesystem full Mar 7 17:45:57 x4 kernel: pid 17609 (rw), uid 1004 inumber 65713 on /mnt: filesystem full Mar 7 17:45:57 x4 kernel: pid 17586 (creat), uid 1004 inumber 65664 on /mnt: filesystem full Mar 7 17:46:47 x4 kernel: pid 17623 (mkdir), uid 1004 inumber 138335 on /mnt: filesystem full Mar 7 17:47:05 x4 kernel: pid 17615 (rw), uid 1004 inumber 120850 on /mnt: filesystem full Mar 7 17:47:19 x4 kernel: pid 17606 (rw), uid 1004 inumber 104895 on /mnt: filesystem full Mar 7 17:47:36 x4 kernel: pid 17868 (rw), uid 1004 inumber 131104 on /mnt: filesystem full Mar 7 17:47:56 x4 kernel: pid 17847 (rw), uid 1004 inumber 131113 on /mnt: filesystem full Mar 7 17:47:56 x4 kernel: pid 17881 (creat), uid 1004 inumber 131080 on /mnt: filesystem full Mar 7 17:47:57 x4 kernel: pid 17878 (mkdir), uid 1004 inumber 66054 on /mnt: filesystem full Mar 7 17:49:40 x4 kernel: pid 18348 (rw), uid 1004 inumber 131084 on /mnt: filesystem full Mar 7 17:49:51 x4 kernel: pid 18357 (mkdir), uid 1004 inumber 210550 on /mnt: filesystem full Mar 7 17:49:51 x4 kernel: pid 18359 (rw), uid 1004 inumber 131100 on /mnt: filesystem full Mar 7 17:50:03 x4 kernel: pid 18972 (rw), uid 1004 inumber 131089 on /mnt: filesystem full Mar 7 17:50:04 x4 kernel: pid 18975 (mkdir), uid 1004 inumber 196847 on /mnt: filesystem full Mar 7 17:50:05 x4 kernel: pid 18970 (rw), uid 1004 inumber 131111 on /mnt: filesystem full Mar 7 17:50:05 x4 kernel: pid 18973 (mkdir), uid 1004 inumber 131289 on /mnt: filesystem full Mar 7 17:52:05 x4 kernel: pid 20264 (rw), uid 1004 inumber 131104 on /mnt: filesystem full Mar 7 17:52:05 x4 kernel: pid 20256 (creat), uid 1004 inumber 131146 on /mnt: filesystem full Mar 7 17:52:05 x4 kernel: pid 20251 (mkdir), uid 1004 inumber 65576 on /mnt: filesystem full Mar 7 17:52:13 x4 kernel: pid 20242 (creat), uid 1004 inumber 131153 on /mnt: filesystem full Mar 7 17:52:13 x4 kernel: pid 20235 (rw), uid 1004 inumber 131075 on /mnt: filesystem full Mar 7 17:52:15 x4 kernel: pid 20263 (rw), uid 1004 inumber 131090 on /mnt: filesystem full Mar 7 17:52:15 x4 kernel: pid 20252 (mkdir), uid 1004 inumber 197174 on /mnt: filesystem full Mar 7 17:52:15 x4 kernel: pid 20258 (creat), uid 1004 inumber 131152 on /mnt: filesystem full Mar 7 17:55:11 x4 kernel: pid 21795 (rw), uid 1004 inumber 131124 on /mnt: filesystem full Mar 7 17:55:11 x4 kernel: pid 21791 (creat), uid 1004 inumber 131109 on /mnt: filesystem full Mar 7 17:55:11 x4 kernel: pid 21829 (mkdir), uid 1004 inumber 133193 on /mnt: filesystem full Mar 7 17:55:13 x4 kernel: pid 21815 (creat), uid 1004 inumber 131157 on /mnt: filesystem full Mar 7 17:55:13 x4 kernel: pid 21797 (rw), uid 1004 inumber 131114 on /mnt: filesystem full Mar 7 17:55:13 x4 kernel: pid 21814 (mkdir), uid 1004 inumber 131252 on /mnt: filesystem full Mar 7 17:58:13 x4 kernel: pid 24909 (rw), uid 1004 inumber 131156 on /mnt: filesystem full Mar 7 17:58:13 x4 kernel: pid 24924 (mkdir), uid 1004 inumber 135257 on /mnt: filesystem full Mar 7 17:59:41 x4 kernel: pid 24906 (rw), uid 1004 inumber 131136 on /mnt: filesystem full Mar 7 17:59:45 x4 kernel: pid 24908 (rw), uid 1004 inumber 131148 on /mnt: filesystem full Mar 7 17:59:45 x4 kernel: pid 24904 (mkdir), uid 1004 inumber 131456 on /mnt: filesystem full Mar 7 17:59:45 x4 kernel: pid 24902 (creat), uid 1004 inumber 65575 on /mnt: filesystem full Mar 7 18:02:36 x4 kernel: pid 28512 (mkdir), uid 1004 inumber 2145 on /mnt: filesystem full Mar 7 18:02:36 x4 kernel: pid 28526 (rw), uid 1004 inumber 66070 on /mnt: filesystem full Mar 7 18:03:14 x4 kernel: pid 31673 (rw), uid 1004 inumber 131402 on /mnt: filesystem full Mar 7 18:03:14 x4 kernel: pid 31695 (creat), uid 1004 inumber 131289 on /mnt: filesystem full Mar 7 18:04:54 x4 kernel: pid 34846 (rw), uid 1004 inumber 131410 on /mnt: filesystem full Mar 7 18:05:06 x4 kernel: pid 34864 (rw), uid 1004 inumber 131422 on /mnt: filesystem full Mar 7 18:05:07 x4 kernel: pid 34850 (mkdir), uid 1004 inumber 134184 on /mnt: filesystem full Mar 7 18:05:07 x4 kernel: pid 34853 (creat), uid 1004 inumber 131433 on /mnt: filesystem full Mar 7 18:05:07 x4 kernel: pid 34866 (rw), uid 1004 inumber 131438 on /mnt: filesystem full Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 18:11:25 all (37/269): msdos4.sh 20140307 18:32:43 all (38/269): mountro2.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 18:32:46 all (39/269): tcp2.sh [zone: mbuf] kern.ipc.nmbufs limit reached [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached 20140307 18:49:27 all (40/269): quota1.sh Start thread /mnt worker Mar 7 18:50:12 x4 kernel: pid 55573 (creat), uid 0 inumber 12298 on /mnt: out of inodes Mar 7 18:53:59 x4 kernel: pid 55692 (mkdir), uid 0 inumber 8136 on /mnt: out of inodes Mar 7 18:59:23 x4 kernel: pid 55796 (rw), uid 0 inumber 10 on /mnt: out of inodes Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 18:59:39 all (41/269): mmap5.sh 20140307 19:00:46 all (42/269): pts.sh 20140307 19:04:48 all (43/269): zfs.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (1 allocations, 16 bytes leaked). 20140307 19:25:55 all (44/269): nullfs13.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 19:25:55 all (45/269): sem.sh 20140307 19:26:14 all (46/269): lockf3.sh 20140307 19:31:23 all (47/269): mlockall2.sh 20140307 19:41:45 all (48/269): fifo2.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 19:43:56 all (49/269): swap.sh 20140307 19:49:25 all (50/269): zfs4.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (2 allocations, 32 bytes leaked). 20140307 20:03:03 all (51/269): f_offset.sh 20140307 20:06:07 all (52/269): umount.sh 20140307 20:10:27 all (53/269): rename5.sh Start thread /mnt worker Stop thread /mnt worker: searchfailed 0, did cleanups 0 20140307 20:10:55 all (54/269): nullfs5.sh 20140307 20:23:22 all (55/269): tmpfs9.sh 20140307 20:31:47 all (56/269): msync2.sh 20140307 20:33:39 all (57/269): md4.sh 20140307 20:33:40 all (58/269): jail4.sh 20140307 20:33:40 all (59/269): lockf.sh 20140307 20:33:51 all (60/269): dup2.sh Stopping inetd. Waiting for PIDS: 1029. Shutting down local packages:. Stopping cron. Waiting for PIDS: 863. Stopping sshd. Waiting for PIDS: 852. Stopping moused. Waiting for PIDS: 802. Stopping winbindd. Waiting for PIDS: 781. Stopping smbd. Waiting for PIDS: 779. Stopping nmbd. Waiting for PIDS: 777. Stopping powerd. Waiting for PIDS: 754. Stopping ntpd. Waiting for PIDS: 751. Stopping nfsd. Waiting for PIDS: 692 698. Stopping mountd. Waiting for PIDS: 686. Stopping nfsuserd. Waiting for PIDS: 675 676 677 678 679. Stopping casperd. Waiting for PIDS: 654. Stopping watchdogd. Waiting for PIDS: 631. rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered Stopping rpcbind. Waiting for PIDS: 622. Stopping devd. Waiting for PIDS: 491. Writing entropy file:. . Terminated Mar 7 23:43:28 x4 syslogd: exiting on signal 15 Mar 7 23:43:48 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 11 398.4 0.0 0 32 - RL 9:03AM 1153:33.14 [idle] 0 0 0 155 0 - root 0 0.0 0.0 0 136 - DLs 9:03AM 10:42.13 [kernel] 0 0 0 -8 0 - root 1 0.0 0.0 9220 804 - SLs 9:03AM 0:00.72 /sbin/init -- 0 0 0 20 0 wait root 2 0.0 0.0 0 16 - DL 9:03AM 3:27.57 [cam] 0 0 0 -16 0 ccb_scan root 3 0.0 0.0 0 8 - DL 9:03AM 0:00.27 [fdc0] 0 0 0 -16 0 - root 4 0.0 0.0 0 8 - DL 9:03AM 0:00.00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 5 0.0 0.0 0 8 - DL 9:03AM 13:47.49 [pagedaemon] 0 0 0 -16 0 psleep root 6 0.0 0.0 0 8 - DL 9:03AM 0:07.95 [vmdaemon] 0 0 0 -16 0 psleep root 7 0.0 0.0 0 8 - DL 9:03AM 0:00.00 [pagezero] 0 0 0 155 0 pgzero root 8 0.0 0.0 0 40 - DL 9:03AM 180:37.87 [bufdaemon] 0 0 0 20 0 sdflush root 9 0.0 0.0 0 8 - DL 9:03AM 276:35.91 [syncer] 0 0 0 16 0 syncer root 10 0.0 0.0 0 8 - DL 9:03AM 0:00.00 [audit] 0 0 0 -16 0 audit_wo root 12 0.0 0.0 0 160 - WL 9:03AM 17:07.53 [intr] 0 0 0 -84 0 - root 13 0.0 0.0 0 24 - DL 9:03AM 2:35.46 [geom] 0 0 0 -8 0 - root 14 0.0 0.0 0 8 - DL 9:03AM 0:23.81 [rand_harvestq] 0 0 0 -16 0 - root 15 0.0 0.0 0 192 - DL 9:03AM 0:02.41 [usb] 0 0 0 -68 0 - root 16 0.0 0.0 0 8 - DL 9:03AM 14:57.54 [vnlru] 0 0 0 -16 0 vlruwt root 16616 0.0 0.0 0 8 - DL 5:30PM 0:00.00 [crypto] 0 0 0 -16 0 crypto_w root 16617 0.0 0.0 0 8 - DL 5:30PM 0:00.00 [crypto returns] 0 0 0 -16 0 crypto_r root 80444 0.0 0.1 10376 2652 u0 Ss 11:43PM 0:00.01 -sh (sh) 0 1 0 20 0 wait root 80445 0.0 0.1 10092 1936 u0 R+ 11:43PM 0:00.00 ps auxwwl 0 80444 0 20 0 - root 79400 0.0 0.0 5772 1520 0- D 8:33PM 0:00.01 /tmp/dup2 0 1 0 20 0 kmem are # ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_break+0x4e: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xc86fc930 kdb_break(c88ef800,c131ef6f,ef,9d67fa3d,1,...) at kdb_break+0x4e/frame 0xe4747ae8 uart_intr(c88ef800,0,c145f4bc,ffffffff,0,...) at uart_intr+0x91/frame 0xe4747b24 intr_event_handle(c8749e00,e4747ba0,5653304a,e4747ba0,30,...) at intr_event_handle+0x7a/frame 0xe4747b50 intr_execute_handlers(c87450d4,e4747ba0,2) at intr_execute_handlers+0x42/frame 0xe4747b70 lapic_handle_intr(30,e4747ba0) at lapic_handle_intr+0x6f/frame 0xe4747b90 Xapic_isr1() at Xapic_isr1+0x35/frame 0xe4747b90 --- interrupt, eip = 0xc0fce735, esp = 0xe4747be0, ebp = 0xe4747be0 --- acpi_cpu_c1(e4734c90,0,c131ef6f,303,e4734c80,...) at acpi_cpu_c1+0x5/frame 0xe4747be0 acpi_cpu_idle(5653304a,0,5653304a,e4747c50,c0fd8c0a,...) at acpi_cpu_idle+0x142/frame 0xe4747c20 cpu_idle_acpi(5653304a,0,c144fe88,c144fe94,c144fe84,...) at cpu_idle_acpi+0x3f/frame 0xe4747c34 cpu_idle(0,2,c115a873,3f4,c144fe88,...) at cpu_idle+0x9a/frame 0xe4747c50 sched_idletd(0,e4747d08,c11519ec,3c9,0,...) at sched_idletd+0x1c8/frame 0xe4747ccc fork_exit(c0b0c390,0,e4747d08) at fork_exit+0x7f/frame 0xe4747cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4747cf4 --- trap 0, eip = 0, esp = 0xe4747d40, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x822c00 curthread = 0xc86fd000: pid 11 "idle: cpu0" curpcb = 0xe4741d60 fpcurthread = none idlethread = 0xc86fd000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x2332ec00 curthread = 0xc86fcc40: pid 11 "idle: cpu1" curpcb = 0xe4744d60 fpcurthread = none idlethread = 0xc86fcc40: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x23331c00 curthread = 0xc86fc930: pid 11 "idle: cpu2" curpcb = 0xe4747d60 fpcurthread = none idlethread = 0xc86fc930: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x23334c00 curthread = 0xc86fc620: pid 11 "idle: cpu3" curpcb = 0xe474ad60 fpcurthread = none idlethread = 0xc86fc620: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc8d792ec /dev/ad4s1a on / (ufs) 0xc8d795d8 devfs on /dev (devfs) 0xc8d79000 /dev/ad4s1f on /home (ufs) 0xc92fa8c4 /dev/ad4s1d on /usr (ufs) 0xc92fa5d8 /dev/ad4s1e on /var (ufs) 0xc8d78bb0 procfs on /proc (procfs) 0xc9219000 /dev/label/tmp on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 80444 1 80444 0 Ss+ ttyin 0xc884aa70 sh 79400 1 25902 0 D+ kmem are 0xc1460800 dup2 16617 0 0 0 DL crypto_r 0xd38b0984 [crypto returns] 16616 0 0 0 DL crypto_w 0xd38b08cc [crypto] 16 0 0 0 DL vlruwt 0xc8c718c4 [vnlru] 9 0 0 0 DL syncer 0xc16fbf5c [syncer] 8 0 0 0 DL (threaded) [bufdaemon] 101212 D sdflush 0xd3cd7484 [/tmp worker] 100086 D sdflush 0xc9303a84 [/var worker] 100085 D sdflush 0xc9303c84 [/usr worker] 100083 D sdflush 0xc9144884 [/home worker] 100072 D psleep 0xc16fba84 [bufdaemon] 7 0 0 0 DL pgzero 0xc1700e20 [pagezero] 6 0 0 0 DL psleep 0xc1700b44 [vmdaemon] 5 0 0 0 DL psleep 0xc175fd84 [pagedaemon] 4 0 0 0 DL waiting_ 0xc175c7bc [sctp_iterator] 3 0 0 0 DL - 0xc88efa3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc894fd34 [usbus5] 100057 D - 0xc894fd04 [usbus5] 100056 D - 0xc894fcd4 [usbus5] 100055 D - 0xc894fca4 [usbus5] 100053 D - 0xc8940b5c [usbus4] 100052 D - 0xc8940b2c [usbus4] 100051 D - 0xc8940afc [usbus4] 100050 D - 0xc8940acc [usbus4] 100049 D - 0xc893eb5c [usbus3] 100048 D - 0xc893eb2c [usbus3] 100047 D - 0xc893eafc [usbus3] 100046 D - 0xc893eacc [usbus3] 100045 D - 0xc8937b5c [usbus2] 100044 D - 0xc8937b2c [usbus2] 100043 D - 0xc8937afc [usbus2] 100042 D - 0xc8937acc [usbus2] 100040 D - 0xc8932b5c [usbus1] 100039 D - 0xc8932b2c [usbus1] 100038 D - 0xc8932afc [usbus1] 100037 D - 0xc8932acc [usbus1] 100035 D - 0xc892bb5c [usbus0] 100034 D - 0xc892bb2c [usbus0] 100033 D - 0xc892bafc [usbus0] 100032 D - 0xc892bacc [usbus0] 2 0 0 0 DL (threaded) [cam] 100068 D ccb_scan 0xc14032a8 [scanner] 100019 D - 0xc1403400 [doneq0] 14 0 0 0 DL - 0xc141d038 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc1758938 [g_down] 100014 D - 0xc1758934 [g_up] 100013 D - 0xc1758930 [g_event] 12 0 0 0 WL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100023 I [swi5: fast taskq] 100021 I [swi6: Giant taskq] 100020 I [swi6: task queue] 100012 I [swi3: vm] 100011 I [swi4: clock (3)] 100010 I [swi4: clock (2)] 100009 I [swi4: clock (1)] 100008 I [swi4: clock (0)] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc86f52ec [init] 10 0 0 0 DL audit_wo 0xc175e698 [audit] 0 0 0 0 DLs (threaded) [kernel] 105226 D - 0xcae93b80 [metaslab_group_task] 105225 D - 0xcae93b80 [metaslab_group_task] 105224 D - 0xcae93c00 [metaslab_group_task] 104970 D - 0xcae93c00 [metaslab_group_task] 105218 D - 0xc94fb880 [metaslab_group_task] 105217 D - 0xc94fb880 [metaslab_group_task] 100067 D - 0xc86e9a80 [CAM taskq] 100065 D - 0xc1441ccc [deadlkres] 100064 D - 0xc86e9680 [mca taskq] 100027 D - 0xc86e9700 [ffs_trim taskq] 100026 D - 0xc86e9780 [acpi_task_2] 100025 D - 0xc86e9780 [acpi_task_1] 100024 D - 0xc86e9780 [acpi_task_0] 100022 D - 0xc86e9880 [thread taskq] 100018 D - 0xc86e9b00 [kqueue taskq] 100016 D - 0xc86e9b80 [firmware taskq] 100000 D swapin 0xc17589b4 [swapper] db:0:ps> allt Tracing command sh pid 80444 tid 100145 td 0xc9919000 sched_switch(c9919000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xf88fc99c mi_switch(104,0,c115e489,1f1,0,...) at mi_switch+0x1c3/frame 0xf88fc9d0 sleepq_switch(c9919000,0,c115e489,1a7,b0,...) at sleepq_switch+0x1c5/frame 0xf88fca04 sleepq_catch_signals(f88fca6c,c0accb29,c884aa14,0,0,...) at sleepq_catch_signals+0x3ce/frame 0xf88fca48 sleepq_wait_sig(c884aa70,0,f88fcaa0,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88fca6c _cv_wait_sig(c884aa70,c884aa04,c1163ab9,558,1,...) at _cv_wait_sig+0x26e/frame 0xf88fcab4 tty_wait(c884aa00,c884aa70,f88fcc10,1,0,...) at tty_wait+0x4b/frame 0xf88fcad8 ttydisc_read(c884aa00,f88fcc10,0,a1,0,...) at ttydisc_read+0x39d/frame 0xf88fcb1c ttydev_read(c85c9c00,f88fcc10,0,1,0,...) at ttydev_read+0x81/frame 0xf88fcb3c devfs_read_f(c8e13ea8,f88fcc10,c85d2800,0,c9919000,...) at devfs_read_f+0x85/frame 0xf88fcb70 dofileread(c8e13ea8,f88fcc10,ffffffff,ffffffff,0,...) at dofileread+0x9e/frame 0xf88fcba8 kern_readv(c9919000,0,f88fcc10,2c,bfbfddab,...) at kern_readv+0x96/frame 0xf88fcbf0 sys_read(c9919000,f88fccc8,c0ff14df,e4734c90,0,...) at sys_read+0x5c/frame 0xf88fcc40 syscall(f88fcd08) at syscall+0x31b/frame 0xf88fccfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88fccfc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x282081d7, esp = 0xbfbfdd5c, ebp = 0xbfbfdd7c --- Tracing command dup2 pid 79400 tid 104108 td 0xc8db4c40 sched_switch(c8db4c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfaee998c mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfaee99c0 sleepq_switch(c8db4c40,0,c115e489,267,c1460800,...) at sleepq_switch+0x1c5/frame 0xfaee99f4 sleepq_wait(c1460800,0,faee9a48,1,0,...) at sleepq_wait+0x48/frame 0xfaee9a18 _cv_wait(c1460800,c1460780,c115fa81,4ab,c1fb7110,...) at _cv_wait+0x263/frame 0xfaee9a5c vmem_xalloc(c1460780,6bf000,0,0,0,...) at vmem_xalloc+0x1f1/frame 0xfaee9ad0 vmem_alloc(c1460780,6bf000,2102,faee9b20,0,...) at vmem_alloc+0xc4/frame 0xfaee9b08 kmem_malloc(c1460780,6bf000,102,6bf000,2,...) at kmem_malloc+0x3d/frame 0xfaee9b30 uma_large_malloc(6bf000,102,80,c0aef070,0,...) at uma_large_malloc+0x40/frame 0xfaee9b50 malloc(6bee08,c13bff74,102,5e4,8,...) at malloc+0x202/frame 0xfaee9ba4 fdgrowtable(c8c3e428,14,c11509b1,5cf,4c1e,...) at fdgrowtable+0x77/frame 0xfaee9be0 do_dup(c8db4c40,1,1,18028,c8db4e9c) at do_dup+0x25a/frame 0xfaee9c20 sys_dup2(c8db4c40,faee9cc8,c131d9ea,d5,0,...) at sys_dup2+0x34/frame 0xfaee9c40 syscall(faee9d08) at syscall+0x31b/frame 0xfaee9cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfaee9cfc --- syscall (90, FreeBSD ELF32, sys_dup2), eip = 0x2810aae3, esp = 0xbfbfda98, ebp = 0xbfbfdac4 --- Tracing command crypto returns pid 16617 tid 102904 td 0xcc02c620 sched_switch(cc02c620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xf9c94b98 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xf9c94bcc sleepq_switch(cc02c620,0,c115e489,267,d38b0984,...) at sleepq_switch+0x1c5/frame 0xf9c94c00 sleepq_wait(d38b0984,6c,d38af315,0,0,...) at sleepq_wait+0x48/frame 0xf9c94c24 _sleep(d38b0984,d38b0970,6c,d38af315,0,...) at _sleep+0x391/frame 0xf9c94c80 crypto_ret_proc(0,f9c94d08,c11519ec,3c9,0,...) at crypto_ret_proc+0x1d2/frame 0xf9c94ccc fork_exit(d3895520,0,f9c94d08) at fork_exit+0x7f/frame 0xf9c94cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf9c94cf4 --- trap 0, eip = 0, esp = 0xf9c94d40, ebp = 0 --- Tracing command crypto pid 16616 tid 102875 td 0xca76d930 sched_switch(ca76d930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xf9c20ba8 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xf9c20bdc sleepq_switch(ca76d930,0,c115e489,267,d38b08cc,...) at sleepq_switch+0x1c5/frame 0xf9c20c10 sleepq_wait(d38b08cc,6c,d38af371,0,0,...) at sleepq_wait+0x48/frame 0xf9c20c34 _sleep(d38b08cc,d38b08b4,6c,d38af371,0,...) at _sleep+0x391/frame 0xf9c20c90 crypto_proc(0,f9c20d08,c11519ec,3c9,0,...) at crypto_proc+0x358/frame 0xf9c20ccc fork_exit(d3895110,0,f9c20d08) at fork_exit+0x7f/frame 0xf9c20cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf9c20cf4 --- trap 0, eip = 0, esp = 0xf9c20d40, ebp = 0 --- Tracing command vnlru pid 16 tid 100074 td 0xc8b92c40 sched_switch(c8b92c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb10b78 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb10bac sleepq_switch(c8b92c40,0,c115e489,28a,c8c718c4,...) at sleepq_switch+0x1c5/frame 0xedb10be0 sleepq_timedwait(c8c718c4,60,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb10c04 _sleep(c8c718c4,c16fbf14,260,c116b015,fffffed8,...) at _sleep+0x35e/frame 0xedb10c60 vnlru_proc(0,edb10d08,c11519ec,3c9,0,...) at vnlru_proc+0xcf/frame 0xedb10ccc fork_exit(c0b9a080,0,edb10d08) at fork_exit+0x7f/frame 0xedb10cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb10cf4 --- trap 0, eip = 0, esp = 0xedb10d40, ebp = 0 --- Tracing command syncer pid 9 tid 100073 td 0xc8b93000 sched_switch(c8b93000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb0db78 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xedb0dbac sleepq_switch(c8b93000,0,c115e489,28a,0,...) at sleepq_switch+0x1c5/frame 0xedb0dbe0 sleepq_timedwait(c16fbf5c,0,edb0dc3c,0,0,...) at sleepq_timedwait+0x48/frame 0xedb0dc04 _cv_timedwait_sbt(c16fbf5c,c16fbf3c,fffffed8,0,0,...) at _cv_timedwait_sbt+0x299/frame 0xedb0dc50 sched_sync(0,edb0dd08,c11519ec,3c9,0,...) at sched_sync+0x67d/frame 0xedb0dccc fork_exit(c0b997e0,0,edb0dd08) at fork_exit+0x7f/frame 0xedb0dcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb0dcf4 --- trap 0, eip = 0, esp = 0xedb0dd40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 101212 td 0xd3d15310 sched_switch(d3d15310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xf8945ba0 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xf8945bd4 sleepq_switch(d3d15310,0,c115e489,28a,d3cd7484,...) at sleepq_switch+0x1c5/frame 0xf8945c08 sleepq_timedwait(d3cd7484,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf8945c2c _sleep(d3cd7484,d3cd7400,54,c11924cb,7fffff6c,...) at _sleep+0x35e/frame 0xf8945c88 softdep_flush(c9219000,f8945d08,c11519ec,3c9,0,...) at softdep_flush+0x152/frame 0xf8945ccc fork_exit(c0d4c270,c9219000,f8945d08) at fork_exit+0x7f/frame 0xf8945cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8945cf4 --- trap 0, eip = 0, esp = 0xf8945d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100086 td 0xc8eaa930 sched_switch(c8eaa930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb4aba0 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb4abd4 sleepq_switch(c8eaa930,0,c115e489,28a,c9303a84,...) at sleepq_switch+0x1c5/frame 0xedb4ac08 sleepq_timedwait(c9303a84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb4ac2c _sleep(c9303a84,c9303a00,54,c11924cb,7fffff6c,...) at _sleep+0x35e/frame 0xedb4ac88 softdep_flush(c92fa5d8,edb4ad08,c11519ec,3c9,0,...) at softdep_flush+0x152/frame 0xedb4accc fork_exit(c0d4c270,c92fa5d8,edb4ad08) at fork_exit+0x7f/frame 0xedb4acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb4acf4 --- trap 0, eip = 0, esp = 0xedb4ad40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100085 td 0xc8e93620 sched_switch(c8e93620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb47ba0 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb47bd4 sleepq_switch(c8e93620,0,c115e489,28a,c9303c84,...) at sleepq_switch+0x1c5/frame 0xedb47c08 sleepq_timedwait(c9303c84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb47c2c _sleep(c9303c84,c9303c00,54,c11924cb,7fffff6c,...) at _sleep+0x35e/frame 0xedb47c88 softdep_flush(c92fa8c4,edb47d08,c11519ec,3c9,0,...) at softdep_flush+0x152/frame 0xedb47ccc fork_exit(c0d4c270,c92fa8c4,edb47d08) at fork_exit+0x7f/frame 0xedb47cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb47cf4 --- trap 0, eip = 0, esp = 0xedb47d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100083 td 0xc8eaac40 sched_switch(c8eaac40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb41ba0 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb41bd4 sleepq_switch(c8eaac40,0,c115e489,28a,c9144884,...) at sleepq_switch+0x1c5/frame 0xedb41c08 sleepq_timedwait(c9144884,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb41c2c _sleep(c9144884,c9144800,54,c11924cb,7fffff6c,...) at _sleep+0x35e/frame 0xedb41c88 softdep_flush(c8d79000,edb41d08,c11519ec,3c9,0,...) at softdep_flush+0x152/frame 0xedb41ccc fork_exit(c0d4c270,c8d79000,edb41d08) at fork_exit+0x7f/frame 0xedb41cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb41cf4 --- trap 0, eip = 0, esp = 0xedb41d40, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100072 td 0xc8b93310 sched_switch(c8b93310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb0abb0 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb0abe4 sleepq_switch(c8b93310,0,c115e489,28a,c16fba84,...) at sleepq_switch+0x1c5/frame 0xedb0ac18 sleepq_timedwait(c16fba84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb0ac3c _sleep(c16fba84,c16fba00,54,c116795a,fffffed8,...) at _sleep+0x35e/frame 0xedb0ac98 buf_daemon(0,edb0ad08,c11519ec,3c9,0,...) at buf_daemon+0x8c/frame 0xedb0accc fork_exit(c0b7c5b0,0,edb0ad08) at fork_exit+0x7f/frame 0xedb0acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb0acf4 --- trap 0, eip = 0, esp = 0xedb0ad40, ebp = 0 --- Tracing command pagezero pid 7 tid 100071 td 0xc8b93620 sched_switch(c8b93620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb07bac mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb07be0 sleepq_switch(c8b93620,0,c115e489,28a,c1700e20,...) at sleepq_switch+0x1c5/frame 0xedb07c14 sleepq_timedwait(c1700e20,0,12b,0,0,...) at sleepq_timedwait+0x48/frame 0xedb07c38 _sleep(c1700e20,c175fd00,0,c1199d80,fffea520,...) at _sleep+0x35e/frame 0xedb07c94 vm_pagezero(0,edb07d08,c11519ec,3c9,0,...) at vm_pagezero+0xc2/frame 0xedb07ccc fork_exit(c0dabc70,0,edb07d08) at fork_exit+0x7f/frame 0xedb07cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb07cf4 --- trap 0, eip = 0, esp = 0xedb07d40, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100070 td 0xc8b93930 sched_switch(c8b93930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb04b78 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xedb04bac sleepq_switch(c8b93930,0,c115e489,267,c1700b44,...) at sleepq_switch+0x1c5/frame 0xedb04be0 sleepq_wait(c1700b44,74,c116795a,0,0,...) at sleepq_wait+0x48/frame 0xedb04c04 _sleep(c1700b44,c1700b2c,74,c116795a,0,...) at _sleep+0x391/frame 0xedb04c60 vm_daemon(0,edb04d08,c11519ec,3c9,0,...) at vm_daemon+0xac/frame 0xedb04ccc fork_exit(c0da3d60,0,edb04d08) at fork_exit+0x7f/frame 0xedb04cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb04cf4 --- trap 0, eip = 0, esp = 0xedb04d40, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100069 td 0xc8b93c40 sched_switch(c8b93c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xedb01b7c mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xedb01bb0 sleepq_switch(c8b93c40,0,c115e489,28a,c175fd84,...) at sleepq_switch+0x1c5/frame 0xedb01be4 sleepq_timedwait(c175fd84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb01c08 _sleep(c175fd84,c175fd00,54,c116795a,fffffed8,...) at _sleep+0x35e/frame 0xedb01c64 vm_pageout(0,edb01d08,c11519ec,3c9,0,...) at vm_pageout+0x244/frame 0xedb01ccc fork_exit(c0da4590,0,edb01d08) at fork_exit+0x7f/frame 0xedb01cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb01cf4 --- trap 0, eip = 0, esp = 0xedb01d40, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100066 td 0xc8b94620 sched_switch(c8b94620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed978bb8 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed978bec sleepq_switch(c8b94620,0,c115e489,267,c175c7bc,...) at sleepq_switch+0x1c5/frame 0xed978c20 sleepq_wait(c175c7bc,0,c117e771,0,0,...) at sleepq_wait+0x48/frame 0xed978c44 _sleep(c175c7bc,c175c784,0,c117e771,0,...) at _sleep+0x391/frame 0xed978ca0 sctp_iterator_thread(0,ed978d08,c11519ec,3c9,0,...) at sctp_iterator_thread+0x7c/frame 0xed978ccc fork_exit(c0c3cc80,0,ed978d08) at fork_exit+0x7f/frame 0xed978cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed978cf4 --- trap 0, eip = 0, esp = 0xed978d40, ebp = 0 --- Tracing command fdc0 pid 3 tid 100061 td 0xc8946310 sched_switch(c8946310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed968b4c mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xed968b80 sleepq_switch(c8946310,0,c115e489,28a,c88efa3c,...) at sleepq_switch+0x1c5/frame 0xed968bb4 sleepq_timedwait(c88efa3c,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xed968bd8 _sleep(c88efa3c,c88efaf0,5c,c1143b4e,fffffed8,...) at _sleep+0x35e/frame 0xed968c34 fdc_thread(c88efa00,ed968d08,c11519ec,3c9,0,...) at fdc_thread+0x80a/frame 0xed968ccc fork_exit(c0f62910,c88efa00,ed968d08) at fork_exit+0x7f/frame 0xed968cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed968cf4 --- trap 0, eip = 0, esp = 0xed968d40, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc8946c40 sched_switch(c8946c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed944bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed944c08 sleepq_switch(c8946c40,0,c115e489,267,c894fd34,...) at sleepq_switch+0x1c5/frame 0xed944c3c sleepq_wait(c894fd34,0,ed944c90,1,0,...) at sleepq_wait+0x48/frame 0xed944c60 _cv_wait(c894fd34,c894fe74,c112e8b7,77,c894fd3c,...) at _cv_wait+0x263/frame 0xed944ca4 usb_process(c894fd2c,ed944d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed944ccc fork_exit(c096ebf0,c894fd2c,ed944d08) at fork_exit+0x7f/frame 0xed944cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed944cf4 --- trap 0, eip = 0, esp = 0xed944d40, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc8947000 sched_switch(c8947000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed941bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed941c08 sleepq_switch(c8947000,0,c115e489,267,c894fd04,...) at sleepq_switch+0x1c5/frame 0xed941c3c sleepq_wait(c894fd04,0,ed941c90,1,0,...) at sleepq_wait+0x48/frame 0xed941c60 _cv_wait(c894fd04,c894fe74,c112e8b7,77,c894fd0c,...) at _cv_wait+0x263/frame 0xed941ca4 usb_process(c894fcfc,ed941d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed941ccc fork_exit(c096ebf0,c894fcfc,ed941d08) at fork_exit+0x7f/frame 0xed941cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed941cf4 --- trap 0, eip = 0, esp = 0xed941d40, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc8947310 sched_switch(c8947310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed93ebd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed93ec08 sleepq_switch(c8947310,0,c115e489,267,c894fcd4,...) at sleepq_switch+0x1c5/frame 0xed93ec3c sleepq_wait(c894fcd4,0,ed93ec90,1,0,...) at sleepq_wait+0x48/frame 0xed93ec60 _cv_wait(c894fcd4,c894fe74,c112e8b7,77,c894fcdc,...) at _cv_wait+0x263/frame 0xed93eca4 usb_process(c894fccc,ed93ed08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed93eccc fork_exit(c096ebf0,c894fccc,ed93ed08) at fork_exit+0x7f/frame 0xed93ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93ecf4 --- trap 0, eip = 0, esp = 0xed93ed40, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc8947620 sched_switch(c8947620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed93bbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed93bc08 sleepq_switch(c8947620,0,c115e489,267,c894fca4,...) at sleepq_switch+0x1c5/frame 0xed93bc3c sleepq_wait(c894fca4,0,ed93bc90,1,0,...) at sleepq_wait+0x48/frame 0xed93bc60 _cv_wait(c894fca4,c894fe74,c112e8b7,77,c894fcac,...) at _cv_wait+0x263/frame 0xed93bca4 usb_process(c894fc9c,ed93bd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed93bccc fork_exit(c096ebf0,c894fc9c,ed93bd08) at fork_exit+0x7f/frame 0xed93bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93bcf4 --- trap 0, eip = 0, esp = 0xed93bd40, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc8947c40 sched_switch(c8947c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed932bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed932c08 sleepq_switch(c8947c40,0,c115e489,267,c8940b5c,...) at sleepq_switch+0x1c5/frame 0xed932c3c sleepq_wait(c8940b5c,0,ed932c90,1,0,...) at sleepq_wait+0x48/frame 0xed932c60 _cv_wait(c8940b5c,c8940c9c,c112e8b7,77,c8940b64,...) at _cv_wait+0x263/frame 0xed932ca4 usb_process(c8940b54,ed932d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed932ccc fork_exit(c096ebf0,c8940b54,ed932d08) at fork_exit+0x7f/frame 0xed932cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed932cf4 --- trap 0, eip = 0, esp = 0xed932d40, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc88b2c40 sched_switch(c88b2c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed92fbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed92fc08 sleepq_switch(c88b2c40,0,c115e489,267,c8940b2c,...) at sleepq_switch+0x1c5/frame 0xed92fc3c sleepq_wait(c8940b2c,0,ed92fc90,1,0,...) at sleepq_wait+0x48/frame 0xed92fc60 _cv_wait(c8940b2c,c8940c9c,c112e8b7,77,c8940b34,...) at _cv_wait+0x263/frame 0xed92fca4 usb_process(c8940b24,ed92fd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed92fccc fork_exit(c096ebf0,c8940b24,ed92fd08) at fork_exit+0x7f/frame 0xed92fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed92fcf4 --- trap 0, eip = 0, esp = 0xed92fd40, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc8934000 sched_switch(c8934000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed92cbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed92cc08 sleepq_switch(c8934000,0,c115e489,267,c8940afc,...) at sleepq_switch+0x1c5/frame 0xed92cc3c sleepq_wait(c8940afc,0,ed92cc90,1,0,...) at sleepq_wait+0x48/frame 0xed92cc60 _cv_wait(c8940afc,c8940c9c,c112e8b7,77,c8940b04,...) at _cv_wait+0x263/frame 0xed92cca4 usb_process(c8940af4,ed92cd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed92cccc fork_exit(c096ebf0,c8940af4,ed92cd08) at fork_exit+0x7f/frame 0xed92ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed92ccf4 --- trap 0, eip = 0, esp = 0xed92cd40, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc8934310 sched_switch(c8934310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed929bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed929c08 sleepq_switch(c8934310,0,c115e489,267,c8940acc,...) at sleepq_switch+0x1c5/frame 0xed929c3c sleepq_wait(c8940acc,0,ed929c90,1,0,...) at sleepq_wait+0x48/frame 0xed929c60 _cv_wait(c8940acc,c8940c9c,c112e8b7,77,c8940ad4,...) at _cv_wait+0x263/frame 0xed929ca4 usb_process(c8940ac4,ed929d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed929ccc fork_exit(c096ebf0,c8940ac4,ed929d08) at fork_exit+0x7f/frame 0xed929cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed929cf4 --- trap 0, eip = 0, esp = 0xed929d40, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc8934620 sched_switch(c8934620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed925bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed925c08 sleepq_switch(c8934620,0,c115e489,267,c893eb5c,...) at sleepq_switch+0x1c5/frame 0xed925c3c sleepq_wait(c893eb5c,0,ed925c90,1,0,...) at sleepq_wait+0x48/frame 0xed925c60 _cv_wait(c893eb5c,c893ec9c,c112e8b7,77,c893eb64,...) at _cv_wait+0x263/frame 0xed925ca4 usb_process(c893eb54,ed925d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed925ccc fork_exit(c096ebf0,c893eb54,ed925d08) at fork_exit+0x7f/frame 0xed925cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed925cf4 --- trap 0, eip = 0, esp = 0xed925d40, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc8934930 sched_switch(c8934930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed922bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed922c08 sleepq_switch(c8934930,0,c115e489,267,c893eb2c,...) at sleepq_switch+0x1c5/frame 0xed922c3c sleepq_wait(c893eb2c,0,ed922c90,1,0,...) at sleepq_wait+0x48/frame 0xed922c60 _cv_wait(c893eb2c,c893ec9c,c112e8b7,77,c893eb34,...) at _cv_wait+0x263/frame 0xed922ca4 usb_process(c893eb24,ed922d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed922ccc fork_exit(c096ebf0,c893eb24,ed922d08) at fork_exit+0x7f/frame 0xed922cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed922cf4 --- trap 0, eip = 0, esp = 0xed922d40, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc8934c40 sched_switch(c8934c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed91fbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed91fc08 sleepq_switch(c8934c40,0,c115e489,267,c893eafc,...) at sleepq_switch+0x1c5/frame 0xed91fc3c sleepq_wait(c893eafc,0,ed91fc90,1,0,...) at sleepq_wait+0x48/frame 0xed91fc60 _cv_wait(c893eafc,c893ec9c,c112e8b7,77,c893eb04,...) at _cv_wait+0x263/frame 0xed91fca4 usb_process(c893eaf4,ed91fd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed91fccc fork_exit(c096ebf0,c893eaf4,ed91fd08) at fork_exit+0x7f/frame 0xed91fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91fcf4 --- trap 0, eip = 0, esp = 0xed91fd40, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc8935000 sched_switch(c8935000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed91cbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed91cc08 sleepq_switch(c8935000,0,c115e489,267,c893eacc,...) at sleepq_switch+0x1c5/frame 0xed91cc3c sleepq_wait(c893eacc,0,ed91cc90,1,0,...) at sleepq_wait+0x48/frame 0xed91cc60 _cv_wait(c893eacc,c893ec9c,c112e8b7,77,c893ead4,...) at _cv_wait+0x263/frame 0xed91cca4 usb_process(c893eac4,ed91cd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed91cccc fork_exit(c096ebf0,c893eac4,ed91cd08) at fork_exit+0x7f/frame 0xed91ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91ccf4 --- trap 0, eip = 0, esp = 0xed91cd40, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc8935310 sched_switch(c8935310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed918bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed918c08 sleepq_switch(c8935310,0,c115e489,267,c8937b5c,...) at sleepq_switch+0x1c5/frame 0xed918c3c sleepq_wait(c8937b5c,0,ed918c90,1,0,...) at sleepq_wait+0x48/frame 0xed918c60 _cv_wait(c8937b5c,c8937c9c,c112e8b7,77,c8937b64,...) at _cv_wait+0x263/frame 0xed918ca4 usb_process(c8937b54,ed918d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed918ccc fork_exit(c096ebf0,c8937b54,ed918d08) at fork_exit+0x7f/frame 0xed918cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed918cf4 --- trap 0, eip = 0, esp = 0xed918d40, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc8935620 sched_switch(c8935620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed915bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed915c08 sleepq_switch(c8935620,0,c115e489,267,c8937b2c,...) at sleepq_switch+0x1c5/frame 0xed915c3c sleepq_wait(c8937b2c,0,ed915c90,1,0,...) at sleepq_wait+0x48/frame 0xed915c60 _cv_wait(c8937b2c,c8937c9c,c112e8b7,77,c8937b34,...) at _cv_wait+0x263/frame 0xed915ca4 usb_process(c8937b24,ed915d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed915ccc fork_exit(c096ebf0,c8937b24,ed915d08) at fork_exit+0x7f/frame 0xed915cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed915cf4 --- trap 0, eip = 0, esp = 0xed915d40, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc8935930 sched_switch(c8935930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed912bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed912c08 sleepq_switch(c8935930,0,c115e489,267,c8937afc,...) at sleepq_switch+0x1c5/frame 0xed912c3c sleepq_wait(c8937afc,0,ed912c90,1,0,...) at sleepq_wait+0x48/frame 0xed912c60 _cv_wait(c8937afc,c8937c9c,c112e8b7,77,c8937b04,...) at _cv_wait+0x263/frame 0xed912ca4 usb_process(c8937af4,ed912d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed912ccc fork_exit(c096ebf0,c8937af4,ed912d08) at fork_exit+0x7f/frame 0xed912cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed912cf4 --- trap 0, eip = 0, esp = 0xed912d40, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc8935c40 sched_switch(c8935c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed90fbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed90fc08 sleepq_switch(c8935c40,0,c115e489,267,c8937acc,...) at sleepq_switch+0x1c5/frame 0xed90fc3c sleepq_wait(c8937acc,0,ed90fc90,1,0,...) at sleepq_wait+0x48/frame 0xed90fc60 _cv_wait(c8937acc,c8937c9c,c112e8b7,77,c8937ad4,...) at _cv_wait+0x263/frame 0xed90fca4 usb_process(c8937ac4,ed90fd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed90fccc fork_exit(c096ebf0,c8937ac4,ed90fd08) at fork_exit+0x7f/frame 0xed90fcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90fcf4 --- trap 0, eip = 0, esp = 0xed90fd40, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc8936310 sched_switch(c8936310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed909bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed909c08 sleepq_switch(c8936310,0,c115e489,267,c8932b5c,...) at sleepq_switch+0x1c5/frame 0xed909c3c sleepq_wait(c8932b5c,0,ed909c90,1,0,...) at sleepq_wait+0x48/frame 0xed909c60 _cv_wait(c8932b5c,c8932c9c,c112e8b7,77,c8932b64,...) at _cv_wait+0x263/frame 0xed909ca4 usb_process(c8932b54,ed909d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed909ccc fork_exit(c096ebf0,c8932b54,ed909d08) at fork_exit+0x7f/frame 0xed909cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed909cf4 --- trap 0, eip = 0, esp = 0xed909d40, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc87b3310 sched_switch(c87b3310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed906bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed906c08 sleepq_switch(c87b3310,0,c115e489,267,c8932b2c,...) at sleepq_switch+0x1c5/frame 0xed906c3c sleepq_wait(c8932b2c,0,ed906c90,1,0,...) at sleepq_wait+0x48/frame 0xed906c60 _cv_wait(c8932b2c,c8932c9c,c112e8b7,77,c8932b34,...) at _cv_wait+0x263/frame 0xed906ca4 usb_process(c8932b24,ed906d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed906ccc fork_exit(c096ebf0,c8932b24,ed906d08) at fork_exit+0x7f/frame 0xed906cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed906cf4 --- trap 0, eip = 0, esp = 0xed906d40, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc87b3620 sched_switch(c87b3620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed903bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed903c08 sleepq_switch(c87b3620,0,c115e489,267,c8932afc,...) at sleepq_switch+0x1c5/frame 0xed903c3c sleepq_wait(c8932afc,0,ed903c90,1,0,...) at sleepq_wait+0x48/frame 0xed903c60 _cv_wait(c8932afc,c8932c9c,c112e8b7,77,c8932b04,...) at _cv_wait+0x263/frame 0xed903ca4 usb_process(c8932af4,ed903d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed903ccc fork_exit(c096ebf0,c8932af4,ed903d08) at fork_exit+0x7f/frame 0xed903cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed903cf4 --- trap 0, eip = 0, esp = 0xed903d40, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc87b3930 sched_switch(c87b3930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed900bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed900c08 sleepq_switch(c87b3930,0,c115e489,267,c8932acc,...) at sleepq_switch+0x1c5/frame 0xed900c3c sleepq_wait(c8932acc,0,ed900c90,1,0,...) at sleepq_wait+0x48/frame 0xed900c60 _cv_wait(c8932acc,c8932c9c,c112e8b7,77,c8932ad4,...) at _cv_wait+0x263/frame 0xed900ca4 usb_process(c8932ac4,ed900d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed900ccc fork_exit(c096ebf0,c8932ac4,ed900d08) at fork_exit+0x7f/frame 0xed900cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed900cf4 --- trap 0, eip = 0, esp = 0xed900d40, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc88b1000 sched_switch(c88b1000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed8fabd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed8fac08 sleepq_switch(c88b1000,0,c115e489,267,c892bb5c,...) at sleepq_switch+0x1c5/frame 0xed8fac3c sleepq_wait(c892bb5c,0,ed8fac90,1,0,...) at sleepq_wait+0x48/frame 0xed8fac60 _cv_wait(c892bb5c,c892bc9c,c112e8b7,77,c892bb64,...) at _cv_wait+0x263/frame 0xed8faca4 usb_process(c892bb54,ed8fad08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed8faccc fork_exit(c096ebf0,c892bb54,ed8fad08) at fork_exit+0x7f/frame 0xed8facf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8facf4 --- trap 0, eip = 0, esp = 0xed8fad40, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc88b1310 sched_switch(c88b1310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed8f7bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed8f7c08 sleepq_switch(c88b1310,0,c115e489,267,c892bb2c,...) at sleepq_switch+0x1c5/frame 0xed8f7c3c sleepq_wait(c892bb2c,0,ed8f7c90,1,0,...) at sleepq_wait+0x48/frame 0xed8f7c60 _cv_wait(c892bb2c,c892bc9c,c112e8b7,77,c892bb34,...) at _cv_wait+0x263/frame 0xed8f7ca4 usb_process(c892bb24,ed8f7d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed8f7ccc fork_exit(c096ebf0,c892bb24,ed8f7d08) at fork_exit+0x7f/frame 0xed8f7cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f7cf4 --- trap 0, eip = 0, esp = 0xed8f7d40, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc88b1620 sched_switch(c88b1620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed8f4bd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed8f4c08 sleepq_switch(c88b1620,0,c115e489,267,c892bafc,...) at sleepq_switch+0x1c5/frame 0xed8f4c3c sleepq_wait(c892bafc,0,ed8f4c90,1,0,...) at sleepq_wait+0x48/frame 0xed8f4c60 _cv_wait(c892bafc,c892bc9c,c112e8b7,77,c892bb04,...) at _cv_wait+0x263/frame 0xed8f4ca4 usb_process(c892baf4,ed8f4d08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xed8f4ccc fork_exit(c096ebf0,c892baf4,ed8f4d08) at fork_exit+0x7f/frame 0xed8f4cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f4cf4 --- trap 0, eip = 0, esp = 0xed8f4d40, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc88b1930 sched_switch(c88b1930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xf0ffdbd4 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xf0ffdc08 sleepq_switch(c88b1930,0,c115e489,267,c892bacc,...) at sleepq_switch+0x1c5/frame 0xf0ffdc3c sleepq_wait(c892bacc,0,f0ffdc90,1,0,...) at sleepq_wait+0x48/frame 0xf0ffdc60 _cv_wait(c892bacc,c892bc9c,c112e8b7,77,c892bad4,...) at _cv_wait+0x263/frame 0xf0ffdca4 usb_process(c892bac4,f0ffdd08,c11519ec,3c9,0,...) at usb_process+0xe6/frame 0xf0ffdccc fork_exit(c096ebf0,c892bac4,f0ffdd08) at fork_exit+0x7f/frame 0xf0ffdcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ffdcf4 --- trap 0, eip = 0, esp = 0xf0ffdd40, ebp = 0 --- Tracing command cam pid 2 tid 100068 td 0xc8b94000 sched_switch(c8b94000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed97eb90 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed97ebc4 sleepq_switch(c8b94000,0,c115e489,267,c14032a8,...) at sleepq_switch+0x1c5/frame 0xed97ebf8 sleepq_wait(c14032a8,5c,c101923a,0,0,...) at sleepq_wait+0x48/frame 0xed97ec1c _sleep(c14032a8,c14032f8,5c,c101923a,0,...) at _sleep+0x391/frame 0xed97ec78 xpt_scanner_thread(0,ed97ed08,c11519ec,3c9,0,...) at xpt_scanner_thread+0xac/frame 0xed97eccc fork_exit(c04bfa70,0,ed97ed08) at fork_exit+0x7f/frame 0xed97ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed97ecf4 --- trap 0, eip = 0, esp = 0xed97ed40, ebp = 0 --- Tracing command cam pid 2 tid 100019 td 0xc87b2000 sched_switch(c87b2000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4773b98 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4773bcc sleepq_switch(c87b2000,0,c115e489,267,c1403400,...) at sleepq_switch+0x1c5/frame 0xe4773c00 sleepq_wait(c1403400,5c,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4773c24 _sleep(c1403400,c1403380,5c,c1143b4e,0,...) at _sleep+0x391/frame 0xe4773c80 xpt_done_td(c1403380,e4773d08,c11519ec,3c9,0,...) at xpt_done_td+0xd2/frame 0xe4773ccc fork_exit(c04bf690,c1403380,e4773d08) at fork_exit+0x7f/frame 0xe4773cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4773cf4 --- trap 0, eip = 0, esp = 0xe4773d40, ebp = 0 --- Tracing command rand_harvestq pid 14 tid 100017 td 0xc87b2620 sched_switch(c87b2620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe476cbb4 mi_switch(104,0,c115e489,1f1,c145f238,...) at mi_switch+0x1c3/frame 0xe476cbe8 sleepq_switch(c87b2620,0,c115e489,28a,c141d038,...) at sleepq_switch+0x1c5/frame 0xe476cc1c sleepq_timedwait(c141d038,0,c1158e9a,c1143b4e,0,...) at sleepq_timedwait+0x48/frame 0xe476cc40 msleep_spin_sbt(c141d038,c174d9ec,c1143b4e,19999999,0,...) at msleep_spin_sbt+0x225/frame 0xe476cc88 random_kthread(c0857380,e476cd08,c11519ec,3c9,0,...) at random_kthread+0x213/frame 0xe476cccc fork_exit(c0856460,c0857380,e476cd08) at fork_exit+0x7f/frame 0xe476ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe476ccf4 --- trap 0, eip = 0, esp = 0xe476cd40, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc87b2c40 sched_switch(c87b2c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4766b94 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4766bc8 sleepq_switch(c87b2c40,0,c115e489,267,c1758938,...) at sleepq_switch+0x1c5/frame 0xe4766bfc sleepq_wait(c1758938,5c,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4766c20 _sleep(c1758938,c1440018,25c,c1143b4e,0,...) at _sleep+0x391/frame 0xe4766c7c g_io_schedule_down(c87b2c40,0,c11458a6,6d,e4766cf4,...) at g_io_schedule_down+0x5c/frame 0xe4766cb4 g_down_procbody(0,e4766d08,c11519ec,3c9,0,...) at g_down_procbody+0x9d/frame 0xe4766ccc fork_exit(c0a3bdf0,0,e4766d08) at fork_exit+0x7f/frame 0xe4766cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4766cf4 --- trap 0, eip = 0, esp = 0xe4766d40, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc87b3000 sched_switch(c87b3000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4763b98 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4763bcc sleepq_switch(c87b3000,0,c115e489,267,c1758934,...) at sleepq_switch+0x1c5/frame 0xe4763c00 sleepq_wait(c1758934,5c,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4763c24 _sleep(c1758934,c1440038,25c,c1143b4e,0,...) at _sleep+0x391/frame 0xe4763c80 g_io_schedule_up(c87b3000,0,c11458a6,60,e4763cf4,...) at g_io_schedule_up+0xe6/frame 0xe4763cb4 g_up_procbody(0,e4763d08,c11519ec,3c9,0,...) at g_up_procbody+0x9d/frame 0xe4763ccc fork_exit(c0a3bd50,0,e4763d08) at fork_exit+0x7f/frame 0xe4763cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4763cf4 --- trap 0, eip = 0, esp = 0xe4763d40, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc86f6000 sched_switch(c86f6000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4760ba8 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4760bdc sleepq_switch(c86f6000,0,c115e489,267,c1758930,...) at sleepq_switch+0x1c5/frame 0xe4760c10 sleepq_wait(c1758930,5c,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4760c34 _sleep(c1758930,c143fff0,25c,c1143b4e,0,...) at _sleep+0x391/frame 0xe4760c90 g_run_events(0,e4760d08,c11519ec,3c9,0,...) at g_run_events+0x62/frame 0xe4760ccc fork_exit(c0a3bcb0,0,e4760d08) at fork_exit+0x7f/frame 0xe4760cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4760cf4 --- trap 0, eip = 0, esp = 0xe4760d40, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc8936c40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xc8946000 sched_switch(c8946000,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xed96bc60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xed96bc94 ithread_loop(c874e600,ed96bd08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xed96bccc fork_exit(c0ab1e60,c874e600,ed96bd08) at fork_exit+0x7f/frame 0xed96bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed96bcf4 --- trap 0, eip = 0, esp = 0xed96bd40, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc8946620 sched_switch(c8946620,0,109,1bb,0,...) at sched_switch+0x4cf/frame 0xed95bc60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xed95bc94 ithread_loop(c874e610,ed95bd08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xed95bccc fork_exit(c0ab1e60,c874e610,ed95bd08) at fork_exit+0x7f/frame 0xed95bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed95bcf4 --- trap 0, eip = 0, esp = 0xed95bd40, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc8946930 sched_switch(c8946930,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xed94ac60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xed94ac94 ithread_loop(c874e630,ed94ad08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xed94accc fork_exit(c0ab1e60,c874e630,ed94ad08) at fork_exit+0x7f/frame 0xed94acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed94acf4 --- trap 0, eip = 0, esp = 0xed94ad40, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc8947930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc8936000 sched_switch(c8936000,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xed90cc60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xed90cc94 ithread_loop(c874e650,ed90cd08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xed90cccc fork_exit(c0ab1e60,c874e650,ed90cd08) at fork_exit+0x7f/frame 0xed90ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90ccf4 --- trap 0, eip = 0, esp = 0xed90cd40, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc87b3c40 sched_switch(c87b3c40,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xed8fdc60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xed8fdc94 ithread_loop(c874e660,ed8fdd08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xed8fdccc fork_exit(c0ab1e60,c874e660,ed8fdd08) at fork_exit+0x7f/frame 0xed8fdcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8fdcf4 --- trap 0, eip = 0, esp = 0xed8fdd40, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc88b1c40 sched_switch(c88b1c40,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xf0ffac60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xf0ffac94 ithread_loop(c874e670,f0ffad08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xf0ffaccc fork_exit(c0ab1e60,c874e670,f0ffad08) at fork_exit+0x7f/frame 0xf0ffacf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ffacf4 --- trap 0, eip = 0, esp = 0xf0ffad40, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc88b2000 sched_switch(c88b2000,0,109,1bb,c17696b4,...) at sched_switch+0x4cf/frame 0xf0ecac60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xf0ecac94 ithread_loop(c874e6c0,f0ecad08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xf0ecaccc fork_exit(c0ab1e60,c874e6c0,f0ecad08) at fork_exit+0x7f/frame 0xf0ecacf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ecacf4 --- trap 0, eip = 0, esp = 0xf0ecad40, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc88b2310 sched_switch(c88b2310,0,109,1bb,c1151c5d,...) at sched_switch+0x4cf/frame 0xf0ec6c60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xf0ec6c94 ithread_loop(c874e6d0,f0ec6d08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xf0ec6ccc fork_exit(c0ab1e60,c874e6d0,f0ec6d08) at fork_exit+0x7f/frame 0xf0ec6cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ec6cf4 --- trap 0, eip = 0, esp = 0xf0ec6d40, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc88b2620 sched_switch(c88b2620,0,109,1bb,c1151c5d,...) at sched_switch+0x4cf/frame 0xe47bdc60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xe47bdc94 ithread_loop(c874e6e0,e47bdd08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xe47bdccc fork_exit(c0ab1e60,c874e6e0,e47bdd08) at fork_exit+0x7f/frame 0xe47bdcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe47bdcf4 --- trap 0, eip = 0, esp = 0xe47bdd40, ebp = 0 --- Tracing command intr pid 12 tid 100023 td 0xc87b1310 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xc87b1930 sched_switch(c87b1930,0,109,1bb,0,...) at sched_switch+0x4cf/frame 0xe4779c60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xe4779c94 ithread_loop(c874e740,e4779d08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xe4779ccc fork_exit(c0ab1e60,c874e740,e4779d08) at fork_exit+0x7f/frame 0xe4779cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4779cf4 --- trap 0, eip = 0, esp = 0xe4779d40, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc87b1c40 sched_switch(c87b1c40,0,109,1bb,c1151c5d,...) at sched_switch+0x4cf/frame 0xe4776c60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xe4776c94 ithread_loop(c874e750,e4776d08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xe4776ccc fork_exit(c0ab1e60,c874e750,e4776d08) at fork_exit+0x7f/frame 0xe4776cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4776cf4 --- trap 0, eip = 0, esp = 0xe4776d40, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc86f6310 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc86f6620 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc86f6930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xc86f6c40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc86fc000 sched_switch(c86fc000,0,109,1bb,c1151c5d,...) at sched_switch+0x4cf/frame 0xe4751c60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xe4751c94 ithread_loop(c874e7b0,e4751d08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xe4751ccc fork_exit(c0ab1e60,c874e7b0,e4751d08) at fork_exit+0x7f/frame 0xe4751cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4751cf4 --- trap 0, eip = 0, esp = 0xe4751d40, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc86fc310 sched_switch(c86fc310,0,109,1bb,c1151c5d,...) at sched_switch+0x4cf/frame 0xe474ec60 mi_switch(109,0,c1151c5d,55a,8,...) at mi_switch+0x1c3/frame 0xe474ec94 ithread_loop(c874e7c0,e474ed08,c11519ec,3c9,0,...) at ithread_loop+0x1fb/frame 0xe474eccc fork_exit(c0ab1e60,c874e7c0,e474ed08) at fork_exit+0x7f/frame 0xe474ecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe474ecf4 --- trap 0, eip = 0, esp = 0xe474ed40, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc86fc620 cpustop_handler(3,13,e474ab94,c0fe925a,3,...) at cpustop_handler+0x25/frame 0xe474aa40 ipi_nmi_handler(3,0,0,b0489,e474aa9c,...) at ipi_nmi_handler+0x37/frame 0xe474aa50 trap(e474aba0) at trap+0x3a/frame 0xe474ab94 calltrap() at calltrap+0x6/frame 0xe474ab94 --- trap 0x13, eip = 0xc0fce735, esp = 0xe474abe0, ebp = 0xe474abe0 --- acpi_cpu_c1(e4737c90,0,c131ef6f,303,e4737c80,...) at acpi_cpu_c1+0x5/frame 0xe474abe0 acpi_cpu_idle(7fc89516,0,7fc89516,e474ac50,c0fd8c0a,...) at acpi_cpu_idle+0x142/frame 0xe474ac20 cpu_idle_acpi(7fc89516,0,c1450588,c1450594,c1450584,...) at cpu_idle_acpi+0x3f/frame 0xe474ac34 cpu_idle(0,2,c115a873,a31,c1450588,...) at cpu_idle+0x9a/frame 0xe474ac50 sched_idletd(0,e474ad08,c11519ec,3c9,0,...) at sched_idletd+0x1c8/frame 0xe474accc fork_exit(c0b0c390,0,e474ad08) at fork_exit+0x7f/frame 0xe474acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe474acf4 --- trap 0, eip = 0, esp = 0xe474ad40, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc86fc930 kdb_break(c88ef800,c131ef6f,ef,9d67fa3d,1,...) at kdb_break+0x4e/frame 0xe4747ae8 uart_intr(c88ef800,0,c145f4bc,ffffffff,0,...) at uart_intr+0x91/frame 0xe4747b24 intr_event_handle(c8749e00,e4747ba0,5653304a,e4747ba0,30,...) at intr_event_handle+0x7a/frame 0xe4747b50 intr_execute_handlers(c87450d4,e4747ba0,2) at intr_execute_handlers+0x42/frame 0xe4747b70 lapic_handle_intr(30,e4747ba0) at lapic_handle_intr+0x6f/frame 0xe4747b90 Xapic_isr1() at Xapic_isr1+0x35/frame 0xe4747b90 --- interrupt, eip = 0xc0fce735, esp = 0xe4747be0, ebp = 0xe4747be0 --- acpi_cpu_c1(e4734c90,0,c131ef6f,303,e4734c80,...) at acpi_cpu_c1+0x5/frame 0xe4747be0 acpi_cpu_idle(5653304a,0,5653304a,e4747c50,c0fd8c0a,...) at acpi_cpu_idle+0x142/frame 0xe4747c20 cpu_idle_acpi(5653304a,0,c144fe88,c144fe94,c144fe84,...) at cpu_idle_acpi+0x3f/frame 0xe4747c34 cpu_idle(0,2,c115a873,3f4,c144fe88,...) at cpu_idle+0x9a/frame 0xe4747c50 sched_idletd(0,e4747d08,c11519ec,3c9,0,...) at sched_idletd+0x1c8/frame 0xe4747ccc fork_exit(c0b0c390,0,e4747d08) at fork_exit+0x7f/frame 0xe4747cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4747cf4 --- trap 0, eip = 0, esp = 0xe4747d40, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc86fcc40 cpustop_handler(1,13,e4744b94,c0fe925a,1,...) at cpustop_handler+0x25/frame 0xe4744a40 ipi_nmi_handler(1,0,0,408821,e4744a9c,...) at ipi_nmi_handler+0x37/frame 0xe4744a50 trap(e4744ba0) at trap+0x3a/frame 0xe4744b94 calltrap() at calltrap+0x6/frame 0xe4744b94 --- trap 0x13, eip = 0xc0fce735, esp = 0xe4744be0, ebp = 0xe4744be0 --- acpi_cpu_c1(e4731c90,0,c131ef6f,303,e4731c80,...) at acpi_cpu_c1+0x5/frame 0xe4744be0 acpi_cpu_idle(26a4d83a,0,26a4d83a,e4744c50,c0fd8c0a,...) at acpi_cpu_idle+0x142/frame 0xe4744c20 cpu_idle_acpi(26a4d83a,0,c144f788,c144f794,c144f784,...) at cpu_idle_acpi+0x3f/frame 0xe4744c34 cpu_idle(0,2,c115a873,a31,c144f788,...) at cpu_idle+0x9a/frame 0xe4744c50 sched_idletd(0,e4744d08,c11519ec,3c9,0,...) at sched_idletd+0x1c8/frame 0xe4744ccc fork_exit(c0b0c390,0,e4744d08) at fork_exit+0x7f/frame 0xe4744cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4744cf4 --- trap 0, eip = 0, esp = 0xe4744d40, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc86fd000 cpustop_handler(0,13,e4741b94,c0fe925a,0,...) at cpustop_handler+0x25/frame 0xe4741a40 ipi_nmi_handler(0,0,0,30c822,e4741a9c,...) at ipi_nmi_handler+0x37/frame 0xe4741a50 trap(e4741ba0) at trap+0x3a/frame 0xe4741b94 calltrap() at calltrap+0x6/frame 0xe4741b94 --- trap 0x13, eip = 0xc0fce735, esp = 0xe4741be0, ebp = 0xe4741be0 --- acpi_cpu_c1(c1c25c90,0,c131ef6f,303,c1c25c80,...) at acpi_cpu_c1+0x5/frame 0xe4741be0 acpi_cpu_idle(653779d,0,653779d,e4741c50,c0fd8c0a,...) at acpi_cpu_idle+0x142/frame 0xe4741c20 cpu_idle_acpi(653779d,0,c144f088,c144f094,c144f084,...) at cpu_idle_acpi+0x3f/frame 0xe4741c34 cpu_idle(0,2,c115a873,a31,c144f088,...) at cpu_idle+0x9a/frame 0xe4741c50 sched_idletd(0,e4741d08,c11519ec,3c9,0,...) at sched_idletd+0x1c8/frame 0xe4741ccc fork_exit(c0b0c390,0,e4741d08) at fork_exit+0x7f/frame 0xe4741cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4741cf4 --- trap 0, eip = 0, esp = 0xe4741d40, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc86fd310 sched_switch(c86fd310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe473d98c mi_switch(104,0,c115e489,1f1,0,...) at mi_switch+0x1c3/frame 0xe473d9c0 sleepq_switch(c86fd310,0,c115e489,1a7,c144f700,...) at sleepq_switch+0x1c5/frame 0xe473d9f4 sleepq_catch_signals(c115e489,15e,c86f5394,11d0,c86f52ec,...) at sleepq_catch_signals+0x3ce/frame 0xe473da38 sleepq_wait_sig(c86f52ec,6c,c11621ad,100,0,...) at sleepq_wait_sig+0x14/frame 0xe473da5c _sleep(c86f52ec,c86f5384,16c,c11621ad,0,...) at _sleep+0x372/frame 0xe473dab8 kern_wait6(c86fd310,7,0,0,e473db4c,...) at kern_wait6+0x47b/frame 0xe473db20 sys_wait4(c86fd310,e473dcc8,0,ab8,e473dccc,...) at sys_wait4+0x94/frame 0xe473dc40 syscall(e473dd08) at syscall+0x31b/frame 0xe473dcfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe473dcfc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x805fa4f, esp = 0xbfbfd87c, ebp = 0xbfbfd890 --- Tracing command audit pid 10 tid 100001 td 0xc86fd620 sched_switch(c86fd620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe473aba8 mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe473abdc sleepq_switch(c86fd620,0,c115e489,267,c175e698,...) at sleepq_switch+0x1c5/frame 0xe473ac10 sleepq_wait(c175e698,0,e473ac64,1,0,...) at sleepq_wait+0x48/frame 0xe473ac34 _cv_wait(c175e698,c175e624,c118b072,18d,0,...) at _cv_wait+0x263/frame 0xe473ac78 audit_worker(0,e473ad08,c11519ec,3c9,0,...) at audit_worker+0x94/frame 0xe473accc fork_exit(c0d15fa0,0,e473ad08) at fork_exit+0x7f/frame 0xe473acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe473acf4 --- trap 0, eip = 0, esp = 0xe473ad40, ebp = 0 --- Tracing command kernel pid 0 tid 105226 td 0xcea67930 sched_switch(cea67930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcac7bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcac7be0 sleepq_switch(cea67930,0,c115e489,267,cae93b80,...) at sleepq_switch+0x1c5/frame 0xfcac7c14 sleepq_wait(cae93b80,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcac7c38 _sleep(cae93b80,cae93b98,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcac7c94 taskqueue_thread_loop(c89869e0,fcac7d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcac7ccc fork_exit(c0b2e580,c89869e0,fcac7d08) at fork_exit+0x7f/frame 0xfcac7cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcac7cf4 --- trap 0, eip = 0, esp = 0xfcac7d40, ebp = 0 --- Tracing command kernel pid 0 tid 105225 td 0xcea67c40 sched_switch(cea67c40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcac4bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcac4be0 sleepq_switch(cea67c40,0,c115e489,267,cae93b80,...) at sleepq_switch+0x1c5/frame 0xfcac4c14 sleepq_wait(cae93b80,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcac4c38 _sleep(cae93b80,cae93b98,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcac4c94 taskqueue_thread_loop(c89869e0,fcac4d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcac4ccc fork_exit(c0b2e580,c89869e0,fcac4d08) at fork_exit+0x7f/frame 0xfcac4cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcac4cf4 --- trap 0, eip = 0, esp = 0xfcac4d40, ebp = 0 --- Tracing command kernel pid 0 tid 105224 td 0xcea68000 sched_switch(cea68000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcac1bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcac1be0 sleepq_switch(cea68000,0,c115e489,267,cae93c00,...) at sleepq_switch+0x1c5/frame 0xfcac1c14 sleepq_wait(cae93c00,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcac1c38 _sleep(cae93c00,cae93c18,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcac1c94 taskqueue_thread_loop(c8986a10,fcac1d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcac1ccc fork_exit(c0b2e580,c8986a10,fcac1d08) at fork_exit+0x7f/frame 0xfcac1cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcac1cf4 --- trap 0, eip = 0, esp = 0xfcac1d40, ebp = 0 --- Tracing command kernel pid 0 tid 104970 td 0xcea68310 sched_switch(cea68310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcabebac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcabebe0 sleepq_switch(cea68310,0,c115e489,267,cae93c00,...) at sleepq_switch+0x1c5/frame 0xfcabec14 sleepq_wait(cae93c00,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcabec38 _sleep(cae93c00,cae93c18,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcabec94 taskqueue_thread_loop(c8986a10,fcabed08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcabeccc fork_exit(c0b2e580,c8986a10,fcabed08) at fork_exit+0x7f/frame 0xfcabecf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcabecf4 --- trap 0, eip = 0, esp = 0xfcabed40, ebp = 0 --- Tracing command kernel pid 0 tid 105218 td 0xd2b08000 sched_switch(d2b08000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcb60bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcb60be0 sleepq_switch(d2b08000,0,c115e489,267,c94fb880,...) at sleepq_switch+0x1c5/frame 0xfcb60c14 sleepq_wait(c94fb880,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcb60c38 _sleep(c94fb880,c94fb898,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcb60c94 taskqueue_thread_loop(c8986f50,fcb60d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcb60ccc fork_exit(c0b2e580,c8986f50,fcb60d08) at fork_exit+0x7f/frame 0xfcb60cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcb60cf4 --- trap 0, eip = 0, esp = 0xfcb60d40, ebp = 0 --- Tracing command kernel pid 0 tid 105217 td 0xd2b08310 sched_switch(d2b08310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xfcb5dbac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xfcb5dbe0 sleepq_switch(d2b08310,0,c115e489,267,c94fb880,...) at sleepq_switch+0x1c5/frame 0xfcb5dc14 sleepq_wait(c94fb880,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xfcb5dc38 _sleep(c94fb880,c94fb898,0,c1143b4e,0,...) at _sleep+0x391/frame 0xfcb5dc94 taskqueue_thread_loop(c8986f50,fcb5dd08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xfcb5dccc fork_exit(c0b2e580,c8986f50,fcb5dd08) at fork_exit+0x7f/frame 0xfcb5dcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xfcb5dcf4 --- trap 0, eip = 0, esp = 0xfcb5dd40, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc8b94310 sched_switch(c8b94310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed97bbac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xed97bbe0 sleepq_switch(c8b94310,0,c115e489,267,c86e9a80,...) at sleepq_switch+0x1c5/frame 0xed97bc14 sleepq_wait(c86e9a80,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xed97bc38 _sleep(c86e9a80,c86e9a98,0,c1143b4e,0,...) at _sleep+0x391/frame 0xed97bc94 taskqueue_thread_loop(c1403320,ed97bd08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xed97bccc fork_exit(c0b2e580,c1403320,ed97bd08) at fork_exit+0x7f/frame 0xed97bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed97bcf4 --- trap 0, eip = 0, esp = 0xed97bd40, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc8936620 sched_switch(c8936620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed974b68 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xed974b9c sleepq_switch(c8936620,0,c115e489,28a,c1441ccc,...) at sleepq_switch+0x1c5/frame 0xed974bd0 sleepq_timedwait(c1441ccc,0,2,0,0,...) at sleepq_timedwait+0x48/frame 0xed974bf4 _sleep(c1441ccc,0,0,c1143b4e,fffffc88,...) at _sleep+0x35e/frame 0xed974c50 pause_sbt(c1143b4e,fffffc88,2,0,0,...) at pause_sbt+0xe9/frame 0xed974c8c deadlkres(0,ed974d08,c11519ec,3c9,0,...) at deadlkres+0x81/frame 0xed974ccc fork_exit(c0a8f100,0,ed974d08) at fork_exit+0x7f/frame 0xed974cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed974cf4 --- trap 0, eip = 0, esp = 0xed974d40, ebp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xc8936930 sched_switch(c8936930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xed971bc0 mi_switch(104,0,c115e489,1f1,c145de88,...) at mi_switch+0x1c3/frame 0xed971bf4 sleepq_switch(c8936930,0,c115e489,267,c86e9680,...) at sleepq_switch+0x1c5/frame 0xed971c28 sleepq_wait(c86e9680,0,c1158e9a,c1143b4e,0,...) at sleepq_wait+0x48/frame 0xed971c4c msleep_spin_sbt(c86e9680,c86e9698,c1143b4e,0,0,...) at msleep_spin_sbt+0x239/frame 0xed971c94 taskqueue_thread_loop(c174a500,ed971d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xed971ccc fork_exit(c0b2e580,c174a500,ed971d08) at fork_exit+0x7f/frame 0xed971cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xed971cf4 --- trap 0, eip = 0, esp = 0xed971d40, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc88b2930 sched_switch(c88b2930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe478bbac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe478bbe0 sleepq_switch(c88b2930,0,c115e489,267,c86e9700,...) at sleepq_switch+0x1c5/frame 0xe478bc14 sleepq_wait(c86e9700,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe478bc38 _sleep(c86e9700,c86e9718,0,c1143b4e,0,...) at _sleep+0x391/frame 0xe478bc94 taskqueue_thread_loop(c175e6d0,e478bd08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xe478bccc fork_exit(c0b2e580,c175e6d0,e478bd08) at fork_exit+0x7f/frame 0xe478bcf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe478bcf4 --- trap 0, eip = 0, esp = 0xe478bd40, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc86fd930 sched_switch(c86fd930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4788bc0 mi_switch(104,0,c115e489,1f1,c145dea0,...) at mi_switch+0x1c3/frame 0xe4788bf4 sleepq_switch(c86fd930,0,c115e489,267,c86e9780,...) at sleepq_switch+0x1c5/frame 0xe4788c28 sleepq_wait(c86e9780,0,c1158e9a,c1143b4e,0,...) at sleepq_wait+0x48/frame 0xe4788c4c msleep_spin_sbt(c86e9780,c86e9798,c1143b4e,0,0,...) at msleep_spin_sbt+0x239/frame 0xe4788c94 taskqueue_thread_loop(c174c044,e4788d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xe4788ccc fork_exit(c0b2e580,c174c044,e4788d08) at fork_exit+0x7f/frame 0xe4788cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4788cf4 --- trap 0, eip = 0, esp = 0xe4788d40, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc86fdc40 sched_switch(c86fdc40,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4785bc0 mi_switch(104,0,c115e489,1f1,c145dea0,...) at mi_switch+0x1c3/frame 0xe4785bf4 sleepq_switch(c86fdc40,0,c115e489,267,c86e9780,...) at sleepq_switch+0x1c5/frame 0xe4785c28 sleepq_wait(c86e9780,0,c1158e9a,c1143b4e,0,...) at sleepq_wait+0x48/frame 0xe4785c4c msleep_spin_sbt(c86e9780,c86e9798,c1143b4e,0,0,...) at msleep_spin_sbt+0x239/frame 0xe4785c94 taskqueue_thread_loop(c174c044,e4785d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xe4785ccc fork_exit(c0b2e580,c174c044,e4785d08) at fork_exit+0x7f/frame 0xe4785cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4785cf4 --- trap 0, eip = 0, esp = 0xe4785d40, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc87b1000 sched_switch(c87b1000,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4782bc0 mi_switch(104,0,c115e489,1f1,c145dea0,...) at mi_switch+0x1c3/frame 0xe4782bf4 sleepq_switch(c87b1000,0,c115e489,267,c86e9780,...) at sleepq_switch+0x1c5/frame 0xe4782c28 sleepq_wait(c86e9780,0,c1158e9a,c1143b4e,0,...) at sleepq_wait+0x48/frame 0xe4782c4c msleep_spin_sbt(c86e9780,c86e9798,c1143b4e,0,0,...) at msleep_spin_sbt+0x239/frame 0xe4782c94 taskqueue_thread_loop(c174c044,e4782d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0xcc/frame 0xe4782ccc fork_exit(c0b2e580,c174c044,e4782d08) at fork_exit+0x7f/frame 0xe4782cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4782cf4 --- trap 0, eip = 0, esp = 0xe4782d40, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc87b1620 sched_switch(c87b1620,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe477cbac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe477cbe0 sleepq_switch(c87b1620,0,c115e489,267,c86e9880,...) at sleepq_switch+0x1c5/frame 0xe477cc14 sleepq_wait(c86e9880,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe477cc38 _sleep(c86e9880,c86e9898,0,c1143b4e,0,...) at _sleep+0x391/frame 0xe477cc94 taskqueue_thread_loop(c175c028,e477cd08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xe477cccc fork_exit(c0b2e580,c175c028,e477cd08) at fork_exit+0x7f/frame 0xe477ccf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe477ccf4 --- trap 0, eip = 0, esp = 0xe477cd40, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc87b2310 sched_switch(c87b2310,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4770bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4770be0 sleepq_switch(c87b2310,0,c115e489,267,c86e9b00,...) at sleepq_switch+0x1c5/frame 0xe4770c14 sleepq_wait(c86e9b00,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4770c38 _sleep(c86e9b00,c86e9b18,0,c1143b4e,0,...) at _sleep+0x391/frame 0xe4770c94 taskqueue_thread_loop(c1759148,e4770d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xe4770ccc fork_exit(c0b2e580,c1759148,e4770d08) at fork_exit+0x7f/frame 0xe4770cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4770cf4 --- trap 0, eip = 0, esp = 0xe4770d40, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc87b2930 sched_switch(c87b2930,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xe4769bac mi_switch(104,0,c115e489,1f1,8,...) at mi_switch+0x1c3/frame 0xe4769be0 sleepq_switch(c87b2930,0,c115e489,267,c86e9b80,...) at sleepq_switch+0x1c5/frame 0xe4769c14 sleepq_wait(c86e9b80,0,c1143b4e,0,0,...) at sleepq_wait+0x48/frame 0xe4769c38 _sleep(c86e9b80,c86e9b98,0,c1143b4e,0,...) at _sleep+0x391/frame 0xe4769c94 taskqueue_thread_loop(c145d140,e4769d08,c11519ec,3c9,0,...) at taskqueue_thread_loop+0x129/frame 0xe4769ccc fork_exit(c0b2e580,c145d140,e4769d08) at fork_exit+0x7f/frame 0xe4769cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4769cf4 --- trap 0, eip = 0, esp = 0xe4769d40, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1758cb0 sched_switch(c1758cb0,0,104,1bb,c0ace17e,...) at sched_switch+0x4cf/frame 0xc1c20c28 mi_switch(104,0,c115e489,1f1,101,...) at mi_switch+0x1c3/frame 0xc1c20c5c sleepq_switch(c1758cb0,0,c115e489,28a,c17589b4,...) at sleepq_switch+0x1c5/frame 0xc1c20c90 sleepq_timedwait(c17589b4,54,9,0,0,...) at sleepq_timedwait+0x48/frame 0xc1c20cb4 _sleep(c17589b4,0,54,c1196eb6,fffff470,...) at _sleep+0x35e/frame 0xc1c20d10 swapper() at swapper+0x283/frame 0xc1c20d58 begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3431 MB Dumping 261 MB: 246 230 214 198 182 166 150 134 118 102 86 70 54 38 22 6 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 79400 [Switching to thread 88 (Thread 104108)]#0 sched_switch (td=0x0, newtd=, flags=0x104) at ../../../kern/sched_ule.c:1933 1933 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=0x0, newtd=, flags=0x104) at ../../../kern/sched_ule.c:1933 #1 0xc0aecb73 in mi_switch (flags=) at ../../../kern/kern_synch.c:493 #2 0xc0b2a475 in sleepq_switch (wchan=, pri=) at ../../../kern/subr_sleepqueue.c:537 #3 0xc0b2a258 in sleepq_wait (wchan=0xc1460800, pri=0x0) at ../../../kern/subr_sleepqueue.c:616 #4 0xc0a8f9d3 in _cv_wait (cvp=0xc1460800, lock=0xc1460780) at ../../../kern/kern_condvar.c:139 #5 0xc0b35251 in vmem_xalloc (vm=0xc1460780, size0=0x6bf000, align=, phase=0xda, nocross=, minaddr=0xc1fb7110, maxaddr=0xc1fb7100, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1204 #6 0xc0b34ff4 in vmem_alloc (vm=0xc1460780, size=0x6bf000, flags=0x2002, addrp=) at ../../../kern/subr_vmem.c:1090 #7 0xc0d8f42d in kmem_malloc (vmem=0xc1460780, size=0x6bf000, flags=0x102) at ../../../vm/vm_kern.c:314 #8 0xc0d876f0 in uma_large_malloc (size=, wait=0x102) at ../../../vm/uma_core.c:1103 #9 0xc0ac8bd2 in malloc (size=0x6bf000, mtp=, flags=) at ../../../kern/kern_malloc.c:519 #10 0xc0a9dae7 in fdgrowtable (fdp=0xc8c3e400, nfd=) at ../../../kern/kern_descrip.c:1530 #11 0xc0a99dba in do_dup (td=, flags=, old=, new=0x18028, retval=0x4c1e) at ../../../kern/kern_descrip.c:1492 #12 0xc0a99b54 in sys_dup2 (td=0xc8db4c40, uap=0xfaee9cc8) at ../../../kern/kern_descrip.c:351 #13 0xc0fea76b in syscall (frame=) at subr_syscall.c:133 #14 0xc0fd2891 in Xint0x80_syscall () at ../../../i386/i386/exception.s:269 #15 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 9 #9 0xc0ac8bd2 in malloc (size=0x6bf000, mtp=, flags=) at ../../../kern/kern_malloc.c:519 519 va = uma_large_malloc(size, flags); (kgdb) p/d size $1 = 7073792 (kgdb) f 10 #10 0xc0a9dae7 in fdgrowtable (fdp=0xc8c3e400, nfd=) at ../../../kern/kern_descrip.c:1530 1530 ntable = malloc(nnfiles * sizeof(ntable[0]) + sizeof(struct freetable), (kgdb) l 1525 * file entries themselves and the struct freetable we will use 1526 * when we decommission the table and place it on the freelist. 1527 * We place the struct freetable in the middle so we don't have 1528 * to worry about padding. 1529 */ 1530 ntable = malloc(nnfiles * sizeof(ntable[0]) + sizeof(struct freetable), 1531 M_FILEDESC, M_ZERO | M_WAITOK); 1532 /* copy the old data over and point at the new tables */ 1533 memcpy(ntable, otable, onfiles * sizeof(*otable)); 1534 fdp->fd_ofiles = ntable; (kgdb) info loc nnfiles = 0x2ff80 onfiles = 0x17fc0 otable = (struct filedescent *) 0xe192f000 omap = (u_long *) 0xceef7000 ntable = nmap = (kgdb) (kgdb) p maxfiles $2 = 0x1ace9 (kgdb) p/d maxfiles $3 = 109801 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 262885) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -99,7 +99,8 @@ struct buf_ops buf_ops_bio = { struct buf *buf; /* buffer header pool */ caddr_t unmapped_buf; -static struct proc *bufdaemonproc; +/* Used below and for softdep flushing threads in ufs/ffs/ffs_softdep.c */ +struct proc *bufdaemonproc; static int inmem(struct vnode *vp, daddr_t blkno); static void vm_hold_free_pages(struct buf *bp, int newbsize); Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 262885) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -908,9 +908,9 @@ static void add_to_worklist(struct worklist *, int static void wake_worklist(struct worklist *); static void wait_worklist(struct worklist *, char *); static void remove_from_worklist(struct worklist *); -static void softdep_flush(void); +static void softdep_flush(void *); static void softdep_flushjournal(struct mount *); -static int softdep_speedup(void); +static int softdep_speedup(struct ufsmount *); static void worklist_speedup(struct mount *); static int journal_mount(struct mount *, struct fs *, struct ucred *); static void journal_unmount(struct ufsmount *); @@ -962,18 +962,21 @@ static int softdep_count_dependencies(struct buf * /* * Global lock over all of soft updates. */ -static struct rwlock lk; -RW_SYSINIT(softdep_lock, &lk, "Softdep Lock"); +static struct mtx lk; +MTX_SYSINIT(softdep_lock, &lk, "Global Softdep Lock", MTX_DEF); +#define ACQUIRE_GBLLOCK(lk) mtx_lock(lk) +#define FREE_GBLLOCK(lk) mtx_unlock(lk) +#define GBLLOCK_OWNED(lk) mtx_assert((lk), MA_OWNED) + /* - * Allow per-filesystem soft-updates locking. - * For now all use the same global lock defined above. + * Per-filesystem soft-updates locking. */ -#define LOCK_PTR(ump) ((ump)->um_softdep->sd_fslock) -#define TRY_ACQUIRE_LOCK(ump) rw_try_wlock((ump)->um_softdep->sd_fslock) -#define ACQUIRE_LOCK(ump) rw_wlock((ump)->um_softdep->sd_fslock) -#define FREE_LOCK(ump) rw_wunlock((ump)->um_softdep->sd_fslock) -#define LOCK_OWNED(ump) rw_assert((ump)->um_softdep->sd_fslock, \ +#define LOCK_PTR(ump) (&(ump)->um_softdep->sd_fslock) +#define TRY_ACQUIRE_LOCK(ump) rw_try_wlock(&(ump)->um_softdep->sd_fslock) +#define ACQUIRE_LOCK(ump) rw_wlock(&(ump)->um_softdep->sd_fslock) +#define FREE_LOCK(ump) rw_wunlock(&(ump)->um_softdep->sd_fslock) +#define LOCK_OWNED(ump) rw_assert(&(ump)->um_softdep->sd_fslock, \ RA_WLOCKED) #define BUF_AREC(bp) lockallowrecurse(&(bp)->b_lock) @@ -1178,7 +1181,7 @@ workitem_free(item, type) KASSERT(ump->softdep_curdeps[item->wk_type] > 0, ("workitem_free: %s: softdep_curdeps[%s] going negative", ump->um_fs->fs_fsmnt, TYPENAME(item->wk_type))); - dep_current[item->wk_type]--; + atomic_subtract_long(&dep_current[item->wk_type], 1); ump->softdep_curdeps[item->wk_type] -= 1; free(item, DtoM(type)); } @@ -1196,11 +1199,13 @@ workitem_alloc(item, type, mp) item->wk_state = 0; ump = VFSTOUFS(mp); - ACQUIRE_LOCK(ump); + ACQUIRE_GBLLOCK(&lk); dep_current[type]++; if (dep_current[type] > dep_highuse[type]) dep_highuse[type] = dep_current[type]; dep_total[type]++; + FREE_GBLLOCK(&lk); + ACQUIRE_LOCK(ump); ump->softdep_curdeps[type] += 1; ump->softdep_deps++; ump->softdep_accdeps++; @@ -1224,11 +1229,13 @@ workitem_reassign(item, newtype) KASSERT(dep_current[item->wk_type] > 0, ("workitem_reassign: %s: dep_current[%s] going negative", VFSTOUFS(item->wk_mp)->um_fs->fs_fsmnt, TYPENAME(item->wk_type))); + ACQUIRE_GBLLOCK(&lk); + dep_current[newtype]++; dep_current[item->wk_type]--; - dep_current[newtype]++; if (dep_current[newtype] > dep_highuse[newtype]) dep_highuse[newtype] = dep_current[newtype]; dep_total[newtype]++; + FREE_GBLLOCK(&lk); item->wk_type = newtype; } @@ -1236,13 +1243,10 @@ workitem_reassign(item, newtype) * Workitem queue management */ static int max_softdeps; /* maximum number of structs before slowdown */ -static int maxindirdeps = 50; /* max number of indirdeps before slowdown */ static int tickdelay = 2; /* number of ticks to pause during slowdown */ static int proc_waiting; /* tracks whether we have a timeout posted */ static int *stat_countp; /* statistic to count in proc_waiting timeout */ static struct callout softdep_callout; -static struct mount *req_pending; -#define ALLCLEAN ((struct mount *)-1) static int req_clear_inodedeps; /* syncer process flush some inodedeps */ static int req_clear_remove; /* syncer process flush some freeblks */ static int softdep_flushcache = 0; /* Should we do BIO_FLUSH? */ @@ -1250,7 +1254,7 @@ static int softdep_flushcache = 0; /* Should we do /* * runtime statistics */ -static int stat_softdep_mounts; /* number of softdep mounted filesystems */ +static int stat_flush_threads; /* number of softdep flushing threads */ static int stat_worklist_push; /* number of worklist cleanups */ static int stat_blk_limit_push; /* number of times block limit neared */ static int stat_ino_limit_push; /* number of times inode limit neared */ @@ -1280,10 +1284,8 @@ SYSCTL_INT(_debug_softdep, OID_AUTO, max_softdeps, &max_softdeps, 0, ""); SYSCTL_INT(_debug_softdep, OID_AUTO, tickdelay, CTLFLAG_RW, &tickdelay, 0, ""); -SYSCTL_INT(_debug_softdep, OID_AUTO, maxindirdeps, CTLFLAG_RW, - &maxindirdeps, 0, ""); -SYSCTL_INT(_debug_softdep, OID_AUTO, softdep_mounts, CTLFLAG_RD, - &stat_softdep_mounts, 0, ""); +SYSCTL_INT(_debug_softdep, OID_AUTO, flush_threads, CTLFLAG_RD, + &stat_flush_threads, 0, ""); SYSCTL_INT(_debug_softdep, OID_AUTO, worklist_push, CTLFLAG_RW, &stat_worklist_push, 0,""); SYSCTL_INT(_debug_softdep, OID_AUTO, blk_limit_push, CTLFLAG_RW, @@ -1341,53 +1343,67 @@ SYSCTL_DECL(_vfs_ffs); static int compute_summary_at_mount = 0; SYSCTL_INT(_vfs_ffs, OID_AUTO, compute_summary_at_mount, CTLFLAG_RW, &compute_summary_at_mount, 0, "Recompute summary at mount"); -static struct proc *softdepproc; -static struct kproc_desc softdep_kp = { - "softdepflush", - softdep_flush, - &softdepproc -}; -SYSINIT(sdproc, SI_SUB_KTHREAD_UPDATE, SI_ORDER_ANY, kproc_start, - &softdep_kp); +static int print_threads = 1; +SYSCTL_INT(_debug_softdep, OID_AUTO, print_threads, CTLFLAG_RW, + &print_threads, 0, "Notify flusher thread start/stop"); +/* List of all filesystems mounted with soft updates */ +static TAILQ_HEAD(, mount_softdeps) softdepmounts; + +/* + * This function cleans the worklist for a filesystem. + * Each filesystem running with soft dependencies gets its own + * thread to run in this function. The thread is started up in + * softdep_mount and shutdown in softdep_unmount. They show up + * as part of the kernel "bufdaemon" process whose process + * entry is available in bufdaemonproc. + */ +static int searchfailed; +extern struct proc *bufdaemonproc; static void -softdep_flush(void) +softdep_flush(addr) + void *addr; { - struct mount *nmp; struct mount *mp; + struct thread *td; struct ufsmount *ump; - struct thread *td; - int remaining; - int progress; td = curthread; td->td_pflags |= TDP_NORUNNINGBUF; - + mp = (struct mount *)addr; + ump = VFSTOUFS(mp); + atomic_add_int(&stat_flush_threads, 1); + if (print_threads) { + if (stat_flush_threads == 1) + printf("Running %s at pid %d\n", bufdaemonproc->p_comm, + bufdaemonproc->p_pid); + printf("Start thread %s\n", td->td_name); + } for (;;) { - kproc_suspend_check(softdepproc); - remaining = progress = 0; - mtx_lock(&mountlist_mtx); - for (mp = TAILQ_FIRST(&mountlist); mp != NULL; mp = nmp) { - nmp = TAILQ_NEXT(mp, mnt_list); - if (MOUNTEDSOFTDEP(mp) == 0) - continue; - if (vfs_busy(mp, MBF_NOWAIT | MBF_MNTLSTLOCK)) - continue; - ump = VFSTOUFS(mp); - progress += softdep_process_worklist(mp, 0); - remaining += ump->softdep_on_worklist; - mtx_lock(&mountlist_mtx); - nmp = TAILQ_NEXT(mp, mnt_list); - vfs_unbusy(mp); + while (softdep_process_worklist(mp, 0) > 0 || + (MOUNTEDSUJ(mp) && + VFSTOUFS(mp)->softdep_jblocks->jb_suspended)) + kthread_suspend_check(); + ACQUIRE_LOCK(ump); + if ((ump->softdep_flags & FLUSH_CLEANUP) == 0) + msleep(&ump->softdep_flushtd, LOCK_PTR(ump), PVM, + "sdflush", hz / 2); + ump->softdep_flags &= ~FLUSH_CLEANUP; + /* + * Check to see if we are done and need to exit. + */ + if ((ump->softdep_flags & FLUSH_EXIT) == 0) { + FREE_LOCK(ump); + continue; } - mtx_unlock(&mountlist_mtx); - if (remaining && progress) - continue; - rw_wlock(&lk); - if (req_pending == NULL) - msleep(&req_pending, &lk, PVM, "sdflush", hz); - req_pending = NULL; - rw_wunlock(&lk); + ump->softdep_flags &= ~FLUSH_EXIT; + FREE_LOCK(ump); + wakeup(&ump->softdep_flags); + if (print_threads) + printf("Stop thread %s: searchfailed %d, did cleanups %d\n", td->td_name, searchfailed, ump->um_softdep->sd_cleanups); + atomic_subtract_int(&stat_flush_threads, 1); + kthread_exit(); + panic("kthread_exit failed\n"); } } @@ -1395,19 +1411,70 @@ static void worklist_speedup(mp) struct mount *mp; { - rw_assert(&lk, RA_WLOCKED); - if (req_pending == 0) { - req_pending = mp; - wakeup(&req_pending); + struct ufsmount *ump; + + ump = VFSTOUFS(mp); + LOCK_OWNED(ump); + if ((ump->softdep_flags & (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + ump->softdep_flags |= FLUSH_CLEANUP; + if (ump->softdep_flushtd->td_wchan == &ump->softdep_flushtd) + wakeup(&ump->softdep_flushtd); } } static int -softdep_speedup(void) +softdep_speedup(ump) + struct ufsmount *ump; { + struct ufsmount *altump; + struct mount_softdeps *sdp; - worklist_speedup(ALLCLEAN); + LOCK_OWNED(ump); + worklist_speedup(ump->um_mountp); bd_speedup(); + /* + * If we have global shortages, then we need other + * filesystems to help with the cleanup. Here we wakeup a + * flusher thread for a filesystem that is over its fair + * share of resources. + */ + if (req_clear_inodedeps || req_clear_remove) { + ACQUIRE_GBLLOCK(&lk); + TAILQ_FOREACH(sdp, &softdepmounts, sd_next) { + if ((altump = sdp->sd_ump) == ump) + continue; + if (((req_clear_inodedeps && + altump->softdep_curdeps[D_INODEDEP] > + max_softdeps / stat_flush_threads) || + (req_clear_remove && + altump->softdep_curdeps[D_DIRREM] > + (max_softdeps / 2) / stat_flush_threads)) && + TRY_ACQUIRE_LOCK(altump)) + break; + } + if (sdp == NULL) { + searchfailed++; + FREE_GBLLOCK(&lk); + } else { + /* + * Move to the end of the list so we pick a + * different one on out next try. + */ + TAILQ_REMOVE(&softdepmounts, sdp, sd_next); + TAILQ_INSERT_TAIL(&softdepmounts, sdp, sd_next); + FREE_GBLLOCK(&lk); + if ((altump->softdep_flags & + (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + altump->softdep_flags |= FLUSH_CLEANUP; + altump->um_softdep->sd_cleanups++; + if (altump->softdep_flushtd->td_wchan == + &altump->softdep_flushtd) { + wakeup(&altump->softdep_flushtd); + } + } + FREE_LOCK(altump); + } + } return (speedup_syncer()); } @@ -2123,9 +2190,14 @@ inodedep_lookup(mp, inum, flags, inodedeppp) if ((flags & DEPALLOC) == 0) return (0); /* - * If we are over our limit, try to improve the situation. + * If the system is over its limit and our filesystem is + * responsible for more than our share of that usage and + * we are not in a rush, request some inodedep cleanup. */ - if (dep_current[D_INODEDEP] > max_softdeps && (flags & NODELAY) == 0) + while (dep_current[D_INODEDEP] > max_softdeps && + (flags & NODELAY) == 0 && + ump->softdep_curdeps[D_INODEDEP] > + max_softdeps / stat_flush_threads) request_cleanup(mp, FLUSH_INODES); FREE_LOCK(ump); inodedep = malloc(sizeof(struct inodedep), @@ -2317,6 +2389,7 @@ void softdep_initialize() { + TAILQ_INIT(&softdepmounts); max_softdeps = desiredvnodes * 4; /* initialise bioops hack */ @@ -2375,7 +2448,9 @@ softdep_mount(devvp, mp, fs, cred) ump = VFSTOUFS(mp); ump->um_softdep = sdp; MNT_IUNLOCK(mp); - LOCK_PTR(ump) = &lk; + rw_init(LOCK_PTR(ump), "Per-Filesystem Softdep Lock"); + TAILQ_INSERT_TAIL(&softdepmounts, sdp, sd_next); + sdp->sd_ump = ump; LIST_INIT(&ump->softdep_workitem_pending); LIST_INIT(&ump->softdep_journal_pending); TAILQ_INIT(&ump->softdep_unlinked); @@ -2406,8 +2481,13 @@ softdep_mount(devvp, mp, fs, cred) softdep_unmount(mp); return (error); } - atomic_add_int(&stat_softdep_mounts, 1); /* + * Start our flushing thread in the bufdaemon process. + */ + kproc_kthread_add(&softdep_flush, mp, &bufdaemonproc, + &ump->softdep_flushtd, 0, 0, "softdepflush", "%s worker", + mp->mnt_stat.f_mntonname); + /* * When doing soft updates, the counters in the * superblock may have gotten out of sync. Recomputation * can take a long time and can be deferred for background @@ -2462,7 +2542,24 @@ softdep_unmount(mp) MNT_IUNLOCK(mp); journal_unmount(ump); } - atomic_subtract_int(&stat_softdep_mounts, 1); + /* + * Shut down our flushing thread. Check for NULL is if + * softdep_mount errors out before the thread has been created. + */ + if (ump->softdep_flushtd != NULL) { + ACQUIRE_LOCK(ump); + ump->softdep_flags |= FLUSH_EXIT; + wakeup(&ump->softdep_flushtd); + msleep(&ump->softdep_flags, LOCK_PTR(ump), PVM | PDROP, + "sdwait", 0); + KASSERT((ump->softdep_flags & FLUSH_EXIT) == 0, + ("Thread shutdown failed")); + } + /* + * Free up our resources. + */ + rw_destroy(LOCK_PTR(ump)); + TAILQ_REMOVE(&softdepmounts, ump->um_softdep, sd_next); hashdestroy(ump->pagedep_hashtbl, M_PAGEDEP, ump->pagedep_hash_size); hashdestroy(ump->inodedep_hashtbl, M_INODEDEP, ump->inodedep_hash_size); hashdestroy(ump->newblk_hashtbl, M_NEWBLK, ump->newblk_hash_size); @@ -2785,7 +2882,7 @@ journal_space(ump, thresh) */ limit = (max_softdeps / 10) * 9; if (dep_current[D_INODEDEP] > limit && - ump->softdep_curdeps[D_INODEDEP] > limit / stat_softdep_mounts) + ump->softdep_curdeps[D_INODEDEP] > limit / stat_flush_threads) return (0); if (thresh) thresh = jblocks->jb_min; @@ -2810,7 +2907,7 @@ journal_suspend(ump) if ((mp->mnt_kern_flag & MNTK_SUSPEND) == 0) { stat_journal_min++; mp->mnt_kern_flag |= MNTK_SUSPEND; - mp->mnt_susp_owner = FIRST_THREAD_IN_PROC(softdepproc); + mp->mnt_susp_owner = ump->softdep_flushtd; } jblocks->jb_suspended = 1; MNT_IUNLOCK(mp); @@ -2885,7 +2982,7 @@ softdep_prealloc(vp, waitok) process_removes(vp); process_truncates(vp); if (journal_space(ump, 0) == 0) { - softdep_speedup(); + softdep_speedup(ump); if (journal_space(ump, 1) == 0) journal_suspend(ump); } @@ -2929,10 +3026,10 @@ softdep_prelink(dvp, vp) } process_removes(dvp); process_truncates(dvp); - softdep_speedup(); + softdep_speedup(ump); process_worklist_item(UFSTOVFS(ump), 2, LK_NOWAIT); if (journal_space(ump, 0) == 0) { - softdep_speedup(); + softdep_speedup(ump); if (journal_space(ump, 1) == 0) journal_suspend(ump); } @@ -3254,7 +3351,7 @@ softdep_process_journal(mp, needwk, flags) if (flags != MNT_WAIT) break; printf("softdep: Out of journal space!\n"); - softdep_speedup(); + softdep_speedup(ump); msleep(jblocks, LOCK_PTR(ump), PRIBIO, "jblocks", hz); } FREE_LOCK(ump); @@ -3944,7 +4041,7 @@ free_freedep(freedep) /* * Allocate a new freework structure that may be a level in an indirect * when parent is not NULL or a top level block when it is. The top level - * freework structures are allocated without the soft updates lock held + * freework structures are allocated without the per-filesystem lock held * and before the freeblks is visible outside of softdep_setup_freeblocks(). */ static struct freework * @@ -4013,7 +4110,7 @@ cancel_jfreeblk(freeblks, blkno) /* * Allocate a new jfreeblk to journal top level block pointer when truncating - * a file. The caller must add this to the worklist when the soft updates + * a file. The caller must add this to the worklist when the per-filesystem * lock is held. */ static struct jfreeblk * @@ -7393,7 +7490,7 @@ softdep_freefile(pvp, ino, mode) clear_unlinked_inodedep(inodedep); /* * Re-acquire inodedep as we've dropped the - * soft updates lock in clear_unlinked_inodedep(). + * per-filesystem lock in clear_unlinked_inodedep(). */ inodedep_lookup(pvp->v_mount, ino, 0, &inodedep); } @@ -7939,10 +8036,8 @@ indir_trunc(freework, dbn, lbn) * If we're goingaway, free the indirdep. Otherwise it will * linger until the write completes. */ - if (goingaway) { + if (goingaway) free_indirdep(indirdep); - ump->softdep_numindirdeps -= 1; - } } FREE_LOCK(ump); /* Initialize pointers depending on block size. */ @@ -8114,7 +8209,7 @@ cancel_allocindir(aip, bp, freeblks, trunc) * Create the mkdir dependencies for . and .. in a new directory. Link them * in to a newdirblk so any subsequent additions are tracked properly. The * caller is responsible for adding the mkdir1 dependency to the journal - * and updating id_mkdiradd. This function returns with the soft updates + * and updating id_mkdiradd. This function returns with the per-filesystem * lock held. */ static struct mkdir * @@ -8932,12 +9027,16 @@ newdirrem(bp, dp, ip, isrmdir, prevdirremp) panic("newdirrem: whiteout"); dvp = ITOV(dp); /* - * If we are over our limit, try to improve the situation. + * If the system is over its limit and our filesystem is + * responsible for more than our share of that usage and + * we are not a snapshot, request some inodedep cleanup. * Limiting the number of dirrem structures will also limit * the number of freefile and freeblks structures. */ ACQUIRE_LOCK(ip->i_ump); - if (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2) + while (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2 && + ip->i_ump->softdep_curdeps[D_DIRREM] > + (max_softdeps / 2) / stat_flush_threads) (void) request_cleanup(ITOV(dp)->v_mount, FLUSH_BLOCKS); FREE_LOCK(ip->i_ump); dirrem = malloc(sizeof(struct dirrem), @@ -9888,7 +9987,7 @@ initiate_write_filepage(pagedep, bp) * Wait for all journal remove dependencies to hit the disk. * We can not allow any potentially conflicting directory adds * to be visible before removes and rollback is too difficult. - * The soft updates lock may be dropped and re-acquired, however + * The per-filesystem lock may be dropped and re-acquired, however * we hold the buf locked so the dependency can not go away. */ LIST_FOREACH(dirrem, &pagedep->pd_dirremhd, dm_next) @@ -10352,7 +10451,6 @@ cancel_indirdep(indirdep, bp, freeblks) LIST_REMOVE(indirdep, ir_next); } indirdep->ir_state |= GOINGAWAY; - VFSTOUFS(indirdep->ir_list.wk_mp)->softdep_numindirdeps += 1; /* * Pass in bp for blocks still have journal writes * pending so we can cancel them on their own. @@ -10779,7 +10877,7 @@ softdep_disk_write_complete(bp) ACQUIRE_LOCK(ump); while ((wk = LIST_FIRST(&bp->b_dep)) != NULL) { WORKLIST_REMOVE(wk); - dep_write[wk->wk_type]++; + atomic_add_long(&dep_write[wk->wk_type], 1); if (wk == owk) panic("duplicate worklist: %p\n", wk); owk = wk; @@ -11462,7 +11560,7 @@ diradd_inode_written(dap, inodedep) /* * Returns true if the bmsafemap will have rollbacks when written. Must only - * be called with the soft updates lock and the buf lock on the cg held. + * be called with the per-filesystem lock and the buf lock on the cg held. */ static int bmsafemap_backgroundwrite(bmsafemap, bp) @@ -12886,18 +12984,42 @@ softdep_slowdown(vp) if (journal_space(ump, 0) == 0) jlow = 1; } + /* + * If the system is under its limits and our filesystem is + * not responsible for more than our share of the usage and + * we are not low on journal space, then no need to slow down. + */ max_softdeps_hard = max_softdeps * 11 / 10; if (dep_current[D_DIRREM] < max_softdeps_hard / 2 && dep_current[D_INODEDEP] < max_softdeps_hard && - VFSTOUFS(vp->v_mount)->softdep_numindirdeps < maxindirdeps && - dep_current[D_FREEBLKS] < max_softdeps_hard && jlow == 0) { + dep_current[D_INDIRDEP] < max_softdeps_hard / 1000 && + dep_current[D_FREEBLKS] < max_softdeps_hard && jlow == 0 && + ump->softdep_curdeps[D_DIRREM] < + (max_softdeps_hard / 2) / stat_flush_threads && + ump->softdep_curdeps[D_INODEDEP] < + max_softdeps_hard / stat_flush_threads && + ump->softdep_curdeps[D_INDIRDEP] < + (max_softdeps_hard / 1000) / stat_flush_threads && + ump->softdep_curdeps[D_FREEBLKS] < + max_softdeps_hard / stat_flush_threads) { FREE_LOCK(ump); return (0); } - if (VFSTOUFS(vp->v_mount)->softdep_numindirdeps >= maxindirdeps || jlow) - softdep_speedup(); + /* + * If the journal is low or our filesystem is over its limit + * then speedup the cleanup. + */ + if (ump->softdep_curdeps[D_INDIRDEP] < + (max_softdeps_hard / 1000) / stat_flush_threads || jlow) + softdep_speedup(ump); stat_sync_limit_hit += 1; FREE_LOCK(ump); + /* + * We only slow down the rate at which new dependencies are + * generated if we are not using journaling. With journaling, + * the cleanup should always be sufficient to keep things + * under control. + */ if (DOINGSUJ(vp)) return (0); return (1); @@ -12955,13 +13077,12 @@ softdep_request_cleanup(fs, vp, cred, resource) return (0); } /* - * If we are in need of resources, consider pausing for - * tickdelay to give ourselves some breathing room. + * If we are in need of resources, start by cleaning up + * any block removals associated with our inode. */ ACQUIRE_LOCK(ump); process_removes(vp); process_truncates(vp); - request_cleanup(UFSTOVFS(ump), resource); FREE_LOCK(ump); /* * Now clean up at least as many resources as we will need. @@ -13094,7 +13215,7 @@ request_cleanup(mp, resource) * Next, we attempt to speed up the syncer process. If that * is successful, then we allow the process to continue. */ - if (softdep_speedup() && + if (softdep_speedup(ump) && resource != FLUSH_BLOCKS_WAIT && resource != FLUSH_INODES_WAIT) return(0); @@ -13112,15 +13233,19 @@ request_cleanup(mp, resource) case FLUSH_INODES: case FLUSH_INODES_WAIT: + ACQUIRE_GBLLOCK(&lk); stat_ino_limit_push += 1; req_clear_inodedeps += 1; + FREE_GBLLOCK(&lk); stat_countp = &stat_ino_limit_hit; break; case FLUSH_BLOCKS: case FLUSH_BLOCKS_WAIT: + ACQUIRE_GBLLOCK(&lk); stat_blk_limit_push += 1; req_clear_remove += 1; + FREE_GBLLOCK(&lk); stat_countp = &stat_blk_limit_hit; break; @@ -13131,6 +13256,8 @@ request_cleanup(mp, resource) * Hopefully the syncer daemon will catch up and awaken us. * We wait at most tickdelay before proceeding in any case. */ + ACQUIRE_GBLLOCK(&lk); + FREE_LOCK(ump); proc_waiting += 1; if (callout_pending(&softdep_callout) == FALSE) callout_reset(&softdep_callout, tickdelay > 2 ? tickdelay : 2, @@ -13138,6 +13265,8 @@ request_cleanup(mp, resource) msleep((caddr_t)&proc_waiting, &lk, PPAUSE, "softupdate", 0); proc_waiting -= 1; + FREE_GBLLOCK(&lk); + ACQUIRE_LOCK(ump); return (1); } @@ -13151,16 +13280,13 @@ pause_timer(arg) void *arg; { - rw_assert(&lk, RA_WLOCKED); + GBLLOCK_OWNED(&lk); /* * The callout_ API has acquired mtx and will hold it around this * function call. */ - *stat_countp += 1; - wakeup_one(&proc_waiting); - if (proc_waiting > 0) - callout_reset(&softdep_callout, tickdelay > 2 ? tickdelay : 2, - pause_timer, 0); + *stat_countp += proc_waiting; + wakeup(&proc_waiting); } /* @@ -13171,7 +13297,6 @@ check_clear_deps(mp) struct mount *mp; { - rw_assert(&lk, RA_WLOCKED); /* * If we are suspended, it may be because of our using * too many inodedeps, so help clear them out. @@ -13181,16 +13306,22 @@ check_clear_deps(mp) /* * General requests for cleanup of backed up dependencies */ + ACQUIRE_GBLLOCK(&lk); if (req_clear_inodedeps) { req_clear_inodedeps -= 1; + FREE_GBLLOCK(&lk); clear_inodedeps(mp); - wakeup_one(&proc_waiting); + ACQUIRE_GBLLOCK(&lk); + wakeup(&proc_waiting); } if (req_clear_remove) { req_clear_remove -= 1; + FREE_GBLLOCK(&lk); clear_remove(mp); - wakeup_one(&proc_waiting); + ACQUIRE_GBLLOCK(&lk); + wakeup(&proc_waiting); } + FREE_GBLLOCK(&lk); } /* Index: /usr/src/sys/ufs/ffs/softdep.h =================================================================== --- /usr/src/sys/ufs/ffs/softdep.h (revision 262885) +++ /usr/src/sys/ufs/ffs/softdep.h (working copy) @@ -1025,7 +1025,7 @@ TAILQ_HEAD(indir_hashhead, freework); * Allocated at mount and freed at unmount. */ struct mount_softdeps { - struct rwlock *sd_fslock; /* softdep lock */ + struct rwlock sd_fslock; /* softdep lock */ struct workhead sd_workitem_pending; /* softdep work queue */ struct worklist *sd_worklist_tail; /* Tail pointer for above */ struct workhead sd_journal_pending; /* journal work queue */ @@ -1046,15 +1046,24 @@ struct mount_softdeps { u_long sd_bmhashsize; /* bmsafemap hash table size-1*/ struct indir_hashhead *sd_indirhash; /* indir hash table */ u_long sd_indirhashsize; /* indir hash table size-1 */ - long sd_numindirdeps; /* outstanding indirdeps */ int sd_on_journal; /* Items on the journal list */ int sd_on_worklist; /* Items on the worklist */ int sd_deps; /* Total dependency count */ int sd_accdeps; /* accumulated dep count */ int sd_req; /* Wakeup when deps hits 0. */ + int sd_flags; /* comm with flushing thread */ + int sd_cleanups; /* Calls to cleanup */ + struct thread *sd_flushtd; /* thread handling flushing */ + TAILQ_ENTRY(mount_softdeps) sd_next; /* List of softdep filesystem */ + struct ufsmount *sd_ump; /* our ufsmount structure */ u_long sd_curdeps[D_LAST + 1]; /* count of current deps */ }; /* + * Flags for communicating with the syncer thread. + */ +#define FLUSH_EXIT 0x0001 /* time to exit */ +#define FLUSH_CLEANUP 0x0002 /* need to clear out softdep structures */ +/* * Keep the old names from when these were in the ufsmount structure. */ #define softdep_workitem_pending um_softdep->sd_workitem_pending @@ -1077,10 +1086,11 @@ struct mount_softdeps { #define bmsafemap_hash_size um_softdep->sd_bmhashsize #define indir_hashtbl um_softdep->sd_indirhash #define indir_hash_size um_softdep->sd_indirhashsize -#define softdep_numindirdeps um_softdep->sd_numindirdeps #define softdep_on_journal um_softdep->sd_on_journal #define softdep_on_worklist um_softdep->sd_on_worklist #define softdep_deps um_softdep->sd_deps #define softdep_accdeps um_softdep->sd_accdeps #define softdep_req um_softdep->sd_req +#define softdep_flags um_softdep->sd_flags +#define softdep_flushtd um_softdep->sd_flushtd #define softdep_curdeps um_softdep->sd_curdeps