GENERIC HEAD from 2017-05-30 12:00:56 UTC, r319211M, vmcore.1018 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #1 r319211M: Tue May 30 15:54:02 CEST 2017 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3525517312 (3362 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... uhub0: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2097312 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 617805 free (2757 frags, 76881 blocks, 0.3% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 859285 free (10653 frags, 106079 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 55549934 free (37934 frags, 6939000 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1704401 free (364465 frags, 167492 blocks, 3.6% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 3392425 free (260985 frags, 391430 blocks, 1.0% fragmentation) Mounting local file systems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc49 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy:uhub1: 10 ports with 10 removable, self powered . re0: link state changed to DOWN ugen2.2: at usbus2 Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d hwaddr 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. uslcom0 on uhub4 uslcom0: on usbus2 add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. Starting ddb. No core dumps found. Additional ABI support: linux. lock order reversal: 1st 0xe49208e0 bufwait (bufwait) @ kern/vfs_bio.c:3509 2nd 0xc97eb400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:281 stack backtrace: #0 0xc0cbd6d1 at witness_debugger+0x81 #1 0xc0cbd5f1 at witness_checkorder+0xd11 #2 0xc0c6192a at _sx_xlock+0x5a #3 0xc0f5f343 at ufsdirhash_add+0x53 #4 0xc0f63dba at ufs_direnter+0x47a #5 0xc0f6d084 at ufs_mkdir+0x8b4 #6 0xc12400a9 at VOP_MKDIR_APV+0xf9 #7 0xc0d365fe at kern_mkdirat+0x21e #8 0xc0d363d1 at sys_mkdir+0x31 #9 0xc120ae5c at syscall+0x75c #10 0xc11f683e at Xint0x80_syscall+0x2e Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:lock order reversal: 1st 0xca0a519c ufs (ufs) @ kern/vfs_lookup.c:669 2nd 0xe4927960 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:280 3rd 0xca083034 ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xc0cbd6d1 at witness_debugger+0x81 #1 0xc0cbd5f1 at witness_checkorder+0xd11 #2 0xc0c2ca47 at __lockmgr_args+0x7a7 #3 0xc0f596e7 at ffs_lock+0xa7 #4 0xc1240e40 at VOP_LOCK1_APV+0xe0 #5 0xc0d39b33 at _vn_lock+0x63 #6 0xc0d26222 at vget+0x92 #7 0xc0d1771d at vfs_hash_get+0xcd #8 0xc0f548b4 at ffs_vgetf+0x44 #9 0xc0f4a8ba at softdep_sync_buf+0xbba #10 0xc0f5a50f at ffs_syncvnode+0x2ef #11 0xc0f2d904 at ffs_truncate+0x974 #12 0xc0f63f62 at ufs_direnter+0x622 #13 0xc0f6debf at ufs_makeinode+0x5af #14 0xc0f69858 at ufs_create+0x38 #15 0xc123da19 at VOP_CREATE_APV+0xf9 #16 0xc0d39361 at vn_open_cred+0x271 #17 0xc0d390dd at vn_open+0x3d . Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Starting smartd. May 30 15:58:06 x4 smartd[651]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, May 30 15:58:06 x4 smartd[651]: see the following Seagate web pages: May 30 15:58:06 x4 smartd[651]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en May 30 15:58:06 x4 smartd[651]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc96bf46c ufs (ufs) @ kern/vfs_mount.c:1271 2nd 0xc9835304 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1908 stack backtrace: #0 0xc0cbd6d1 at witness_debugger+0x81 #1 0xc0cbd5f1 at witness_checkorder+0xd11 #2 0xc0c2c1cf at lockmgr_lock_fast_path+0x1df #3 0xc0d137ab at vop_stdlock+0x3b #4 0xc1240e40 at VOP_LOCK1_APV+0xe0 #5 0xc0d39b33 at _vn_lock+0x63 #6 0xc0f37426 at softdep_flushworklist+0x66 #7 0xc0f579e2 at ffs_sync+0xd2 #8 0xc0d3b335 at vfs_write_suspend+0x1c5 #9 0xc0d3b565 at vfs_write_suspend_umnt+0x45 #10 0xc0f57329 at ffs_unmount+0xa9 #11 0xc0d1de1d at dounmount+0x64d #12 0xc0d1d6c5 at sys_unmount+0x315 #13 0xc120ae5c at syscall+0x75c #14 0xc11f683e at Xint0x80_syscall+0x2e ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 921097 files, 62764047 used, 55549934 free (37934 frags, 6939000 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp. Starting nfscbd. Starting default moused. Starting inetd. Tue May 30 16:00 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: May 30 16:40:21 x4 su: pho to root on /dev/pts/1 20170530 16:40:54 all (1/1): crossmp11.sh May 30 16:41:13 x4 kernel: pid 5383 (mkdir), uid 1004 inumber 153 on /mnt7: filesystem full May 30 16:41:14 x4 kernel: pid 5380 (mkdir), uid 1004 inumber 49315 on /mnt7: filesystem full May 30 16:41:15 x4 kernel: pid 5371 (mkdir), uid 1004 inumber 49337 on /mnt7: filesystem full May 30 16:41:16 x4 kernel: pid 5413 (rw), uid 1004 inumber 49171 on /mnt7: filesystem full May 30 16:41:16 x4 kernel: pid 5375 (mkdir), uid 1004 inumber 181 on /mnt7: filesystem full May 30 16:41:20 x4 kernel: pid 5428 (rw), uid 1004 inumber 49161 on /mnt7: filesystem full May 30 16:41:20 x4 kernel: pid 5370 (mkdir), uid 1004 inumber 43 on /mnt7: filesystem full May 30 16:41:30 x4 kernel: pid 5418 (rw), uid 1004 inumber 49165 on /mnt7: filesystem full May 30 16:41:31 x4 kernel: pid 5434 (rw), uid 1004 inumber 24592 on /mnt7: filesystem full May 30 16:41:34 x4 kernel: pid 5318 (rw), uid 1004 inumber 24583 on /mnt5: filesystem full May 30 16:41:38 x4 kernel: pid 5362 (creat), uid 1004 inumber 24595 on /mnt5: filesystem full May 30 16:42:02 x4 kernel: pid 5432 (rw), uid 1004 inumber 24591 on /mnt7: filesystem full May 30 16:44:12 x4 kernel: pid 7964 (rw), uid 1004 inumber 49369 on /mnt7: filesystem full May 30 16:44:12 x4 kernel: pid 7853 (creat), uid 1004 inumber 49155 on /mnt7: filesystem full May 30 16:44:12 x4 kernel: pid 7996 (mkdir), uid 1004 inumber 598 on /mnt6: filesystem full May 30 16:44:13 x4 kernel: pid 7942 (rw), uid 1004 inumber 49164 on /mnt6: filesystem full May 30 16:44:14 x4 kernel: pid 7921 (rw), uid 1004 inumber 49164 on /mnt8: filesystem full May 30 16:44:15 x4 kernel: pid 7973 (mkdir), uid 1004 inumber 73740 on /mnt8: filesystem full May 30 16:44:15 x4 kernel: pid 7930 (creat), uid 1004 inumber 24587 on /mnt8: filesystem full May 30 16:44:17 x4 kernel: pid 7947 (creat), uid 1004 inumber 49180 on /mnt8: filesystem full May 30 16:44:18 x4 kernel: pid 7893 (creat), uid 1004 inumber 49166 on /mnt7: filesystem full May 30 16:44:19 x4 kernel: pid 7943 (creat), uid 1004 inumber 49177 on /mnt8: filesystem full May 30 16:44:20 x4 kernel: pid 7928 (creat), uid 1004 inumber 24588 on /mnt8: filesystem full May 30 16:44:25 x4 kernel: pid 7850 (creat), uid 1004 inumber 49153 on /mnt7: filesystem full May 30 16:44:26 x4 kernel: pid 7932 (creat), uid 1004 inumber 49170 on /mnt8: filesystem full May 30 16:44:28 x4 kernel: pid 7844 (creat), uid 1004 inumber 49158 on /mnt7: filesystem full May 30 16:44:29 x4 kernel: pid 7958 (rw), uid 1004 inumber 24586 on /mnt7: filesystem full May 30 16:44:32 x4 kernel: pid 7929 (creat), uid 1004 inumber 49178 on /mnt8: filesystem full May 30 16:44:34 x4 kernel: pid 7845 (creat), uid 1004 inumber 49156 on /mnt7: filesystem full May 30 16:44:36 x4 kernel: pid 7854 (creat), uid 1004 inumber 49160 on /mnt7: filesystem full May 30 16:44:37 x4 kernel: pid 7890 (creat), uid 1004 inumber 49153 on /mnt8: filesystem full May 30 16:44:42 x4 kernel: pid 8004 (mkdir), uid 1004 inumber 73848 on /mnt5: filesystem full May 30 16:44:42 x4 kernel: pid 7903 (rw), uid 1004 inumber 73950 on /mnt5: filesystem full May 30 16:44:43 x4 kernel: pid 7847 (creat), uid 1004 inumber 49161 on /mnt7: filesystem full May 30 16:44:43 x4 kernel: pid 8002 (mkdir), uid 1004 inumber 50523 on /mnt5: filesystem full May 30 16:44:45 x4 kernel: pid 7853 (creat), uid 1004 inumber 49155 on /mnt7: filesystem full May 30 16:44:49 x4 kernel: pid 7854 (creat), uid 1004 inumber 49160 on /mnt7: filesystem full May 30 16:44:50 x4 kernel: pid 7851 (creat), uid 1004 inumber 49159 on /mnt7: filesystem full May 30 16:44:53 x4 kernel: pid 7892 (creat), uid 1004 inumber 49163 on /mnt7: filesystem full May 30 16:44:54 x4 kernel: pid 7845 (creat), uid 1004 inumber 49156 on /mnt7: filesystem full May 30 16:45:13 x4 kernel: pid 8934 (rw), uid 1004 inumber 15 on /mnt7: filesystem full May 30 16:45:13 x4 kernel: pid 8942 (mkdir), uid 1004 inumber 73755 on /mnt7: filesystem full May 30 16:45:18 x4 kernel: pid 9183 (rw), uid 1004 inumber 73753 on /mnt5: filesystem full May 30 16:45:18 x4 kernel: pid 9169 (mkdir), uid 1004 inumber 74236 on /mnt5: filesystem full May 30 16:45:18 x4 kernel: pid 9154 (creat), uid 1004 inumber 73733 on /mnt5: filesystem full May 30 16:47:23 x4 kernel: pid 9632 (rw), uid 1004 inumber 73745 on /mnt8: filesystem full May 30 16:47:24 x4 kernel: pid 9642 (creat), uid 1004 inumber 73741 on /mnt8: filesystem full May 30 16:47:24 x4 kernel: pid 9657 (mkdir), uid 1004 inumber 19 on /mnt8: filesystem full 20170530 16:47:53 all (1/1): crossmp11.sh May 30 16:48:35 x4 kernel: pid 12573 (rw), uid 1004 inumber 49511 on /mnt6: filesystem full May 30 16:48:35 x4 kernel: pid 12547 (mkdir), uid 1004 inumber 30 on /mnt6: filesystem full May 30 16:48:38 x4 kernel: pid 12541 (mkdir), uid 1004 inumber 688 on /mnt6: filesystem full May 30 16:48:38 x4 kernel: pid 12567 (creat), uid 1004 inumber 287 on /mnt6: filesystem full May 30 16:48:39 x4 kernel: pid 12502 (mkdir), uid 1004 inumber 809 on /mnt6: filesystem full May 30 16:48:39 x4 kernel: pid 12528 (rw), uid 1004 inumber 49158 on /mnt6: filesystem full May 30 16:48:48 x4 kernel: pid 12515 (mkdir), uid 1004 inumber 49202 on /mnt5: filesystem full May 30 16:48:49 x4 kernel: pid 12452 (rw), uid 1004 inumber 8 on /mnt5: filesystem full May 30 16:48:51 x4 kernel: pid 12465 (rw), uid 1004 inumber 73740 on /mnt5: filesystem full May 30 16:48:54 x4 kernel: pid 12458 (rw), uid 1004 inumber 6 on /mnt5: filesystem full May 30 16:48:54 x4 kernel: pid 12562 (creat), uid 1004 inumber 49163 on /mnt6: filesystem full May 30 16:48:55 x4 kernel: pid 12557 (rw), uid 1004 inumber 24581 on /mnt7: filesystem full May 30 16:48:56 x4 kernel: pid 12540 (creat), uid 1004 inumber 49161 on /mnt7: filesystem full May 30 16:48:59 x4 kernel: pid 12479 (rw), uid 1004 inumber 73736 on /mnt5: filesystem full May 30 16:50:38 x4 kernel: pid 13992 (mkdir), uid 1004 inumber 186 on /mnt5: filesystem full May 30 16:50:39 x4 kernel: pid 14002 (mkdir), uid 1004 inumber 73889 on /mnt5: filesystem full May 30 16:50:41 x4 kernel: pid 14007 (mkdir), uid 1004 inumber 49318 on /mnt5: filesystem full May 30 16:50:42 x4 kernel: pid 13997 (mkdir), uid 1004 inumber 74016 on /mnt5: filesystem full May 30 16:50:43 x4 kernel: pid 14029 (rw), uid 1004 inumber 49282 on /mnt5: filesystem full May 30 16:50:43 x4 kernel: pid 13993 (mkdir), uid 1004 inumber 74107 on /mnt5: filesystem full May 30 16:50:43 x4 kernel: pid 14016 (creat), uid 1004 inumber 24577 on /mnt5: filesystem full May 30 16:50:46 x4 last message repeated 2 times May 30 16:50:46 x4 kernel: pid 14028 (rw), uid 1004 inumber 49274 on /mnt5: filesystem full May 30 16:50:47 x4 kernel: pid 14016 (creat), uid 1004 inumber 24577 on /mnt5: filesystem full May 30 16:51:13 x4 kernel: pid 14247 (mkdir), uid 1004 inumber 1297 on /mnt6: filesystem full May 30 16:51:13 x4 kernel: pid 14262 (creat), uid 1004 inumber 49162 on /mnt6: filesystem full May 30 16:51:13 x4 kernel: pid 14284 (rw), uid 1004 inumber 73739 on /mnt6: filesystem full May 30 16:52:04 x4 kernel: pid 14270 (creat), uid 1004 inumber 49163 on /mnt6: filesystem full May 30 16:52:04 x4 kernel: pid 14269 (rw), uid 1004 inumber 49167 on /mnt6: filesystem full May 30 16:52:05 x4 kernel: pid 14241 (creat), uid 1004 inumber 49155 on /mnt6: filesystem full May 30 16:52:06 x4 kernel: pid 14258 (creat), uid 1004 inumber 49157 on /mnt6: filesystem full May 30 16:53:26 x4 kernel: pid 16339 (rw), uid 1004 inumber 73757 on /mnt6: filesystem full May 30 16:54:20 x4 kernel: pid 16376 (rw), uid 1004 inumber 16 on /mnt8: filesystem full 20170530 16:54:34 all (1/1): crossmp11.sh May 30 16:54:51 x4 kernel: pid 19870 (mkdir), uid 1004 inumber 56079 on /mnt7: filesystem full May 30 16:54:51 x4 kernel: pid 19852 (creat), uid 1004 inumber 24600 on /mnt7: filesystem full May 30 16:54:51 x4 kernel: pid 19860 (rw), uid 1004 inumber 24596 on /mnt7: filesystem full May 30 16:54:59 x4 kernel: pid 19936 (mkdir), uid 1004 inumber 232 on /mnt5: filesystem full May 30 16:54:59 x4 kernel: pid 19944 (rw), uid 1004 inumber 49172 on /mnt5: filesystem full May 30 16:55:00 x4 kernel: pid 19938 (mkdir), uid 1004 inumber 73969 on /mnt5: filesystem full May 30 16:55:00 x4 kernel: pid 19959 (rw), uid 1004 inumber 49169 on /mnt5: filesystem full May 30 16:55:04 x4 kernel: pid 19930 (mkdir), uid 1004 inumber 650 on /mnt5: filesystem full May 30 16:55:04 x4 kernel: pid 19935 (rw), uid 1004 inumber 49167 on /mnt5: filesystem full May 30 16:55:05 x4 kernel: pid 19973 (mkdir), uid 1004 inumber 73745 on /mnt8: filesystem full May 30 16:55:05 x4 kernel: pid 19919 (rw), uid 1004 inumber 24587 on /mnt8: filesystem full May 30 16:55:09 x4 kernel: pid 19962 (rw), uid 1004 inumber 49173 on /mnt5: filesystem full May 30 16:55:12 x4 kernel: pid 19904 (rw), uid 1004 inumber 24608 on /mnt7: filesystem full May 30 16:55:52 x4 kernel: pid 19966 (rw), uid 1004 inumber 24583 on /mnt5: filesystem full May 30 16:57:02 x4 kernel: pid 21272 (creat), uid 1004 inumber 24593 on /mnt7: filesystem full May 30 16:57:02 x4 kernel: pid 21243 (rw), uid 1004 inumber 49166 on /mnt7: filesystem full May 30 16:57:06 x4 kernel: pid 21277 (creat), uid 1004 inumber 24602 on /mnt7: filesystem full May 30 16:57:07 x4 kernel: pid 21246 (rw), uid 1004 inumber 49168 on /mnt7: filesystem full May 30 16:57:18 x4 kernel: pid 21213 (mkdir), uid 1004 inumber 25961 on /mnt8: filesystem full May 30 16:57:20 x4 kernel: pid 21212 (mkdir), uid 1004 inumber 73736 on /mnt8: filesystem full May 30 16:57:20 x4 kernel: pid 21230 (creat), uid 1004 inumber 24579 on /mnt8: filesystem full May 30 16:57:20 x4 kernel: pid 21223 (rw), uid 1004 inumber 49160 on /mnt8: filesystem full May 30 16:57:22 x4 kernel: pid 21230 (creat), uid 1004 inumber 24579 on /mnt8: filesystem full May 30 16:57:33 x4 kernel: pid 21364 (rw), uid 1004 inumber 24598 on /mnt5: filesystem full May 30 16:57:33 x4 kernel: pid 21327 (mkdir), uid 1004 inumber 73732 on /mnt5: filesystem full May 30 16:57:33 x4 kernel: pid 21330 (creat), uid 1004 inumber 24581 on /mnt5: filesystem full May 30 16:58:03 x4 kernel: pid 21359 (rw), uid 1004 inumber 27292 on /mnt5: filesystem full May 30 16:58:03 x4 kernel: pid 21343 (creat), uid 1004 inumber 25225 on /mnt5: filesystem full May 30 16:58:08 x4 kernel: pid 21241 (rw), uid 1004 inumber 49165 on /mnt7: filesystem full May 30 16:58:08 x4 kernel: pid 21260 (creat), uid 1004 inumber 24587 on /mnt7: filesystem full May 30 16:58:09 x4 kernel: pid 21274 (creat), uid 1004 inumber 24596 on /mnt7: filesystem full May 30 16:58:10 x4 kernel: pid 21240 (rw), uid 1004 inumber 73731 on /mnt7: filesystem full May 30 16:58:10 x4 kernel: pid 21274 (creat), uid 1004 inumber 24596 on /mnt7: filesystem full May 30 16:58:11 x4 kernel: pid 21363 (rw), uid 1004 inumber 24604 on /mnt5: filesystem full May 30 16:58:11 x4 kernel: pid 21265 (creat), uid 1004 inumber 24590 on /mnt7: filesystem full May 30 16:58:13 x4 kernel: pid 21342 (creat), uid 1004 inumber 24591 on /mnt5: filesystem full May 30 16:58:14 x4 kernel: pid 21336 (rw), uid 1004 inumber 24600 on /mnt5: filesystem full May 30 16:58:14 x4 kernel: pid 21344 (creat), uid 1004 inumber 24781 on /mnt5: filesystem full May 30 16:58:16 x4 kernel: pid 21330 (creat), uid 1004 inumber 24581 on /mnt5: filesystem full May 30 16:58:17 x4 kernel: pid 21273 (creat), uid 1004 inumber 24595 on /mnt7: filesystem full May 30 16:58:19 x4 kernel: pid 21341 (creat), uid 1004 inumber 25223 on /mnt5: filesystem full May 30 16:58:20 x4 kernel: pid 21273 (creat), uid 1004 inumber 24595 on /mnt7: filesystem full May 30 16:58:21 x4 kernel: pid 21267 (creat), uid 1004 inumber 24597 on /mnt7: filesystem full May 30 16:58:22 x4 kernel: pid 21357 (rw), uid 1004 inumber 24601 on /mnt5: filesystem full May 30 16:58:29 x4 kernel: pid 21267 (creat), uid 1004 inumber 24597 on /mnt7: filesystem full May 30 16:58:31 x4 kernel: pid 21265 (creat), uid 1004 inumber 24590 on /mnt7: filesystem full May 30 16:58:46 x4 kernel: pid 21355 (rw), uid 1004 inumber 24608 on /mnt5: filesystem full May 30 16:59:27 x4 kernel: pid 22581 (mkdir), uid 1004 inumber 2134 on /mnt5: filesystem full May 30 16:59:27 x4 kernel: pid 22564 (rw), uid 1004 inumber 49158 on /mnt5: filesystem full May 30 16:59:56 x4 kernel: pid 23196 (mkdir), uid 1004 inumber 49536 on /mnt8: filesystem full May 30 16:59:56 x4 kernel: pid 23190 (rw), uid 1004 inumber 49169 on /mnt8: filesystem full May 30 16:59:57 x4 kernel: pid 23176 (creat), uid 1004 inumber 49160 on /mnt8: filesystem full May 30 17:00:39 x4 kernel: pid 22579 (mkdir), uid 1004 inumber 61 on /mnt5: filesystem full May 30 17:00:41 x4 kernel: pid 22584 (mkdir), uid 1004 inumber 24801 on /mnt5: filesystem full May 30 17:00:48 x4 kernel: pid 22571 (mkdir), uid 1004 inumber 73803 on /mnt5: filesystem full May 30 17:00:51 x4 kernel: pid 22565 (rw), uid 1004 inumber 49156 on /mnt5: filesystem full May 30 17:01:23 x4 kernel: pid 23670 (mkdir), uid 1004 inumber 1540 on /mnt6: filesystem full May 30 17:01:23 x4 kernel: pid 23645 (rw), uid 1004 inumber 73738 on /mnt6: filesystem full FreeBSD/i386 (x4.osted.lan) (ttyu0) login: root Password: May 30 17:46:01 x4 login: ROOT LOGIN (root) ON ttyu0 Last login: Tue May 23 15:41:05 on ttyu0 FreeBSD 12.0-CURRENT (PHO) #1 r319211M: Tue May 30 15:54:02 CEST 2017 You have new mail. root@x4:~ # init 1 root@x4:~ # Stopping inetd. Waiting for PIDS: 843. Stopping moused. Waiting for PIDS: 813. Stopping nfscbd. Waiting for PIDS: 808 809. Shutting down local packages:. Stopping cron. Waiting for PIDS: 701. Stopping sshd. Waiting for PIDS: 690. Stopping smartd. Waiting for PIDS: 653. Stopping powerd. Waiting for PIDS: 623. Stopping ntpd. Waiting for PIDS: 620. Stopping lockd. Waiting for PIDS: 563. Stopping statd. Waiting for PIDS: 560. Stopping nfsd. Waiting for PIDS: 556 559. Stopping mountd. Waiting for PIDS: 550. Stopping nfsuserd. Waiting for PIDS: 539 540 541 542 543. Stopping watchdogd. Waiting for PIDS: 493. Stopping rpcbind. Waiting for PIDS: 409. Stopping devd. Waiting for PIDS: 328. Writing entropy file:. Writing early boot entropy file:. . Terminated May 30 17:46:26 x4 syslogd: exiting on signal 15 May 30 17:46:46 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 69287 1 0 20 0 6628 2912 wait Ss u0 0:00.01 -sh (sh) 0 69288 69287 0 20 0 6564 2344 - R+ u0 0:00.00 ps -l 0 23808 1 0 20 0 6012 1812 mntref D 1- 0:00.34 umount -f /mnt8 # KDB: enter: Break to debugger [ thread pid 11 tid 100006 ] Stopped at kdb_alt_break_internal+0x101: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100006 td 0xc8e66a20 kdb_alt_break_internal(0,e4b42ac4,c0a5b2a7,2,c91ae888,...) at kdb_alt_break_internal+0x101/frame 0xe4b42a98 kdb_alt_break(2,c91ae888,c15c926b,0,40000,...) at kdb_alt_break+0x16/frame 0xe4b42aa4 uart_intr_rxready(c91ae800,6e2e5f16,e4b20c80,c8e66a20,e4b42ae8,...) at uart_intr_rxready+0x87/frame 0xe4b42ac4 uart_intr(c91ae800,1a2e,1,6e2e5f16,0,...) at uart_intr+0x120/frame 0xe4b42b04 intr_event_handle(c8eb3e00,e4b42b98,c1b05c00,40000000,1,...) at intr_event_handle+0x75/frame 0xe4b42b30 intr_execute_handlers(c8eae908,e4b42b98,3) at intr_execute_handlers+0xb0/frame 0xe4b42b68 lapic_handle_intr(32,e4b42b98) at lapic_handle_intr+0x5b/frame 0xe4b42b88 Xapic_isr1() at Xapic_isr1+0x55/frame 0xe4b42b88 --- interrupt, eip = 0xc1224725, esp = 0xe4b42bd8, ebp = 0xe4b42bd8 --- acpi_cpu_c1(e4b42c2c,c1213a39,e4b20c90,0,c18183c3,...) at acpi_cpu_c1+0x5/frame 0xe4b42bd8 acpi_cpu_idle(1dfbd839,0,1dfbd839,e4b42c48,c1224af6,...) at acpi_cpu_idle+0x325/frame 0xe4b42c18 cpu_idle_acpi(1dfbd839,0,c1b05c00,2,c8e66a20,...) at cpu_idle_acpi+0x3f/frame 0xe4b42c2c cpu_idle(0,e4b42c80,c160ccd7,394,0,...) at cpu_idle+0xa6/frame 0xe4b42c48 sched_idletd(0,e4b42ce8,c1601280,406,0,...) at sched_idletd+0x16a/frame 0xe4b42cac fork_exit(c0c88250,0,e4b42ce8) at fork_exit+0x7e/frame 0xe4b42cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b42cd4 --- trap 0, eip = 0, esp = 0xe4b42d20, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x70c400 curthread = 0xc8e676c0: pid 11 tid 100003 "idle: cpu0" curpcb = 0xe4b36d40 fpcurthread = none idlethread = 0xc8e676c0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x231fd400 curthread = 0xc8e67360: pid 11 tid 100004 "idle: cpu1" curpcb = 0xe4b3ad40 fpcurthread = none idlethread = 0xc8e67360: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x23201400 curthread = 0xc8e67000: pid 11 tid 100005 "idle: cpu2" curpcb = 0xe4b3ed40 fpcurthread = none idlethread = 0xc8e67000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x23205400 curthread = 0xc8e66a20: pid 11 tid 100006 "idle: cpu3" curpcb = 0xe4b42d40 fpcurthread = none idlethread = 0xc8e66a20: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc96bb000 /dev/ada0s1a on / (ufs) 0xc96bc000 devfs on /dev (devfs) 0xc9acf000 /dev/ada0s1f on /home (ufs) 0xc9acd000 /dev/ada0s1d on /usr (ufs) 0xc9bb4000 /dev/ada0s1e on /var (ufs) 0xc9bb3000 procfs on /proc (procfs) 0xc9c24000 fdescfs on /dev/fd (fdescfs) 0xc9c23000 linprocfs on /compat/linux/proc (linprocfs) 0xc9ace000 /dev/label/tmp on /tmp (ufs) More info: show mount db:0:mount> show page vm_cnt.v_free_count: 43439 vm_cnt.v_inactive_count: 411494 vm_cnt.v_active_count: 262670 vm_cnt.v_laundry_count: 112219 vm_cnt.v_wire_count: 32962 vm_cnt.v_free_reserved: 1173 vm_cnt.v_free_min: 5485 vm_cnt.v_free_target: 18421 vm_cnt.v_inactive_target: 27631 db:0:page> ps pid ppid pgrp uid state wmesg wchan cmd 69287 1 69287 0 Ss+ ttyin 0xc91ae674 sh 23808 1 17658 0 D+ mntref 0xcc97a000 umount 18490 0 0 0 DL mdwait 0xcaed9000 [md8] 18444 0 0 0 DL mdwait 0xcc0c5000 [md7] 18398 0 0 0 DL mdwait 0xcc0c4800 [md6] 18331 0 0 0 DL mdwait 0xcbb90000 [md5] 22 0 0 0 DL syncer 0xc1b71ddc [syncer] 21 0 0 0 DL vlruwt 0xc914fa2c [vnlru] 20 0 0 0 DL (threaded) [bufdaemon] 100092 D psleep 0xc1b70b04 [bufdaemon] 100107 D sdflush 0xc90ce284 [/home worker] 100109 D sdflush 0xc90cde84 [/usr worker] 100110 D sdflush 0xc90cdc84 [/var worker] 100186 D sdflush 0xc90ce084 [/tmp worker] 19 0 0 0 DL - 0xc1b7174c [bufspacedaemon] 18 0 0 0 DL psleep 0xc1b77cac [vmdaemon] 17 0 0 0 DL (threaded) [pagedaemon] 100089 D psleep 0xc1bdaf05 [pagedaemon] 100093 D launds 0xc1b77c74 [laundry: dom0] 100095 D umarcl 0xc1b778a0 [uma] 16 0 0 0 DL - 0xc19cf768 [rand_harvestq] 15 0 0 0 DL waiting 0xc1bd67f4 [sctp_iterator] 9 0 0 0 DL - 0xc1b704dc [soaiod4] 8 0 0 0 DL - 0xc1b704dc [soaiod3] 7 0 0 0 DL - 0xc1b704dc [soaiod2] 6 0 0 0 DL - 0xc1b704dc [soaiod1] 5 0 0 0 DL - 0xc9174a3c [fdc0] 14 0 0 0 DL (threaded) [usb] 100042 D - 0xc90eaacc [usbus0] 100043 D - 0xc90eaafc [usbus0] 100044 D - 0xc90eab2c [usbus0] 100045 D - 0xc90eab5c [usbus0] 100046 D - 0xc90eab8c [usbus0] 100048 D - 0xc9156acc [usbus1] 100049 D - 0xc9156afc [usbus1] 100050 D - 0xc9156b2c [usbus1] 100051 D - 0xc9156b5c [usbus1] 100052 D - 0xc9156b8c [usbus1] 100054 D - 0xc915dacc [usbus2] 100055 D - 0xc915dafc [usbus2] 100056 D - 0xc915db2c [usbus2] 100057 D - 0xc915db5c [usbus2] 100058 D - 0xc915db8c [usbus2] 100059 D - 0xc9164acc [usbus3] 100060 D - 0xc9164afc [usbus3] 100061 D - 0xc9164b2c [usbus3] 100062 D - 0xc9164b5c [usbus3] 100063 D - 0xc9164b8c [usbus3] 100064 D - 0xc9169acc [usbus4] 100065 D - 0xc9169afc [usbus4] 100066 D - 0xc9169b2c [usbus4] 100067 D - 0xc9169b5c [usbus4] 100068 D - 0xc9169b8c [usbus4] 100070 D - 0xc90d5ca4 [usbus5] 100071 D - 0xc90d5cd4 [usbus5] 100072 D - 0xc90d5d04 [usbus5] 100073 D - 0xc90d5d34 [usbus5] 100074 D - 0xc90d5d64 [usbus5] 100117 D - 0xc8e23408 [ucom] 4 0 0 0 DL (threaded) [cam] 100037 D - 0xc191bc00 [doneq0] 100088 D - 0xc191baac [scanner] 3 0 0 0 DL crypto_ 0xc1b7683c [crypto returns] 2 0 0 0 DL crypto_ 0xc1b76784 [crypto] 13 0 0 0 DL (threaded) [geom] 100028 D - 0xc1bd18c4 [g_event] 100029 D - 0xc1bd18c8 [g_up] 100030 D - 0xc1bd18cc [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi3: vm] 100008 I [swi4: clock (0)] 100009 I [swi4: clock (1)] 100010 I [swi4: clock (2)] 100011 I [swi4: clock (3)] 100012 I [swi1: netisr 0] 100023 I [swi6: task queue] 100024 I [swi6: Giant taskq] 100027 I [swi5: fast taskq] 100038 I [irq256: hdac0] 100039 I [irq257: re0] 100040 I [irq22: ahci0] 100041 I [irq16: hdac1 ohci0] 100047 I [irq17: ohci1 ohci3] 100053 I [irq18: ohci2 ohci4] 100069 I [irq19: ehci0] 100075 I [irq14: ata0] 100076 I [swi0: uart] 100078 I [irq1: atkbd0] 100079 I [irq12: psm0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc8e59a2c [init] 10 0 0 0 DL audit_w 0xc1bd9710 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xc1bd18e4 [swapper] 100013 D - 0xc8e34f00 [if_config_tqg_0] 100014 D - 0xc8e34e80 [softirq_0] 100015 D - 0xc8e34e00 [softirq_1] 100016 D - 0xc8e34d80 [softirq_2] 100017 D - 0xc8e34d00 [softirq_3] 100018 D - 0xc8e34c80 [if_io_tqg_0] 100019 D - 0xc8e34c00 [if_io_tqg_1] 100020 D - 0xc8e34b80 [if_io_tqg_2] 100021 D - 0xc8e34b00 [if_io_tqg_3] 100022 D - 0xc8e34a80 [kqueue_ctx taskq] 100025 D - 0xc8e34900 [thread taskq] 100026 D - 0xc8e34880 [aiod_kick taskq] 100031 D - 0xc8e34680 [firmware taskq] 100034 D - 0xc8e33d80 [acpi_task_0] 100035 D - 0xc8e33d80 [acpi_task_1] 100036 D - 0xc8e33d80 [acpi_task_2] 100080 D - 0xc9191980 [mca taskq] 100087 D - 0xc8e33d00 [CAM taskq] db:0:ps> allt Tracing command sh pid 69287 tid 100205 td 0xc92b2360 sched_switch(c92b2360,0,104,18a,c92b2360,...) at sched_switch+0x512/frame 0xf36af900 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xf36af948 sleepq_switch(c92b2360,0,c1610fcd,1c5,490,...) at sleepq_switch+0x11c/frame 0xf36af980 sleepq_catch_signals(f36af9e4,c0c362b0,c91ae614,0,0,...) at sleepq_catch_signals+0x36e/frame 0xf36af9c0 sleepq_wait_sig(c91ae674,0,f36afa04,101,0,...) at sleepq_wait_sig+0x14/frame 0xf36af9e4 _cv_wait_sig(c91ae674,c91ae604,c1616ec5,5a8,c91ae600,...) at _cv_wait_sig+0x21f/frame 0xf36afa28 tty_wait(c91ae600,c91ae674,f36afb90,1,0,...) at tty_wait+0x4b/frame 0xf36afa4c ttydisc_read(c91ae600,f36afb90,0,d0,c18dd5c4,...) at ttydisc_read+0x235/frame 0xf36afa94 ttydev_read(c8f19e00,f36afb90,0,c9ced028,f36afaf4,...) at ttydev_read+0x55/frame 0xf36afab4 devfs_read_f(c980edc8,f36afb90,c8e2fd00,0,c92b2360,...) at devfs_read_f+0xc5/frame 0xf36afaec dofileread(c980edc8,f36afb90,ffffffff,ffffffff,0,...) at dofileread+0xa7/frame 0xf36afb20 kern_readv(c92b2360,0,f36afb90,f36afbc0,bfbfec6a,...) at kern_readv+0x93/frame 0xf36afb70 sys_read(c92b2360,f36afc00,c1600cbf,4,0,...) at sys_read+0x54/frame 0xf36afbc0 syscall(f36afce8) at syscall+0x75c/frame 0xf36afcdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xf36afcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2820a7a7, esp = 0xbfbfec24, ebp = 0xbfbfec38 --- Tracing command umount pid 23808 tid 100318 td 0xccae2a20 sched_switch(ccae2a20,0,104,18a,ccae2a20,...) at sched_switch+0x512/frame 0xc8563958 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xc85639a0 sleepq_switch(ccae2a20,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xc85639d8 sleepq_wait(cc97a000,60,c161d2e3,0,0,...) at sleepq_wait+0x48/frame 0xc85639fc _sleep(cc97a000,cc97a000,60,c161d2e3,0,...) at _sleep+0x38f/frame 0xc8563a50 vfs_mount_destroy(cc97a000,10,0,591,c0c31a00,...) at vfs_mount_destroy+0xa4/frame 0xc8563a88 dounmount(cc97a000,8080000,ccae2a20,490,72417122,...) at dounmount+0x88c/frame 0xc8563ae8 sys_unmount(ccae2a20,c8563c00,c1600cbf,4,c1b05500,...) at sys_unmount+0x315/frame 0xc8563bc0 syscall(c8563ce8) at syscall+0x75c/frame 0xc8563cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xc8563cdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280a9bef, esp = 0xbfbfd424, ebp = 0xbfbfd900 --- Tracing command md8 pid 18490 tid 100468 td 0xcbb94360 sched_switch(cbb94360,0,104,18a,c681cb9c,...) at sched_switch+0x512/frame 0xc681cb78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xc681cbbc sleepq_switch(cbb94360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xc681cbf4 sleepq_wait(caed9000,5c,c132afda,0,0,...) at sleepq_wait+0x48/frame 0xc681cc18 _sleep(caed9000,caed9020,25c,c132afda,0,...) at _sleep+0x38f/frame 0xc681cc6c md_kthread(caed9000,c681cce8,c1601280,406,0,...) at md_kthread+0x1da/frame 0xc681ccac fork_exit(c07e2da0,caed9000,c681cce8) at fork_exit+0x7e/frame 0xc681ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xc681ccd4 --- trap 0, eip = 0, esp = 0xc681cd20, ebp = 0 --- Tracing command md7 pid 18444 tid 100268 td 0xcca4d360 sched_switch(cca4d360,0,104,18a,c8470b9c,...) at sched_switch+0x512/frame 0xc8470b78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xc8470bbc sleepq_switch(cca4d360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xc8470bf4 sleepq_wait(cc0c5000,5c,c132afda,0,0,...) at sleepq_wait+0x48/frame 0xc8470c18 _sleep(cc0c5000,cc0c5020,25c,c132afda,0,...) at _sleep+0x38f/frame 0xc8470c6c md_kthread(cc0c5000,c8470ce8,c1601280,406,0,...) at md_kthread+0x1da/frame 0xc8470cac fork_exit(c07e2da0,cc0c5000,c8470ce8) at fork_exit+0x7e/frame 0xc8470cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xc8470cd4 --- trap 0, eip = 0, esp = 0xc8470d20, ebp = 0 --- Tracing command md6 pid 18398 tid 100251 td 0xca6a7a20 sched_switch(ca6a7a20,0,104,18a,f37cdb9c,...) at sched_switch+0x512/frame 0xf37cdb78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xf37cdbbc sleepq_switch(ca6a7a20,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xf37cdbf4 sleepq_wait(cc0c4800,5c,c132afda,0,0,...) at sleepq_wait+0x48/frame 0xf37cdc18 _sleep(cc0c4800,cc0c4820,25c,c132afda,0,...) at _sleep+0x38f/frame 0xf37cdc6c md_kthread(cc0c4800,f37cdce8,c1601280,406,0,...) at md_kthread+0x1da/frame 0xf37cdcac fork_exit(c07e2da0,cc0c4800,f37cdce8) at fork_exit+0x7e/frame 0xf37cdcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf37cdcd4 --- trap 0, eip = 0, esp = 0xf37cdd20, ebp = 0 --- Tracing command md5 pid 18331 tid 100266 td 0xcca4da20 sched_switch(cca4da20,0,104,18a,c8468b9c,...) at sched_switch+0x512/frame 0xc8468b78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xc8468bbc sleepq_switch(cca4da20,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xc8468bf4 sleepq_wait(cbb90000,5c,c132afda,0,0,...) at sleepq_wait+0x48/frame 0xc8468c18 _sleep(cbb90000,cbb90020,25c,c132afda,0,...) at _sleep+0x38f/frame 0xc8468c6c md_kthread(cbb90000,c8468ce8,c1601280,406,0,...) at md_kthread+0x1da/frame 0xc8468cac fork_exit(c07e2da0,cbb90000,c8468ce8) at fork_exit+0x7e/frame 0xc8468cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xc8468cd4 --- trap 0, eip = 0, esp = 0xc8468d20, ebp = 0 --- Tracing command syncer pid 22 tid 100096 td 0xc91bf360 sched_switch(c91bf360,0,104,18a,c91bf360,...) at sched_switch+0x512/frame 0xedf9cb40 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf9cb88 sleepq_switch(c91bf360,0,c1610fcd,2d1,c91bf360,...) at sleepq_switch+0x11c/frame 0xedf9cbc0 sleepq_timedwait(c1b71ddc,0,edf9cc24,0,0,...) at sleepq_timedwait+0x53/frame 0xedf9cc00 _cv_timedwait_sbt(c1b71ddc,c1b71dbc,fffffed8,0,0,...) at _cv_timedwait_sbt+0x24b/frame 0xedf9cc48 sched_sync(0,edf9cce8,c1601280,406,0,...) at sched_sync+0x1fd/frame 0xedf9ccac fork_exit(c0d2b750,0,edf9cce8) at fork_exit+0x7e/frame 0xedf9ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf9ccd4 --- trap 0, eip = 0, esp = 0xedf9cd20, ebp = 0 --- Tracing command vnlru pid 21 tid 100094 td 0xc91bf6c0 sched_switch(c91bf6c0,0,104,18a,c91bf6c0,...) at sched_switch+0x512/frame 0xedf98b20 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf98b68 sleepq_switch(c91bf6c0,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedf98ba0 sleepq_timedwait(c914fa2c,60,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf98bdc _sleep(c914fa2c,c1b71d88,260,c161ebf0,fffffed8,...) at _sleep+0x363/frame 0xedf98c30 vnlru_proc(0,edf98ce8,c1601280,406,0,...) at vnlru_proc+0xbf/frame 0xedf98cac fork_exit(c0d2adc0,0,edf98ce8) at fork_exit+0x7e/frame 0xedf98cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf98cd4 --- trap 0, eip = 0, esp = 0xedf98d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100092 td 0xc91bfa20 sched_switch(c91bfa20,0,104,18a,c91bfa20,...) at sched_switch+0x512/frame 0xedf8cb68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf8cbb0 sleepq_switch(c91bfa20,0,c1610fcd,2d1,1a2f,...) at sleepq_switch+0x11c/frame 0xedf8cbe8 sleepq_timedwait(c1b70b04,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf8cc24 _sleep(c1b70b04,c1b70a80,54,c161b6ea,fffffed8,...) at _sleep+0x363/frame 0xedf8cc78 buf_daemon(0,edf8cce8,c1601280,406,0,...) at buf_daemon+0x8c/frame 0xedf8ccac fork_exit(c0d08ef0,0,edf8cce8) at fork_exit+0x7e/frame 0xedf8ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf8ccd4 --- trap 0, eip = 0, esp = 0xedf8cd20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100107 td 0xc9b186c0 sched_switch(c9b186c0,0,104,18a,c9b186c0,...) at sched_switch+0x512/frame 0xedfd8b58 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedfd8ba0 sleepq_switch(c9b186c0,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedfd8bd8 sleepq_timedwait(c90ce284,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfd8c14 _sleep(c90ce284,c90ce200,54,c1655f51,7fffff6c,...) at _sleep+0x363/frame 0xedfd8c68 softdep_flush(c9acf000,edfd8ce8,c1601280,406,0,...) at softdep_flush+0x196/frame 0xedfd8cac fork_exit(c0f3ab20,c9acf000,edfd8ce8) at fork_exit+0x7e/frame 0xedfd8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfd8cd4 --- trap 0, eip = 0, esp = 0xedfd8d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100109 td 0xc9b18000 sched_switch(c9b18000,0,104,18a,c9b18000,...) at sched_switch+0x512/frame 0xedfe0b58 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedfe0ba0 sleepq_switch(c9b18000,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedfe0bd8 sleepq_timedwait(c90cde84,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfe0c14 _sleep(c90cde84,c90cde00,54,c1655f51,7fffff6c,...) at _sleep+0x363/frame 0xedfe0c68 softdep_flush(c9acd000,edfe0ce8,c1601280,406,0,...) at softdep_flush+0x196/frame 0xedfe0cac fork_exit(c0f3ab20,c9acd000,edfe0ce8) at fork_exit+0x7e/frame 0xedfe0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfe0cd4 --- trap 0, eip = 0, esp = 0xedfe0d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100110 td 0xc9b17a20 sched_switch(c9b17a20,0,104,18a,c9b17a20,...) at sched_switch+0x512/frame 0xedfe4b58 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedfe4ba0 sleepq_switch(c9b17a20,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedfe4bd8 sleepq_timedwait(c90cdc84,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfe4c14 _sleep(c90cdc84,c90cdc00,54,c1655f51,7fffff6c,...) at _sleep+0x363/frame 0xedfe4c68 softdep_flush(c9bb4000,edfe4ce8,c1601280,406,0,...) at softdep_flush+0x196/frame 0xedfe4cac fork_exit(c0f3ab20,c9bb4000,edfe4ce8) at fork_exit+0x7e/frame 0xedfe4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfe4cd4 --- trap 0, eip = 0, esp = 0xedfe4d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100186 td 0xca65c000 sched_switch(ca65c000,0,104,18a,ca65c000,...) at sched_switch+0x512/frame 0xf3639b58 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xf3639ba0 sleepq_switch(ca65c000,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xf3639bd8 sleepq_timedwait(c90ce084,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xf3639c14 _sleep(c90ce084,c90ce000,54,c1655f51,7fffff6c,...) at _sleep+0x363/frame 0xf3639c68 softdep_flush(c9ace000,f3639ce8,c1601280,406,0,...) at softdep_flush+0x196/frame 0xf3639cac fork_exit(c0f3ab20,c9ace000,f3639ce8) at fork_exit+0x7e/frame 0xf3639cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf3639cd4 --- trap 0, eip = 0, esp = 0xf3639d20, ebp = 0 --- Tracing command bufspacedaemon pid 19 tid 100091 td 0xc91c0000 sched_switch(c91c0000,0,104,18a,c91c0000,...) at sched_switch+0x512/frame 0xedf88b68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf88bb0 sleepq_switch(c91c0000,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedf88be8 sleepq_timedwait(c1b7174c,5c,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf88c24 _sleep(c1b7174c,c1b71500,25c,c15f0e1b,fffffed8,...) at _sleep+0x363/frame 0xedf88c78 bufspace_daemon(0,edf88ce8,c1601280,406,0,...) at bufspace_daemon+0x1ff/frame 0xedf88cac fork_exit(c0d08800,0,edf88ce8) at fork_exit+0x7e/frame 0xedf88cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf88cd4 --- trap 0, eip = 0, esp = 0xedf88d20, ebp = 0 --- Tracing command vmdaemon pid 18 tid 100090 td 0xc91c0360 sched_switch(c91c0360,0,104,18a,c91c0360,...) at sched_switch+0x512/frame 0xedf84b38 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf84b80 sleepq_switch(c91c0360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xedf84bb8 sleepq_wait(c1b77cac,74,c161b6ea,0,0,...) at sleepq_wait+0x48/frame 0xedf84bdc _sleep(c1b77cac,c1b77c94,74,c161b6ea,0,...) at _sleep+0x38f/frame 0xedf84c30 vm_daemon(0,edf84ce8,c1601280,406,0,...) at vm_daemon+0xa9/frame 0xedf84cac fork_exit(c0f9bbe0,0,edf84ce8) at fork_exit+0x7e/frame 0xedf84cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf84cd4 --- trap 0, eip = 0, esp = 0xedf84d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100089 td 0xc91c06c0 sched_switch(c91c06c0,0,104,18a,c91c06c0,...) at sched_switch+0x512/frame 0xedf80b28 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf80b70 sleepq_switch(c91c06c0,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedf80ba8 sleepq_timedwait(c1bdaf05,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf80be4 _sleep(c1bdaf05,c1bdae80,254,c161b6ea,fffffed8,...) at _sleep+0x363/frame 0xedf80c38 vm_pageout(0,edf80ce8,c1601280,406,0,...) at vm_pageout+0x467/frame 0xedf80cac fork_exit(c0f990e0,0,edf80ce8) at fork_exit+0x7e/frame 0xedf80cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf80cd4 --- trap 0, eip = 0, esp = 0xedf80d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100093 td 0xc93ce000 sched_switch(c93ce000,0,104,18a,c93ce000,...) at sched_switch+0x512/frame 0xedf90b08 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf90b50 sleepq_switch(c93ce000,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xedf90b88 sleepq_wait(c1b77c74,54,c165e4da,0,0,...) at sleepq_wait+0x48/frame 0xedf90bac _sleep(c1b77c74,c1bdac00,54,c165e4da,0,...) at _sleep+0x38f/frame 0xedf90c00 vm_pageout_laundry_worker(0,edf90ce8,c1601280,406,0,...) at vm_pageout_laundry_worker+0x37e/frame 0xedf90cac fork_exit(c0f9a410,0,edf90ce8) at fork_exit+0x7e/frame 0xedf90cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf90cd4 --- trap 0, eip = 0, esp = 0xedf90d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100095 td 0xc93cda20 sched_switch(c93cda20,0,104,18a,edf94ba4,...) at sched_switch+0x512/frame 0xedf94b80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedf94bc4 sleepq_switch(c93cda20,0,c1610fcd,2ae,c1b7788c,...) at sleepq_switch+0x11c/frame 0xedf94bfc sleepq_wait(c1b778a0,54,edf94c50,0,0,...) at sleepq_wait+0x48/frame 0xedf94c20 _sleep(c1b778a0,c1b7788c,54,c165ac25,0,...) at _sleep+0x38f/frame 0xedf94c74 uma_reclaim_worker(0,edf94ce8,c1601280,406,0,...) at uma_reclaim_worker+0xaa/frame 0xedf94cac fork_exit(c0f795f0,0,edf94ce8) at fork_exit+0x7e/frame 0xedf94cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf94cd4 --- trap 0, eip = 0, esp = 0xedf94d20, ebp = 0 --- Tracing command rand_harvestq pid 16 tid 100086 td 0xc93cea20 sched_switch(c93cea20,0,104,18a,c93cea20,...) at sched_switch+0x512/frame 0xeddf4b38 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddf4b80 sleepq_switch(c93cea20,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xeddf4bb8 sleepq_timedwait(c19cf768,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xeddf4bf4 _sleep(c19cf768,0,0,c15f0e1b,19999999,...) at _sleep+0x363/frame 0xeddf4c48 random_kthread(0,eddf4ce8,c1601280,406,0,...) at random_kthread+0x7c/frame 0xeddf4cac fork_exit(c096b270,0,eddf4ce8) at fork_exit+0x7e/frame 0xeddf4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf4cd4 --- trap 0, eip = 0, esp = 0xeddf4d20, ebp = 0 --- Tracing command sctp_iterator pid 15 tid 100085 td 0xc93cf000 sched_switch(c93cf000,0,104,18a,c93cf000,...) at sched_switch+0x512/frame 0xeddf0b88 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddf0bd0 sleepq_switch(c93cf000,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddf0c08 sleepq_wait(c1bd67f4,0,c16391e7,0,0,...) at sleepq_wait+0x48/frame 0xeddf0c2c _sleep(c1bd67f4,c1bd67bc,0,c16391e7,0,...) at _sleep+0x38f/frame 0xeddf0c80 sctp_iterator_thread(0,eddf0ce8,c1601280,406,0,...) at sctp_iterator_thread+0x7c/frame 0xeddf0cac fork_exit(c0dedca0,0,eddf0ce8) at fork_exit+0x7e/frame 0xeddf0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf0cd4 --- trap 0, eip = 0, esp = 0xeddf0d20, ebp = 0 --- Tracing command soaiod4 pid 9 tid 100084 td 0xc92b36c0 sched_switch(c92b36c0,0,104,18a,c92b36c0,...) at sched_switch+0x512/frame 0xeddebb60 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddebba8 sleepq_switch(c92b36c0,0,c1610fcd,2d1,1a30,...) at sleepq_switch+0x11c/frame 0xeddebbe0 sleepq_timedwait(c1b704dc,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xeddebc1c _sleep(c1b704dc,c1b704bc,0,c15f0e1b,ffffdd50,...) at _sleep+0x363/frame 0xeddebc70 soaio_kproc_loop(4,eddebce8,c1601280,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xeddebcac fork_exit(c0ccc9a0,4,eddebce8) at fork_exit+0x7e/frame 0xeddebcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddebcd4 --- trap 0, eip = 0, esp = 0xeddebd20, ebp = 0 --- Tracing command soaiod3 pid 8 tid 100083 td 0xc92b3a20 sched_switch(c92b3a20,0,104,18a,c92b3a20,...) at sched_switch+0x512/frame 0xedde7b60 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedde7ba8 sleepq_switch(c92b3a20,0,c1610fcd,2d1,1a2e,...) at sleepq_switch+0x11c/frame 0xedde7be0 sleepq_timedwait(c1b704dc,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedde7c1c _sleep(c1b704dc,c1b704bc,0,c15f0e1b,ffffdd50,...) at _sleep+0x363/frame 0xedde7c70 soaio_kproc_loop(3,edde7ce8,c1601280,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedde7cac fork_exit(c0ccc9a0,3,edde7ce8) at fork_exit+0x7e/frame 0xedde7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedde7cd4 --- trap 0, eip = 0, esp = 0xedde7d20, ebp = 0 --- Tracing command soaiod2 pid 7 tid 100082 td 0xc8f246c0 sched_switch(c8f246c0,0,104,18a,c8f246c0,...) at sched_switch+0x512/frame 0xedde3b60 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedde3ba8 sleepq_switch(c8f246c0,0,c1610fcd,2d1,1a2f,...) at sleepq_switch+0x11c/frame 0xedde3be0 sleepq_timedwait(c1b704dc,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedde3c1c _sleep(c1b704dc,c1b704bc,0,c15f0e1b,ffffdd50,...) at _sleep+0x363/frame 0xedde3c70 soaio_kproc_loop(2,edde3ce8,c1601280,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedde3cac fork_exit(c0ccc9a0,2,edde3ce8) at fork_exit+0x7e/frame 0xedde3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedde3cd4 --- trap 0, eip = 0, esp = 0xedde3d20, ebp = 0 --- Tracing command soaiod1 pid 6 tid 100081 td 0xc8f24a20 sched_switch(c8f24a20,0,104,18a,c8f24a20,...) at sched_switch+0x512/frame 0xedddfb60 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedddfba8 sleepq_switch(c8f24a20,0,c1610fcd,2d1,1a2f,...) at sleepq_switch+0x11c/frame 0xedddfbe0 sleepq_timedwait(c1b704dc,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedddfc1c _sleep(c1b704dc,c1b704bc,0,c15f0e1b,ffffdd50,...) at _sleep+0x363/frame 0xedddfc70 soaio_kproc_loop(1,edddfce8,c1601280,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedddfcac fork_exit(c0ccc9a0,1,edddfce8) at fork_exit+0x7e/frame 0xedddfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedddfcd4 --- trap 0, eip = 0, esp = 0xedddfd20, ebp = 0 --- Tracing command fdc0 pid 5 tid 100077 td 0xc91c16c0 sched_switch(c91c16c0,0,104,18a,c91c16c0,...) at sched_switch+0x512/frame 0xeddcfb30 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddcfb78 sleepq_switch(c91c16c0,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddcfbb0 sleepq_wait(c9174a3c,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xeddcfbd4 _sleep(c9174a3c,c9174af0,5c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xeddcfc28 fdc_thread(c9174a00,eddcfce8,c1601280,406,0,...) at fdc_thread+0x14bc/frame 0xeddcfcac fork_exit(c11741e0,c9174a00,eddcfce8) at fork_exit+0x7e/frame 0xeddcfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddcfcd4 --- trap 0, eip = 0, esp = 0xeddcfd20, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc910ea20 sched_switch(c910ea20,0,104,18a,c910ea20,...) at sched_switch+0x512/frame 0xedd23b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd23be0 sleepq_switch(c910ea20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd23c18 sleepq_wait(c90eaacc,0,edd23c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd23c3c _cv_wait(c90eaacc,c90eacf8,c15d31b6,76,c90eaad4,...) at _cv_wait+0x21f/frame 0xedd23c80 usb_process(c90eaac4,edd23ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd23cac fork_exit(c0a98e00,c90eaac4,edd23ce8) at fork_exit+0x7e/frame 0xedd23cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd23cd4 --- trap 0, eip = 0, esp = 0xedd23d20, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc910e6c0 sched_switch(c910e6c0,0,104,18a,c910e6c0,...) at sched_switch+0x512/frame 0xedd27b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd27be0 sleepq_switch(c910e6c0,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd27c18 sleepq_wait(c90eaafc,0,edd27c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd27c3c _cv_wait(c90eaafc,c90eacf8,c15d31b6,76,c90eab04,...) at _cv_wait+0x21f/frame 0xedd27c80 usb_process(c90eaaf4,edd27ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd27cac fork_exit(c0a98e00,c90eaaf4,edd27ce8) at fork_exit+0x7e/frame 0xedd27cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd27cd4 --- trap 0, eip = 0, esp = 0xedd27d20, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc910e360 sched_switch(c910e360,0,104,18a,c910e360,...) at sched_switch+0x512/frame 0xedd2bb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd2bbe0 sleepq_switch(c910e360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd2bc18 sleepq_wait(c90eab2c,0,edd2bc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd2bc3c _cv_wait(c90eab2c,c90eacf8,c15d31b6,76,c90eab34,...) at _cv_wait+0x21f/frame 0xedd2bc80 usb_process(c90eab24,edd2bce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd2bcac fork_exit(c0a98e00,c90eab24,edd2bce8) at fork_exit+0x7e/frame 0xedd2bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2bcd4 --- trap 0, eip = 0, esp = 0xedd2bd20, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc910e000 sched_switch(c910e000,0,104,18a,c910e000,...) at sched_switch+0x512/frame 0xedd2fb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd2fbe0 sleepq_switch(c910e000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd2fc18 sleepq_wait(c90eab5c,0,edd2fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd2fc3c _cv_wait(c90eab5c,c90eacf8,c15d31b6,76,c90eab64,...) at _cv_wait+0x21f/frame 0xedd2fc80 usb_process(c90eab54,edd2fce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd2fcac fork_exit(c0a98e00,c90eab54,edd2fce8) at fork_exit+0x7e/frame 0xedd2fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2fcd4 --- trap 0, eip = 0, esp = 0xedd2fd20, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc90fda20 sched_switch(c90fda20,0,104,18a,c90fda20,...) at sched_switch+0x512/frame 0xedd33b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd33be0 sleepq_switch(c90fda20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd33c18 sleepq_wait(c90eab8c,0,edd33c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd33c3c _cv_wait(c90eab8c,c90eacf8,c15d31b6,76,c90eab94,...) at _cv_wait+0x21f/frame 0xedd33c80 usb_process(c90eab84,edd33ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd33cac fork_exit(c0a98e00,c90eab84,edd33ce8) at fork_exit+0x7e/frame 0xedd33cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd33cd4 --- trap 0, eip = 0, esp = 0xedd33d20, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc90fd360 sched_switch(c90fd360,0,104,18a,c90fd360,...) at sched_switch+0x512/frame 0xedd3cb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd3cbe0 sleepq_switch(c90fd360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd3cc18 sleepq_wait(c9156acc,0,edd3cc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd3cc3c _cv_wait(c9156acc,c9156cf8,c15d31b6,76,c9156ad4,...) at _cv_wait+0x21f/frame 0xedd3cc80 usb_process(c9156ac4,edd3cce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd3ccac fork_exit(c0a98e00,c9156ac4,edd3cce8) at fork_exit+0x7e/frame 0xedd3ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd3ccd4 --- trap 0, eip = 0, esp = 0xedd3cd20, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc90fd000 sched_switch(c90fd000,0,104,18a,c90fd000,...) at sched_switch+0x512/frame 0xedd40b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd40be0 sleepq_switch(c90fd000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd40c18 sleepq_wait(c9156afc,0,edd40c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd40c3c _cv_wait(c9156afc,c9156cf8,c15d31b6,76,c9156b04,...) at _cv_wait+0x21f/frame 0xedd40c80 usb_process(c9156af4,edd40ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd40cac fork_exit(c0a98e00,c9156af4,edd40ce8) at fork_exit+0x7e/frame 0xedd40cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd40cd4 --- trap 0, eip = 0, esp = 0xedd40d20, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc8f49a20 sched_switch(c8f49a20,0,104,18a,c8f49a20,...) at sched_switch+0x512/frame 0xedd44b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd44be0 sleepq_switch(c8f49a20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd44c18 sleepq_wait(c9156b2c,0,edd44c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd44c3c _cv_wait(c9156b2c,c9156cf8,c15d31b6,76,c9156b34,...) at _cv_wait+0x21f/frame 0xedd44c80 usb_process(c9156b24,edd44ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd44cac fork_exit(c0a98e00,c9156b24,edd44ce8) at fork_exit+0x7e/frame 0xedd44cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd44cd4 --- trap 0, eip = 0, esp = 0xedd44d20, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc915c000 sched_switch(c915c000,0,104,18a,c915c000,...) at sched_switch+0x512/frame 0xedd48b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd48be0 sleepq_switch(c915c000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd48c18 sleepq_wait(c9156b5c,0,edd48c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd48c3c _cv_wait(c9156b5c,c9156cf8,c15d31b6,76,c9156b64,...) at _cv_wait+0x21f/frame 0xedd48c80 usb_process(c9156b54,edd48ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd48cac fork_exit(c0a98e00,c9156b54,edd48ce8) at fork_exit+0x7e/frame 0xedd48cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd48cd4 --- trap 0, eip = 0, esp = 0xedd48d20, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc915ba20 sched_switch(c915ba20,0,104,18a,c915ba20,...) at sched_switch+0x512/frame 0xedd4cb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd4cbe0 sleepq_switch(c915ba20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd4cc18 sleepq_wait(c9156b8c,0,edd4cc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd4cc3c _cv_wait(c9156b8c,c9156cf8,c15d31b6,76,c9156b94,...) at _cv_wait+0x21f/frame 0xedd4cc80 usb_process(c9156b84,edd4cce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd4ccac fork_exit(c0a98e00,c9156b84,edd4cce8) at fork_exit+0x7e/frame 0xedd4ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd4ccd4 --- trap 0, eip = 0, esp = 0xedd4cd20, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc915b360 sched_switch(c915b360,0,104,18a,c915b360,...) at sched_switch+0x512/frame 0xedd55b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd55be0 sleepq_switch(c915b360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd55c18 sleepq_wait(c915dacc,0,edd55c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd55c3c _cv_wait(c915dacc,c915dcf8,c15d31b6,76,c915dad4,...) at _cv_wait+0x21f/frame 0xedd55c80 usb_process(c915dac4,edd55ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd55cac fork_exit(c0a98e00,c915dac4,edd55ce8) at fork_exit+0x7e/frame 0xedd55cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd55cd4 --- trap 0, eip = 0, esp = 0xedd55d20, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc915b000 sched_switch(c915b000,0,104,18a,c915b000,...) at sched_switch+0x512/frame 0xedd59b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd59be0 sleepq_switch(c915b000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd59c18 sleepq_wait(c915dafc,0,edd59c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd59c3c _cv_wait(c915dafc,c915dcf8,c15d31b6,76,c915db04,...) at _cv_wait+0x21f/frame 0xedd59c80 usb_process(c915daf4,edd59ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd59cac fork_exit(c0a98e00,c915daf4,edd59ce8) at fork_exit+0x7e/frame 0xedd59cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd59cd4 --- trap 0, eip = 0, esp = 0xedd59d20, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc915aa20 sched_switch(c915aa20,0,104,18a,c915aa20,...) at sched_switch+0x512/frame 0xedd5db98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd5dbe0 sleepq_switch(c915aa20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd5dc18 sleepq_wait(c915db2c,0,edd5dc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd5dc3c _cv_wait(c915db2c,c915dcf8,c15d31b6,76,c915db34,...) at _cv_wait+0x21f/frame 0xedd5dc80 usb_process(c915db24,edd5dce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd5dcac fork_exit(c0a98e00,c915db24,edd5dce8) at fork_exit+0x7e/frame 0xedd5dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd5dcd4 --- trap 0, eip = 0, esp = 0xedd5dd20, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc915a6c0 sched_switch(c915a6c0,0,104,18a,c915a6c0,...) at sched_switch+0x512/frame 0xedd61b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd61be0 sleepq_switch(c915a6c0,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd61c18 sleepq_wait(c915db5c,0,edd61c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd61c3c _cv_wait(c915db5c,c915dcf8,c15d31b6,76,c915db64,...) at _cv_wait+0x21f/frame 0xedd61c80 usb_process(c915db54,edd61ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd61cac fork_exit(c0a98e00,c915db54,edd61ce8) at fork_exit+0x7e/frame 0xedd61cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd61cd4 --- trap 0, eip = 0, esp = 0xedd61d20, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xc915a360 sched_switch(c915a360,0,104,18a,c915a360,...) at sched_switch+0x512/frame 0xedd65b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd65be0 sleepq_switch(c915a360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd65c18 sleepq_wait(c915db8c,0,edd65c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd65c3c _cv_wait(c915db8c,c915dcf8,c15d31b6,76,c915db94,...) at _cv_wait+0x21f/frame 0xedd65c80 usb_process(c915db84,edd65ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd65cac fork_exit(c0a98e00,c915db84,edd65ce8) at fork_exit+0x7e/frame 0xedd65cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd65cd4 --- trap 0, eip = 0, esp = 0xedd65d20, ebp = 0 --- Tracing command usb pid 14 tid 100059 td 0xc915a000 sched_switch(c915a000,0,104,18a,c915a000,...) at sched_switch+0x512/frame 0xedd6ab98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd6abe0 sleepq_switch(c915a000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd6ac18 sleepq_wait(c9164acc,0,edd6ac5c,1,0,...) at sleepq_wait+0x48/frame 0xedd6ac3c _cv_wait(c9164acc,c9164cf8,c15d31b6,76,c9164ad4,...) at _cv_wait+0x21f/frame 0xedd6ac80 usb_process(c9164ac4,edd6ace8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd6acac fork_exit(c0a98e00,c9164ac4,edd6ace8) at fork_exit+0x7e/frame 0xedd6acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd6acd4 --- trap 0, eip = 0, esp = 0xedd6ad20, ebp = 0 --- Tracing command usb pid 14 tid 100060 td 0xc9159a20 sched_switch(c9159a20,0,104,18a,c9159a20,...) at sched_switch+0x512/frame 0xedd6eb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd6ebe0 sleepq_switch(c9159a20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd6ec18 sleepq_wait(c9164afc,0,edd6ec5c,1,0,...) at sleepq_wait+0x48/frame 0xedd6ec3c _cv_wait(c9164afc,c9164cf8,c15d31b6,76,c9164b04,...) at _cv_wait+0x21f/frame 0xedd6ec80 usb_process(c9164af4,edd6ece8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd6ecac fork_exit(c0a98e00,c9164af4,edd6ece8) at fork_exit+0x7e/frame 0xedd6ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd6ecd4 --- trap 0, eip = 0, esp = 0xedd6ed20, ebp = 0 --- Tracing command usb pid 14 tid 100061 td 0xc91596c0 sched_switch(c91596c0,0,104,18a,c91596c0,...) at sched_switch+0x512/frame 0xedd72b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd72be0 sleepq_switch(c91596c0,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd72c18 sleepq_wait(c9164b2c,0,edd72c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd72c3c _cv_wait(c9164b2c,c9164cf8,c15d31b6,76,c9164b34,...) at _cv_wait+0x21f/frame 0xedd72c80 usb_process(c9164b24,edd72ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd72cac fork_exit(c0a98e00,c9164b24,edd72ce8) at fork_exit+0x7e/frame 0xedd72cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd72cd4 --- trap 0, eip = 0, esp = 0xedd72d20, ebp = 0 --- Tracing command usb pid 14 tid 100062 td 0xc9159360 sched_switch(c9159360,0,104,18a,c9159360,...) at sched_switch+0x512/frame 0xedd76b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd76be0 sleepq_switch(c9159360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd76c18 sleepq_wait(c9164b5c,0,edd76c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd76c3c _cv_wait(c9164b5c,c9164cf8,c15d31b6,76,c9164b64,...) at _cv_wait+0x21f/frame 0xedd76c80 usb_process(c9164b54,edd76ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd76cac fork_exit(c0a98e00,c9164b54,edd76ce8) at fork_exit+0x7e/frame 0xedd76cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd76cd4 --- trap 0, eip = 0, esp = 0xedd76d20, ebp = 0 --- Tracing command usb pid 14 tid 100063 td 0xc9159000 sched_switch(c9159000,0,104,18a,c9159000,...) at sched_switch+0x512/frame 0xedd7ab98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd7abe0 sleepq_switch(c9159000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd7ac18 sleepq_wait(c9164b8c,0,edd7ac5c,1,0,...) at sleepq_wait+0x48/frame 0xedd7ac3c _cv_wait(c9164b8c,c9164cf8,c15d31b6,76,c9164b94,...) at _cv_wait+0x21f/frame 0xedd7ac80 usb_process(c9164b84,edd7ace8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd7acac fork_exit(c0a98e00,c9164b84,edd7ace8) at fork_exit+0x7e/frame 0xedd7acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd7acd4 --- trap 0, eip = 0, esp = 0xedd7ad20, ebp = 0 --- Tracing command usb pid 14 tid 100064 td 0xc9171360 sched_switch(c9171360,0,104,18a,c9171360,...) at sched_switch+0x512/frame 0xedd7fb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd7fbe0 sleepq_switch(c9171360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd7fc18 sleepq_wait(c9169acc,0,edd7fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd7fc3c _cv_wait(c9169acc,c9169cf8,c15d31b6,76,c9169ad4,...) at _cv_wait+0x21f/frame 0xedd7fc80 usb_process(c9169ac4,edd7fce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd7fcac fork_exit(c0a98e00,c9169ac4,edd7fce8) at fork_exit+0x7e/frame 0xedd7fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd7fcd4 --- trap 0, eip = 0, esp = 0xedd7fd20, ebp = 0 --- Tracing command usb pid 14 tid 100065 td 0xc9171000 sched_switch(c9171000,0,104,18a,c9171000,...) at sched_switch+0x512/frame 0xedd83b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd83be0 sleepq_switch(c9171000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd83c18 sleepq_wait(c9169afc,0,edd83c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd83c3c _cv_wait(c9169afc,c9169cf8,c15d31b6,76,c9169b04,...) at _cv_wait+0x21f/frame 0xedd83c80 usb_process(c9169af4,edd83ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd83cac fork_exit(c0a98e00,c9169af4,edd83ce8) at fork_exit+0x7e/frame 0xedd83cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd83cd4 --- trap 0, eip = 0, esp = 0xedd83d20, ebp = 0 --- Tracing command usb pid 14 tid 100066 td 0xc9170a20 sched_switch(c9170a20,0,104,18a,c9170a20,...) at sched_switch+0x512/frame 0xedd87b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd87be0 sleepq_switch(c9170a20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd87c18 sleepq_wait(c9169b2c,0,edd87c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd87c3c _cv_wait(c9169b2c,c9169cf8,c15d31b6,76,c9169b34,...) at _cv_wait+0x21f/frame 0xedd87c80 usb_process(c9169b24,edd87ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd87cac fork_exit(c0a98e00,c9169b24,edd87ce8) at fork_exit+0x7e/frame 0xedd87cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd87cd4 --- trap 0, eip = 0, esp = 0xedd87d20, ebp = 0 --- Tracing command usb pid 14 tid 100067 td 0xc91706c0 sched_switch(c91706c0,0,104,18a,c91706c0,...) at sched_switch+0x512/frame 0xedd8bb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd8bbe0 sleepq_switch(c91706c0,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd8bc18 sleepq_wait(c9169b5c,0,edd8bc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd8bc3c _cv_wait(c9169b5c,c9169cf8,c15d31b6,76,c9169b64,...) at _cv_wait+0x21f/frame 0xedd8bc80 usb_process(c9169b54,edd8bce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd8bcac fork_exit(c0a98e00,c9169b54,edd8bce8) at fork_exit+0x7e/frame 0xedd8bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd8bcd4 --- trap 0, eip = 0, esp = 0xedd8bd20, ebp = 0 --- Tracing command usb pid 14 tid 100068 td 0xc9170360 sched_switch(c9170360,0,104,18a,c9170360,...) at sched_switch+0x512/frame 0xedd8fb98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd8fbe0 sleepq_switch(c9170360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd8fc18 sleepq_wait(c9169b8c,0,edd8fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd8fc3c _cv_wait(c9169b8c,c9169cf8,c15d31b6,76,c9169b94,...) at _cv_wait+0x21f/frame 0xedd8fc80 usb_process(c9169b84,edd8fce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd8fcac fork_exit(c0a98e00,c9169b84,edd8fce8) at fork_exit+0x7e/frame 0xedd8fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd8fcd4 --- trap 0, eip = 0, esp = 0xedd8fd20, ebp = 0 --- Tracing command usb pid 14 tid 100070 td 0xc916fa20 sched_switch(c916fa20,0,104,18a,c916fa20,...) at sched_switch+0x512/frame 0xedd99b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd99be0 sleepq_switch(c916fa20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd99c18 sleepq_wait(c90d5ca4,0,edd99c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd99c3c _cv_wait(c90d5ca4,c90d5ed0,c15d31b6,76,c90d5cac,...) at _cv_wait+0x21f/frame 0xedd99c80 usb_process(c90d5c9c,edd99ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd99cac fork_exit(c0a98e00,c90d5c9c,edd99ce8) at fork_exit+0x7e/frame 0xedd99cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd99cd4 --- trap 0, eip = 0, esp = 0xedd99d20, ebp = 0 --- Tracing command usb pid 14 tid 100071 td 0xc916f6c0 sched_switch(c916f6c0,0,104,18a,c916f6c0,...) at sched_switch+0x512/frame 0xedd9db98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedd9dbe0 sleepq_switch(c916f6c0,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedd9dc18 sleepq_wait(c90d5cd4,0,edd9dc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd9dc3c _cv_wait(c90d5cd4,c90d5ed0,c15d31b6,76,c90d5cdc,...) at _cv_wait+0x21f/frame 0xedd9dc80 usb_process(c90d5ccc,edd9dce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedd9dcac fork_exit(c0a98e00,c90d5ccc,edd9dce8) at fork_exit+0x7e/frame 0xedd9dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd9dcd4 --- trap 0, eip = 0, esp = 0xedd9dd20, ebp = 0 --- Tracing command usb pid 14 tid 100072 td 0xc916f360 sched_switch(c916f360,0,104,18a,c916f360,...) at sched_switch+0x512/frame 0xedda1b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedda1be0 sleepq_switch(c916f360,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedda1c18 sleepq_wait(c90d5d04,0,edda1c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda1c3c _cv_wait(c90d5d04,c90d5ed0,c15d31b6,76,c90d5d0c,...) at _cv_wait+0x21f/frame 0xedda1c80 usb_process(c90d5cfc,edda1ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedda1cac fork_exit(c0a98e00,c90d5cfc,edda1ce8) at fork_exit+0x7e/frame 0xedda1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda1cd4 --- trap 0, eip = 0, esp = 0xedda1d20, ebp = 0 --- Tracing command usb pid 14 tid 100073 td 0xc916f000 sched_switch(c916f000,0,104,18a,c916f000,...) at sched_switch+0x512/frame 0xedda5b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedda5be0 sleepq_switch(c916f000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedda5c18 sleepq_wait(c90d5d34,0,edda5c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda5c3c _cv_wait(c90d5d34,c90d5ed0,c15d31b6,76,c90d5d3c,...) at _cv_wait+0x21f/frame 0xedda5c80 usb_process(c90d5d2c,edda5ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedda5cac fork_exit(c0a98e00,c90d5d2c,edda5ce8) at fork_exit+0x7e/frame 0xedda5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda5cd4 --- trap 0, eip = 0, esp = 0xedda5d20, ebp = 0 --- Tracing command usb pid 14 tid 100074 td 0xc915ca20 sched_switch(c915ca20,0,104,18a,c915ca20,...) at sched_switch+0x512/frame 0xedda9b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedda9be0 sleepq_switch(c915ca20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xedda9c18 sleepq_wait(c90d5d64,0,edda9c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda9c3c _cv_wait(c90d5d64,c90d5ed0,c15d31b6,76,c90d5d6c,...) at _cv_wait+0x21f/frame 0xedda9c80 usb_process(c90d5d5c,edda9ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xedda9cac fork_exit(c0a98e00,c90d5d5c,edda9ce8) at fork_exit+0x7e/frame 0xedda9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda9cd4 --- trap 0, eip = 0, esp = 0xedda9d20, ebp = 0 --- Tracing command usb pid 14 tid 100117 td 0xc9b16a20 sched_switch(c9b16a20,0,104,18a,c9b16a20,...) at sched_switch+0x512/frame 0xf34d3b98 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xf34d3be0 sleepq_switch(c9b16a20,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xf34d3c18 sleepq_wait(c8e23408,0,f34d3c5c,1,0,...) at sleepq_wait+0x48/frame 0xf34d3c3c _cv_wait(c8e23408,c8e23734,c15d31b6,76,c8e23410,...) at _cv_wait+0x21f/frame 0xf34d3c80 usb_process(c8e23400,f34d3ce8,c1601280,406,0,...) at usb_process+0x190/frame 0xf34d3cac fork_exit(c0a98e00,c8e23400,f34d3ce8) at fork_exit+0x7e/frame 0xf34d3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf34d3cd4 --- trap 0, eip = 0, esp = 0xf34d3d20, ebp = 0 --- Tracing command cam pid 4 tid 100037 td 0xc8f47000 sched_switch(c8f47000,0,104,18a,c8f47000,...) at sched_switch+0x512/frame 0xe4bbfb68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4bbfbb0 sleepq_switch(c8f47000,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4bbfbe8 sleepq_wait(c191bc00,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4bbfc0c _sleep(c191bc00,c191bb80,5c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4bbfc60 xpt_done_td(c191bb80,e4bbfce8,c1601280,406,0,...) at xpt_done_td+0xee/frame 0xe4bbfcac fork_exit(c04e2870,c191bb80,e4bbfce8) at fork_exit+0x7e/frame 0xe4bbfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bbfcd4 --- trap 0, eip = 0, esp = 0xe4bbfd20, ebp = 0 --- Tracing command cam pid 4 tid 100088 td 0xc93ce360 sched_switch(c93ce360,0,104,18a,c93ce360,...) at sched_switch+0x512/frame 0xeddfcb60 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddfcba8 sleepq_switch(c93ce360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddfcbe0 sleepq_wait(c191baac,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xeddfcc04 _sleep(c191baac,c191bb00,5c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xeddfcc58 xpt_scanner_thread(0,eddfcce8,c1601280,406,0,...) at xpt_scanner_thread+0x1b2/frame 0xeddfccac fork_exit(c04e2c30,0,eddfcce8) at fork_exit+0x7e/frame 0xeddfccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddfccd4 --- trap 0, eip = 0, esp = 0xeddfcd20, ebp = 0 --- Tracing command crypto returns pid 3 tid 100033 td 0xc8f48000 sched_switch(c8f48000,0,104,18a,c8f48000,...) at sched_switch+0x512/frame 0xe4bafb68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4bafbb0 sleepq_switch(c8f48000,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4bafbe8 sleepq_wait(c1b7683c,6c,c164d400,0,0,...) at sleepq_wait+0x48/frame 0xe4bafc0c _sleep(c1b7683c,c1b76828,6c,c164d400,0,...) at _sleep+0x38f/frame 0xe4bafc60 crypto_ret_proc(0,e4bafce8,c1601280,406,0,...) at crypto_ret_proc+0x1f5/frame 0xe4bafcac fork_exit(c0ee15a0,0,e4bafce8) at fork_exit+0x7e/frame 0xe4bafcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bafcd4 --- trap 0, eip = 0, esp = 0xe4bafd20, ebp = 0 --- Tracing command crypto pid 2 tid 100032 td 0xc8f48360 sched_switch(c8f48360,0,104,18a,e4babb9c,...) at sched_switch+0x512/frame 0xe4babb78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4babbbc sleepq_switch(c8f48360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4babbf4 sleepq_wait(c1b76784,6c,c164d3f4,0,0,...) at sleepq_wait+0x48/frame 0xe4babc18 _sleep(c1b76784,c1b7676c,6c,c164d3f4,0,...) at _sleep+0x38f/frame 0xe4babc6c crypto_proc(0,e4babce8,c1601280,406,0,...) at crypto_proc+0x9d/frame 0xe4babcac fork_exit(c0ee11a0,0,e4babce8) at fork_exit+0x7e/frame 0xe4babcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4babcd4 --- trap 0, eip = 0, esp = 0xe4babd20, ebp = 0 --- Tracing command geom pid 13 tid 100028 td 0xc8f49360 sched_switch(c8f49360,0,104,18a,c8f49360,...) at sched_switch+0x512/frame 0xe4b9bb78 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b9bbc0 sleepq_switch(c8f49360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b9bbf8 sleepq_wait(c1bd18c4,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4b9bc1c _sleep(c1bd18c4,c1af41f8,25c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4b9bc70 g_run_events(0,e4b9bce8,c1601280,406,0,...) at g_run_events+0x141/frame 0xe4b9bcac fork_exit(c0b9d960,0,e4b9bce8) at fork_exit+0x7e/frame 0xe4b9bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b9bcd4 --- trap 0, eip = 0, esp = 0xe4b9bd20, ebp = 0 --- Tracing command geom pid 13 tid 100029 td 0xc8f49000 sched_switch(c8f49000,0,104,18a,c8f49000,...) at sched_switch+0x512/frame 0xe4b9fb68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b9fbb0 sleepq_switch(c8f49000,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b9fbe8 sleepq_wait(c1bd18c8,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4b9fc0c _sleep(c1bd18c8,c1af4240,25c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4b9fc60 g_io_schedule_up(c8f49000,0,c15f2e52,5f,e4b9fcd4,...) at g_io_schedule_up+0xcf/frame 0xe4b9fc94 g_up_procbody(0,e4b9fce8,c1601280,406,0,...) at g_up_procbody+0x7d/frame 0xe4b9fcac fork_exit(c0b9d9e0,0,e4b9fce8) at fork_exit+0x7e/frame 0xe4b9fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b9fcd4 --- trap 0, eip = 0, esp = 0xe4b9fd20, ebp = 0 --- Tracing command geom pid 13 tid 100030 td 0xc8f48a20 sched_switch(c8f48a20,0,104,18a,e4ba3b8c,...) at sched_switch+0x512/frame 0xe4ba3b68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4ba3bac sleepq_switch(c8f48a20,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4ba3be4 sleepq_wait(c1bd18cc,5c,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4ba3c08 _sleep(c1bd18cc,c1af4220,25c,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4ba3c5c g_io_schedule_down(c8f48a20,0,c15f2e52,6b,e4ba3cd4,...) at g_io_schedule_down+0xec/frame 0xe4ba3c94 g_down_procbody(0,e4ba3ce8,c1601280,406,0,...) at g_down_procbody+0x7d/frame 0xe4ba3cac fork_exit(c0b9da60,0,e4ba3ce8) at fork_exit+0x7e/frame 0xe4ba3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4ba3cd4 --- trap 0, eip = 0, esp = 0xe4ba3d20, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc8e666c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc8e66360 sched_switch(c8e66360,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b4bc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xe4b4bc70 ithread_loop(c8d01d90,e4b4bce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xe4b4bcac fork_exit(c0c1a910,c8d01d90,e4b4bce8) at fork_exit+0x7e/frame 0xe4b4bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b4bcd4 --- trap 0, eip = 0, esp = 0xe4b4bd20, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc8e66000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc8e5ba20 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc8e5b6c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc8e5b360 sched_switch(c8e5b360,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b5bc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xe4b5bc70 ithread_loop(c8d01d50,e4b5bce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xe4b5bcac fork_exit(c0c1a910,c8d01d50,e4b5bce8) at fork_exit+0x7e/frame 0xe4b5bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5bcd4 --- trap 0, eip = 0, esp = 0xe4b5bd20, ebp = 0 --- Tracing command intr pid 12 tid 100023 td 0xc8f22000 sched_switch(c8f22000,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b87c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xe4b87c70 ithread_loop(c8d01d40,e4b87ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xe4b87cac fork_exit(c0c1a910,c8d01d40,e4b87ce8) at fork_exit+0x7e/frame 0xe4b87cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b87cd4 --- trap 0, eip = 0, esp = 0xe4b87d20, ebp = 0 --- Tracing command intr pid 12 tid 100024 td 0xc8e68a20 sched_switch(c8e68a20,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b8bc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xe4b8bc70 ithread_loop(c8d01d30,e4b8bce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xe4b8bcac fork_exit(c0c1a910,c8d01d30,e4b8bce8) at fork_exit+0x7e/frame 0xe4b8bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b8bcd4 --- trap 0, eip = 0, esp = 0xe4b8bd20, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc8f496c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xc910fa20 sched_switch(c910fa20,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4bf2c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xe4bf2c70 ithread_loop(c9085d40,e4bf2ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xe4bf2cac fork_exit(c0c1a910,c9085d40,e4bf2ce8) at fork_exit+0x7e/frame 0xe4bf2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bf2cd4 --- trap 0, eip = 0, esp = 0xe4bf2d20, ebp = 0 --- Tracing command intr pid 12 tid 100039 td 0xc910f6c0 sched_switch(c910f6c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xf0efbc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xf0efbc70 ithread_loop(c9085780,f0efbce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xf0efbcac fork_exit(c0c1a910,c9085780,f0efbce8) at fork_exit+0x7e/frame 0xf0efbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0efbcd4 --- trap 0, eip = 0, esp = 0xf0efbd20, ebp = 0 --- Tracing command intr pid 12 tid 100040 td 0xc910f360 sched_switch(c910f360,0,109,18a,0,...) at sched_switch+0x512/frame 0xf0f00c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xf0f00c70 ithread_loop(c90853e0,f0f00ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xf0f00cac fork_exit(c0c1a910,c90853e0,f0f00ce8) at fork_exit+0x7e/frame 0xf0f00cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0f00cd4 --- trap 0, eip = 0, esp = 0xf0f00d20, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc910f000 sched_switch(c910f000,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd1fc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xedd1fc70 ithread_loop(c90f5e60,edd1fce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xedd1fcac fork_exit(c0c1a910,c90f5e60,edd1fce8) at fork_exit+0x7e/frame 0xedd1fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd1fcd4 --- trap 0, eip = 0, esp = 0xedd1fd20, ebp = 0 --- Tracing command intr pid 12 tid 100047 td 0xc90fd6c0 sched_switch(c90fd6c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd38c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xedd38c70 ithread_loop(c90f5c10,edd38ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xedd38cac fork_exit(c0c1a910,c90f5c10,edd38ce8) at fork_exit+0x7e/frame 0xedd38cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd38cd4 --- trap 0, eip = 0, esp = 0xedd38d20, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc915b6c0 sched_switch(c915b6c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd51c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xedd51c70 ithread_loop(c90f5a20,edd51ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xedd51cac fork_exit(c0c1a910,c90f5a20,edd51ce8) at fork_exit+0x7e/frame 0xedd51cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd51cd4 --- trap 0, eip = 0, esp = 0xedd51d20, ebp = 0 --- Tracing command intr pid 12 tid 100069 td 0xc9170000 sched_switch(c9170000,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd95c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xedd95c70 ithread_loop(c90f5410,edd95ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xedd95cac fork_exit(c0c1a910,c90f5410,edd95ce8) at fork_exit+0x7e/frame 0xedd95cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd95cd4 --- trap 0, eip = 0, esp = 0xedd95d20, ebp = 0 --- Tracing command intr pid 12 tid 100075 td 0xc915c6c0 sched_switch(c915c6c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xeddafc28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xeddafc70 ithread_loop(c919e0d0,eddafce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xeddafcac fork_exit(c0c1a910,c919e0d0,eddafce8) at fork_exit+0x7e/frame 0xeddafcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddafcd4 --- trap 0, eip = 0, esp = 0xeddafd20, ebp = 0 --- Tracing command intr pid 12 tid 100076 td 0xc915c360 sched_switch(c915c360,0,109,18a,0,...) at sched_switch+0x512/frame 0xeddc1c28 mi_switch(109,0,c16015fd,59f,c1601280,...) at mi_switch+0x226/frame 0xeddc1c70 ithread_loop(c919ece0,eddc1ce8,c1601280,406,0,...) at ithread_loop+0x200/frame 0xeddc1cac fork_exit(c0c1a910,c919ece0,eddc1ce8) at fork_exit+0x7e/frame 0xeddc1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddc1cd4 --- trap 0, eip = 0, esp = 0xeddc1d20, ebp = 0 --- Tracing command intr pid 12 tid 100078 td 0xc91c1360 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100079 td 0xc91c1000 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc8e676c0 (CPU 0) cpustop_handler(0,13,e4b36b8c,c12095ee,1,...) at cpustop_handler+0x25/frame 0xe4b36a40 ipi_nmi_handler(1,0,a,c8eb3380,c1b9190c,...) at ipi_nmi_handler+0x36/frame 0xe4b36a50 trap(e4b36b98) at trap+0xae/frame 0xe4b36b8c calltrap() at calltrap+0x6/frame 0xe4b36b8c --- trap 0x13, eip = 0xc1224725, esp = 0xe4b36bd8, ebp = 0xe4b36bd8 --- acpi_cpu_c1(e4b36c2c,c1213a39,c2027c90,0,c18183c3,...) at acpi_cpu_c1+0x5/frame 0xe4b36bd8 acpi_cpu_idle(9c1d73b,0,9c1d73b,e4b36c48,c1224af6,...) at acpi_cpu_idle+0x325/frame 0xe4b36c18 cpu_idle_acpi(9c1d73b,0,c1b04700,1,c8e676c0,...) at cpu_idle_acpi+0x3f/frame 0xe4b36c2c cpu_idle(0,e4b36c80,c160ccd7,a6a,0,...) at cpu_idle+0xa6/frame 0xe4b36c48 sched_idletd(0,e4b36ce8,c1601280,406,0,...) at sched_idletd+0x16a/frame 0xe4b36cac fork_exit(c0c88250,0,e4b36ce8) at fork_exit+0x7e/frame 0xe4b36cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b36cd4 --- trap 0, eip = 0, esp = 0xe4b36d20, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc8e67360 (CPU 1) cpustop_handler(1,13,e4b3ab8c,c12095ee,a,...) at cpustop_handler+0x25/frame 0xe4b3aa40 ipi_nmi_handler(a,c160ba6d,c1b91c54,1,0,...) at ipi_nmi_handler+0x36/frame 0xe4b3aa50 trap(e4b3ab98) at trap+0xae/frame 0xe4b3ab8c calltrap() at calltrap+0x6/frame 0xe4b3ab8c --- trap 0x13, eip = 0xc1224725, esp = 0xe4b3abd8, ebp = 0xe4b3abd8 --- acpi_cpu_c1(e4b3ac2c,c1213a39,e4b18c90,0,c18183c3,...) at acpi_cpu_c1+0x5/frame 0xe4b3abd8 acpi_cpu_idle(7fd7e8a1,0,7fd7e8a1,e4b3ac48,c1224af6,...) at acpi_cpu_idle+0x325/frame 0xe4b3ac18 cpu_idle_acpi(7fd7e8a1,0,c1b04e00,3,c8e67360,...) at cpu_idle_acpi+0x3f/frame 0xe4b3ac2c cpu_idle(0,e4b3ac80,c160ccd7,407,0,...) at cpu_idle+0xa6/frame 0xe4b3ac48 sched_idletd(0,e4b3ace8,c1601280,406,0,...) at sched_idletd+0x16a/frame 0xe4b3acac fork_exit(c0c88250,0,e4b3ace8) at fork_exit+0x7e/frame 0xe4b3acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3acd4 --- trap 0, eip = 0, esp = 0xe4b3ad20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc8e67000 (CPU 2) cpustop_handler(2,13,e4b3eb8c,c12095ee,a,...) at cpustop_handler+0x25/frame 0xe4b3ea40 ipi_nmi_handler(a,c160ba6d,c1b91f9c,2,0,...) at ipi_nmi_handler+0x36/frame 0xe4b3ea50 trap(e4b3eb98) at trap+0xae/frame 0xe4b3eb8c calltrap() at calltrap+0x6/frame 0xe4b3eb8c --- trap 0x13, eip = 0xc1224725, esp = 0xe4b3ebd8, ebp = 0xe4b3ebd8 --- acpi_cpu_c1(e4b3ec2c,c1213a39,e4b1cc90,0,c18183c3,...) at acpi_cpu_c1+0x5/frame 0xe4b3ebd8 acpi_cpu_idle(254c99d4,0,254c99d4,e4b3ec48,c1224af6,...) at acpi_cpu_idle+0x325/frame 0xe4b3ec18 cpu_idle_acpi(254c99d4,0,c1b05500,3,c8e67000,...) at cpu_idle_acpi+0x3f/frame 0xe4b3ec2c cpu_idle(0,e4b3ec80,c160ccd7,a6a,0,...) at cpu_idle+0xa6/frame 0xe4b3ec48 sched_idletd(0,e4b3ece8,c1601280,406,0,...) at sched_idletd+0x16a/frame 0xe4b3ecac fork_exit(c0c88250,0,e4b3ece8) at fork_exit+0x7e/frame 0xe4b3ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3ecd4 --- trap 0, eip = 0, esp = 0xe4b3ed20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc8e66a20 (CPU 3) kdb_alt_break_internal(0,e4b42ac4,c0a5b2a7,2,c91ae888,...) at kdb_alt_break_internal+0x101/frame 0xe4b42a98 kdb_alt_break(2,c91ae888,c15c926b,0,40000,...) at kdb_alt_break+0x16/frame 0xe4b42aa4 uart_intr_rxready(c91ae800,6e2e5f16,e4b20c80,c8e66a20,e4b42ae8,...) at uart_intr_rxready+0x87/frame 0xe4b42ac4 uart_intr(c91ae800,1a2e,1,6e2e5f16,0,...) at uart_intr+0x120/frame 0xe4b42b04 intr_event_handle(c8eb3e00,e4b42b98,c1b05c00,40000000,1,...) at intr_event_handle+0x75/frame 0xe4b42b30 intr_execute_handlers(c8eae908,e4b42b98,3) at intr_execute_handlers+0xb0/frame 0xe4b42b68 lapic_handle_intr(32,e4b42b98) at lapic_handle_intr+0x5b/frame 0xe4b42b88 Xapic_isr1() at Xapic_isr1+0x55/frame 0xe4b42b88 --- interrupt, eip = 0xc1224725, esp = 0xe4b42bd8, ebp = 0xe4b42bd8 --- acpi_cpu_c1(e4b42c2c,c1213a39,e4b20c90,0,c18183c3,...) at acpi_cpu_c1+0x5/frame 0xe4b42bd8 acpi_cpu_idle(1dfbd839,0,1dfbd839,e4b42c48,c1224af6,...) at acpi_cpu_idle+0x325/frame 0xe4b42c18 cpu_idle_acpi(1dfbd839,0,c1b05c00,2,c8e66a20,...) at cpu_idle_acpi+0x3f/frame 0xe4b42c2c cpu_idle(0,e4b42c80,c160ccd7,394,0,...) at cpu_idle+0xa6/frame 0xe4b42c48 sched_idletd(0,e4b42ce8,c1601280,406,0,...) at sched_idletd+0x16a/frame 0xe4b42cac fork_exit(c0c88250,0,e4b42ce8) at fork_exit+0x7e/frame 0xe4b42cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b42cd4 --- trap 0, eip = 0, esp = 0xe4b42d20, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc8e67a20 sched_switch(c8e67a20,0,104,18a,e4b3192c,...) at sched_switch+0x512/frame 0xe4b31908 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b3194c sleepq_switch(c8e67a20,0,c1610fcd,1c5,5b0,...) at sleepq_switch+0x11c/frame 0xe4b31984 sleepq_catch_signals(c1610fcd,173,c8e59ae4,222,100,...) at sleepq_catch_signals+0x36e/frame 0xe4b319c4 sleepq_wait_sig(c8e59a2c,6c,c161525f,100,0,...) at sleepq_wait_sig+0x14/frame 0xe4b319e8 _sleep(c8e59a2c,c8e59ad4,16c,c161525f,0,...) at _sleep+0x37c/frame 0xe4b31a3c kern_wait6(c8e67a20,7,0,0,e4b31acc,...) at kern_wait6+0x23d/frame 0xe4b31aa0 sys_wait4(c8e67a20,e4b31c00,c1600cbf,4,ffffffff,...) at sys_wait4+0x87/frame 0xe4b31bc0 syscall(e4b31ce8) at syscall+0x75c/frame 0xe4b31cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xe4b31cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8065ad7, esp = 0xbfbfe834, ebp = 0xbfbfe848 --- Tracing command audit pid 10 tid 100001 td 0xc8e68000 sched_switch(c8e68000,0,104,18a,c8e68000,...) at sched_switch+0x512/frame 0xe4b2db68 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b2dbb0 sleepq_switch(c8e68000,0,c1610fcd,2ae,c18d3e58,...) at sleepq_switch+0x11c/frame 0xe4b2dbe8 sleepq_wait(c1bd9710,0,e4b2dc2c,1,0,...) at sleepq_wait+0x48/frame 0xe4b2dc0c _cv_wait(c1bd9710,c1bd9694,c164fa4e,1bc,0,...) at _cv_wait+0x21f/frame 0xe4b2dc50 audit_worker(0,e4b2dce8,c1601280,406,0,...) at audit_worker+0x3c4/frame 0xe4b2dcac fork_exit(c0f04960,0,e4b2dce8) at fork_exit+0x7e/frame 0xe4b2dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b2dcd4 --- trap 0, eip = 0, esp = 0xe4b2dd20, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1bd1c60 sched_switch(c1bd1c60,0,104,18a,c1bd1c60,...) at sched_switch+0x512/frame 0xc2022be0 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xc2022c28 sleepq_switch(c1bd1c60,0,c1610fcd,2d1,1a33,...) at sleepq_switch+0x11c/frame 0xc2022c60 sleepq_timedwait(c1bd18e4,54,9,0,0,...) at sleepq_timedwait+0x53/frame 0xc2022c9c _sleep(c1bd18e4,0,54,c165b7da,fffff470,...) at _sleep+0x363/frame 0xc2022cf0 swapper() at swapper+0xfa/frame 0xc2022d38 begin() at begin+0x22 Tracing command kernel pid 0 tid 100013 td 0xc8e5b000 sched_switch(c8e5b000,0,104,18a,e4b5fbb4,...) at sched_switch+0x512/frame 0xe4b5fb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b5fbd4 sleepq_switch(c8e5b000,0,c1610fcd,2ae,c8e34f00,...) at sleepq_switch+0x11c/frame 0xe4b5fc0c sleepq_wait(c8e34f00,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b5fc30 msleep_spin_sbt(c8e34f00,c8e34f1c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b5fc74 gtaskqueue_thread_loop(c8e24c04,e4b5fce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b5fcac fork_exit(c0c9c240,c8e24c04,e4b5fce8) at fork_exit+0x7e/frame 0xe4b5fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5fcd4 --- trap 0, eip = 0, esp = 0xe4b5fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc8f24360 sched_switch(c8f24360,0,104,18a,e4b63bb4,...) at sched_switch+0x512/frame 0xe4b63b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b63bd4 sleepq_switch(c8f24360,0,c1610fcd,2ae,c8e34e80,...) at sleepq_switch+0x11c/frame 0xe4b63c0c sleepq_wait(c8e34e80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b63c30 msleep_spin_sbt(c8e34e80,c8e34e9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b63c74 gtaskqueue_thread_loop(c8e24804,e4b63ce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b63cac fork_exit(c0c9c240,c8e24804,e4b63ce8) at fork_exit+0x7e/frame 0xe4b63cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b63cd4 --- trap 0, eip = 0, esp = 0xe4b63d20, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc8f24000 sched_switch(c8f24000,0,104,18a,e4b67bb4,...) at sched_switch+0x512/frame 0xe4b67b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b67bd4 sleepq_switch(c8f24000,0,c1610fcd,2ae,c8e34e00,...) at sleepq_switch+0x11c/frame 0xe4b67c0c sleepq_wait(c8e34e00,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b67c30 msleep_spin_sbt(c8e34e00,c8e34e1c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b67c74 gtaskqueue_thread_loop(c8e24814,e4b67ce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b67cac fork_exit(c0c9c240,c8e24814,e4b67ce8) at fork_exit+0x7e/frame 0xe4b67cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b67cd4 --- trap 0, eip = 0, esp = 0xe4b67d20, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc8f23a20 sched_switch(c8f23a20,0,104,18a,e4b6bbb4,...) at sched_switch+0x512/frame 0xe4b6bb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b6bbd4 sleepq_switch(c8f23a20,0,c1610fcd,2ae,c8e34d80,...) at sleepq_switch+0x11c/frame 0xe4b6bc0c sleepq_wait(c8e34d80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b6bc30 msleep_spin_sbt(c8e34d80,c8e34d9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b6bc74 gtaskqueue_thread_loop(c8e24824,e4b6bce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b6bcac fork_exit(c0c9c240,c8e24824,e4b6bce8) at fork_exit+0x7e/frame 0xe4b6bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6bcd4 --- trap 0, eip = 0, esp = 0xe4b6bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc8f236c0 sched_switch(c8f236c0,0,104,18a,e4b6fbb4,...) at sched_switch+0x512/frame 0xe4b6fb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b6fbd4 sleepq_switch(c8f236c0,0,c1610fcd,2ae,c8e34d00,...) at sleepq_switch+0x11c/frame 0xe4b6fc0c sleepq_wait(c8e34d00,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b6fc30 msleep_spin_sbt(c8e34d00,c8e34d1c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b6fc74 gtaskqueue_thread_loop(c8e24834,e4b6fce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b6fcac fork_exit(c0c9c240,c8e24834,e4b6fce8) at fork_exit+0x7e/frame 0xe4b6fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6fcd4 --- trap 0, eip = 0, esp = 0xe4b6fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc8f23360 sched_switch(c8f23360,0,104,18a,e4b73bb4,...) at sched_switch+0x512/frame 0xe4b73b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b73bd4 sleepq_switch(c8f23360,0,c1610fcd,2ae,c8e34c80,...) at sleepq_switch+0x11c/frame 0xe4b73c0c sleepq_wait(c8e34c80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b73c30 msleep_spin_sbt(c8e34c80,c8e34c9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b73c74 gtaskqueue_thread_loop(c8e24404,e4b73ce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b73cac fork_exit(c0c9c240,c8e24404,e4b73ce8) at fork_exit+0x7e/frame 0xe4b73cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b73cd4 --- trap 0, eip = 0, esp = 0xe4b73d20, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc8f23000 sched_switch(c8f23000,0,104,18a,e4b77bb4,...) at sched_switch+0x512/frame 0xe4b77b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b77bd4 sleepq_switch(c8f23000,0,c1610fcd,2ae,c8e34c00,...) at sleepq_switch+0x11c/frame 0xe4b77c0c sleepq_wait(c8e34c00,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b77c30 msleep_spin_sbt(c8e34c00,c8e34c1c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b77c74 gtaskqueue_thread_loop(c8e24414,e4b77ce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b77cac fork_exit(c0c9c240,c8e24414,e4b77ce8) at fork_exit+0x7e/frame 0xe4b77cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b77cd4 --- trap 0, eip = 0, esp = 0xe4b77d20, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc8f22a20 sched_switch(c8f22a20,0,104,18a,e4b7bbb4,...) at sched_switch+0x512/frame 0xe4b7bb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b7bbd4 sleepq_switch(c8f22a20,0,c1610fcd,2ae,c8e34b80,...) at sleepq_switch+0x11c/frame 0xe4b7bc0c sleepq_wait(c8e34b80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b7bc30 msleep_spin_sbt(c8e34b80,c8e34b9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b7bc74 gtaskqueue_thread_loop(c8e24424,e4b7bce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b7bcac fork_exit(c0c9c240,c8e24424,e4b7bce8) at fork_exit+0x7e/frame 0xe4b7bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7bcd4 --- trap 0, eip = 0, esp = 0xe4b7bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc8f226c0 sched_switch(c8f226c0,0,104,18a,e4b7fbb4,...) at sched_switch+0x512/frame 0xe4b7fb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b7fbd4 sleepq_switch(c8f226c0,0,c1610fcd,2ae,c8e34b00,...) at sleepq_switch+0x11c/frame 0xe4b7fc0c sleepq_wait(c8e34b00,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4b7fc30 msleep_spin_sbt(c8e34b00,c8e34b1c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b7fc74 gtaskqueue_thread_loop(c8e24434,e4b7fce8,c1601280,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b7fcac fork_exit(c0c9c240,c8e24434,e4b7fce8) at fork_exit+0x7e/frame 0xe4b7fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7fcd4 --- trap 0, eip = 0, esp = 0xe4b7fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc8f22360 sched_switch(c8f22360,0,104,18a,e4b83ba4,...) at sched_switch+0x512/frame 0xe4b83b80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b83bc4 sleepq_switch(c8f22360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b83bfc sleepq_wait(c8e34a80,0,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4b83c20 _sleep(c8e34a80,c8e34a9c,0,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4b83c74 taskqueue_thread_loop(c1bd21e8,e4b83ce8,c1601280,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b83cac fork_exit(c0cb0dc0,c1bd21e8,e4b83ce8) at fork_exit+0x7e/frame 0xe4b83cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b83cd4 --- trap 0, eip = 0, esp = 0xe4b83d20, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc8e686c0 sched_switch(c8e686c0,0,104,18a,e4b8fba4,...) at sched_switch+0x512/frame 0xe4b8fb80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b8fbc4 sleepq_switch(c8e686c0,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b8fbfc sleepq_wait(c8e34900,0,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4b8fc20 _sleep(c8e34900,c8e3491c,0,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4b8fc74 taskqueue_thread_loop(c1bd6024,e4b8fce8,c1601280,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b8fcac fork_exit(c0cb0dc0,c1bd6024,e4b8fce8) at fork_exit+0x7e/frame 0xe4b8fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b8fcd4 --- trap 0, eip = 0, esp = 0xe4b8fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc8e68360 sched_switch(c8e68360,0,104,18a,e4b93ba4,...) at sched_switch+0x512/frame 0xe4b93b80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4b93bc4 sleepq_switch(c8e68360,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b93bfc sleepq_wait(c8e34880,0,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4b93c20 _sleep(c8e34880,c8e3489c,0,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4b93c74 taskqueue_thread_loop(c1bd60a0,e4b93ce8,c1601280,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b93cac fork_exit(c0cb0dc0,c1bd60a0,e4b93ce8) at fork_exit+0x7e/frame 0xe4b93cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b93cd4 --- trap 0, eip = 0, esp = 0xe4b93d20, ebp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xc8f486c0 sched_switch(c8f486c0,0,104,18a,e4ba7ba4,...) at sched_switch+0x512/frame 0xe4ba7b80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4ba7bc4 sleepq_switch(c8f486c0,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4ba7bfc sleepq_wait(c8e34680,0,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xe4ba7c20 _sleep(c8e34680,c8e3469c,0,c15f0e1b,0,...) at _sleep+0x38f/frame 0xe4ba7c74 taskqueue_thread_loop(c1b12820,e4ba7ce8,c1601280,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4ba7cac fork_exit(c0cb0dc0,c1b12820,e4ba7ce8) at fork_exit+0x7e/frame 0xe4ba7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4ba7cd4 --- trap 0, eip = 0, esp = 0xe4ba7d20, ebp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xc8f47a20 sched_switch(c8f47a20,0,104,18a,e4bb3bb4,...) at sched_switch+0x512/frame 0xe4bb3b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4bb3bd4 sleepq_switch(c8f47a20,0,c1610fcd,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bb3c0c sleepq_wait(c8e33d80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4bb3c30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bb3c74 taskqueue_thread_loop(c1bc7a1c,e4bb3ce8,c1601280,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bb3cac fork_exit(c0cb0dc0,c1bc7a1c,e4bb3ce8) at fork_exit+0x7e/frame 0xe4bb3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bb3cd4 --- trap 0, eip = 0, esp = 0xe4bb3d20, ebp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xc8f476c0 sched_switch(c8f476c0,0,104,18a,e4bb7bb4,...) at sched_switch+0x512/frame 0xe4bb7b90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4bb7bd4 sleepq_switch(c8f476c0,0,c1610fcd,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bb7c0c sleepq_wait(c8e33d80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4bb7c30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bb7c74 taskqueue_thread_loop(c1bc7a1c,e4bb7ce8,c1601280,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bb7cac fork_exit(c0cb0dc0,c1bc7a1c,e4bb7ce8) at fork_exit+0x7e/frame 0xe4bb7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bb7cd4 --- trap 0, eip = 0, esp = 0xe4bb7d20, ebp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xc8f47360 sched_switch(c8f47360,0,104,18a,e4bbbbb4,...) at sched_switch+0x512/frame 0xe4bbbb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xe4bbbbd4 sleepq_switch(c8f47360,0,c1610fcd,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bbbc0c sleepq_wait(c8e33d80,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xe4bbbc30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bbbc74 taskqueue_thread_loop(c1bc7a1c,e4bbbce8,c1601280,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bbbcac fork_exit(c0cb0dc0,c1bc7a1c,e4bbbce8) at fork_exit+0x7e/frame 0xe4bbbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bbbcd4 --- trap 0, eip = 0, esp = 0xe4bbbd20, ebp = 0 --- Tracing command kernel pid 0 tid 100080 td 0xc91c0a20 sched_switch(c91c0a20,0,104,18a,edddbbb4,...) at sched_switch+0x512/frame 0xedddbb90 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xedddbbd4 sleepq_switch(c91c0a20,0,c1610fcd,2ae,c9191980,...) at sleepq_switch+0x11c/frame 0xedddbc0c sleepq_wait(c9191980,0,c160ab79,c15f0e1b,0,...) at sleepq_wait+0x48/frame 0xedddbc30 msleep_spin_sbt(c9191980,c919199c,c15f0e1b,0,0,...) at msleep_spin_sbt+0x226/frame 0xedddbc74 taskqueue_thread_loop(c1bc5e14,edddbce8,c1601280,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xedddbcac fork_exit(c0cb0dc0,c1bc5e14,edddbce8) at fork_exit+0x7e/frame 0xedddbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedddbcd4 --- trap 0, eip = 0, esp = 0xedddbd20, ebp = 0 --- Tracing command kernel pid 0 tid 100087 td 0xc93ce6c0 sched_switch(c93ce6c0,0,104,18a,eddf8ba4,...) at sched_switch+0x512/frame 0xeddf8b80 mi_switch(104,0,c1610fcd,226,0,...) at mi_switch+0x226/frame 0xeddf8bc4 sleepq_switch(c93ce6c0,0,c1610fcd,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddf8bfc sleepq_wait(c8e33d00,0,c15f0e1b,0,0,...) at sleepq_wait+0x48/frame 0xeddf8c20 _sleep(c8e33d00,c8e33d1c,0,c15f0e1b,0,...) at _sleep+0x38f/frame 0xeddf8c74 taskqueue_thread_loop(c191bb28,eddf8ce8,c1601280,406,0,...) at taskqueue_thread_loop+0x144/frame 0xeddf8cac fork_exit(c0cb0dc0,c191bb28,eddf8ce8) at fork_exit+0x7e/frame 0xeddf8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf8cd4 --- trap 0, eip = 0, esp = 0xeddf8d20, ebp = 0 --- db:0:allt> dump Physical memory: 3427 MB Dumping 235 MB: 220 204 188 172 156 140 124 108 92 76 60 44 28 12 Dump complete db:0:dump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) proc 23808 [Switching to thread 107 (Thread 100318)]#0 sched_switch (td=, newtd=, flags=) at ../../../kern/sched_ule.c:1988 1988 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=, newtd=, flags=) at ../../../kern/sched_ule.c:1988 #1 0xc0c63ec6 in mi_switch (flags=) at ../../../kern/kern_synch.c:438 #2 0xc0cab43c in sleepq_switch (wchan=, pri=0x60) at ../../../kern/subr_sleepqueue.c:608 #3 0xc0cab2c8 in sleepq_wait (wchan=0xcc97a000, pri=0x60) at ../../../kern/subr_sleepqueue.c:687 #4 0xc0c6362f in _sleep (lock=, priority=, wmesg=, sbt=0x0, pr=0x0, flags=) at ../../../kern/kern_synch.c:216 #5 0xc0d1ccd4 in vfs_mount_destroy (mp=0xcc97a000) at ../../../kern/vfs_mount.c:506 #6 0xc0d1e05c in dounmount (mp=, flags=, td=) at ../../../kern/vfs_mount.c:1434 #7 0xc0d1d6c5 in sys_unmount (td=, uap=) at ../../../kern/vfs_mount.c:1209 #8 0xc120ae5c in syscall (frame=) at subr_syscall.c:136 #9 0xc11f683e in Xint0x80_syscall () at ../../../i386/i386/exception.s:277 #10 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xc0d1ccd4 in vfs_mount_destroy (mp=0xcc97a000) at ../../../kern/vfs_mount.c:506 506 msleep(mp, MNT_MTX(mp), PVFS, "mntref", 0); (kgdb) l 501 if (mp->mnt_kern_flag & MNTK_MWAIT) { 502 mp->mnt_kern_flag &= ~MNTK_MWAIT; 503 wakeup(mp); 504 } 505 while (mp->mnt_ref) 506 msleep(mp, MNT_MTX(mp), PVFS, "mntref", 0); 507 KASSERT(mp->mnt_ref == 0, 508 ("%s: invalid refcount in the drain path @ %s:%d", __func__, 509 __FILE__, __LINE__)); 510 if (mp->mnt_writeopcount != 0) (kgdb) info loc No locals. (kgdb) p *mp $1 = {mnt_mtx = {lock_object = {lo_name = 0xc1613ba5 "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc89ac588}, mtx_lock = 0x4}, mnt_gen = 0x4, mnt_list = {tqe_next = 0xc91bb000, tqe_prev = 0xc9ace018}, mnt_op = 0xc19006bc, mnt_vfc = 0xc1900754, mnt_vnodecovered = 0xcaa10870, mnt_syncer = 0x0, mnt_ref = 0x1, mnt_nvnodelist = {tqh_first = 0x0, tqh_last = 0xcc97a034}, mnt_nvnodelistsize = 0x0, mnt_writeopcount = 0x0, mnt_kern_flag = 0x4180616d, mnt_flag = 0x0, mnt_opt = 0xc9ac61e0, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x78, mnt_stat = {f_version = 0x20140518, f_type = 0x35, f_flags = 0x201000, f_bsize = 0x1000, f_iosize = 0x8000, f_blocks = 0x2e7a5, f_bfree = 0x2e78f, f_bavail = 0x2ac12, f_files = 0x17ffe, f_ffree = 0x15b50, f_syncwrites = 0x64f5f, f_asyncwrites = 0x14c4d, f_syncreads = 0xebd, f_asyncreads = 0x0, f_spare = 0xcc97a0c4, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = 0xcc97a11c}, f_charspare = 0xcc97a124 "", f_fstypename = 0xcc97a174 "ufs", f_mntfromname = 0xcc97a184 "/dev/md8a", f_mntonname = 0xcc97a584 "/mnt8"}, mnt_cred = 0xccd8e800, mnt_data = 0x0, mnt_time = 0x0, mnt_iosize_max = 0x20000, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 0xf29b70fd, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x1a91e2, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_listmtx = {lock_object = {lo_name = 0xc161d685 "struct mount vlist mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc89b5690}, mtx_lock = 0x4}, mnt_activevnodelist = {tqh_first = 0x0, tqh_last = 0xcc97a9c8}, mnt_activevnodelistsize = 0x0, mnt_tmpfreevnodelist = {tqh_first = 0x0, tqh_last = 0xcc97a9d4}, mnt_tmpfreevnodelistsize = 0x0, mnt_explock = {lock_object = {lo_name = 0xc15f0877 "explock", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xc89b3d60}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x0, pcs = 0xcc97aa04}}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0x0}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xcc97aa54}} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 319211) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -13253,6 +13253,35 @@ softdep_slowdown(vp) return (1); } +static void +softdep_request_cleanup_flush(mp, ump) + struct mount *mp; + struct ufsmount *ump; +{ + struct vnode *lvp, *mvp; + + MNT_VNODE_FOREACH_ALL(lvp, mp, mvp) { + if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0) { + VI_UNLOCK(lvp); + continue; + } + if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, + curthread)) + continue; + if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ + vput(lvp); + continue; + } + (void) ffs_syncvnode(lvp, MNT_NOWAIT, 0); + vput(lvp); + } + lvp = ump->um_devvp; + if (vn_lock(lvp, LK_EXCLUSIVE | LK_NOWAIT) == 0) { + VOP_FSYNC(lvp, MNT_NOWAIT, curthread); + VOP_UNLOCK(lvp, 0); + } +} + /* * Called by the allocation routines when they are about to fail * in the hope that we can free up the requested resource (inodes @@ -13274,7 +13303,6 @@ softdep_request_cleanup(fs, vp, cred, resource) { struct ufsmount *ump; struct mount *mp; - struct vnode *lvp, *mvp; long starttime; ufs2_daddr_t needed; int error; @@ -13369,40 +13397,33 @@ retry: * resources. We walk the vnodes associated with the mount point * until we get the needed worklist requests that we can reap. */ + error = 1; if ((resource == FLUSH_BLOCKS_WAIT && fs->fs_cstotal.cs_nbfree <= needed) || (resource == FLUSH_INODES_WAIT && fs->fs_pendinginodes > 0 && fs->fs_cstotal.cs_nifree <= needed)) { - MNT_VNODE_FOREACH_ALL(lvp, mp, mvp) { - if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0) { - VI_UNLOCK(lvp); - continue; + ACQUIRE_LOCK(ump); + if ((ump->um_softdep->sd_flags & FLUSH_RC_ACTIVE) == 0) { + ump->um_softdep->sd_flags |= FLUSH_RC_ACTIVE; + FREE_LOCK(ump); + softdep_request_cleanup_flush(mp, ump); + ACQUIRE_LOCK(ump); + ump->um_softdep->sd_flags &= ~FLUSH_RC_ACTIVE; + FREE_LOCK(ump); + if (ump->softdep_on_worklist > 0) { + stat_cleanup_retries += 1; + goto retry; } - if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, - curthread)) - continue; - if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ - vput(lvp); - continue; - } - (void) ffs_syncvnode(lvp, MNT_NOWAIT, 0); - vput(lvp); + } else { + FREE_LOCK(ump); + error = 0; } - lvp = ump->um_devvp; - if (vn_lock(lvp, LK_EXCLUSIVE | LK_NOWAIT) == 0) { - VOP_FSYNC(lvp, MNT_NOWAIT, curthread); - VOP_UNLOCK(lvp, 0); - } - if (ump->softdep_on_worklist > 0) { - stat_cleanup_retries += 1; - goto retry; - } stat_cleanup_failures += 1; } if (time_second - starttime > stat_cleanup_high_delay) stat_cleanup_high_delay = time_second - starttime; UFS_LOCK(ump); - return (1); + return (error); } static bool Index: /usr/src/sys/ufs/ffs/softdep.h =================================================================== --- /usr/src/sys/ufs/ffs/softdep.h (revision 319211) +++ /usr/src/sys/ufs/ffs/softdep.h (working copy) @@ -1065,6 +1065,7 @@ struct mount_softdeps { #define FLUSH_EXIT 0x0001 /* time to exit */ #define FLUSH_CLEANUP 0x0002 /* need to clear out softdep structures */ #define FLUSH_STARTING 0x0004 /* flush thread not yet started */ +#define FLUSH_RC_ACTIVE 0x0008 /* * Keep the old names from when these were in the ufsmount structure. -- Test scenario: misc/crossmp11.sh