GENERIC HEAD from 2011-01-04 10:59:38 UTC, r216952M, vmcore.47 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0 r216952M: Tue Jan 4 13:13:19 CET 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3536162816 (3372 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 808207 free (855 frags, 100919 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 673813 free (1285 frags, 84066 blocks, 0.1% fragmentation) /dev/ad4s1g: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1g: clean, 117564619 free (827 frags, 14695474 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 4136564 free (161964 frags, 496825 blocks, 1.6% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 14455959 free (28535 frags, 1803428 blocks, 0.1% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb ch 1 dma -1 re0: link state changed to UP Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xdad1af60 bufwait (bufwait) @ kern/vfs_bio.c:2659 2nd 0xc77e4200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0cf71cb,2e687361,38323a63,a0d34,a7b,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0900d2b,c0cfa3d0,c6d55238,c6d58978,e9776894,...) at kdb_backtrace+0x2a _witness_debugger(c0cfa3d0,c77e4200,c0d1f39f,c6d58978,c0d1f031,...) at _witness_debugger+0x25 witness_checkorder(c77e4200,9,c0d1f028,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c77e4200,0,c0d1f028,11c,c74b3074,...) at _sx_xlock+0x85 ufsdirhash_acquire(dad1af00,e97769ec,9c,db770974,e9776964,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c74b3074,e97769ec,974,e9776950,e9776954,...) at ufsdirhash_add+0x13 ufs_direnter(c7564d98,c79ac414,e97769ec,e9776bd0,dad1b3e0,...) at ufs_direnter+0x6f9 ufs_mkdir(e9776bf8,c0d35397,0,0,e9776b3c,...) at ufs_mkdir+0x85f VOP_MKDIR_APV(c0e08e60,e9776bf8,e9776bd0,e9776b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c74f05a0,ffffff9c,28404020,0,1c0,...) at kern_mkdirat+0x225 kern_mkdir(c74f05a0,28404020,0,1c0,e9776c7c,...) at kern_mkdir+0x2e mkdir(c74f05a0,e9776cec,e9776d28,c0cf8c4b,0,...) at mkdir+0x29 syscallenter(c74f05a0,e9776ce4,e9776ce4,0,0,...) at syscallenter+0x263 syscall(e9776d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172603, esp = 0xbfbfe8ec, ebp = 0xbfbfed98 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Starting default moused. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc7564df0 ufs (ufs) @ kern/vfs_subr.c:2117 2nd 0xdad1af60 bufwait (bufwait) @ ufs/ffs/ffs_softdep.c:11337 3rd 0xc7989df0 ufs (ufs) @ kern/vfs_subr.c:2117 KDB: stack backtrace: db_trace_self_wrapper(c0cf71cb,2e706564,31313a63,d373333,c0cf000a,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0900d2b,c0cfa3e9,c6d55238,c6d58910,e9798890,...) at kdb_backtrace+0x2a _witness_debugger(c0cfa3e9,c7989df0,c0ceba26,c6d58910,c0d019a8,...) at _witness_debugger+0x25 witness_checkorder(c7989df0,9,c0d0199f,845,0,...) at witness_checkorder+0x839 __lockmgr_args(c7989df0,80100,c7989e5c,0,0,...) at __lockmgr_args+0x814 ffs_lock(e97989b4,c09119db,c0d00d1e,80100,c7989d98,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c0e08e60,e97989b4,109,c0e23d60,c7989d98,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7989d98,80100,c0d0199f,845,4,...) at _vn_lock+0x78 vget(c7989d98,80100,c7546b40,50,0,...) at vget+0xbb vfs_hash_get(c77ce2d4,6fc400,80000,c7546b40,e9798b08,...) at vfs_hash_get+0xed ffs_vgetf(c77ce2d4,6fc400,80000,e9798b08,1,...) at ffs_vgetf+0x49 softdep_sync_metadata(c7564d98,0,c0d1ebbf,144,0,...) at softdep_sync_metadata+0x683 ffs_syncvnode(c7564d98,1,c7546b40,565,c0d0199f,...) at ffs_syncvnode+0x3e2 ffs_sync(c77ce2d4,1,c0d0105f,4ff,80,...) at ffs_sync+0x26f dounmount(c77ce2d4,8000000,c7546b40,484,5b85c5ea,...) at dounmount+0x447 unmount(c7546b40,e9798cec,281766a5,1,0,...) at unmount+0x310 syscallenter(c7546b40,e9798ce4,c0c0bebd,c0e588d0,0,...) at syscallenter+0x263 syscall(e9798d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dbd3f, esp = 0xbfbfe66c, ebp = 0xbfbfe738 --- lock order reversal: 1st 0xc7572724 ufs (ufs) @ kern/vfs_mount.c:1210 2nd 0xc755b310 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1599 KDB: stack backtrace: db_trace_self_wrapper(c0cf71cb,35313a63,a0d3939,c0c72b00,e97989a8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0900d2b,c0cfa3d0,c6d58910,c6d58840,e9798a04,...) at kdb_backtrace+0x2a _witness_debugger(c0cfa3d0,c755b310,c0ce75ce,c6d58840,c0d1bcc6,...) at _witness_debugger+0x25 witness_checkorder(c755b310,9,c0d1bcbd,63f,c755b37c,...) at witness_checkorder+0x839 __lockmgr_args(c755b310,80400,c755b37c,0,0,...) at __lockmgr_args+0x814 vop_stdlock(e9798b24,c0cf2216,df,80400,c755b2b8,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c0de0f40,e9798b24,0,c0e23d60,c755b2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c755b2b8,80400,c0d1bcbd,63f,c77ce2d4,...) at _vn_lock+0x78 softdep_flushworklist(c77ce2d4,e9798bd0,c7546b40,573,c0d0199f,...) at softdep_flushworklist+0x47 ffs_sync(c77ce2d4,1,c0d0105f,4ff,80,...) at ffs_sync+0x2fd dounmount(c77ce2d4,8000000,c7546b40,484,5b85c5ea,...) at dounmount+0x447 unmount(c7546b40,e9798cec,281766a5,1,0,...) at unmount+0x310 syscallenter(c7546b40,e9798ce4,c0c0bebd,c0e588d0,0,...) at syscallenter+0x263 syscall(e9798d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dbd3f, esp = 0xbfbfe66c, ebp = 0xbfbfe738 --- ** /dev/ad4s1g ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 40086 files, 749362 used, 117564619 free (827 frags, 14695474 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** usage: kill [-s signal_name] pid ... kill -l [exit_status] kill -signal_name pid ... kill -signal_number pid ... fsck -y /tmp. Starting inetd. Tue Jan 4 23:52:43 CET 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Jan 4 23:53:00 x4 su: pho to root on /dev/pts/1 lock order reversal: 1st 0xc780a02c filedesc structure (filedesc structure) @ kern/kern_descrip.c:773 2nd 0xc7986c94 ufs (ufs) @ kern/vfs_subr.c:4197 KDB: stack backtrace: db_trace_self_wrapper(c0cf71cb,20296572,656b2040,6b2f6e72,5f6e7265,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0900d2b,c0cfa3d0,c6d555e0,c6d58910,e9849a38,...) at kdb_backtrace+0x2a _witness_debugger(c0cfa3d0,c7986c94,c0ceba26,c6d58910,c0d019a8,...) at _witness_debugger+0x25 witness_checkorder(c7986c94,9,c0d0199f,1065,c7986d00,...) at witness_checkorder+0x839 __lockmgr_args(c7986c94,80400,c7986d00,0,0,...) at __lockmgr_args+0x814 ffs_lock(e9849b5c,c,0,80400,c7986c3c,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c0e08e60,e9849b5c,e9849b78,c0e23d60,c7986c3c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7986c3c,80400,c0d0199f,1065,e9849bb8,...) at _vn_lock+0x78 vfs_knllock(c7986c3c,0,c0cee458,723,c7d571b0,...) at vfs_knllock+0x29 knlist_remove_kq(0,e9849bd8,c095cee9,c7d8401c,c7d571b0,...) at knlist_remove_kq+0x85 knlist_remove(c7d8401c,c7d571b0,0,e9849c04,c0895eb5,...) at knlist_remove+0x1b filt_vfsdetach(c7d571b0,0,c0cee458,804,305,...) at filt_vfsdetach+0x39 knote_fdclose(c74c8870,3,c0cede90,305,5,...) at knote_fdclose+0xf5 do_dup(3,c74c8ab0,e9849c7c,c090a263,c74c8870,...) at do_dup+0x351 dup2(c74c8870,e9849cec,2810cee8,1,0,...) at dup2+0x28 syscallenter(c74c8870,e9849ce4,c0c0bebd,c0e588d0,0,...) at syscallenter+0x263 syscall(e9849d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (90, FreeBSD ELF32, dup2), eip = 0x2810ceef, esp = 0xbfbfeaec, ebp = 0xbfbfeb38 --- Jan 5 08:59:54 x4 kernel: pid 73694 (as), uid 0 inumber 642032 on /usr: filesystem full Jan 5 08:59:54 x4 kernel: pid 73623 (as), uid 0 inumber 2043770 on /usr: filesystem full Jan 5 08:59:55 x4 kernel: pid 73832 (ld), uid 0 inumber 1499126 on /usr: filesystem full Jan 5 08:59:56 x4 kernel: pid 32541 (make), uid 0 inumber 1439115 on /usr: filesystem full Jan 5 08:59:57 x4 kernel: pid 73941 (as), uid 0 inumber 642036 on /usr: filesystem full Jan 5 08:59:57 x4 kernel: pid 32536 (make), uid 0 inumber 1438080 on /usr: filesystem full Jan 5 08:59:58 x4 kernel: pid 74024 (as), uid 0 inumber 1499151 on /usr: filesystem full Jan 5 08:59:58 x4 kernel: pid 32536 (make), uid 0 inumber 1438080 on /usr: filesystem full Jan 5 08:59:59 x4 kernel: pid 74094 (as), uid 0 inumber 642039 on /usr: filesystem full Jan 5 08:59:59 x4 kernel: pid 74103 (as), uid 0 inumber 1499158 on /usr: filesystem full : : Jan 5 10:09:06 x4 kernel: pid 91019 (cc1), uid 0 inumber 342513 on /usr: filesystem full Jan 5 10:09:06 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full Jan 5 10:09:07 x4 kernel: pid 81641 (make), uid 0 inumber 1476907 on /usr: filesystem full Jan 5 10:09:07 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full Jan 5 10:09:08 x4 kernel: pid 91138 (cc1), uid 0 inumber 342513 on /usr: filesystem full Jan 5 10:09:08 x4 kernel: pid 81988 (make), uid 0 inumber 1476902 on /usr: filesystem full Jan 5 10:09:09 x4 kernel: pid 91151 (as), uid 0 inumber 1211763 on /usr: filesystem full Jan 5 10:09:09 x4 kernel: pid 91169 (cc1), uid 0 inumber 342513 on /usr: filesystem full Jan 5 10:09:10 x4 kernel: pid 91204 (cc1), uid 0 inumber 342513 on /usr: filesystem full Jan 5 10:09:10 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full Jan 5 10:09:11 x4 kernel: pid 91314 (cc1), uid 0 inumber 342513 on /usr: filesystem full Jan 5 10:09:11 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full Jan 5 10:09:12 x4 kernel: pid 91357 (as), uid 0 inumber 1863537 on /usr: filesystem full Jan 5 10:09:12 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full Jan 5 10:09:13 x4 kernel: pid 91439 (cc1), uid 0 inumber 342515 on /usr: filesystem full Jan 5 10:09:13 x4 kernel: pid 32542 (make), uid 0 inumber 1476920 on /usr: filesystem full 30 second watchdog timeout expired. Shutdown terminated. Wed Jan 5 12:50:09 CET 2011 Jan 5 12:50:10 x4 init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode Jan 5 12:50:11 x4 syslogd: exiting on signal 15 Jan 5 12:50:30 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: sh Jan 5 12:57:41 init: can't exec sh for single user: No such file or directory # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 70099 1 0 44 0 9896 1480 wait Ss u0 0:00.01 -sh (sh) 0 70100 70099 0 44 0 9688 1032 - R+ u0 0:00.00 ps -l # ps auxww USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 400.0 0.0 0 32 ?? RL 11:51PM 1581:36.24 [idle] root 18 0.3 0.0 0 8 ?? DL 11:51PM 5:23.77 [syncer] root 0 0.0 0.0 0 72 ?? DLs 11:51PM 0:07.21 [kernel] root 1 0.0 0.0 8032 428 ?? ILs 11:51PM 0:00.25 /sbin/init -- root 2 0.0 0.0 0 8 ?? DL 11:51PM 0:00.02 [g_event] root 3 0.0 0.0 0 8 ?? DL 11:51PM 1:56.96 [g_up] root 4 0.0 0.0 0 8 ?? DL 11:51PM 1:11.82 [g_down] root 5 0.0 0.0 0 8 ?? IL 11:51PM 0:00.00 [fw0_probe] root 6 0.0 0.0 0 8 ?? DL 11:51PM 0:00.05 [fdc0] root 7 0.0 0.0 0 8 ?? DL 11:51PM 0:00.00 [sctp_iterator] root 8 0.0 0.0 0 8 ?? DL 11:51PM 0:00.00 [xpt_thrd] root 9 0.0 0.0 0 8 ?? DL 11:51PM 0:12.50 [pagedaemon] root 10 0.0 0.0 0 8 ?? DL 11:51PM 0:00.00 [audit] root 12 0.0 0.0 0 168 ?? WL 11:51PM 4:01.77 [intr] root 13 0.0 0.0 0 8 ?? DL 11:51PM 0:00.97 [yarrow] root 14 0.0 0.0 0 192 ?? DL 11:51PM 0:01.78 [usb] root 15 0.0 0.0 0 8 ?? DL 11:51PM 0:00.00 [vmdaemon] root 16 0.0 0.0 0 8 ?? DL 11:51PM 0:00.00 [pagezero] root 17 0.0 0.0 0 8 ?? DL 11:51PM 0:03.31 [bufdaemon] root 19 0.0 0.0 0 8 ?? DL 11:51PM 0:21.81 [vnlru] root 20 0.0 0.0 0 8 ?? DL 11:51PM 2:16.20 [softdepflush] root 21 0.0 0.0 0 8 ?? DL 11:51PM 0:00.46 [flowcleaner] root 1848 0.0 0.0 9536 976 ?? D 11:51PM 0:00.26 nfsd: server (nfsd) root 70099 0.0 0.0 9896 1480 u0 Ss 12:50PM 0:00.01 -sh (sh) root 70101 0.0 0.0 9688 1036 u0 R+ 12:57PM 0:00.00 ps auxww # exit ~KDB: enter: Line break on console [ thread pid 11 tid 100004 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> ps pid ppid pgrp uid state wmesg wchan cmd 70107 70102 70102 0 D+ vlruwk 0xc0fd0378 rcorder 70102 1 70102 0 Ss+ piperd 0xc77ef930 sh 1848 1 1847 0 D (threaded) nfsd 100140 D nfsd: service 100098 D svcexit 0xc7243380 nfsd: master 21 0 0 0 DL c_flowcl 0xc0fd0550 [flowcleaner] 20 0 0 0 DL sdflush 0xc0fdbd40 [softdepflush] 19 0 0 0 DL [vnlru] 18 0 0 0 DL syncer 0xc0fd0354 [syncer] 17 0 0 0 DL [bufdaemon] 16 0 0 0 DL pgzero 0xc0fddbdc [pagezero] 15 0 0 0 DL psleep 0xc0fdd7d0 [vmdaemon] 9 0 0 0 DL psleep 0xc0fdd798 [pagedaemon] 8 0 0 0 DL ccb_scan 0xc0e25254 [xpt_thrd] 7 0 0 0 DL waiting_ 0xc0fd1db8 [sctp_iterator] 6 0 0 0 DL - 0xc704b03c [fdc0] 5 0 0 0 SL - 0xc7052000 [fw0_probe] 14 0 0 0 DL (threaded) [usb] 100057 D - 0xc7033d34 [usbus5] 100056 D - 0xc7033d04 [usbus5] 100055 D - 0xc7033cd4 [usbus5] 100054 D - 0xc7033ca4 [usbus5] 100052 D - 0xc7027b5c [usbus4] 100051 D - 0xc7027b2c [usbus4] 100050 D - 0xc7027afc [usbus4] 100049 D - 0xc7027acc [usbus4] 100048 D - 0xc7023b5c [usbus3] 100047 D - 0xc7023b2c [usbus3] 100046 D - 0xc7023afc [usbus3] 100045 D - 0xc7023acc [usbus3] 100044 D - 0xc7017b5c [usbus2] 100043 D - 0xc7017b2c [usbus2] 100042 D - 0xc7017afc [usbus2] 100041 D - 0xc7017acc [usbus2] 100039 D - 0xc7012b5c [usbus1] 100038 D - 0xc7012b2c [usbus1] 100037 D - 0xc7012afc [usbus1] 100036 D - 0xc7012acc [usbus1] 100034 D - 0xc7007b5c [usbus0] 100033 D - 0xc7007b2c [usbus0] 100032 D - 0xc7007afc [usbus0] 100031 D - 0xc7007acc [usbus0] 13 0 0 0 DL - 0xc0e58fe4 [yarrow] 4 0 0 0 DL - 0xc0e56c24 [g_down] 3 0 0 0 DL - 0xc0e56c20 [g_up] 2 0 0 0 DL - 0xc0e56c18 [g_event] 12 0 0 0 WL (threaded) [intr] 100065 I [irq12: psm0] 100064 I [irq1: atkbd0] 100062 I [swi0: uart] 100059 I [irq20: fwohci0] 100058 I [irq14: ata0] 100053 I [irq19: ehci0] 100040 I [irq18: ohci2 ohci4] 100035 I [irq17: ohci1 ohci3] 100030 I [irq16: ohci0] 100029 I [irq22: atapci0] 100028 I [irq9: acpi0] 100027 I [swi5: +] 100026 I [swi2: cambio] 100020 I [swi6: task queue] 100019 I [swi6: Giant taskq] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 0 [idle: cpu0] 100005 Run CPU 1 [idle: cpu1] 100004 Run CPU 2 [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc6db5d48 [init] 10 0 0 0 DL audit_wo 0xc0fdb560 [audit] 0 0 0 0 DLs (threaded) [kernel] 100066 D - 0xc0e58fe4 [deadlkres] 100060 D - 0xc704c440 [fw0_taskq] 100025 D - 0xc6f488c0 [ffs_trim taskq] 100024 D - 0xc6f48900 [acpi_task_2] 100023 D - 0xc6f48900 [acpi_task_1] 100022 D - 0xc6f48900 [acpi_task_0] 100021 D - 0xc6f48940 [kqueue taskq] 100018 D - 0xc6f48a00 [thread taskq] 100013 D - 0xc6d87c40 [firmware taskq] 100000 D sched 0xc0e56d00 [swapper] db> run pho db:0:pho> bt Tracing pid 11 tid 100004 td 0xc6db75a0 kdb_enter(c0c986eb,c0cdaef2,0,c6f22880,0,...) at kdb_enter+0x3a uart_intr(c6f22800,c6b1dbcc,c6b1dc34,c6b1dbe8,0,...) at uart_intr+0x126 intr_event_handle(c6daf800,c6b1dc10,c6b1dc34,0,c6fd6c00,...) at intr_event_handle+0x76 intr_execute_handlers(c6dfc0d0,c6b1dc10,2,c6b1dc50,c0bf5225,...) at intr_execute_handlers+0x49 lapic_handle_intr(30,c6b1dc10) at lapic_handle_intr+0x4c Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0bf0825, esp = 0xc6b1dc50, ebp = 0xc6b1dc50 --- acpi_cpu_c1(c6b1dc6c,c0e254d0,2,2f7,b88a,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6b1dca0,c0bfd480,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c0e67b40,2,c6b1dce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6b1dcc4,c0cf5741,9f1,c6db75a0,...) at cpu_idle+0x90 sched_idletd(0,c6b1dd28,c0ceeaab,354,c6db5aa0,...) at sched_idletd+0x263 fork_exit(c08ee590,0,c6b1dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b1dd60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x600b80 curthread = 0xc6db7000: pid 11 "idle: cpu0" curpcb = 0xc6b23d80 fpcurthread = none idlethread = 0xc6db7000: tid 100006 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5ce5b80 curthread = 0xc6db72d0: pid 11 "idle: cpu1" curpcb = 0xc6b20d80 fpcurthread = none idlethread = 0xc6db72d0: tid 100005 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5ce8b80 curthread = 0xc6db75a0: pid 11 "idle: cpu2" curpcb = 0xc6b1dd80 fpcurthread = none idlethread = 0xc6db75a0: tid 100004 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5cebb80 curthread = 0xc6db7870: pid 11 "idle: cpu3" curpcb = 0xc6b1ad80 fpcurthread = none idlethread = 0xc6db7870: tid 100003 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 70107 (rcorder) thread 0xc99ea870 (100571) shared lockmgr ufs (ufs) r = 0 (0xcd9d3310) locked @ kern/vfs_subr.c:2117 db:0:alllocks> show lockedvnods Locked vnodes 0xcd9d32b8: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xcba51330 ref 0 pages 1 lock type ufs: SHARED (count 1) #0 0xc08b0f05 at __lockmgr_args+0x5a5 #1 0xc0b1dc01 at ffs_lock+0xa1 #2 0xc0c2d735 at VOP_LOCK1_APV+0xb5 #3 0xc096e3e8 at _vn_lock+0x78 #4 0xc096197b at vget+0xbb #5 0xc094c65b at cache_lookup+0x67b #6 0xc094c83d at vfs_cache_lookup+0xad #7 0xc0c2e8b5 at VOP_LOOKUP_APV+0xe5 #8 0xc0953d4b at lookup+0x67b #9 0xc0954bb2 at namei+0x592 #10 0xc096dd9f at vn_open_cred+0x31f #11 0xc096e09b at vn_open+0x3b #12 0xc096bad8 at kern_openat+0x128 #13 0xc096bf25 at kern_open+0x35 #14 0xc096bf60 at open+0x30 #15 0xc090a263 at syscallenter+0x263 #16 0xc0c0b65f at syscall+0x4f #17 0xc0bf4ec1 at Xint0x80_syscall+0x21 ino 23572, on dev ad4s1a db:0:lockedvnods> show mount 0xc7493b50 /dev/ad4s1a on / (ufs) 0xc7494000 devfs on /dev (devfs) 0xc77ce5a8 /dev/ad4s1f on /home (ufs) 0xc77ce000 /dev/ad4s1d on /usr (ufs) 0xc7494b50 /dev/ad4s1e on /var (ufs) 0xc749487c procfs on /proc (procfs) 0xc74945a8 /dev/ad4s1g on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 70107 70102 70102 0 D+ vlruwk 0xc0fd0378 rcorder 70102 1 70102 0 Ss+ piperd 0xc77ef930 sh 1848 1 1847 0 D (threaded) nfsd 100140 D nfsd: service 100098 D svcexit 0xc7243380 nfsd: master 21 0 0 0 DL c_flowcl 0xc0fd0550 [flowcleaner] 20 0 0 0 DL sdflush 0xc0fdbd40 [softdepflush] 19 0 0 0 DL [vnlru] 18 0 0 0 DL syncer 0xc0fd0354 [syncer] 17 0 0 0 DL [bufdaemon] 16 0 0 0 DL pgzero 0xc0fddbdc [pagezero] 15 0 0 0 DL psleep 0xc0fdd7d0 [vmdaemon] 9 0 0 0 DL psleep 0xc0fdd798 [pagedaemon] 8 0 0 0 DL ccb_scan 0xc0e25254 [xpt_thrd] 7 0 0 0 DL waiting_ 0xc0fd1db8 [sctp_iterator] 6 0 0 0 DL - 0xc704b03c [fdc0] 5 0 0 0 SL - 0xc7052000 [fw0_probe] 14 0 0 0 DL (threaded) [usb] 100057 D - 0xc7033d34 [usbus5] 100056 D - 0xc7033d04 [usbus5] 100055 D - 0xc7033cd4 [usbus5] 100054 D - 0xc7033ca4 [usbus5] 100052 D - 0xc7027b5c [usbus4] 100051 D - 0xc7027b2c [usbus4] 100050 D - 0xc7027afc [usbus4] 100049 D - 0xc7027acc [usbus4] 100048 D - 0xc7023b5c [usbus3] 100047 D - 0xc7023b2c [usbus3] 100046 D - 0xc7023afc [usbus3] 100045 D - 0xc7023acc [usbus3] 100044 D - 0xc7017b5c [usbus2] 100043 D - 0xc7017b2c [usbus2] 100042 D - 0xc7017afc [usbus2] 100041 D - 0xc7017acc [usbus2] 100039 D - 0xc7012b5c [usbus1] 100038 D - 0xc7012b2c [usbus1] 100037 D - 0xc7012afc [usbus1] 100036 D - 0xc7012acc [usbus1] 100034 D - 0xc7007b5c [usbus0] 100033 D - 0xc7007b2c [usbus0] 100032 D - 0xc7007afc [usbus0] 100031 D - 0xc7007acc [usbus0] 13 0 0 0 DL - 0xc0e58fe4 [yarrow] 4 0 0 0 DL - 0xc0e56c24 [g_down] 3 0 0 0 DL - 0xc0e56c20 [g_up] 2 0 0 0 DL - 0xc0e56c18 [g_event] 12 0 0 0 WL (threaded) [intr] 100065 I [irq12: psm0] 100064 I [irq1: atkbd0] 100062 I [swi0: uart] 100059 I [irq20: fwohci0] 100058 I [irq14: ata0] 100053 I [irq19: ehci0] 100040 I [irq18: ohci2 ohci4] 100035 I [irq17: ohci1 ohci3] 100030 I [irq16: ohci0] 100029 I [irq22: atapci0] 100028 I [irq9: acpi0] 100027 I [swi5: +] 100026 I [swi2: cambio] 100020 I [swi6: task queue] 100019 I [swi6: Giant taskq] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 0 [idle: cpu0] 100005 Run CPU 1 [idle: cpu1] 100004 Run CPU 2 [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc6db5d48 [init] 10 0 0 0 DL audit_wo 0xc0fdb560 [audit] 0 0 0 0 DLs (threaded) [kernel] 100066 D - 0xc0e58fe4 [deadlkres] 100060 D - 0xc704c440 [fw0_taskq] 100025 D - 0xc6f488c0 [ffs_trim taskq] 100024 D - 0xc6f48900 [acpi_task_2] 100023 D - 0xc6f48900 [acpi_task_1] 100022 D - 0xc6f48900 [acpi_task_0] 100021 D - 0xc6f48940 [kqueue taskq] 100018 D - 0xc6f48a00 [thread taskq] 100013 D - 0xc6d87c40 [firmware taskq] 100000 D sched 0xc0e56d00 [swapper] db:0:ps> allt Tracing command rcorder pid 70107 tid 100571 td 0xc99ea870 sched_switch(c99ea870,0,104,191,26ac7bf7,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,50,...) at mi_switch+0x20a sleepq_switch(c99ea870,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0fd0378,50,c0d028df,0,0,...) at sleepq_timedwait+0x6b _sleep(c0fd0378,c0fd0314,50,c0d028df,3e8,...) at _sleep+0x342 getnewvnode(c0ceba26,c7493b50,c0e08e60,ea02e820,ea02e8c4,...) at getnewvnode+0xce ffs_vgetf(c7493b50,5d80,200000,ea02e8c4,0,...) at ffs_vgetf+0x112 ffs_vget(c7493b50,5d80,200000,ea02e8c4,ea02e8d4,...) at ffs_vget+0x2e ufs_lookup_ino(cd9d32b8,ea02eb90,ea02eba4,0,ea02e918,...) at ufs_lookup_ino+0xafc ufs_lookup(ea02e938,c0d34d54,cd9d32b8,cd9d32b8,ea02eba4,...) at ufs_lookup+0x2a VOP_CACHEDLOOKUP_APV(c0e08e60,ea02e938,ea02eba4,ea02eb90,c6d95e80,...) at VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(ea02e9cc,c0d02710,c0e239a0,200000,ea02eb78,...) at vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c0e08e60,ea02e9cc,ea02eba4,1f5,ea02eb90,...) at VOP_LOOKUP_APV+0xe5 lookup(ea02eb78,c0d00e04,ee,c9,c99dad48,...) at lookup+0x67b namei(ea02eb78,ea02eac4,c0cede99,c99dadd0,0,...) at namei+0x592 vn_open_cred(ea02eb78,ea02ec2c,1a4,0,c6d95e80,...) at vn_open_cred+0x31f vn_open(ea02eb78,ea02ec2c,1a4,c74d6038,ea02eb9c,...) at vn_open+0x3b kern_openat(c99ea870,ffffff9c,bfbfe82a,0,1,...) at kern_openat+0x128 kern_open(c99ea870,bfbfe82a,0,0,1b6,...) at kern_open+0x35 open(c99ea870,ea02ecec,ea02ed28,c0cf8c4b,0,...) at open+0x30 syscallenter(c99ea870,ea02ece4,ea02ece4,0,0,...) at syscallenter+0x263 syscall(ea02ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, open), eip = 0x28184643, esp = 0xbfbfe06c, ebp = 0xbfbfe0a8 --- Tracing command sh pid 70102 tid 100214 td 0xc7e83000 sched_switch(c7e83000,0,104,191,fde288ef,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c7e83000,0,c0cf8412,1a7,4c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0cf8412,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c77ef930,4c,c0cfa981,100,0,...) at sleepq_wait_sig+0x17 _sleep(c77ef930,c77efaa0,14c,c0cfa981,0,...) at _sleep+0x35c pipe_read(c791f6c8,e996fc28,c6d95e80,0,c7e83000,...) at pipe_read+0x417 dofileread(e996fc28,ffffffff,ffffffff,0,c791f6c8,...) at dofileread+0x9e kern_readv(c7e83000,3,e996fc28,e996fc48,1,...) at kern_readv+0x58 read(c7e83000,e996fcec,28435010,1,0,...) at read+0x4f syscallenter(c7e83000,e996fce4,c0c0bebd,c0e588d0,0,...) at syscallenter+0x263 syscall(e996fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x281f4893, esp = 0xbfbfeb2c, ebp = 0xbfbfec68 --- Tracing command nfsd pid 1848 tid 100140 td 0xc7907000 sched_switch(c7907000,0,205,191,6e11cec7,...) at sched_switch+0x3bc mi_switch(205,0,c0cf8412,229,0,...) at mi_switch+0x20a sleepq_check_timeout(e9867be0,c08b8e9a,c7243380,0,c7907000,...) at sleepq_check_timeout+0xa5 sleepq_timedwait_sig(c7a0d3d0,0,e9867c18,101,0,...) at sleepq_timedwait_sig+0x21 _cv_timedwait_sig(c7a0d3d0,c7243380,1388,3b1,1,...) at _cv_timedwait_sig+0x252 svc_run_internal(e9867d14,c089cd78,c7243380,e9867d28,c0ceeaab,...) at svc_run_internal+0x356 svc_thread_start(c7243380,e9867d28,c0ceeaab,354,c7542550,...) at svc_thread_start+0x10 fork_exit(c0acac80,c7243380,e9867d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c62e, eip = 0xc, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1848 tid 100098 td 0xc754bb40 sched_switch(c754bb40,0,104,191,6e1262b3,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c754bb40,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7243380,0,c0d18ea7,0,0,...) at sleepq_wait+0x63 _sleep(c7243380,c7243380,0,c0d18ea7,0,...) at _sleep+0x372 svc_run(c7243380,0,c0d17c3b,1c2,0,...) at svc_run+0xc7 nfssvc_nfsd(bfbfe8a0,e9784c0c,c,c6d95e80,e9784c20,...) at nfssvc_nfsd+0xad nfssvc_nfsserver(c754bb40,e9784cec,c7542550,0,c754bb40,...) at nfssvc_nfsserver+0x24f nfssvc(c754bb40,e9784cec,e9784d28,c0cf8c4b,0,...) at nfssvc+0x83 syscallenter(c754bb40,e9784ce4,e9784ce4,0,0,...) at syscallenter+0x263 syscall(e9784d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280dc6db, esp = 0xbfbfe85c, ebp = 0xbfbfeac8 --- Tracing command flowcleaner pid 21 tid 100076 td 0xc702e870 sched_switch(c702e870,0,104,191,8ef34a14,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702e870,0,c0cf8412,28b,c702e870,...) at sleepq_switch+0x162 sleepq_timedwait(c0fd0550,0,e753ecac,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0fd0550,c0fd0558,4e20,620,c702e870,...) at _cv_timedwait+0x252 flowtable_cleaner(0,e753ed28,c0ceeaab,354,c7470000,...) at flowtable_cleaner+0x1ae fork_exit(c0978ab0,0,e753ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe753ed60, ebp = 0 --- Tracing command softdepflush pid 20 tid 100075 td 0xc702eb40 sched_switch(c702eb40,0,104,191,77f226a8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,44,...) at mi_switch+0x20a sleepq_switch(c702eb40,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0fdbd40,44,c0d1db1f,0,0,...) at sleepq_timedwait+0x6b _sleep(c0fdbd40,c0fdbcfc,44,c0d1db1f,3e8,...) at _sleep+0x342 softdep_flush(0,e753bd28,c0ceeaab,354,c74702a8,...) at softdep_flush+0x25a fork_exit(c0b11f50,0,e753bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe753bd60, ebp = 0 --- Tracing command vnlru pid 19 tid 100074 td 0xc7159000 sched_switch(c7159000,0,205,191,20ba5fc6,...) at sched_switch+0x3bc mi_switch(205,0,c0cf8412,229,0,...) at mi_switch+0x20a sleepq_check_timeout(c7159000,0,c0cf8412,28b,0,...) at sleepq_check_timeout+0xa5 sleepq_timedwait(c7470550,50,c0d02911,0,0,...) at sleepq_timedwait+0x70 _sleep(c7470550,c0fd0314,250,c0d02911,3e8,...) at _sleep+0x342 vnlru_proc(0,e7538d28,c0ceeaab,354,c7470550,...) at vnlru_proc+0xe7 fork_exit(c0963190,0,e7538d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7538d60, ebp = 0 --- Tracing command syncer pid 18 tid 100073 td 0xc71592d0 sched_switch(c71592d0,0,104,191,3d27a0a2,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c71592d0,0,c0cf8412,28b,c71592d0,...) at sleepq_switch+0x162 sleepq_timedwait(c0fd0354,0,e7535c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0fd0354,c0fd0340,3e8,6ee,4e20,...) at _cv_timedwait+0x252 sched_sync(0,e7535d28,c0ceeaab,354,c74707f8,...) at sched_sync+0x512 fork_exit(c0961200,0,e7535d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7535d60, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100072 td 0xc71595a0 sched_switch(c71595a0,0,205,191,5fd05ffe,...) at sched_switch+0x3bc mi_switch(205,0,c0cf8412,229,0,...) at mi_switch+0x20a sleepq_check_timeout(c71595a0,0,c0cf8412,28b,0,...) at sleepq_check_timeout+0xa5 sleepq_timedwait(c0fcffe8,44,c0cff984,0,0,...) at sleepq_timedwait+0x70 _sleep(c0fcffe8,c0fcfff0,44,c0cff984,3e8,...) at _sleep+0x342 buf_daemon(0,e7532d28,c0ceeaab,354,c7470aa0,...) at buf_daemon+0x16e fork_exit(c0948e50,0,e7532d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7532d60, ebp = 0 --- Tracing command pagezero pid 16 tid 100071 td 0xc7159870 sched_switch(c7159870,0,104,191,e8522f73,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c7159870,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0fddbdc,0,c0d23efb,0,0,...) at sleepq_timedwait+0x6b _sleep(c0fddbdc,c0fdc680,0,c0d23efb,493e0,...) at _sleep+0x342 vm_pagezero(0,e752fd28,c0ceeaab,354,c7470d48,...) at vm_pagezero+0xdc fork_exit(c0b561f0,0,e752fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe752fd60, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100070 td 0xc7159b40 sched_switch(c7159b40,0,104,191,94b63da8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,68,...) at mi_switch+0x20a sleepq_switch(c7159b40,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0fdd7d0,68,c0cff984,0,0,...) at sleepq_wait+0x63 _sleep(c0fdd7d0,c0fdd7d4,68,c0cff984,0,...) at _sleep+0x372 vm_daemon(0,e752cd28,c0ceeaab,354,c6db62a8,...) at vm_daemon+0x59 fork_exit(c0b50030,0,e752cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe752cd60, ebp = 0 --- Tracing command pagedaemon pid 9 tid 100069 td 0xc715a000 sched_switch(c715a000,0,104,191,61854f1d,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,44,...) at mi_switch+0x20a sleepq_switch(c715a000,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0fdd798,44,c0cff984,0,0,...) at sleepq_timedwait+0x6b _sleep(c0fdd798,c0fdc680,44,c0cff984,1388,...) at _sleep+0x342 vm_pageout(0,e7529d28,c0ceeaab,354,c6db6550,...) at vm_pageout+0x2c7 fork_exit(c0b510e0,0,e7529d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7529d60, ebp = 0 --- Tracing command xpt_thrd pid 8 tid 100068 td 0xc715a2d0 sched_switch(c715a2d0,0,104,191,8d3e42b3,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c715a2d0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0e25254,4c,c0c88eb0,0,0,...) at sleepq_wait+0x63 _sleep(c0e25254,c0e25298,4c,c0c88eb0,0,...) at _sleep+0x372 xpt_scanner_thread(0,e7526d28,c0ceeaab,354,c6db67f8,...) at xpt_scanner_thread+0x47 fork_exit(c0486730,0,e7526d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7526d60, ebp = 0 --- Tracing command sctp_iterator pid 7 tid 100067 td 0xc715a5a0 sched_switch(c715a5a0,0,104,191,f53f9aa6,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c715a5a0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0fd1db8,0,c0d0ec79,0,0,...) at sleepq_wait+0x63 _sleep(c0fd1db8,c0fd1d80,0,c0d0ec79,0,...) at _sleep+0x372 sctp_iterator_thread(0,e7523d28,c0ceeaab,354,c6db6aa0,...) at sctp_iterator_thread+0x5c fork_exit(c09f38f0,0,e7523d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7523d60, ebp = 0 --- Tracing command fdc0 pid 6 tid 100063 td 0xc70202d0 sched_switch(c70202d0,0,104,191,41392f90,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c70202d0,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c704b03c,4c,c0ce864a,0,0,...) at sleepq_timedwait+0x6b _sleep(c704b03c,c704b0f0,4c,c0ce864a,3e8,...) at _sleep+0x342 fdc_thread(c704b000,e7517d28,c0ceeaab,354,c6db6d48,...) at fdc_thread+0x27d fork_exit(c0bd4f40,c704b000,e7517d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7517d60, ebp = 0 --- Tracing command fw0_probe pid 5 tid 100061 td 0xc7020870 sched_switch(c7020870,0,104,191,96c741ad,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,5c,...) at mi_switch+0x20a sleepq_switch(c7020870,0,c0cf8412,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0cf8412,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7052000,5c,c0ce864a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7052000,c7056488,15c,c0ce864a,0,...) at _sleep+0x35c fw_bus_probe_thread(c7052000,e7507d28,c0ceeaab,354,c6f7a000,...) at fw_bus_probe_thread+0xa08 fork_exit(c067fbb0,c7052000,e7507d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7507d60, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc702c5a0 sched_switch(c702c5a0,0,104,191,83e20ae0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702c5a0,0,c0cf8412,268,c702c5a0,...) at sleepq_switch+0x162 sleepq_wait(c7033d34,0,e74f4cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033d34,c7033dd4,c0ce093c,6c,c7033d3c,...) at _cv_wait+0x243 usb_process(c7033d2c,e74f4d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7033d2c,e74f4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74f4d60, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc702c870 sched_switch(c702c870,0,104,191,26310bb6,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702c870,0,c0cf8412,268,c702c870,...) at sleepq_switch+0x162 sleepq_wait(c7033d04,0,e74f1cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033d04,c7033dd4,c0ce093c,6c,c7033d0c,...) at _cv_wait+0x243 usb_process(c7033cfc,e74f1d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7033cfc,e74f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74f1d60, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc702cb40 sched_switch(c702cb40,0,104,191,837ae9ef,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702cb40,0,c0cf8412,268,c702cb40,...) at sleepq_switch+0x162 sleepq_wait(c7033cd4,0,e74eecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033cd4,c7033dd4,c0ce093c,6c,c7033cdc,...) at _cv_wait+0x243 usb_process(c7033ccc,e74eed28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7033ccc,e74eed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74eed60, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc702e000 sched_switch(c702e000,0,104,191,837ad887,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702e000,0,c0cf8412,268,c702e000,...) at sleepq_switch+0x162 sleepq_wait(c7033ca4,0,e74ebcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7033ca4,c7033dd4,c0ce093c,6c,c7033cac,...) at _cv_wait+0x243 usb_process(c7033c9c,e74ebd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7033c9c,e74ebd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74ebd60, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc702e5a0 sched_switch(c702e5a0,0,104,191,837ac3e5,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c702e5a0,0,c0cf8412,268,c702e5a0,...) at sleepq_switch+0x162 sleepq_wait(c7027b5c,0,e74e2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027b5c,c7027bfc,c0ce093c,6c,c7027b64,...) at _cv_wait+0x243 usb_process(c7027b54,e74e2d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7027b54,e74e2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74e2d60, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc701d000 sched_switch(c701d000,0,104,191,1d2d888f,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701d000,0,c0cf8412,268,c701d000,...) at sleepq_switch+0x162 sleepq_wait(c7027b2c,0,e74dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027b2c,c7027bfc,c0ce093c,6c,c7027b34,...) at _cv_wait+0x243 usb_process(c7027b24,e74dfd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7027b24,e74dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74dfd60, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc701d2d0 sched_switch(c701d2d0,0,104,191,83161ce6,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701d2d0,0,c0cf8412,268,c701d2d0,...) at sleepq_switch+0x162 sleepq_wait(c7027afc,0,e74dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027afc,c7027bfc,c0ce093c,6c,c7027b04,...) at _cv_wait+0x243 usb_process(c7027af4,e74dcd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7027af4,e74dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74dcd60, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc701d5a0 sched_switch(c701d5a0,0,104,191,83160a20,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701d5a0,0,c0cf8412,268,c701d5a0,...) at sleepq_switch+0x162 sleepq_wait(c7027acc,0,e74d9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7027acc,c7027bfc,c0ce093c,6c,c7027ad4,...) at _cv_wait+0x243 usb_process(c7027ac4,e74d9d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7027ac4,e74d9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d9d60, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc701d870 sched_switch(c701d870,0,104,191,8315f5a0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701d870,0,c0cf8412,268,c701d870,...) at sleepq_switch+0x162 sleepq_wait(c7023b5c,0,e74d5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7023b5c,c7023bfc,c0ce093c,6c,c7023b64,...) at _cv_wait+0x243 usb_process(c7023b54,e74d5d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7023b54,e74d5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d5d60, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc701db40 sched_switch(c701db40,0,104,191,1c5199a6,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701db40,0,c0cf8412,268,c701db40,...) at sleepq_switch+0x162 sleepq_wait(c7023b2c,0,e74d2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7023b2c,c7023bfc,c0ce093c,6c,c7023b34,...) at _cv_wait+0x243 usb_process(c7023b24,e74d2d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7023b24,e74d2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74d2d60, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc701f000 sched_switch(c701f000,0,104,191,7d52df85,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701f000,0,c0cf8412,268,c701f000,...) at sleepq_switch+0x162 sleepq_wait(c7023afc,0,e74cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7023afc,c7023bfc,c0ce093c,6c,c7023b04,...) at _cv_wait+0x243 usb_process(c7023af4,e74cfd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7023af4,e74cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74cfd60, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc701f2d0 sched_switch(c701f2d0,0,104,191,7d52cc59,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701f2d0,0,c0cf8412,268,c701f2d0,...) at sleepq_switch+0x162 sleepq_wait(c7023acc,0,e74cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7023acc,c7023bfc,c0ce093c,6c,c7023ad4,...) at _cv_wait+0x243 usb_process(c7023ac4,e74ccd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7023ac4,e74ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74ccd60, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc701f5a0 sched_switch(c701f5a0,0,104,191,7d52b94e,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701f5a0,0,c0cf8412,268,c701f5a0,...) at sleepq_switch+0x162 sleepq_wait(c7017b5c,0,e74c8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7017b5c,c7017bfc,c0ce093c,6c,c7017b64,...) at _cv_wait+0x243 usb_process(c7017b54,e74c8d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7017b54,e74c8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c8d60, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc701f870 sched_switch(c701f870,0,104,191,18abb10c,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701f870,0,c0cf8412,268,c701f870,...) at sleepq_switch+0x162 sleepq_wait(c7017b2c,0,e74c5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7017b2c,c7017bfc,c0ce093c,6c,c7017b34,...) at _cv_wait+0x243 usb_process(c7017b24,e74c5d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7017b24,e74c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c5d60, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc701fb40 sched_switch(c701fb40,0,104,191,7cee3a8e,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c701fb40,0,c0cf8412,268,c701fb40,...) at sleepq_switch+0x162 sleepq_wait(c7017afc,0,e74c2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7017afc,c7017bfc,c0ce093c,6c,c7017b04,...) at _cv_wait+0x243 usb_process(c7017af4,e74c2d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7017af4,e74c2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74c2d60, ebp = 0 --- Tracing command usb pid 14 tid 100041 td 0xc7020000 sched_switch(c7020000,0,104,191,7cee27bc,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c7020000,0,c0cf8412,268,c7020000,...) at sleepq_switch+0x162 sleepq_wait(c7017acc,0,e74bfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7017acc,c7017bfc,c0ce093c,6c,c7017ad4,...) at _cv_wait+0x243 usb_process(c7017ac4,e74bfd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7017ac4,e74bfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74bfd60, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc700e2d0 sched_switch(c700e2d0,0,104,191,7cee1257,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c700e2d0,0,c0cf8412,268,c700e2d0,...) at sleepq_switch+0x162 sleepq_wait(c7012b5c,0,e74b8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7012b5c,c7012bfc,c0ce093c,6c,c7012b64,...) at _cv_wait+0x243 usb_process(c7012b54,e74b8d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7012b54,e74b8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b8d60, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc700e5a0 sched_switch(c700e5a0,0,104,191,1807d7f8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c700e5a0,0,c0cf8412,268,c700e5a0,...) at sleepq_switch+0x162 sleepq_wait(c7012b2c,0,e74b5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7012b2c,c7012bfc,c0ce093c,6c,c7012b34,...) at _cv_wait+0x243 usb_process(c7012b24,e74b5d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7012b24,e74b5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b5d60, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc700e870 sched_switch(c700e870,0,104,191,7c89c2db,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c700e870,0,c0cf8412,268,c700e870,...) at sleepq_switch+0x162 sleepq_wait(c7012afc,0,e74b2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7012afc,c7012bfc,c0ce093c,6c,c7012b04,...) at _cv_wait+0x243 usb_process(c7012af4,e74b2d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7012af4,e74b2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74b2d60, ebp = 0 --- Tracing command usb pid 14 tid 100036 td 0xc700eb40 sched_switch(c700eb40,0,104,191,7c89af01,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c700eb40,0,c0cf8412,268,c700eb40,...) at sleepq_switch+0x162 sleepq_wait(c7012acc,0,e74afcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7012acc,c7012bfc,c0ce093c,6c,c7012ad4,...) at _cv_wait+0x243 usb_process(c7012ac4,e74afd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7012ac4,e74afd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74afd60, ebp = 0 --- Tracing command usb pid 14 tid 100034 td 0xc70102d0 sched_switch(c70102d0,0,104,191,7c899840,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c70102d0,0,c0cf8412,268,c70102d0,...) at sleepq_switch+0x162 sleepq_wait(c7007b5c,0,e74a8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7007b5c,c7007bfc,c0ce093c,6c,c7007b64,...) at _cv_wait+0x243 usb_process(c7007b54,e74a8d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7007b54,e74a8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a8d60, ebp = 0 --- Tracing command usb pid 14 tid 100033 td 0xc70105a0 sched_switch(c70105a0,0,104,191,18073303,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c70105a0,0,c0cf8412,268,c70105a0,...) at sleepq_switch+0x162 sleepq_wait(c7007b2c,0,e74a5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7007b2c,c7007bfc,c0ce093c,6c,c7007b34,...) at _cv_wait+0x243 usb_process(c7007b24,e74a5d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7007b24,e74a5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a5d60, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc7010870 sched_switch(c7010870,0,104,191,7c1e7ef0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c7010870,0,c0cf8412,268,c7010870,...) at sleepq_switch+0x162 sleepq_wait(c7007afc,0,e74a2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7007afc,c7007bfc,c0ce093c,6c,c7007b04,...) at _cv_wait+0x243 usb_process(c7007af4,e74a2d28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7007af4,e74a2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74a2d60, ebp = 0 --- Tracing command usb pid 14 tid 100031 td 0xc7010b40 sched_switch(c7010b40,0,104,191,7c1e6c56,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c7010b40,0,c0cf8412,268,c7010b40,...) at sleepq_switch+0x162 sleepq_wait(c7007acc,0,e749fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7007acc,c7007bfc,c0ce093c,6c,c7007ad4,...) at _cv_wait+0x243 usb_process(c7007ac4,e749fd28,c0ceeaab,354,c6f7a2a8,...) at usb_process+0x193 fork_exit(c07ed520,c7007ac4,e749fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe749fd60, ebp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xc6ec35a0 sched_switch(c6ec35a0,0,104,191,2a4b07b3,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6ec35a0,0,c0cf8412,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c0e58fe4,0,c0ce864a,2,0,...) at sleepq_timedwait+0x6b _sleep(c0e58fe4,0,0,c0ce864a,64,...) at _sleep+0x342 pause(c0ce864a,64,c0cd4002,111,0,...) at pause+0x47 random_kthread(0,c6b45d28,c0ceeaab,354,c6f7a550,...) at random_kthread+0x1ef fork_exit(c075e4d0,0,c6b45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b45d60, ebp = 0 --- Tracing command g_down pid 4 tid 100016 td 0xc6ec3870 sched_switch(c6ec3870,0,104,191,26a2b411,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c6ec3870,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0e56c24,4c,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c0e56c24,c0e56b88,24c,c0ce864a,0,...) at _sleep+0x372 g_io_schedule_down(c6ec3870,0,c0ce9dd1,74,0,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6b42d28,c0ceeaab,354,c6db5000,...) at g_down_procbody+0x8d fork_exit(c0860f90,0,c6b42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b42d60, ebp = 0 --- Tracing command g_up pid 3 tid 100015 td 0xc6ec3b40 sched_switch(c6ec3b40,0,104,191,26a75475,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c6ec3b40,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0e56c20,4c,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c0e56c20,c0e56ba8,24c,c0ce864a,0,...) at _sleep+0x372 g_io_schedule_up(c6ec3b40,0,c0ce9dd1,5d,0,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6b3fd28,c0ceeaab,354,c6db52a8,...) at g_up_procbody+0x8d fork_exit(c0861020,0,c6b3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b3fd60, ebp = 0 --- Tracing command g_event pid 2 tid 100014 td 0xc6ec5000 sched_switch(c6ec5000,0,104,191,3607e4c6,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,4c,...) at mi_switch+0x20a sleepq_switch(c6ec5000,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0e56c18,4c,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c0e56c18,c0e56b68,24c,c0ce864a,0,...) at _sleep+0x372 g_run_events(c0e66e40,0,c0ce9dd1,8b) at g_run_events+0x526 g_event_procbody(0,c6b3cd28,c0ceeaab,354,c6db5550,...) at g_event_procbody+0x8a fork_exit(c08610b0,0,c6b3cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b3cd60, ebp = 0 --- Tracing command intr pid 12 tid 100065 td 0xc715ab40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100064 td 0xc715e000 sched_switch(c715e000,0,109,191,7c1b574b,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6daf9f0,...) at mi_switch+0x20a ithread_loop(c6d4a310,e751ad28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a310,e751ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe751ad60, ebp = 0 --- Tracing command intr pid 12 tid 100062 td 0xc70205a0 sched_switch(c70205a0,0,109,191,39a3b8c7,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6ecf9f0,...) at mi_switch+0x20a ithread_loop(c6d4a320,e750ad28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a320,e750ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe750ad60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc702c000 sched_switch(c702c000,0,109,191,7c03a7ee,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd270,...) at mi_switch+0x20a ithread_loop(c6d4a3c0,e74fed28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a3c0,e74fed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74fed60, ebp = 0 --- Tracing command intr pid 12 tid 100058 td 0xc702c2d0 sched_switch(c702c2d0,0,109,191,88f4118c,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6daf370,...) at mi_switch+0x20a ithread_loop(c6d4a3d0,e74fad28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a3d0,e74fad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74fad60, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc702e2d0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100040 td 0xc700e000 sched_switch(c700e000,0,109,191,a17a79f9,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd370,...) at mi_switch+0x20a ithread_loop(c6d4a490,e74bcd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a490,e74bcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74bcd60, ebp = 0 --- Tracing command intr pid 12 tid 100035 td 0xc7010000 sched_switch(c7010000,0,109,191,9bafb6fa,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd3f0,...) at mi_switch+0x20a ithread_loop(c6d4a4c0,e74acd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a4c0,e74acd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe74acd60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc6ec52d0 sched_switch(c6ec52d0,0,109,191,99c27bea,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd470,...) at mi_switch+0x20a ithread_loop(c6d4a4f0,e749cd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a4f0,e749cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe749cd60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc6ec55a0 sched_switch(c6ec55a0,0,109,191,26a64409,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd170,...) at mi_switch+0x20a ithread_loop(c6d4a500,e738dd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a500,e738dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe738dd60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc6ec5870 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100027 td 0xc6ec5b40 sched_switch(c6ec5b40,0,109,191,41a4000c,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfdb70,...) at mi_switch+0x20a ithread_loop(c6d4a580,c6b63d28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a580,c6b63d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b63d60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc6fa1000 sched_switch(c6fa1000,0,109,191,93392146,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfdbf0,...) at mi_switch+0x20a ithread_loop(c6d4a590,c6b60d28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a590,c6b60d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b60d60, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc6dfeb40 sched_switch(c6dfeb40,0,109,191,cba10101,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfdef0,...) at mi_switch+0x20a ithread_loop(c6d4a620,c6b4ed28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a620,c6b4ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b4ed60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc6ec3000 sched_switch(c6ec3000,0,109,191,188e165a,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6ecf070,...) at mi_switch+0x20a ithread_loop(c6d4a630,c6b4bd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a630,c6b4bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b4bd60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc6db85a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc6db8870 sched_switch(c6db8870,0,109,191,8eebd9d7,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6db0af0,...) at mi_switch+0x20a ithread_loop(c6d4a720,c6b33d28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a720,c6b33d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b33d60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc6db8b40 sched_switch(c6db8b40,0,109,191,1a12e469,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6db0b70,...) at mi_switch+0x20a ithread_loop(c6d4a730,c6b30d28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a730,c6b30d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b30d60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc6dfe000 sched_switch(c6dfe000,0,109,191,77f1a853,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6db0bf0,...) at mi_switch+0x20a ithread_loop(c6d4a740,c6b2dd28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a740,c6b2dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b2dd60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc6dfe2d0 sched_switch(c6dfe2d0,0,109,191,81eb8c24,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6db0c70,...) at mi_switch+0x20a ithread_loop(c6d4a750,c6b2ad28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a750,c6b2ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b2ad60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc6dfe5a0 sched_switch(c6dfe5a0,0,109,191,820158f0,...) at sched_switch+0x3bc mi_switch(109,0,c0ceedb9,52d,c6dfd070,...) at mi_switch+0x20a ithread_loop(c6d4a760,c6b27d28,c0ceeaab,354,c6db57f8,...) at ithread_loop+0x209 fork_exit(c08a0640,c6d4a760,c6b27d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b27d60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc6db7000 cpustop_handler(1,c6b23c04,c0c0b776,f1,0,...) at cpustop_handler+0x34 ipi_nmi_handler(f1,0,15f80e8,c0bf0825,c6db5aa0,...) at ipi_nmi_handler+0x2f trap(c6b23c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0bf0825, esp = 0xc6b23c50, ebp = 0xc6b23c50 --- acpi_cpu_c1(c6b23c6c,c0e254d0,0,2f7,b88a,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6b23ca0,c0bfd480,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c0e66e40,2,c6b23ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6b23cc4,c0cf5741,9f1,c6db7000,...) at cpu_idle+0x90 sched_idletd(0,c6b23d28,c0ceeaab,354,c6db5aa0,...) at sched_idletd+0x263 fork_exit(c08ee590,0,c6b23d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b23d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc6db72d0 cpustop_handler(2,c6b20c04,c0c0b776,f1,0,...) at cpustop_handler+0x34 ipi_nmi_handler(f1,0,18612a8,c0bf0825,c6db5aa0,...) at ipi_nmi_handler+0x2f trap(c6b20c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0bf0825, esp = 0xc6b20c50, ebp = 0xc6b20c50 --- acpi_cpu_c1(c6b20c6c,c0e254d0,1,2f7,b88a,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6b20ca0,c0bfd480,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c0e674c0,2,c6b20ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6b20cc4,c0cf5741,9f1,c6db72d0,...) at cpu_idle+0x90 sched_idletd(0,c6b20d28,c0ceeaab,354,c6db5aa0,...) at sched_idletd+0x263 fork_exit(c08ee590,0,c6b20d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b20d60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc6db75a0 kdb_enter(c0c986eb,c0cdaef2,0,c6f22880,0,...) at kdb_enter+0x3a uart_intr(c6f22800,c6b1dbcc,c6b1dc34,c6b1dbe8,0,...) at uart_intr+0x126 intr_event_handle(c6daf800,c6b1dc10,c6b1dc34,0,c6fd6c00,...) at intr_event_handle+0x76 intr_execute_handlers(c6dfc0d0,c6b1dc10,2,c6b1dc50,c0bf5225,...) at intr_execute_handlers+0x49 lapic_handle_intr(30,c6b1dc10) at lapic_handle_intr+0x4c Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0bf0825, esp = 0xc6b1dc50, ebp = 0xc6b1dc50 --- acpi_cpu_c1(c6b1dc6c,c0e254d0,2,2f7,b88a,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6b1dca0,c0bfd480,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c0e67b40,2,c6b1dce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6b1dcc4,c0cf5741,9f1,c6db75a0,...) at cpu_idle+0x90 sched_idletd(0,c6b1dd28,c0ceeaab,354,c6db5aa0,...) at sched_idletd+0x263 fork_exit(c08ee590,0,c6b1dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b1dd60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc6db7870 cpustop_handler(8,c6b1ac04,c0c0b776,f1,0,...) at cpustop_handler+0x34 ipi_nmi_handler(f1,0,1578180,c6b1ac04,c6db5aa0,...) at ipi_nmi_handler+0x2f trap(c6b1ac10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0bf0825, esp = 0xc6b1ac50, ebp = 0xc6b1ac50 --- acpi_cpu_c1(c6b1ac6c,c0e254d0,3,2f7,b88a,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6b1aca0,c0bfd480,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c0e681c0,2,c6b1ace8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6b1acc4,c0cf5741,9f1,c6db7870,...) at cpu_idle+0x90 sched_idletd(0,c6b1ad28,c0ceeaab,354,c6db5aa0,...) at sched_idletd+0x263 fork_exit(c08ee590,0,c6b1ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b1ad60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc6db7b40 sched_switch(c6db7b40,0,104,191,8efdf7f0,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,5c,...) at mi_switch+0x20a sleepq_switch(c6db7b40,0,c0cf8412,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0cf8412,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6db5d48,5c,c0cfad6a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c6db5d48,c6db5dd0,15c,c0cfad6a,0,...) at _sleep+0x35c kern_wait(c6db7b40,ffffffff,c6b16c44,2,0,...) at kern_wait+0xb66 wait4(c6db7b40,c6b16cec,bfbfe798,1,0,...) at wait4+0x3b syscallenter(c6db7b40,c6b16ce4,c0c0bebd,c0e588d0,0,...) at syscallenter+0x263 syscall(c6b16d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054fc3, esp = 0xbfbfe76c, ebp = 0xbfbfe788 --- Tracing command audit pid 10 tid 100001 td 0xc6db8000 sched_switch(c6db8000,0,104,191,7c1c4ae3,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6db8000,0,c0cf8412,268,c6db8000,...) at sleepq_switch+0x162 sleepq_wait(c0fdb560,0,c6b13c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c0fdb560,c0fdb544,c0d19c03,194,0,...) at _cv_wait+0x243 audit_worker(0,c6b13d28,c0ceeaab,354,c6db6000,...) at audit_worker+0x84 fork_exit(c0adae10,0,c6b13d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b13d60, ebp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xc715a870 sched_switch(c715a870,0,104,191,5bf2f42a,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c715a870,0,c0cf8412,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c0e58fe4,0,c0ce864a,2,0,...) at sleepq_timedwait+0x6b _sleep(c0e58fe4,0,0,c0ce864a,bb8,...) at _sleep+0x342 pause(c0ce864a,bb8,120,11e,c0e56d88,...) at pause+0x47 deadlkres(0,e7520d28,c0ceeaab,354,c0e56d00,...) at deadlkres+0x302 fork_exit(c0881f10,0,e7520d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7520d60, ebp = 0 --- Tracing command kernel pid 0 tid 100060 td 0xc7020b40 sched_switch(c7020b40,0,104,191,7c8f92c4,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c7020b40,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c704c440,0,c0cf4656,c0ce864a,0,...) at sleepq_wait+0x63 msleep_spin(c704c440,c704c458,c0ce864a,0,c0cf2216,...) at msleep_spin+0x21d taskqueue_thread_loop(c705649c,e7504d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0x8f fork_exit(c0909ea0,c705649c,e7504d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7504d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc6fa12d0 sched_switch(c6fa12d0,0,104,191,83e9ec0a,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6fa12d0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f488c0,0,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c6f488c0,c6f488d8,0,c0ce864a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0fdbb44,c6b5dd28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0909ea0,c0fdbb44,c6b5dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b5dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc6fa15a0 sched_switch(c6fa15a0,0,104,191,83e9d81a,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6fa15a0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f48900,0,c0cf4656,c0ce864a,0,...) at sleepq_wait+0x63 msleep_spin(c6f48900,c6f48918,c0ce864a,0,c0cf2216,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e28100,c6b5ad28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0x8f fork_exit(c0909ea0,c0e28100,c6b5ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b5ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc6fa1870 sched_switch(c6fa1870,0,104,191,83e9c8d8,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6fa1870,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f48900,0,c0cf4656,c0ce864a,0,...) at sleepq_wait+0x63 msleep_spin(c6f48900,c6f48918,c0ce864a,0,c0cf2216,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e28100,c6b57d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0x8f fork_exit(c0909ea0,c0e28100,c6b57d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b57d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc6fa1b40 sched_switch(c6fa1b40,0,104,191,83e9b8a2,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6fa1b40,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f48900,0,c0cf4656,c0ce864a,0,...) at sleepq_wait+0x63 msleep_spin(c6f48900,c6f48918,c0ce864a,0,c0cf2216,...) at msleep_spin+0x21d taskqueue_thread_loop(c0e28100,c6b54d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0x8f fork_exit(c0909ea0,c0e28100,c6b54d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b54d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc6dfe870 sched_switch(c6dfe870,0,104,191,83e80bdf,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6dfe870,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f48940,0,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c6f48940,c6f48958,0,c0ce864a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e57678,c6b51d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0909ea0,c0e57678,c6b51d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b51d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc6ec32d0 sched_switch(c6ec32d0,0,104,191,a78d3a61,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6ec32d0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6f48a00,0,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c6f48a00,c6f48a18,0,c0ce864a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e75b68,c6b48d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0909ea0,c0e75b68,c6b48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b48d60, ebp = 0 --- Tracing command kernel pid 0 tid 100013 td 0xc6db82d0 sched_switch(c6db82d0,0,104,191,99bcf373,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,0,...) at mi_switch+0x20a sleepq_switch(c6db82d0,0,c0cf8412,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6d87c40,0,c0ce864a,0,0,...) at sleepq_wait+0x63 _sleep(c6d87c40,c6d87c58,0,c0ce864a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0e745e0,c6b39d28,c0ceeaab,354,c0e56d00,...) at taskqueue_thread_loop+0xb5 fork_exit(c0909ea0,c0e745e0,c6b39d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b39d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0e56fb0 sched_switch(c0e56fb0,0,104,191,e813fd29,...) at sched_switch+0x3bc mi_switch(104,0,c0cf8412,1f3,44,...) at mi_switch+0x20a sleepq_switch(c0e56fb0,0,c0cf8412,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0e56d00,44,c0cf6055,0,0,...) at sleepq_timedwait+0x6b _sleep(c0e56d00,0,44,c0cf6055,2710,...) at _sleep+0x342 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x247 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3439 MB Dumping 337 MB: 322 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2 Dump complete = 0xf db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 70107 [Switching to thread 81 (Thread 100571)]#0 sched_switch (td=0xc99ea870, newtd=0xc6db72d0, flags=0x104) at ../../../kern/sched_ule.c:1838 1838 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xc99ea870, newtd=0xc6db72d0, flags=0x104) at ../../../kern/sched_ule.c:1838 #1 0xc08d153a in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:449 #2 0xc0907192 in sleepq_switch (wchan=0xc0fd0378, pri=0x50) at ../../../kern/subr_sleepqueue.c:538 #3 0xc0907dcb in sleepq_timedwait (wchan=0xc0fd0378, pri=0x50) at ../../../kern/subr_sleepqueue.c:652 #4 0xc08d1a32 in _sleep (ident=0xc0fd0378, lock=0xc0fd0314, priority=0x50, wmesg=0xc0d028df "vlruwk", timo=0x3e8) at ../../../kern/kern_synch.c:230 #5 0xc0962fbe in getnewvnode (tag=0xc0ceba26 "ufs", mp=0xc7493b50, vops=0xc0e08e60, vpp=0xea02e820) at ../../../kern/vfs_subr.c:976 #6 0xc0b183d2 in ffs_vgetf (mp=0xc7493b50, ino=0x5d80, flags=0x80000, vpp=0xea02e8c4, ffs_flags=0x0) at ../../../ufs/ffs/ffs_vfsops.c:1511 #7 0xc0b1885e in ffs_vget (mp=0xc7493b50, ino=0x5d80, flags=0x200000, vpp=0xea02e8c4) at ../../../ufs/ffs/ffs_vfsops.c:1455 #8 0xc0b2445c in ufs_lookup_ino (vdp=0xcd9d32b8, vpp=0xea02eb90, cnp=0xea02eba4, dd_ino=0x0) at ../../../ufs/ufs/ufs_lookup.c:750 #9 0xc0b244ea in ufs_lookup (ap=0xea02e938) at ../../../ufs/ufs/ufs_lookup.c:215 #10 0xc0c2bda5 in VOP_CACHEDLOOKUP_APV (vop=0xc0e09320, a=0xea02e938) at vnode_if.c:187 #11 0xc094c866 in vfs_cache_lookup (ap=0xea02e9cc) at vnode_if.h:80 #12 0xc0c2e8b5 in VOP_LOOKUP_APV (vop=0xc0e09320, a=0xea02e9cc) at vnode_if.c:123 #13 0xc0953d4b in lookup (ndp=0xea02eb78) at vnode_if.h:54 #14 0xc0954bb2 in namei (ndp=0xea02eb78) at ../../../kern/vfs_lookup.c:268 #15 0xc096dd9f in vn_open_cred (ndp=0xea02eb78, flagp=0xea02ec2c, cmode=0x1a4, vn_open_flags=Variable "vn_open_flags" is not available. ) at ../../../kern/vfs_vnops.c:189 #16 0xc096e09b in vn_open (ndp=0xea02eb78, flagp=0xea02ec2c, cmode=0x1a4, fp=0xc74d6038) at ../../../kern/vfs_vnops.c:95 #17 0xc096bad8 in kern_openat (td=0xc99ea870, fd=0xffffff9c, path=0xbfbfe82a
, pathseg=UIO_USERSPACE, flags=0x1, mode=0x1b6) at ../../../kern/vfs_syscalls.c:1083 #18 0xc096bf25 in kern_open (td=0xc99ea870, path=0xbfbfe82a
, pathseg=UIO_USERSPACE, flags=0x0, mode=0x1b6) at ../../../kern/vfs_syscalls.c:1039 #19 0xc096bf60 in open (td=0xc99ea870, uap=0xea02ecec) at ../../../kern/vfs_syscalls.c:1015 #20 0xc090a263 in syscallenter (td=0xc99ea870, sa=0xea02ece4) at ../../../kern/subr_trap.c:318 #21 0xc0c0b65f in syscall (frame=0xea02ed28) at ../../../i386/i386/trap.c:1094 #22 0xc0bf4ec1 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #23 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xc0962fbe in getnewvnode (tag=0xc0ceba26 "ufs", mp=0xc7493b50, vops=0xc0e08e60, vpp=0xea02e820) at ../../../kern/vfs_subr.c:976 976 msleep(&vnlruproc_sig, &vnode_free_list_mtx, PVFS, (kgdb) p vnlruproc_sig $1 = 0x1 (kgdb) p vnode_free_list_mtx $2 = {lock_object = {lo_name = 0xc0d0252d "vnode_free_list", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc6d571e8}, mtx_lock = 0x4} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_timeout.c =================================================================== --- /usr/src/sys/kern/kern_timeout.c (revision 216952) +++ /usr/src/sys/kern/kern_timeout.c (working copy) @@ -550,7 +550,7 @@ softclock(void *arg) /* * There must not be any waiting * thread now because the callout - * has a blocked CPU. + * is migrating. * Also, the callout must not be * freed, but that is not easy to * assert. @@ -564,6 +564,7 @@ softclock(void *arg) cc->cc_migration_ticks = 0; cc->cc_migration_func = NULL; cc->cc_migration_arg = NULL; + c->c_cpu = CPUBLOCK; CC_UNLOCK(cc); new_cc = CC_CPU(new_cpu); CC_LOCK(new_cc); @@ -733,7 +734,6 @@ callout_reset_on(struct callout *c, int to_ticks, * to a more appropriate moment. */ if (c->c_cpu != cpu) { - c->c_cpu = CPUBLOCK; if (cc->cc_curr == c) { cc->cc_migration_cpu = cpu; cc->cc_migration_ticks = to_ticks; @@ -745,9 +745,11 @@ callout_reset_on(struct callout *c, int to_ticks, CC_UNLOCK(cc); return (cancelled); } + c->c_cpu = CPUBLOCK; CC_UNLOCK(cc); cc = CC_CPU(cpu); CC_LOCK(cc); + MPASS(c->c_cpu == CPUBLOCK); c->c_cpu = cpu; } #endif @@ -817,6 +819,14 @@ again: goto again; } + /* If the callout is scheduled for migration, cancel that. */ + if (cc->cc_migration_cpu != CPUBLOCK) { + cc->cc_migration_cpu = CPUBLOCK; + cc->cc_migration_ticks = 0; + cc->cc_migration_func = NULL; + cc->cc_migration_arg = NULL; + } + /* * If the callout isn't pending, it's not on the queue, so * don't attempt to remove it from the queue. We can try to -- Test scenario: make universe + (cd /usr/ports; du -s *)