GENERIC HEAD from 2017-05-30 12:00:56 UTC, r319211M, vmcore.1019 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #2 r319211M: Tue May 30 21:03:04 CEST 2017 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3525517312 (3362 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2097312 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 617791 free (2759 frags, 76879 blocks, 0.3% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 55549933 free (37933 frags, 6939000 blocks, 0.0% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 859275 free (10643 frags, 106079 blocks, 1.1% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1704186 free (366442 frags, 167218 blocks, 3.6% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 3154061 free (260981 frags, 361635 blocks, 1.0% fragmentation) Mounting local file systems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc49 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy:uhub0: 10 ports with 10 removable, self powered . re0: link state changed to DOWN ugen2.2: at usbus2 Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d hwaddr 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. uslcom0 on uhub5 uslcom0: on usbus2 re0: link state changed to UP add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. Starting ddb. No core dumps found. Additional ABI support: linux. lock order reversal: 1st 0xe491ae40 bufwait (bufwait) @ kern/vfs_bio.c:3509 2nd 0xc9413600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:281 stack backtrace: #0 0xc0cbd7b1 at witness_debugger+0x81 #1 0xc0cbd6d1 at witness_checkorder+0xd11 #2 0xc0c61a0a at _sx_xlock+0x5a #3 0xc0f62bf3 at ufsdirhash_add+0x53 #4 0xc0f6766a at ufs_direnter+0x47a #5 0xc0f70964 at ufs_mkdir+0x8b4 #6 0xc1243989 at VOP_MKDIR_APV+0xf9 #7 0xc0d39c6e at kern_mkdirat+0x21e #8 0xc0d39a41 at sys_mkdir+0x31 #9 0xc120e73c at syscall+0x75c #10 0xc11fa11e at Xint0x80_syscall+0x2e Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:lock order reversal: 1st 0xca09746c ufs (ufs) @ kern/vfs_lookup.c:669 2nd 0xe492a840 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:280 3rd 0xca09719c ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xc0cbd7b1 at witness_debugger+0x81 #1 0xc0cbd6d1 at witness_checkorder+0xd11 #2 0xc0c2cae7 at __lockmgr_args+0x7a7 #3 0xc0f5cf87 at ffs_lock+0xa7 #4 0xc1244720 at VOP_LOCK1_APV+0xe0 #5 0xc0d3d1f3 at _vn_lock+0x63 #6 0xc0d29752 at vget+0x92 #7 0xc0d1781d at vfs_hash_get+0xcd #8 0xc0f58144 at ffs_vgetf+0x44 #9 0xc0f4e11a at softdep_sync_buf+0xbba #10 0xc0f5ddaf at ffs_syncvnode+0x2ef #11 0xc0f31124 at ffs_truncate+0x974 #12 0xc0f67812 at ufs_direnter+0x622 #13 0xc0f7179f at ufs_makeinode+0x5af #14 0xc0f6d138 at ufs_create+0x38 #15 0xc12412f9 at VOP_CREATE_APV+0xf9 #16 0xc0d3c9e9 at vn_open_cred+0x289 #17 0xc0d3c74d at vn_open+0x3d . Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Starting smartd. May 30 21:08:45 x4 smartd[651]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, May 30 21:08:45 x4 smartd[651]: see the following Seagate web pages: May 30 21:08:45 x4 smartd[651]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en May 30 21:08:45 x4 smartd[651]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc927e8a4 ufs (ufs) @ kern/vfs_mount.c:1355 2nd 0xc9873304 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1908 stack backtrace: #0 0xc0cbd7b1 at witness_debugger+0x81 #1 0xc0cbd6d1 at witness_checkorder+0xd11 #2 0xc0c2c26f at lockmgr_lock_fast_path+0x1df #3 0xc0d1388b at vop_stdlock+0x3b #4 0xc1244720 at VOP_LOCK1_APV+0xe0 #5 0xc0d3d1f3 at _vn_lock+0x63 #6 0xc0f3ac56 at softdep_flushworklist+0x66 #7 0xc0f5b282 at ffs_sync+0xd2 #8 0xc0d3ea95 at vfs_write_suspend+0x1c5 #9 0xc0d3ecf5 at vfs_write_suspend_umnt+0x45 #10 0xc0f5abc9 at ffs_unmount+0xa9 #11 0xc0d1e734 at dounmount+0xc54 #12 0xc0d1dab7 at sys_unmount+0x667 #13 0xc120e73c at syscall+0x75c #14 0xc11fa11e at Xint0x80_syscall+0x2e ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames May 30 21:11:10 x4 su: pho to root on /dev/pts/1 ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 921098 files, 62764048 used, 55549933 free (37933 frags, 6939000 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp. Starting nfscbd. Starting default moused. Starting inetd. Tue May 30 21:11 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: 20170530 21:11:49 all (1/1): crossmp11.sh May 30 21:12:32 x4 kernel: pid 5411 (rw), uid 1004 inumber 73739 on /mnt7: filesystem full May 30 21:12:53 x4 kernel: pid 5440 (rw), uid 1004 inumber 73747 on /mnt8: filesystem full May 30 21:13:10 x4 kernel: pid 5438 (rw), uid 1004 inumber 73743 on /mnt8: filesystem full May 30 21:13:37 x4 kernel: pid 5426 (rw), uid 1004 inumber 73732 on /mnt8: filesystem full May 30 21:14:13 x4 kernel: pid 8413 (rw), uid 1004 inumber 37 on /mnt6: filesystem full May 30 21:14:14 x4 kernel: pid 8377 (creat), uid 1004 inumber 7 on /mnt6: filesystem full May 30 21:14:15 x4 kernel: pid 8380 (creat), uid 1004 inumber 4 on /mnt6: filesystem full May 30 21:14:16 x4 kernel: pid 8420 (rw), uid 1004 inumber 38 on /mnt6: filesystem full May 30 21:14:16 x4 kernel: pid 8412 (creat), uid 1004 inumber 73738 on /mnt6: filesystem full May 30 21:14:17 x4 kernel: pid 8408 (creat), uid 1004 inumber 73732 on /mnt6: filesystem full May 30 21:14:18 x4 kernel: pid 8379 (creat), uid 1004 inumber 3 on /mnt6: filesystem full May 30 21:14:19 x4 kernel: pid 8416 (rw), uid 1004 inumber 73742 on /mnt6: filesystem full May 30 21:14:20 x4 kernel: pid 8424 (rw), uid 1004 inumber 24 on /mnt6: filesystem full May 30 21:14:29 x4 kernel: pid 8431 (rw), uid 1004 inumber 49179 on /mnt5: filesystem full May 30 21:14:29 x4 kernel: pid 8479 (mkdir), uid 1004 inumber 50441 on /mnt5: filesystem full May 30 21:14:36 x4 kernel: pid 8469 (mkdir), uid 1004 inumber 58234 on /mnt5: filesystem full May 30 21:14:36 x4 kernel: pid 8393 (creat), uid 1004 inumber 73729 on /mnt5: filesystem full May 30 21:14:36 x4 kernel: pid 8440 (rw), uid 1004 inumber 49173 on /mnt5: filesystem full May 30 21:14:56 x4 kernel: pid 8430 (rw), uid 1004 inumber 73733 on /mnt5: filesystem full May 30 21:14:56 x4 kernel: pid 8477 (mkdir), uid 1004 inumber 49553 on /mnt5: filesystem full May 30 21:15:00 x4 kernel: pid 8472 (mkdir), uid 1004 inumber 54123 on /mnt5: filesystem full May 30 21:15:00 x4 kernel: pid 8426 (rw), uid 1004 inumber 49156 on /mnt5: filesystem full May 30 21:15:00 x4 kernel: pid 8392 (creat), uid 1004 inumber 49171 on /mnt5: filesystem full May 30 21:15:03 x4 kernel: pid 8463 (mkdir), uid 1004 inumber 7936 on /mnt7: filesystem full May 30 21:15:03 x4 kernel: pid 8451 (rw), uid 1004 inumber 6 on /mnt7: filesystem full May 30 21:15:04 x4 kernel: pid 8462 (mkdir), uid 1004 inumber 24619 on /mnt7: filesystem full May 30 21:15:04 x4 kernel: pid 8398 (creat), uid 1004 inumber 3 on /mnt7: filesystem full May 30 21:15:04 x4 kernel: pid 8452 (rw), uid 1004 inumber 73733 on /mnt7: filesystem full May 30 21:16:44 x4 kernel: pid 11347 (mkdir), uid 1004 inumber 49180 on /mnt5: filesystem full May 30 21:16:44 x4 kernel: pid 11266 (rw), uid 1004 inumber 24608 on /mnt5: filesystem full May 30 21:16:44 x4 kernel: pid 11252 (creat), uid 1004 inumber 24591 on /mnt5: filesystem full May 30 21:16:45 x4 kernel: pid 11252 (creat), uid 1004 inumber 24591 on /mnt5: filesystem full May 30 21:16:46 x4 kernel: pid 11284 (rw), uid 1004 inumber 49158 on /mnt5: filesystem full May 30 21:16:46 x4 kernel: pid 11241 (creat), uid 1004 inumber 24593 on /mnt5: filesystem full May 30 21:16:47 x4 kernel: pid 11239 (creat), uid 1004 inumber 24584 on /mnt5: filesystem full May 30 21:16:48 x4 kernel: pid 11286 (rw), uid 1004 inumber 49156 on /mnt5: filesystem full May 30 21:16:49 x4 kernel: pid 11239 (creat), uid 1004 inumber 24584 on /mnt5: filesystem full May 30 21:16:50 x4 kernel: pid 11343 (mkdir), uid 1004 inumber 24577 on /mnt5: filesystem full May 30 21:16:52 x4 kernel: pid 11318 (rw), uid 1004 inumber 19 on /mnt7: filesystem full May 30 21:17:33 x4 kernel: pid 11331 (mkdir), uid 1004 inumber 25109 on /mnt7: filesystem full May 30 21:17:33 x4 kernel: pid 11293 (creat), uid 1004 inumber 5 on /mnt7: filesystem full May 30 21:17:35 x4 kernel: pid 11327 (rw), uid 1004 inumber 32 on /mnt7: filesystem full May 30 21:17:35 x4 kernel: pid 11335 (mkdir), uid 1004 inumber 73754 on /mnt7: filesystem full May 30 21:17:35 x4 kernel: pid 11274 (creat), uid 1004 inumber 15 on /mnt7: filesystem full May 30 21:17:37 x4 kernel: pid 11309 (creat), uid 1004 inumber 12 on /mnt7: filesystem full May 30 21:17:38 x4 kernel: pid 11263 (rw), uid 1004 inumber 24595 on /mnt5: filesystem full May 30 21:17:38 x4 kernel: pid 11239 (creat), uid 1004 inumber 24584 on /mnt5: filesystem full May 30 21:17:39 x4 kernel: pid 11243 (creat), uid 1004 inumber 24622 on /mnt5: filesystem full May 30 21:17:39 x4 kernel: pid 11260 (rw), uid 1004 inumber 24587 on /mnt5: filesystem full May 30 21:17:41 x4 kernel: pid 11238 (creat), uid 1004 inumber 49175 on /mnt6: filesystem full May 30 21:17:41 x4 kernel: pid 11301 (rw), uid 1004 inumber 76935 on /mnt6: filesystem full May 30 21:18:21 x4 kernel: pid 11315 (rw), uid 1004 inumber 29 on /mnt7: filesystem full Stopping inetd. Waiting for PIDS: 3295. Stopping moused. Waiting for PIDS: 3265. Stopping nfscbd. Waiting for PIDS: 3260 3261. Shutting down local packages:. Stopping cron. Waiting for PIDS: 701. Stopping sshd. Waiting for PIDS: 690. Stopping smartd. Waiting for PIDS: 653. Stopping powerd. Waiting for PIDS: 623. Stopping ntpd. Waiting for PIDS: 620. Stopping lockd. Waiting for PIDS: 563. Stopping statd. Waiting for PIDS: 560. Stopping nfsd. Waiting for PIDS: 556 559. Stopping mountd. Waiting for PIDS: 550. Stopping nfsuserd. Waiting for PIDS: 539 540 541 542 543. Stopping watchdogd. Waiting for PIDS: 497. Stopping rpcbind. Waiting for PIDS: 413. Stopping devd. Waiting for PIDS: 328. Writing entropy file:. Writing early boot entropy file:. . Terminated May 30 22:02:57 x4 syslogd: exiting on signal 15 May 30 22:03:17 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 55402 1 0 20 0 6628 2800 wait Ss u0 0:00.01 -sh (sh) 0 55403 55402 0 20 0 6564 2336 - R+ u0 0:00.00 ps -l 0 6816 1 0 20 0 6012 1816 mntref D 1- 0:00.87 umount -f /mnt8 # ~KDB: enter: Break to debugger [ thread pid 11 tid 100006 ] Stopped at kdb_alt_break_internal+0x101: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100006 td 0xc8e66a20 kdb_alt_break_internal(0,e4b42ac4,c0a5b2a7,2,c91a6888,...) at kdb_alt_break_internal+0x101/frame 0xe4b42a98 kdb_alt_break(2,c91a6888,c15ccb4b,0,40000,...) at kdb_alt_break+0x16/frame 0xe4b42aa4 uart_intr_rxready(c91a6800,0,e4b42b04,c122d2e3,8,...) at uart_intr_rxready+0x87/frame 0xe4b42ac4 uart_intr(c91a6800,c1bd7910,3,c160f34d,0,...) at uart_intr+0x120/frame 0xe4b42b04 intr_event_handle(c8eb3e00,e4b42b98,ffffffff,80000000,1,...) at intr_event_handle+0x75/frame 0xe4b42b30 intr_execute_handlers(c8eae908,e4b42b98,3) at intr_execute_handlers+0xb0/frame 0xe4b42b68 lapic_handle_intr(32,e4b42b98) at lapic_handle_intr+0x5b/frame 0xe4b42b88 Xapic_isr1() at Xapic_isr1+0x55/frame 0xe4b42b88 --- interrupt, eip = 0xc1228005, esp = 0xe4b42bd8, ebp = 0xe4b42bd8 --- acpi_cpu_c1(e4b42c2c,c1217319,e4b20c90,0,c181bc83,...) at acpi_cpu_c1+0x5/frame 0xe4b42bd8 acpi_cpu_idle(7665f0cc,0,7665f0cc,e4b42c48,c12283d6,...) at acpi_cpu_idle+0x325/frame 0xe4b42c18 cpu_idle_acpi(7665f0cc,0,c1b09c80,4,c8e66a20,...) at cpu_idle_acpi+0x3f/frame 0xe4b42c2c cpu_idle(0,e4b42c80,c16105b7,a6a,0,...) at cpu_idle+0xa6/frame 0xe4b42c48 sched_idletd(0,e4b42ce8,c1604b60,406,0,...) at sched_idletd+0x16a/frame 0xe4b42cac fork_exit(c0c88330,0,e4b42ce8) at fork_exit+0x7e/frame 0xe4b42cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b42cd4 --- trap 0, eip = 0, esp = 0xe4b42d20, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x708380 curthread = 0xc8e676c0: pid 11 tid 100003 "idle: cpu0" curpcb = 0xe4b36d40 fpcurthread = none idlethread = 0xc8e676c0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x231f9380 curthread = 0xc8e67360: pid 11 tid 100004 "idle: cpu1" curpcb = 0xe4b3ad40 fpcurthread = none idlethread = 0xc8e67360: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x231fd380 curthread = 0xc8e67000: pid 11 tid 100005 "idle: cpu2" curpcb = 0xe4b3ed40 fpcurthread = none idlethread = 0xc8e67000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x23201380 curthread = 0xc8e66a20: pid 11 tid 100006 "idle: cpu3" curpcb = 0xe4b42d40 fpcurthread = none idlethread = 0xc8e66a20: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc977d000 /dev/ada0s1a on / (ufs) 0xc96b4000 devfs on /dev (devfs) 0xc9ac5000 /dev/ada0s1f on /home (ufs) 0xc9b5e000 /dev/ada0s1d on /usr (ufs) 0xc9ba6000 /dev/ada0s1e on /var (ufs) 0xc9bfe000 procfs on /proc (procfs) 0xc9c0a000 fdescfs on /dev/fd (fdescfs) 0xc9c9f000 linprocfs on /compat/linux/proc (linprocfs) 0xca698000 /dev/label/tmp on /tmp (ufs) More info: show mount db:0:mount> show page vm_cnt.v_free_count: 32254 vm_cnt.v_inactive_count: 287671 vm_cnt.v_active_count: 401001 vm_cnt.v_laundry_count: 101207 vm_cnt.v_wire_count: 40651 vm_cnt.v_free_reserved: 1173 vm_cnt.v_free_min: 5485 vm_cnt.v_free_target: 18421 vm_cnt.v_inactive_target: 27631 db:0:page> ps pid ppid pgrp uid state wmesg wchan cmd 55402 1 55402 0 Ss+ ttyin 0xc91a6674 sh 6816 1 2017 0 D+ mntref 0xcae98000 umount 4225 0 0 0 DL mdwait 0xc9724000 [md8] 4214 0 0 0 DL mdwait 0xc94df800 [md7] 4203 0 0 0 DL mdwait 0xc9724800 [md6] 4192 0 0 0 DL mdwait 0xc9520000 [md5] 22 0 0 0 DL syncer 0xc1b75e5c [syncer] 21 0 0 0 DL vlruwt 0xc92a6000 [vnlru] 20 0 0 0 DL (threaded) [bufdaemon] 100092 D psleep 0xc1b74b84 [bufdaemon] 100105 D sdflush 0xc9364484 [/home worker] 100107 D sdflush 0xc9364084 [/usr worker] 100108 D sdflush 0xc9363e84 [/var worker] 100252 D sdflush 0xc9363a84 [/tmp worker] 19 0 0 0 DL - 0xc1b757cc [bufspacedaemon] 18 0 0 0 DL psleep 0xc1b7bd2c [vmdaemon] 17 0 0 0 DL (threaded) [pagedaemon] 100089 D psleep 0xc1bdf585 [pagedaemon] 100093 D launds 0xc1b7bcf4 [laundry: dom0] 100095 D umarcl 0xc1b7b920 [uma] 16 0 0 0 DL - 0xc19d37e8 [rand_harvestq] 15 0 0 0 DL waiting 0xc1bdae74 [sctp_iterator] 9 0 0 0 DL - 0xc1b7455c [soaiod4] 8 0 0 0 DL - 0xc1b7455c [soaiod3] 7 0 0 0 DL - 0xc1b7455c [soaiod2] 6 0 0 0 DL - 0xc1b7455c [soaiod1] 5 0 0 0 DL - 0xc9176a3c [fdc0] 14 0 0 0 DL (threaded) [usb] 100042 D - 0xc90cfacc [usbus0] 100043 D - 0xc90cfafc [usbus0] 100044 D - 0xc90cfb2c [usbus0] 100045 D - 0xc90cfb5c [usbus0] 100046 D - 0xc90cfb8c [usbus0] 100048 D - 0xc913bacc [usbus1] 100049 D - 0xc913bafc [usbus1] 100050 D - 0xc913bb2c [usbus1] 100051 D - 0xc913bb5c [usbus1] 100052 D - 0xc913bb8c [usbus1] 100054 D - 0xc9145acc [usbus2] 100055 D - 0xc9145afc [usbus2] 100056 D - 0xc9145b2c [usbus2] 100057 D - 0xc9145b5c [usbus2] 100058 D - 0xc9145b8c [usbus2] 100059 D - 0xc914cacc [usbus3] 100060 D - 0xc914cafc [usbus3] 100061 D - 0xc914cb2c [usbus3] 100062 D - 0xc914cb5c [usbus3] 100063 D - 0xc914cb8c [usbus3] 100064 D - 0xc916bacc [usbus4] 100065 D - 0xc916bafc [usbus4] 100066 D - 0xc916bb2c [usbus4] 100067 D - 0xc916bb5c [usbus4] 100068 D - 0xc916bb8c [usbus4] 100070 D - 0xc90baca4 [usbus5] 100071 D - 0xc90bacd4 [usbus5] 100072 D - 0xc90bad04 [usbus5] 100073 D - 0xc90bad34 [usbus5] 100074 D - 0xc90bad64 [usbus5] 100114 D - 0xc8e22c08 [ucom] 4 0 0 0 DL (threaded) [cam] 100037 D - 0xc191fc80 [doneq0] 100088 D - 0xc191fb2c [scanner] 3 0 0 0 DL crypto_ 0xc1b7a8bc [crypto returns] 2 0 0 0 DL crypto_ 0xc1b7a804 [crypto] 13 0 0 0 DL (threaded) [geom] 100028 D - 0xc1bd5f44 [g_event] 100029 D - 0xc1bd5f48 [g_up] 100030 D - 0xc1bd5f4c [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi3: vm] 100008 I [swi4: clock (0)] 100009 I [swi4: clock (1)] 100010 I [swi4: clock (2)] 100011 I [swi4: clock (3)] 100012 I [swi1: netisr 0] 100023 I [swi6: task queue] 100024 I [swi6: Giant taskq] 100027 I [swi5: fast taskq] 100038 I [irq256: hdac0] 100039 I [irq257: re0] 100040 I [irq22: ahci0] 100041 I [irq16: hdac1 ohci0] 100047 I [irq17: ohci1 ohci3] 100053 I [irq18: ohci2 ohci4] 100069 I [irq19: ehci0] 100075 I [irq14: ata0] 100076 I [swi0: uart] 100078 I [irq1: atkbd0] 100079 I [irq12: psm0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc8e59a2c [init] 10 0 0 0 DL audit_w 0xc1bddd90 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xc1bd5f64 [swapper] 100013 D - 0xc8e34f00 [if_config_tqg_0] 100014 D - 0xc8e34e80 [softirq_0] 100015 D - 0xc8e34e00 [softirq_1] 100016 D - 0xc8e34d80 [softirq_2] 100017 D - 0xc8e34d00 [softirq_3] 100018 D - 0xc8e34c80 [if_io_tqg_0] 100019 D - 0xc8e34c00 [if_io_tqg_1] 100020 D - 0xc8e34b80 [if_io_tqg_2] 100021 D - 0xc8e34b00 [if_io_tqg_3] 100022 D - 0xc8e34a80 [kqueue_ctx taskq] 100025 D - 0xc8e34900 [thread taskq] 100026 D - 0xc8e34880 [aiod_kick taskq] 100031 D - 0xc8e34680 [firmware taskq] 100034 D - 0xc8e33d80 [acpi_task_0] 100035 D - 0xc8e33d80 [acpi_task_1] 100036 D - 0xc8e33d80 [acpi_task_2] 100080 D - 0xc918a100 [mca taskq] 100087 D - 0xc8e33d00 [CAM taskq] db:0:ps> allt Tracing command sh pid 55402 tid 100185 td 0xca5cd6c0 sched_switch(ca5cd6c0,0,104,18a,ca5cd6c0,...) at sched_switch+0x512/frame 0xf3648900 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf3648948 sleepq_switch(ca5cd6c0,0,c16148ad,1c5,90,...) at sleepq_switch+0x11c/frame 0xf3648980 sleepq_catch_signals(f36489e4,c0c36350,c91a6614,0,0,...) at sleepq_catch_signals+0x36e/frame 0xf36489c0 sleepq_wait_sig(c91a6674,0,f3648a04,101,0,...) at sleepq_wait_sig+0x14/frame 0xf36489e4 _cv_wait_sig(c91a6674,c91a6604,c161a7a5,5a8,c91a6600,...) at _cv_wait_sig+0x21f/frame 0xf3648a28 tty_wait(c91a6600,c91a6674,f3648b90,1,0,...) at tty_wait+0x4b/frame 0xf3648a4c ttydisc_read(c91a6600,f3648b90,0,d0,c18e1644,...) at ttydisc_read+0x235/frame 0xf3648a94 ttydev_read(c8f19e00,f3648b90,0,ca5eea90,f3648af4,...) at ttydev_read+0x55/frame 0xf3648ab4 devfs_read_f(c9804498,f3648b90,c8e2fd00,0,ca5cd6c0,...) at devfs_read_f+0xc5/frame 0xf3648aec dofileread(c9804498,f3648b90,ffffffff,ffffffff,0,...) at dofileread+0xa7/frame 0xf3648b20 kern_readv(ca5cd6c0,0,f3648b90,f3648bc0,bfbfec6a,...) at kern_readv+0x93/frame 0xf3648b70 sys_read(ca5cd6c0,f3648c00,c160459f,4,0,...) at sys_read+0x54/frame 0xf3648bc0 syscall(f3648ce8) at syscall+0x75c/frame 0xf3648cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xf3648cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2820a7a7, esp = 0xbfbfec24, ebp = 0xbfbfec38 --- Tracing command umount pid 6816 tid 100302 td 0xcafb3a20 sched_switch(cafb3a20,0,104,18a,c84d7744,...) at sched_switch+0x512/frame 0xc84d7720 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xc84d7764 sleepq_switch(cafb3a20,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xc84d779c sleepq_wait(cae98000,60,c1620b14,0,0,...) at sleepq_wait+0x48/frame 0xc84d77c0 _sleep(cae98000,cae98000,60,c1620b14,0,...) at _sleep+0x38f/frame 0xc84d7814 vfs_mount_destroy(cae98000,10,0,5e5,0,...) at vfs_mount_destroy+0xdf/frame 0xc84d7878 dounmount(cae98000,8080000,cafb3a20,4e4,43e,...) at dounmount+0x14a5/frame 0xc84d7a70 sys_unmount(cafb3a20,c84d7c00,c160459f,4,c1b09580,...) at sys_unmount+0x667/frame 0xc84d7bc0 syscall(c84d7ce8) at syscall+0x75c/frame 0xc84d7cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xc84d7cdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280a9bef, esp = 0xbfbfd424, ebp = 0xbfbfd900 --- Tracing command md8 pid 4225 tid 100238 td 0xca6186c0 sched_switch(ca6186c0,0,104,18a,f374ab9c,...) at sched_switch+0x512/frame 0xf374ab78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf374abbc sleepq_switch(ca6186c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xf374abf4 sleepq_wait(c9724000,5c,c132e8ba,0,0,...) at sleepq_wait+0x48/frame 0xf374ac18 _sleep(c9724000,c9724020,25c,c132e8ba,0,...) at _sleep+0x38f/frame 0xf374ac6c md_kthread(c9724000,f374ace8,c1604b60,406,0,...) at md_kthread+0x1da/frame 0xf374acac fork_exit(c07e2da0,c9724000,f374ace8) at fork_exit+0x7e/frame 0xf374acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf374acd4 --- trap 0, eip = 0, esp = 0xf374ad20, ebp = 0 --- Tracing command md7 pid 4214 tid 100165 td 0xca06c360 sched_switch(ca06c360,0,104,18a,f35f5b9c,...) at sched_switch+0x512/frame 0xf35f5b78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf35f5bbc sleepq_switch(ca06c360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xf35f5bf4 sleepq_wait(c94df800,5c,c132e8ba,0,0,...) at sleepq_wait+0x48/frame 0xf35f5c18 _sleep(c94df800,c94df820,25c,c132e8ba,0,...) at _sleep+0x38f/frame 0xf35f5c6c md_kthread(c94df800,f35f5ce8,c1604b60,406,0,...) at md_kthread+0x1da/frame 0xf35f5cac fork_exit(c07e2da0,c94df800,f35f5ce8) at fork_exit+0x7e/frame 0xf35f5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf35f5cd4 --- trap 0, eip = 0, esp = 0xf35f5d20, ebp = 0 --- Tracing command md6 pid 4203 tid 100192 td 0xca5dc000 sched_switch(ca5dc000,0,104,18a,f3664b9c,...) at sched_switch+0x512/frame 0xf3664b78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf3664bbc sleepq_switch(ca5dc000,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xf3664bf4 sleepq_wait(c9724800,5c,c132e8ba,0,0,...) at sleepq_wait+0x48/frame 0xf3664c18 _sleep(c9724800,c9724820,25c,c132e8ba,0,...) at _sleep+0x38f/frame 0xf3664c6c md_kthread(c9724800,f3664ce8,c1604b60,406,0,...) at md_kthread+0x1da/frame 0xf3664cac fork_exit(c07e2da0,c9724800,f3664ce8) at fork_exit+0x7e/frame 0xf3664cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf3664cd4 --- trap 0, eip = 0, esp = 0xf3664d20, ebp = 0 --- Tracing command md5 pid 4192 tid 100250 td 0xca696a20 sched_switch(ca696a20,0,104,18a,f3789b9c,...) at sched_switch+0x512/frame 0xf3789b78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf3789bbc sleepq_switch(ca696a20,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xf3789bf4 sleepq_wait(c9520000,5c,c132e8ba,0,0,...) at sleepq_wait+0x48/frame 0xf3789c18 _sleep(c9520000,c9520020,25c,c132e8ba,0,...) at _sleep+0x38f/frame 0xf3789c6c md_kthread(c9520000,f3789ce8,c1604b60,406,0,...) at md_kthread+0x1da/frame 0xf3789cac fork_exit(c07e2da0,c9520000,f3789ce8) at fork_exit+0x7e/frame 0xf3789cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf3789cd4 --- trap 0, eip = 0, esp = 0xf3789d20, ebp = 0 --- Tracing command syncer pid 22 tid 100096 td 0xc96b0000 sched_switch(c96b0000,0,104,18a,c96b0000,...) at sched_switch+0x512/frame 0xedf9cb40 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf9cb88 sleepq_switch(c96b0000,0,c16148ad,2d1,c96b0000,...) at sleepq_switch+0x11c/frame 0xedf9cbc0 sleepq_timedwait(c1b75e5c,0,edf9cc24,0,0,...) at sleepq_timedwait+0x53/frame 0xedf9cc00 _cv_timedwait_sbt(c1b75e5c,c1b75e3c,fffffed8,0,0,...) at _cv_timedwait_sbt+0x24b/frame 0xedf9cc48 sched_sync(0,edf9cce8,c1604b60,406,0,...) at sched_sync+0x1fd/frame 0xedf9ccac fork_exit(c0d2ed40,0,edf9cce8) at fork_exit+0x7e/frame 0xedf9ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf9ccd4 --- trap 0, eip = 0, esp = 0xedf9cd20, ebp = 0 --- Tracing command vnlru pid 21 tid 100094 td 0xc91b9a20 sched_switch(c91b9a20,0,104,18a,c91b9a20,...) at sched_switch+0x512/frame 0xedf94b20 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf94b68 sleepq_switch(c91b9a20,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedf94ba0 sleepq_timedwait(c92a6000,60,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf94bdc _sleep(c92a6000,c1b75e08,260,c16224b3,fffffed8,...) at _sleep+0x363/frame 0xedf94c30 vnlru_proc(0,edf94ce8,c1604b60,406,0,...) at vnlru_proc+0xbf/frame 0xedf94cac fork_exit(c0d2e3b0,0,edf94ce8) at fork_exit+0x7e/frame 0xedf94cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf94cd4 --- trap 0, eip = 0, esp = 0xedf94d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100092 td 0xc92b2000 sched_switch(c92b2000,0,104,18a,c92b2000,...) at sched_switch+0x512/frame 0xedf8cb68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf8cbb0 sleepq_switch(c92b2000,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedf8cbe8 sleepq_timedwait(c1b74b84,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf8cc24 _sleep(c1b74b84,c1b74b00,54,c161efca,fffffed8,...) at _sleep+0x363/frame 0xedf8cc78 buf_daemon(0,edf8cce8,c1604b60,406,0,...) at buf_daemon+0x8c/frame 0xedf8ccac fork_exit(c0d08fd0,0,edf8cce8) at fork_exit+0x7e/frame 0xedf8ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf8ccd4 --- trap 0, eip = 0, esp = 0xedf8cd20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100105 td 0xc96af6c0 sched_switch(c96af6c0,0,104,18a,c96af6c0,...) at sched_switch+0x512/frame 0xedfd6b58 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedfd6ba0 sleepq_switch(c96af6c0,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedfd6bd8 sleepq_timedwait(c9364484,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfd6c14 _sleep(c9364484,c9364400,54,c1659811,7fffff6c,...) at _sleep+0x363/frame 0xedfd6c68 softdep_flush(c9ac5000,edfd6ce8,c1604b60,406,0,...) at softdep_flush+0x196/frame 0xedfd6cac fork_exit(c0f3e360,c9ac5000,edfd6ce8) at fork_exit+0x7e/frame 0xedfd6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfd6cd4 --- trap 0, eip = 0, esp = 0xedfd6d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100107 td 0xc96af000 sched_switch(c96af000,0,104,18a,c96af000,...) at sched_switch+0x512/frame 0xedfdeb58 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedfdeba0 sleepq_switch(c96af000,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedfdebd8 sleepq_timedwait(c9364084,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfdec14 _sleep(c9364084,c9364000,54,c1659811,7fffff6c,...) at _sleep+0x363/frame 0xedfdec68 softdep_flush(c9b5e000,edfdece8,c1604b60,406,0,...) at softdep_flush+0x196/frame 0xedfdecac fork_exit(c0f3e360,c9b5e000,edfdece8) at fork_exit+0x7e/frame 0xedfdecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfdecd4 --- trap 0, eip = 0, esp = 0xedfded20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100108 td 0xc96aea20 sched_switch(c96aea20,0,104,18a,c96aea20,...) at sched_switch+0x512/frame 0xedfe2b58 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedfe2ba0 sleepq_switch(c96aea20,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedfe2bd8 sleepq_timedwait(c9363e84,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedfe2c14 _sleep(c9363e84,c9363e00,54,c1659811,7fffff6c,...) at _sleep+0x363/frame 0xedfe2c68 softdep_flush(c9ba6000,edfe2ce8,c1604b60,406,0,...) at softdep_flush+0x196/frame 0xedfe2cac fork_exit(c0f3e360,c9ba6000,edfe2ce8) at fork_exit+0x7e/frame 0xedfe2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedfe2cd4 --- trap 0, eip = 0, esp = 0xedfe2d20, ebp = 0 --- Tracing command bufdaemon pid 20 tid 100252 td 0xca696360 sched_switch(ca696360,0,104,18a,ca696360,...) at sched_switch+0x512/frame 0xf379eb58 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf379eba0 sleepq_switch(ca696360,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xf379ebd8 sleepq_timedwait(c9363a84,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xf379ec14 _sleep(c9363a84,c9363a00,54,c1659811,7fffff6c,...) at _sleep+0x363/frame 0xf379ec68 softdep_flush(ca698000,f379ece8,c1604b60,406,0,...) at softdep_flush+0x196/frame 0xf379ecac fork_exit(c0f3e360,ca698000,f379ece8) at fork_exit+0x7e/frame 0xf379ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf379ecd4 --- trap 0, eip = 0, esp = 0xf379ed20, ebp = 0 --- Tracing command bufspacedaemon pid 19 tid 100091 td 0xc92b2360 sched_switch(c92b2360,0,104,18a,c92b2360,...) at sched_switch+0x512/frame 0xedf88b68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf88bb0 sleepq_switch(c92b2360,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedf88be8 sleepq_timedwait(c1b757cc,5c,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf88c24 _sleep(c1b757cc,c1b75580,25c,c15f46fb,fffffed8,...) at _sleep+0x363/frame 0xedf88c78 bufspace_daemon(0,edf88ce8,c1604b60,406,0,...) at bufspace_daemon+0x1ff/frame 0xedf88cac fork_exit(c0d088e0,0,edf88ce8) at fork_exit+0x7e/frame 0xedf88cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf88cd4 --- trap 0, eip = 0, esp = 0xedf88d20, ebp = 0 --- Tracing command vmdaemon pid 18 tid 100090 td 0xc92b26c0 sched_switch(c92b26c0,0,104,18a,c92b26c0,...) at sched_switch+0x512/frame 0xedf84b38 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf84b80 sleepq_switch(c92b26c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xedf84bb8 sleepq_wait(c1b7bd2c,74,c161efca,0,0,...) at sleepq_wait+0x48/frame 0xedf84bdc _sleep(c1b7bd2c,c1b7bd14,74,c161efca,0,...) at _sleep+0x38f/frame 0xedf84c30 vm_daemon(0,edf84ce8,c1604b60,406,0,...) at vm_daemon+0xa9/frame 0xedf84cac fork_exit(c0f9f4c0,0,edf84ce8) at fork_exit+0x7e/frame 0xedf84cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf84cd4 --- trap 0, eip = 0, esp = 0xedf84d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100089 td 0xc92b2a20 sched_switch(c92b2a20,0,104,18a,c92b2a20,...) at sched_switch+0x512/frame 0xedf80b28 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf80b70 sleepq_switch(c92b2a20,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xedf80ba8 sleepq_timedwait(c1bdf585,54,0,0,0,...) at sleepq_timedwait+0x53/frame 0xedf80be4 _sleep(c1bdf585,c1bdf500,254,c161efca,fffffed8,...) at _sleep+0x363/frame 0xedf80c38 vm_pageout(0,edf80ce8,c1604b60,406,0,...) at vm_pageout+0x467/frame 0xedf80cac fork_exit(c0f9c9c0,0,edf80ce8) at fork_exit+0x7e/frame 0xedf80cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf80cd4 --- trap 0, eip = 0, esp = 0xedf80d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100093 td 0xc91b86c0 sched_switch(c91b86c0,0,104,18a,c91b86c0,...) at sched_switch+0x512/frame 0xedf90b08 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf90b50 sleepq_switch(c91b86c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xedf90b88 sleepq_wait(c1b7bcf4,54,c1661d9a,0,0,...) at sleepq_wait+0x48/frame 0xedf90bac _sleep(c1b7bcf4,c1bdf280,54,c1661d9a,0,...) at _sleep+0x38f/frame 0xedf90c00 vm_pageout_laundry_worker(0,edf90ce8,c1604b60,406,0,...) at vm_pageout_laundry_worker+0x37e/frame 0xedf90cac fork_exit(c0f9dcf0,0,edf90ce8) at fork_exit+0x7e/frame 0xedf90cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf90cd4 --- trap 0, eip = 0, esp = 0xedf90d20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100095 td 0xc91b8360 sched_switch(c91b8360,0,104,18a,edf98ba4,...) at sched_switch+0x512/frame 0xedf98b80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedf98bc4 sleepq_switch(c91b8360,0,c16148ad,2ae,c1b7b90c,...) at sleepq_switch+0x11c/frame 0xedf98bfc sleepq_wait(c1b7b920,54,edf98c50,0,0,...) at sleepq_wait+0x48/frame 0xedf98c20 _sleep(c1b7b920,c1b7b90c,54,c165e4e5,0,...) at _sleep+0x38f/frame 0xedf98c74 uma_reclaim_worker(0,edf98ce8,c1604b60,406,0,...) at uma_reclaim_worker+0xaa/frame 0xedf98cac fork_exit(c0f7ced0,0,edf98ce8) at fork_exit+0x7e/frame 0xedf98cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf98cd4 --- trap 0, eip = 0, esp = 0xedf98d20, ebp = 0 --- Tracing command rand_harvestq pid 16 tid 100086 td 0xc92c96c0 sched_switch(c92c96c0,0,104,18a,c92c96c0,...) at sched_switch+0x512/frame 0xeddf4b38 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddf4b80 sleepq_switch(c92c96c0,0,c16148ad,2d1,cdf,...) at sleepq_switch+0x11c/frame 0xeddf4bb8 sleepq_timedwait(c19d37e8,0,0,0,0,...) at sleepq_timedwait+0x53/frame 0xeddf4bf4 _sleep(c19d37e8,0,0,c15f46fb,19999999,...) at _sleep+0x363/frame 0xeddf4c48 random_kthread(0,eddf4ce8,c1604b60,406,0,...) at random_kthread+0x7c/frame 0xeddf4cac fork_exit(c096b270,0,eddf4ce8) at fork_exit+0x7e/frame 0xeddf4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf4cd4 --- trap 0, eip = 0, esp = 0xeddf4d20, ebp = 0 --- Tracing command sctp_iterator pid 15 tid 100085 td 0xc92c9a20 sched_switch(c92c9a20,0,104,18a,c92c9a20,...) at sched_switch+0x512/frame 0xeddf0b88 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddf0bd0 sleepq_switch(c92c9a20,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddf0c08 sleepq_wait(c1bdae74,0,c163caa7,0,0,...) at sleepq_wait+0x48/frame 0xeddf0c2c _sleep(c1bdae74,c1bdae3c,0,c163caa7,0,...) at _sleep+0x38f/frame 0xeddf0c80 sctp_iterator_thread(0,eddf0ce8,c1604b60,406,0,...) at sctp_iterator_thread+0x7c/frame 0xeddf0cac fork_exit(c0df1480,0,eddf0ce8) at fork_exit+0x7e/frame 0xeddf0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf0cd4 --- trap 0, eip = 0, esp = 0xeddf0d20, ebp = 0 --- Tracing command soaiod4 pid 9 tid 100084 td 0xc92ca000 sched_switch(c92ca000,0,104,18a,c92ca000,...) at sched_switch+0x512/frame 0xeddebb60 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddebba8 sleepq_switch(c92ca000,0,c16148ad,2d1,cea,...) at sleepq_switch+0x11c/frame 0xeddebbe0 sleepq_timedwait(c1b7455c,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xeddebc1c _sleep(c1b7455c,c1b7453c,0,c15f46fb,ffffdd50,...) at _sleep+0x363/frame 0xeddebc70 soaio_kproc_loop(4,eddebce8,c1604b60,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xeddebcac fork_exit(c0ccca80,4,eddebce8) at fork_exit+0x7e/frame 0xeddebcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddebcd4 --- trap 0, eip = 0, esp = 0xeddebd20, ebp = 0 --- Tracing command soaiod3 pid 8 tid 100083 td 0xc92ca360 sched_switch(c92ca360,0,104,18a,c92ca360,...) at sched_switch+0x512/frame 0xedde7b60 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedde7ba8 sleepq_switch(c92ca360,0,c16148ad,2d1,ce9,...) at sleepq_switch+0x11c/frame 0xedde7be0 sleepq_timedwait(c1b7455c,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedde7c1c _sleep(c1b7455c,c1b7453c,0,c15f46fb,ffffdd50,...) at _sleep+0x363/frame 0xedde7c70 soaio_kproc_loop(3,edde7ce8,c1604b60,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedde7cac fork_exit(c0ccca80,3,edde7ce8) at fork_exit+0x7e/frame 0xedde7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedde7cd4 --- trap 0, eip = 0, esp = 0xedde7d20, ebp = 0 --- Tracing command soaiod2 pid 7 tid 100082 td 0xc92ca6c0 sched_switch(c92ca6c0,0,104,18a,c92ca6c0,...) at sched_switch+0x512/frame 0xedde3b60 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedde3ba8 sleepq_switch(c92ca6c0,0,c16148ad,2d1,ce9,...) at sleepq_switch+0x11c/frame 0xedde3be0 sleepq_timedwait(c1b7455c,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedde3c1c _sleep(c1b7455c,c1b7453c,0,c15f46fb,ffffdd50,...) at _sleep+0x363/frame 0xedde3c70 soaio_kproc_loop(2,edde3ce8,c1604b60,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedde3cac fork_exit(c0ccca80,2,edde3ce8) at fork_exit+0x7e/frame 0xedde3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedde3cd4 --- trap 0, eip = 0, esp = 0xedde3d20, ebp = 0 --- Tracing command soaiod1 pid 6 tid 100081 td 0xc92caa20 sched_switch(c92caa20,0,104,18a,c92caa20,...) at sched_switch+0x512/frame 0xedddfb60 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedddfba8 sleepq_switch(c92caa20,0,c16148ad,2d1,ce9,...) at sleepq_switch+0x11c/frame 0xedddfbe0 sleepq_timedwait(c1b7455c,0,1d,0,0,...) at sleepq_timedwait+0x53/frame 0xedddfc1c _sleep(c1b7455c,c1b7453c,0,c15f46fb,ffffdd50,...) at _sleep+0x363/frame 0xedddfc70 soaio_kproc_loop(1,edddfce8,c1604b60,406,0,...) at soaio_kproc_loop+0x1fd/frame 0xedddfcac fork_exit(c0ccca80,1,edddfce8) at fork_exit+0x7e/frame 0xedddfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedddfcd4 --- trap 0, eip = 0, esp = 0xedddfd20, ebp = 0 --- Tracing command fdc0 pid 5 tid 100077 td 0xc91b96c0 sched_switch(c91b96c0,0,104,18a,c91b96c0,...) at sched_switch+0x512/frame 0xeddcfb30 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddcfb78 sleepq_switch(c91b96c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddcfbb0 sleepq_wait(c9176a3c,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xeddcfbd4 _sleep(c9176a3c,c9176af0,5c,c15f46fb,0,...) at _sleep+0x38f/frame 0xeddcfc28 fdc_thread(c9176a00,eddcfce8,c1604b60,406,0,...) at fdc_thread+0x14bc/frame 0xeddcfcac fork_exit(c1177ac0,c9176a00,eddcfce8) at fork_exit+0x7e/frame 0xeddcfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddcfcd4 --- trap 0, eip = 0, esp = 0xeddcfd20, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc90f7a20 sched_switch(c90f7a20,0,104,18a,c90f7a20,...) at sched_switch+0x512/frame 0xedd23b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd23be0 sleepq_switch(c90f7a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd23c18 sleepq_wait(c90cfacc,0,edd23c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd23c3c _cv_wait(c90cfacc,c90cfcf8,c15d6a96,76,c90cfad4,...) at _cv_wait+0x21f/frame 0xedd23c80 usb_process(c90cfac4,edd23ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd23cac fork_exit(c0a98e00,c90cfac4,edd23ce8) at fork_exit+0x7e/frame 0xedd23cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd23cd4 --- trap 0, eip = 0, esp = 0xedd23d20, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc90f76c0 sched_switch(c90f76c0,0,104,18a,c90f76c0,...) at sched_switch+0x512/frame 0xedd27b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd27be0 sleepq_switch(c90f76c0,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd27c18 sleepq_wait(c90cfafc,0,edd27c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd27c3c _cv_wait(c90cfafc,c90cfcf8,c15d6a96,76,c90cfb04,...) at _cv_wait+0x21f/frame 0xedd27c80 usb_process(c90cfaf4,edd27ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd27cac fork_exit(c0a98e00,c90cfaf4,edd27ce8) at fork_exit+0x7e/frame 0xedd27cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd27cd4 --- trap 0, eip = 0, esp = 0xedd27d20, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc90f7360 sched_switch(c90f7360,0,104,18a,c90f7360,...) at sched_switch+0x512/frame 0xedd2bb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd2bbe0 sleepq_switch(c90f7360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd2bc18 sleepq_wait(c90cfb2c,0,edd2bc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd2bc3c _cv_wait(c90cfb2c,c90cfcf8,c15d6a96,76,c90cfb34,...) at _cv_wait+0x21f/frame 0xedd2bc80 usb_process(c90cfb24,edd2bce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd2bcac fork_exit(c0a98e00,c90cfb24,edd2bce8) at fork_exit+0x7e/frame 0xedd2bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2bcd4 --- trap 0, eip = 0, esp = 0xedd2bd20, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc90f7000 sched_switch(c90f7000,0,104,18a,c90f7000,...) at sched_switch+0x512/frame 0xedd2fb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd2fbe0 sleepq_switch(c90f7000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd2fc18 sleepq_wait(c90cfb5c,0,edd2fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd2fc3c _cv_wait(c90cfb5c,c90cfcf8,c15d6a96,76,c90cfb64,...) at _cv_wait+0x21f/frame 0xedd2fc80 usb_process(c90cfb54,edd2fce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd2fcac fork_exit(c0a98e00,c90cfb54,edd2fce8) at fork_exit+0x7e/frame 0xedd2fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2fcd4 --- trap 0, eip = 0, esp = 0xedd2fd20, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc90e2a20 sched_switch(c90e2a20,0,104,18a,c90e2a20,...) at sched_switch+0x512/frame 0xedd33b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd33be0 sleepq_switch(c90e2a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd33c18 sleepq_wait(c90cfb8c,0,edd33c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd33c3c _cv_wait(c90cfb8c,c90cfcf8,c15d6a96,76,c90cfb94,...) at _cv_wait+0x21f/frame 0xedd33c80 usb_process(c90cfb84,edd33ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd33cac fork_exit(c0a98e00,c90cfb84,edd33ce8) at fork_exit+0x7e/frame 0xedd33cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd33cd4 --- trap 0, eip = 0, esp = 0xedd33d20, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc90e2360 sched_switch(c90e2360,0,104,18a,c90e2360,...) at sched_switch+0x512/frame 0xedd3cb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd3cbe0 sleepq_switch(c90e2360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd3cc18 sleepq_wait(c913bacc,0,edd3cc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd3cc3c _cv_wait(c913bacc,c913bcf8,c15d6a96,76,c913bad4,...) at _cv_wait+0x21f/frame 0xedd3cc80 usb_process(c913bac4,edd3cce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd3ccac fork_exit(c0a98e00,c913bac4,edd3cce8) at fork_exit+0x7e/frame 0xedd3ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd3ccd4 --- trap 0, eip = 0, esp = 0xedd3cd20, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc90e2000 sched_switch(c90e2000,0,104,18a,c90e2000,...) at sched_switch+0x512/frame 0xedd40b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd40be0 sleepq_switch(c90e2000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd40c18 sleepq_wait(c913bafc,0,edd40c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd40c3c _cv_wait(c913bafc,c913bcf8,c15d6a96,76,c913bb04,...) at _cv_wait+0x21f/frame 0xedd40c80 usb_process(c913baf4,edd40ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd40cac fork_exit(c0a98e00,c913baf4,edd40ce8) at fork_exit+0x7e/frame 0xedd40cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd40cd4 --- trap 0, eip = 0, esp = 0xedd40d20, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc8f49a20 sched_switch(c8f49a20,0,104,18a,c8f49a20,...) at sched_switch+0x512/frame 0xedd44b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd44be0 sleepq_switch(c8f49a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd44c18 sleepq_wait(c913bb2c,0,edd44c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd44c3c _cv_wait(c913bb2c,c913bcf8,c15d6a96,76,c913bb34,...) at _cv_wait+0x21f/frame 0xedd44c80 usb_process(c913bb24,edd44ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd44cac fork_exit(c0a98e00,c913bb24,edd44ce8) at fork_exit+0x7e/frame 0xedd44cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd44cd4 --- trap 0, eip = 0, esp = 0xedd44d20, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc9144000 sched_switch(c9144000,0,104,18a,c9144000,...) at sched_switch+0x512/frame 0xedd48b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd48be0 sleepq_switch(c9144000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd48c18 sleepq_wait(c913bb5c,0,edd48c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd48c3c _cv_wait(c913bb5c,c913bcf8,c15d6a96,76,c913bb64,...) at _cv_wait+0x21f/frame 0xedd48c80 usb_process(c913bb54,edd48ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd48cac fork_exit(c0a98e00,c913bb54,edd48ce8) at fork_exit+0x7e/frame 0xedd48cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd48cd4 --- trap 0, eip = 0, esp = 0xedd48d20, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc9143a20 sched_switch(c9143a20,0,104,18a,c9143a20,...) at sched_switch+0x512/frame 0xedd4cb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd4cbe0 sleepq_switch(c9143a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd4cc18 sleepq_wait(c913bb8c,0,edd4cc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd4cc3c _cv_wait(c913bb8c,c913bcf8,c15d6a96,76,c913bb94,...) at _cv_wait+0x21f/frame 0xedd4cc80 usb_process(c913bb84,edd4cce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd4ccac fork_exit(c0a98e00,c913bb84,edd4cce8) at fork_exit+0x7e/frame 0xedd4ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd4ccd4 --- trap 0, eip = 0, esp = 0xedd4cd20, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc9143360 sched_switch(c9143360,0,104,18a,c9143360,...) at sched_switch+0x512/frame 0xedd55b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd55be0 sleepq_switch(c9143360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd55c18 sleepq_wait(c9145acc,0,edd55c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd55c3c _cv_wait(c9145acc,c9145cf8,c15d6a96,76,c9145ad4,...) at _cv_wait+0x21f/frame 0xedd55c80 usb_process(c9145ac4,edd55ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd55cac fork_exit(c0a98e00,c9145ac4,edd55ce8) at fork_exit+0x7e/frame 0xedd55cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd55cd4 --- trap 0, eip = 0, esp = 0xedd55d20, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc9143000 sched_switch(c9143000,0,104,18a,c9143000,...) at sched_switch+0x512/frame 0xedd59b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd59be0 sleepq_switch(c9143000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd59c18 sleepq_wait(c9145afc,0,edd59c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd59c3c _cv_wait(c9145afc,c9145cf8,c15d6a96,76,c9145b04,...) at _cv_wait+0x21f/frame 0xedd59c80 usb_process(c9145af4,edd59ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd59cac fork_exit(c0a98e00,c9145af4,edd59ce8) at fork_exit+0x7e/frame 0xedd59cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd59cd4 --- trap 0, eip = 0, esp = 0xedd59d20, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc9142a20 sched_switch(c9142a20,0,104,18a,c9142a20,...) at sched_switch+0x512/frame 0xedd5db98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd5dbe0 sleepq_switch(c9142a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd5dc18 sleepq_wait(c9145b2c,0,edd5dc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd5dc3c _cv_wait(c9145b2c,c9145cf8,c15d6a96,76,c9145b34,...) at _cv_wait+0x21f/frame 0xedd5dc80 usb_process(c9145b24,edd5dce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd5dcac fork_exit(c0a98e00,c9145b24,edd5dce8) at fork_exit+0x7e/frame 0xedd5dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd5dcd4 --- trap 0, eip = 0, esp = 0xedd5dd20, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc91426c0 sched_switch(c91426c0,0,104,18a,c91426c0,...) at sched_switch+0x512/frame 0xedd61b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd61be0 sleepq_switch(c91426c0,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd61c18 sleepq_wait(c9145b5c,0,edd61c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd61c3c _cv_wait(c9145b5c,c9145cf8,c15d6a96,76,c9145b64,...) at _cv_wait+0x21f/frame 0xedd61c80 usb_process(c9145b54,edd61ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd61cac fork_exit(c0a98e00,c9145b54,edd61ce8) at fork_exit+0x7e/frame 0xedd61cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd61cd4 --- trap 0, eip = 0, esp = 0xedd61d20, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xc9142360 sched_switch(c9142360,0,104,18a,c9142360,...) at sched_switch+0x512/frame 0xedd65b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd65be0 sleepq_switch(c9142360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd65c18 sleepq_wait(c9145b8c,0,edd65c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd65c3c _cv_wait(c9145b8c,c9145cf8,c15d6a96,76,c9145b94,...) at _cv_wait+0x21f/frame 0xedd65c80 usb_process(c9145b84,edd65ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd65cac fork_exit(c0a98e00,c9145b84,edd65ce8) at fork_exit+0x7e/frame 0xedd65cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd65cd4 --- trap 0, eip = 0, esp = 0xedd65d20, ebp = 0 --- Tracing command usb pid 14 tid 100059 td 0xc9142000 sched_switch(c9142000,0,104,18a,c9142000,...) at sched_switch+0x512/frame 0xedd6ab98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd6abe0 sleepq_switch(c9142000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd6ac18 sleepq_wait(c914cacc,0,edd6ac5c,1,0,...) at sleepq_wait+0x48/frame 0xedd6ac3c _cv_wait(c914cacc,c914ccf8,c15d6a96,76,c914cad4,...) at _cv_wait+0x21f/frame 0xedd6ac80 usb_process(c914cac4,edd6ace8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd6acac fork_exit(c0a98e00,c914cac4,edd6ace8) at fork_exit+0x7e/frame 0xedd6acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd6acd4 --- trap 0, eip = 0, esp = 0xedd6ad20, ebp = 0 --- Tracing command usb pid 14 tid 100060 td 0xc9141a20 sched_switch(c9141a20,0,104,18a,c9141a20,...) at sched_switch+0x512/frame 0xedd6eb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd6ebe0 sleepq_switch(c9141a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd6ec18 sleepq_wait(c914cafc,0,edd6ec5c,1,0,...) at sleepq_wait+0x48/frame 0xedd6ec3c _cv_wait(c914cafc,c914ccf8,c15d6a96,76,c914cb04,...) at _cv_wait+0x21f/frame 0xedd6ec80 usb_process(c914caf4,edd6ece8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd6ecac fork_exit(c0a98e00,c914caf4,edd6ece8) at fork_exit+0x7e/frame 0xedd6ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd6ecd4 --- trap 0, eip = 0, esp = 0xedd6ed20, ebp = 0 --- Tracing command usb pid 14 tid 100061 td 0xc91416c0 sched_switch(c91416c0,0,104,18a,c91416c0,...) at sched_switch+0x512/frame 0xedd72b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd72be0 sleepq_switch(c91416c0,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd72c18 sleepq_wait(c914cb2c,0,edd72c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd72c3c _cv_wait(c914cb2c,c914ccf8,c15d6a96,76,c914cb34,...) at _cv_wait+0x21f/frame 0xedd72c80 usb_process(c914cb24,edd72ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd72cac fork_exit(c0a98e00,c914cb24,edd72ce8) at fork_exit+0x7e/frame 0xedd72cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd72cd4 --- trap 0, eip = 0, esp = 0xedd72d20, ebp = 0 --- Tracing command usb pid 14 tid 100062 td 0xc9141360 sched_switch(c9141360,0,104,18a,c9141360,...) at sched_switch+0x512/frame 0xedd76b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd76be0 sleepq_switch(c9141360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd76c18 sleepq_wait(c914cb5c,0,edd76c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd76c3c _cv_wait(c914cb5c,c914ccf8,c15d6a96,76,c914cb64,...) at _cv_wait+0x21f/frame 0xedd76c80 usb_process(c914cb54,edd76ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd76cac fork_exit(c0a98e00,c914cb54,edd76ce8) at fork_exit+0x7e/frame 0xedd76cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd76cd4 --- trap 0, eip = 0, esp = 0xedd76d20, ebp = 0 --- Tracing command usb pid 14 tid 100063 td 0xc9141000 sched_switch(c9141000,0,104,18a,c9141000,...) at sched_switch+0x512/frame 0xedd7ab98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd7abe0 sleepq_switch(c9141000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd7ac18 sleepq_wait(c914cb8c,0,edd7ac5c,1,0,...) at sleepq_wait+0x48/frame 0xedd7ac3c _cv_wait(c914cb8c,c914ccf8,c15d6a96,76,c914cb94,...) at _cv_wait+0x21f/frame 0xedd7ac80 usb_process(c914cb84,edd7ace8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd7acac fork_exit(c0a98e00,c914cb84,edd7ace8) at fork_exit+0x7e/frame 0xedd7acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd7acd4 --- trap 0, eip = 0, esp = 0xedd7ad20, ebp = 0 --- Tracing command usb pid 14 tid 100064 td 0xc9173360 sched_switch(c9173360,0,104,18a,c9173360,...) at sched_switch+0x512/frame 0xedd7fb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd7fbe0 sleepq_switch(c9173360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd7fc18 sleepq_wait(c916bacc,0,edd7fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd7fc3c _cv_wait(c916bacc,c916bcf8,c15d6a96,76,c916bad4,...) at _cv_wait+0x21f/frame 0xedd7fc80 usb_process(c916bac4,edd7fce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd7fcac fork_exit(c0a98e00,c916bac4,edd7fce8) at fork_exit+0x7e/frame 0xedd7fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd7fcd4 --- trap 0, eip = 0, esp = 0xedd7fd20, ebp = 0 --- Tracing command usb pid 14 tid 100065 td 0xc9173000 sched_switch(c9173000,0,104,18a,c9173000,...) at sched_switch+0x512/frame 0xedd83b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd83be0 sleepq_switch(c9173000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd83c18 sleepq_wait(c916bafc,0,edd83c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd83c3c _cv_wait(c916bafc,c916bcf8,c15d6a96,76,c916bb04,...) at _cv_wait+0x21f/frame 0xedd83c80 usb_process(c916baf4,edd83ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd83cac fork_exit(c0a98e00,c916baf4,edd83ce8) at fork_exit+0x7e/frame 0xedd83cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd83cd4 --- trap 0, eip = 0, esp = 0xedd83d20, ebp = 0 --- Tracing command usb pid 14 tid 100066 td 0xc9172a20 sched_switch(c9172a20,0,104,18a,c9172a20,...) at sched_switch+0x512/frame 0xedd87b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd87be0 sleepq_switch(c9172a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd87c18 sleepq_wait(c916bb2c,0,edd87c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd87c3c _cv_wait(c916bb2c,c916bcf8,c15d6a96,76,c916bb34,...) at _cv_wait+0x21f/frame 0xedd87c80 usb_process(c916bb24,edd87ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd87cac fork_exit(c0a98e00,c916bb24,edd87ce8) at fork_exit+0x7e/frame 0xedd87cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd87cd4 --- trap 0, eip = 0, esp = 0xedd87d20, ebp = 0 --- Tracing command usb pid 14 tid 100067 td 0xc91726c0 sched_switch(c91726c0,0,104,18a,c91726c0,...) at sched_switch+0x512/frame 0xedd8bb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd8bbe0 sleepq_switch(c91726c0,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd8bc18 sleepq_wait(c916bb5c,0,edd8bc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd8bc3c _cv_wait(c916bb5c,c916bcf8,c15d6a96,76,c916bb64,...) at _cv_wait+0x21f/frame 0xedd8bc80 usb_process(c916bb54,edd8bce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd8bcac fork_exit(c0a98e00,c916bb54,edd8bce8) at fork_exit+0x7e/frame 0xedd8bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd8bcd4 --- trap 0, eip = 0, esp = 0xedd8bd20, ebp = 0 --- Tracing command usb pid 14 tid 100068 td 0xc9172360 sched_switch(c9172360,0,104,18a,c9172360,...) at sched_switch+0x512/frame 0xedd8fb98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd8fbe0 sleepq_switch(c9172360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd8fc18 sleepq_wait(c916bb8c,0,edd8fc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd8fc3c _cv_wait(c916bb8c,c916bcf8,c15d6a96,76,c916bb94,...) at _cv_wait+0x21f/frame 0xedd8fc80 usb_process(c916bb84,edd8fce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd8fcac fork_exit(c0a98e00,c916bb84,edd8fce8) at fork_exit+0x7e/frame 0xedd8fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd8fcd4 --- trap 0, eip = 0, esp = 0xedd8fd20, ebp = 0 --- Tracing command usb pid 14 tid 100070 td 0xc9171a20 sched_switch(c9171a20,0,104,18a,c9171a20,...) at sched_switch+0x512/frame 0xedd99b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd99be0 sleepq_switch(c9171a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd99c18 sleepq_wait(c90baca4,0,edd99c5c,1,0,...) at sleepq_wait+0x48/frame 0xedd99c3c _cv_wait(c90baca4,c90baed0,c15d6a96,76,c90bacac,...) at _cv_wait+0x21f/frame 0xedd99c80 usb_process(c90bac9c,edd99ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd99cac fork_exit(c0a98e00,c90bac9c,edd99ce8) at fork_exit+0x7e/frame 0xedd99cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd99cd4 --- trap 0, eip = 0, esp = 0xedd99d20, ebp = 0 --- Tracing command usb pid 14 tid 100071 td 0xc91716c0 sched_switch(c91716c0,0,104,18a,c91716c0,...) at sched_switch+0x512/frame 0xedd9db98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedd9dbe0 sleepq_switch(c91716c0,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedd9dc18 sleepq_wait(c90bacd4,0,edd9dc5c,1,0,...) at sleepq_wait+0x48/frame 0xedd9dc3c _cv_wait(c90bacd4,c90baed0,c15d6a96,76,c90bacdc,...) at _cv_wait+0x21f/frame 0xedd9dc80 usb_process(c90baccc,edd9dce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedd9dcac fork_exit(c0a98e00,c90baccc,edd9dce8) at fork_exit+0x7e/frame 0xedd9dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd9dcd4 --- trap 0, eip = 0, esp = 0xedd9dd20, ebp = 0 --- Tracing command usb pid 14 tid 100072 td 0xc9171360 sched_switch(c9171360,0,104,18a,c9171360,...) at sched_switch+0x512/frame 0xedda1b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedda1be0 sleepq_switch(c9171360,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedda1c18 sleepq_wait(c90bad04,0,edda1c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda1c3c _cv_wait(c90bad04,c90baed0,c15d6a96,76,c90bad0c,...) at _cv_wait+0x21f/frame 0xedda1c80 usb_process(c90bacfc,edda1ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedda1cac fork_exit(c0a98e00,c90bacfc,edda1ce8) at fork_exit+0x7e/frame 0xedda1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda1cd4 --- trap 0, eip = 0, esp = 0xedda1d20, ebp = 0 --- Tracing command usb pid 14 tid 100073 td 0xc9171000 sched_switch(c9171000,0,104,18a,c9171000,...) at sched_switch+0x512/frame 0xedda5b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedda5be0 sleepq_switch(c9171000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedda5c18 sleepq_wait(c90bad34,0,edda5c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda5c3c _cv_wait(c90bad34,c90baed0,c15d6a96,76,c90bad3c,...) at _cv_wait+0x21f/frame 0xedda5c80 usb_process(c90bad2c,edda5ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedda5cac fork_exit(c0a98e00,c90bad2c,edda5ce8) at fork_exit+0x7e/frame 0xedda5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda5cd4 --- trap 0, eip = 0, esp = 0xedda5d20, ebp = 0 --- Tracing command usb pid 14 tid 100074 td 0xc9144a20 sched_switch(c9144a20,0,104,18a,c9144a20,...) at sched_switch+0x512/frame 0xedda9b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedda9be0 sleepq_switch(c9144a20,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xedda9c18 sleepq_wait(c90bad64,0,edda9c5c,1,0,...) at sleepq_wait+0x48/frame 0xedda9c3c _cv_wait(c90bad64,c90baed0,c15d6a96,76,c90bad6c,...) at _cv_wait+0x21f/frame 0xedda9c80 usb_process(c90bad5c,edda9ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xedda9cac fork_exit(c0a98e00,c90bad5c,edda9ce8) at fork_exit+0x7e/frame 0xedda9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedda9cd4 --- trap 0, eip = 0, esp = 0xedda9d20, ebp = 0 --- Tracing command usb pid 14 tid 100114 td 0xc97f6000 sched_switch(c97f6000,0,104,18a,c97f6000,...) at sched_switch+0x512/frame 0xf34e3b98 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xf34e3be0 sleepq_switch(c97f6000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xf34e3c18 sleepq_wait(c8e22c08,0,f34e3c5c,1,0,...) at sleepq_wait+0x48/frame 0xf34e3c3c _cv_wait(c8e22c08,c8e22f34,c15d6a96,76,c8e22c10,...) at _cv_wait+0x21f/frame 0xf34e3c80 usb_process(c8e22c00,f34e3ce8,c1604b60,406,0,...) at usb_process+0x190/frame 0xf34e3cac fork_exit(c0a98e00,c8e22c00,f34e3ce8) at fork_exit+0x7e/frame 0xf34e3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf34e3cd4 --- trap 0, eip = 0, esp = 0xf34e3d20, ebp = 0 --- Tracing command cam pid 4 tid 100037 td 0xc8f47000 sched_switch(c8f47000,0,104,18a,c8f47000,...) at sched_switch+0x512/frame 0xe4bbfb68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4bbfbb0 sleepq_switch(c8f47000,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4bbfbe8 sleepq_wait(c191fc80,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4bbfc0c _sleep(c191fc80,c191fc00,5c,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4bbfc60 xpt_done_td(c191fc00,e4bbfce8,c1604b60,406,0,...) at xpt_done_td+0xee/frame 0xe4bbfcac fork_exit(c04e2870,c191fc00,e4bbfce8) at fork_exit+0x7e/frame 0xe4bbfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bbfcd4 --- trap 0, eip = 0, esp = 0xe4bbfd20, ebp = 0 --- Tracing command cam pid 4 tid 100088 td 0xc92c9000 sched_switch(c92c9000,0,104,18a,c92c9000,...) at sched_switch+0x512/frame 0xeddfcb60 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddfcba8 sleepq_switch(c92c9000,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddfcbe0 sleepq_wait(c191fb2c,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xeddfcc04 _sleep(c191fb2c,c191fb80,5c,c15f46fb,0,...) at _sleep+0x38f/frame 0xeddfcc58 xpt_scanner_thread(0,eddfcce8,c1604b60,406,0,...) at xpt_scanner_thread+0x1b2/frame 0xeddfccac fork_exit(c04e2c30,0,eddfcce8) at fork_exit+0x7e/frame 0xeddfccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddfccd4 --- trap 0, eip = 0, esp = 0xeddfcd20, ebp = 0 --- Tracing command crypto returns pid 3 tid 100033 td 0xc8f48000 sched_switch(c8f48000,0,104,18a,c8f48000,...) at sched_switch+0x512/frame 0xe4bafb68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4bafbb0 sleepq_switch(c8f48000,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4bafbe8 sleepq_wait(c1b7a8bc,6c,c1650cc0,0,0,...) at sleepq_wait+0x48/frame 0xe4bafc0c _sleep(c1b7a8bc,c1b7a8a8,6c,c1650cc0,0,...) at _sleep+0x38f/frame 0xe4bafc60 crypto_ret_proc(0,e4bafce8,c1604b60,406,0,...) at crypto_ret_proc+0x1f5/frame 0xe4bafcac fork_exit(c0ee4dc0,0,e4bafce8) at fork_exit+0x7e/frame 0xe4bafcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bafcd4 --- trap 0, eip = 0, esp = 0xe4bafd20, ebp = 0 --- Tracing command crypto pid 2 tid 100032 td 0xc8f48360 sched_switch(c8f48360,0,104,18a,e4babb9c,...) at sched_switch+0x512/frame 0xe4babb78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4babbbc sleepq_switch(c8f48360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4babbf4 sleepq_wait(c1b7a804,6c,c1650cb4,0,0,...) at sleepq_wait+0x48/frame 0xe4babc18 _sleep(c1b7a804,c1b7a7ec,6c,c1650cb4,0,...) at _sleep+0x38f/frame 0xe4babc6c crypto_proc(0,e4babce8,c1604b60,406,0,...) at crypto_proc+0x9d/frame 0xe4babcac fork_exit(c0ee49c0,0,e4babce8) at fork_exit+0x7e/frame 0xe4babcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4babcd4 --- trap 0, eip = 0, esp = 0xe4babd20, ebp = 0 --- Tracing command geom pid 13 tid 100028 td 0xc8f49360 sched_switch(c8f49360,0,104,18a,c8f49360,...) at sched_switch+0x512/frame 0xe4b9bb78 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b9bbc0 sleepq_switch(c8f49360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b9bbf8 sleepq_wait(c1bd5f44,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4b9bc1c _sleep(c1bd5f44,c1af8278,25c,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4b9bc70 g_run_events(0,e4b9bce8,c1604b60,406,0,...) at g_run_events+0x141/frame 0xe4b9bcac fork_exit(c0b9da00,0,e4b9bce8) at fork_exit+0x7e/frame 0xe4b9bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b9bcd4 --- trap 0, eip = 0, esp = 0xe4b9bd20, ebp = 0 --- Tracing command geom pid 13 tid 100029 td 0xc8f49000 sched_switch(c8f49000,0,104,18a,c8f49000,...) at sched_switch+0x512/frame 0xe4b9fb68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b9fbb0 sleepq_switch(c8f49000,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b9fbe8 sleepq_wait(c1bd5f48,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4b9fc0c _sleep(c1bd5f48,c1af82c0,25c,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4b9fc60 g_io_schedule_up(c8f49000,0,c15f6732,5f,e4b9fcd4,...) at g_io_schedule_up+0xcf/frame 0xe4b9fc94 g_up_procbody(0,e4b9fce8,c1604b60,406,0,...) at g_up_procbody+0x7d/frame 0xe4b9fcac fork_exit(c0b9da80,0,e4b9fce8) at fork_exit+0x7e/frame 0xe4b9fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b9fcd4 --- trap 0, eip = 0, esp = 0xe4b9fd20, ebp = 0 --- Tracing command geom pid 13 tid 100030 td 0xc8f48a20 sched_switch(c8f48a20,0,104,18a,e4ba3b8c,...) at sched_switch+0x512/frame 0xe4ba3b68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4ba3bac sleepq_switch(c8f48a20,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4ba3be4 sleepq_wait(c1bd5f4c,5c,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4ba3c08 _sleep(c1bd5f4c,c1af82a0,25c,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4ba3c5c g_io_schedule_down(c8f48a20,0,c15f6732,6b,e4ba3cd4,...) at g_io_schedule_down+0xec/frame 0xe4ba3c94 g_down_procbody(0,e4ba3ce8,c1604b60,406,0,...) at g_down_procbody+0x7d/frame 0xe4ba3cac fork_exit(c0b9db00,0,e4ba3ce8) at fork_exit+0x7e/frame 0xe4ba3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4ba3cd4 --- trap 0, eip = 0, esp = 0xe4ba3d20, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc8e666c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xc8e66360 sched_switch(c8e66360,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b4bc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xe4b4bc70 ithread_loop(c8d01d90,e4b4bce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xe4b4bcac fork_exit(c0c1a9b0,c8d01d90,e4b4bce8) at fork_exit+0x7e/frame 0xe4b4bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b4bcd4 --- trap 0, eip = 0, esp = 0xe4b4bd20, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc8e66000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc8e5ba20 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc8e5b6c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc8e5b360 sched_switch(c8e5b360,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b5bc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xe4b5bc70 ithread_loop(c8d01d50,e4b5bce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xe4b5bcac fork_exit(c0c1a9b0,c8d01d50,e4b5bce8) at fork_exit+0x7e/frame 0xe4b5bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5bcd4 --- trap 0, eip = 0, esp = 0xe4b5bd20, ebp = 0 --- Tracing command intr pid 12 tid 100023 td 0xc8f22000 sched_switch(c8f22000,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b87c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xe4b87c70 ithread_loop(c8d01d40,e4b87ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xe4b87cac fork_exit(c0c1a9b0,c8d01d40,e4b87ce8) at fork_exit+0x7e/frame 0xe4b87cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b87cd4 --- trap 0, eip = 0, esp = 0xe4b87d20, ebp = 0 --- Tracing command intr pid 12 tid 100024 td 0xc8e68a20 sched_switch(c8e68a20,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4b8bc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xe4b8bc70 ithread_loop(c8d01d30,e4b8bce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xe4b8bcac fork_exit(c0c1a9b0,c8d01d30,e4b8bce8) at fork_exit+0x7e/frame 0xe4b8bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b8bcd4 --- trap 0, eip = 0, esp = 0xe4b8bd20, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc8f496c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xc90f8a20 sched_switch(c90f8a20,0,109,18a,0,...) at sched_switch+0x512/frame 0xe4bf2c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xe4bf2c70 ithread_loop(c9085d40,e4bf2ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xe4bf2cac fork_exit(c0c1a9b0,c9085d40,e4bf2ce8) at fork_exit+0x7e/frame 0xe4bf2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bf2cd4 --- trap 0, eip = 0, esp = 0xe4bf2d20, ebp = 0 --- Tracing command intr pid 12 tid 100039 td 0xc90f86c0 sched_switch(c90f86c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xf0efbc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xf0efbc70 ithread_loop(c9085780,f0efbce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xf0efbcac fork_exit(c0c1a9b0,c9085780,f0efbce8) at fork_exit+0x7e/frame 0xf0efbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0efbcd4 --- trap 0, eip = 0, esp = 0xf0efbd20, ebp = 0 --- Tracing command intr pid 12 tid 100040 td 0xc90f8360 sched_switch(c90f8360,0,109,18a,0,...) at sched_switch+0x512/frame 0xf0f00c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xf0f00c70 ithread_loop(c90853e0,f0f00ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xf0f00cac fork_exit(c0c1a9b0,c90853e0,f0f00ce8) at fork_exit+0x7e/frame 0xf0f00cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0f00cd4 --- trap 0, eip = 0, esp = 0xf0f00d20, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc90f8000 sched_switch(c90f8000,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd1fc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xedd1fc70 ithread_loop(c90dae60,edd1fce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xedd1fcac fork_exit(c0c1a9b0,c90dae60,edd1fce8) at fork_exit+0x7e/frame 0xedd1fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd1fcd4 --- trap 0, eip = 0, esp = 0xedd1fd20, ebp = 0 --- Tracing command intr pid 12 tid 100047 td 0xc90e26c0 sched_switch(c90e26c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd38c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xedd38c70 ithread_loop(c90dac10,edd38ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xedd38cac fork_exit(c0c1a9b0,c90dac10,edd38ce8) at fork_exit+0x7e/frame 0xedd38cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd38cd4 --- trap 0, eip = 0, esp = 0xedd38d20, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc91436c0 sched_switch(c91436c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd51c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xedd51c70 ithread_loop(c90daa20,edd51ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xedd51cac fork_exit(c0c1a9b0,c90daa20,edd51ce8) at fork_exit+0x7e/frame 0xedd51cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd51cd4 --- trap 0, eip = 0, esp = 0xedd51d20, ebp = 0 --- Tracing command intr pid 12 tid 100069 td 0xc9172000 sched_switch(c9172000,0,109,18a,0,...) at sched_switch+0x512/frame 0xedd95c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xedd95c70 ithread_loop(c90da410,edd95ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xedd95cac fork_exit(c0c1a9b0,c90da410,edd95ce8) at fork_exit+0x7e/frame 0xedd95cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd95cd4 --- trap 0, eip = 0, esp = 0xedd95d20, ebp = 0 --- Tracing command intr pid 12 tid 100075 td 0xc91446c0 sched_switch(c91446c0,0,109,18a,0,...) at sched_switch+0x512/frame 0xeddafc28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xeddafc70 ithread_loop(c91960d0,eddafce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xeddafcac fork_exit(c0c1a9b0,c91960d0,eddafce8) at fork_exit+0x7e/frame 0xeddafcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddafcd4 --- trap 0, eip = 0, esp = 0xeddafd20, ebp = 0 --- Tracing command intr pid 12 tid 100076 td 0xc9144360 sched_switch(c9144360,0,109,18a,0,...) at sched_switch+0x512/frame 0xeddc1c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xeddc1c70 ithread_loop(c9196ce0,eddc1ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xeddc1cac fork_exit(c0c1a9b0,c9196ce0,eddc1ce8) at fork_exit+0x7e/frame 0xeddc1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddc1cd4 --- trap 0, eip = 0, esp = 0xeddc1d20, ebp = 0 --- Tracing command intr pid 12 tid 100078 td 0xc91b9360 sched_switch(c91b9360,0,109,18a,0,...) at sched_switch+0x512/frame 0xeddd3c28 mi_switch(109,0,c1604edd,59f,c1604b60,...) at mi_switch+0x226/frame 0xeddd3c70 ithread_loop(c91968c0,eddd3ce8,c1604b60,406,0,...) at ithread_loop+0x200/frame 0xeddd3cac fork_exit(c0c1a9b0,c91968c0,eddd3ce8) at fork_exit+0x7e/frame 0xeddd3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddd3cd4 --- trap 0, eip = 0, esp = 0xeddd3d20, ebp = 0 --- Tracing command intr pid 12 tid 100079 td 0xc91b9000 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc8e676c0 (CPU 0) cpustop_handler(0,13,e4b36b8c,c120cece,c160eaef,...) at cpustop_handler+0x25/frame 0xe4b36a40 ipi_nmi_handler(c160eaef,e4b36a90,c0c3747b,c1afa4e8,c1b9598c,...) at ipi_nmi_handler+0x36/frame 0xe4b36a50 trap(e4b36b98) at trap+0xae/frame 0xe4b36b8c calltrap() at calltrap+0x6/frame 0xe4b36b8c --- trap 0x13, eip = 0xc1228005, esp = 0xe4b36bd8, ebp = 0xe4b36bd8 --- acpi_cpu_c1(e4b36c2c,c1217319,c2027c90,0,c181bc83,...) at acpi_cpu_c1+0x5/frame 0xe4b36bd8 acpi_cpu_idle(a316b5a,0,a316b5a,e4b36c48,c12283d6,...) at acpi_cpu_idle+0x325/frame 0xe4b36c18 cpu_idle_acpi(a316b5a,0,c1b08780,4,c8e676c0,...) at cpu_idle_acpi+0x3f/frame 0xe4b36c2c cpu_idle(0,e4b36c80,c16105b7,407,0,...) at cpu_idle+0xa6/frame 0xe4b36c48 sched_idletd(0,e4b36ce8,c1604b60,406,0,...) at sched_idletd+0x16a/frame 0xe4b36cac fork_exit(c0c88330,0,e4b36ce8) at fork_exit+0x7e/frame 0xe4b36cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b36cd4 --- trap 0, eip = 0, esp = 0xe4b36d20, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc8e67360 (CPU 1) cpustop_handler(1,13,e4b3ab8c,c120cece,a,...) at cpustop_handler+0x25/frame 0xe4b3aa40 ipi_nmi_handler(a,c160f34d,c1b95cd4,1,0,...) at ipi_nmi_handler+0x36/frame 0xe4b3aa50 trap(e4b3ab98) at trap+0xae/frame 0xe4b3ab8c calltrap() at calltrap+0x6/frame 0xe4b3ab8c --- trap 0x13, eip = 0xc1228005, esp = 0xe4b3abd8, ebp = 0xe4b3abd8 --- acpi_cpu_c1(e4b3ac2c,c1217319,e4b18c90,0,c181bc83,...) at acpi_cpu_c1+0x5/frame 0xe4b3abd8 acpi_cpu_idle(2c691aae,0,2c691aae,e4b3ac48,c12283d6,...) at acpi_cpu_idle+0x325/frame 0xe4b3ac18 cpu_idle_acpi(2c691aae,0,c1b08e80,6,c8e67360,...) at cpu_idle_acpi+0x3f/frame 0xe4b3ac2c cpu_idle(0,e4b3ac80,c16105b7,407,0,...) at cpu_idle+0xa6/frame 0xe4b3ac48 sched_idletd(0,e4b3ace8,c1604b60,406,0,...) at sched_idletd+0x16a/frame 0xe4b3acac fork_exit(c0c88330,0,e4b3ace8) at fork_exit+0x7e/frame 0xe4b3acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3acd4 --- trap 0, eip = 0, esp = 0xe4b3ad20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc8e67000 (CPU 2) cpustop_handler(2,13,e4b3eb8c,c120cece,a,...) at cpustop_handler+0x25/frame 0xe4b3ea40 ipi_nmi_handler(a,c160f34d,c1b9601c,2,0,...) at ipi_nmi_handler+0x36/frame 0xe4b3ea50 trap(e4b3eb98) at trap+0xae/frame 0xe4b3eb8c calltrap() at calltrap+0x6/frame 0xe4b3eb8c --- trap 0x13, eip = 0xc1228005, esp = 0xe4b3ebd8, ebp = 0xe4b3ebd8 --- acpi_cpu_c1(e4b3ec2c,c1217319,e4b1cc90,0,c181bc83,...) at acpi_cpu_c1+0x5/frame 0xe4b3ebd8 acpi_cpu_idle(78024508,0,78024508,e4b3ec48,c12283d6,...) at acpi_cpu_idle+0x325/frame 0xe4b3ec18 cpu_idle_acpi(78024508,0,c1b09580,5,c8e67000,...) at cpu_idle_acpi+0x3f/frame 0xe4b3ec2c cpu_idle(0,e4b3ec80,c16105b7,a6a,0,...) at cpu_idle+0xa6/frame 0xe4b3ec48 sched_idletd(0,e4b3ece8,c1604b60,406,0,...) at sched_idletd+0x16a/frame 0xe4b3ecac fork_exit(c0c88330,0,e4b3ece8) at fork_exit+0x7e/frame 0xe4b3ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3ecd4 --- trap 0, eip = 0, esp = 0xe4b3ed20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc8e66a20 (CPU 3) kdb_alt_break_internal(0,e4b42ac4,c0a5b2a7,2,c91a6888,...) at kdb_alt_break_internal+0x101/frame 0xe4b42a98 kdb_alt_break(2,c91a6888,c15ccb4b,0,40000,...) at kdb_alt_break+0x16/frame 0xe4b42aa4 uart_intr_rxready(c91a6800,0,e4b42b04,c122d2e3,8,...) at uart_intr_rxready+0x87/frame 0xe4b42ac4 uart_intr(c91a6800,c1bd7910,3,c160f34d,0,...) at uart_intr+0x120/frame 0xe4b42b04 intr_event_handle(c8eb3e00,e4b42b98,ffffffff,80000000,1,...) at intr_event_handle+0x75/frame 0xe4b42b30 intr_execute_handlers(c8eae908,e4b42b98,3) at intr_execute_handlers+0xb0/frame 0xe4b42b68 lapic_handle_intr(32,e4b42b98) at lapic_handle_intr+0x5b/frame 0xe4b42b88 Xapic_isr1() at Xapic_isr1+0x55/frame 0xe4b42b88 --- interrupt, eip = 0xc1228005, esp = 0xe4b42bd8, ebp = 0xe4b42bd8 --- acpi_cpu_c1(e4b42c2c,c1217319,e4b20c90,0,c181bc83,...) at acpi_cpu_c1+0x5/frame 0xe4b42bd8 acpi_cpu_idle(7665f0cc,0,7665f0cc,e4b42c48,c12283d6,...) at acpi_cpu_idle+0x325/frame 0xe4b42c18 cpu_idle_acpi(7665f0cc,0,c1b09c80,4,c8e66a20,...) at cpu_idle_acpi+0x3f/frame 0xe4b42c2c cpu_idle(0,e4b42c80,c16105b7,a6a,0,...) at cpu_idle+0xa6/frame 0xe4b42c48 sched_idletd(0,e4b42ce8,c1604b60,406,0,...) at sched_idletd+0x16a/frame 0xe4b42cac fork_exit(c0c88330,0,e4b42ce8) at fork_exit+0x7e/frame 0xe4b42cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b42cd4 --- trap 0, eip = 0, esp = 0xe4b42d20, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc8e67a20 sched_switch(c8e67a20,0,104,18a,e4b3192c,...) at sched_switch+0x512/frame 0xe4b31908 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b3194c sleepq_switch(c8e67a20,0,c16148ad,1c5,5b0,...) at sleepq_switch+0x11c/frame 0xe4b31984 sleepq_catch_signals(c16148ad,173,c8e59ae4,222,100,...) at sleepq_catch_signals+0x36e/frame 0xe4b319c4 sleepq_wait_sig(c8e59a2c,6c,c1618b3f,100,0,...) at sleepq_wait_sig+0x14/frame 0xe4b319e8 _sleep(c8e59a2c,c8e59ad4,16c,c1618b3f,0,...) at _sleep+0x37c/frame 0xe4b31a3c kern_wait6(c8e67a20,7,0,0,e4b31acc,...) at kern_wait6+0x23d/frame 0xe4b31aa0 sys_wait4(c8e67a20,e4b31c00,c160459f,4,ffffffff,...) at sys_wait4+0x87/frame 0xe4b31bc0 syscall(e4b31ce8) at syscall+0x75c/frame 0xe4b31cdc Xint0x80_syscall() at Xint0x80_syscall+0x2e/frame 0xe4b31cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8065ad7, esp = 0xbfbfe834, ebp = 0xbfbfe848 --- Tracing command audit pid 10 tid 100001 td 0xc8e68000 sched_switch(c8e68000,0,104,18a,c8e68000,...) at sched_switch+0x512/frame 0xe4b2db68 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b2dbb0 sleepq_switch(c8e68000,0,c16148ad,2ae,c18d7ed8,...) at sleepq_switch+0x11c/frame 0xe4b2dbe8 sleepq_wait(c1bddd90,0,e4b2dc2c,1,0,...) at sleepq_wait+0x48/frame 0xe4b2dc0c _cv_wait(c1bddd90,c1bddd14,c165330e,1bc,0,...) at _cv_wait+0x21f/frame 0xe4b2dc50 audit_worker(0,e4b2dce8,c1604b60,406,0,...) at audit_worker+0x3c4/frame 0xe4b2dcac fork_exit(c0f08180,0,e4b2dce8) at fork_exit+0x7e/frame 0xe4b2dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b2dcd4 --- trap 0, eip = 0, esp = 0xe4b2dd20, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1bd62e0 sched_switch(c1bd62e0,0,104,18a,c1bd62e0,...) at sched_switch+0x512/frame 0xc2022be0 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xc2022c28 sleepq_switch(c1bd62e0,0,c16148ad,2d1,ce1,...) at sleepq_switch+0x11c/frame 0xc2022c60 sleepq_timedwait(c1bd5f64,54,9,0,0,...) at sleepq_timedwait+0x53/frame 0xc2022c9c _sleep(c1bd5f64,0,54,c165f09a,fffff470,...) at _sleep+0x363/frame 0xc2022cf0 swapper() at swapper+0xfa/frame 0xc2022d38 begin() at begin+0x22 Tracing command kernel pid 0 tid 100013 td 0xc8e5b000 sched_switch(c8e5b000,0,104,18a,e4b5fbb4,...) at sched_switch+0x512/frame 0xe4b5fb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b5fbd4 sleepq_switch(c8e5b000,0,c16148ad,2ae,c8e34f00,...) at sleepq_switch+0x11c/frame 0xe4b5fc0c sleepq_wait(c8e34f00,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b5fc30 msleep_spin_sbt(c8e34f00,c8e34f1c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b5fc74 gtaskqueue_thread_loop(c8e24c04,e4b5fce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b5fcac fork_exit(c0c9c320,c8e24c04,e4b5fce8) at fork_exit+0x7e/frame 0xe4b5fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5fcd4 --- trap 0, eip = 0, esp = 0xe4b5fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc8f24360 sched_switch(c8f24360,0,104,18a,e4b63bb4,...) at sched_switch+0x512/frame 0xe4b63b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b63bd4 sleepq_switch(c8f24360,0,c16148ad,2ae,c8e34e80,...) at sleepq_switch+0x11c/frame 0xe4b63c0c sleepq_wait(c8e34e80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b63c30 msleep_spin_sbt(c8e34e80,c8e34e9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b63c74 gtaskqueue_thread_loop(c8e24804,e4b63ce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b63cac fork_exit(c0c9c320,c8e24804,e4b63ce8) at fork_exit+0x7e/frame 0xe4b63cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b63cd4 --- trap 0, eip = 0, esp = 0xe4b63d20, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc8f24000 sched_switch(c8f24000,0,104,18a,e4b67bb4,...) at sched_switch+0x512/frame 0xe4b67b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b67bd4 sleepq_switch(c8f24000,0,c16148ad,2ae,c8e34e00,...) at sleepq_switch+0x11c/frame 0xe4b67c0c sleepq_wait(c8e34e00,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b67c30 msleep_spin_sbt(c8e34e00,c8e34e1c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b67c74 gtaskqueue_thread_loop(c8e24814,e4b67ce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b67cac fork_exit(c0c9c320,c8e24814,e4b67ce8) at fork_exit+0x7e/frame 0xe4b67cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b67cd4 --- trap 0, eip = 0, esp = 0xe4b67d20, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc8f23a20 sched_switch(c8f23a20,0,104,18a,e4b6bbb4,...) at sched_switch+0x512/frame 0xe4b6bb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b6bbd4 sleepq_switch(c8f23a20,0,c16148ad,2ae,c8e34d80,...) at sleepq_switch+0x11c/frame 0xe4b6bc0c sleepq_wait(c8e34d80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b6bc30 msleep_spin_sbt(c8e34d80,c8e34d9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b6bc74 gtaskqueue_thread_loop(c8e24824,e4b6bce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b6bcac fork_exit(c0c9c320,c8e24824,e4b6bce8) at fork_exit+0x7e/frame 0xe4b6bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6bcd4 --- trap 0, eip = 0, esp = 0xe4b6bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc8f236c0 sched_switch(c8f236c0,0,104,18a,e4b6fbb4,...) at sched_switch+0x512/frame 0xe4b6fb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b6fbd4 sleepq_switch(c8f236c0,0,c16148ad,2ae,c8e34d00,...) at sleepq_switch+0x11c/frame 0xe4b6fc0c sleepq_wait(c8e34d00,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b6fc30 msleep_spin_sbt(c8e34d00,c8e34d1c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b6fc74 gtaskqueue_thread_loop(c8e24834,e4b6fce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b6fcac fork_exit(c0c9c320,c8e24834,e4b6fce8) at fork_exit+0x7e/frame 0xe4b6fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6fcd4 --- trap 0, eip = 0, esp = 0xe4b6fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc8f23360 sched_switch(c8f23360,0,104,18a,e4b73bb4,...) at sched_switch+0x512/frame 0xe4b73b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b73bd4 sleepq_switch(c8f23360,0,c16148ad,2ae,c8e34c80,...) at sleepq_switch+0x11c/frame 0xe4b73c0c sleepq_wait(c8e34c80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b73c30 msleep_spin_sbt(c8e34c80,c8e34c9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b73c74 gtaskqueue_thread_loop(c8e24404,e4b73ce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b73cac fork_exit(c0c9c320,c8e24404,e4b73ce8) at fork_exit+0x7e/frame 0xe4b73cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b73cd4 --- trap 0, eip = 0, esp = 0xe4b73d20, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc8f23000 sched_switch(c8f23000,0,104,18a,e4b77bb4,...) at sched_switch+0x512/frame 0xe4b77b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b77bd4 sleepq_switch(c8f23000,0,c16148ad,2ae,c8e34c00,...) at sleepq_switch+0x11c/frame 0xe4b77c0c sleepq_wait(c8e34c00,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b77c30 msleep_spin_sbt(c8e34c00,c8e34c1c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b77c74 gtaskqueue_thread_loop(c8e24414,e4b77ce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b77cac fork_exit(c0c9c320,c8e24414,e4b77ce8) at fork_exit+0x7e/frame 0xe4b77cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b77cd4 --- trap 0, eip = 0, esp = 0xe4b77d20, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc8f22a20 sched_switch(c8f22a20,0,104,18a,e4b7bbb4,...) at sched_switch+0x512/frame 0xe4b7bb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b7bbd4 sleepq_switch(c8f22a20,0,c16148ad,2ae,c8e34b80,...) at sleepq_switch+0x11c/frame 0xe4b7bc0c sleepq_wait(c8e34b80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b7bc30 msleep_spin_sbt(c8e34b80,c8e34b9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b7bc74 gtaskqueue_thread_loop(c8e24424,e4b7bce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b7bcac fork_exit(c0c9c320,c8e24424,e4b7bce8) at fork_exit+0x7e/frame 0xe4b7bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7bcd4 --- trap 0, eip = 0, esp = 0xe4b7bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc8f226c0 sched_switch(c8f226c0,0,104,18a,e4b7fbb4,...) at sched_switch+0x512/frame 0xe4b7fb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b7fbd4 sleepq_switch(c8f226c0,0,c16148ad,2ae,c8e34b00,...) at sleepq_switch+0x11c/frame 0xe4b7fc0c sleepq_wait(c8e34b00,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4b7fc30 msleep_spin_sbt(c8e34b00,c8e34b1c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4b7fc74 gtaskqueue_thread_loop(c8e24434,e4b7fce8,c1604b60,406,0,...) at gtaskqueue_thread_loop+0xf3/frame 0xe4b7fcac fork_exit(c0c9c320,c8e24434,e4b7fce8) at fork_exit+0x7e/frame 0xe4b7fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7fcd4 --- trap 0, eip = 0, esp = 0xe4b7fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc8f22360 sched_switch(c8f22360,0,104,18a,e4b83ba4,...) at sched_switch+0x512/frame 0xe4b83b80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b83bc4 sleepq_switch(c8f22360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b83bfc sleepq_wait(c8e34a80,0,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4b83c20 _sleep(c8e34a80,c8e34a9c,0,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4b83c74 taskqueue_thread_loop(c1bd6868,e4b83ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b83cac fork_exit(c0cb0ea0,c1bd6868,e4b83ce8) at fork_exit+0x7e/frame 0xe4b83cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b83cd4 --- trap 0, eip = 0, esp = 0xe4b83d20, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc8e686c0 sched_switch(c8e686c0,0,104,18a,e4b8fba4,...) at sched_switch+0x512/frame 0xe4b8fb80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b8fbc4 sleepq_switch(c8e686c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b8fbfc sleepq_wait(c8e34900,0,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4b8fc20 _sleep(c8e34900,c8e3491c,0,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4b8fc74 taskqueue_thread_loop(c1bda6a4,e4b8fce8,c1604b60,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b8fcac fork_exit(c0cb0ea0,c1bda6a4,e4b8fce8) at fork_exit+0x7e/frame 0xe4b8fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b8fcd4 --- trap 0, eip = 0, esp = 0xe4b8fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc8e68360 sched_switch(c8e68360,0,104,18a,e4b93ba4,...) at sched_switch+0x512/frame 0xe4b93b80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4b93bc4 sleepq_switch(c8e68360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4b93bfc sleepq_wait(c8e34880,0,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4b93c20 _sleep(c8e34880,c8e3489c,0,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4b93c74 taskqueue_thread_loop(c1bda720,e4b93ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b93cac fork_exit(c0cb0ea0,c1bda720,e4b93ce8) at fork_exit+0x7e/frame 0xe4b93cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b93cd4 --- trap 0, eip = 0, esp = 0xe4b93d20, ebp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xc8f486c0 sched_switch(c8f486c0,0,104,18a,e4ba7ba4,...) at sched_switch+0x512/frame 0xe4ba7b80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4ba7bc4 sleepq_switch(c8f486c0,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xe4ba7bfc sleepq_wait(c8e34680,0,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xe4ba7c20 _sleep(c8e34680,c8e3469c,0,c15f46fb,0,...) at _sleep+0x38f/frame 0xe4ba7c74 taskqueue_thread_loop(c1b168a0,e4ba7ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0x144/frame 0xe4ba7cac fork_exit(c0cb0ea0,c1b168a0,e4ba7ce8) at fork_exit+0x7e/frame 0xe4ba7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4ba7cd4 --- trap 0, eip = 0, esp = 0xe4ba7d20, ebp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xc8f47a20 sched_switch(c8f47a20,0,104,18a,e4bb3bb4,...) at sched_switch+0x512/frame 0xe4bb3b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4bb3bd4 sleepq_switch(c8f47a20,0,c16148ad,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bb3c0c sleepq_wait(c8e33d80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4bb3c30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bb3c74 taskqueue_thread_loop(c1bcba9c,e4bb3ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bb3cac fork_exit(c0cb0ea0,c1bcba9c,e4bb3ce8) at fork_exit+0x7e/frame 0xe4bb3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bb3cd4 --- trap 0, eip = 0, esp = 0xe4bb3d20, ebp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xc8f476c0 sched_switch(c8f476c0,0,104,18a,e4bb7bb4,...) at sched_switch+0x512/frame 0xe4bb7b90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4bb7bd4 sleepq_switch(c8f476c0,0,c16148ad,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bb7c0c sleepq_wait(c8e33d80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4bb7c30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bb7c74 taskqueue_thread_loop(c1bcba9c,e4bb7ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bb7cac fork_exit(c0cb0ea0,c1bcba9c,e4bb7ce8) at fork_exit+0x7e/frame 0xe4bb7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bb7cd4 --- trap 0, eip = 0, esp = 0xe4bb7d20, ebp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xc8f47360 sched_switch(c8f47360,0,104,18a,e4bbbbb4,...) at sched_switch+0x512/frame 0xe4bbbb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xe4bbbbd4 sleepq_switch(c8f47360,0,c16148ad,2ae,c8e33d80,...) at sleepq_switch+0x11c/frame 0xe4bbbc0c sleepq_wait(c8e33d80,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xe4bbbc30 msleep_spin_sbt(c8e33d80,c8e33d9c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xe4bbbc74 taskqueue_thread_loop(c1bcba9c,e4bbbce8,c1604b60,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4bbbcac fork_exit(c0cb0ea0,c1bcba9c,e4bbbce8) at fork_exit+0x7e/frame 0xe4bbbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bbbcd4 --- trap 0, eip = 0, esp = 0xe4bbbd20, ebp = 0 --- Tracing command kernel pid 0 tid 100080 td 0xc91b8a20 sched_switch(c91b8a20,0,104,18a,edddbbb4,...) at sched_switch+0x512/frame 0xedddbb90 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xedddbbd4 sleepq_switch(c91b8a20,0,c16148ad,2ae,c918a100,...) at sleepq_switch+0x11c/frame 0xedddbc0c sleepq_wait(c918a100,0,c160e459,c15f46fb,0,...) at sleepq_wait+0x48/frame 0xedddbc30 msleep_spin_sbt(c918a100,c918a11c,c15f46fb,0,0,...) at msleep_spin_sbt+0x226/frame 0xedddbc74 taskqueue_thread_loop(c1bc9e94,edddbce8,c1604b60,406,0,...) at taskqueue_thread_loop+0xf3/frame 0xedddbcac fork_exit(c0cb0ea0,c1bc9e94,edddbce8) at fork_exit+0x7e/frame 0xedddbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedddbcd4 --- trap 0, eip = 0, esp = 0xedddbd20, ebp = 0 --- Tracing command kernel pid 0 tid 100087 td 0xc92c9360 sched_switch(c92c9360,0,104,18a,eddf8ba4,...) at sched_switch+0x512/frame 0xeddf8b80 mi_switch(104,0,c16148ad,226,0,...) at mi_switch+0x226/frame 0xeddf8bc4 sleepq_switch(c92c9360,0,c16148ad,2ae,0,...) at sleepq_switch+0x11c/frame 0xeddf8bfc sleepq_wait(c8e33d00,0,c15f46fb,0,0,...) at sleepq_wait+0x48/frame 0xeddf8c20 _sleep(c8e33d00,c8e33d1c,0,c15f46fb,0,...) at _sleep+0x38f/frame 0xeddf8c74 taskqueue_thread_loop(c191fba8,eddf8ce8,c1604b60,406,0,...) at taskqueue_thread_loop+0x144/frame 0xeddf8cac fork_exit(c0cb0ea0,c191fba8,eddf8ce8) at fork_exit+0x7e/frame 0xeddf8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xeddf8cd4 --- trap 0, eip = 0, esp = 0xeddf8d20, ebp = 0 --- db:0:allt> dump Physical memory: 3427 MB Dumping 253 MB: 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 Dump complete db:0:dump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) proc 6816 [Switching to thread 107 (Thread 100302)]#0 sched_switch (td=, newtd=, flags=) at ../../../kern/sched_ule.c:1988 1988 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=, newtd=, flags=) at ../../../kern/sched_ule.c:1988 #1 0xc0c63fa6 in mi_switch (flags=) at ../../../kern/kern_synch.c:438 #2 0xc0cab51c in sleepq_switch (wchan=, pri=0x60) at ../../../kern/subr_sleepqueue.c:608 #3 0xc0cab3a8 in sleepq_wait (wchan=0xcae98000, pri=0x60) at ../../../kern/subr_sleepqueue.c:687 #4 0xc0c6370f in _sleep (lock=, priority=, wmesg=, sbt=0x0, pr=0x0, flags=) at ../../../kern/kern_synch.c:216 #5 0xc0d1c61f in vfs_mount_destroy (mp=0xcae98000) at ../../../kern/vfs_mount.c:590 #6 0xc0d1ef85 in dounmount (mp=0xcae98000, flags=0x8080000, td=0xcafb3a20) at ../../../kern/vfs_mount.c:1518 #7 0xc0d1dab7 in sys_unmount (td=0xcafb3a20, uap=0xc84d7c00) at ../../../kern/vfs_mount.c:1293 #8 0xc120e73c in syscall (frame=) at subr_syscall.c:136 #9 0xc11fa11e in Xint0x80_syscall () at ../../../i386/i386/exception.s:277 #10 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xc0d1c61f in vfs_mount_destroy (mp=0xcae98000) at ../../../kern/vfs_mount.c:590 590 msleep(mp, MNT_MTX(mp), PVFS, "mntref", 0); (kgdb) l 585 if (mp->mnt_kern_flag & MNTK_MWAIT) { 586 mp->mnt_kern_flag &= ~MNTK_MWAIT; 587 wakeup(mp); 588 } 589 while (mp->mnt_ref) 590 msleep(mp, MNT_MTX(mp), PVFS, "mntref", 0); 591 KASSERT(mp->mnt_ref == 0, 592 ("%s: invalid refcount in the drain path @ %s:%d", __func__, 593 __FILE__, __LINE__)); 594 if (mp->mnt_writeopcount != 0) (kgdb) p *mp $1 = {mnt_mtx = {lock_object = {lo_name = 0xc1617485 "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc89ac588}, mtx_lock = 0x4}, mnt_gen = 0x1, mnt_list = {tqe_next = 0xcae9a000, tqe_prev = 0xcae96018}, mnt_op = 0xc190475c, mnt_vfc = 0xc19047f4, mnt_vnodecovered = 0xca99a708, mnt_syncer = 0x0, mnt_ref = 0x1, mnt_nvnodelist = {tqh_first = 0x0, tqh_last = 0xcae98034}, mnt_nvnodelistsize = 0x0, mnt_writeopcount = 0x0, mnt_kern_flag = 0x4180616d, mnt_flag = 0x0, mnt_opt = 0xca0720b0, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x78, mnt_stat = {f_version = 0x20140518, f_type = 0x35, f_flags = 0x201000, f_bsize = 0x1000, f_iosize = 0x8000, f_blocks = 0x2e7a5, f_bfree = 0x2e7a2, f_bavail = 0x2ac25, f_files = 0x17ffe, f_ffree = 0x15793, f_syncwrites = 0x2e75a, f_asyncwrites = 0xc6fc, f_syncreads = 0x282, f_asyncreads = 0x0, f_spare = 0xcae980c4, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = 0xcae9811c}, f_charspare = 0xcae98124 "", f_fstypename = 0xcae98174 "ufs", f_mntfromname = 0xcae98184 "/dev/md8a", f_mntonname = 0xcae98584 "/mnt8"}, mnt_cred = 0xc97cb700, mnt_data = 0x0, mnt_time = 0x0, mnt_iosize_max = 0x20000, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 0xfba53a9b, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0xfbbb2, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_listmtx = {lock_object = {lo_name = 0xc1620ef7 "struct mount vlist mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc89b5690}, mtx_lock = 0x4}, mnt_activevnodelist = {tqh_first = 0x0, tqh_last = 0xcae989c8}, mnt_activevnodelistsize = 0x0, mnt_tmpfreevnodelist = {tqh_first = 0x0, tqh_last = 0xcae989d4}, mnt_tmpfreevnodelistsize = 0x0, mnt_explock = {lock_object = {lo_name = 0xc15f4157 "explock", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xc89b3d60}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x0, pcs = 0xcae98a04}}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0x0}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xcae98a54}, mnt_refs = 0xcae98a5c, mnt_unrefs = 0xcae98d5c} (kgdb) (kgdb) p mp->mnt_refs[0] $4 = {refs = 0x16e98, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x220} (kgdb) p mp->mnt_refs[1] $5 = {refs = 0x6b9f7, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x63f} (kgdb) p mp->mnt_refs[2] $6 = {refs = 0x1d3b30, file = 0xc162040b "../../../kern/vfs_default.c", line = 0x263} (kgdb) p mp->mnt_refs[3] $7 = {refs = 0x133e, file = 0xc1623120 "../../../kern/vfs_syscalls.c", line = 0x169} (kgdb) p mp->mnt_refs[4] $8 = {refs = 0x1, file = 0xc1623120 "../../../kern/vfs_syscalls.c", line = 0x138} (kgdb) p mp->mnt_refs[5] $9 = {refs = 0x733f5, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x6c4} (kgdb) p mp->mnt_refs[6] $10 = {refs = 0x3cb4c, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x690} (kgdb) p mp->mnt_refs[7] $11 = {refs = 0x2fae, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x1582} (kgdb) p mp->mnt_refs[8] $13 = {refs = 0x6f2, file = 0xc1657e63 "../../../ufs/ffs/ffs_softdep.c", line = 0x3495} (kgdb) p mp->mnt_refs[9] $14 = {refs = 0x2c, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x14b9} (kgdb) p mp->mnt_refs[10] $15 = {refs = 0x1, file = 0xc1620abf "../../../kern/vfs_mount.c", line = 0x4e0} (kgdb) p mp->mnt_refs[11] $16 = {refs = 0x1, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x748} (kgdb) p mp->mnt_refs[12] $17 = {refs = 0x0, file = 0x0, line = 0x0} (kgdb) (kgdb) p mp->mnt_unrefs[0] $19 = {refs = 0xfbbb2, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x6fd} (kgdb) p mp->mnt_unrefs[1] $20 = {refs = 0x16e91, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x24d} (kgdb) p mp->mnt_unrefs[2] $21 = {refs = 0x15f186, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x6e7} (kgdb) p mp->mnt_unrefs[3] $22 = {refs = 0x133f, file = 0xc1623120 "../../../kern/vfs_syscalls.c", line = 0xf7} (kgdb) p mp->mnt_unrefs[4] $23 = {refs = 0x6b9f7, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x60e} (kgdb) p mp->mnt_unrefs[5] $24 = {refs = 0x2fae, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x14ef} (kgdb) p mp->mnt_unrefs[6] $25 = {refs = 0x6e5, file = 0xc1657e63 "../../../ufs/ffs/ffs_softdep.c", line = 0x34a7} (kgdb) p mp->mnt_unrefs[7] $26 = {refs = 0x2b, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x14de} (kgdb) p mp->mnt_unrefs[8] $27 = {refs = 0xc, file = 0xc1657e63 "../../../ufs/ffs/ffs_softdep.c", line = 0x3494} (kgdb) p mp->mnt_unrefs[9] $28 = {refs = 0x28d3b, file = 0xc1623396 "../../../kern/vfs_vnops.c", line = 0x663} (kgdb) p mp->mnt_unrefs[10] $29 = {refs = 0x7, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x22f} (kgdb) p mp->mnt_unrefs[11] $30 = {refs = 0x1, file = 0xc1621658 "../../../kern/vfs_subr.c", line = 0x14c3} (kgdb) p mp->mnt_unrefs[12] $31 = {refs = 0x0, file = 0x0, line = 0x0} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/vfs_mount.c =================================================================== --- /usr/src/sys/kern/vfs_mount.c (revision 319211) +++ /usr/src/sys/kern/vfs_mount.c (working copy) @@ -430,25 +430,107 @@ sys_nmount(struct thread *td, struct nmount_args * */ void -vfs_ref(struct mount *mp) +vfs_ref1(struct mount *mp, const char *file, int line) { CTR2(KTR_VFS, "%s: mp %p", __func__, mp); MNT_ILOCK(mp); - MNT_REF(mp); + mp->mnt_ref++; + mnt_record_ref(mp, file, line); MNT_IUNLOCK(mp); } void -vfs_rel(struct mount *mp) +vfs_rel1(struct mount *mp, const char *file, int line) { CTR2(KTR_VFS, "%s: mp %p", __func__, mp); MNT_ILOCK(mp); - MNT_REL(mp); + KASSERT(mp->mnt_ref > 0, ("negative mnt_ref")); + mp->mnt_ref--; + mnt_record_unref(mp, file, line); + if (mp->mnt_ref == 0) + wakeup(mp); MNT_IUNLOCK(mp); } +static void +mnt_check_refs(struct mount *mp) +{ + struct mount_refs *mr; + int refs, i; + + refs = 0; + for (i = 0; i < nitems(mp->mnt_refs); i++) { + mr = &mp->mnt_refs[i]; + refs += mr->refs; + } + for (i = 0; i < nitems(mp->mnt_unrefs); i++) { + mr = &mp->mnt_unrefs[i]; + refs -= mr->refs; + } + if (refs != mp->mnt_ref) + panic("mp %p refs %d mnt_ref %d\n", mp, refs, mp->mnt_ref); +} + +void +mnt_record_ref(struct mount *mp, const char *file, int line) +{ + struct mount_refs *mr; + int i; + + mtx_assert(MNT_MTX(mp), MA_OWNED); + for (i = 0; i < nitems(mp->mnt_refs); i++) { + mr = &mp->mnt_refs[i]; + if (mr->file != NULL && strcmp(mr->file, file) == 0 && + mr->line == line) { + mr->refs++; + mnt_check_refs(mp); + return; + } + } + for (i = 0; i < nitems(mp->mnt_refs); i++) { + mr = &mp->mnt_refs[i]; + if (mr->file == NULL) { + mr->file = file; + mr->line = line; + mr->refs++; + mnt_check_refs(mp); + return; + } + } + panic("mount_refs overflow"); +} + +void +mnt_record_unref(struct mount *mp, const char *file, int line) +{ + struct mount_refs *mr; + int i; + + mtx_assert(MNT_MTX(mp), MA_OWNED); + for (i = 0; i < nitems(mp->mnt_unrefs); i++) { + mr = &mp->mnt_unrefs[i]; + if (mr->file != NULL && strcmp(mr->file, file) == 0 && + mr->line == line) { + mr->refs++; + mnt_check_refs(mp); + return; + } + } + for (i = 0; i < nitems(mp->mnt_unrefs); i++) { + mr = &mp->mnt_unrefs[i]; + if (mr->file == NULL) { + mr->file = file; + mr->line = line; + mr->refs++; + mnt_check_refs(mp); + return; + } + } + panic("mount_unrefs overflow"); +} + /* * Allocate and initialize the mount point struct. */ @@ -461,6 +543,8 @@ vfs_mount_alloc(struct vnode *vp, struct vfsconf * mp = uma_zalloc(mount_zone, M_WAITOK); bzero(&mp->mnt_startzero, __rangeof(struct mount, mnt_startzero, mnt_endzero)); +bzero(&mp->mnt_refs, sizeof(mp->mnt_refs)); +bzero(&mp->mnt_unrefs, sizeof(mp->mnt_unrefs)); TAILQ_INIT(&mp->mnt_nvnodelist); mp->mnt_nvnodelistsize = 0; TAILQ_INIT(&mp->mnt_activevnodelist); Index: /usr/src/sys/sys/mount.h =================================================================== --- /usr/src/sys/sys/mount.h (revision 319211) +++ /usr/src/sys/sys/mount.h (working copy) @@ -183,6 +183,11 @@ struct vfsopt { * Unmarked fields are considered stable as long as a ref is held. * */ +struct mount_refs { + int refs; + const char *file; + int line; +}; struct mount { struct mtx mnt_mtx; /* mount structure interlock */ int mnt_gen; /* struct mount generation */ @@ -223,6 +228,8 @@ struct mount { struct lock mnt_explock; /* vfs_export walkers lock */ TAILQ_ENTRY(mount) mnt_upper_link; /* (m) we in the all uppers */ TAILQ_HEAD(, mount) mnt_uppers; /* (m) upper mounts over us*/ +struct mount_refs mnt_refs[64]; +struct mount_refs mnt_unrefs[64]; }; /* @@ -258,14 +265,17 @@ void __mnt_vnode_markerfree_active(struct #define MNT_VNODE_FOREACH_ACTIVE_ABORT(mp, mvp) \ __mnt_vnode_markerfree_active(&(mvp), (mp)) +void mnt_record_ref(struct mount *mp, const char *file, int line); +void mnt_record_unref(struct mount *mp, const char *file, int line); #define MNT_ILOCK(mp) mtx_lock(&(mp)->mnt_mtx) #define MNT_ITRYLOCK(mp) mtx_trylock(&(mp)->mnt_mtx) #define MNT_IUNLOCK(mp) mtx_unlock(&(mp)->mnt_mtx) #define MNT_MTX(mp) (&(mp)->mnt_mtx) -#define MNT_REF(mp) (mp)->mnt_ref++ +#define MNT_REF(mp) do { (mp)->mnt_ref++; mnt_record_ref((mp), __FILE__, __LINE__);} while (0) #define MNT_REL(mp) do { \ KASSERT((mp)->mnt_ref > 0, ("negative mnt_ref")); \ (mp)->mnt_ref--; \ + mnt_record_unref((mp), __FILE__, __LINE__); \ if ((mp)->mnt_ref == 0) \ wakeup((mp)); \ } while (0) @@ -920,8 +930,10 @@ void vfs_mountedfrom(struct mount *, const char *f void vfs_notify_upper(struct vnode *, int); void vfs_oexport_conv(const struct oexport_args *oexp, struct export_args *exp); -void vfs_ref(struct mount *); -void vfs_rel(struct mount *); +void vfs_ref1(struct mount *, const char *, int); +#define vfs_ref(mp) vfs_ref1((mp), __FILE__, __LINE__) +void vfs_rel1(struct mount *, const char *, int); +#define vfs_rel(mp) vfs_rel1((mp), __FILE__, __LINE__) struct mount *vfs_mount_alloc(struct vnode *, struct vfsconf *, const char *, struct ucred *); int vfs_suser(struct mount *, struct thread *); Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 319211) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -13253,6 +13253,35 @@ softdep_slowdown(vp) return (1); } +static void +softdep_request_cleanup_flush(mp, ump) + struct mount *mp; + struct ufsmount *ump; +{ + struct vnode *lvp, *mvp; + + MNT_VNODE_FOREACH_ALL(lvp, mp, mvp) { + if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0) { + VI_UNLOCK(lvp); + continue; + } + if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, + curthread)) + continue; + if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ + vput(lvp); + continue; + } + (void) ffs_syncvnode(lvp, MNT_NOWAIT, 0); + vput(lvp); + } + lvp = ump->um_devvp; + if (vn_lock(lvp, LK_EXCLUSIVE | LK_NOWAIT) == 0) { + VOP_FSYNC(lvp, MNT_NOWAIT, curthread); + VOP_UNLOCK(lvp, 0); + } +} + /* * Called by the allocation routines when they are about to fail * in the hope that we can free up the requested resource (inodes @@ -13274,7 +13303,6 @@ softdep_request_cleanup(fs, vp, cred, resource) { struct ufsmount *ump; struct mount *mp; - struct vnode *lvp, *mvp; long starttime; ufs2_daddr_t needed; int error; @@ -13369,40 +13397,33 @@ retry: * resources. We walk the vnodes associated with the mount point * until we get the needed worklist requests that we can reap. */ + error = 1; if ((resource == FLUSH_BLOCKS_WAIT && fs->fs_cstotal.cs_nbfree <= needed) || (resource == FLUSH_INODES_WAIT && fs->fs_pendinginodes > 0 && fs->fs_cstotal.cs_nifree <= needed)) { - MNT_VNODE_FOREACH_ALL(lvp, mp, mvp) { - if (TAILQ_FIRST(&lvp->v_bufobj.bo_dirty.bv_hd) == 0) { - VI_UNLOCK(lvp); - continue; + ACQUIRE_LOCK(ump); + if ((ump->um_softdep->sd_flags & FLUSH_RC_ACTIVE) == 0) { + ump->um_softdep->sd_flags |= FLUSH_RC_ACTIVE; + FREE_LOCK(ump); + softdep_request_cleanup_flush(mp, ump); + ACQUIRE_LOCK(ump); + ump->um_softdep->sd_flags &= ~FLUSH_RC_ACTIVE; + FREE_LOCK(ump); + if (ump->softdep_on_worklist > 0) { + stat_cleanup_retries += 1; + goto retry; } - if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, - curthread)) - continue; - if (lvp->v_vflag & VV_NOSYNC) { /* unlinked */ - vput(lvp); - continue; - } - (void) ffs_syncvnode(lvp, MNT_NOWAIT, 0); - vput(lvp); + } else { + FREE_LOCK(ump); + error = 0; } - lvp = ump->um_devvp; - if (vn_lock(lvp, LK_EXCLUSIVE | LK_NOWAIT) == 0) { - VOP_FSYNC(lvp, MNT_NOWAIT, curthread); - VOP_UNLOCK(lvp, 0); - } - if (ump->softdep_on_worklist > 0) { - stat_cleanup_retries += 1; - goto retry; - } stat_cleanup_failures += 1; } if (time_second - starttime > stat_cleanup_high_delay) stat_cleanup_high_delay = time_second - starttime; UFS_LOCK(ump); - return (1); + return (error); } static bool Index: /usr/src/sys/ufs/ffs/softdep.h =================================================================== --- /usr/src/sys/ufs/ffs/softdep.h (revision 319211) +++ /usr/src/sys/ufs/ffs/softdep.h (working copy) @@ -1065,6 +1065,7 @@ struct mount_softdeps { #define FLUSH_EXIT 0x0001 /* time to exit */ #define FLUSH_CLEANUP 0x0002 /* need to clear out softdep structures */ #define FLUSH_STARTING 0x0004 /* flush thread not yet started */ +#define FLUSH_RC_ACTIVE 0x0008 /* * Keep the old names from when these were in the ufsmount structure. -- Test scenario: misc/crossmp11.sh