GENERIC HEAD from 2012-02-12 05:14:12 UTC, r231557, vmcore.283 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r231557M: Sun Feb 12 13:38:01 CET 2012 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531640832 (3368 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 117169 free (1273 frags, 14487 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 143317 free (4813 frags, 17313 blocks, 0.5% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 114631673 free (2001 frags, 14328709 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 941474 free (366658 frags, 71852 blocks, 3.6% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 12289588 free (53564 frags, 1529503 blocks, 0.2% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=209b ether 00:21:85:15:33:7d nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xe191eb80 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xca79a400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f90ff3,2e687361,38323a63,a34,f27c47e8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a957db,c0f94a80,c12729f8,11c,f27c4878,...) at kdb_backtrace+0x2a _witness_debugger(c0f94a80,ca79a400,c0fc4b3a,c7d79678,c0fc47cc,...) at _witness_debugger+0x25 witness_checkorder(ca79a400,9,c0fc47c3,11c,0,...) at witness_checkorder+0x86f _sx_xlock(ca79a400,0,c0fc47c3,11c,ca81dae0,...) at _sx_xlock+0xa0 ufsdirhash_acquire(e191eb20,ca81dae0,f27c49f4,e2406474,f27c4948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(ca81dae0,f27c49f4,2474,f27c4934,f27c4938,...) at ufsdirhash_add+0x13 ufs_direnter(caaeec3c,cac26570,f27c49f4,f27c4b84,e191f1a0,...) at ufs_direnter+0x7a9 ufs_mkdir(f27c4c14,c0fe5da5,0,0,f27c4bac,...) at ufs_mkdir+0xa6f VOP_MKDIR_APV(c10dbd20,f27c4c14,f27c4b84,f27c4bac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(caadab80,ffffff9c,28404020,0,1c0,...) at kern_mkdirat+0x291 kern_mkdir(caadab80,28404020,0,1c0,f27c4d1c,...) at kern_mkdir+0x2e sys_mkdir(caadab80,f27c4cec,c0fdec2a,c0f9575e,202,...) at sys_mkdir+0x29 syscall(f27c4d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x2817f743, esp = 0xbfbfe8cc, ebp = 0xbfbfed78 --- Clearing /tmp (X related). Starting mountd. Feb 12 13:51:52 x4 mountd[1836]: can't delete exports for /dev: Invalid argument mount option is unknown NFSv4 is disabled Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xcaddddf0 ufs (ufs) @ kern/vfs_lookup.c:542 2nd 0xe1926a40 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 3rd 0xcadddb38 ufs (ufs) @ kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f90ff3,3a632e73,a323632,2807f000,2807f0dd,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a957db,c0f94a99,c1272770,877,f2765334,...) at kdb_backtrace+0x2a _witness_debugger(c0f94a99,cadddb38,c0f835ec,c7d79610,c0f9ca35,...) at _witness_debugger+0x25 witness_checkorder(cadddb38,9,c0f9ca2c,877,0,...) at witness_checkorder+0x86f __lockmgr_args(cadddb38,80100,cadddba4,0,0,...) at __lockmgr_args+0x834 ffs_lock(f2765458,60,0,80100,cadddae0,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c10dbd20,f2765458,c7d721e0,c10ecd40,cadddae0,...) at VOP_LOCK1_APV+0xb5 _vn_lock(cadddae0,80100,c0f9ca2c,877,4,...) at _vn_lock+0x78 vget(cadddae0,80100,cab355c0,50,0,...) at vget+0xbb vfs_hash_get(ca76bb50,15edde,80000,cab355c0,f276559c,...) at vfs_hash_get+0xed ffs_vgetf(ca76bb50,15edde,80000,f276559c,1,...) at ffs_vgetf+0x49 softdep_sync_buf(cadddd98,e19269e0,1,108,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(cadddd98,1,c1272760,caddde90,c7d75370,...) at ffs_syncvnode+0x24c ffs_truncate(cadddd98,200,0,880,ca85ec00,...) at ffs_truncate+0x827 ufs_direnter(cadddd98,cadddae0,f2765918,f2765bc0,0,...) at ufs_direnter+0x9a1 ufs_makeinode(f2765bc0,c10dc2a0,f2765b04,f2765a60,c0de4115,...) at ufs_makeinode+0x65d ufs_create(f2765b04,c0fe5f6b,0,0,f2765b80,...) at ufs_create+0x30 VOP_CREATE_APV(c10dbd20,f2765b04,f2765bc0,f2765a9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(f2765b80,f2765c48,1b0,0,ca85ec00,...) at vn_open_cred+0x205 vn_open(f2765b80,f2765c48,1b0,ca835230,4f37b594,...) at vn_open+0x3b kern_openat(cab355c0,ffffff9c,28839f60,0,a03,...) at kern_openat+0x1ec kern_open(cab355c0,28839f60,0,a02,1b0,...) at kern_open+0x35 sys_open(cab355c0,f2765cec,c0fdec2a,c0f9575e,206,...) at sys_open+0x30 syscall(f2765d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283eada3, esp = 0xbfbfc50c, ebp = 0xbfbfc598 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xca8309dc ufs (ufs) @ kern/vfs_mount.c:1249 2nd 0xca4751b4 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1835 KDB: stack backtrace: db_trace_self_wrapper(c0f90ff3,38313a63,a3533,c0aa7dcc,8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a957db,c0f94a80,c1272340,72b,f285f9a0,...) at kdb_backtrace+0x2a _witness_debugger(c0f94a80,ca4751b4,c0f7b527,c7d79470,c0fc0c39,...) at _witness_debugger+0x25 witness_checkorder(ca4751b4,9,c0fc0c30,72b,ca475220,...) at witness_checkorder+0x86f __lockmgr_args(ca4751b4,80400,ca475220,0,0,...) at __lockmgr_args+0x834 vop_stdlock(f285fac0,4,c0f8b3cb,80400,ca47515c,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10b19a0,f285fac0,c12a5f60,c10ecd40,ca47515c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(ca47515c,80400,c0fc0c30,72b,caaf42d4,...) at _vn_lock+0x78 softdep_flushworklist(caaf42d4,f285fb6c,cab5d2e0,5f2,0,...) at softdep_flushworklist+0x47 ffs_sync(caaf42d4,1,f285fbc0,51a,c7c30000,...) at ffs_sync+0x32d dounmount(caaf42d4,8080000,cab5d2e0,49c,0,...) at dounmount+0x3f8 sys_unmount(cab5d2e0,f285fcec,c,c,206,...) at sys_unmount+0x421 syscall(f285fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280e7e0b, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- lock order reversal: 1st 0xc12a5f60 Softdep Lock (Softdep Lock) @ ufs/ffs/ffs_softdep.c:10486 2nd 0xca475254 bufobj interlock (bufobj interlock) @ kern/vfs_subr.c:2011 KDB: stack backtrace: db_trace_self_wrapper(c0f90ff3,656b2040,762f6e72,735f7366,2e726275,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a957db,c0f94a80,c1273e68,7db,c7b48a68,...) at kdb_backtrace+0x2a _witness_debugger(c0f94a80,ca475254,c0f9d972,c7d77c10,c0f9ca35,...) at _witness_debugger+0x25 witness_checkorder(ca475254,9,c0f9ca2c,7db,0,...) at witness_checkorder+0x86f _mtx_lock_flags(ca475254,0,c0f9ca2c,7db,ca475254,...) at _mtx_lock_flags+0xcc reassignbuf(e191f000,cb0a3fa8,cb0a3900,caafc000,e191f000,...) at reassignbuf+0x53 bdirty(e191f000,c10d92f8,100,28f6,c0afa03a,...) at bdirty+0x68 softdep_disk_write_complete(e191f000,c0aa599a,c1273e68,c7edb0b0,c1273e68,...) at softdep_disk_write_complete+0x4e3 bufdone_finish(e191f000,caaf6080,caaf6080,e191f000,c7b48c80,...) at bufdone_finish+0x26 bufdone(e191f000,0,c0f824b7,74,caaf7340) at bufdone+0x6a g_vfs_done(ca6bb1c8,0,c0f99f8a,c58,c7edb000,...) at g_vfs_done+0xf8 biodone(ca6bb1c8,c1126468,25c,c0f7f9f2,0,...) at biodone+0xa5 g_io_schedule_up(c7edb000,0,c0f81541,5f,c7b48d14,...) at g_io_schedule_up+0xde g_up_procbody(0,c7b48d28,c0f876b0,3db,c7dc72c4,...) at g_up_procbody+0x9d fork_exit(c09ef110,0,c7b48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b48d60, ebp = 0 --- panic: _mtx_lock_sleep: recursed on non-recursive mutex bufobj interlock @ ../../../kern/vfs_bio.c:2644 cpuid = 3 KDB: enter: panic [ thread pid 2128 tid 100149 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 2128 tid 100149 td 0xcab5d2e0 kdb_enter(c0f8cf98,c0f8cf98,c0f8b611,f285f838,3,...) at kdb_enter+0x3a panic(c0f8b611,c0f9d972,c0f99f8a,a54,ca475254,...) at panic+0x1ac _mtx_lock_sleep(ca475254,cab5d2e0,0,c0f99f8a,a54,...) at _mtx_lock_sleep+0x58 _mtx_lock_flags(ca475254,0,c0f99f8a,a54,c0f8b3cb,...) at _mtx_lock_flags+0xff getblk(ca47515c,e0,0,4000,0,...) at getblk+0xe3 breadn(ca47515c,e0,0,4000,0,...) at breadn+0x44 bread(ca47515c,e0,0,4000,0,...) at bread+0x4c ffs_update(caaeec3c,0,c0fc0c30,246b,1,...) at ffs_update+0x192 handle_workitem_remove(0,f285f9e4,2,6bb,f285f9e8,...) at handle_workitem_remove+0x478 process_worklist_item(c12a5f60,0,c0fc0c30,5ea,cab5d2e0,...) at process_worklist_item+0x2b4 softdep_process_worklist(caaf42d4,1,c0a4b7fc,ca475254,8,...) at softdep_process_worklist+0x88 softdep_flushworklist(caaf42d4,f285fa98,cab5d2e0,296,0,...) at softdep_flushworklist+0xac vop_stdfsync(f285faf8,f285fab8,0,c10b19a0,f285faf8,...) at vop_stdfsync+0x1bb devfs_fsync(f285faf8,c0fe5903,ca47515c,ca47515c,0,...) at devfs_fsync+0x7c VOP_FSYNC_APV(c10b19a0,f285faf8,c0fc0c30,72b,c10ecbe0,...) at VOP_FSYNC_APV+0xc5 softdep_flushworklist(caaf42d4,f285fb6c,cab5d2e0,5f2,0,...) at softdep_flushworklist+0x70 ffs_sync(caaf42d4,1,f285fbc0,51a,c7c30000,...) at ffs_sync+0x32d dounmount(caaf42d4,8080000,cab5d2e0,49c,0,...) at dounmount+0x3f8 sys_unmount(cab5d2e0,f285fcec,c,c,206,...) at sys_unmount+0x421 syscall(f285fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280e7e0b, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- db:0:bt> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x737a80 curthread = 0xc7dc98a0: pid 11 "idle: cpu0" curpcb = 0xc7b26d80 fpcurthread = none idlethread = 0xc7dc98a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x6a28a80 curthread = 0xc7dc95c0: pid 11 "idle: cpu1" curpcb = 0xc7b29d80 fpcurthread = none idlethread = 0xc7dc95c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x6a2ba80 curthread = 0xc7dc92e0: pid 11 "idle: cpu2" curpcb = 0xc7b2cd80 fpcurthread = none idlethread = 0xc7dc92e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x6a2ea80 curthread = 0xcab5d2e0: pid 2128 "umount" curpcb = 0xf285fd80 fpcurthread = none idlethread = 0xc7dc9000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 2128 (umount) thread 0xcab5d2e0 (100149) exclusive lockmgr ufs (ufs) r = 0 (0xcaaeec94) locked @ kern/vfs_subr.c:2167 exclusive sleep mutex bufobj interlock (bufobj interlock) r = 0 (0xca475254) locked @ kern/vfs_default.c:662 exclusive sleep mutex Giant (Giant) r = 0 (0xc1128190) locked @ kern/vfs_mount.c:1164 exclusive lockmgr devfs (devfs) r = 0 (0xca4751b4) locked @ ufs/ffs/ffs_softdep.c:1835 exclusive lockmgr ufs (ufs) r = 0 (0xca8309dc) locked @ kern/vfs_mount.c:1249 Process 2108 (sendmail) thread 0xca7e7000 (100080) exclusive lockmgr ufs (ufs) r = 0 (0xcb0a646c) locked @ kern/vfs_syscalls.c:3583 Process 1710 (syslogd) thread 0xca7e52e0 (100091) exclusive lockmgr ufs (ufs) r = 0 (0xcaccab38) locked @ kern/vfs_syscalls.c:3583 db:0:alllocks> show lockedvnods Locked vnodes 0xca830984: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xcaaf42d4 flags () lock type ufs: EXCL by thread 0xcab5d2e0 (pid 2128) #0 0xc0a4348e at __lockmgr_args+0xc1e #1 0xc0ca50c1 at ffs_lock+0xa1 #2 0xc0de3145 at VOP_LOCK1_APV+0xb5 #3 0xc0b09348 at _vn_lock+0x78 #4 0xc0af1b8c at dounmount+0x8c #5 0xc0af25e1 at sys_unmount+0x421 #6 0xc0dbfde7 at syscall+0x2c7 #7 0xc0da9111 at Xint0x80_syscall+0x21 ino 211968, on dev ada0s1a 0xca47515c: tag devfs, type VCHR usecount 1, writecount 0, refcount 23 mountedhere 0xca6d1500 flags () v_object 0xca8283f0 ref 0 pages 79 lock type devfs: EXCL by thread 0xcab5d2e0 (pid 2128) #0 0xc0a4348e at __lockmgr_args+0xc1e #1 0xc0ae7255 at vop_stdlock+0x65 #2 0xc0de3145 at VOP_LOCK1_APV+0xb5 #3 0xc0b09348 at _vn_lock+0x78 #4 0xc0c9a377 at softdep_flushworklist+0x47 #5 0xc0c9f3cd at ffs_sync+0x32d #6 0xc0af1ef8 at dounmount+0x3f8 #7 0xc0af25e1 at sys_unmount+0x421 #8 0xc0dbfde7 at syscall+0x2c7 #9 0xc0da9111 at Xint0x80_syscall+0x21 dev label/tmp 0xcaaeec3c: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags (VV_ROOT) v_object 0xcac21c60 ref 0 pages 6 lock type ufs: EXCL by thread 0xcab5d2e0 (pid 2128) #0 0xc0a4348e at __lockmgr_args+0xc1e #1 0xc0ca50c1 at ffs_lock+0xa1 #2 0xc0de3145 at VOP_LOCK1_APV+0xb5 #3 0xc0b09348 at _vn_lock+0x78 #4 0xc0afbdfb at vget+0xbb #5 0xc0aeb80d at vfs_hash_get+0xed #6 0xc0c9e3b9 at ffs_vgetf+0x49 #7 0xc0c8af60 at handle_workitem_remove+0x60 #8 0xc0c8a7c4 at process_worklist_item+0x2b4 #9 0xc0c9a218 at softdep_process_worklist+0x88 #10 0xc0c9a3dc at softdep_flushworklist+0xac #11 0xc0ae77cb at vop_stdfsync+0x1bb #12 0xc096ebec at devfs_fsync+0x7c #13 0xc0de1ed5 at VOP_FSYNC_APV+0xc5 #14 0xc0c9a3a0 at softdep_flushworklist+0x70 #15 0xc0c9f3cd at ffs_sync+0x32d #16 0xc0af1ef8 at dounmount+0x3f8 #17 0xc0af25e1 at sys_unmount+0x421 ino 2, on dev label/tmp 0xcaccaae0: tag ufs, type VREG usecount 1, writecount 1, refcount 4 mountedhere 0 flags () v_object 0xcaadb240 ref 0 pages 6 lock type ufs: EXCL by thread 0xca7e52e0 (pid 1710) #0 0xc0a4348e at __lockmgr_args+0xc1e #1 0xc0ca50c1 at ffs_lock+0xa1 #2 0xc0de3145 at VOP_LOCK1_APV+0xb5 #3 0xc0b09348 at _vn_lock+0x78 #4 0xc0b074e6 at sys_fsync+0x116 #5 0xc0dbfde7 at syscall+0x2c7 #6 0xc0da9111 at Xint0x80_syscall+0x21 ino 6052869, on dev ada0s1e 0xcb0a6414: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xcadd2e10 ref 0 pages 1 lock type ufs: EXCL by thread 0xca7e7000 (pid 2108) #0 0xc0a4348e at __lockmgr_args+0xc1e #1 0xc0ca50c1 at ffs_lock+0xa1 #2 0xc0de3145 at VOP_LOCK1_APV+0xb5 #3 0xc0b09348 at _vn_lock+0x78 #4 0xc0b074e6 at sys_fsync+0x116 #5 0xc0dbfde7 at syscall+0x2c7 #6 0xc0da9111 at Xint0x80_syscall+0x21 ino 1437266, on dev ada0s1e db:0:lockedvnods> show mount 0xca76ab50 /dev/ad4s1a on / (ufs) 0xca76b000 devfs on /dev (devfs) 0xcaaf45a8 /dev/ad4s1f on /home (ufs) 0xcaaf42d4 /dev/label/tmp on /tmp (ufs) 0xcaaf4000 /dev/ad4s1d on /usr (ufs) 0xca76bb50 /dev/ad4s1e on /var (ufs) 0xca76b87c procfs on /proc (procfs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 2128 2125 20 0 RL+ CPU 3 umount 2126 2123 20 0 S+ nanslp 0xc1129ae4 sleep 2125 2124 20 0 S+ wait 0xcab5b588 sh 2124 2116 20 0 S+ wait 0xcb07684c sh 2123 1 20 0 S+ wait 0xcb076b10 sh 2116 20 20 0 S+ wait 0xcaacfb10 sh 2112 1 2112 0 Ss nanslp 0xc1129ae4 cron 2108 2100 2100 0 D bo_wwait 0xcb0a6540 sendmail 2105 2104 2104 25 S select 0xca7696e4 sendmail 2104 1 2104 25 Ss pause 0xcaebe5e0 sendmail 2100 1 2100 0 Ss select 0xca7693a4 sendmail 2092 1 2092 0 Ss select 0xca7c8324 sshd 2032 1 2032 0 Ss select 0xca7c8624 moused 1954 1 1954 0 Ss select 0xca768ca4 ntpd 1853 1847 1847 0 S (threaded) nfsd 100138 S rpcsvc 0xcab69a90 nfsd: service 100137 S rpcsvc 0xcab69ad0 nfsd: service 100136 S rpcsvc 0xcab69b10 nfsd: service 100126 S rpcsvc 0xc8180410 nfsd: master 1847 1 1847 0 Ss select 0xca7c8ca4 nfsd 1837 1 1837 0 Ss select 0xca769864 mountd 1738 1 1738 0 Ss select 0xca7698e4 rpcbind 1710 1 1710 0 Ds bo_wwait 0xcaccac0c syslogd 1503 1 1503 0 Ss select 0xca690d24 devd 20 1 20 0 Ss+ wait 0xca7a2b10 sh 19 0 0 0 DL sdflush 0xc12a5fa0 [softdepflush] 18 0 0 0 DL vlruwt 0xca6c02c4 [vnlru] 17 0 0 0 DL syncer 0xc129fcb4 [syncer] 16 0 0 0 DL psleep 0xc129f948 [bufdaemon] 9 0 0 0 DL pgzero 0xc12a7e5c [pagezero] 8 0 0 0 DL psleep 0xc12a7a48 [vmdaemon] 7 0 0 0 DL psleep 0xc12a7a10 [pagedaemon] 6 0 0 0 DL ccb_scan 0xc10ee2d4 [xpt_thrd] 5 0 0 0 DL waiting_ 0xc12a17d8 [sctp_iterator] 4 0 0 0 DL ctl_work 0xc819c000 [ctl_thrd] 3 0 0 0 DL - 0xc7ffa03c [fdc0] 2 0 0 0 SL - 0xc808f000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc8063d34 [usbus5] 100057 D - 0xc8063d04 [usbus5] 100056 D - 0xc8063cd4 [usbus5] 100055 D - 0xc8063ca4 [usbus5] 100053 D - 0xc8057b5c [usbus4] 100052 D - 0xc8057b2c [usbus4] 100051 D - 0xc8057afc [usbus4] 100050 D - 0xc8057acc [usbus4] 100049 D - 0xc8052b5c [usbus3] 100048 D - 0xc8052b2c [usbus3] 100047 D - 0xc8052afc [usbus3] 100046 D - 0xc8052acc [usbus3] 100045 D - 0xc804ab5c [usbus2] 100044 D - 0xc804ab2c [usbus2] 100043 D - 0xc804aafc [usbus2] 100042 D - 0xc804aacc [usbus2] 100040 D - 0xc8044b5c [usbus1] 100039 D - 0xc8044b2c [usbus1] 100038 D - 0xc8044afc [usbus1] 100037 D - 0xc8044acc [usbus1] 100035 D - 0xc8039b5c [usbus0] 100034 D - 0xc8039b2c [usbus0] 100033 D - 0xc8039afc [usbus0] 100032 D - 0xc8039acc [usbus0] 14 0 0 0 DL - 0xc11288e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc11264e8 [g_down] 100014 D - 0xc11264e4 [g_up] 100013 D - 0xc11264dc [g_event] 12 0 0 0 WL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100027 I [swi6: task queue] 100026 I [swi6: Giant taskq] 100024 I [swi5: +] 100018 I [swi2: cambio] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc7dc7b10 [init] 10 0 0 0 DL audit_wo 0xc12a56e0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100078 D - 0xc8036380 [mca taskq] 100068 D - 0xc11288e4 [deadlkres] 100061 D - 0xc80366c0 [fw0_taskq] 100025 D - 0xc7f9ee80 [thread taskq] 100023 D - 0xc7f9f040 [acpi_task_2] 100022 D - 0xc7f9f040 [acpi_task_1] 100021 D - 0xc7f9f040 [acpi_task_0] 100020 D - 0xc7f9f080 [kqueue taskq] 100019 D - 0xc7f9f0c0 [ffs_trim taskq] 100016 D - 0xc7dade80 [firmware taskq] 100000 D sched 0xc11265c0 [swapper] db:0:ps> allt Tracing command umount pid 2128 tid 100149 td 0xcab5d2e0 kdb_enter(c0f8cf98,c0f8cf98,c0f8b611,f285f838,3,...) at kdb_enter+0x3a panic(c0f8b611,c0f9d972,c0f99f8a,a54,ca475254,...) at panic+0x1ac _mtx_lock_sleep(ca475254,cab5d2e0,0,c0f99f8a,a54,...) at _mtx_lock_sleep+0x58 _mtx_lock_flags(ca475254,0,c0f99f8a,a54,c0f8b3cb,...) at _mtx_lock_flags+0xff getblk(ca47515c,e0,0,4000,0,...) at getblk+0xe3 breadn(ca47515c,e0,0,4000,0,...) at breadn+0x44 bread(ca47515c,e0,0,4000,0,...) at bread+0x4c ffs_update(caaeec3c,0,c0fc0c30,246b,1,...) at ffs_update+0x192 handle_workitem_remove(0,f285f9e4,2,6bb,f285f9e8,...) at handle_workitem_remove+0x478 process_worklist_item(c12a5f60,0,c0fc0c30,5ea,cab5d2e0,...) at process_worklist_item+0x2b4 softdep_process_worklist(caaf42d4,1,c0a4b7fc,ca475254,8,...) at softdep_process_worklist+0x88 softdep_flushworklist(caaf42d4,f285fa98,cab5d2e0,296,0,...) at softdep_flushworklist+0xac vop_stdfsync(f285faf8,f285fab8,0,c10b19a0,f285faf8,...) at vop_stdfsync+0x1bb devfs_fsync(f285faf8,c0fe5903,ca47515c,ca47515c,0,...) at devfs_fsync+0x7c VOP_FSYNC_APV(c10b19a0,f285faf8,c0fc0c30,72b,c10ecbe0,...) at VOP_FSYNC_APV+0xc5 softdep_flushworklist(caaf42d4,f285fb6c,cab5d2e0,5f2,0,...) at softdep_flushworklist+0x70 ffs_sync(caaf42d4,1,f285fbc0,51a,c7c30000,...) at ffs_sync+0x32d dounmount(caaf42d4,8080000,cab5d2e0,49c,0,...) at dounmount+0x3f8 sys_unmount(cab5d2e0,f285fcec,c,c,206,...) at sys_unmount+0x421 syscall(f285fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280e7e0b, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- Tracing command sleep pid 2126 tid 100147 td 0xcab5d8a0 sched_switch(cab5d8a0,0,104,1a3,cb4ea533,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cab5d8a0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(30d41,c0a9c010,cab5d8a0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1129ae4,6c,c0f8e4f7,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1129ae4,0,16c,c0f8e4f7,30d41,...) at _sleep+0x34f kern_nanosleep(cab5d8a0,f2859c50,f2859c58,c8,0,...) at kern_nanosleep+0xc1 sys_nanosleep(cab5d8a0,f2859cec,c0fdec2a,c0f96138,246,...) at sys_nanosleep+0x6f syscall(f2859d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2816de27, esp = 0xbfbfedbc, ebp = 0xbfbfee08 --- Tracing command sh pid 2125 tid 100116 td 0xcab33b80 sched_switch(cab33b80,0,104,1a3,d0133cb3,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cab33b80,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cab5b588,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(cab5b588,cab5b610,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(cab33b80,ffffffff,f27b5c60,2,0,...) at kern_wait+0x626 sys_wait4(cab33b80,f27b5cec,c0fdec2a,c0f95896,216,...) at sys_wait4+0x3b syscall(f27b5d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2817645b, esp = 0xbfbfeb0c, ebp = 0xbfbfeb28 --- Tracing command sh pid 2124 tid 100159 td 0xc81852e0 sched_switch(c81852e0,0,104,1a3,c5c53bce,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c81852e0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cb07684c,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(cb07684c,cb0768d4,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(c81852e0,ffffffff,f2880c60,2,0,...) at kern_wait+0x626 sys_wait4(c81852e0,f2880cec,c0fdec2a,c0f95896,216,...) at sys_wait4+0x3b syscall(f2880d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2817645b, esp = 0xbfbfdd4c, ebp = 0xbfbfdd68 --- Tracing command sh pid 2123 tid 100158 td 0xc81855c0 sched_switch(c81855c0,0,104,1a3,cb306c17,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c81855c0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cb076b10,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(cb076b10,cb076b98,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(c81855c0,ffffffff,f287dc60,2,0,...) at kern_wait+0x626 sys_wait4(c81855c0,f287dcec,c,c,216,...) at sys_wait4+0x3b syscall(f287dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2817645b, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command sh pid 2116 tid 100131 td 0xcaad92e0 sched_switch(caad92e0,0,104,1a3,c5ba8c00,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(caad92e0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(caacfb10,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(caacfb10,caacfb98,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(caad92e0,ffffffff,f2802c60,2,0,...) at kern_wait+0x626 sys_wait4(caad92e0,f2802cec,c0fdec2a,c0f95896,216,...) at sys_wait4+0x3b syscall(f2802d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2817645b, esp = 0xbfbfdeac, ebp = 0xbfbfdec8 --- Tracing command cron pid 2112 tid 100144 td 0xcab612e0 sched_switch(cab612e0,0,104,1a3,c329137b,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cab612e0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(1389,c0a9c010,cab612e0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c1129ae4,6c,c0f8e4f7,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c1129ae4,0,16c,c0f8e4f7,1389,...) at _sleep+0x34f kern_nanosleep(cab612e0,f2850c50,f2850c58,5,0,...) at kern_nanosleep+0xc1 sys_nanosleep(cab612e0,f2850cec,c,c,206,...) at sys_nanosleep+0x6f syscall(f2850d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2818fe27, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sendmail pid 2108 tid 100080 td 0xca7e7000 sched_switch(ca7e7000,0,104,1a3,1e4a9d2e,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5d,...) at mi_switch+0x1ff sleepq_switch(ca7e7000,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(cb0a6540,5d,c0f99fd8,0,0,...) at sleepq_wait+0x63 _sleep(cb0a6540,cb0a650c,5d,c0f99fd8,0,...) at _sleep+0x39c bufobj_wwait(cb0a650c,0,0,12c,0,...) at bufobj_wwait+0x73 ffs_syncvnode(cb0a6414,1,c0afaf3f,c10dbd20,cb0a650c,...) at ffs_syncvnode+0x46b ffs_fsync(f2703c48,c0fe5903,f2703cec,f2703c48,0,...) at ffs_fsync+0x27 VOP_FSYNC_APV(c10dbd20,f2703c48,c0f9e047,e04,0,...) at VOP_FSYNC_APV+0xc5 sys_fsync(ca7e7000,f2703cec,c0fdec2a,c0f5b6a7,202,...) at sys_fsync+0x1df syscall(f2703d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (95, FreeBSD ELF32, sys_fsync), eip = 0x283f0207, esp = 0xbfbfad9c, ebp = 0xbfbfb6d8 --- Tracing command sendmail pid 2105 tid 100121 td 0xcaadab80 sched_switch(caadab80,0,104,1a3,c99e6d93,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(caadab80,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f27c4a20,c0a4b42a,ca7696d0,0,caadab80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(ca7696e4,0,f27c4a58,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(ca7696e4,ca7696d0,36ee81,6a4,cac27070,...) at _cv_timedwait_sig+0x270 seltdwait(f27c4c10,f27c4c18,ca76c600,caadab80,f27c4b00,...) at seltdwait+0x8a kern_select(caadab80,8,bfbf9c44,0,bfbf9bc4,f27c4c5c,20,e10,0) at kern_select+0x655 sys_select(caadab80,f27c4cec,c0fdec2a,c0f959f4,202,...) at sys_select+0x66 syscall(f27c4d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283f32b3, esp = 0xbfbf9b8c, ebp = 0xbfbf9ce8 --- Tracing command sendmail pid 2104 tid 100141 td 0xcab61b80 sched_switch(cab61b80,0,104,1a3,becf52c1,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(cab61b80,0,c0f929a0,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(caebe5e0,74,c0e141ce,100,0,...) at sleepq_wait_sig+0x17 _sleep(caebe5e0,caebe610,174,c0e141ce,0,...) at _sleep+0x385 kern_sigsuspend(cab61b80,0,0,0,0,...) at kern_sigsuspend+0xba sys_sigsuspend(cab61b80,f2847cec,c0fdec2a,c0f957a6,202,...) at sys_sigsuspend+0x4d syscall(f2847d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x28348ccb, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 --- Tracing command sendmail pid 2100 tid 100115 td 0xcab34000 sched_switch(cab34000,0,104,1a3,bf477905,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cab34000,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f27b2a20,c0a4b42a,ca769390,0,cab34000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(ca7693a4,0,f27b2a58,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(ca7693a4,ca769390,1389,6a4,cac278f8,...) at _cv_timedwait_sig+0x270 seltdwait(f27b2c10,f27b2c18,ca7d8600,cab34000,c7dcb570,...) at seltdwait+0x8a kern_select(cab34000,5,bfbfc510,0,0,f27b2c5c,20,5,0) at kern_select+0x655 sys_select(cab34000,f27b2cec,c,c,206,...) at sys_select+0x66 syscall(f27b2d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283f32b3, esp = 0xbfbfc47c, ebp = 0xbfbfcfa8 --- Tracing command sshd pid 2092 tid 100090 td 0xca7e55c0 sched_switch(ca7e55c0,0,104,1a3,a1867e09,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca7e55c0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca7c8310,0,c0f8b3cb,ca7e55c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca7c8324,0,f272ba58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca7c8324,ca7c8310,c0f950f0,6a4,cac27690,...) at _cv_wait_sig+0x260 seltdwait(cac27690,58,ca85ec00,ca7e55c0,cacae488,...) at seltdwait+0xa2 kern_select(ca7e55c0,5,2880c0b8,0,0,0,20,ca7dd2c4,a) at kern_select+0x655 sys_select(ca7e55c0,f272bcec,c,c,206,...) at sys_select+0x66 syscall(f272bd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d72b3, esp = 0xbfbfdeec, ebp = 0xbfbfedf8 --- Tracing command moused pid 2032 tid 100100 td 0xcab355c0 sched_switch(cab355c0,0,104,1a3,6fe34640,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cab355c0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca7c8610,0,c0f8b3cb,cab355c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca7c8624,0,f2765a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca7c8624,ca7c8610,c0f950f0,6a4,ca7f9230,...) at _cv_wait_sig+0x260 seltdwait(ca7f9230,58,c7dc4e80,cab355c0,f6,...) at seltdwait+0xa2 kern_select(cab355c0,400,bfbfea04,0,0,0,20,cab2d000,1) at kern_select+0x655 sys_select(cab355c0,f2765cec,c0fdec2a,c0df2696,246,...) at sys_select+0x66 syscall(f2765d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281c52b3, esp = 0xbfbfe99c, ebp = 0xbfbfeb28 --- Tracing command ntpd pid 1954 tid 100151 td 0xcab35b80 sched_switch(cab35b80,0,104,1a3,be52668f,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cab35b80,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca768c90,0,c0f8b3cb,cab35b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca768ca4,0,f2865a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca768ca4,ca768c90,c0f950f0,6a4,cac278c0,...) at _cv_wait_sig+0x260 seltdwait(cac278c0,58,c7dc4e80,cab35b80,c0aa599a,...) at seltdwait+0xa2 kern_select(cab35b80,17,bfbfece8,0,0,0,20,cab5c588,0) at kern_select+0x655 sys_select(cab35b80,f2865cec,c0fdec2a,c0f7da99,206,...) at sys_select+0x66 syscall(f2865d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283722b3, esp = 0xbfbfecbc, ebp = 0xbfbfed88 --- Tracing command nfsd pid 1853 tid 100138 td 0xca6cc8a0 sched_switch(ca6cc8a0,0,104,1a3,70a95cee,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca6cc8a0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f2826be0,c0a4b42a,c800f900,0,ca6cc8a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cab69a90,0,f2826c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cab69a90,c800f900,1388,3ec,2,...) at _cv_timedwait_sig+0x270 svc_run_internal(f2826d14,c0a2df68,c800f900,f2826d28,c0f876b0,...) at svc_run_internal+0x356 svc_thread_start(c800f900,f2826d28,c0f876b0,3db,caad2b10,...) at svc_thread_start+0x10 fork_exit(c0c4b090,c800f900,f2826d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1853 tid 100137 td 0xca6ccb80 sched_switch(ca6ccb80,0,104,1a3,70a8fc0d,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca6ccb80,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f2823be0,c0a4b42a,c800f900,0,ca6ccb80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cab69ad0,0,f2823c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cab69ad0,c800f900,1388,3ec,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(f2823d14,c0a2df68,c800f900,f2823d28,c0f876b0,...) at svc_run_internal+0x356 svc_thread_start(c800f900,f2823d28,c0f876b0,3db,caad2b10,...) at svc_thread_start+0x10 fork_exit(c0c4b090,c800f900,f2823d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1853 tid 100136 td 0xca6cd000 sched_switch(ca6cd000,0,104,1a3,70a8c3a0,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca6cd000,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f2820be0,c0a4b42a,c800f900,0,ca6cd000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cab69b10,0,f2820c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cab69b10,c800f900,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(f2820d14,c0a2df68,c800f900,f2820d28,c0f876b0,...) at svc_run_internal+0x356 svc_thread_start(c800f900,f2820d28,c0f876b0,3db,caad2b10,...) at svc_thread_start+0x10 fork_exit(c0c4b090,c800f900,f2820d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1853 tid 100126 td 0xcaada2e0 sched_switch(caada2e0,0,104,1a3,70a95194,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(caada2e0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f27ee498,c0a4b42a,c800f900,0,caada2e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c8180410,0,f27ee4d0,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c8180410,c800f900,1388,3ec,f27ee518,...) at _cv_timedwait_sig+0x270 svc_run_internal(caada490,14,c0fbc15a,c0f7ef7b,f27ee810,...) at svc_run_internal+0x356 svc_run(c800f900,0,c0f7ef56,19f,0,...) at svc_run+0x7b nfsrvd_nfsd(caada2e0,f27ee810,c,c1272bd8,0,...) at nfsrvd_nfsd+0x1c3 nfssvc_nfsd(caada2e0,f27eecec,bfbfe870,caada2e0,f27eed1c,...) at nfssvc_nfsd+0x177 sys_nfssvc(caada2e0,f27eecec,c0fdec2a,c0f95cc3,206,...) at sys_nfssvc+0xeb syscall(f27eed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280e890b, esp = 0xbfbfe82c, ebp = 0xbfbfea98 --- Tracing command nfsd pid 1847 tid 100107 td 0xca7a92e0 sched_switch(ca7a92e0,0,104,1a3,7493b436,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca7a92e0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca7c8c90,0,c0f8b3cb,ca7a92e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca7c8ca4,0,f2791a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca7c8ca4,ca7c8c90,c0f950f0,6a4,cac377e0,...) at _cv_wait_sig+0x260 seltdwait(cac377e0,58,c7dc4e80,ca7a92e0,cacae828,...) at seltdwait+0xa2 kern_select(ca7a92e0,7,bfbfecb4,0,0,0,20,ca7a22c4,1) at kern_select+0x655 sys_select(ca7a92e0,f2791cec,c,c,202,...) at sys_select+0x66 syscall(f2791d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281922b3, esp = 0xbfbfea9c, ebp = 0xbfbfede8 --- Tracing command mountd pid 1837 tid 100120 td 0xcab33000 sched_switch(cab33000,0,104,1a3,6cafdc62,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cab33000,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca769850,0,c0f8b3cb,cab33000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca769864,0,f27c1a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca769864,ca769850,c0f950f0,6a4,cac37a10,...) at _cv_wait_sig+0x260 seltdwait(cac37a10,58,c7dc4e80,cab33000,c0aa599a,...) at seltdwait+0xa2 kern_select(cab33000,a,bfbfed04,0,0,0,20,cab5984c,bfbfed04) at kern_select+0x655 sys_select(cab33000,f27c1cec,c0fdec2a,f27c1d80,206,...) at sys_select+0x66 syscall(f27c1d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281aa2b3, esp = 0xbfbfec9c, ebp = 0xbfbfede8 --- Tracing command rpcbind pid 1738 tid 100095 td 0xca7e08a0 sched_switch(ca7e08a0,0,104,1a3,749078d8,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca7e08a0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f2756a84,c0a4b42a,ca7698d0,0,ca7e08a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(ca7698e4,0,f2756abc,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(ca7698e4,ca7698d0,7531,6a4,f2756b74,...) at _cv_timedwait_sig+0x270 seltdwait(f2756c44,f2756c4c,5a1,ca7e08a0,c7d75ac0,...) at seltdwait+0x8a sys_poll(ca7e08a0,f2756cec,c0fdec2a,c0f9585b,212,...) at sys_poll+0x321 syscall(f2756d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2814a14b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1710 tid 100091 td 0xca7e52e0 sched_switch(ca7e52e0,0,104,1a3,1e4b248d,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5d,...) at mi_switch+0x1ff sleepq_switch(ca7e52e0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(caccac0c,5d,c0f99fd8,0,0,...) at sleepq_wait+0x63 _sleep(caccac0c,caccabd8,5d,c0f99fd8,0,...) at _sleep+0x39c bufobj_wwait(caccabd8,0,0,12c,0,...) at bufobj_wwait+0x73 ffs_syncvnode(caccaae0,1,c0afaf3f,c10dbd20,caccabd8,...) at ffs_syncvnode+0x46b ffs_fsync(f2730c48,c0fe5903,f2730cec,f2730c48,0,...) at ffs_fsync+0x27 VOP_FSYNC_APV(c10dbd20,f2730c48,c0f9e047,e04,0,...) at VOP_FSYNC_APV+0xc5 sys_fsync(ca7e52e0,f2730cec,c0fdec2a,c0f96800,202,...) at sys_fsync+0x1df syscall(f2730d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (95, FreeBSD ELF32, sys_fsync), eip = 0x281a6207, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1503 tid 100096 td 0xca7e05c0 sched_switch(ca7e05c0,0,104,1a3,aaeb3b34,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca7e05c0,0,c0f929a0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b42a,ca690d10,0,c0f8b3cb,ca7e05c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca690d24,0,f2759a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(ca690d24,ca690d10,c0f950f0,6a4,ca835ce8,...) at _cv_wait_sig+0x260 seltdwait(ca835ce8,58,c7dc4e80,ca7e05c0,ca79e000,...) at seltdwait+0xa2 kern_select(ca7e05c0,5,bfbfe960,0,0,0,20,cab2db10,29) at kern_select+0x655 sys_select(ca7e05c0,f2759cec,c,c,206,...) at sys_select+0x66 syscall(f2759d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x809152b, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command sh pid 20 tid 100079 td 0xca7a9b80 sched_switch(ca7a9b80,0,104,1a3,c3819c4e,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(ca7a9b80,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(ca7a2b10,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(ca7a2b10,ca7a2b98,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(ca7a9b80,ffffffff,f26ffc60,2,0,...) at kern_wait+0x626 sys_wait4(ca7a9b80,f26ffcec,c0fdec2a,c0f95896,216,...) at sys_wait4+0x3b syscall(f26ffd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2817645b, esp = 0xbfbfeacc, ebp = 0xbfbfeae8 --- Tracing command softdepflush pid 19 tid 100077 td 0xca6cdb80 sched_switch(ca6cdb80,0,104,1a3,1e4b58dd,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(ca6cdb80,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a5fa0,54,c0fc3170,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a5fa0,c12a5f60,54,c0fc3170,3e8,...) at _sleep+0x36a softdep_flush(0,ee320d28,c0f876b0,3db,ca6c0000,...) at softdep_flush+0x266 fork_exit(c0c9a550,0,ee320d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee320d60, ebp = 0 --- Tracing command vnlru pid 18 tid 100076 td 0xc805e8a0 sched_switch(c805e8a0,0,104,1a3,7088c72,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c805e8a0,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(ca6c02c4,60,c0f9d99d,0,0,...) at sleepq_timedwait+0x6b _sleep(ca6c02c4,c129fc74,260,c0f9d99d,3e8,...) at _sleep+0x36a vnlru_proc(0,ee31dd28,c0f876b0,3db,ca6c02c4,...) at vnlru_proc+0xe7 fork_exit(c0afcdd0,0,ee31dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee31dd60, ebp = 0 --- Tracing command syncer pid 17 tid 100075 td 0xc805eb80 sched_switch(c805eb80,0,104,1a3,70868cb,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805eb80,0,c0f929a0,28b,c805eb80,...) at sleepq_switch+0x162 sleepq_timedwait(c129fcb4,0,ee31ac78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c129fcb4,c129fca0,3e8,78b,4e20,...) at _cv_timedwait+0x270 sched_sync(0,ee31ad28,c0f876b0,3db,ca6c0588,...) at sched_sync+0x595 fork_exit(c0afc140,0,ee31ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee31ad60, ebp = 0 --- Tracing command bufdaemon pid 16 tid 100074 td 0xc8183000 sched_switch(c8183000,0,104,1a3,708636a,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c8183000,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c129f948,54,c0f9a8d1,0,0,...) at sleepq_timedwait+0x6b _sleep(c129f948,c129f950,54,c0f9a8d1,3e8,...) at _sleep+0x36a buf_daemon(0,ee317d28,c0f876b0,3db,ca6c084c,...) at buf_daemon+0x175 fork_exit(c0ae08e0,0,ee317d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee317d60, ebp = 0 --- Tracing command pagezero pid 9 tid 100073 td 0xc81832e0 sched_switch(c81832e0,0,104,1a3,29123c2a,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c81832e0,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a7e5c,0,c0fca19e,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a7e5c,c12a6900,0,c0fca19e,493e0,...) at _sleep+0x36a vm_pagezero(0,ee314d28,c0f876b0,3db,ca6c0b10,...) at vm_pagezero+0xdc fork_exit(c0ce1840,0,ee314d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee314d60, ebp = 0 --- Tracing command vmdaemon pid 8 tid 100072 td 0xc81835c0 sched_switch(c81835c0,0,104,1a3,29004e27,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(c81835c0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a7a48,74,c0f9a8d1,0,0,...) at sleepq_wait+0x63 _sleep(c12a7a48,c12a7a4c,74,c0f9a8d1,0,...) at _sleep+0x39c vm_daemon(0,ee311d28,c0f876b0,3db,ca6c1000,...) at vm_daemon+0x59 fork_exit(c0cdb150,0,ee311d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee311d60, ebp = 0 --- Tracing command pagedaemon pid 7 tid 100071 td 0xc81838a0 sched_switch(c81838a0,0,104,1a3,59e2ee66,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c81838a0,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a7a10,54,c0f9a8d1,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a7a10,c12a6900,54,c0f9a8d1,1388,...) at _sleep+0x36a vm_pageout(0,ee30ed28,c0f876b0,3db,c7dc82c4,...) at vm_pageout+0x2c7 fork_exit(c0cdc210,0,ee30ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee30ed60, ebp = 0 --- Tracing command xpt_thrd pid 6 tid 100070 td 0xc8183b80 sched_switch(c8183b80,0,104,1a3,9fdd464a,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8183b80,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10ee2d4,5c,c0de8167,0,0,...) at sleepq_wait+0x63 _sleep(c10ee2d4,c10ee318,5c,c0de8167,0,...) at _sleep+0x39c xpt_scanner_thread(0,ee18bd28,c0f876b0,3db,c7dc8588,...) at xpt_scanner_thread+0x47 fork_exit(c049d350,0,ee18bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee18bd60, ebp = 0 --- Tracing command sctp_iterator pid 5 tid 100069 td 0xc8184000 sched_switch(c8184000,0,104,1a3,d223b5d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8184000,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a17d8,0,c0fb103f,0,0,...) at sleepq_wait+0x63 _sleep(c12a17d8,c12a17a0,0,c0fb103f,0,...) at _sleep+0x39c sctp_iterator_thread(0,ee188d28,c0f876b0,3db,c7dc884c,...) at sctp_iterator_thread+0x5c fork_exit(c0b8e610,0,ee188d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee188d60, ebp = 0 --- Tracing command ctl_thrd pid 4 tid 100067 td 0xc81845c0 sched_switch(c81845c0,0,104,1a3,8d7d73fd,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c81845c0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c819c000,5c,c0df2b72,0,0,...) at sleepq_wait+0x63 _sleep(c819c000,c819c000,5c,c0df2b72,0,...) at _sleep+0x39c ctl_work_thread(c819c000,ee182d28,c0f876b0,3db,c7dc8b10,...) at ctl_work_thread+0x22f6 fork_exit(c04c6770,c819c000,ee182d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee182d60, ebp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xc8185000 sched_switch(c8185000,0,104,1a3,18a37807,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8185000,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c7ffa03c,5c,c0f7f9f2,0,0,...) at sleepq_timedwait+0x6b _sleep(c7ffa03c,c7ffa0f0,5c,c0f7f9f2,3e8,...) at _sleep+0x36a fdc_thread(c7ffa000,ee179d28,c0f876b0,3db,c8042000,...) at fdc_thread+0x27d fork_exit(c0d60a90,c7ffa000,ee179d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee179d60, ebp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xc80515c0 sched_switch(c80515c0,0,104,1a3,9fdd8331,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c80515c0,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c808f000,6c,c0f7f9f2,100,0,...) at sleepq_wait_sig+0x17 _sleep(c808f000,c8093488,16c,c0f7f9f2,0,...) at _sleep+0x385 fw_bus_probe_thread(c808f000,ee169d28,c0f876b0,3db,c80422c4,...) at fw_bus_probe_thread+0xa08 fork_exit(c06f09f0,c808f000,ee169d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee169d60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc805c2e0 sched_switch(c805c2e0,0,104,1a3,8d784a18,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805c2e0,0,c0f929a0,268,c805c2e0,...) at sleepq_switch+0x162 sleepq_wait(c8063d34,0,ee152cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8063d34,c8063e4c,c0f660b1,74,c8063d3c,...) at _cv_wait+0x260 usb_process(c8063d2c,ee152d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8063d2c,ee152d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee152d60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc805c5c0 sched_switch(c805c5c0,0,104,1a3,6fc7179e,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805c5c0,0,c0f929a0,268,c805c5c0,...) at sleepq_switch+0x162 sleepq_wait(c8063d04,0,ee14fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8063d04,c8063e4c,c0f660b1,74,c8063d0c,...) at _cv_wait+0x260 usb_process(c8063cfc,ee14fd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8063cfc,ee14fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee14fd60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc805c8a0 sched_switch(c805c8a0,0,104,1a3,8d11fbdc,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805c8a0,0,c0f929a0,268,c805c8a0,...) at sleepq_switch+0x162 sleepq_wait(c8063cd4,0,ee14ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8063cd4,c8063e4c,c0f660b1,74,c8063cdc,...) at _cv_wait+0x260 usb_process(c8063ccc,ee14cd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8063ccc,ee14cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee14cd60, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc805cb80 sched_switch(c805cb80,0,104,1a3,8d11e9d3,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805cb80,0,c0f929a0,268,c805cb80,...) at sleepq_switch+0x162 sleepq_wait(c8063ca4,0,ee149cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8063ca4,c8063e4c,c0f660b1,74,c8063cac,...) at _cv_wait+0x260 usb_process(c8063c9c,ee149d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8063c9c,ee149d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee149d60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc805e2e0 sched_switch(c805e2e0,0,104,1a3,8d11d599,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805e2e0,0,c0f929a0,268,c805e2e0,...) at sleepq_switch+0x162 sleepq_wait(c8057b5c,0,ee140cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8057b5c,c8057c74,c0f660b1,74,c8057b64,...) at _cv_wait+0x260 usb_process(c8057b54,ee140d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8057b54,ee140d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee140d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc805e5c0 sched_switch(c805e5c0,0,104,1a3,5f3c930c,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c805e5c0,0,c0f929a0,268,c805e5c0,...) at sleepq_switch+0x162 sleepq_wait(c8057b2c,0,ee13dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8057b2c,c8057c74,c0f660b1,74,c8057b34,...) at _cv_wait+0x260 usb_process(c8057b24,ee13dd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8057b24,ee13dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee13dd60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc804e000 sched_switch(c804e000,0,104,1a3,8cae6bfb,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804e000,0,c0f929a0,268,c804e000,...) at sleepq_switch+0x162 sleepq_wait(c8057afc,0,ee13acac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8057afc,c8057c74,c0f660b1,74,c8057b04,...) at _cv_wait+0x260 usb_process(c8057af4,ee13ad28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8057af4,ee13ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee13ad60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc804e2e0 sched_switch(c804e2e0,0,104,1a3,8cae5aa1,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804e2e0,0,c0f929a0,268,c804e2e0,...) at sleepq_switch+0x162 sleepq_wait(c8057acc,0,ee137cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8057acc,c8057c74,c0f660b1,74,c8057ad4,...) at _cv_wait+0x260 usb_process(c8057ac4,ee137d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8057ac4,ee137d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee137d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc804e5c0 sched_switch(c804e5c0,0,104,1a3,8cae45b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804e5c0,0,c0f929a0,268,c804e5c0,...) at sleepq_switch+0x162 sleepq_wait(c8052b5c,0,ee133cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8052b5c,c8052c74,c0f660b1,74,c8052b64,...) at _cv_wait+0x260 usb_process(c8052b54,ee133d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8052b54,ee133d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee133d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc804e8a0 sched_switch(c804e8a0,0,104,1a3,5c204725,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804e8a0,0,c0f929a0,268,c804e8a0,...) at sleepq_switch+0x162 sleepq_wait(c8052b2c,0,ee130cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8052b2c,c8052c74,c0f660b1,74,c8052b34,...) at _cv_wait+0x260 usb_process(c8052b24,ee130d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8052b24,ee130d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee130d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc804eb80 sched_switch(c804eb80,0,104,1a3,86dd0e8b,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804eb80,0,c0f929a0,268,c804eb80,...) at sleepq_switch+0x162 sleepq_wait(c8052afc,0,ee12dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8052afc,c8052c74,c0f660b1,74,c8052b04,...) at _cv_wait+0x260 usb_process(c8052af4,ee12dd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8052af4,ee12dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee12dd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc804f000 sched_switch(c804f000,0,104,1a3,86dcfd8a,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804f000,0,c0f929a0,268,c804f000,...) at sleepq_switch+0x162 sleepq_wait(c8052acc,0,ee12acac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8052acc,c8052c74,c0f660b1,74,c8052ad4,...) at _cv_wait+0x260 usb_process(c8052ac4,ee12ad28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8052ac4,ee12ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee12ad60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc804f2e0 sched_switch(c804f2e0,0,104,1a3,86dce9c9,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804f2e0,0,c0f929a0,268,c804f2e0,...) at sleepq_switch+0x162 sleepq_wait(c804ab5c,0,ee126cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c804ab5c,c804ac74,c0f660b1,74,c804ab64,...) at _cv_wait+0x260 usb_process(c804ab54,ee126d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c804ab54,ee126d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee126d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc804f5c0 sched_switch(c804f5c0,0,104,1a3,5af232c6,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804f5c0,0,c0f929a0,268,c804f5c0,...) at sleepq_switch+0x162 sleepq_wait(c804ab2c,0,ee123cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c804ab2c,c804ac74,c0f660b1,74,c804ab34,...) at _cv_wait+0x260 usb_process(c804ab24,ee123d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c804ab24,ee123d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee123d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc804f8a0 sched_switch(c804f8a0,0,104,1a3,86796d82,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804f8a0,0,c0f929a0,268,c804f8a0,...) at sleepq_switch+0x162 sleepq_wait(c804aafc,0,ee120cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c804aafc,c804ac74,c0f660b1,74,c804ab04,...) at _cv_wait+0x260 usb_process(c804aaf4,ee120d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c804aaf4,ee120d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee120d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc804fb80 sched_switch(c804fb80,0,104,1a3,86795a5e,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c804fb80,0,c0f929a0,268,c804fb80,...) at sleepq_switch+0x162 sleepq_wait(c804aacc,0,ee11dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c804aacc,c804ac74,c0f660b1,74,c804aad4,...) at _cv_wait+0x260 usb_process(c804aac4,ee11dd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c804aac4,ee11dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee11dd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc803e000 sched_switch(c803e000,0,104,1a3,8679471b,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c803e000,0,c0f929a0,268,c803e000,...) at sleepq_switch+0x162 sleepq_wait(c8044b5c,0,ee116cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8044b5c,c8044c74,c0f660b1,74,c8044b64,...) at _cv_wait+0x260 usb_process(c8044b54,ee116d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8044b54,ee116d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee116d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc803e2e0 sched_switch(c803e2e0,0,104,1a3,59c3da2c,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c803e2e0,0,c0f929a0,268,c803e2e0,...) at sleepq_switch+0x162 sleepq_wait(c8044b2c,0,ee113cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8044b2c,c8044c74,c0f660b1,74,c8044b34,...) at _cv_wait+0x260 usb_process(c8044b24,ee113d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8044b24,ee113d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee113d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc803e5c0 sched_switch(c803e5c0,0,104,1a3,8615b44f,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c803e5c0,0,c0f929a0,268,c803e5c0,...) at sleepq_switch+0x162 sleepq_wait(c8044afc,0,ee110cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8044afc,c8044c74,c0f660b1,74,c8044b04,...) at _cv_wait+0x260 usb_process(c8044af4,ee110d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8044af4,ee110d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee110d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc803e8a0 sched_switch(c803e8a0,0,104,1a3,8615a19e,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c803e8a0,0,c0f929a0,268,c803e8a0,...) at sleepq_switch+0x162 sleepq_wait(c8044acc,0,ee10dcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8044acc,c8044c74,c0f660b1,74,c8044ad4,...) at _cv_wait+0x260 usb_process(c8044ac4,ee10dd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8044ac4,ee10dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee10dd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc8040000 sched_switch(c8040000,0,104,1a3,86158996,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8040000,0,c0f929a0,268,c8040000,...) at sleepq_switch+0x162 sleepq_wait(c8039b5c,0,ee106cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8039b5c,c8039c74,c0f660b1,74,c8039b64,...) at _cv_wait+0x260 usb_process(c8039b54,ee106d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8039b54,ee106d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee106d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc80402e0 sched_switch(c80402e0,0,104,1a3,57d5b836,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c80402e0,0,c0f929a0,268,c80402e0,...) at sleepq_switch+0x162 sleepq_wait(c8039b2c,0,ee103cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8039b2c,c8039c74,c0f660b1,74,c8039b34,...) at _cv_wait+0x260 usb_process(c8039b24,ee103d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8039b24,ee103d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee103d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc80405c0 sched_switch(c80405c0,0,104,1a3,85b160c7,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c80405c0,0,c0f929a0,268,c80405c0,...) at sleepq_switch+0x162 sleepq_wait(c8039afc,0,ee100cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8039afc,c8039c74,c0f660b1,74,c8039b04,...) at _cv_wait+0x260 usb_process(c8039af4,ee100d28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8039af4,ee100d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee100d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc80408a0 sched_switch(c80408a0,0,104,1a3,85b14ce1,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c80408a0,0,c0f929a0,268,c80408a0,...) at sleepq_switch+0x162 sleepq_wait(c8039acc,0,ee0fdcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c8039acc,c8039c74,c0f660b1,74,c8039ad4,...) at _cv_wait+0x260 usb_process(c8039ac4,ee0fdd28,c0f876b0,3db,c8042588,...) at usb_process+0x1a3 fork_exit(c08fd9d0,c8039ac4,ee0fdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee0fdd60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc7eda5c0 sched_switch(c7eda5c0,0,104,1a3,1b6b0973,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7eda5c0,0,c0f929a0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c11288e4,0,c0f7f9f2,2,0,...) at sleepq_timedwait+0x6b _sleep(c11288e4,0,0,c0f7f9f2,64) at _sleep+0x36a pause(c0f7f9f2,64,c0f41e70,111,0,...) at pause+0xa0 random_kthread(0,c7b51d28,c0f876b0,3db,c7dc7000,...) at random_kthread+0x1ef fork_exit(c0805c50,0,c7b51d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b51d60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc7edab80 sched_switch(c7edab80,0,104,1a3,1e4c872f,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c7edab80,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11264e8,5c,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c11264e8,c1126448,25c,c0f7f9f2,0,...) at _sleep+0x39c g_io_schedule_down(c7edab80,0,c0f81541,6c,c7b4bd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c7b4bd28,c0f876b0,3db,c7dc72c4,...) at g_down_procbody+0x9d fork_exit(c09ef070,0,c7b4bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b4bd60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc7edb000 sched_switch(c7edb000,0,104,1a3,1e4b6714,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c7edb000,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11264e4,5c,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c11264e4,c1126468,25c,c0f7f9f2,0,...) at _sleep+0x39c g_io_schedule_up(c7edb000,0,c0f81541,5f,c7b48d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c7b48d28,c0f876b0,3db,c7dc72c4,...) at g_up_procbody+0x9d fork_exit(c09ef110,0,c7b48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b48d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc7dca2e0 sched_switch(c7dca2e0,0,104,1a3,37da5308,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c7dca2e0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11264dc,5c,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c11264dc,c112640c,25c,c0f7f9f2,0,...) at _sleep+0x39c g_run_events(c1136740,0,c0f81541,79,c7b45d14,...) at g_run_events+0x54b g_event_procbody(0,c7b45d28,c0f876b0,3db,c7dc72c4,...) at g_event_procbody+0x90 fork_exit(c09ef1b0,0,c7b45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b45d60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc81848a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc8184b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc80512e0 sched_switch(c80512e0,0,109,1a3,c5ad489b,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c80512e0,...) at mi_switch+0x1ff ithread_loop(c7e12690,ee16cd28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12690,ee16cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee16cd60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc8051b80 sched_switch(c8051b80,0,109,1a3,75b1cd2d,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c8051b80,...) at mi_switch+0x1ff ithread_loop(c7e126a0,ee160d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e126a0,ee160d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee160d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc805c000 sched_switch(c805c000,0,109,1a3,2770efe0,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c805c000,...) at mi_switch+0x1ff ithread_loop(c7e126b0,ee158d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e126b0,ee158d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee158d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc805e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc8051000 sched_switch(c8051000,0,109,1a3,aab7d0b0,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c8051000,...) at mi_switch+0x1ff ithread_loop(c7e126d0,ee11ad28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e126d0,ee11ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee11ad60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc803eb80 sched_switch(c803eb80,0,109,1a3,a53ef5b5,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c803eb80,...) at mi_switch+0x1ff ithread_loop(c7e126e0,ee10ad28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e126e0,ee10ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee10ad60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc8040b80 sched_switch(c8040b80,0,109,1a3,df8d189c,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c8040b80,...) at mi_switch+0x1ff ithread_loop(c7e126f0,ee0fad28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e126f0,ee0fad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee0fad60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc7edb2e0 sched_switch(c7edb2e0,0,109,1a3,f7f963b3,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7edb2e0,...) at mi_switch+0x1ff ithread_loop(c7e12700,edfa4d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12700,edfa4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xedfa4d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc7edb5c0 sched_switch(c7edb5c0,0,109,1a3,c952f82e,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7edb5c0,...) at mi_switch+0x1ff ithread_loop(c7e12710,edfa0d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12710,edfa0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xedfa0d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc7edb8a0 sched_switch(c7edb8a0,0,109,1a3,778fc731,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7edb8a0,...) at mi_switch+0x1ff ithread_loop(c7e12720,c7ba0d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12720,c7ba0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7ba0d60, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc7edbb80 sched_switch(c7edbb80,0,109,1a3,a8c49180,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7edbb80,...) at mi_switch+0x1ff ithread_loop(c7e12730,c7b6fd28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12730,c7b6fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b6fd60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc7fc7000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xc7fc75c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100018 td 0xc7eda2e0 sched_switch(c7eda2e0,0,109,1a3,f7fa8380,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7eda2e0,...) at mi_switch+0x1ff ithread_loop(c7e12760,c7b54d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e12760,c7b54d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b54d60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc7dca5c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc7dca8a0 sched_switch(c7dca8a0,0,109,1a3,1b6ad225,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7dca8a0,...) at mi_switch+0x1ff ithread_loop(c7e127b0,c7b3fd28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e127b0,c7b3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b3fd60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc7dcab80 sched_switch(c7dcab80,0,109,1a3,18a07b65,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7dcab80,...) at mi_switch+0x1ff ithread_loop(c7e127c0,c7b3cd28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e127c0,c7b3cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b3cd60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc7e13000 sched_switch(c7e13000,0,109,1a3,7084548,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7e13000,...) at mi_switch+0x1ff ithread_loop(c7e127d0,c7b39d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e127d0,c7b39d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b39d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc7e132e0 sched_switch(c7e132e0,0,109,1a3,1b36c656,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7e132e0,...) at mi_switch+0x1ff ithread_loop(c7e127e0,c7b36d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e127e0,c7b36d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b36d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc7e135c0 sched_switch(c7e135c0,0,109,1a3,12bc848,...) at sched_switch+0x3ec mi_switch(109,0,c0f879d3,554,c7e135c0,...) at mi_switch+0x1ff ithread_loop(c7e127f0,c7b33d28,c0f876b0,3db,c7dc7588,...) at ithread_loop+0x1fb fork_exit(c0a31a80,c7e127f0,c7b33d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b33d60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc7dc9000 sched_switch(c7dc9000,0,608,19e,1e4b7994,...) at sched_switch+0x3ec mi_switch(608,0,c0f8d7df,cd,0,...) at mi_switch+0x1ff critical_exit(0,ffffffff,c1137ac0,2,c7b2fce8,...) at critical_exit+0xb5 cpu_idle(0,c7b2fcc4,c0f8f062,a0b,c7dc9000,...) at cpu_idle+0x97 sched_idletd(0,c7b2fd28,c0f876b0,3db,c7dc784c,...) at sched_idletd+0x263 fork_exit(c0a828a0,0,c7b2fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b2fd60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc7dc92e0 cpustop_handler(2,13,c7b2cc04,c0dc01e6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,0,c12d4c38,c7dc784c,...) at ipi_nmi_handler+0x47 trap(c7b2cc10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0da4a75, esp = 0xc7b2cc50, ebp = 0xc7b2cc50 --- acpi_cpu_c1(c7b2cc6c,c10ee8b0,2,301,13,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b2cca0,c0db2d09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1137440,2,c7b2cce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b2ccc4,c0f8f062,3c0,c7dc92e0,...) at cpu_idle+0x89 sched_idletd(0,c7b2cd28,c0f876b0,3db,c7dc784c,...) at sched_idletd+0x263 fork_exit(c0a828a0,0,c7b2cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b2cd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc7dc95c0 cpustop_handler(1,13,c7b29c04,c0dc01e6,c7dc95c0,...) at cpustop_handler+0x28 ipi_nmi_handler(c7dc95c0,c7b29b88,0,c12d490c,c7dc784c,...) at ipi_nmi_handler+0x47 trap(c7b29c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0da4a75, esp = 0xc7b29c50, ebp = 0xc7b29c50 --- acpi_cpu_c1(c7b29c6c,c10ee8b0,1,301,13,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b29ca0,c0db2d09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1136dc0,2,c7b29ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b29cc4,c0f8f062,a0b,c7dc95c0,...) at cpu_idle+0x89 sched_idletd(0,c7b29d28,c0f876b0,3db,c7dc784c,...) at sched_idletd+0x263 fork_exit(c0a828a0,0,c7b29d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b29d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc7dc98a0 cpustop_handler(0,13,c7b26c04,c0dc01e6,c1136740,...) at cpustop_handler+0x28 ipi_nmi_handler(c1136740,0,0,c12d45e0,c7dc784c,...) at ipi_nmi_handler+0x47 trap(c7b26c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0da4a75, esp = 0xc7b26c50, ebp = 0xc7b26c50 --- acpi_cpu_c1(c7b26c6c,c10ee8b0,0,301,13,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b26ca0,c0db2d09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1136740,2,c7b26ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b26cc4,c0f8f062,a0b,c7dc98a0,...) at cpu_idle+0x89 sched_idletd(0,c7b26d28,c0f876b0,3db,c7dc784c,...) at sched_idletd+0x263 fork_exit(c0a828a0,0,c7b26d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b26d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc7dc9b80 sched_switch(c7dc9b80,0,104,1a3,c5a6627d,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c7dc9b80,0,c0f929a0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f929a0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7dc7b10,6c,c0f958b7,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7dc7b10,c7dc7b98,16c,c0f958b7,0,...) at _sleep+0x385 kern_wait(c7dc9b80,ffffffff,c7b22c60,2,0,...) at kern_wait+0x626 sys_wait4(c7dc9b80,c7b22cec,c,c,286,...) at sys_wait4+0x3b syscall(c7b22d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8060f97, esp = 0xbfbfe74c, ebp = 0xbfbfe768 --- Tracing command audit pid 10 tid 100001 td 0xc7dca000 sched_switch(c7dca000,0,104,1a3,85af786b,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7dca000,0,c0f929a0,268,c7dca000,...) at sleepq_switch+0x162 sleepq_wait(c12a56e0,0,c7b1fc8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c12a56e0,c12a56c4,c0fbdbd7,194,0,...) at _cv_wait+0x260 audit_worker(0,c7b1fd28,c0f876b0,3db,c7dc8000,...) at audit_worker+0x84 fork_exit(c0c5b3e0,0,c7b1fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b1fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100078 td 0xca6cd8a0 sched_switch(ca6cd8a0,0,104,1a3,290082db,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ca6cd8a0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8036380,0,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c8036380,c8036398,0,c0f7f9f2,0,...) at _sleep+0x39c taskqueue_thread_loop(c1306f60,ee323d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9f710,c1306f60,ee323d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee323d60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xc81842e0 sched_switch(c81842e0,0,104,1a3,18a1d557,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c81842e0,0,c0f929a0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c11288e4,0,c0f7f9f2,2,0,...) at sleepq_timedwait+0x6b _sleep(c11288e4,0,0,c0f7f9f2,bb8) at _sleep+0x36a pause(c0f7f9f2,bb8,124,122,c1126648,...) at pause+0xa0 deadlkres(0,ee185d28,c0f876b0,3db,c11265c0,...) at deadlkres+0x324 fork_exit(c0a11250,0,ee185d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee185d60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc80518a0 sched_switch(c80518a0,0,104,1a3,169ea854,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c80518a0,0,c0f929a0,268,c80518a0,...) at sleepq_switch+0x162 sleepq_wait(c80366c0,0,c0f8dd01,c0f7f9f2,0,...) at sleepq_wait+0x63 msleep_spin(c80366c0,c80366d8,c0f7f9f2,0,c0f8b3cb,...) at msleep_spin+0x259 taskqueue_thread_loop(c809349c,ee166d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9f710,c809349c,ee166d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee166d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc7fc72e0 sched_switch(c7fc72e0,0,104,1a3,278b5beb,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7fc72e0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7f9ee80,0,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c7f9ee80,c7f9ee98,0,c0f7f9f2,0,...) at _sleep+0x39c taskqueue_thread_loop(c11454ac,c7b69d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9f710,c11454ac,c7b69d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b69d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc7fc78a0 sched_switch(c7fc78a0,0,104,1a3,9fe1a257,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7fc78a0,0,c0f929a0,268,c7fc78a0,...) at sleepq_switch+0x162 sleepq_wait(c7f9f040,0,c0f8dd01,c0f7f9f2,0,...) at sleepq_wait+0x63 msleep_spin(c7f9f040,c7f9f058,c0f7f9f2,0,c0f8b3cb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f1538,c7b63d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9f710,c10f1538,c7b63d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b63d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7fc7b80 sched_switch(c7fc7b80,0,104,1a3,9fe19336,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7fc7b80,0,c0f929a0,268,c7fc7b80,...) at sleepq_switch+0x162 sleepq_wait(c7f9f040,0,c0f8dd01,c0f7f9f2,0,...) at sleepq_wait+0x63 msleep_spin(c7f9f040,c7f9f058,c0f7f9f2,0,c0f8b3cb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f1538,c7b60d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9f710,c10f1538,c7b60d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b60d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc7e138a0 sched_switch(c7e138a0,0,104,1a3,9fe180f6,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7e138a0,0,c0f929a0,268,c7e138a0,...) at sleepq_switch+0x162 sleepq_wait(c7f9f040,0,c0f8dd01,c0f7f9f2,0,...) at sleepq_wait+0x63 msleep_spin(c7f9f040,c7f9f058,c0f7f9f2,0,c0f8b3cb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f1538,c7b5dd28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9f710,c10f1538,c7b5dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b5dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc7e13b80 sched_switch(c7e13b80,0,104,1a3,9fddb94c,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7e13b80,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7f9f080,0,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c7f9f080,c7f9f098,0,c0f7f9f2,0,...) at _sleep+0x39c taskqueue_thread_loop(c1126f38,c7b5ad28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9f710,c1126f38,c7b5ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b5ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc7eda000 sched_switch(c7eda000,0,104,1a3,9fdda7ff,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7eda000,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7f9f0c0,0,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c7f9f0c0,c7f9f0d8,0,c0f7f9f2,0,...) at _sleep+0x39c taskqueue_thread_loop(c12a5cc4,c7b57d28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9f710,c12a5cc4,c7b57d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b57d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc7eda8a0 sched_switch(c7eda8a0,0,104,1a3,63d47a1a,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c7eda8a0,0,c0f929a0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7dade80,0,c0f7f9f2,0,0,...) at sleepq_wait+0x63 _sleep(c7dade80,c7dade98,0,c0f7f9f2,0,...) at _sleep+0x39c taskqueue_thread_loop(c1143ec0,c7b4ed28,c0f876b0,3db,c11265c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9f710,c1143ec0,c7b4ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b4ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1126890 sched_switch(c1126890,0,104,1a3,5c23b1eb,...) at sched_switch+0x3ec mi_switch(104,0,c0f929a0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c1126890,0,c0f929a0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c11265c0,54,c0f8f974,0,0,...) at sleepq_timedwait+0x6b _sleep(c11265c0,0,54,c0f8f974,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Dumping 156 MB: 141 125 109 93 77 61 45 29 13 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) bt #0 doadump (textdump=0xf285f434) at pcpu.h:244 #1 0xc0510a89 in db_fncall (dummy1=0xc0a90e3a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xf285f4c8 "¿\205¿") at ../../../ddb/db_command.c:573 #2 0xc0510ebf in db_command (last_cmdp=0xc10ef6fc, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:449 #3 0xc0510f74 in db_command_script (command=0xc10f0608 "call doadump") at ../../../ddb/db_command.c:520 #4 0xc0515360 in db_script_exec (scriptname=0xc10eff60 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc05153f1 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc12d2860, modif=0xf285f600 "") at ../../../ddb/db_script.c:375 #6 0xc0510e81 in db_command (last_cmdp=0xc10ef6fc, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:449 #7 0xc0510fda in db_command_loop () at ../../../ddb/db_command.c:502 #8 0xc0512fcd in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc0a912f6 in kdb_trap (type=0x3, code=0x0, tf=0xf285f7b4) at ../../../kern/subr_kdb.c:629 #10 0xc0dc081b in trap (frame=0xf285f7b4) at ../../../i386/i386/trap.c:722 #11 0xc0da90ac in calltrap () at ../../../i386/i386/exception.s:168 #12 0xc0a90e3a in kdb_enter (why=0xc0f8cf98 "panic", msg=0xc0f8cf98 "panic") at cpufunc.h:71 #13 0xc0a5bd0c in panic (fmt=0xc0f8b611 "_mtx_lock_sleep: recursed on non-recursive mutex %s @ %s:%d\n") at ../../../kern/kern_shutdown.c:633 #14 0xc0a4b5b8 in _mtx_lock_sleep (m=0xca475254, tid=0xcab5d2e0, opts=0x0, file=0xc0f99f8a "../../../kern/vfs_bio.c", line=0xa54) at ../../../kern/kern_mutex.c:357 #15 0xc0a4b7df in _mtx_lock_flags (m=0xca475254, opts=0x0, file=0xc0f99f8a "../../../kern/vfs_bio.c", line=0xa54) at ../../../kern/kern_mutex.c:206 #16 0xc0ae17f3 in getblk (vp=0xca47515c, blkno=0xe0, size=0x4000, slpflag=0x0, slptimeo=0x0, flags=0x0) at ../../../kern/vfs_bio.c:2644 #17 0xc0ae20e4 in breadn (vp=0xca47515c, blkno=Unhandled dwarf expression opcode 0x93 ) at ../../../kern/vfs_bio.c:849 #18 0xc0ae225c in bread (vp=0xca47515c, blkno=Unhandled dwarf expression opcode 0x93 ) at ../../../kern/vfs_bio.c:797 #19 0xc0c7be12 in ffs_update (vp=0xcaaeec3c, waitfor=0x0) at ../../../ufs/ffs/ffs_inode.c:95 #20 0xc0c8b378 in handle_workitem_remove (dirrem=0xcab2bc00, flags=0x80200) at ../../../ufs/ffs/ffs_softdep.c:9378 #21 0xc0c8a7c4 in process_worklist_item (mp=0xcaaf42d4, target=0xa, flags=0x200) at ../../../ufs/ffs/ffs_softdep.c:1729 #22 0xc0c9a218 in softdep_process_worklist (mp=0xcaaf42d4, full=0x1) at ../../../ufs/ffs/ffs_softdep.c:1518 #23 0xc0c9a3dc in softdep_flushworklist (oldmnt=0xcaaf42d4, countp=0xf285fa98, td=0xcab5d2e0) at ../../../ufs/ffs/ffs_softdep.c:1833 #24 0xc0ae77cb in vop_stdfsync (ap=0xf285faf8) at ../../../kern/vfs_default.c:674 #25 0xc096ebec in devfs_fsync (ap=0xf285faf8) at ../../../fs/devfs/devfs_vnops.c:646 #26 0xc0de1ed5 in VOP_FSYNC_APV (vop=0xc10b19a0, a=0xf285faf8) at vnode_if.c:1267 #27 0xc0c9a3a0 in softdep_flushworklist (oldmnt=0xcaaf42d4, countp=0xf285fb6c, td=0xcab5d2e0) at vnode_if.h:549 #28 0xc0c9f3cd in ffs_sync (mp=0xcaaf42d4, waitfor=0x1) at ../../../ufs/ffs/ffs_vfsops.c:1527 #29 0xc0af1ef8 in dounmount (mp=0xcaaf42d4, flags=0x8080000, td=0xcab5d2e0) at ../../../kern/vfs_mount.c:1324 #30 0xc0af25e1 in sys_unmount (td=0xcab5d2e0, uap=0xf285fcec) at ../../../kern/vfs_mount.c:1224 #31 0xc0dbfde7 in syscall (frame=0xf285fd28) at subr_syscall.c:135 #32 0xc0da9111 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #33 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 16 #16 0xc0ae17f3 in getblk (vp=0xca47515c, blkno=0xe0, size=0x4000, slpflag=0x0, slptimeo=0x0, flags=0x0) at ../../../kern/vfs_bio.c:2644 2644 BO_LOCK(bo); (kgdb) l 2639 mtx_lock(&nblock); 2640 needsbuffer |= VFS_BIO_NEED_ANY; 2641 mtx_unlock(&nblock); 2642 } 2643 2644 BO_LOCK(bo); 2645 bp = gbincore(bo, blkno); 2646 if (bp != NULL) { 2647 int lockflags; 2648 /* (kgdb) info loc bp = (struct buf *) 0xcaafc000 bo = (struct bufobj *) 0xca475254 error = Variable "error" is not available. (kgdb) p *bo $1 = {bo_mtx = {lock_object = {lo_name = 0xc0f9d972 "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d77c10}, mtx_lock = 0xcab5d2e0}, bo_clean = {bv_hd = { tqh_first = 0xe1904e60, tqh_last = 0xe19203b8}, bv_root = 0xe191f000, bv_cnt = 0x15}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xca475278}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x1, bo_flag = 0x0, bo_ops = 0xc10db9b4, bo_bsize = 0x200, bo_object = 0xca8283f0, bo_synclist = {le_next = 0xcab36e90, le_prev = 0xca43102c}, bo_private = 0xcaaf7340, __bo_vnode = 0xca47515c} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/sys/conf.h =================================================================== --- /usr/src/sys/sys/conf.h (revision 231557) +++ /usr/src/sys/sys/conf.h (working copy) @@ -84,6 +84,7 @@ struct cdev { int si_iosize_max; /* maximum I/O size (for physio &al) */ u_long si_usecount; u_long si_threadcount; + struct mount *si_mountpt; union { struct snapdata *__sid_snapdata; } __si_u; Index: /usr/src/sys/ufs/ufs/inode.h =================================================================== --- /usr/src/sys/ufs/ufs/inode.h (revision 231557) +++ /usr/src/sys/ufs/ufs/inode.h (working copy) @@ -158,6 +158,7 @@ struct inode { #define SHORTLINK(ip) \ (((ip)->i_ump->um_fstype == UFS1) ? \ (caddr_t)(ip)->i_din1->di_db : (caddr_t)(ip)->i_din2->di_db) +#define IS_SNAPSHOT(ip) ((ip)->i_flags & SF_SNAPSHOT) /* * Structure used to pass around logical block paths generated by Index: /usr/src/sys/ufs/ufs/ufs_quota.c =================================================================== --- /usr/src/sys/ufs/ufs/ufs_quota.c (revision 231557) +++ /usr/src/sys/ufs/ufs/ufs_quota.c (working copy) @@ -80,7 +80,7 @@ static int dqopen(struct vnode *, struct ufsmount static int dqget(struct vnode *, u_long, struct ufsmount *, int, struct dquot **); static int dqsync(struct vnode *, struct dquot *); -static void dqflush(struct vnode *); +static int dqflush(struct vnode *); static int quotaoff1(struct thread *td, struct mount *mp, int type); static int quotaoff_inchange(struct thread *td, struct mount *mp, int type); @@ -694,7 +694,9 @@ again: } MNT_IUNLOCK(mp); - dqflush(qvp); + if ((error = dqflush(qvp)) != 0) + return (error); + /* Clear um_quotas before closing the quota vnode to prevent * access to the closed vnode from dqget/dqsync */ @@ -1601,17 +1603,19 @@ out: /* * Flush all entries from the cache for a particular vnode. */ -static void +static int dqflush(struct vnode *vp) { struct dquot *dq, *nextdq; struct dqhash *dqh; + int error; /* * Move all dquot's that used to refer to this quota * file off their hash chains (they will eventually * fall off the head of the free list and be re-used). */ + error = 0; DQH_LOCK(); for (dqh = &dqhashtbl[dqhash]; dqh >= dqhashtbl; dqh--) { for (dq = LIST_FIRST(dqh); dq; dq = nextdq) { @@ -1619,12 +1623,13 @@ dqflush(struct vnode *vp) if (dq->dq_ump->um_quotas[dq->dq_type] != vp) continue; if (dq->dq_cnt) - panic("dqflush: stray dquot"); + error = EBUSY; LIST_REMOVE(dq, dq_hash); dq->dq_ump = (struct ufsmount *)0; } } DQH_UNLOCK(); + return (error); } /* Index: /usr/src/sys/ufs/ufs/ufs_vfsops.c =================================================================== --- /usr/src/sys/ufs/ufs/ufs_vfsops.c (revision 231557) +++ /usr/src/sys/ufs/ufs/ufs_vfsops.c (working copy) @@ -125,6 +125,14 @@ ufs_quotactl(mp, cmds, id, arg) case Q_QUOTAOFF: error = quotaoff(td, mp, type); + if (error == EBUSY) { + /* + * If busy, flush the filesystem and try again. + * If still busy, return error. + */ + (void) VFS_SYNC(mp, MNT_WAIT); + error = quotaoff(td, mp, type); + } break; case Q_SETQUOTA32: Index: /usr/src/sys/ufs/ffs/ffs_softdep.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_softdep.c (revision 231557) +++ /usr/src/sys/ufs/ffs/ffs_softdep.c (working copy) @@ -1895,11 +1895,21 @@ retry_flush: * Do another flush in case any vnodes were brought in * as part of the cleanup operations. */ - if ((error = ffs_flushfiles(oldmnt, flags, td)) != 0) + if ((error = ffs_flushfiles(oldmnt, flags, td)) != 0 && + error != EBUSY) + return (error); + if ((error = softdep_flushworklist(oldmnt, &depcount, td)) != 0) + return (error); + /* + * Normally ffs_flushfiles() will shut down quotas. If + * it finds quotas still referenced it will return EBUSY. + * When we are running with journaled soft updates we may + * have to flush the worklist (softdep_flushworklist()) to + * clear the quota references. Thus we cannot exit this loop + * until quotas are shut down and the worklist is clear. + */ + if ((oldmnt->mnt_flag & MNT_QUOTA) == 0 && depcount == 0) break; - if ((error = softdep_flushworklist(oldmnt, &depcount, td)) != 0 || - depcount == 0) - break; } /* * If we are unmounting then it is an error to fail. If we @@ -1909,27 +1919,24 @@ retry_flush: if (loopcnt == 0) { if (oldmnt->mnt_kern_flag & MNTK_UNMOUNT) panic("softdep_flushfiles: looping"); - error = EBUSY; + return (EBUSY); } - if (!error) - error = softdep_waitidle(oldmnt); - if (!error) { - if (oldmnt->mnt_kern_flag & MNTK_UNMOUNT) { - retry = 0; - MNT_ILOCK(oldmnt); - KASSERT((oldmnt->mnt_kern_flag & MNTK_NOINSMNTQ) != 0, - ("softdep_flushfiles: !MNTK_NOINSMNTQ")); - if (oldmnt->mnt_nvnodelistsize > 0) { - if (--retry_flush_count > 0) { - retry = 1; - loopcnt = 3; - } else - error = EBUSY; - } - MNT_IUNLOCK(oldmnt); - if (retry) - goto retry_flush; + if ((error = softdep_waitidle(oldmnt)) == 0 && + (oldmnt->mnt_kern_flag & MNTK_UNMOUNT) != 0) { + retry = 0; + MNT_ILOCK(oldmnt); + KASSERT((oldmnt->mnt_kern_flag & MNTK_NOINSMNTQ) != 0, + ("softdep_flushfiles: !MNTK_NOINSMNTQ")); + if (oldmnt->mnt_nvnodelistsize > 0) { + if (--retry_flush_count > 0) { + retry = 1; + loopcnt = 3; + } else + error = EBUSY; } + MNT_IUNLOCK(oldmnt); + if (retry) + goto retry_flush; } return (error); } @@ -2827,7 +2834,12 @@ softdep_prealloc(vp, waitok) { struct ufsmount *ump; - if (DOINGSUJ(vp) == 0) + /* + * Nothing to do if we are not running journaled soft updates. + * If we currently hold the snapshot lock, we must avoid handling + * other resources that could cause deadlock. + */ + if (DOINGSUJ(vp) == 0 || IS_SNAPSHOT(VTOI(vp))) return (0); ump = VFSTOUFS(vp->v_mount); ACQUIRE_LOCK(&lk); @@ -2873,7 +2885,12 @@ softdep_prelink(dvp, vp) ump = VFSTOUFS(dvp->v_mount); mtx_assert(&lk, MA_OWNED); - if (journal_space(ump, 0)) + /* + * Nothing to do if we have sufficient journal space. + * If we currently hold the snapshot lock, we must avoid + * handling other resources that could cause deadlock. + */ + if (journal_space(ump, 0) || (vp && IS_SNAPSHOT(VTOI(vp)))) return; stat_journal_low++; FREE_LOCK(&lk); @@ -4304,11 +4321,15 @@ inodedep_lookup_ip(ip) struct inode *ip; { struct inodedep *inodedep; + int dflags; KASSERT(ip->i_nlink >= ip->i_effnlink, ("inodedep_lookup_ip: bad delta")); - (void) inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, - DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, + &inodedep); inodedep->id_nlinkdelta = ip->i_nlink - ip->i_effnlink; return (inodedep); @@ -4696,7 +4717,7 @@ softdep_setup_inomapdep(bp, ip, newinum, mode) * the cylinder group map from which it was allocated. */ ACQUIRE_LOCK(&lk); - if ((inodedep_lookup(mp, newinum, DEPALLOC|NODELAY, &inodedep))) + if ((inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep))) panic("softdep_setup_inomapdep: dependency %p for new" "inode already exists", inodedep); bmsafemap = bmsafemap_lookup(mp, bp, ino_to_cg(fs, newinum)); @@ -5437,6 +5458,7 @@ softdep_setup_allocindir_page(ip, lbn, bp, ptrno, struct allocindir *aip; struct pagedep *pagedep; struct mount *mp; + int dflags; if (lbn != nbp->b_lblkno) panic("softdep_setup_allocindir_page: lbn %jd != lblkno %jd", @@ -5444,7 +5466,10 @@ softdep_setup_allocindir_page(ip, lbn, bp, ptrno, ASSERT_VOP_LOCKED(ITOV(ip), "softdep_setup_allocindir_page"); mp = UFSTOVFS(ip->i_ump); aip = newallocindir(ip, ptrno, newblkno, oldblkno, lbn); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); /* * If we are allocating a directory page, then we must * allocate an associated pagedep to track additions and @@ -5474,11 +5499,15 @@ softdep_setup_allocindir_meta(nbp, ip, bp, ptrno, struct inodedep *inodedep; struct allocindir *aip; ufs_lbn_t lbn; + int dflags; lbn = nbp->b_lblkno; ASSERT_VOP_LOCKED(ITOV(ip), "softdep_setup_allocindir_meta"); aip = newallocindir(ip, ptrno, newblkno, 0, lbn); - inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, &inodedep); WORKLIST_INSERT(&nbp->b_dep, &aip->ai_block.nb_list); if (setup_allocindir_phase2(bp, ip, inodedep, aip, lbn)) panic("softdep_setup_allocindir_meta: Block already existed"); @@ -6085,11 +6114,7 @@ softdep_journal_freeblocks(ip, cred, length, flags struct mount *mp; ufs2_daddr_t extblocks, datablocks; ufs_lbn_t tmpval, lbn, lastlbn; - int frags; - int lastoff, iboff; - int allocblock; - int error, i; - int needj; + int frags, lastoff, iboff, allocblock, needj, dflags, error, i; fs = ip->i_fs; mp = UFSTOVFS(ip->i_ump); @@ -6107,7 +6132,10 @@ softdep_journal_freeblocks(ip, cred, length, flags * we don't need to journal the block frees. The canceled journals * for the allocations will suffice. */ - inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & (UNLINKED | DEPCOMPLETE)) == UNLINKED && length == 0) needj = 0; @@ -6232,7 +6260,7 @@ softdep_journal_freeblocks(ip, cred, length, flags *((struct ufs2_dinode *)bp->b_data + ino_to_fsbo(fs, ip->i_number)) = *ip->i_din2; ACQUIRE_LOCK(&lk); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & IOSTARTED) != 0) panic("softdep_setup_freeblocks: inode busy"); /* @@ -6310,7 +6338,7 @@ softdep_journal_freeblocks(ip, cred, length, flags } ACQUIRE_LOCK(&lk); - inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + inodedep_lookup(mp, ip->i_number, dflags, &inodedep); TAILQ_INSERT_TAIL(&inodedep->id_freeblklst, freeblks, fb_next); freeblks->fb_state |= DEPCOMPLETE | ONDEPLIST; /* @@ -6398,7 +6426,7 @@ softdep_setup_freeblocks(ip, length, flags) struct fs *fs; ufs2_daddr_t extblocks, datablocks; struct mount *mp; - int i, delay, error; + int i, delay, error, dflags; ufs_lbn_t tmpval; ufs_lbn_t lbn; @@ -6463,7 +6491,10 @@ softdep_setup_freeblocks(ip, length, flags) * Find and eliminate any inode dependencies. */ ACQUIRE_LOCK(&lk); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & IOSTARTED) != 0) panic("softdep_setup_freeblocks: inode busy"); /* @@ -8026,7 +8057,7 @@ softdep_setup_directory_add(bp, dp, diroffset, new dap->da_pagedep = pagedep; LIST_INSERT_HEAD(&pagedep->pd_diraddhd[DIRADDHASH(offset)], dap, da_pdlist); - inodedep_lookup(mp, newinum, DEPALLOC, &inodedep); + inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep); /* * If we're journaling, link the diradd into the jaddref so it * may be completed after the journal entry is written. Otherwise, @@ -8628,8 +8659,7 @@ newdirrem(bp, dp, ip, isrmdir, prevdirremp) * the number of freefile and freeblks structures. */ ACQUIRE_LOCK(&lk); - if (!(ip->i_flags & SF_SNAPSHOT) && - dep_current[D_DIRREM] > max_softdeps / 2) + if (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2) (void) request_cleanup(ITOV(dp)->v_mount, FLUSH_BLOCKS); FREE_LOCK(&lk); dirrem = malloc(sizeof(struct dirrem), @@ -8863,11 +8893,11 @@ softdep_setup_directory_change(bp, dp, ip, newinum /* * Lookup the jaddref for this journal entry. We must finish * initializing it and make the diradd write dependent on it. - * If we're not journaling Put it on the id_bufwait list if the inode - * is not yet written. If it is written, do the post-inode write - * processing to put it on the id_pendinghd list. + * If we're not journaling, put it on the id_bufwait list if the + * inode is not yet written. If it is written, do the post-inode + * write processing to put it on the id_pendinghd list. */ - inodedep_lookup(mp, newinum, DEPALLOC, &inodedep); + inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep); if (MOUNTEDSUJ(mp)) { jaddref = (struct jaddref *)TAILQ_LAST(&inodedep->id_inoreflst, inoreflst); @@ -8909,9 +8939,13 @@ softdep_change_linkcnt(ip) struct inode *ip; /* the inode with the increased link count */ { struct inodedep *inodedep; + int dflags; ACQUIRE_LOCK(&lk); - inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, &inodedep); if (ip->i_nlink < ip->i_effnlink) panic("softdep_change_linkcnt: bad delta"); inodedep->id_nlinkdelta = ip->i_nlink - ip->i_effnlink; @@ -12541,22 +12575,25 @@ softdep_request_cleanup(fs, vp, cred, resource) ufs2_daddr_t needed; int error; - mp = vp->v_mount; - ump = VFSTOUFS(mp); - mtx_assert(UFS_MTX(ump), MA_OWNED); - if (resource == FLUSH_BLOCKS_WAIT) - stat_cleanup_blkrequests += 1; - else - stat_cleanup_inorequests += 1; - /* * If we are being called because of a process doing a * copy-on-write, then it is not safe to process any * worklist items as we will recurse into the copyonwrite * routine. This will result in an incoherent snapshot. + * If the vnode that we hold is a snapshot, we must avoid + * handling other resources that could cause deadlock. */ - if (curthread->td_pflags & TDP_COWINPROGRESS) + if ((curthread->td_pflags & TDP_COWINPROGRESS) || IS_SNAPSHOT(VTOI(vp))) return (0); + + if (resource == FLUSH_BLOCKS_WAIT) + stat_cleanup_blkrequests += 1; + else + stat_cleanup_inorequests += 1; + + mp = vp->v_mount; + ump = VFSTOUFS(mp); + mtx_assert(UFS_MTX(ump), MA_OWNED); UFS_UNLOCK(ump); error = ffs_update(vp, 1); if (error != 0) { Index: /usr/src/sys/ufs/ffs/ffs_vnops.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_vnops.c (revision 231557) +++ /usr/src/sys/ufs/ffs/ffs_vnops.c (working copy) @@ -216,8 +216,10 @@ ffs_syncvnode(struct vnode *vp, int waitfor) struct buf *bp; struct buf *nbp; ufs_lbn_t lbn; - int error, wait, passes; + int error, wait, passes, noupdate; + noupdate = waitfor & NO_INO_UPDT; + waitfor &= ~NO_INO_UPDT; ip = VTOI(vp); ip->i_flag &= ~IN_NEEDSYNC; bo = &vp->v_bufobj; @@ -300,7 +302,10 @@ next: } if (waitfor != MNT_WAIT) { BO_UNLOCK(bo); - return (ffs_update(vp, waitfor)); + if (noupdate) + return (0); + else + return (ffs_update(vp, waitfor)); } /* Drain IO to see if we're done. */ bufobj_wwait(bo, 0, 0); @@ -317,7 +322,7 @@ next: */ if (bo->bo_dirty.bv_cnt > 0) { /* Write the inode after sync passes to flush deps. */ - if (wait && DOINGSOFTDEP(vp)) { + if (wait && DOINGSOFTDEP(vp) && noupdate == 0) { BO_UNLOCK(bo); ffs_update(vp, MNT_WAIT); BO_LOCK(bo); @@ -332,7 +337,9 @@ next: #endif } BO_UNLOCK(bo); - error = ffs_update(vp, MNT_WAIT); + error = 0; + if (noupdate == 0) + error = ffs_update(vp, MNT_WAIT); if (DOINGSUJ(vp)) softdep_journal_fsync(VTOI(vp)); return (error); Index: /usr/src/sys/ufs/ffs/ffs_extern.h =================================================================== --- /usr/src/sys/ufs/ffs/ffs_extern.h (revision 231557) +++ /usr/src/sys/ufs/ffs/ffs_extern.h (working copy) @@ -167,6 +167,13 @@ void softdep_freework(struct workhead *); #define FLUSH_INODES_WAIT 2 #define FLUSH_BLOCKS 3 #define FLUSH_BLOCKS_WAIT 4 +/* + * Flag to ffs_syncinode() to request flushing of data only, + * but skip the ffs_update() on the inode itself. Used to avoid + * deadlock when flushing snapshot inodes while holding snaplk. + * Avoid bit conflicts with MNT_WAIT values in sys/mount.h + */ +#define NO_INO_UPDT 0x10 int ffs_rdonly(struct inode *); Index: /usr/src/sys/ufs/ffs/ffs_vfsops.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_vfsops.c (revision 231557) +++ /usr/src/sys/ufs/ffs/ffs_vfsops.c (working copy) @@ -405,6 +405,8 @@ ffs_mount(struct mount *mp) vn_finished_write(mp); return (error); } + if (MOUNTEDSOFTDEP(mp)) + devvp->v_rdev->si_mountpt = mp; if (fs->fs_snapinum[0] != 0) ffs_snapshot_mount(mp); vn_finished_write(mp); @@ -1048,6 +1050,8 @@ ffs_mountfs(devvp, mp, td) ffs_flushfiles(mp, FORCECLOSE, td); goto out; } + if (MOUNTEDSOFTDEP(mp)) + devvp->v_rdev->si_mountpt = mp; if (fs->fs_snapinum[0] != 0) ffs_snapshot_mount(mp); fs->fs_fmod = 1; @@ -1293,6 +1297,7 @@ ffs_unmount(mp, mntflags) g_vfs_close(ump->um_cp); g_topology_unlock(); PICKUP_GIANT(); + ump->um_devvp->v_rdev->si_mountpt = NULL; vrele(ump->um_devvp); dev_rel(ump->um_dev); mtx_destroy(UFS_MTX(ump)); @@ -1341,13 +1346,17 @@ ffs_flushfiles(mp, flags, td) ump = VFSTOUFS(mp); #ifdef QUOTA if (mp->mnt_flag & MNT_QUOTA) { - int i; - error = vflush(mp, 0, SKIPSYSTEM|flags, td); - if (error) + int i, allerror; + + if ((error = vflush(mp, 0, SKIPSYSTEM|flags, td)) != 0) return (error); for (i = 0; i < MAXQUOTAS; i++) { - quotaoff(td, mp, i); + allerror = quotaoff(td, mp, i); + if (allerror != 0 && (error == 0 || error == EBUSY)) + error = allerror; } + if (error) + return (error); /* * Here we fall through to vflush again to ensure * that we have gotten rid of all the system vnodes. Index: /usr/src/sys/ufs/ffs/ffs_snapshot.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_snapshot.c (revision 231557) +++ /usr/src/sys/ufs/ffs/ffs_snapshot.c (working copy) @@ -225,18 +225,10 @@ ffs_snapshot(mp, snapfile) ump = VFSTOUFS(mp); fs = ump->um_fs; sn = NULL; - /* - * At the moment, journaled soft updates cannot support - * taking snapshots. - */ - if (MOUNTEDSUJ(mp)) { - vfs_mount_error(mp, "%s: Snapshots are not yet supported when " - "running with journaled soft updates", fs->fs_fsmnt); - return (EOPNOTSUPP); - } MNT_ILOCK(mp); flag = mp->mnt_flag; MNT_IUNLOCK(mp); + /* * Need to serialize access to snapshot code per filesystem. */ @@ -1902,7 +1894,7 @@ retry: bawrite(cbp); if ((vtype == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); continue; } /* @@ -1913,7 +1905,7 @@ retry: bawrite(cbp); if ((vtype == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); break; } savedcbp = cbp; @@ -1931,7 +1923,7 @@ retry: bawrite(savedcbp); if ((vtype == VDIR || dopersistence) && VTOI(vp)->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); } /* * If we have been unable to allocate a block in which to do @@ -2395,7 +2387,7 @@ ffs_copyonwrite(devvp, bp) bawrite(cbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; continue; @@ -2408,7 +2400,7 @@ ffs_copyonwrite(devvp, bp) bawrite(cbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; break; @@ -2428,7 +2420,7 @@ ffs_copyonwrite(devvp, bp) bawrite(savedcbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && VTOI(vp)->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; } @@ -2478,7 +2470,7 @@ ffs_sync_snap(mp, waitfor) } TAILQ_FOREACH(ip, &sn->sn_head, i_nextsnap) { vp = ITOV(ip); - ffs_syncvnode(vp, waitfor); + ffs_syncvnode(vp, waitfor|NO_INO_UPDT); } lockmgr(&sn->sn_lock, LK_RELEASE, NULL); } Index: /usr/src/sys/kern/vfs_default.c =================================================================== --- /usr/src/sys/kern/vfs_default.c (revision 231557) +++ /usr/src/sys/kern/vfs_default.c (working copy) @@ -608,6 +608,8 @@ vop_stdbmap(ap) return (0); } +int softdep_flushworklist(struct mount *, int *, struct thread *); + int vop_stdfsync(ap) struct vop_fsync_args /* { @@ -621,7 +623,7 @@ vop_stdfsync(ap) struct buf *bp; struct bufobj *bo; struct buf *nbp; - int error = 0; + int count, error = 0; int maxretry = 1000; /* large, arbitrarily chosen */ bo = &vp->v_bufobj; @@ -668,6 +670,9 @@ loop2: * retry if dirty blocks still exist. */ if (ap->a_waitfor == MNT_WAIT) { + if (vp->v_type == VCHR && vp->v_rdev->si_mountpt != NULL) + softdep_flushworklist(vp->v_rdev->si_mountpt, + &count, ap->a_td); bufobj_wwait(bo, 0, 0); if (bo->bo_dirty.bv_cnt > 0) { /*