GENERIC HEAD from 2010-05-17 03:51:57 UTC, r208190, vmcore.19 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0 r208190: Mon May 17 15:45:38 CEST 2010 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.49-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3973947392 (3789 MB) : Trying to mount root from ufs:/dev/ad4s2a Entropy harvesting: interrupts ethernet point_to_point kickstart. /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 378778 free (1970 frags, 47101 blocks, 0.2% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 48930204 free (540 frags, 6116208 blocks, 0.0% fragmentation) /dev/ad4s2f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2f: clean, 2102495 free (224719 frags, 234722 blocks, 4.5% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 30479808 free (124320 frags, 3794436 blocks, 0.2% fragmentation) Setting hostname: x4.osted.lan. lock order reversal: 1st 0xffffff0004326cf0 ufs (ufs) @ kern/vfs_lookup.c:502 2nd 0xffffff804fbb1bf0 bufwait (bufwait) @ ufs/ffs/ffs_softdep.c:11363 3rd 0xffffff00043af800 ufs (ufs) @ kern/vfs_subr.c:2103 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd32 ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b vfs_hash_get() at vfs_hash_get+0xd5 ffs_vgetf() at ffs_vgetf+0x49 softdep_sync_metadata() at softdep_sync_metadata+0x5d9 ffs_syncvnode() at ffs_syncvnode+0x213 ffs_truncate() at ffs_truncate+0x55a ufs_direnter() at ufs_direnter+0x493 ufs_makeinode() at ufs_makeinode+0x291 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800733b6c, rsp = 0x7fffffffeb48, rbp = 0 --- re0: link state changed to DOWN Starting Network: lo0 re0. add net default: gateway 192.168.1.1 lock order reversal: 1st 0xffffff804fbcc478 bufwait (bufwait) @ kern/vfs_bio.c:2579 2nd 0xffffff0004556000 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:283 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x677 ufs_mkdir() at ufs_mkdir+0x444 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800730b2c, rsp = 0x7fffffffec88, rbp = 0x7fffffffef66 --- Recovering vi editor sessions:. Expensive timeout(9) function: 0xffffffff8048f590(0xffffffff80ec5620) 0.011252547 s Configuring syscons: keymap blanktime. Local package initialization:lock order reversal: 1st 0xffffff00042fc310 ufs (ufs) @ kern/vfs_mount.c:1201 2nd 0xffffff000430e310 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1597 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd32 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 softdep_flushworklist() at softdep_flushworklist+0x61 ffs_sync() at ffs_sync+0x43a dounmount() at dounmount+0x2ca unmount() at unmount+0x27e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a1fdc, rsp = 0x7fffffffe568, rbp = 0x800c07fb0 --- ** /dev/ad4s2e ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 391495 files, 1846871 used, 48930204 free (540 frags, 6116208 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** usage: kill [-s signal_name] pid ... kill -l [exit_status] kill -signal_name pid ... kill -signal_number pid ... fsck -y /tmp. Mon May 17 19:17:25 CEST 2010 FreeBSD/amd64 (x4.osted.lan) (console) login: Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex struct mount mtx (struct mount mtx) r = 0 (0xffffff00041c3ab0) locked @ ufs/ffs/ffs_softdep.c:2433 exclusive sleep mutex Softdep Lock (Softdep Lock) r = 0 (0xffffffff80e61360) locked @ ufs/ffs/ffs_softdep.c:1539 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2d2 trap() at trap+0x2ce calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff807d3bd2, rsp = 0xffffff80756db870, rbp = 0xffffff80756db890 --- journal_suspend() at journal_suspend+0x52 softdep_setup_unlink() at softdep_setup_unlink+0x55 ufs_dirremove() at ufs_dirremove+0x24e ufs_remove() at ufs_remove+0x92 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7 kern_unlinkat() at kern_unlinkat+0x2cb syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084343c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x54 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff807d3bd2 stack pointer = 0x28:0xffffff80756db870 frame pointer = 0x28:0xffffff80756db890 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1125 (creat) [ thread pid 1125 tid 100145 ] Stopped at journal_suspend+0x52: movl $0x1,0x54(%r12) db> run pho db:0:pho> bt Tracing pid 1125 tid 100145 td 0xffffff000427e780 journal_suspend() at journal_suspend+0x52 softdep_setup_unlink() at softdep_setup_unlink+0x55 ufs_dirremove() at ufs_dirremove+0x24e ufs_remove() at ufs_remove+0x92 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7 kern_unlinkat() at kern_unlinkat+0x2cb syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084343c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x4f4600 curthread = 0xffffff00024f4b40: pid 12 "swi4: clock" curpcb = 0xffffff800005fd40 fpcurthread = none idlethread = 0xffffff00024ff780: pid 11 "idle: cpu0" curpmap = 0 tssp = 0xffffffff80ea3c80 commontssp = 0xffffffff80ea3c80 rsp0 = 0xffffff800005fd40 gs32p = 0xffffffff80ea2ab8 ldt = 0xffffffff80ea2af8 tss = 0xffffffff80ea2ae8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f3cb600 curthread = 0xffffff0002505b40: pid 3 "g_up" curpcb = 0xffffff8000078d40 fpcurthread = none idlethread = 0xffffff00024f3000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff80ea3ce8 commontssp = 0xffffffff80ea3ce8 rsp0 = 0xffffff8000078d40 gs32p = 0xffffffff80ea2b20 ldt = 0xffffffff80ea2b60 tss = 0xffffffff80ea2b50 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f3d2600 curthread = 0xffffff000427e780: pid 1125 "creat" curpcb = 0xffffff80756dbd40 fpcurthread = none idlethread = 0xffffff00024f33c0: pid 11 "idle: cpu2" curpmap = 0 tssp = 0xffffffff80ea3d50 commontssp = 0xffffffff80ea3d50 rsp0 = 0xffffff80756dbd40 gs32p = 0xffffffff80ea2b88 ldt = 0xffffffff80ea2bc8 tss = 0xffffffff80ea2bb8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f3d9600 curthread = 0xffffff00024f3780: pid 11 "idle: cpu3" curpcb = 0xffffff8000041d40 fpcurthread = none idlethread = 0xffffff00024f3780: pid 11 "idle: cpu3" curpmap = 0 tssp = 0xffffffff80ea3db8 commontssp = 0xffffffff80ea3db8 rsp0 = 0xffffff8000041d40 gs32p = 0xffffffff80ea2bf0 ldt = 0xffffffff80ea2c30 tss = 0xffffffff80ea2c20 spin locks held: db:0:allpcpu> show alllocks Process 1128 (creat) thread 0xffffff000418b3c0 (100090) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff804ff97fd0) locked @ kern/vfs_bio.c:2579 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015ae1310) locked @ kern/vfs_subr.c:2103 exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff80500fa328) locked @ ufs/ffs/ffs_softdep.c:11363 exclusive lockmgr ufs (ufs) r = 0 (0xffffff00155cecf0) locked @ ufs/ffs/ffs_vfsops.c:1512 exclusive lockmgr ufs (ufs) r = 0 (0xffffff00175ca098) locked @ kern/vfs_lookup.c:502 Process 1127 (creat) thread 0xffffff0004884780 (100151) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015897098) locked @ kern/vfs_subr.c:2103 exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff804fd0ec28) locked @ ufs/ffs/ffs_softdep.c:11363 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0017478800) locked @ ufs/ffs/ffs_vfsops.c:1512 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0004841098) locked @ kern/vfs_lookup.c:502 Process 1126 (creat) thread 0xffffff000432e780 (100112) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015a60098) locked @ ufs/ffs/ffs_vfsops.c:1512 exclusive lockmgr ufs (ufs) r = 0 (0xffffff00151e1800) locked @ kern/vfs_lookup.c:502 Process 1125 (creat) thread 0xffffff000427e780 (100145) exclusive sleep mutex struct mount mtx (struct mount mtx) r = 0 (0xffffff00041c3ab0) locked @ ufs/ffs/ffs_softdep.c:2433 exclusive sleep mutex Softdep Lock (Softdep Lock) r = 0 (0xffffffff80e61360) locked @ ufs/ffs/ffs_softdep.c:1539 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015a6a098) locked @ ufs/ffs/ffs_vfsops.c:1512 exclusive lockmgr ufs (ufs) r = 0 (0xffffff00175ca310) locked @ kern/vfs_lookup.c:502 Process 1124 (creat) thread 0xffffff0004271780 (100107) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015797310) locked @ ufs/ffs/ffs_vfsops.c:1512 exclusive lockmgr ufs (ufs) r = 0 (0xffffff001562f800) locked @ kern/vfs_lookup.c:502 Process 1030 (sshd) thread 0xffffff000430b780 (100103) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00045493a0) locked @ kern/uipc_sockbuf.c:148 Process 714 (syslogd) thread 0xffffff00042833c0 (100117) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff804fbcd058) locked @ kern/vfs_bio.c:2579 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0004550310) locked @ kern/vfs_vnops.c:612 Process 17 (bufdaemon) thread 0xffffff000418d3c0 (100072) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff804fe8f958) locked @ kern/vfs_bio.c:2257 exclusive sleep mutex buf queue lock (buf queue lock) r = 0 (0xffffffff80e51c40) locked @ kern/vfs_bio.c:2327 Process 12 (intr) thread 0xffffff00024f4b40 (100009) exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff80c903e0) locked @ kern/kern_mutex.c:147 db:0:alllocks> show lockedvnods Locked vnodes 0xffffff0004550278: tag ufs, type VREG usecount 1, writecount 1, refcount 4 mountedhere 0 flags () v_object 0xffffff000459d6c0 ref 0 pages 5 lock type ufs: EXCL by thread 0xffffff00042833c0 (pid 714) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff806514e9 at vn_write+0x149 #5 0xffffffff805f7065 at dofilewrite+0x85 #6 0xffffffff805f8880 at kern_writev+0x60 #7 0xffffffff805f8901 at writev+0x41 #8 0xffffffff808ab2f8 at syscall+0x118 #9 0xffffffff80890911 at Xfast_syscall+0xe1 ino 235556, on dev ad4s2d 0xffffff001562f768: tag ufs, type VDIR usecount 2, writecount 0, refcount 8 mountedhere 0 flags () v_object 0xffffff00186030d8 ref 0 pages 14 lock type ufs: EXCL by thread 0xffffff0004271780 (pid 1124) #0 0xffffffff805963ff at __lockmgr_args+0x4ef #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff80637666 at lookup+0xe6 #5 0xffffffff80638901 at namei+0x481 #6 0xffffffff8064f821 at vn_open_cred+0xc1 #7 0xffffffff8064cd99 at kern_openat+0x179 #8 0xffffffff808ab2f8 at syscall+0x118 #9 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12906552, on dev ad4s2e 0xffffff0004841000: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xffffff00042fa288 ref 0 pages 1 lock type ufs: EXCL by thread 0xffffff0004884780 (pid 1127) #0 0xffffffff805963ff at __lockmgr_args+0x4ef #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff80637666 at lookup+0xe6 #5 0xffffffff80638901 at namei+0x481 #6 0xffffffff8064f821 at vn_open_cred+0xc1 #7 0xffffffff8064cd99 at kern_openat+0x179 #8 0xffffffff808ab2f8 at syscall+0x118 #9 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12930104, on dev ad4s2e 0xffffff00151e1768: tag ufs, type VDIR usecount 2, writecount 0, refcount 9 mountedhere 0 flags () v_object 0xffffff000452f870 ref 0 pages 20 lock type ufs: EXCL by thread 0xffffff000432e780 (pid 1126) #0 0xffffffff805963ff at __lockmgr_args+0x4ef #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff80637666 at lookup+0xe6 #5 0xffffffff80638901 at namei+0x481 #6 0xffffffff8064f821 at vn_open_cred+0xc1 #7 0xffffffff8064cd99 at kern_openat+0x179 #8 0xffffffff808ab2f8 at syscall+0x118 #9 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12930107, on dev ad4s2e 0xffffff00175ca000: tag ufs, type VDIR usecount 2, writecount 0, refcount 17 mountedhere 0 flags () v_object 0xffffff0018b51ca8 ref 0 pages 53 lock type ufs: EXCL by thread 0xffffff000418b3c0 (pid 1128) #0 0xffffffff805963ff at __lockmgr_args+0x4ef #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff80637666 at lookup+0xe6 #5 0xffffffff80638901 at namei+0x481 #6 0xffffffff8064f821 at vn_open_cred+0xc1 #7 0xffffffff8064cd99 at kern_openat+0x179 #8 0xffffffff808ab2f8 at syscall+0x118 #9 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12930109, on dev ad4s2e 0xffffff00175ca278: tag ufs, type VDIR usecount 2, writecount 0, refcount 52 mountedhere 0 flags () v_object 0xffffff000459da20 ref 0 pages 192 lock type ufs: EXCL by thread 0xffffff000427e780 (pid 1125) #0 0xffffffff805963ff at __lockmgr_args+0x4ef #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff80637666 at lookup+0xe6 #5 0xffffffff80638901 at namei+0x481 #6 0xffffffff80648088 at kern_unlinkat+0x128 #7 0xffffffff808ab2f8 at syscall+0x118 #8 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12930113, on dev ad4s2e 0xffffff0015ae1278: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () v_object 0xffffff0018f32000 ref 0 pages 0 lock type ufs: EXCL by thread 0xffffff000418b3c0 (pid 1128) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff8064433b at vget+0x7b #5 0xffffffff806368d5 at vfs_hash_get+0xd5 #6 0xffffffff807e7d59 at ffs_vgetf+0x49 #7 0xffffffff807e5099 at softdep_sync_metadata+0x5d9 #8 0xffffffff807ec7b3 at ffs_syncvnode+0x213 #9 0xffffffff807df77d at softdep_prelink+0x13d #10 0xffffffff807dfd34 at softdep_setup_create+0x94 #11 0xffffffff807f9817 at ufs_makeinode+0x567 #12 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #13 0xffffffff8064fbcb at vn_open_cred+0x46b #14 0xffffffff8064cd99 at kern_openat+0x179 #15 0xffffffff808ab2f8 at syscall+0x118 #16 0xffffffff80890911 at Xfast_syscall+0xe1 ino 5811734, on dev ad4s2e 0xffffff0015897000: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () v_object 0xffffff001d170510 ref 0 pages 0 lock type ufs: EXCL by thread 0xffffff0004884780 (pid 1127) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807ecb3c at ffs_lock+0x9c #2 0xffffffff808fd53b at VOP_LOCK1_APV+0x9b #3 0xffffffff8064ffb7 at _vn_lock+0x57 #4 0xffffffff8064433b at vget+0x7b #5 0xffffffff806368d5 at vfs_hash_get+0xd5 #6 0xffffffff807e7d59 at ffs_vgetf+0x49 #7 0xffffffff807e5099 at softdep_sync_metadata+0x5d9 #8 0xffffffff807ec7b3 at ffs_syncvnode+0x213 #9 0xffffffff807df77d at softdep_prelink+0x13d #10 0xffffffff807dfd34 at softdep_setup_create+0x94 #11 0xffffffff807f9817 at ufs_makeinode+0x567 #12 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #13 0xffffffff8064fbcb at vn_open_cred+0x46b #14 0xffffffff8064cd99 at kern_openat+0x179 #15 0xffffffff808ab2f8 at syscall+0x118 #16 0xffffffff80890911 at Xfast_syscall+0xe1 ino 5811735, on dev ad4s2e 0xffffff0015a6a000: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff000427e780 (pid 1125) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807e7e97 at ffs_vgetf+0x187 #2 0xffffffff807f30c0 at ufs_lookup_ino+0xd60 #3 0xffffffff808fc0af at VOP_CACHEDLOOKUP_APV+0xaf #4 0xffffffff80630c10 at vfs_cache_lookup+0xf0 #5 0xffffffff808fe417 at VOP_LOOKUP_APV+0xb7 #6 0xffffffff80637953 at lookup+0x3d3 #7 0xffffffff80638901 at namei+0x481 #8 0xffffffff80648088 at kern_unlinkat+0x128 #9 0xffffffff808ab2f8 at syscall+0x118 #10 0xffffffff80890911 at Xfast_syscall+0xe1 ino 12968401, on dev ad4s2e 0xffffff00155cec58: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff000418b3c0 (pid 1128) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807e7e97 at ffs_vgetf+0x187 #2 0xffffffff807c4b17 at ffs_valloc+0x177 #3 0xffffffff807f935f at ufs_makeinode+0xaf #4 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #5 0xffffffff8064fbcb at vn_open_cred+0x46b #6 0xffffffff8064cd99 at kern_openat+0x179 #7 0xffffffff808ab2f8 at syscall+0x118 #8 0xffffffff80890911 at Xfast_syscall+0xe1 ino 5811742, on dev ad4s2e 0xffffff0017478768: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff0004884780 (pid 1127) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807e7e97 at ffs_vgetf+0x187 #2 0xffffffff807c4b17 at ffs_valloc+0x177 #3 0xffffffff807f935f at ufs_makeinode+0xaf #4 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #5 0xffffffff8064fbcb at vn_open_cred+0x46b #6 0xffffffff8064cd99 at kern_openat+0x179 #7 0xffffffff808ab2f8 at syscall+0x118 #8 0xffffffff80890911 at Xfast_syscall+0xe1 ino 5811743, on dev ad4s2e 0xffffff0015a60000: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff000432e780 (pid 1126) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807e7e97 at ffs_vgetf+0x187 #2 0xffffffff807c4b17 at ffs_valloc+0x177 #3 0xffffffff807f935f at ufs_makeinode+0xaf #4 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #5 0xffffffff8064fbcb at vn_open_cred+0x46b #6 0xffffffff8064cd99 at kern_openat+0x179 #7 0xffffffff808ab2f8 at syscall+0x118 #8 0xffffffff80890911 at Xfast_syscall+0xe1 ino 5811744, on dev ad4s2e 0xffffff0015797278: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff0004271780 (pid 1124) #0 0xffffffff80596687 at __lockmgr_args+0x777 #1 0xffffffff807e7e97 at ffs_vgetf+0x187 #2 0xffffffff807c4b17 at ffs_valloc+0x177 #3 0xffffffff807f935f at ufs_makeinode+0xaf #4 0xffffffff808fe2a3 at VOP_CREATE_APV+0xb3 #5 0xffffffff8064fbcb at vn_open_cred+0x46b #6 0xffffffff8064cd99 at kern_openat+0x179 #7 0xffffffff808ab2f8 at syscall+0x118 #8 0xffffffff80890911 at Xfast_syscall+0xe1 ino 13094834, on dev ad4s2e db:0:lockedvnods> show mount 0xffffff00041c2ab0 /dev/ad4s2a on / (ufs) 0xffffff00041c3000 devfs on /dev (devfs) 0xffffff000433e390 /dev/ad4s2f on /usr (ufs) 0xffffff000433e000 /dev/ad4s2d on /var (ufs) 0xffffff00041c3ab0 /dev/ad4s2e on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 1128 1123 1112 1001 D+ biowr 0xffffff804ff97f38 creat 1127 1123 1112 1001 D+ getblk 0xffffff804ff97fd0 creat 1126 1123 1112 1001 D+ bo_wwait 0xffffff00151e1980 creat 1125 1123 1112 1001 R+ CPU 2 creat 1124 1123 1112 1001 D+ getbuf 0xffffff804fcf1710 creat 1123 1122 1112 1001 S+ wait 0xffffff001857e000 creat 1122 1121 1112 1001 S+ nanslp 0xffffffff80c91068 creat 1121 1120 1112 1001 S+ wait 0xffffff000431f8c0 run 1120 1119 1112 1001 S+ wait 0xffffff001857e8c0 run 1119 1112 1112 1001 S+ nanslp 0xffffffff80c91068 run 1112 1033 1112 1001 S+ wait 0xffffff000421c8c0 sh 1110 1 1110 0 Ss+ ttyin 0xffffff00040298a8 getty 1109 1 1109 0 Ss+ ttyin 0xffffff0004029ca8 getty 1108 1 1108 0 Ss+ ttyin 0xffffff000402a0a8 getty 1107 1 1107 0 Ss+ ttyin 0xffffff000402a4a8 getty 1106 1 1106 0 Ss+ ttyin 0xffffff000402a8a8 getty 1105 1 1105 0 Ss+ ttyin 0xffffff000402aca8 getty 1104 1 1104 0 Ss+ ttyin 0xffffff000402c0a8 getty 1103 1 1103 0 Ss+ ttyin 0xffffff000402c4a8 getty 1102 1 1102 0 Ss+ ttyin 0xffffff0004019ca8 getty 1076 1 1076 0 Ss select 0xffffff000426d240 inetd 1033 1032 1033 1001 Ss+ wait 0xffffff000485d000 bash 1032 1030 1030 1001 S select 0xffffff000439a740 sshd 1030 999 1030 0 Ss sbwait 0xffffff00045493ec sshd 1017 1 1017 0 Ss nanslp 0xffffffff80c91068 cron 1010 1 1010 25 Ss pause 0xffffff0004353500 sendmail 1006 1 1006 0 Ss select 0xffffff000426b740 sendmail 999 1 999 0 Ss select 0xffffff00041c9640 sshd 962 1 962 0 Ss select 0xffffff00041827c0 ntpd 847 1 847 0 Ss select 0xffffff000426c3c0 moused 818 817 817 0 S (threaded) nfsd 100134 S rpcsvc 0xffffff0004399020 nfsd: service 100133 S rpcsvc 0xffffff00043990a0 nfsd: service 100132 S rpcsvc 0xffffff0004399120 nfsd: service 100105 S rpcsvc 0xffffff0004182720 nfsd: master 817 1 817 0 Ss select 0xffffff00041806c0 nfsd 809 1 809 0 Ss select 0xffffff000426c440 mountd 731 1 731 0 Ss select 0xffffff000439a340 rpcbind 714 1 714 0 Ls *buf queu 0xffffff000426a840 syslogd 580 1 580 0 Ss select 0xffffff000439b040 devd 21 0 0 0 DL flowclea 0xffffffff80e52310 [flowcleaner] 20 0 0 0 LL *Softdep 0xffffff00023a6300 [softdepflush] 19 0 0 0 DL syncer 0xffffffff80e52060 [syncer] 18 0 0 0 DL vlruwt 0xffffff0004193460 [vnlru] 17 0 0 0 LL *Softdep 0xffffff00023a6300 [bufdaemon] 16 0 0 0 DL pgzero 0xffffffff80e6b14c [pagezero] 15 0 0 0 DL psleep 0xffffffff80e6a4e8 [vmdaemon] 9 0 0 0 DL psleep 0xffffffff80e6a4ac [pagedaemon] 8 0 0 0 DL ccb_scan 0xffffffff80c59460 [xpt_thrd] 7 0 0 0 DL waiting_ 0xffffffff80e54ea0 [sctp_iterator] 6 0 0 0 DL - 0xffffff0004003a48 [fdc0] 5 0 0 0 SL - 0xffffff80003ee000 [fw0_probe] 14 0 0 0 DL (threaded) [usb] 100057 D - 0xffffff80003ebdd0 [usbus5] 100056 D - 0xffffff80003ebd78 [usbus5] 100055 D - 0xffffff80003ebd20 [usbus5] 100054 D - 0xffffff80003ebcc8 [usbus5] 100052 D - 0xffffff80003e3460 [usbus4] 100051 D - 0xffffff80003e3408 [usbus4] 100050 D - 0xffffff80003e33b0 [usbus4] 100049 D - 0xffffff80003e3358 [usbus4] 100048 D - 0xffffff80003e0460 [usbus3] 100047 D - 0xffffff80003e0408 [usbus3] 100046 D - 0xffffff80003e03b0 [usbus3] 100045 D - 0xffffff80003e0358 [usbus3] 100044 D - 0xffffff80003dd460 [usbus2] 100043 D - 0xffffff80003dd408 [usbus2] 100042 D - 0xffffff80003dd3b0 [usbus2] 100041 D - 0xffffff80003dd358 [usbus2] 100039 D - 0xffffff80003da460 [usbus1] 100038 D - 0xffffff80003da408 [usbus1] 100037 D - 0xffffff80003da3b0 [usbus1] 100036 D - 0xffffff80003da358 [usbus1] 100034 D - 0xffffff80003d7460 [usbus0] 100033 D - 0xffffff80003d7408 [usbus0] 100032 D - 0xffffff80003d73b0 [usbus0] 100031 D - 0xffffff80003d7358 [usbus0] 13 0 0 0 DL - 0xffffffff80c90d44 [yarrow] 4 0 0 0 DL - 0xffffffff80c8d488 [g_down] 3 0 0 0 RL CPU 1 [g_up] 2 0 0 0 DL - 0xffffffff80c8d470 [g_event] 12 0 0 0 RL (threaded) [intr] 100065 I [irq12: psm0] 100064 I [irq1: atkbd0] 100062 I [swi0: uart] 100059 I [irq20: fwohci0] 100058 I [irq14: ata0] 100053 I [irq19: ehci0] 100040 I [irq18: ohci2 ohci4] 100035 I [irq17: ohci1 ohci3] 100030 I [irq16: hdac1 ohci0] 100029 I [irq22: atapci0] 100028 I [irq256: hdac0] 100027 I [irq9: acpi0] 100026 I [swi6: task queue] 100025 I [swi6: Giant taskq] 100023 I [swi5: +] 100021 I [swi2: cambio] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 Run CPU 0 [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 CanRun [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00024f08c0 [init] 10 0 0 0 DL audit_wo 0xffffffff80e60510 [audit] 0 0 0 0 DLs (threaded) [kernel] 100066 D deadlkre 0xffffffff80c90d44 [deadlkres] 100060 D - 0xffffff0003f70800 [fw0_taskq] 100024 D - 0xffffff000263a980 [thread taskq] 100022 D - 0xffffff000263ab00 [kqueue taskq] 100020 D - 0xffffff0002674780 [acpi_task_2] 100019 D - 0xffffff0002674780 [acpi_task_1] 100018 D - 0xffffff0002674780 [acpi_task_0] 100016 D - 0xffffff0002507700 [firmware taskq] 100000 D sched 0xffffffff80c8d5c0 [swapper] db:0:ps> allt Tracing command creat pid 1128 tid 100090 td 0xffffff000418b3c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 bwait() at bwait+0x64 bufwait() at bufwait+0x23 bufwrite() at bufwrite+0x158 ffs_update() at ffs_update+0x1b3 softdep_sync_metadata() at softdep_sync_metadata+0x5f1 ffs_syncvnode() at ffs_syncvnode+0x213 softdep_prelink() at softdep_prelink+0x13d softdep_setup_create() at softdep_setup_create+0x94 ufs_makeinode() at ufs_makeinode+0x567 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800845b6c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Tracing command creat pid 1127 tid 100151 td 0xffffff0004884780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e getblk() at getblk+0xf4 breadn() at breadn+0x3f bread() at bread+0x1e ffs_update() at ffs_update+0x105 softdep_sync_metadata() at softdep_sync_metadata+0x5f1 ffs_syncvnode() at ffs_syncvnode+0x213 softdep_prelink() at softdep_prelink+0x13d softdep_setup_create() at softdep_setup_create+0x94 ufs_makeinode() at ufs_makeinode+0x567 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800845b6c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Tracing command creat pid 1126 tid 100112 td 0xffffff000432e780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 bufobj_wwait() at bufobj_wwait+0x63 ffs_syncvnode() at ffs_syncvnode+0x1f4 softdep_prelink() at softdep_prelink+0x13d softdep_setup_create() at softdep_setup_create+0x94 ufs_makeinode() at ufs_makeinode+0x567 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800845b6c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Tracing command creat pid 1125 tid 100145 td 0xffffff000427e780 journal_suspend() at journal_suspend+0x52 softdep_setup_unlink() at softdep_setup_unlink+0x55 ufs_dirremove() at ufs_dirremove+0x24e ufs_remove() at ufs_remove+0x92 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xd7 kern_unlinkat() at kern_unlinkat+0x2cb syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084343c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Tracing command creat pid 1124 tid 100107 td 0xffffff0004271780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 getdirtybuf() at getdirtybuf+0x21e flush_deplist() at flush_deplist+0x6f softdep_sync_metadata() at softdep_sync_metadata+0x10f ffs_syncvnode() at ffs_syncvnode+0x213 softdep_prelink() at softdep_prelink+0x13d softdep_setup_create() at softdep_setup_create+0x94 ufs_makeinode() at ufs_makeinode+0x567 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800845b6c, rsp = 0x7fffffffe578, rbp = 0x7fffffffe620 --- Tracing command creat pid 1123 tid 100160 td 0xffffff00048803c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb57c, rsp = 0x7fffffffe658, rbp = 0x7fffffffe680 --- Tracing command creat pid 1122 tid 100159 td 0xffffff0004880780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800839fac, rsp = 0x7fffffffe658, rbp = 0x7fffffffe6b0 --- Tracing command run pid 1121 tid 100089 td 0xffffff000418b780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb57c, rsp = 0x7fffffffe178, rbp = 0x7fffffffe620 --- Tracing command run pid 1120 tid 100158 td 0xffffff0004880b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb57c, rsp = 0x7fffffffe658, rbp = 0x7fffffffe680 --- Tracing command run pid 1119 tid 100122 td 0xffffff0004341000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800839fac, rsp = 0x7fffffffe658, rbp = 0x7fffffffe6b0 --- Tracing command sh pid 1112 tid 100125 td 0xffffff0004221b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80093857c, rsp = 0x7fffffffe5d8, rbp = 0x7fffffffe798 --- Tracing command getty pid 1110 tid 100147 td 0xffffff0004886780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1109 tid 100121 td 0xffffff00043413c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1108 tid 100155 td 0xffffff0004883780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1107 tid 100156 td 0xffffff00048833c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1106 tid 100153 td 0xffffff0004884000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1105 tid 100152 td 0xffffff00048843c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1104 tid 100150 td 0xffffff0004884b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1103 tid 100149 td 0xffffff0004886000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1102 tid 100085 td 0xffffff000418c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800854edc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command inetd pid 1076 tid 100144 td 0xffffff000427eb40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a69e5c, rsp = 0x7fffffffdda8, rbp = 0x7878787878787878 --- Tracing command bash pid 1033 tid 100146 td 0xffffff0004886b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b8f57c, rsp = 0x7fffffffe7d8, rbp = 0x8011d99c0 --- Tracing command sshd pid 1032 tid 100082 td 0xffffff00042703c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8016d4e5c, rsp = 0x7fffffffe028, rbp = 0x7fffffffe0b0 --- Tracing command sshd pid 1030 tid 100103 td 0xffffff000430b780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0x100b dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8016d4edc, rsp = 0x7fffffffdc28, rbp = 0 --- Tracing command cron pid 1017 tid 100100 td 0xffffff000435c3c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800944fac, rsp = 0x7fffffffeb28, rbp = 0x3c --- Tracing command sendmail pid 1010 tid 100102 td 0xffffff000430bb40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d41d7c, rsp = 0x7fffffffcd78, rbp = 0x1 --- Tracing command sendmail pid 1006 tid 100096 td 0xffffff00043083c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800de2e5c, rsp = 0x7fffffffc1e8, rbp = 0x7fffffffc280 --- Tracing command sshd pid 999 tid 100115 td 0xffffff0004283b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8016d4e5c, rsp = 0x7fffffffe148, rbp = 0xa --- Tracing command ntpd pid 962 tid 100081 td 0xffffff0004270780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800d57e5c, rsp = 0x7fffffffec18, rbp = 0x7fffffffed48 --- Tracing command moused pid 847 tid 100110 td 0xffffff0004330000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800976e5c, rsp = 0x7fffffffe878, rbp = 0x7fffffffed40 --- Tracing command nfsd pid 818 tid 100134 td 0xffffff000421e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8006a295c, rsp = 0x7fffffffe6d8, rbp = 0x5 --- Tracing command nfsd pid 818 tid 100133 td 0xffffff000421e3c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8006a295c, rsp = 0x7fffffffe6d8, rbp = 0x5 --- Tracing command nfsd pid 818 tid 100132 td 0xffffff000421e780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8006a295c, rsp = 0x7fffffffe6d8, rbp = 0x5 --- Tracing command nfsd pid 818 tid 100105 td 0xffffff000430b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_run() at svc_run+0x94 nfssvc_nfsd() at nfssvc_nfsd+0xa2 nfssvc_nfsserver() at nfssvc_nfsserver+0x5b nfssvc() at nfssvc+0x7c syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006a295c, rsp = 0x7fffffffe6d8, rbp = 0x5 --- Tracing command nfsd pid 817 tid 100091 td 0xffffff000418b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800742e5c, rsp = 0x7fffffffe978, rbp = 0x7fffffffec50 --- Tracing command mountd pid 809 tid 100116 td 0xffffff0004283780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800858e5c, rsp = 0x7fffffffec78, rbp = 0x7fffffffed70 --- Tracing command rpcbind pid 731 tid 100095 td 0xffffff0004308780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x406 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80090ccbc, rsp = 0x7fffffffca88, rbp = 0x800c640c0 --- Tracing command syslogd pid 714 tid 100117 td 0xffffff00042833c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 bqrelse() at bqrelse+0x9b bdwrite() at bdwrite+0xb0 ffs_write() at ffs_write+0x589 VOP_WRITE_APV() at VOP_WRITE_APV+0x129 vn_write() at vn_write+0x2a2 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 writev() at writev+0x41 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (121, FreeBSD ELF64, writev), rip = 0x80084088c, rsp = 0x7fffffffd3c8, rbp = 0x7fffffffd450 --- Tracing command devd pid 580 tid 100093 td 0xffffff000427e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x44413c, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 --- Tracing command flowcleaner pid 21 tid 100076 td 0xffffff000418c3c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xd8 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807327fd30, rbp = 0 --- Tracing command softdepflush pid 20 tid 100075 td 0xffffff000418c780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 process_worklist_item() at process_worklist_item+0x234 softdep_process_worklist() at softdep_process_worklist+0x8b softdep_flush() at softdep_flush+0xba fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807327ad30, rbp = 0 --- Tracing command syncer pid 19 tid 100074 td 0xffffff000418cb40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4de fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073275d30, rbp = 0 --- Tracing command vnlru pid 18 tid 100073 td 0xffffff000418d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vnlru_proc() at vnlru_proc+0x4df fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073270d30, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100072 td 0xffffff000418d3c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 softdep_count_dependencies() at softdep_count_dependencies+0x27 flushbufqueues() at flushbufqueues+0x335 buf_do_flush() at buf_do_flush+0x16 buf_daemon() at buf_daemon+0xdf fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807326bd30, rbp = 0 --- Tracing command pagezero pid 16 tid 100071 td 0xffffff000418d780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073266d30, rbp = 0 --- Tracing command vmdaemon pid 15 tid 100070 td 0xffffff000418db40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073261d30, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100069 td 0xffffff000418e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x8d2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807325cd30, rbp = 0 --- Tracing command xpt_thrd pid 8 tid 100068 td 0xffffff0003e01000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731f7d30, rbp = 0 --- Tracing command sctp_iterator pid 7 tid 100067 td 0xffffff0003e013c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 sctp_iterator_thread() at sctp_iterator_thread+0x49 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731f2d30, rbp = 0 --- Tracing command fdc0 pid 6 tid 100063 td 0xffffff0003e023c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x74a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731dad30, rbp = 0 --- Tracing command fw0_probe pid 5 tid 100061 td 0xffffff0003e02b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 fw_bus_probe_thread() at fw_bus_probe_thread+0xbc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731c2d30, rbp = 0 --- Tracing command usb pid 14 tid 100057 td 0xffffff0003e04b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f41d30, rbp = 0 --- Tracing command usb pid 14 tid 100056 td 0xffffff0003a833c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f3cd30, rbp = 0 --- Tracing command usb pid 14 tid 100055 td 0xffffff0003a83780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f37d30, rbp = 0 --- Tracing command usb pid 14 tid 100054 td 0xffffff0003a83b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f32d30, rbp = 0 --- Tracing command usb pid 14 tid 100052 td 0xffffff0003b443c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072da6d30, rbp = 0 --- Tracing command usb pid 14 tid 100051 td 0xffffff0003b44780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072da1d30, rbp = 0 --- Tracing command usb pid 14 tid 100050 td 0xffffff0003b44b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d9cd30, rbp = 0 --- Tracing command usb pid 14 tid 100049 td 0xffffff0003b45000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d97d30, rbp = 0 --- Tracing command usb pid 14 tid 100048 td 0xffffff0003b453c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d4ed30, rbp = 0 --- Tracing command usb pid 14 tid 100047 td 0xffffff0003b45780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d49d30, rbp = 0 --- Tracing command usb pid 14 tid 100046 td 0xffffff0003b45b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d44d30, rbp = 0 --- Tracing command usb pid 14 tid 100045 td 0xffffff00026a4b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072d3fd30, rbp = 0 --- Tracing command usb pid 14 tid 100044 td 0xffffff0003a80000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072cf6d30, rbp = 0 --- Tracing command usb pid 14 tid 100043 td 0xffffff0003a803c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072cf1d30, rbp = 0 --- Tracing command usb pid 14 tid 100042 td 0xffffff0003a80780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072cecd30, rbp = 0 --- Tracing command usb pid 14 tid 100041 td 0xffffff0003a80b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072ce7d30, rbp = 0 --- Tracing command usb pid 14 tid 100039 td 0xffffff0003a813c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c99d30, rbp = 0 --- Tracing command usb pid 14 tid 100038 td 0xffffff0003a81780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c94d30, rbp = 0 --- Tracing command usb pid 14 tid 100037 td 0xffffff0003a81b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c8fd30, rbp = 0 --- Tracing command usb pid 14 tid 100036 td 0xffffff0003a83000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c8ad30, rbp = 0 --- Tracing command usb pid 14 tid 100034 td 0xffffff0002677b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c3cd30, rbp = 0 --- Tracing command usb pid 14 tid 100033 td 0xffffff00026a2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c37d30, rbp = 0 --- Tracing command usb pid 14 tid 100032 td 0xffffff00026a23c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c32d30, rbp = 0 --- Tracing command usb pid 14 tid 100031 td 0xffffff00026a2780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c2dd30, rbp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xffffff0002505000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000087d30, rbp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xffffff0002505780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_down() at g_io_schedule_down+0x245 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800007dd30, rbp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xffffff0002505b40 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff8059dec2, rsp = 0xffffff8000013fe0, rbp = 0xffffff8000078a60 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x102 _mtx_lock_flags() at _mtx_lock_flags+0xe1 softdep_disk_write_complete() at softdep_disk_write_complete+0x4e bufdone_finish() at bufdone_finish+0x2c bufdone() at bufdone+0x6c g_io_schedule_up() at g_io_schedule_up+0xed g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000078d30, rbp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xffffff0002506000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000073d30, rbp = 0 --- Tracing command intr pid 12 tid 100065 td 0xffffff0003e01b40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100064 td 0xffffff0003e02000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731dfd30, rbp = 0 --- Tracing command intr pid 12 tid 100062 td 0xffffff0003e02780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731cbd30, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xffffff0003e043c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f6ed30, rbp = 0 --- Tracing command intr pid 12 tid 100058 td 0xffffff0003e04780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072f69d30, rbp = 0 --- Tracing command intr pid 12 tid 100053 td 0xffffff0003b44000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100040 td 0xffffff0003a81000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072ce2d30, rbp = 0 --- Tracing command intr pid 12 tid 100035 td 0xffffff0002677780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c85d30, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xffffff00026a2b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8072c28d30, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff00026a4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80729d3d30, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xffffff00026a43c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000bed30, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xffffff00026a4780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b9d30, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff0002506780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b4d30, rbp = 0 --- Tracing command intr pid 12 tid 100025 td 0xffffff0002506b40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xffffff00026753c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a5d30, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xffffff0002675b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800009bd30, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00025063c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800006ed30, rbp = 0 --- Tracing command intr pid 12 tid 100011 td 0xffffff00024f43c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000069d30, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xffffff00024f4780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000064d30, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xffffff00024f4b40 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff802058e6, rsp = 0xffffffff80eaba40, rbp = 0xffffff800005faa0 --- acpi_timer_read() at acpi_timer_read+0x16 acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x14 binuptime() at binuptime+0x3b softclock() at softclock+0x2d5 intr_event_execute_handlers() at intr_event_execute_handlers+0x66 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800005fd30, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xffffff00024ff000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800005ad30, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xffffff00024ff3c0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xffffff00024ff780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d critical_exit() at critical_exit+0xa3 lapic_handle_timer() at lapic_handle_timer+0xe0 Xtimerint() at Xtimerint+0x8c --- interrupt, rip = 0xffffffff80887476, rsp = 0xffffff8000050b50, rbp = 0xffffff8000050b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000050d30, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff00024f3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff80887476, rsp = 0xffffff800004bb50, rbp = 0xffffff800004bb60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800004bd30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00024f33c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_idletd() at sched_idletd+0x26c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000046d30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff00024f3780 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80887476, rsp = 0xffffff8000021fe0, rbp = 0xffffff8000041b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000041d30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00024f3b40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c30c, rsp = 0x7fffffffe808, rbp = 0x401ce0 --- Tracing command audit pid 10 tid 100001 td 0xffffff00024f4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000037d30, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xffffff0003e01780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 deadlkres() at deadlkres+0x1ec fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80731edd30, rbp = 0 --- Tracing command kernel pid 0 tid 100060 td 0xffffff0003e04000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80730fdd30, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xffffff0002675000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000aad30, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xffffff0002675780 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a0d30, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xffffff0002677000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000096d30, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xffffff00026773c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000091d30, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xffffff00024ffb40 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800008cd30, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00025053c0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000082d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c8da20 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c db:0:allt> call doadump Physical memory: 3951 MB Dumping 1558 MB: 1543 1527 1511 1495 1479 1463 1447 1431 1415 1399 1383 1367 1351 1335 1319 1303 1287 1271 1255 1239 1223 1207 1191 1175 1159 1143 1127 1111 1095 1079 1063 1047 1031 1015 999 983 967 951 935 919 903 887 871 855 839 823 807 791 775 759 743 727 711 695 679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) bt #0 doadump () at pcpu.h:223 #1 0xffffffff801e19ec in db_fncall (dummy1=Variable "dummy1" is not available. ) at ../../../ddb/db_command.c:548 #2 0xffffffff801e1c9d in db_command (last_cmdp=0xffffffff80c5ab60, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #3 0xffffffff801e6593 in db_script_exec (scriptname=0xffffffff80c5b400 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff801e1d21 in db_command (last_cmdp=0xffffffff80c5ab60, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #5 0xffffffff801e1f70 in db_command_loop () at ../../../ddb/db_command.c:498 #6 0xffffffff801e3f49 in db_trap (type=Variable "type" is not available. ) at ../../../ddb/db_main.c:229 #7 0xffffffff805e0555 in kdb_trap (type=0xc, code=0x0, tf=0xffffff80756db7c0) at ../../../kern/subr_kdb.c:535 #8 0xffffffff808aae6d in trap_fatal (frame=0xffffff80756db7c0, eva=Variable "eva" is not available. ) at ../../../amd64/amd64/trap.c:773 #9 0xffffffff808ab975 in trap (frame=0xffffff80756db7c0) at ../../../amd64/amd64/trap.c:300 #10 0xffffffff80890633 in calltrap () at ../../../amd64/amd64/exception.S:223 #11 0xffffffff807d3bd2 in journal_suspend (ump=Variable "ump" is not available. ) at ../../../ufs/ffs/ffs_softdep.c:2437 #12 0xffffffff807df7f5 in softdep_setup_unlink (dp=0xffffff00175ba690, ip=0xffffff001757b0a8) at ../../../ufs/ffs/ffs_softdep.c:3967 #13 0xffffffff807f362e in ufs_dirremove (dvp=0xffffff00175ca278, ip=0xffffff001757b0a8, flags=0x500800c, isrmdir=0x0) at ../../../ufs/ufs/ufs_lookup.c:1141 #14 0xffffffff807faca2 in ufs_remove (ap=0xffffff80756dbaf0) at ../../../ufs/ufs/ufs_vnops.c:932 #15 0xffffffff808fdcd7 in VOP_REMOVE_APV (vop=0xffffffff80c1f7e0, a=0xffffff80756dbaf0) at vnode_if.c:1333 #16 0xffffffff8064822b in kern_unlinkat (td=0xffffff000427e780, fd=0xffffff9c, path=0x7fffffffe590
, pathseg=UIO_USERSPACE, oldinum=0x0) at vnode_if.h:575 #17 0xffffffff808ab2f8 in syscall (frame=0xffffff80756dbc80) at ../../../amd64/amd64/trap.c:946 #18 0xffffffff80890911 in Xfast_syscall () at ../../../amd64/amd64/exception.S:374 #19 0x000000080084343c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 11 #11 0xffffffff807d3bd2 in journal_suspend (ump=Variable "ump" is not available. ) at ../../../ufs/ffs/ffs_softdep.c:2437 2437 mp->mnt_susp_owner = FIRST_THREAD_IN_PROC(softdepproc); (kgdb) l *0xffffffff807d3bd2 0xffffffff807d3bd2 is in journal_suspend (../../../ufs/ffs/ffs_softdep.c:2439). 2434 if ((mp->mnt_kern_flag & MNTK_SUSPEND) == 0) { 2435 stat_journal_min++; 2436 mp->mnt_kern_flag |= MNTK_SUSPEND; 2437 mp->mnt_susp_owner = FIRST_THREAD_IN_PROC(softdepproc); 2438 } 2439 jblocks->jb_suspended = 1; 2440 MNT_IUNLOCK(mp); 2441 } 2442 2443 /* (kgdb) p jblocks $3 = (struct jblocks *) 0x0 (kgdb) f 13 #13 0xffffffff807f362e in ufs_dirremove (dvp=0xffffff00175ca278, ip=0xffffff001757b0a8, flags=0x500800c, isrmdir=0x0) at ../../../ufs/ufs/ufs_lookup.c:1141 1141 softdep_setup_unlink(dp, ip); (kgdb) l 1136 * Adjust the link count early so softdep can block if necessary. 1137 */ 1138 if (ip) { 1139 ip->i_effnlink--; 1140 if (DOINGSOFTDEP(dvp)) { 1141 softdep_setup_unlink(dp, ip); 1142 } else { 1143 ip->i_nlink--; 1144 DIP_SET(ip, i_nlink, ip->i_nlink); 1145 ip->i_flag |= IN_CHANGE; (kgdb) p dvp $4 = (struct vnode *) 0xffffff00175ca278 (kgdb) p *dvp $5 = {v_type = VDIR, v_tag = 0xffffffff809b061d "ufs", v_op = 0xffffffff80c1efa0, v_data = 0xffffff00175ba690, v_mount = 0xffffff00041c3ab0, v_nmntvnodes = {tqe_next = 0xffffff0015cba278, tqe_prev = 0xffffff00175ca028}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0, vu_yield = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xffffff80005758b0}, v_hash = 0xc54c41, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0xffffff0015cc1bd0, tqh_last = 0xffffff0015cc1bf0}, v_cache_dd = 0xffffff0015cc1bd0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff809b061d "ufs", lo_flags = 0x57b0000, lo_data = 0x0, lo_witness = 0xffffff8000223400}, lk_lock = 0xffffff000427e780, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x50, lk_stack = {depth = 0x9, pcs = {0xffffffff805963ff, 0xffffffff807ecb3c, 0xffffffff808fd53b, 0xffffffff8064ffb7, 0xffffffff80637666, 0xffffffff80638901, 0xffffffff80648088, 0xffffffff808ab2f8, 0xffffffff80890911, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = {lo_name = 0xffffffff809c0be6 "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff800021b580}, mtx_lock = 0x4}, v_vnlock = 0xffffff00175ca310, v_holdcnt = 0x34, v_usecount = 0x2, v_iflag = 0x0, v_vflag = 0x0, v_writecount = 0x0, v_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xffffffff809c9bbb "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff8000221600}, mtx_lock = 0x4}, bo_clean = {bv_hd = { tqh_first = 0xffffff805019bc58, tqh_last = 0xffffff80501a5a00}, bv_root = 0xffffff80501a59b0, bv_cnt = 0x31}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff00175ca470}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xffffffff80c01b80, bo_bsize = 0x4000, bo_object = 0xffffff000459da20, bo_synclist = {le_next = 0x0, le_prev = 0xffffff000402e0f8}, bo_private = 0xffffff00175ca278, __bo_vnode = 0xffffff00175ca278}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) p *dvp->v_mount $6 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff809c0c0f "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff800021b880}, mtx_lock = 0xffffff000427e780}, mnt_gen = 0x1, mnt_list = {tqe_next = 0x0, tqe_prev = 0xffffff000433e028}, mnt_op = 0xffffffff80c1eb20, mnt_vfc = 0xffffffff80c1e820, mnt_vnodecovered = 0xffffff00042fc278, mnt_syncer = 0xffffff00048a3768, mnt_ref = 0x61bd, mnt_nvnodelist = {tqh_first = 0xffffff00048a39e0, tqh_last = 0xffffff00157972a0}, mnt_nvnodelistsize = 0x61b7, mnt_writeopcount = 0x5, mnt_kern_flag = 0x68000044, mnt_flag = 0x201300, mnt_xflag = 0x0, mnt_noasync = 0x1, mnt_opt = 0xffffff00041cb550, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x78, mnt_stat = {f_version = 0x20030518, f_type = 0x1, f_flags = 0x201300, f_bsize = 0x800, f_iosize = 0x4000, f_blocks = 0x306cbf3, f_bfree = 0x2eaa0ac, f_bavail = 0x2aca4d6, f_files = 0xc887fe, f_ffree = 0xc307fb, f_syncwrites = 0x0, f_asyncwrites = 0x0, f_syncreads = 0x0, f_asyncreads = 0x0, f_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = {0x48f3a647, 0x122b5359}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/ad4s2e", '\0' , f_mntonname = "/tmp", '\0' }, mnt_cred = 0xffffff00047dd600, mnt_data = 0xffffff0004305000, mnt_time = 0x0, mnt_iosize_max = 0x20000, mnt_export = 0xffffff0004306800, mnt_label = 0x0, mnt_hashseed = 0xfd789ed5, mnt_lockref = 0x1, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x2d523f, mnt_susp_owner = 0xffffff000418c780, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xffffffff809c8efe "explock", lo_flags = 0x5730000, lo_data = 0x0, lo_witness = 0xffffff8000223200}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x50, lk_stack = {depth = 0x6, pcs = {0xffffffff80596687, 0xffffffff80633f69, 0xffffffff8063cd28, 0xffffffff8063e033, 0xffffffff808ab2f8, 0xffffffff80890911, 0x0 }}}} (kgdb) $ svn diff -x -p /usr/src/sys -- Test scenario: Index: stress2/testcases/creat/creat.c =================================================================== --- stress2/testcases/creat/creat.c (revision 187224) +++ stress2/testcases/creat/creat.c (working copy) @@ -64,10 +64,8 @@ setup(int nb) pct = random_int(1, 90); size = size / 100 * pct + 1; - if (size > 20000) - size = 20000; /* arbitrary limit number of files pr. dir */ - if (size > 1000) - size = 1000; /* XXX Soft Update */ + if (size > 32000) + size = 32000; /* arbitrary limit number of files pr. dir */ /* Resource requirements: */ while (size > 0) { $ cd stress2; ./run.sh creat.cfg run: run time 3+00:00:00, incarnations 1, load 100, verbose 1 19:18:21 Loop #1 creat: run time 0+00:02:00, incarnations 5, load 100, verbose 1