GENERIC HEAD from 2013-02-25 11:22:54 UTC, r247264M, vmcore.40 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r247264M: Mon Feb 25 13:37:17 CET 2013 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 0x10 Model = 0x2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8099631104 (7724 MB) : Trying to mount root from ufs:/dev/ufs/root [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ufs/root: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/root: clean, 448786 free (890 frags, 55987 blocks, 0.1% fragmentation) /dev/ufs/home: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/home: clean, 113613 free (5277 frags, 13542 blocks, 0.5% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 42611274 free (1226 frags, 5326256 blocks, 0.0% fragmentation) /dev/ufs/usr: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/usr: clean, 4275364 free (178068 frags, 512162 blocks, 1.8% fragmentation) /dev/ufs/var: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/var: clean, 8486866 free (43850 frags, 1055377 blocks, 0.2% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat/pkg /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. savecore: unable to read from bounds, using 0 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device Feb 25 13:43:02 x4 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device savecore: unable to read from bounds, using 0 No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xffffff81e6e69968 bufwait (bufwait) @ kern/vfs_bio.c:2673 2nd 0xfffffe00053e5400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f7373f0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f7374b0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f7374d0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f737580 _sx_xlock() at _sx_xlock+0x74/frame 0xffffff823f7375b0 ufsdirhash_acquire() at ufsdirhash_acquire+0x44/frame 0xffffff823f7375d0 ufsdirhash_add() at ufsdirhash_add+0x19/frame 0xffffff823f737600 ufs_direnter() at ufs_direnter+0x6c1/frame 0xffffff823f7376d0 ufs_mkdir() at ufs_mkdir+0x50e/frame 0xffffff823f7378c0 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc9/frame 0xffffff823f7378f0 kern_mkdirat() at kern_mkdirat+0x212/frame 0xffffff823f737ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f737bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f737bf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092532a, rsp = 0x7fffffffd788, rbp = 0x801006050 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting default moused. Starting inetd. Mon Feb 25 13:43:06 CET 2013 FreeBSD/amd64 (x4.osted.lan) (console) login: Feb 25 13:43:22 x4 su: pho to root on /dev/pts/1 20130225 13:43:27 all: quota8.sh lock order reversal: 1st 0xfffffe013acd12d8 ufs (ufs) @ kern/vfs_subr.c:2176 2nd 0xffffff81e788d638 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xfffffe0015b3e548 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f89eb60 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f89ec20 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f89ec40 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f89ecf0 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f89edd0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f89ee20 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f89ee50 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f89eed0 vget() at vget+0x63/frame 0xffffff823f89ef20 vfs_hash_get() at vfs_hash_get+0xd5/frame 0xffffff823f89ef70 ffs_vgetf() at ffs_vgetf+0x48/frame 0xffffff823f89f000 softdep_sync_buf() at softdep_sync_buf+0x3b7/frame 0xffffff823f89f0f0 ffs_syncvnode() at ffs_syncvnode+0x311/frame 0xffffff823f89f170 ffs_sync() at ffs_sync+0x1b3/frame 0xffffff823f89f230 vfs_write_suspend() at vfs_write_suspend+0x105/frame 0xffffff823f89f260 ffs_snapshot() at ffs_snapshot+0xb87/frame 0xffffff823f89f5a0 ffs_mount() at ffs_mount+0x8d9/frame 0xffffff823f89f790 vfs_donmount() at vfs_donmount+0xee0/frame 0xffffff823f89fa90 sys_nmount() at sys_nmount+0x66/frame 0xffffff823f89fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f89fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f89fbf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800880dfa, rsp = 0x7fffffffcea8, rbp = 0x7fffffffda60 --- lock order reversal: 1st 0xffffff81e78ac8f8 bufwait (bufwait) @ kern/vfs_bio.c:2673 2nd 0xfffffe0008c48430 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2298 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f7dc060 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f7dc120 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f7dc140 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f7dc1f0 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f7dc2d0 ffs_copyonwrite() at ffs_copyonwrite+0x199/frame 0xffffff823f7dc370 ffs_geom_strategy() at ffs_geom_strategy+0x12b/frame 0xffffff823f7dc3a0 ufs_strategy() at ufs_strategy+0x45/frame 0xffffff823f7dc3d0 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0xd1/frame 0xffffff823f7dc400 bufstrategy() at bufstrategy+0x4b/frame 0xffffff823f7dc440 bufwrite() at bufwrite+0x125/frame 0xffffff823f7dc470 cluster_wbuild() at cluster_wbuild+0x544/frame 0xffffff823f7dc510 cluster_write() at cluster_write+0x323/frame 0xffffff823f7dc620 ffs_write() at ffs_write+0x5cb/frame 0xffffff823f7dc6c0 VOP_WRITE_APV() at VOP_WRITE_APV+0x135/frame 0xffffff823f7dc7d0 vn_write() at vn_write+0x281/frame 0xffffff823f7dc860 vn_io_fault() at vn_io_fault+0x196/frame 0xffffff823f7dc9f0 dofilewrite() at dofilewrite+0x85/frame 0xffffff823f7dca40 kern_writev() at kern_writev+0x6c/frame 0xffffff823f7dca80 sys_write() at sys_write+0x64/frame 0xffffff823f7dcad0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff823f7dcbf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff823f7dcbf0 --- syscall (4, FreeBSD ELF32, sys_write), rip = 0x2819249f, rsp = 0xffffb6c8, rbp = 0xffffc770 --- lock order reversal: 1st 0xfffffe0008c48430 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:968 2nd 0xfffffe0015b3e068 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1627 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f9172a0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f917360 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f917380 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f917430 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f917510 ffs_snapremove() at ffs_snapremove+0xe2/frame 0xffffff823f917590 ffs_truncate() at ffs_truncate+0xd35/frame 0xffffff823f9177e0 ufs_inactive() at ufs_inactive+0x28c/frame 0xffffff823f917820 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xc9/frame 0xffffff823f917850 vinactive() at vinactive+0xb2/frame 0xffffff823f9178b0 vputx() at vputx+0x375/frame 0xffffff823f917910 kern_unlinkat() at kern_unlinkat+0x19c/frame 0xffffff823f917ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f917bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f917bf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x8009249da, rsp = 0x7fffffffd5a8, rbp = 0 --- lock order reversal: 1st 0xfffffe0015544c98 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe013acd1068 syncer (syncer) @ kern/vfs_subr.c:2287 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f89a5d0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f89a690 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f89a6b0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f89a760 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f89a840 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff823f89a860 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f89a890 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f89a910 vputx() at vputx+0x315/frame 0xffffff823f89a970 dounmount() at dounmount+0x288/frame 0xffffff823f89a9e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823f89aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f89abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f89abf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffce78, rbp = 0x801006ce8 --- lock order reversal: 1st 0xfffffe0015544c98 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe00158ff548 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f89a500 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f89a5c0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f89a5e0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f89a690 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f89a770 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff823f89a790 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f89a7c0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f89a840 softdep_flushworklist() at softdep_flushworklist+0x61/frame 0xffffff823f89a8b0 ffs_sync() at ffs_sync+0x3be/frame 0xffffff823f89a970 dounmount() at dounmount+0x318/frame 0xffffff823f89a9e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823f89aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f89abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f89abf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffce78, rbp = 0x801006ce8 --- 20130225 13:45:47 all: quota8.sh 20130225 13:48:35 all: quota8.sh 20130225 13:52:18 all: quota8.sh 20130225 13:55:19 all: quota8.sh 20130225 13:57:57 all: quota8.sh lock order reversal: 1st 0xfffffe0008f20248 filedesc structure (filedesc structure) @ kern/kern_descrip.c:840 2nd 0xfffffe0015040068 ufs (ufs) @ kern/vfs_subr.c:4401 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f82c630 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f82c6f0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f82c710 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f82c7c0 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f82c8a0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f82c8f0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f82c920 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f82c9a0 knlist_remove_kq() at knlist_remove_kq+0x175/frame 0xffffff823f82c9d0 knote_fdclose() at knote_fdclose+0x177/frame 0xffffff823f82ca10 closefp() at closefp+0x6c/frame 0xffffff823f82ca60 do_dup() at do_dup+0x237/frame 0xffffff823f82cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f82cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f82cbf0 --- syscall (90, FreeBSD ELF64, sys_dup2), rip = 0x8008bde6a, rsp = 0x7fffffffd998, rbp = 0x800b8eb80 --- 20130225 14:00:16 all: quota8.sh Feb 25 14:02:41 x4 kernel: pid 7878 (rw), uid 1004 inumber 1061 on /mnt: filesystem full 20130225 14:03:38 all: quota8.sh Feb 25 14:06:05 x4 kernel: pid 9089 (rw), uid 1004 inumber 66955 on /mnt: filesystem full 20130225 14:06:32 all: quota8.sh 20130225 14:10:53 all: quota8.sh panic: dqflush: stray dquot cpuid = 2 KDB: enter: panic [ thread pid 15098 tid 100276 ] Stopped at kdb_enter+0x3b: movq $0,0xa97a22(%rip) db> run pho db:0:pho> bt Tracing pid 15098 tid 100276 td 0xfffffe013af8b900 kdb_enter() at kdb_enter+0x3b/frame 0xffffff823fb10650 vpanic() at vpanic+0xe1/frame 0xffffff823fb10690 panic() at panic+0x67/frame 0xffffff823fb10770 quotaoff1() at quotaoff1+0x31d/frame 0xffffff823fb10800 quotaoff_inchange() at quotaoff_inchange+0x29/frame 0xffffff823fb10840 ffs_flushfiles() at ffs_flushfiles+0x82/frame 0xffffff823fb108a0 softdep_flushfiles() at softdep_flushfiles+0x64/frame 0xffffff823fb10900 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff823fb10970 dounmount() at dounmount+0x349/frame 0xffffff823fb109e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823fb10ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fb10bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fb10bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffcec8, rbp = 0x801006ce8 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x5d6b00 curthread = 0xfffffe00051d4480: pid 11 "idle: cpu0" curpcb = 0xffffff800023dcc0 fpcurthread = none idlethread = 0xfffffe00051d4480: tid 100003 "idle: cpu0" curpmap = 0xffffffff8135aff0 tssp = 0xffffffff8156aa80 commontssp = 0xffffffff8156aa80 rsp0 = 0xffffff800023dcc0 gs32p = 0xffffffff81568bb8 ldt = 0xffffffff81568bf8 tss = 0xffffffff81568be8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807ef00b00 curthread = 0xfffffe00051d4000: pid 11 "idle: cpu1" curpcb = 0xffffff8000242cc0 fpcurthread = none idlethread = 0xfffffe00051d4000: tid 100004 "idle: cpu1" curpmap = 0xffffffff8135aff0 tssp = 0xffffffff8156aae8 commontssp = 0xffffffff8156aae8 rsp0 = 0xffffff8000242cc0 gs32p = 0xffffffff81568c20 ldt = 0xffffffff81568c60 tss = 0xffffffff81568c50 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807ef07b00 curthread = 0xfffffe013af8b900: pid 15098 "umount" curpcb = 0xffffff823fb10cc0 fpcurthread = none idlethread = 0xfffffe00051e0900: tid 100005 "idle: cpu2" curpmap = 0xfffffe0015064750 tssp = 0xffffffff8156ab50 commontssp = 0xffffffff8156ab50 rsp0 = 0xffffff823fb10cc0 gs32p = 0xffffffff81568c88 ldt = 0xffffffff81568cc8 tss = 0xffffffff81568cb8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807ef0eb00 curthread = 0xfffffe00051e0480: pid 11 "idle: cpu3" curpcb = 0xffffff800024ccc0 fpcurthread = none idlethread = 0xfffffe00051e0480: tid 100006 "idle: cpu3" curpmap = 0xffffffff8135aff0 tssp = 0xffffffff8156abb8 commontssp = 0xffffffff8156abb8 rsp0 = 0xffffff800024ccc0 gs32p = 0xffffffff81568cf0 ldt = 0xffffffff81568d30 tss = 0xffffffff81568d20 spin locks held: db:0:allpcpu> show alllocks Process 15098 (umount) thread 0xfffffe013af8b900 (100276) exclusive sleep mutex dqhlock (dqhlock) r = 0 (0xffffffff8155dc20) locked @ ufs/ufs/ufs_quota.c:1608 exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff8135d6a0) locked @ kern/vfs_mount.c:1155 exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0015544c98) locked @ kern/vfs_mount.c:1236 Process 15097 (sh) thread 0xfffffe0015dae000 (100168) shared lockmgr ufs (ufs) r = 0 (0xfffffe0008ba7a28) locked @ kern/vfs_lookup.c:516 db:0:alllocks> show lockedvnods Locked vnodes 0xfffffe0008ba79c0: tag ufs, type VDIR usecount 87, writecount 0, refcount 90 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) v_object 0xfffffe0008e082d0 ref 0 pages 1 lock type ufs: SHARED (count 1) #0 0xffffffff808c1aac at __lockmgr_args+0xd7c #1 0xffffffff80b35b7b at ffs_lock+0x9b #2 0xffffffff80d1f42b at VOP_LOCK1_APV+0xab #3 0xffffffff809910fe at _vn_lock+0x8e #4 0xffffffff80975c17 at lookup+0xc7 #5 0xffffffff80976bfc at namei+0x3fc #6 0xffffffff80988d9e at kern_statat_vnhook+0xbe #7 0xffffffff80988f95 at kern_statat+0x15 #8 0xffffffff8098913a at sys_stat+0x2a #9 0xffffffff80c80523 at amd64_syscall+0x2d3 #10 0xffffffff80c6a767 at Xfast_syscall+0xf7 ino 2, on dev ufs/root 0xfffffe0015544c30: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0xfffffe0015b2d3c8 flags (VI_ACTIVE) v_object 0xfffffe001530a000 ref 0 pages 0 lock type ufs: EXCL by thread 0xfffffe013af8b900 (pid 15098, umount, tid 100276) with shared waiters pending #0 0xffffffff808c1433 at __lockmgr_args+0x703 #1 0xffffffff80b35b7b at ffs_lock+0x9b #2 0xffffffff80d1f42b at VOP_LOCK1_APV+0xab #3 0xffffffff809910fe at _vn_lock+0x8e #4 0xffffffff80979043 at dounmount+0x93 #5 0xffffffff80979a8e at sys_unmount+0x38e #6 0xffffffff80c80523 at amd64_syscall+0x2d3 #7 0xffffffff80c6a767 at Xfast_syscall+0xf7 ino 47105, on dev ufs/root db:0:lockedvnods> show mount 0xfffffe0008debb58 /dev/ufs/root on / (ufs) 0xfffffe0008dec000 devfs on /dev (devfs) 0xfffffe0008deb790 /dev/ufs/home on /home (ufs) 0xfffffe0008deb3c8 /dev/label/tmp on /tmp (ufs) 0xfffffe0008deb000 /dev/ufs/usr on /usr (ufs) 0xfffffe0008f2d790 /dev/ufs/var on /var (ufs) 0xfffffe0008f2d3c8 procfs on /proc (procfs) 0xfffffe0015b2d3c8 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 15098 11392 1131 0 R+ CPU 2 umount 15097 9916 9904 1004 D+ ufs 0xfffffe0015544c98 sh 14043 1084 1081 1001 S nanslp 0xffffffff81362568 sleep 11392 1131 1131 0 S+ wait 0xfffffe0015dab950 sh 9916 9915 9904 1004 S+ wait 0xfffffe013afb6000 run 9915 9914 9904 1004 S+ wait 0xfffffe0015a34000 run 9914 9905 9904 1004 S+ nanslp 0xffffffff81362568 run 9905 1 9904 1004 S+ wait 0xfffffe0181ade000 sh 9890 0 0 0 DL mdwait 0xfffffe0015027000 [md5] 1131 1127 1131 0 S+ wait 0xfffffe0008c694a8 sh 1127 1124 1127 0 S+ wait 0xfffffe00156dc000 bash 1124 1108 1124 0 S+ pause 0xfffffe0015a330a0 csh 1108 1104 1108 1001 S+ wait 0xfffffe0015782000 su 1104 1103 1104 1001 Ss+ wait 0xfffffe0015786000 bash 1103 1101 1101 1001 S select 0xfffffe0008e044c0 sshd 1101 942 1101 0 Ss select 0xfffffe00081a3140 sshd 1085 1081 1081 1001 S piperd 0xfffffe00154f85d0 awk 1084 1081 1081 1001 S wait 0xfffffe0008e2f000 sh 1083 1080 1083 1001 Ss+ select 0xfffffe0008e045c0 top 1082 1079 1082 1001 Ss kqread 0xfffffe00150dcb00 tail 1081 1078 1081 1001 Ss wait 0xfffffe0015311950 sh 1080 1072 1072 1001 S select 0xfffffe0008e04540 sshd 1079 1073 1073 1001 S select 0xfffffe0008ec9c40 sshd 1078 1074 1074 1001 S select 0xfffffe00081a30c0 sshd 1074 942 1074 0 Ss select 0xfffffe0008ec9cc0 sshd 1073 942 1073 0 Ss select 0xfffffe0005401e40 sshd 1072 942 1072 0 Ss select 0xfffffe0008e04640 sshd 1071 1 1071 0 Ss+ ttyin 0xfffffe00052ba8a8 getty 1070 1 1070 0 Ss+ ttyin 0xfffffe00052baca8 getty 1069 1 1069 0 Ss+ ttyin 0xfffffe00080090a8 getty 1068 1 1068 0 Ss+ ttyin 0xfffffe00080094a8 getty 1067 1 1067 0 Ss+ ttyin 0xfffffe00080098a8 getty 1066 1 1066 0 Ss+ ttyin 0xfffffe00052b90a8 getty 1065 1 1065 0 Ss+ ttyin 0xfffffe00052b94a8 getty 1064 1 1064 0 Ss+ ttyin 0xfffffe00052b98a8 getty 1063 1 1063 0 Ss+ ttyin 0xfffffe00052b9ca8 getty 1024 1 1024 0 Ss select 0xfffffe0005401dc0 inetd 991 1 991 0 Ss select 0xfffffe0005401cc0 moused 971 1 971 0 Ss nanslp 0xffffffff81362568 watchdogd 961 1 961 0 Ss nanslp 0xffffffff81362568 cron 954 1 954 25 Ss pause 0xfffffe0008f38548 sendmail 950 1 950 0 Ss select 0xfffffe0008e04740 sendmail 942 1 942 0 Ss select 0xfffffe0008e033c0 sshd 851 1 851 0 Ss select 0xfffffe0005401940 ntpd 756 755 755 0 S (threaded) nfsd 100120 S rpcsvc 0xfffffe0008d0c820 nfsd: service 100119 S rpcsvc 0xfffffe0008d0c8a0 nfsd: service 100118 S rpcsvc 0xfffffe0008d0c920 nfsd: service 100111 S rpcsvc 0xfffffe0008db1aa0 nfsd: master 755 1 755 0 Ss select 0xfffffe0008e047c0 nfsd 746 1 746 0 Ss select 0xfffffe0008e03540 mountd 641 1 641 0 Ss select 0xfffffe00054019c0 rpcbind 615 1 615 0 Ss select 0xfffffe0008f140c0 syslogd 434 1 434 0 Ss select 0xfffffe0005401b40 devd 18 0 0 0 DL sdflush 0xffffffff8155da98 [softdepflush] 17 0 0 0 DL vlruwt 0xfffffe0008c68950 [vnlru] 16 0 0 0 DL syncer 0xffffffff81552100 [syncer] 9 0 0 0 DL psleep 0xffffffff81551b60 [bufdaemon] 8 0 0 0 DL pgzero 0xffffffff81567a3c [pagezero] 7 0 0 0 DL psleep 0xffffffff81566bf0 [vmdaemon] 6 0 0 0 DL psleep 0xffffffff81566bcc [pagedaemon] 5 0 0 0 DL ccb_scan 0xffffffff813203e0 [xpt_thrd] 4 0 0 0 DL waiting_ 0xffffffff81557de0 [sctp_iterator] 3 0 0 0 DL ctl_work 0xffffff80008b5000 [ctl_thrd] 2 0 0 0 DL - 0xfffffe00055de248 [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xffffff80008b0e18 [usbus5] 100057 D - 0xffffff80008b0dc0 [usbus5] 100056 D - 0xffffff80008b0d68 [usbus5] 100055 D - 0xffffff80008b0d10 [usbus5] 100053 D - 0xffffff80008a8460 [usbus4] 100052 D - 0xffffff80008a8408 [usbus4] 100051 D - 0xffffff80008a83b0 [usbus4] 100050 D - 0xffffff80008a8358 [usbus4] 100049 D - 0xffffff80008a5460 [usbus3] 100048 D - 0xffffff80008a5408 [usbus3] 100047 D - 0xffffff80008a53b0 [usbus3] 100046 D - 0xffffff80008a5358 [usbus3] 100045 D - 0xffffff80008a2460 [usbus2] 100044 D - 0xffffff80008a2408 [usbus2] 100043 D - 0xffffff80008a23b0 [usbus2] 100042 D - 0xffffff80008a2358 [usbus2] 100040 D - 0xffffff800089f460 [usbus1] 100039 D - 0xffffff800089f408 [usbus1] 100038 D - 0xffffff800089f3b0 [usbus1] 100037 D - 0xffffff800089f358 [usbus1] 100035 D - 0xffffff800089c460 [usbus0] 100034 D - 0xffffff800089c408 [usbus0] 100033 D - 0xffffff800089c3b0 [usbus0] 100032 D - 0xffffff800089c358 [usbus0] 14 0 0 0 DL - 0xffffffff81361204 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xffffffff8135a290 [g_down] 100014 D - 0xffffffff8135a288 [g_up] 100013 D - 0xffffffff8135a278 [g_event] 12 0 0 0 WL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100023 I [swi2: cambio] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100012 I [swi3: vm] 100011 I [swi1: netisr 0] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi4: clock] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe00051d2950 [init] 10 0 0 0 DL audit_wo 0xffffffff8155ca10 [audit] 0 0 0 0 DLs (threaded) [kernel] 100066 D - 0xffffffff81361204 [deadlkres] 100065 D - 0xfffffe00055fd780 [mca taskq] 100027 D - 0xfffffe0005386200 [acpi_task_2] 100026 D - 0xfffffe0005386200 [acpi_task_1] 100025 D - 0xfffffe0005386200 [acpi_task_0] 100024 D - 0xfffffe0005386280 [kqueue taskq] 100020 D - 0xfffffe0005386400 [thread taskq] 100018 D - 0xfffffe0005386500 [ffs_trim taskq] 100016 D - 0xfffffe0002f63900 [firmware taskq] 100000 D sched 0xffffffff8135a5a0 [swapper] db:0:ps> allt Tracing command umount pid 15098 tid 100276 td 0xfffffe013af8b900 kdb_enter() at kdb_enter+0x3b/frame 0xffffff823fb10650 vpanic() at vpanic+0xe1/frame 0xffffff823fb10690 panic() at panic+0x67/frame 0xffffff823fb10770 quotaoff1() at quotaoff1+0x31d/frame 0xffffff823fb10800 quotaoff_inchange() at quotaoff_inchange+0x29/frame 0xffffff823fb10840 ffs_flushfiles() at ffs_flushfiles+0x82/frame 0xffffff823fb108a0 softdep_flushfiles() at softdep_flushfiles+0x64/frame 0xffffff823fb10900 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff823fb10970 dounmount() at dounmount+0x349/frame 0xffffff823fb109e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823fb10ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fb10bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fb10bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffcec8, rbp = 0x801006ce8 --- Tracing command sh pid 15097 tid 100168 td 0xfffffe0015dae000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f8f4330 mi_switch() at mi_switch+0x238/frame 0xffffff823f8f4380 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f8f43c0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823f8f43f0 __lockmgr_args() at __lockmgr_args+0x108b/frame 0xffffff823f8f44d0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f8f4520 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f8f4550 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f8f45d0 vget() at vget+0x63/frame 0xffffff823f8f4620 cache_lookup() at cache_lookup+0x55a/frame 0xffffff823f8f46b0 vfs_cache_lookup() at vfs_cache_lookup+0xcb/frame 0xffffff823f8f4720 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc6/frame 0xffffff823f8f4750 lookup() at lookup+0x332/frame 0xffffff823f8f47e0 namei() at namei+0x3fc/frame 0xffffff823f8f4890 kern_statat_vnhook() at kern_statat_vnhook+0xbe/frame 0xffffff823f8f4a10 kern_statat() at kern_statat+0x15/frame 0xffffff823f8f4a30 sys_stat() at sys_stat+0x2a/frame 0xffffff823f8f4ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f8f4bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f8f4bf0 --- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800dbb30a, rsp = 0x7fffffffd178, rbp = 0 --- Tracing command sleep pid 14043 tid 100122 td 0xfffffe0008c70900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f80e850 mi_switch() at mi_switch+0x238/frame 0xffffff823f80e8a0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f80e8e0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f80e940 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f80e970 _sleep() at _sleep+0x3c3/frame 0xffffff823f80ea00 kern_nanosleep() at kern_nanosleep+0x118/frame 0xffffff823f80ea70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff823f80ead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f80ebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f80ebf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800913baa, rsp = 0x7fffffffdb48, rbp = 0x7fffffffdbd0 --- Tracing command sh pid 11392 tid 100237 td 0xfffffe0015db6900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fa4d670 mi_switch() at mi_switch+0x238/frame 0xffffff823fa4d6c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fa4d700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823fa4d760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823fa4d780 _sleep() at _sleep+0x37d/frame 0xffffff823fa4d810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823fa4d8b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823fa4da10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823fa4dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fa4dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fa4dbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd558, rbp = 0x1 --- Tracing command run pid 9916 tid 100205 td 0xfffffe013af4d480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f9ad630 mi_switch() at mi_switch+0x238/frame 0xffffff823f9ad680 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f9ad6c0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f9ad720 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f9ad740 _sleep() at _sleep+0x37d/frame 0xffffff823f9ad7d0 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f9ad870 kern_wait() at kern_wait+0x9c/frame 0xffffff823f9ad9d0 freebsd32_wait4() at freebsd32_wait4+0x35/frame 0xffffff823f9adad0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff823f9adbf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff823f9adbf0 --- syscall (7, FreeBSD ELF32, freebsd32_wait4), rip = 0x280eeb57, rsp = 0xffffc144, rbp = 0xffffc1f0 --- Tracing command run pid 9915 tid 100138 td 0xfffffe0008f4b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f85e630 mi_switch() at mi_switch+0x238/frame 0xffffff823f85e680 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f85e6c0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f85e720 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f85e740 _sleep() at _sleep+0x37d/frame 0xffffff823f85e7d0 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f85e870 kern_wait() at kern_wait+0x9c/frame 0xffffff823f85e9d0 freebsd32_wait4() at freebsd32_wait4+0x35/frame 0xffffff823f85ead0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff823f85ebf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff823f85ebf0 --- syscall (7, FreeBSD ELF32, freebsd32_wait4), rip = 0x280eeb57, rsp = 0xffffc6e4, rbp = 0xffffc700 --- Tracing command run pid 9914 tid 100266 td 0xfffffe018167a480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fade840 mi_switch() at mi_switch+0x238/frame 0xffffff823fade890 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fade8d0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823fade930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823fade960 _sleep() at _sleep+0x3c3/frame 0xffffff823fade9f0 kern_nanosleep() at kern_nanosleep+0x118/frame 0xffffff823fadea60 freebsd32_nanosleep() at freebsd32_nanosleep+0x71/frame 0xffffff823fadead0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff823fadebf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff823fadebf0 --- syscall (240, FreeBSD ELF32, freebsd32_nanosleep), rip = 0x2816f99f, rsp = 0xffffc6d4, rbp = 0xffffc700 --- Tracing command sh pid 9905 tid 100257 td 0xfffffe00155ae900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fab1670 mi_switch() at mi_switch+0x238/frame 0xffffff823fab16c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fab1700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823fab1760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823fab1780 _sleep() at _sleep+0x37d/frame 0xffffff823fab1810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823fab18b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823fab1a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823fab1ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fab1bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fab1bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd288, rbp = 0x1 --- Tracing command md5 pid 9890 tid 100233 td 0xfffffe0177193000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fa39a10 mi_switch() at mi_switch+0x238/frame 0xffffff823fa39a60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fa39aa0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823fa39ad0 _sleep() at _sleep+0x3e9/frame 0xffffff823fa39b60 md_kthread() at md_kthread+0x17b/frame 0xffffff823fa39ba0 fork_exit() at fork_exit+0x139/frame 0xffffff823fa39bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823fa39bf0 --- trap 0, rip = 0, rsp = 0xffffff823fa39cb0, rbp = 0 --- Tracing command sh pid 1131 tid 100094 td 0xfffffe0008e34480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f782670 mi_switch() at mi_switch+0x238/frame 0xffffff823f7826c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f782700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f782760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f782780 _sleep() at _sleep+0x37d/frame 0xffffff823f782810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f7828b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f782a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f782ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f782bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f782bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd408, rbp = 0x1 --- Tracing command bash pid 1127 tid 100145 td 0xfffffe001566a000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f881670 mi_switch() at mi_switch+0x238/frame 0xffffff823f8816c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f881700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f881760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f881780 _sleep() at _sleep+0x37d/frame 0xffffff823f881810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f8818b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f881a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f881ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f881bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f881bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010a12aa, rsp = 0x7fffffffd648, rbp = 0x801922e40 --- Tracing command csh pid 1124 tid 100144 td 0xfffffe0015162000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f87c8b0 mi_switch() at mi_switch+0x238/frame 0xffffff823f87c900 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f87c940 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f87c9a0 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f87c9c0 _sleep() at _sleep+0x37d/frame 0xffffff823f87ca50 kern_sigsuspend() at kern_sigsuspend+0xaa/frame 0xffffff823f87caa0 sys_sigsuspend() at sys_sigsuspend+0x34/frame 0xffffff823f87cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f87cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f87cbf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800d4509a, rsp = 0x7fffffffcd88, rbp = 0x80189a400 --- Tracing command su pid 1108 tid 100133 td 0xfffffe0015a39000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f845670 mi_switch() at mi_switch+0x238/frame 0xffffff823f8456c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f845700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f845760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f845780 _sleep() at _sleep+0x37d/frame 0xffffff823f845810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f8458b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f845a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f845ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f845bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f845bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ed42aa, rsp = 0x7fffffffd458, rbp = 0x464 --- Tracing command bash pid 1104 tid 100129 td 0xfffffe0008e7f900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f831670 mi_switch() at mi_switch+0x238/frame 0xffffff823f8316c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f831700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f831760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f831780 _sleep() at _sleep+0x37d/frame 0xffffff823f831810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f8318b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f831a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f831ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f831bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f831bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010a12aa, rsp = 0x7fffffffd8c8, rbp = 0x8018c5280 --- Tracing command sshd pid 1103 tid 100084 td 0xfffffe0008e36480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f750690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7506e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f750720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f750780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7507a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f750800 seltdwait() at seltdwait+0xad/frame 0xffffff823f750830 kern_select() at kern_select+0x79f/frame 0xffffff823f750a80 sys_select() at sys_select+0x5d/frame 0xffffff823f750ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f750bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f750bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1101 tid 100103 td 0xfffffe0008c6e480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7af760 mi_switch() at mi_switch+0x238/frame 0xffffff823f7af7b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7af7f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7af850 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7af870 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7af8d0 seltdwait() at seltdwait+0xad/frame 0xffffff823f7af900 sys_poll() at sys_poll+0x3e6/frame 0xffffff823f7afad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7afbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7afbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command awk pid 1085 tid 100106 td 0xfffffe0008c6e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7be800 mi_switch() at mi_switch+0x238/frame 0xffffff823f7be850 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7be890 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7be8f0 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7be910 _sleep() at _sleep+0x37d/frame 0xffffff823f7be9a0 pipe_read() at pipe_read+0x432/frame 0xffffff823f7be9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f7bea40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f7bea80 sys_read() at sys_read+0x64/frame 0xffffff823f7bead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7bebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7bebf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7792a, rsp = 0x7fffffffd848, rbp = 0x800db5d60 --- Tracing command sh pid 1084 tid 100085 td 0xfffffe0008e36000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f755670 mi_switch() at mi_switch+0x238/frame 0xffffff823f7556c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f755700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f755760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f755780 _sleep() at _sleep+0x37d/frame 0xffffff823f755810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f7558b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f755a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f755ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f755bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f755bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd678, rbp = 0x1 --- Tracing command top pid 1083 tid 100116 td 0xfffffe0008f7e480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7f0680 mi_switch() at mi_switch+0x238/frame 0xffffff823f7f06d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7f0710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7f0770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7f07a0 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f7f0800 seltdwait() at seltdwait+0x57/frame 0xffffff823f7f0830 kern_select() at kern_select+0x79f/frame 0xffffff823f7f0a80 sys_select() at sys_select+0x5d/frame 0xffffff823f7f0ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7f0bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7f0bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fb98aa, rsp = 0x7fffffffd988, rbp = 0xe --- Tracing command tail pid 1082 tid 100128 td 0xfffffe0008e84000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f82c680 mi_switch() at mi_switch+0x238/frame 0xffffff823f82c6d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f82c710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f82c770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f82c7a0 _sleep() at _sleep+0x3c3/frame 0xffffff823f82c830 kern_kevent() at kern_kevent+0x33a/frame 0xffffff823f82ca10 sys_kevent() at sys_kevent+0x90/frame 0xffffff823f82cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f82cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f82cbf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800918b2a, rsp = 0x7fffffffd9e8, rbp = 0x8010060a8 --- Tracing command sh pid 1081 tid 100107 td 0xfffffe0008c6d900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7c3670 mi_switch() at mi_switch+0x238/frame 0xffffff823f7c36c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7c3700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7c3760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7c3780 _sleep() at _sleep+0x37d/frame 0xffffff823f7c3810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823f7c38b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823f7c3a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823f7c3ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7c3bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7c3bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd938, rbp = 0x1 --- Tracing command sshd pid 1080 tid 100127 td 0xfffffe00154ce480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f827690 mi_switch() at mi_switch+0x238/frame 0xffffff823f8276e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f827720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f827780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f8277a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f827800 seltdwait() at seltdwait+0xad/frame 0xffffff823f827830 kern_select() at kern_select+0x79f/frame 0xffffff823f827a80 sys_select() at sys_select+0x5d/frame 0xffffff823f827ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f827bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f827bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1079 tid 100083 td 0xfffffe0008e85900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f74b690 mi_switch() at mi_switch+0x238/frame 0xffffff823f74b6e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f74b720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f74b780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f74b7a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f74b800 seltdwait() at seltdwait+0xad/frame 0xffffff823f74b830 kern_select() at kern_select+0x79f/frame 0xffffff823f74ba80 sys_select() at sys_select+0x5d/frame 0xffffff823f74bad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f74bbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f74bbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1078 tid 100126 td 0xfffffe0008e84480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f822690 mi_switch() at mi_switch+0x238/frame 0xffffff823f8226e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f822720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f822780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f8227a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f822800 seltdwait() at seltdwait+0xad/frame 0xffffff823f822830 kern_select() at kern_select+0x79f/frame 0xffffff823f822a80 sys_select() at sys_select+0x5d/frame 0xffffff823f822ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f822bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f822bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1074 tid 100113 td 0xfffffe001531a480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7e1760 mi_switch() at mi_switch+0x238/frame 0xffffff823f7e17b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7e17f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7e1850 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7e1870 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7e18d0 seltdwait() at seltdwait+0xad/frame 0xffffff823f7e1900 sys_poll() at sys_poll+0x3e6/frame 0xffffff823f7e1ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7e1bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7e1bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command sshd pid 1073 tid 100114 td 0xfffffe001531a000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7e6760 mi_switch() at mi_switch+0x238/frame 0xffffff823f7e67b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7e67f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7e6850 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7e6870 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7e68d0 seltdwait() at seltdwait+0xad/frame 0xffffff823f7e6900 sys_poll() at sys_poll+0x3e6/frame 0xffffff823f7e6ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7e6bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7e6bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command sshd pid 1072 tid 100100 td 0xfffffe0008f3e900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7a0760 mi_switch() at mi_switch+0x238/frame 0xffffff823f7a07b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7a07f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7a0850 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7a0870 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7a08d0 seltdwait() at seltdwait+0xad/frame 0xffffff823f7a0900 sys_poll() at sys_poll+0x3e6/frame 0xffffff823f7a0ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7a0bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7a0bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command getty pid 1071 tid 100101 td 0xfffffe0008c6f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7a5710 mi_switch() at mi_switch+0x238/frame 0xffffff823f7a5760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7a57a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7a5800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7a5820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7a5880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f7a58b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f7a5950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f7a5980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f7a59f0 dofileread() at dofileread+0xa1/frame 0xffffff823f7a5a40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f7a5a80 sys_read() at sys_read+0x64/frame 0xffffff823f7a5ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7a5bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7a5bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1070 tid 100090 td 0xfffffe0008e35000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f76e710 mi_switch() at mi_switch+0x238/frame 0xffffff823f76e760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f76e7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f76e800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f76e820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f76e880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f76e8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f76e950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f76e980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f76e9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f76ea40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f76ea80 sys_read() at sys_read+0x64/frame 0xffffff823f76ead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f76ebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f76ebf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1069 tid 100086 td 0xfffffe0008e35900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f75a710 mi_switch() at mi_switch+0x238/frame 0xffffff823f75a760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f75a7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f75a800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f75a820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f75a880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f75a8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f75a950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f75a980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f75a9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f75aa40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f75aa80 sys_read() at sys_read+0x64/frame 0xffffff823f75aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f75abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f75abf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1068 tid 100087 td 0xfffffe0008f41000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f75f710 mi_switch() at mi_switch+0x238/frame 0xffffff823f75f760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f75f7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f75f800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f75f820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f75f880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f75f8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f75f950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f75f980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f75f9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f75fa40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f75fa80 sys_read() at sys_read+0x64/frame 0xffffff823f75fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f75fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f75fbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1067 tid 100104 td 0xfffffe0008f3e480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7b4710 mi_switch() at mi_switch+0x238/frame 0xffffff823f7b4760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7b47a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7b4800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7b4820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7b4880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f7b48b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f7b4950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f7b4980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f7b49f0 dofileread() at dofileread+0xa1/frame 0xffffff823f7b4a40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f7b4a80 sys_read() at sys_read+0x64/frame 0xffffff823f7b4ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7b4bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7b4bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1066 tid 100115 td 0xfffffe0008f7e900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7eb710 mi_switch() at mi_switch+0x238/frame 0xffffff823f7eb760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7eb7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7eb800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7eb820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7eb880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f7eb8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f7eb950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f7eb980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f7eb9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f7eba40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f7eba80 sys_read() at sys_read+0x64/frame 0xffffff823f7ebad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7ebbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7ebbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1065 tid 100099 td 0xfffffe0008f3f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f79b710 mi_switch() at mi_switch+0x238/frame 0xffffff823f79b760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f79b7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f79b800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f79b820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f79b880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f79b8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f79b950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f79b980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f79b9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f79ba40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f79ba80 sys_read() at sys_read+0x64/frame 0xffffff823f79bad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f79bbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f79bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1064 tid 100077 td 0xfffffe0008e87900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f72d710 mi_switch() at mi_switch+0x238/frame 0xffffff823f72d760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f72d7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f72d800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f72d820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f72d880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f72d8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f72d950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f72d980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f72d9f0 dofileread() at dofileread+0xa1/frame 0xffffff823f72da40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f72da80 sys_read() at sys_read+0x64/frame 0xffffff823f72dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f72dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f72dbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1063 tid 100076 td 0xfffffe0008e36900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f728710 mi_switch() at mi_switch+0x238/frame 0xffffff823f728760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7287a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f728800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f728820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f728880 tty_wait() at tty_wait+0x4c/frame 0xffffff823f7288b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823f728950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823f728980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823f7289f0 dofileread() at dofileread+0xa1/frame 0xffffff823f728a40 kern_readv() at kern_readv+0x6c/frame 0xffffff823f728a80 sys_read() at sys_read+0x64/frame 0xffffff823f728ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f728bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f728bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command inetd pid 1024 tid 100096 td 0xfffffe0008f40000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f78c690 mi_switch() at mi_switch+0x238/frame 0xffffff823f78c6e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f78c720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f78c780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f78c7a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f78c800 seltdwait() at seltdwait+0xad/frame 0xffffff823f78c830 kern_select() at kern_select+0x79f/frame 0xffffff823f78ca80 sys_select() at sys_select+0x5d/frame 0xffffff823f78cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f78cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f78cbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800f608aa, rsp = 0x7fffffffcd68, rbp = 0x1 --- Tracing command moused pid 991 tid 100110 td 0xfffffe0008c6c900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7d2690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7d26e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7d2720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7d2780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7d27a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7d2800 seltdwait() at seltdwait+0xad/frame 0xffffff823f7d2830 kern_select() at kern_select+0x79f/frame 0xffffff823f7d2a80 sys_select() at sys_select+0x5d/frame 0xffffff823f7d2ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7d2bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7d2bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d728aa, rsp = 0x7fffffffd858, rbp = 0x7fffffffdf31 --- Tracing command watchdogd pid 971 tid 100080 td 0xfffffe0008e86900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f73c850 mi_switch() at mi_switch+0x238/frame 0xffffff823f73c8a0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f73c8e0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f73c940 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f73c970 _sleep() at _sleep+0x3c3/frame 0xffffff823f73ca00 kern_nanosleep() at kern_nanosleep+0x118/frame 0xffffff823f73ca70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff823f73cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f73cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f73cbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b27baa, rsp = 0x7fffffffdba8, rbp = 0x7fffffffdbf0 --- Tracing command cron pid 961 tid 100093 td 0xfffffe0008e85000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f77d850 mi_switch() at mi_switch+0x238/frame 0xffffff823f77d8a0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f77d8e0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f77d940 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f77d970 _sleep() at _sleep+0x3c3/frame 0xffffff823f77da00 kern_nanosleep() at kern_nanosleep+0x118/frame 0xffffff823f77da70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff823f77dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f77dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f77dbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d39baa, rsp = 0x7fffffffdac8, rbp = 0x3c --- Tracing command sendmail pid 954 tid 100117 td 0xfffffe0008f7e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7f58b0 mi_switch() at mi_switch+0x238/frame 0xffffff823f7f5900 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7f5940 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7f59a0 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7f59c0 _sleep() at _sleep+0x37d/frame 0xffffff823f7f5a50 kern_sigsuspend() at kern_sigsuspend+0xaa/frame 0xffffff823f7f5aa0 sys_sigsuspend() at sys_sigsuspend+0x34/frame 0xffffff823f7f5ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7f5bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7f5bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80139609a, rsp = 0x7fffffffbda8, rbp = 0x1 --- Tracing command sendmail pid 950 tid 100082 td 0xfffffe0008e86000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f746680 mi_switch() at mi_switch+0x238/frame 0xffffff823f7466d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f746710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f746770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7467a0 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f746800 seltdwait() at seltdwait+0x57/frame 0xffffff823f746830 kern_select() at kern_select+0x79f/frame 0xffffff823f746a80 sys_select() at sys_select+0x5d/frame 0xffffff823f746ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f746bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f746bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8014498aa, rsp = 0x7fffffffb218, rbp = 0x7fffffffb2b0 --- Tracing command sshd pid 942 tid 100089 td 0xfffffe0008e35480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f769690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7696e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f769720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f769780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7697a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f769800 seltdwait() at seltdwait+0xad/frame 0xffffff823f769830 kern_select() at kern_select+0x79f/frame 0xffffff823f769a80 sys_select() at sys_select+0x5d/frame 0xffffff823f769ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f769bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f769bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcd78, rbp = 0x2 --- Tracing command ntpd pid 851 tid 100109 td 0xfffffe0008c6d000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7cd690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7cd6e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7cd720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7cd780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7cd7a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f7cd800 seltdwait() at seltdwait+0xad/frame 0xffffff823f7cd830 kern_select() at kern_select+0x79f/frame 0xffffff823f7cda80 sys_select() at sys_select+0x5d/frame 0xffffff823f7cdad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7cdbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7cdbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8013a98aa, rsp = 0x7fffffffdbc8, rbp = 0x7fffffffdd08 --- Tracing command nfsd pid 756 tid 100120 td 0xfffffe0008e30480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f8048e0 mi_switch() at mi_switch+0x238/frame 0xffffff823f804930 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f804970 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f8049d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f804a00 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f804a60 svc_run_internal() at svc_run_internal+0x895/frame 0xffffff823f804b90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff823f804ba0 fork_exit() at fork_exit+0x139/frame 0xffffff823f804bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823f804bf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 756 tid 100119 td 0xfffffe0008e30900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7ff8e0 mi_switch() at mi_switch+0x238/frame 0xffffff823f7ff930 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7ff970 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7ff9d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7ffa00 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f7ffa60 svc_run_internal() at svc_run_internal+0x895/frame 0xffffff823f7ffb90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff823f7ffba0 fork_exit() at fork_exit+0x139/frame 0xffffff823f7ffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823f7ffbf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 756 tid 100118 td 0xfffffe0008e34000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7fa8e0 mi_switch() at mi_switch+0x238/frame 0xffffff823f7fa930 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7fa970 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7fa9d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7faa00 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f7faa60 svc_run_internal() at svc_run_internal+0x895/frame 0xffffff823f7fab90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff823f7faba0 fork_exit() at fork_exit+0x139/frame 0xffffff823f7fabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823f7fabf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 756 tid 100111 td 0xfffffe0008c6c480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7d7070 mi_switch() at mi_switch+0x238/frame 0xffffff823f7d70c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7d7100 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f7d7160 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7d7190 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f7d71f0 svc_run_internal() at svc_run_internal+0x895/frame 0xffffff823f7d7320 svc_run() at svc_run+0x94/frame 0xffffff823f7d7340 nfsrvd_nfsd() at nfsrvd_nfsd+0x1c7/frame 0xffffff823f7d7490 nfssvc_nfsd() at nfssvc_nfsd+0x9b/frame 0xffffff823f7d7ab0 sys_nfssvc() at sys_nfssvc+0xb0/frame 0xffffff823f7d7ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7d7bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7d7bf0 --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 755 tid 100078 td 0xfffffe0008e87480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f732690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7326e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f732720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f732780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7327a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f732800 seltdwait() at seltdwait+0xad/frame 0xffffff823f732830 kern_select() at kern_select+0x79f/frame 0xffffff823f732a80 sys_select() at sys_select+0x5d/frame 0xffffff823f732ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f732bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f732bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8009388aa, rsp = 0x7fffffffd928, rbp = 0x7fffffffdc00 --- Tracing command mountd pid 746 tid 100088 td 0xfffffe0008f40900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f764690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7646e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f764720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f764780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7647a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f764800 seltdwait() at seltdwait+0xad/frame 0xffffff823f764830 kern_select() at kern_select+0x79f/frame 0xffffff823f764a80 sys_select() at sys_select+0x5d/frame 0xffffff823f764ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f764bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f764bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b508aa, rsp = 0x7fffffffdb78, rbp = 0x801419060 --- Tracing command rpcbind pid 641 tid 100091 td 0xfffffe0008e85480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f773750 mi_switch() at mi_switch+0x238/frame 0xffffff823f7737a0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7737e0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f773840 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f773870 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f7738d0 seltdwait() at seltdwait+0x57/frame 0xffffff823f773900 sys_poll() at sys_poll+0x3e6/frame 0xffffff823f773ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f773bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f773bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800cf350a, rsp = 0x7fffffffba48, rbp = 0x80141b020 --- Tracing command syslogd pid 615 tid 100098 td 0xfffffe0008f3f480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f796690 mi_switch() at mi_switch+0x238/frame 0xffffff823f7966e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f796720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f796780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823f7967a0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823f796800 seltdwait() at seltdwait+0xad/frame 0xffffff823f796830 kern_select() at kern_select+0x79f/frame 0xffffff823f796a80 sys_select() at sys_select+0x5d/frame 0xffffff823f796ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f796bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f796bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b4e8aa, rsp = 0x7fffffffd118, rbp = 0x80142c108 --- Tracing command devd pid 434 tid 100092 td 0xfffffe0008e34900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f778680 mi_switch() at mi_switch+0x238/frame 0xffffff823f7786d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f778710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823f778770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823f7787a0 _cv_timedwait_sig() at _cv_timedwait_sig+0x18f/frame 0xffffff823f778800 seltdwait() at seltdwait+0x57/frame 0xffffff823f778830 kern_select() at kern_select+0x79f/frame 0xffffff823f778a80 sys_select() at sys_select+0x5d/frame 0xffffff823f778ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f778bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f778bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x442aaa, rsp = 0x7fffffffd828, rbp = 0x10 --- Tracing command softdepflush pid 18 tid 100075 td 0xfffffe0008c6f480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbbca10 mi_switch() at mi_switch+0x238/frame 0xffffff822cbbca60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbbcaa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbbcad0 _sleep() at _sleep+0x29a/frame 0xffffff822cbbcb60 softdep_flush() at softdep_flush+0x20d/frame 0xffffff822cbbcba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbbcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbbcbf0 --- trap 0, rip = 0, rsp = 0xffffff822cbbccb0, rbp = 0 --- Tracing command vnlru pid 17 tid 100074 td 0xfffffe0008c6f900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbb79d0 mi_switch() at mi_switch+0x238/frame 0xffffff822cbb7a20 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbb7a60 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbb7a90 _sleep() at _sleep+0x29a/frame 0xffffff822cbb7b20 vnlru_proc() at vnlru_proc+0x537/frame 0xffffff822cbb7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbb7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbb7bf0 --- trap 0, rip = 0, rsp = 0xffffff822cbb7cb0, rbp = 0 --- Tracing command syncer pid 16 tid 100073 td 0xfffffe0008c70000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbb29f0 mi_switch() at mi_switch+0x238/frame 0xffffff822cbb2a40 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbb2a80 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbb2ab0 _cv_timedwait() at _cv_timedwait+0x18f/frame 0xffffff822cbb2b10 sched_sync() at sched_sync+0x4ee/frame 0xffffff822cbb2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbb2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbb2bf0 --- trap 0, rip = 0, rsp = 0xffffff822cbb2cb0, rbp = 0 --- Tracing command bufdaemon pid 9 tid 100072 td 0xfffffe00055ee000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbada30 mi_switch() at mi_switch+0x238/frame 0xffffff822cbada80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbadac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbadaf0 _sleep() at _sleep+0x29a/frame 0xffffff822cbadb80 buf_daemon() at buf_daemon+0x192/frame 0xffffff822cbadba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbadbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbadbf0 --- trap 0, rip = 0, rsp = 0xffffff822cbadcb0, rbp = 0 --- Tracing command pagezero pid 8 tid 100071 td 0xfffffe00055ee480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cba8a30 mi_switch() at mi_switch+0x238/frame 0xffffff822cba8a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cba8ac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cba8af0 _sleep() at _sleep+0x29a/frame 0xffffff822cba8b80 vm_pagezero() at vm_pagezero+0x73/frame 0xffffff822cba8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cba8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cba8bf0 --- trap 0, rip = 0, rsp = 0xffffff822cba8cb0, rbp = 0 --- Tracing command vmdaemon pid 7 tid 100070 td 0xfffffe00055ee900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cba39e0 mi_switch() at mi_switch+0x238/frame 0xffffff822cba3a30 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cba3a70 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822cba3aa0 _sleep() at _sleep+0x3e9/frame 0xffffff822cba3b30 vm_daemon() at vm_daemon+0x4d/frame 0xffffff822cba3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cba3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cba3bf0 --- trap 0, rip = 0, rsp = 0xffffff822cba3cb0, rbp = 0 --- Tracing command pagedaemon pid 6 tid 100069 td 0xfffffe00055f1000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cb9e940 mi_switch() at mi_switch+0x238/frame 0xffffff822cb9e990 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cb9e9d0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cb9ea00 _sleep() at _sleep+0x29a/frame 0xffffff822cb9ea90 vm_pageout() at vm_pageout+0xb8f/frame 0xffffff822cb9eba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cb9ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cb9ebf0 --- trap 0, rip = 0, rsp = 0xffffff822cb9ecb0, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100068 td 0xfffffe00055f1480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca19a30 mi_switch() at mi_switch+0x238/frame 0xffffff822ca19a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca19ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca19af0 _sleep() at _sleep+0x3e9/frame 0xffffff822ca19b80 xpt_scanner_thread() at xpt_scanner_thread+0xdd/frame 0xffffff822ca19ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca19bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca19bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca19cb0, rbp = 0 --- Tracing command sctp_iterator pid 4 tid 100067 td 0xfffffe00055f1900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca14a40 mi_switch() at mi_switch+0x238/frame 0xffffff822ca14a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca14ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca14b00 _sleep() at _sleep+0x3e9/frame 0xffffff822ca14b90 sctp_iterator_thread() at sctp_iterator_thread+0x3f/frame 0xffffff822ca14ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca14bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca14bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca14cb0, rbp = 0 --- Tracing command ctl_thrd pid 3 tid 100064 td 0xfffffe00055f3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca043f0 mi_switch() at mi_switch+0x238/frame 0xffffff822ca04440 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca04480 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca044b0 _sleep() at _sleep+0x3e9/frame 0xffffff822ca04540 ctl_work_thread() at ctl_work_thread+0x1ce8/frame 0xffffff822ca04ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca04bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca04bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca04cb0, rbp = 0 --- Tracing command fdc0 pid 2 tid 100061 td 0xfffffe00055f4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9f1990 mi_switch() at mi_switch+0x238/frame 0xffffff822c9f19e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9f1a20 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822c9f1a50 _sleep() at _sleep+0x29a/frame 0xffffff822c9f1ae0 fdc_thread() at fdc_thread+0x7f4/frame 0xffffff822c9f1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9f1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9f1bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9f1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100058 td 0xfffffe00055c3000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9aba50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9abaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9abae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9abb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9abb70 usb_process() at usb_process+0x172/frame 0xffffff822c9abba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9abbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9abbf0 --- trap 0, rip = 0, rsp = 0xffffff822c9abcb0, rbp = 0 --- Tracing command usb pid 15 tid 100057 td 0xfffffe00055c3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9a6a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9a6aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9a6ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9a6b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9a6b70 usb_process() at usb_process+0x172/frame 0xffffff822c9a6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9a6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9a6bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9a6cb0, rbp = 0 --- Tracing command usb pid 15 tid 100056 td 0xfffffe00055c3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9a1a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9a1aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9a1ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9a1b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9a1b70 usb_process() at usb_process+0x172/frame 0xffffff822c9a1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9a1bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9a1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100055 td 0xfffffe00055c4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c99ca50 mi_switch() at mi_switch+0x238/frame 0xffffff822c99caa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c99cae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c99cb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c99cb70 usb_process() at usb_process+0x172/frame 0xffffff822c99cba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c99cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c99cbf0 --- trap 0, rip = 0, rsp = 0xffffff822c99ccb0, rbp = 0 --- Tracing command usb pid 15 tid 100053 td 0xfffffe00055c4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c80fa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c80faa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c80fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c80fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c80fb70 usb_process() at usb_process+0x172/frame 0xffffff822c80fba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c80fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c80fbf0 --- trap 0, rip = 0, rsp = 0xffffff822c80fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100052 td 0xfffffe00055c5000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c80aa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c80aaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c80aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c80ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c80ab70 usb_process() at usb_process+0x172/frame 0xffffff822c80aba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c80abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c80abf0 --- trap 0, rip = 0, rsp = 0xffffff822c80acb0, rbp = 0 --- Tracing command usb pid 15 tid 100051 td 0xfffffe00055c5480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c805a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c805aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c805ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c805b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c805b70 usb_process() at usb_process+0x172/frame 0xffffff822c805ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c805bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c805bf0 --- trap 0, rip = 0, rsp = 0xffffff822c805cb0, rbp = 0 --- Tracing command usb pid 15 tid 100050 td 0xfffffe00055c5900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c800a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c800aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c800ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c800b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c800b70 usb_process() at usb_process+0x172/frame 0xffffff822c800ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c800bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c800bf0 --- trap 0, rip = 0, rsp = 0xffffff822c800cb0, rbp = 0 --- Tracing command usb pid 15 tid 100049 td 0xfffffe00055b7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7b7a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7b7aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7b7ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7b7b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7b7b70 usb_process() at usb_process+0x172/frame 0xffffff822c7b7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7b7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7b7bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7b7cb0, rbp = 0 --- Tracing command usb pid 15 tid 100048 td 0xfffffe00055b7480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7b2a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7b2aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7b2ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7b2b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7b2b70 usb_process() at usb_process+0x172/frame 0xffffff822c7b2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7b2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7b2bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7b2cb0, rbp = 0 --- Tracing command usb pid 15 tid 100047 td 0xfffffe00055b7900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7ada50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7adaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7adae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7adb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7adb70 usb_process() at usb_process+0x172/frame 0xffffff822c7adba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7adbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7adbf0 --- trap 0, rip = 0, rsp = 0xffffff822c7adcb0, rbp = 0 --- Tracing command usb pid 15 tid 100046 td 0xfffffe00055b8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7a8a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7a8aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7a8ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7a8b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7a8b70 usb_process() at usb_process+0x172/frame 0xffffff822c7a8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7a8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7a8bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7a8cb0, rbp = 0 --- Tracing command usb pid 15 tid 100045 td 0xfffffe00055b8480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c75fa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c75faa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c75fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c75fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c75fb70 usb_process() at usb_process+0x172/frame 0xffffff822c75fba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c75fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c75fbf0 --- trap 0, rip = 0, rsp = 0xffffff822c75fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100044 td 0xfffffe00055b8900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c75aa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c75aaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c75aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c75ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c75ab70 usb_process() at usb_process+0x172/frame 0xffffff822c75aba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c75abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c75abf0 --- trap 0, rip = 0, rsp = 0xffffff822c75acb0, rbp = 0 --- Tracing command usb pid 15 tid 100043 td 0xfffffe00055ba000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c755a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c755aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c755ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c755b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c755b70 usb_process() at usb_process+0x172/frame 0xffffff822c755ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c755bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c755bf0 --- trap 0, rip = 0, rsp = 0xffffff822c755cb0, rbp = 0 --- Tracing command usb pid 15 tid 100042 td 0xfffffe00055ba480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c750a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c750aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c750ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c750b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c750b70 usb_process() at usb_process+0x172/frame 0xffffff822c750ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c750bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c750bf0 --- trap 0, rip = 0, rsp = 0xffffff822c750cb0, rbp = 0 --- Tracing command usb pid 15 tid 100040 td 0xfffffe00055bc000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c702a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c702aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c702ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c702b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c702b70 usb_process() at usb_process+0x172/frame 0xffffff822c702ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c702bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c702bf0 --- trap 0, rip = 0, rsp = 0xffffff822c702cb0, rbp = 0 --- Tracing command usb pid 15 tid 100039 td 0xfffffe00055b0000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6fda50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6fdaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6fdae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6fdb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6fdb70 usb_process() at usb_process+0x172/frame 0xffffff822c6fdba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6fdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6fdbf0 --- trap 0, rip = 0, rsp = 0xffffff822c6fdcb0, rbp = 0 --- Tracing command usb pid 15 tid 100038 td 0xfffffe00055b0480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6f8a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6f8aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6f8ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6f8b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6f8b70 usb_process() at usb_process+0x172/frame 0xffffff822c6f8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6f8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6f8bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6f8cb0, rbp = 0 --- Tracing command usb pid 15 tid 100037 td 0xfffffe00055b0900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6f3a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6f3aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6f3ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6f3b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6f3b70 usb_process() at usb_process+0x172/frame 0xffffff822c6f3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6f3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6f3bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6f3cb0, rbp = 0 --- Tracing command usb pid 15 tid 100035 td 0xfffffe00055b1480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6a5a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6a5aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6a5ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6a5b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6a5b70 usb_process() at usb_process+0x172/frame 0xffffff822c6a5ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6a5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6a5bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6a5cb0, rbp = 0 --- Tracing command usb pid 15 tid 100034 td 0xfffffe00055b1900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6a0a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6a0aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6a0ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6a0b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6a0b70 usb_process() at usb_process+0x172/frame 0xffffff822c6a0ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6a0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6a0bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6a0cb0, rbp = 0 --- Tracing command usb pid 15 tid 100033 td 0xfffffe00055b2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c69ba50 mi_switch() at mi_switch+0x238/frame 0xffffff822c69baa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c69bae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c69bb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c69bb70 usb_process() at usb_process+0x172/frame 0xffffff822c69bba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c69bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c69bbf0 --- trap 0, rip = 0, rsp = 0xffffff822c69bcb0, rbp = 0 --- Tracing command usb pid 15 tid 100032 td 0xfffffe00055b2480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c696a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c696aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c696ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c696b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c696b70 usb_process() at usb_process+0x172/frame 0xffffff822c696ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c696bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c696bf0 --- trap 0, rip = 0, rsp = 0xffffff822c696cb0, rbp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xfffffe0005209900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000283a10 mi_switch() at mi_switch+0x238/frame 0xffffff8000283a60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000283aa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8000283ad0 _sleep() at _sleep+0x29a/frame 0xffffff8000283b60 random_kthread() at random_kthread+0x1ad/frame 0xffffff8000283ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000283bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000283bf0 --- trap 0, rip = 0, rsp = 0xffffff8000283cb0, rbp = 0 --- Tracing command geom pid 13 tid 100015 td 0xfffffe00051f2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000279a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000279a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000279ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000279ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000279b70 g_io_schedule_down() at g_io_schedule_down+0x25f/frame 0xffffff8000279b90 g_down_procbody() at g_down_procbody+0x7c/frame 0xffffff8000279ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000279bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000279bf0 --- trap 0, rip = 0, rsp = 0xffffff8000279cb0, rbp = 0 --- Tracing command geom pid 13 tid 100014 td 0xfffffe00051f2480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000274a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000274a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000274ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000274ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000274b70 g_io_schedule_up() at g_io_schedule_up+0x138/frame 0xffffff8000274b90 g_up_procbody() at g_up_procbody+0x7c/frame 0xffffff8000274ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000274bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000274bf0 --- trap 0, rip = 0, rsp = 0xffffff8000274cb0, rbp = 0 --- Tracing command geom pid 13 tid 100013 td 0xfffffe00051f2900 sched_switch() at sched_switch+0x1b4/frame 0xffffff800026fa20 mi_switch() at mi_switch+0x238/frame 0xffffff800026fa70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800026fab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800026fae0 _sleep() at _sleep+0x3e9/frame 0xffffff800026fb70 g_run_events() at g_run_events+0x449/frame 0xffffff800026fba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026fbf0 --- trap 0, rip = 0, rsp = 0xffffff800026fcb0, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xfffffe00055f4000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xfffffe00055f4480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9f6b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9f6b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9f6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9f6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9f6bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9f6cb0, rbp = 0 --- Tracing command intr pid 12 tid 100060 td 0xfffffe00055bc480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9e2b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9e2b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9e2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9e2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9e2bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9e2cb0, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffffe00055bc900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9d3b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9d3b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9d3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9d3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9d3bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9d3cb0, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffffe00055c4480 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffffe00055ba900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c74bb00 mi_switch() at mi_switch+0x238/frame 0xffffff822c74bb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c74bba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c74bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c74bbf0 --- trap 0, rip = 0, rsp = 0xffffff822c74bcb0, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xfffffe00055b1000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6eeb00 mi_switch() at mi_switch+0x238/frame 0xffffff822c6eeb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c6eeba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6eebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6eebf0 --- trap 0, rip = 0, rsp = 0xffffff822c6eecb0, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xfffffe00055b2900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c691b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c691b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c691ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c691bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c691bf0 --- trap 0, rip = 0, rsp = 0xffffff822c691cb0, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xfffffe000520b480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003ccb00 mi_switch() at mi_switch+0x238/frame 0xffffff80003ccb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80003ccba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003ccbf0 --- trap 0, rip = 0, rsp = 0xffffff80003cccb0, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xfffffe000520b900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003c7b00 mi_switch() at mi_switch+0x238/frame 0xffffff80003c7b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80003c7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003c7bf0 --- trap 0, rip = 0, rsp = 0xffffff80003c7cb0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffffe00053a8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002bbb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002bbb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80002bbba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002bbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bbbf0 --- trap 0, rip = 0, rsp = 0xffffff80002bbcb0, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xfffffe00053a9900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a2b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002a2b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80002a2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a2bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a2cb0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xfffffe0005208000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800029db00 mi_switch() at mi_switch+0x238/frame 0xffffff800029db50 ithread_loop() at ithread_loop+0x273/frame 0xffffff800029dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800029dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800029dbf0 --- trap 0, rip = 0, rsp = 0xffffff800029dcb0, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xfffffe0005208480 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xfffffe0005209000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xfffffe00051f3000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xfffffe00051f3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000265b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000265b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000265ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000265bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000265bf0 --- trap 0, rip = 0, rsp = 0xffffff8000265cb0, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xfffffe00051f3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000260b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000260b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000260ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000260bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000260bf0 --- trap 0, rip = 0, rsp = 0xffffff8000260cb0, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xfffffe00051d7480 sched_switch() at sched_switch+0x1b4/frame 0xffffff800025bb00 mi_switch() at mi_switch+0x238/frame 0xffffff800025bb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff800025bba0 fork_exit() at fork_exit+0x139/frame 0xffffff800025bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800025bbf0 --- trap 0, rip = 0, rsp = 0xffffff800025bcb0, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xfffffe00051d7900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000256b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000256b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000256ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000256bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000256bf0 --- trap 0, rip = 0, rsp = 0xffffff8000256cb0, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xfffffe00051e0000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000251b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000251b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000251ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000251bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000251bf0 --- trap 0, rip = 0, rsp = 0xffffff8000251cb0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00051e0480 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800022cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800022cd20 trap() at trap+0x325/frame 0xffffff800022cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800022cf20 --- trap 0x13, rip = 0xffffffff80c6ea1a, rsp = 0xffffff800022cfe0, rbp = 0xffffff800024cb10 --- cpu_idle_mwait() at cpu_idle_mwait+0x5a/frame 0xffffff800024cb10 sched_idletd() at sched_idletd+0x186/frame 0xffffff800024cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800024cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800024cbf0 --- trap 0, rip = 0, rsp = 0xffffff800024ccb0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00051e0900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000247aa0 mi_switch() at mi_switch+0x238/frame 0xffffff8000247af0 critical_exit() at critical_exit+0xb0/frame 0xffffff8000247b10 sched_idletd() at sched_idletd+0x186/frame 0xffffff8000247ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000247bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000247bf0 --- trap 0, rip = 0, rsp = 0xffffff8000247cb0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00051d4000 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800021ed00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800021ed20 trap() at trap+0x325/frame 0xffffff800021ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800021ef20 --- trap 0x13, rip = 0xffffffff80c6ea1a, rsp = 0xffffff800021efe0, rbp = 0xffffff8000242b10 --- cpu_idle_mwait() at cpu_idle_mwait+0x5a/frame 0xffffff8000242b10 sched_idletd() at sched_idletd+0x186/frame 0xffffff8000242ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000242bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000242bf0 --- trap 0, rip = 0, rsp = 0xffffff8000242cb0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00051d4480 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffffff81578260 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff81578280 trap() at trap+0x325/frame 0xffffffff81578480 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81578480 --- trap 0x13, rip = 0xffffffff80c6ea1a, rsp = 0xffffffff81578540, rbp = 0xffffff800023db10 --- cpu_idle_mwait() at cpu_idle_mwait+0x5a/frame 0xffffff800023db10 sched_idletd() at sched_idletd+0x186/frame 0xffffff800023dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800023dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800023dbf0 --- trap 0, rip = 0, rsp = 0xffffff800023dcb0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00051d4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000238670 mi_switch() at mi_switch+0x238/frame 0xffffff80002386c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000238700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8000238760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8000238780 _sleep() at _sleep+0x37d/frame 0xffffff8000238810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff80002388b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8000238a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8000238ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8000238bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8000238bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41242a, rsp = 0x7fffffffd798, rbp = 0x8a --- Tracing command audit pid 10 tid 100001 td 0xfffffe00051d7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000233a00 mi_switch() at mi_switch+0x238/frame 0xffffff8000233a50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000233a90 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000233ac0 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8000233b20 audit_worker() at audit_worker+0x77/frame 0xffffff8000233ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000233bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000233bf0 --- trap 0, rip = 0, rsp = 0xffffff8000233cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffffe00055f3000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca0ea10 mi_switch() at mi_switch+0x238/frame 0xffffff822ca0ea60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca0eaa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822ca0ead0 _sleep() at _sleep+0x29a/frame 0xffffff822ca0eb60 deadlkres() at deadlkres+0x2c3/frame 0xffffff822ca0eba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca0ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca0ebf0 --- trap 0, rip = 0, rsp = 0xffffff822ca0ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe00055f3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca09a40 mi_switch() at mi_switch+0x238/frame 0xffffff822ca09a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca09ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca09b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff822ca09b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff822ca09ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca09bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca09bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca09cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe00053a8480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b6a40 mi_switch() at mi_switch+0x238/frame 0xffffff80002b6a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002b6ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b6b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002b6b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002b6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b6bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b6cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00053a8900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b1a40 mi_switch() at mi_switch+0x238/frame 0xffffff80002b1a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002b1ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b1b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002b1b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002b1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b1bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b1cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00053a9000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002aca40 mi_switch() at mi_switch+0x238/frame 0xffffff80002aca90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002acad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002acb00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002acb70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002acba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002acbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002acbf0 --- trap 0, rip = 0, rsp = 0xffffff80002accb0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00053a9480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a7a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002a7a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002a7ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002a7ae0 _sleep() at _sleep+0x3e9/frame 0xffffff80002a7b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff80002a7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a7bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a7cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe0005208900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000293a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000293a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000293ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000293ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000293b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff8000293ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000293bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000293bf0 --- trap 0, rip = 0, rsp = 0xffffff8000293cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe0005209480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000289a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000289a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000289ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000289ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000289b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff8000289ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000289bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000289bf0 --- trap 0, rip = 0, rsp = 0xffffff8000289cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe000520b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800027ea20 mi_switch() at mi_switch+0x238/frame 0xffffff800027ea70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800027eab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800027eae0 _sleep() at _sleep+0x3e9/frame 0xffffff800027eb70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff800027eba0 fork_exit() at fork_exit+0x139/frame 0xffffff800027ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800027ebf0 --- trap 0, rip = 0, rsp = 0xffffff800027ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff8135aa50 sched_switch() at sched_switch+0x1b4/frame 0xffffffff818f4b00 mi_switch() at mi_switch+0x238/frame 0xffffffff818f4b50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffffff818f4b90 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffffff818f4bc0 _sleep() at _sleep+0x29a/frame 0xffffffff818f4c50 scheduler() at scheduler+0x2b0/frame 0xffffffff818f4c90 mi_startup() at mi_startup+0x77/frame 0xffffffff818f4cb0 btext() at btext+0x2c db:0:allt> call doadump Dumping 1311 out of 8040 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) bt #0 doadump (textdump=0x51d3000) at ../../../kern/kern_shutdown.c:263 #1 0xffffffff80342fdc in db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:578 #2 0xffffffff8034328d in db_command (last_cmdp=0xffffffff81321fe0, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:449 #3 0xffffffff80347e23 in db_script_exec (scriptname=0xffffffff81322880 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff80343311 in db_command (last_cmdp=0xffffffff81321fe0, cmd_table=, dopager=0x1) at ../../../ddb/db_command.c:449 #5 0xffffffff80343560 in db_command_loop () at ../../../ddb/db_command.c:502 #6 0xffffffff803456d9 in db_trap (type=, code=) at ../../../ddb/db_main.c:231 #7 0xffffffff80918558 in kdb_trap (type=0x3, code=0x0, tf=0xffffff823fb10580) at ../../../kern/subr_kdb.c:654 #8 0xffffffff80c8174d in trap (frame=0xffffff823fb10580) at ../../../amd64/amd64/trap.c:579 #9 0xffffffff80c6a483 in calltrap () at ../../../amd64/amd64/exception.S:228 #10 0xffffffff80917fdb in kdb_enter (why=0xffffffff80ef4fce "panic", msg=0x80
) at cpufunc.h:63 #11 0xffffffff808ddb51 in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:746 #12 0xffffffff808ddc57 in panic (fmt=) at ../../../kern/kern_shutdown.c:682 #13 0xffffffff80b4289d in quotaoff1 (td=0xfffffe013af8b900, mp=, type=0x0) at ../../../ufs/ufs/ufs_quota.c:1615 #14 0xffffffff80b42929 in quotaoff_inchange (td=, mp=0xfffffe0015b2d3c8, type=0x0) at ../../../ufs/ufs/ufs_quota.c:707 #15 0xffffffff80b30392 in ffs_flushfiles (mp=0xfffffe0015b2d3c8, flags=0x2, td=0xfffffe013af8b900) at ../../../ufs/ffs/ffs_vfsops.c:1364 #16 0xffffffff80b2d4c4 in softdep_flushfiles (oldmnt=0xfffffe0015b2d3c8, flags=0x2, td=0xfffffe013af8b900) at ../../../ufs/ffs/ffs_softdep.c:1929 #17 0xffffffff80b31761 in ffs_unmount (mp=0xfffffe0015b2d3c8, mntflags=) at ../../../ufs/ffs/ffs_vfsops.c:1273 #18 0xffffffff809792f9 in dounmount (mp=0xfffffe0015b2d3c8, flags=0x8080000, td=) at ../../../kern/vfs_mount.c:1314 #19 0xffffffff80979a8e in sys_unmount (td=0xfffffe013af8b900, uap=0xffffff823fb10b70) at ../../../kern/vfs_mount.c:1211 #20 0xffffffff80c80523 in amd64_syscall (td=0xfffffe013af8b900, traced=0x0) at subr_syscall.c:134 #21 0xffffffff80c6a767 in Xfast_syscall () at ../../../amd64/amd64/exception.S:387 #22 0x00000008008841fa in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 13 #13 0xffffffff80b4289d in quotaoff1 (td=0xfffffe013af8b900, mp=, type=0x0) at ../../../ufs/ufs/ufs_quota.c:1615 1615 panic("dqflush: stray dquot"); (kgdb) l 1610 for (dq = LIST_FIRST(dqh); dq; dq = nextdq) { 1611 nextdq = LIST_NEXT(dq, dq_hash); 1612 if (dq->dq_ump->um_quotas[dq->dq_type] != vp) 1613 continue; 1614 if (dq->dq_cnt) 1615 panic("dqflush: stray dquot"); 1616 LIST_REMOVE(dq, dq_hash); 1617 dq->dq_ump = (struct ufsmount *)0; 1618 } 1619 } (kgdb) info loc vp = (struct vnode *) 0x0 qvp = (struct vnode *) 0xfffffe017766b000 mvp = (struct vnode *) 0x0 ump = (struct ufsmount *) 0xfffffe0015b72a00 dq = cr = (struct ucred *) 0xfffffe0008eae300 error = (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_alloc.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_alloc.c (revision 247264) +++ /usr/src/sys/ufs/ffs/ffs_alloc.c (working copy) @@ -1790,6 +1790,17 @@ fail: return (0); } +static inline struct buf * +getinobuf(struct inode *ip, u_int cg, u_int32_t cginoblk, int gbflags) +{ + struct fs *fs; + + fs = ip->i_fs; + return (getblk(ip->i_devvp, fsbtodb(fs, ino_to_fsba(fs, + cg * fs->fs_ipg + cginoblk)), (int)fs->fs_bsize, 0, 0, + gbflags)); +} + /* * Determine whether an inode can be allocated. * @@ -1814,9 +1825,11 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) u_int8_t *inosused, *loc; struct ufs2_dinode *dp2; int error, start, len, i; + u_int32_t old_initediblk; fs = ip->i_fs; ump = ip->i_ump; +check_nifree: if (fs->fs_cs(fs, cg).cs_nifree == 0) return (0); UFS_UNLOCK(ump); @@ -1828,13 +1841,13 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) return (0); } cgp = (struct cg *)bp->b_data; +restart: if (!cg_chkmagic(cgp) || cgp->cg_cs.cs_nifree == 0) { brelse(bp); UFS_LOCK(ump); return (0); } bp->b_xflags |= BX_BKGRDWRITE; - cgp->cg_old_time = cgp->cg_time = time_second; inosused = cg_inosused(cgp); if (ipref) { ipref %= fs->fs_ipg; @@ -1856,7 +1869,6 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) } } ipref = (loc - inosused) * NBBY + ffs(~*loc) - 1; - cgp->cg_irotor = ipref; gotit: /* * Check to see if we need to initialize more inodes. @@ -1864,9 +1876,37 @@ gotit: if (fs->fs_magic == FS_UFS2_MAGIC && ipref + INOPB(fs) > cgp->cg_initediblk && cgp->cg_initediblk < cgp->cg_niblk) { - ibp = getblk(ip->i_devvp, fsbtodb(fs, - ino_to_fsba(fs, cg * fs->fs_ipg + cgp->cg_initediblk)), - (int)fs->fs_bsize, 0, 0, 0); + old_initediblk = cgp->cg_initediblk; + + /* + * Free the cylinder group lock before writing the + * initialized inode block. Entering the + * babarrierwrite() with the cylinder group lock + * causes lock order violation between the lock and + * snaplk. + * + * Another thread can decide to initialize the same + * inode block, but whichever thread first gets the + * cylinder group lock after writing the newly + * allocated inode block will update it and the other + * will realize that it has lost and leave the + * cylinder group unchanged. + */ + ibp = getinobuf(ip, cg, old_initediblk, GB_LOCK_NOWAIT); + brelse(bp); + if (ibp == NULL) { + /* + * The inode block buffer is already owned by + * another thread, which must initialize it. + * Wait on the buffer to allow another thread + * to finish the updates, with dropped cg + * buffer lock, then retry. + */ + ibp = getinobuf(ip, cg, old_initediblk, 0); + brelse(ibp); + UFS_LOCK(ump); + goto check_nifree; + } bzero(ibp->b_data, (int)fs->fs_bsize); dp2 = (struct ufs2_dinode *)(ibp->b_data); for (i = 0; i < INOPB(fs); i++) { @@ -1883,8 +1923,29 @@ gotit: * loading of newly created filesystems. */ babarrierwrite(ibp); - cgp->cg_initediblk += INOPB(fs); + + /* + * After the inode block is written, try to update the + * cg initediblk pointer. If another thread beat us + * to it, then leave it unchanged as the other thread + * has already set it correctly. + */ + error = bread(ip->i_devvp, fsbtodb(fs, cgtod(fs, cg)), + (int)fs->fs_cgsize, NOCRED, &bp); + UFS_LOCK(ump); + ACTIVECLEAR(fs, cg); + UFS_UNLOCK(ump); + if (error != 0) { + brelse(bp); + return (error); + } + cgp = (struct cg *)bp->b_data; + if (cgp->cg_initediblk == old_initediblk) + cgp->cg_initediblk += INOPB(fs); + goto restart; } + cgp->cg_old_time = cgp->cg_time = time_second; + cgp->cg_irotor = ipref; UFS_LOCK(ump); ACTIVECLEAR(fs, cg); setbit(inosused, ipref); -- Test scenario: quota8.sh