GENERIC HEAD from 2013-02-25 11:22:54 UTC, r247264M, vmcore.41 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #2 r247264M: Mon Feb 25 20:29:28 CET 2013 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 0x10 Model = 0x2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8099631104 (7724 MB) : Trying to mount root from ufs:/dev/ufs/root [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ufs/root: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/root: clean, 448786 free (890 frags, 55987 blocks, 0.1% fragmentation) /dev/ufs/home: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/home: clean, 113608 free (5272 frags, 13542 blocks, 0.5% fragmentation) /dev/ufs/usr: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/var: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/var: clean, 8256225 free (43857 frags, 1026546 blocks, 0.2% fragmentation) /dev/ufs/usr: clean, 4275342 free (178070 frags, 512159 blocks, 1.8% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 42659716 free (1324 frags, 5332299 blocks, 0.0% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat/pkg /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. savecore: unable to read from bounds, using 0 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device Feb 25 20:35:18 x4 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device savecore: unable to read from bounds, using 0 No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xffffff81e6e69670 bufwait (bufwait) @ kern/vfs_bio.c:2673 2nd 0xfffffe0015254400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f75f3f0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f75f4b0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f75f4d0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f75f580 _sx_xlock() at _sx_xlock+0x74/frame 0xffffff823f75f5b0 ufsdirhash_acquire() at ufsdirhash_acquire+0x44/frame 0xffffff823f75f5d0 ufsdirhash_add() at ufsdirhash_add+0x19/frame 0xffffff823f75f600 ufs_direnter() at ufs_direnter+0x6c1/frame 0xffffff823f75f6d0 ufs_mkdir() at ufs_mkdir+0x50e/frame 0xffffff823f75f8c0 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc9/frame 0xffffff823f75f8f0 kern_mkdirat() at kern_mkdirat+0x212/frame 0xffffff823f75fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f75fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f75fbf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092532a, rsp = 0x7fffffffd788, rbp = 0x801006050 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting default moused. Starting inetd. Mon Feb 25 20:35:22 CET 2013 FreeBSD/amd64 (x4.osted.lan) (console) login: Feb 25 20:37:20 x4 su: pho to root on /dev/pts/1 20130225 20:38:17 all: quota8.sh lock order reversal: 1st 0xfffffe012d83c068 ufs (ufs) @ kern/vfs_subr.c:2176 2nd 0xffffff81e7b4a038 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xfffffe012d8342d8 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f78bb60 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f78bc20 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f78bc40 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f78bcf0 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f78bdd0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f78be20 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f78be50 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f78bed0 vget() at vget+0x63/frame 0xffffff823f78bf20 vfs_hash_get() at vfs_hash_get+0xd5/frame 0xffffff823f78bf70 ffs_vgetf() at ffs_vgetf+0x48/frame 0xffffff823f78c000 softdep_sync_buf() at softdep_sync_buf+0x3b7/frame 0xffffff823f78c0f0 ffs_syncvnode() at ffs_syncvnode+0x311/frame 0xffffff823f78c170 ffs_sync() at ffs_sync+0x1b3/frame 0xffffff823f78c230 vfs_write_suspend() at vfs_write_suspend+0x105/frame 0xffffff823f78c260 ffs_snapshot() at ffs_snapshot+0xb87/frame 0xffffff823f78c5a0 ffs_mount() at ffs_mount+0x8d9/frame 0xffffff823f78c790 vfs_donmount() at vfs_donmount+0xee0/frame 0xffffff823f78ca90 sys_nmount() at sys_nmount+0x66/frame 0xffffff823f78cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f78cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f78cbf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800880dfa, rsp = 0x7fffffffcea8, rbp = 0x7fffffffda60 --- lock order reversal: 1st 0xffffff81e7b6dd30 bufwait (bufwait) @ kern/vfs_bio.c:2673 2nd 0xfffffe0008e8f330 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2298 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f999060 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f999120 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f999140 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f9991f0 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f9992d0 ffs_copyonwrite() at ffs_copyonwrite+0x199/frame 0xffffff823f999370 ffs_geom_strategy() at ffs_geom_strategy+0x12b/frame 0xffffff823f9993a0 ufs_strategy() at ufs_strategy+0x45/frame 0xffffff823f9993d0 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0xd1/frame 0xffffff823f999400 bufstrategy() at bufstrategy+0x4b/frame 0xffffff823f999440 bufwrite() at bufwrite+0x125/frame 0xffffff823f999470 cluster_wbuild() at cluster_wbuild+0x544/frame 0xffffff823f999510 cluster_write() at cluster_write+0x323/frame 0xffffff823f999620 ffs_write() at ffs_write+0x5cb/frame 0xffffff823f9996c0 VOP_WRITE_APV() at VOP_WRITE_APV+0x135/frame 0xffffff823f9997d0 vn_write() at vn_write+0x281/frame 0xffffff823f999860 vn_io_fault() at vn_io_fault+0x196/frame 0xffffff823f9999f0 dofilewrite() at dofilewrite+0x85/frame 0xffffff823f999a40 kern_writev() at kern_writev+0x6c/frame 0xffffff823f999a80 sys_write() at sys_write+0x64/frame 0xffffff823f999ad0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff823f999bf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff823f999bf0 --- syscall (4, FreeBSD ELF32, sys_write), rip = 0x2819249f, rsp = 0xffffb6c8, rbp = 0xffffc770 --- lock order reversal: 1st 0xfffffe0008e8f330 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:968 2nd 0xfffffe0134acc548 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1627 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823f9622a0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823f962360 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823f962380 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823f962430 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823f962510 ffs_snapremove() at ffs_snapremove+0xe2/frame 0xffffff823f962590 ffs_truncate() at ffs_truncate+0xd35/frame 0xffffff823f9627e0 ufs_inactive() at ufs_inactive+0x28c/frame 0xffffff823f962820 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xc9/frame 0xffffff823f962850 vinactive() at vinactive+0xb2/frame 0xffffff823f9628b0 vputx() at vputx+0x375/frame 0xffffff823f962910 kern_unlinkat() at kern_unlinkat+0x19c/frame 0xffffff823f962ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f962bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f962bf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x8009249da, rsp = 0x7fffffffd5a8, rbp = 0 --- lock order reversal: 1st 0xfffffe0015239068 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe012d835068 syncer (syncer) @ kern/vfs_subr.c:2287 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823fa2a5d0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823fa2a690 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823fa2a6b0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823fa2a760 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823fa2a840 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff823fa2a860 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823fa2a890 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823fa2a910 vputx() at vputx+0x315/frame 0xffffff823fa2a970 dounmount() at dounmount+0x288/frame 0xffffff823fa2a9e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823fa2aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fa2abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fa2abf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffce68, rbp = 0x801006ce8 --- lock order reversal: 1st 0xfffffe0015239068 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe012d83c2d8 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823fa2a500 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823fa2a5c0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823fa2a5e0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823fa2a690 __lockmgr_args() at __lockmgr_args+0x1178/frame 0xffffff823fa2a770 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff823fa2a790 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823fa2a7c0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823fa2a840 softdep_flushworklist() at softdep_flushworklist+0x61/frame 0xffffff823fa2a8b0 ffs_sync() at ffs_sync+0x3be/frame 0xffffff823fa2a970 dounmount() at dounmount+0x318/frame 0xffffff823fa2a9e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823fa2aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fa2abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fa2abf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffce68, rbp = 0x801006ce8 --- 20130225 20:40:55 all: quota8.sh Feb 25 20:41:28 x4 kernel: pid 2631 (rw), uid 1004 inumber 68360 on /mnt: filesystem full Feb 25 20:43:19 x4 kernel: pid 3098 (rw), uid 1004 inumber 98880 on /mnt: filesystem full 20130225 20:44:12 all: quota8.sh 20130225 20:47:09 all: quota8.sh 20130225 20:51:31 all: quota8.sh Stopping inetd. Waiting for PIDS: 1024. Stopping moused. Waiting for PIDS: 991, 991. Shutting down local packages:. Stopping cron. Waiting for PIDS: 961. Stopping sshd. Waiting for PIDS: 942. Stopping watchdogd. Waiting for PIDS: 97130 second watchdog timeout expired. Shutdown terminated. Mon Feb 25 21:06:33 CET 2013 Feb 25 21:06:33 x4 init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode Feb 25 21:06:33 x4 syslogd: exiting on signal 15 Feb 25 21:06:53 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # umount -t ufs -a -f & # mount -u -o ro / ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x3b: movq $0,0xa97aa2(%rip) db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xfffffe00051e0900 kdb_enter() at kdb_enter+0x3b/frame 0xffffff80002478c0 kdb_break() at kdb_break+0x27/frame 0xffffff80002478d0 uart_intr() at uart_intr+0x35a/frame 0xffffff8000247910 intr_event_handle() at intr_event_handle+0xb7/frame 0xffffff8000247960 intr_execute_handlers() at intr_execute_handlers+0x5f/frame 0xffffff8000247990 lapic_handle_intr() at lapic_handle_intr+0x32/frame 0xffffff80002479c0 Xapic_isr1() at Xapic_isr1+0xa0/frame 0xffffff80002479c0 --- interrupt, rip = 0xffffffff80c658d6, rsp = 0xffffff8000247a80, rbp = 0xffffff8000247a90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000247a90 acpi_cpu_idle() at acpi_cpu_idle+0x2ac/frame 0xffffff8000247ad0 cpu_idle_acpi() at cpu_idle_acpi+0x45/frame 0xffffff8000247af0 cpu_idle() at cpu_idle+0x78/frame 0xffffff8000247b10 sched_idletd() at sched_idletd+0x186/frame 0xffffff8000247ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000247bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000247bf0 --- trap 0, rip = 0, rsp = 0xffffff8000247cb0, rbp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x5d6a80 curthread = 0xfffffe00051d4480: pid 11 "idle: cpu0" curpcb = 0xffffff800023dcc0 fpcurthread = none idlethread = 0xfffffe00051d4480: tid 100003 "idle: cpu0" curpmap = 0xffffffff8135b070 tssp = 0xffffffff8156ab00 commontssp = 0xffffffff8156ab00 rsp0 = 0xffffff800023dcc0 gs32p = 0xffffffff81568c38 ldt = 0xffffffff81568c78 tss = 0xffffffff81568c68 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807ef00a80 curthread = 0xfffffe00051d4000: pid 11 "idle: cpu1" curpcb = 0xffffff8000242cc0 fpcurthread = none idlethread = 0xfffffe00051d4000: tid 100004 "idle: cpu1" curpmap = 0xffffffff8135b070 tssp = 0xffffffff8156ab68 commontssp = 0xffffffff8156ab68 rsp0 = 0xffffff8000242cc0 gs32p = 0xffffffff81568ca0 ldt = 0xffffffff81568ce0 tss = 0xffffffff81568cd0 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807ef07a80 curthread = 0xfffffe00051e0900: pid 11 "idle: cpu2" curpcb = 0xffffff8000247cc0 fpcurthread = none idlethread = 0xfffffe00051e0900: tid 100005 "idle: cpu2" curpmap = 0xffffffff8135b070 tssp = 0xffffffff8156abd0 commontssp = 0xffffffff8156abd0 rsp0 = 0xffffff8000247cc0 gs32p = 0xffffffff81568d08 ldt = 0xffffffff81568d48 tss = 0xffffffff81568d38 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807ef0ea80 curthread = 0xfffffe00051e0480: pid 11 "idle: cpu3" curpcb = 0xffffff800024ccc0 fpcurthread = none idlethread = 0xfffffe00051e0480: tid 100006 "idle: cpu3" curpmap = 0xffffffff8135b070 tssp = 0xffffffff8156ac38 commontssp = 0xffffffff8156ac38 rsp0 = 0xffffff800024ccc0 gs32p = 0xffffffff81568d70 ldt = 0xffffffff81568db0 tss = 0xffffffff81568da0 spin locks held: db:0:allpcpu> show alllocks Process 7191 (ls) thread 0xfffffe017b8c6000 (100188) shared lockmgr ufs (ufs) r = 0 (0xfffffe0008e10548) locked @ kern/vfs_lookup.c:516 Process 7128 (umount) thread 0xfffffe017bcc9000 (100256) exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0015239068) locked @ kern/vfs_mount.c:1236 db:0:alllocks> show lockedvnods Locked vnodes 0xfffffe0008e104e0: tag ufs, type VDIR usecount 19, writecount 0, refcount 22 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) v_object 0xfffffe0008c72e10 ref 0 pages 1 lock type ufs: SHARED (count 1) with exclusive waiters pending #0 0xffffffff808c1aac at __lockmgr_args+0xd7c #1 0xffffffff80b35b7b at ffs_lock+0x9b #2 0xffffffff80d1f47b at VOP_LOCK1_APV+0xab #3 0xffffffff809910fe at _vn_lock+0x8e #4 0xffffffff80975c17 at lookup+0xc7 #5 0xffffffff80976bfc at namei+0x3fc #6 0xffffffff80979f45 at vfs_donmount+0x495 #7 0xffffffff8097b346 at sys_nmount+0x66 #8 0xffffffff80c80573 at amd64_syscall+0x2d3 #9 0xffffffff80c6a7b7 at Xfast_syscall+0xf7 ino 2, on dev ufs/root 0xfffffe0015239000: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0xfffffe014b193000 flags (VI_ACTIVE) v_object 0xfffffe00153e5780 ref 0 pages 0 lock type ufs: EXCL by thread 0xfffffe017bcc9000 (pid 7128, umount, tid 100256) with shared waiters pending #0 0xffffffff808c1433 at __lockmgr_args+0x703 #1 0xffffffff80b35b7b at ffs_lock+0x9b #2 0xffffffff80d1f47b at VOP_LOCK1_APV+0xab #3 0xffffffff809910fe at _vn_lock+0x8e #4 0xffffffff80979043 at dounmount+0x93 #5 0xffffffff80979a8e at sys_unmount+0x38e #6 0xffffffff80c80573 at amd64_syscall+0x2d3 #7 0xffffffff80c6a7b7 at Xfast_syscall+0xf7 ino 47105, on dev ufs/root db:0:lockedvnods> show mount 0xfffffe0008dd8b58 /dev/ufs/root on / (ufs) 0xfffffe0008dd9000 devfs on /dev (devfs) 0xfffffe0008dd9790 procfs on /proc (procfs) 0xfffffe014b193000 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 8573 8571 8573 0 D+ ufs 0xfffffe0008e10548 mount 8571 1 8571 0 Ss+ wait 0xfffffe01e048b4a8 sh 7191 1 971 0 D ufs 0xfffffe0015239068 ls 7129 1 5114 1004 D+ vfs_busy 0xfffffe014b193000 sh 7128 1 1135 0 D+ dqflsh 0xffffffff8155dcd8 umount 5100 0 0 0 DL mdwait 0xfffffe014b195800 [md5] 18 0 0 0 DL sdflush 0xffffffff8155db18 [softdepflush] 17 0 0 0 DL vlruwt 0xfffffe0008c68950 [vnlru] 16 0 0 0 DL syncer 0xffffffff81552180 [syncer] 9 0 0 0 DL psleep 0xffffffff81551be0 [bufdaemon] 8 0 0 0 DL pgzero 0xffffffff81567abc [pagezero] 7 0 0 0 DL psleep 0xffffffff81566c70 [vmdaemon] 6 0 0 0 DL psleep 0xffffffff81566c4c [pagedaemon] 5 0 0 0 DL ccb_scan 0xffffffff81320460 [xpt_thrd] 4 0 0 0 DL waiting_ 0xffffffff81557e60 [sctp_iterator] 3 0 0 0 DL ctl_work 0xffffff80008b5000 [ctl_thrd] 2 0 0 0 DL - 0xfffffe00055de248 [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xffffff80008b0e18 [usbus5] 100057 D - 0xffffff80008b0dc0 [usbus5] 100056 D - 0xffffff80008b0d68 [usbus5] 100055 D - 0xffffff80008b0d10 [usbus5] 100053 D - 0xffffff80008a8460 [usbus4] 100052 D - 0xffffff80008a8408 [usbus4] 100051 D - 0xffffff80008a83b0 [usbus4] 100050 D - 0xffffff80008a8358 [usbus4] 100049 D - 0xffffff80008a5460 [usbus3] 100048 D - 0xffffff80008a5408 [usbus3] 100047 D - 0xffffff80008a53b0 [usbus3] 100046 D - 0xffffff80008a5358 [usbus3] 100045 D - 0xffffff80008a2460 [usbus2] 100044 D - 0xffffff80008a2408 [usbus2] 100043 D - 0xffffff80008a23b0 [usbus2] 100042 D - 0xffffff80008a2358 [usbus2] 100040 D - 0xffffff800089f460 [usbus1] 100039 D - 0xffffff800089f408 [usbus1] 100038 D - 0xffffff800089f3b0 [usbus1] 100037 D - 0xffffff800089f358 [usbus1] 100035 D - 0xffffff800089c460 [usbus0] 100034 D - 0xffffff800089c408 [usbus0] 100033 D - 0xffffff800089c3b0 [usbus0] 100032 D - 0xffffff800089c358 [usbus0] 14 0 0 0 DL - 0xffffffff81361284 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xffffffff8135a310 [g_down] 100014 D - 0xffffffff8135a308 [g_up] 100013 D - 0xffffffff8135a2f8 [g_event] 12 0 0 0 WL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100023 I [swi2: cambio] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100012 I [swi3: vm] 100011 I [swi1: netisr 0] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi4: clock] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe00051d2950 [init] 10 0 0 0 DL audit_wo 0xffffffff8155ca90 [audit] 0 0 0 0 DLs (threaded) [kernel] 100066 D - 0xffffffff81361284 [deadlkres] 100065 D - 0xfffffe00055fd780 [mca taskq] 100027 D - 0xfffffe0005386200 [acpi_task_2] 100026 D - 0xfffffe0005386200 [acpi_task_1] 100025 D - 0xfffffe0005386200 [acpi_task_0] 100024 D - 0xfffffe0005386280 [kqueue taskq] 100020 D - 0xfffffe0005386400 [thread taskq] 100018 D - 0xfffffe0005386500 [ffs_trim taskq] 100016 D - 0xfffffe0002f63900 [firmware taskq] 100000 D sched 0xffffffff8135a620 [swapper] db:0:ps> allt Tracing command mount pid 8573 tid 100114 td 0xfffffe0015050000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f7e63b0 mi_switch() at mi_switch+0x238/frame 0xffffff823f7e6400 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f7e6440 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823f7e6470 __lockmgr_args() at __lockmgr_args+0x713/frame 0xffffff823f7e6550 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f7e65a0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f7e65d0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f7e6650 lookup() at lookup+0x95e/frame 0xffffff823f7e66e0 namei() at namei+0x3fc/frame 0xffffff823f7e6790 vfs_donmount() at vfs_donmount+0x495/frame 0xffffff823f7e6a90 sys_nmount() at sys_nmount+0x66/frame 0xffffff823f7e6ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f7e6bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f7e6bf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a97dfa, rsp = 0x7fffffffcc88, rbp = 0x7fffffffcca0 --- Tracing command sh pid 8571 tid 100338 td 0xfffffe017bf13480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fc46670 mi_switch() at mi_switch+0x238/frame 0xffffff823fc466c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fc46700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823fc46760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823fc46780 _sleep() at _sleep+0x37d/frame 0xffffff823fc46810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff823fc468b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823fc46a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823fc46ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fc46bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fc46bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd9b8, rbp = 0x1 --- Tracing command ls pid 7191 tid 100188 td 0xfffffe017b8c6000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f958330 mi_switch() at mi_switch+0x238/frame 0xffffff823f958380 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f9583c0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823f9583f0 __lockmgr_args() at __lockmgr_args+0x108b/frame 0xffffff823f9584d0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823f958520 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823f958550 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823f9585d0 vget() at vget+0x63/frame 0xffffff823f958620 cache_lookup() at cache_lookup+0x55a/frame 0xffffff823f9586b0 vfs_cache_lookup() at vfs_cache_lookup+0xcb/frame 0xffffff823f958720 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc6/frame 0xffffff823f958750 lookup() at lookup+0x332/frame 0xffffff823f9587e0 namei() at namei+0x3fc/frame 0xffffff823f958890 kern_statat_vnhook() at kern_statat_vnhook+0xbe/frame 0xffffff823f958a10 kern_statat() at kern_statat+0x15/frame 0xffffff823f958a30 sys_lstat() at sys_lstat+0x2a/frame 0xffffff823f958ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823f958bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823f958bf0 --- syscall (190, FreeBSD ELF64, sys_lstat), rip = 0x800d8c2ea, rsp = 0x7fffffffd448, rbp = 0x80141e590 --- Tracing command sh pid 7129 tid 100268 td 0xfffffe017b18e480 sched_switch() at sched_switch+0x1b4/frame 0xffffff823fae8500 mi_switch() at mi_switch+0x238/frame 0xffffff823fae8550 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823fae8590 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823fae85c0 _sleep() at _sleep+0x3e9/frame 0xffffff823fae8650 vfs_busy() at vfs_busy+0x125/frame 0xffffff823fae8680 kern_fstatfs() at kern_fstatfs+0x126/frame 0xffffff823fae88d0 sys_fstatfs() at sys_fstatfs+0x34/frame 0xffffff823fae8ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823fae8bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823fae8bf0 --- syscall (397, FreeBSD ELF64, sys_fstatfs), rip = 0x800d30ada, rsp = 0x7fffffffcf78, rbp = 0x1000 --- Tracing command umount pid 7128 tid 100256 td 0xfffffe017bcc9000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823faac620 mi_switch() at mi_switch+0x238/frame 0xffffff823faac670 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823faac6b0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823faac6e0 _sleep() at _sleep+0x3e9/frame 0xffffff823faac770 quotaoff1() at quotaoff1+0x203/frame 0xffffff823faac800 quotaoff_inchange() at quotaoff_inchange+0x29/frame 0xffffff823faac840 ffs_flushfiles() at ffs_flushfiles+0x82/frame 0xffffff823faac8a0 softdep_flushfiles() at softdep_flushfiles+0x64/frame 0xffffff823faac900 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff823faac970 dounmount() at dounmount+0x349/frame 0xffffff823faac9e0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff823faacad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823faacbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823faacbf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffcec8, rbp = 0x801006ce8 --- Tracing command md5 pid 5100 tid 100152 td 0xfffffe001597d900 sched_switch() at sched_switch+0x1b4/frame 0xffffff823f8a4a10 mi_switch() at mi_switch+0x238/frame 0xffffff823f8a4a60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823f8a4aa0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823f8a4ad0 _sleep() at _sleep+0x3e9/frame 0xffffff823f8a4b60 md_kthread() at md_kthread+0x17b/frame 0xffffff823f8a4ba0 fork_exit() at fork_exit+0x139/frame 0xffffff823f8a4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823f8a4bf0 --- trap 0, rip = 0, rsp = 0xffffff823f8a4cb0, rbp = 0 --- Tracing command softdepflush pid 18 tid 100075 td 0xfffffe0008c6f480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbbca10 mi_switch() at mi_switch+0x238/frame 0xffffff822cbbca60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbbcaa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbbcad0 _sleep() at _sleep+0x29a/frame 0xffffff822cbbcb60 softdep_flush() at softdep_flush+0x20d/frame 0xffffff822cbbcba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbbcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbbcbf0 --- trap 0, rip = 0, rsp = 0xffffff822cbbccb0, rbp = 0 --- Tracing command vnlru pid 17 tid 100074 td 0xfffffe0008c6f900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbb79d0 mi_switch() at mi_switch+0x238/frame 0xffffff822cbb7a20 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbb7a60 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbb7a90 _sleep() at _sleep+0x29a/frame 0xffffff822cbb7b20 vnlru_proc() at vnlru_proc+0x537/frame 0xffffff822cbb7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbb7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbb7bf0 --- trap 0, rip = 0, rsp = 0xffffff822cbb7cb0, rbp = 0 --- Tracing command syncer pid 16 tid 100073 td 0xfffffe0008c70000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbb29f0 mi_switch() at mi_switch+0x238/frame 0xffffff822cbb2a40 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbb2a80 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbb2ab0 _cv_timedwait() at _cv_timedwait+0x18f/frame 0xffffff822cbb2b10 sched_sync() at sched_sync+0x4ee/frame 0xffffff822cbb2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbb2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbb2bf0 --- trap 0, rip = 0, rsp = 0xffffff822cbb2cb0, rbp = 0 --- Tracing command bufdaemon pid 9 tid 100072 td 0xfffffe00055ee000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cbada30 mi_switch() at mi_switch+0x238/frame 0xffffff822cbada80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cbadac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cbadaf0 _sleep() at _sleep+0x29a/frame 0xffffff822cbadb80 buf_daemon() at buf_daemon+0x192/frame 0xffffff822cbadba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cbadbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cbadbf0 --- trap 0, rip = 0, rsp = 0xffffff822cbadcb0, rbp = 0 --- Tracing command pagezero pid 8 tid 100071 td 0xfffffe00055ee480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cba8a30 mi_switch() at mi_switch+0x238/frame 0xffffff822cba8a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cba8ac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cba8af0 _sleep() at _sleep+0x29a/frame 0xffffff822cba8b80 vm_pagezero() at vm_pagezero+0x73/frame 0xffffff822cba8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cba8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cba8bf0 --- trap 0, rip = 0, rsp = 0xffffff822cba8cb0, rbp = 0 --- Tracing command vmdaemon pid 7 tid 100070 td 0xfffffe00055ee900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cba39e0 mi_switch() at mi_switch+0x238/frame 0xffffff822cba3a30 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cba3a70 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822cba3aa0 _sleep() at _sleep+0x3e9/frame 0xffffff822cba3b30 vm_daemon() at vm_daemon+0x4d/frame 0xffffff822cba3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cba3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cba3bf0 --- trap 0, rip = 0, rsp = 0xffffff822cba3cb0, rbp = 0 --- Tracing command pagedaemon pid 6 tid 100069 td 0xfffffe00055f1000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822cb9e940 mi_switch() at mi_switch+0x238/frame 0xffffff822cb9e990 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822cb9e9d0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822cb9ea00 _sleep() at _sleep+0x29a/frame 0xffffff822cb9ea90 vm_pageout() at vm_pageout+0xb8f/frame 0xffffff822cb9eba0 fork_exit() at fork_exit+0x139/frame 0xffffff822cb9ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822cb9ebf0 --- trap 0, rip = 0, rsp = 0xffffff822cb9ecb0, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100068 td 0xfffffe00055f1480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca19a30 mi_switch() at mi_switch+0x238/frame 0xffffff822ca19a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca19ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca19af0 _sleep() at _sleep+0x3e9/frame 0xffffff822ca19b80 xpt_scanner_thread() at xpt_scanner_thread+0xdd/frame 0xffffff822ca19ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca19bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca19bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca19cb0, rbp = 0 --- Tracing command sctp_iterator pid 4 tid 100067 td 0xfffffe00055f1900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca14a40 mi_switch() at mi_switch+0x238/frame 0xffffff822ca14a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca14ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca14b00 _sleep() at _sleep+0x3e9/frame 0xffffff822ca14b90 sctp_iterator_thread() at sctp_iterator_thread+0x3f/frame 0xffffff822ca14ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca14bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca14bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca14cb0, rbp = 0 --- Tracing command ctl_thrd pid 3 tid 100064 td 0xfffffe00055f3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca043f0 mi_switch() at mi_switch+0x238/frame 0xffffff822ca04440 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca04480 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca044b0 _sleep() at _sleep+0x3e9/frame 0xffffff822ca04540 ctl_work_thread() at ctl_work_thread+0x1ce8/frame 0xffffff822ca04ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca04bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca04bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca04cb0, rbp = 0 --- Tracing command fdc0 pid 2 tid 100061 td 0xfffffe00055f4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9f1990 mi_switch() at mi_switch+0x238/frame 0xffffff822c9f19e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9f1a20 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822c9f1a50 _sleep() at _sleep+0x29a/frame 0xffffff822c9f1ae0 fdc_thread() at fdc_thread+0x7f4/frame 0xffffff822c9f1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9f1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9f1bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9f1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100058 td 0xfffffe00055c3000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9aba50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9abaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9abae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9abb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9abb70 usb_process() at usb_process+0x172/frame 0xffffff822c9abba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9abbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9abbf0 --- trap 0, rip = 0, rsp = 0xffffff822c9abcb0, rbp = 0 --- Tracing command usb pid 15 tid 100057 td 0xfffffe00055c3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9a6a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9a6aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9a6ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9a6b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9a6b70 usb_process() at usb_process+0x172/frame 0xffffff822c9a6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9a6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9a6bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9a6cb0, rbp = 0 --- Tracing command usb pid 15 tid 100056 td 0xfffffe00055c3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9a1a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c9a1aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c9a1ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c9a1b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c9a1b70 usb_process() at usb_process+0x172/frame 0xffffff822c9a1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9a1bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9a1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100055 td 0xfffffe00055c4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c99ca50 mi_switch() at mi_switch+0x238/frame 0xffffff822c99caa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c99cae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c99cb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c99cb70 usb_process() at usb_process+0x172/frame 0xffffff822c99cba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c99cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c99cbf0 --- trap 0, rip = 0, rsp = 0xffffff822c99ccb0, rbp = 0 --- Tracing command usb pid 15 tid 100053 td 0xfffffe00055c4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c80fa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c80faa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c80fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c80fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c80fb70 usb_process() at usb_process+0x172/frame 0xffffff822c80fba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c80fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c80fbf0 --- trap 0, rip = 0, rsp = 0xffffff822c80fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100052 td 0xfffffe00055c5000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c80aa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c80aaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c80aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c80ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c80ab70 usb_process() at usb_process+0x172/frame 0xffffff822c80aba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c80abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c80abf0 --- trap 0, rip = 0, rsp = 0xffffff822c80acb0, rbp = 0 --- Tracing command usb pid 15 tid 100051 td 0xfffffe00055c5480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c805a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c805aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c805ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c805b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c805b70 usb_process() at usb_process+0x172/frame 0xffffff822c805ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c805bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c805bf0 --- trap 0, rip = 0, rsp = 0xffffff822c805cb0, rbp = 0 --- Tracing command usb pid 15 tid 100050 td 0xfffffe00055c5900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c800a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c800aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c800ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c800b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c800b70 usb_process() at usb_process+0x172/frame 0xffffff822c800ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c800bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c800bf0 --- trap 0, rip = 0, rsp = 0xffffff822c800cb0, rbp = 0 --- Tracing command usb pid 15 tid 100049 td 0xfffffe00055b7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7b7a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7b7aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7b7ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7b7b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7b7b70 usb_process() at usb_process+0x172/frame 0xffffff822c7b7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7b7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7b7bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7b7cb0, rbp = 0 --- Tracing command usb pid 15 tid 100048 td 0xfffffe00055b7480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7b2a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7b2aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7b2ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7b2b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7b2b70 usb_process() at usb_process+0x172/frame 0xffffff822c7b2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7b2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7b2bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7b2cb0, rbp = 0 --- Tracing command usb pid 15 tid 100047 td 0xfffffe00055b7900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7ada50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7adaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7adae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7adb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7adb70 usb_process() at usb_process+0x172/frame 0xffffff822c7adba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7adbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7adbf0 --- trap 0, rip = 0, rsp = 0xffffff822c7adcb0, rbp = 0 --- Tracing command usb pid 15 tid 100046 td 0xfffffe00055b8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c7a8a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c7a8aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c7a8ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c7a8b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c7a8b70 usb_process() at usb_process+0x172/frame 0xffffff822c7a8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c7a8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c7a8bf0 --- trap 0, rip = 0, rsp = 0xffffff822c7a8cb0, rbp = 0 --- Tracing command usb pid 15 tid 100045 td 0xfffffe00055b8480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c75fa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c75faa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c75fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c75fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c75fb70 usb_process() at usb_process+0x172/frame 0xffffff822c75fba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c75fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c75fbf0 --- trap 0, rip = 0, rsp = 0xffffff822c75fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100044 td 0xfffffe00055b8900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c75aa50 mi_switch() at mi_switch+0x238/frame 0xffffff822c75aaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c75aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c75ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c75ab70 usb_process() at usb_process+0x172/frame 0xffffff822c75aba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c75abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c75abf0 --- trap 0, rip = 0, rsp = 0xffffff822c75acb0, rbp = 0 --- Tracing command usb pid 15 tid 100043 td 0xfffffe00055ba000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c755a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c755aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c755ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c755b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c755b70 usb_process() at usb_process+0x172/frame 0xffffff822c755ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c755bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c755bf0 --- trap 0, rip = 0, rsp = 0xffffff822c755cb0, rbp = 0 --- Tracing command usb pid 15 tid 100042 td 0xfffffe00055ba480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c750a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c750aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c750ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c750b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c750b70 usb_process() at usb_process+0x172/frame 0xffffff822c750ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c750bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c750bf0 --- trap 0, rip = 0, rsp = 0xffffff822c750cb0, rbp = 0 --- Tracing command usb pid 15 tid 100040 td 0xfffffe00055bc000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c702a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c702aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c702ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c702b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c702b70 usb_process() at usb_process+0x172/frame 0xffffff822c702ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c702bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c702bf0 --- trap 0, rip = 0, rsp = 0xffffff822c702cb0, rbp = 0 --- Tracing command usb pid 15 tid 100039 td 0xfffffe00055b0000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6fda50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6fdaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6fdae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6fdb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6fdb70 usb_process() at usb_process+0x172/frame 0xffffff822c6fdba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6fdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6fdbf0 --- trap 0, rip = 0, rsp = 0xffffff822c6fdcb0, rbp = 0 --- Tracing command usb pid 15 tid 100038 td 0xfffffe00055b0480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6f8a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6f8aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6f8ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6f8b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6f8b70 usb_process() at usb_process+0x172/frame 0xffffff822c6f8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6f8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6f8bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6f8cb0, rbp = 0 --- Tracing command usb pid 15 tid 100037 td 0xfffffe00055b0900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6f3a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6f3aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6f3ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6f3b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6f3b70 usb_process() at usb_process+0x172/frame 0xffffff822c6f3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6f3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6f3bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6f3cb0, rbp = 0 --- Tracing command usb pid 15 tid 100035 td 0xfffffe00055b1480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6a5a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6a5aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6a5ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6a5b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6a5b70 usb_process() at usb_process+0x172/frame 0xffffff822c6a5ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6a5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6a5bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6a5cb0, rbp = 0 --- Tracing command usb pid 15 tid 100034 td 0xfffffe00055b1900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6a0a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c6a0aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c6a0ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c6a0b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c6a0b70 usb_process() at usb_process+0x172/frame 0xffffff822c6a0ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6a0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6a0bf0 --- trap 0, rip = 0, rsp = 0xffffff822c6a0cb0, rbp = 0 --- Tracing command usb pid 15 tid 100033 td 0xfffffe00055b2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c69ba50 mi_switch() at mi_switch+0x238/frame 0xffffff822c69baa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c69bae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c69bb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c69bb70 usb_process() at usb_process+0x172/frame 0xffffff822c69bba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c69bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c69bbf0 --- trap 0, rip = 0, rsp = 0xffffff822c69bcb0, rbp = 0 --- Tracing command usb pid 15 tid 100032 td 0xfffffe00055b2480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c696a50 mi_switch() at mi_switch+0x238/frame 0xffffff822c696aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822c696ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822c696b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff822c696b70 usb_process() at usb_process+0x172/frame 0xffffff822c696ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c696bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c696bf0 --- trap 0, rip = 0, rsp = 0xffffff822c696cb0, rbp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xfffffe0005209900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000283a10 mi_switch() at mi_switch+0x238/frame 0xffffff8000283a60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000283aa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8000283ad0 _sleep() at _sleep+0x29a/frame 0xffffff8000283b60 random_kthread() at random_kthread+0x1ad/frame 0xffffff8000283ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000283bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000283bf0 --- trap 0, rip = 0, rsp = 0xffffff8000283cb0, rbp = 0 --- Tracing command geom pid 13 tid 100015 td 0xfffffe00051f2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000279a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000279a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000279ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000279ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000279b70 g_io_schedule_down() at g_io_schedule_down+0x25f/frame 0xffffff8000279b90 g_down_procbody() at g_down_procbody+0x7c/frame 0xffffff8000279ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000279bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000279bf0 --- trap 0, rip = 0, rsp = 0xffffff8000279cb0, rbp = 0 --- Tracing command geom pid 13 tid 100014 td 0xfffffe00051f2480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000274a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000274a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000274ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000274ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000274b70 g_io_schedule_up() at g_io_schedule_up+0x138/frame 0xffffff8000274b90 g_up_procbody() at g_up_procbody+0x7c/frame 0xffffff8000274ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000274bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000274bf0 --- trap 0, rip = 0, rsp = 0xffffff8000274cb0, rbp = 0 --- Tracing command geom pid 13 tid 100013 td 0xfffffe00051f2900 sched_switch() at sched_switch+0x1b4/frame 0xffffff800026fa20 mi_switch() at mi_switch+0x238/frame 0xffffff800026fa70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800026fab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800026fae0 _sleep() at _sleep+0x3e9/frame 0xffffff800026fb70 g_run_events() at g_run_events+0x449/frame 0xffffff800026fba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026fbf0 --- trap 0, rip = 0, rsp = 0xffffff800026fcb0, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xfffffe00055f4000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xfffffe00055f4480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9f6b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9f6b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9f6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9f6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9f6bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9f6cb0, rbp = 0 --- Tracing command intr pid 12 tid 100060 td 0xfffffe00055bc480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9e2b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9e2b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9e2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9e2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9e2bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9e2cb0, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffffe00055bc900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c9d3b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c9d3b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c9d3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c9d3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c9d3bf0 --- trap 0, rip = 0, rsp = 0xffffff822c9d3cb0, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffffe00055c4480 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffffe00055ba900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c74bb00 mi_switch() at mi_switch+0x238/frame 0xffffff822c74bb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c74bba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c74bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c74bbf0 --- trap 0, rip = 0, rsp = 0xffffff822c74bcb0, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xfffffe00055b1000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c6eeb00 mi_switch() at mi_switch+0x238/frame 0xffffff822c6eeb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c6eeba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c6eebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c6eebf0 --- trap 0, rip = 0, rsp = 0xffffff822c6eecb0, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xfffffe00055b2900 sched_switch() at sched_switch+0x1b4/frame 0xffffff822c691b00 mi_switch() at mi_switch+0x238/frame 0xffffff822c691b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff822c691ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822c691bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822c691bf0 --- trap 0, rip = 0, rsp = 0xffffff822c691cb0, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xfffffe000520b480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003ccb00 mi_switch() at mi_switch+0x238/frame 0xffffff80003ccb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80003ccba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003ccbf0 --- trap 0, rip = 0, rsp = 0xffffff80003cccb0, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xfffffe000520b900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003c7b00 mi_switch() at mi_switch+0x238/frame 0xffffff80003c7b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80003c7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003c7bf0 --- trap 0, rip = 0, rsp = 0xffffff80003c7cb0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffffe00053a8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002bbb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002bbb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80002bbba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002bbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bbbf0 --- trap 0, rip = 0, rsp = 0xffffff80002bbcb0, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xfffffe00053a9900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a2b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002a2b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff80002a2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a2bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a2cb0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xfffffe0005208000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800029db00 mi_switch() at mi_switch+0x238/frame 0xffffff800029db50 ithread_loop() at ithread_loop+0x273/frame 0xffffff800029dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800029dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800029dbf0 --- trap 0, rip = 0, rsp = 0xffffff800029dcb0, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xfffffe0005208480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000298b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000298b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000298ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000298bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000298bf0 --- trap 0, rip = 0, rsp = 0xffffff8000298cb0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffffe0005209000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xfffffe00051f3000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xfffffe00051f3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000265b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000265b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000265ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000265bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000265bf0 --- trap 0, rip = 0, rsp = 0xffffff8000265cb0, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xfffffe00051f3900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000260b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000260b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000260ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000260bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000260bf0 --- trap 0, rip = 0, rsp = 0xffffff8000260cb0, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xfffffe00051d7480 sched_switch() at sched_switch+0x1b4/frame 0xffffff800025bb00 mi_switch() at mi_switch+0x238/frame 0xffffff800025bb50 ithread_loop() at ithread_loop+0x273/frame 0xffffff800025bba0 fork_exit() at fork_exit+0x139/frame 0xffffff800025bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800025bbf0 --- trap 0, rip = 0, rsp = 0xffffff800025bcb0, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xfffffe00051d7900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000256b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000256b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000256ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000256bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000256bf0 --- trap 0, rip = 0, rsp = 0xffffff8000256cb0, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xfffffe00051e0000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000251b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000251b50 ithread_loop() at ithread_loop+0x273/frame 0xffffff8000251ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000251bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000251bf0 --- trap 0, rip = 0, rsp = 0xffffff8000251cb0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00051e0480 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800022cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800022cd20 trap() at trap+0x325/frame 0xffffff800022cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800022cf20 --- trap 0x13, rip = 0xffffffff80c658d6, rsp = 0xffffff800022cfe0, rbp = 0xffffff800024ca90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800024ca90 acpi_cpu_idle() at acpi_cpu_idle+0x2ac/frame 0xffffff800024cad0 cpu_idle_acpi() at cpu_idle_acpi+0x45/frame 0xffffff800024caf0 cpu_idle() at cpu_idle+0x78/frame 0xffffff800024cb10 sched_idletd() at sched_idletd+0x186/frame 0xffffff800024cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800024cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800024cbf0 --- trap 0, rip = 0, rsp = 0xffffff800024ccb0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00051e0900 kdb_enter() at kdb_enter+0x3b/frame 0xffffff80002478c0 kdb_break() at kdb_break+0x27/frame 0xffffff80002478d0 uart_intr() at uart_intr+0x35a/frame 0xffffff8000247910 intr_event_handle() at intr_event_handle+0xb7/frame 0xffffff8000247960 intr_execute_handlers() at intr_execute_handlers+0x5f/frame 0xffffff8000247990 lapic_handle_intr() at lapic_handle_intr+0x32/frame 0xffffff80002479c0 Xapic_isr1() at Xapic_isr1+0xa0/frame 0xffffff80002479c0 --- interrupt, rip = 0xffffffff80c658d6, rsp = 0xffffff8000247a80, rbp = 0xffffff8000247a90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000247a90 acpi_cpu_idle() at acpi_cpu_idle+0x2ac/frame 0xffffff8000247ad0 cpu_idle_acpi() at cpu_idle_acpi+0x45/frame 0xffffff8000247af0 cpu_idle() at cpu_idle+0x78/frame 0xffffff8000247b10 sched_idletd() at sched_idletd+0x186/frame 0xffffff8000247ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000247bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000247bf0 --- trap 0, rip = 0, rsp = 0xffffff8000247cb0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00051d4000 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800021ed00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800021ed20 trap() at trap+0x325/frame 0xffffff800021ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800021ef20 --- trap 0x13, rip = 0xffffffff80c658d6, rsp = 0xffffff800021efe0, rbp = 0xffffff8000242a90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000242a90 acpi_cpu_idle() at acpi_cpu_idle+0x2ac/frame 0xffffff8000242ad0 cpu_idle_acpi() at cpu_idle_acpi+0x45/frame 0xffffff8000242af0 cpu_idle() at cpu_idle+0x78/frame 0xffffff8000242b10 sched_idletd() at sched_idletd+0x186/frame 0xffffff8000242ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000242bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000242bf0 --- trap 0, rip = 0, rsp = 0xffffff8000242cb0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00051d4480 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffffff815782e0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff81578300 trap() at trap+0x325/frame 0xffffffff81578500 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81578500 --- trap 0x13, rip = 0xffffffff80c658d6, rsp = 0xffffffff815785c0, rbp = 0xffffff800023da90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800023da90 acpi_cpu_idle() at acpi_cpu_idle+0x2ac/frame 0xffffff800023dad0 cpu_idle_acpi() at cpu_idle_acpi+0x45/frame 0xffffff800023daf0 cpu_idle() at cpu_idle+0x78/frame 0xffffff800023db10 sched_idletd() at sched_idletd+0x186/frame 0xffffff800023dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800023dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800023dbf0 --- trap 0, rip = 0, rsp = 0xffffff800023dcb0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00051d4900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000238670 mi_switch() at mi_switch+0x238/frame 0xffffff80002386c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000238700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8000238760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8000238780 _sleep() at _sleep+0x37d/frame 0xffffff8000238810 kern_wait6() at kern_wait6+0x5f1/frame 0xffffff80002388b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8000238a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8000238ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8000238bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8000238bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41242a, rsp = 0x7fffffffd6c8, rbp = 0x217b --- Tracing command audit pid 10 tid 100001 td 0xfffffe00051d7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000233a00 mi_switch() at mi_switch+0x238/frame 0xffffff8000233a50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000233a90 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000233ac0 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8000233b20 audit_worker() at audit_worker+0x77/frame 0xffffff8000233ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000233bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000233bf0 --- trap 0, rip = 0, rsp = 0xffffff8000233cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffffe00055f3000 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca0ea10 mi_switch() at mi_switch+0x238/frame 0xffffff822ca0ea60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca0eaa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff822ca0ead0 _sleep() at _sleep+0x29a/frame 0xffffff822ca0eb60 deadlkres() at deadlkres+0x2c3/frame 0xffffff822ca0eba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca0ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca0ebf0 --- trap 0, rip = 0, rsp = 0xffffff822ca0ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe00055f3480 sched_switch() at sched_switch+0x1b4/frame 0xffffff822ca09a40 mi_switch() at mi_switch+0x238/frame 0xffffff822ca09a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff822ca09ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff822ca09b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff822ca09b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff822ca09ba0 fork_exit() at fork_exit+0x139/frame 0xffffff822ca09bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff822ca09bf0 --- trap 0, rip = 0, rsp = 0xffffff822ca09cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe00053a8480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b6a40 mi_switch() at mi_switch+0x238/frame 0xffffff80002b6a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002b6ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b6b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002b6b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002b6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b6bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b6cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00053a8900 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b1a40 mi_switch() at mi_switch+0x238/frame 0xffffff80002b1a90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002b1ad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b1b00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002b1b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002b1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b1bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b1cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00053a9000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002aca40 mi_switch() at mi_switch+0x238/frame 0xffffff80002aca90 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002acad0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002acb00 msleep_spin() at msleep_spin+0x22d/frame 0xffffff80002acb70 taskqueue_thread_loop() at taskqueue_thread_loop+0x6f/frame 0xffffff80002acba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002acbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002acbf0 --- trap 0, rip = 0, rsp = 0xffffff80002accb0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00053a9480 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a7a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002a7a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002a7ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002a7ae0 _sleep() at _sleep+0x3e9/frame 0xffffff80002a7b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff80002a7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a7bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a7cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe0005208900 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000293a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000293a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000293ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000293ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000293b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff8000293ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000293bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000293bf0 --- trap 0, rip = 0, rsp = 0xffffff8000293cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe0005209480 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000289a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000289a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000289ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000289ae0 _sleep() at _sleep+0x3e9/frame 0xffffff8000289b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff8000289ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000289bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000289bf0 --- trap 0, rip = 0, rsp = 0xffffff8000289cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe000520b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800027ea20 mi_switch() at mi_switch+0x238/frame 0xffffff800027ea70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800027eab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800027eae0 _sleep() at _sleep+0x3e9/frame 0xffffff800027eb70 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xffffff800027eba0 fork_exit() at fork_exit+0x139/frame 0xffffff800027ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800027ebf0 --- trap 0, rip = 0, rsp = 0xffffff800027ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff8135aad0 sched_switch() at sched_switch+0x1b4/frame 0xffffffff818f4b00 mi_switch() at mi_switch+0x238/frame 0xffffffff818f4b50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffffff818f4b90 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffffff818f4bc0 _sleep() at _sleep+0x29a/frame 0xffffffff818f4c50 scheduler() at scheduler+0x2b0/frame 0xffffffff818f4c90 mi_startup() at mi_startup+0x77/frame 0xffffffff818f4cb0 btext() at btext+0x2c db:0:allt> call doadump Dumping 620 out of 8040 MB:..3%..11%..21%..31%..42%..52%..62%..73%..83%..91% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 7128 [Switching to thread 78 (Thread 100256)]#0 sched_switch (td=0xfffffe017bcc9000, newtd=0xfffffe00051e0900, flags=) at ../../../kern/sched_ule.c:1954 1954 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffffe017bcc9000, newtd=0xfffffe00051e0900, flags=) at ../../../kern/sched_ule.c:1954 #1 0xffffffff808e69f8 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:485 #2 0xffffffff80923d0e in sleepq_switch (wchan=0xffffffff8155dcd8, pri=0x0) at ../../../kern/subr_sleepqueue.c:543 #3 0xffffffff80924a2d in sleepq_wait (wchan=0xffffffff8155dcd8, pri=0x0) at ../../../kern/subr_sleepqueue.c:622 #4 0xffffffff808e7279 in _sleep (ident=, lock=0xffffffff8155dca0, priority=, wmesg=0xffffffff80f379f9 "dqflsh", timo=0x0) at ../../../kern/kern_synch.c:250 #5 0xffffffff80b427a3 in quotaoff1 (td=0xfffffe017bcc9000, mp=0xfffffe00055dec00, type=0x0) at ../../../ufs/ufs/ufs_quota.c:1644 #6 0xffffffff80b42979 in quotaoff_inchange (td=, mp=0xfffffe014b193000, type=0x0) at ../../../ufs/ufs/ufs_quota.c:709 #7 0xffffffff80b30392 in ffs_flushfiles (mp=0xfffffe014b193000, flags=0x2, td=0xfffffe017bcc9000) at ../../../ufs/ffs/ffs_vfsops.c:1364 #8 0xffffffff80b2d4c4 in softdep_flushfiles (oldmnt=0xfffffe014b193000, flags=0x2, td=0xfffffe017bcc9000) at ../../../ufs/ffs/ffs_softdep.c:1929 #9 0xffffffff80b31761 in ffs_unmount (mp=0xfffffe014b193000, mntflags=) at ../../../ufs/ffs/ffs_vfsops.c:1273 #10 0xffffffff809792f9 in dounmount (mp=0xfffffe014b193000, flags=0x8080000, td=) at ../../../kern/vfs_mount.c:1314 #11 0xffffffff80979a8e in sys_unmount (td=0xfffffe017bcc9000, uap=0xffffff823faacb70) at ../../../kern/vfs_mount.c:1211 #12 0xffffffff80c80573 in amd64_syscall (td=0xfffffe017bcc9000, traced=0x0) at subr_syscall.c:134 #13 0xffffffff80c6a7b7 in Xfast_syscall () at ../../../amd64/amd64/exception.S:387 #14 0x00000008008841fa in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xffffffff80b427a3 in quotaoff1 (td=0xfffffe017bcc9000, mp=0xfffffe00055dec00, type=0x0) at ../../../ufs/ufs/ufs_quota.c:1644 1644 msleep(&dqhashtbl_wakeup, &dqhlock, 0, "dqflsh", 0); (kgdb) l 1639 } 1640 } 1641 if (refs == 0) 1642 break; 1643 dqhashtbl_wakeup = 1; 1644 msleep(&dqhashtbl_wakeup, &dqhlock, 0, "dqflsh", 0); 1645 } 1646 DQH_UNLOCK(); 1647 } 1648 (kgdb) info loc vp = (struct vnode *) 0x0 qvp = (struct vnode *) 0xfffffe0135573000 mvp = (struct vnode *) 0x0 ump = (struct ufsmount *) 0xfffffe00055dec00 dq = cr = (struct ucred *) 0xfffffe0008c48600 error = (kgdb) p dqhashtbl_wakeup $1 = 0x1 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ufs/ufs_quota.c =================================================================== --- /usr/src/sys/ufs/ufs/ufs_quota.c (revision 247264) +++ /usr/src/sys/ufs/ufs/ufs_quota.c (working copy) @@ -675,7 +675,9 @@ again: } dqflush(qvp); - /* Clear um_quotas before closing the quota vnode to prevent + + /* + * Clear um_quotas before closing the quota vnode to prevent * access to the closed vnode from dqget/dqsync */ UFS_LOCK(ump); @@ -1099,6 +1101,7 @@ qsyncvp(struct vnode *vp) (&dqhashtbl[((((intptr_t)(dqvp)) >> 8) + id) & dqhash]) static LIST_HEAD(dqhash, dquot) *dqhashtbl; static u_long dqhash; +static int dqhashtbl_wakeup; /* * Dquot free list. @@ -1497,6 +1500,10 @@ sync: goto sync; } TAILQ_INSERT_TAIL(&dqfreelist, dq, dq_freelist); + if (dqhashtbl_wakeup != 0) { + dqhashtbl_wakeup = 0; + wakeup(&dqhashtbl_wakeup); + } DQH_UNLOCK(); } @@ -1599,23 +1606,42 @@ dqflush(struct vnode *vp) { struct dquot *dq, *nextdq; struct dqhash *dqh; + int refs; /* * Move all dquot's that used to refer to this quota - * file off their hash chains (they will eventually - * fall off the head of the free list and be re-used). + * file off their hash chains. They will eventually + * fall off the head of the free list and be re-used. */ DQH_LOCK(); - for (dqh = &dqhashtbl[dqhash]; dqh >= dqhashtbl; dqh--) { - for (dq = LIST_FIRST(dqh); dq; dq = nextdq) { - nextdq = LIST_NEXT(dq, dq_hash); - if (dq->dq_ump->um_quotas[dq->dq_type] != vp) - continue; - if (dq->dq_cnt) - panic("dqflush: stray dquot"); - LIST_REMOVE(dq, dq_hash); - dq->dq_ump = (struct ufsmount *)0; + for (;;) { + refs = 0; + for (dqh = &dqhashtbl[dqhash]; dqh >= dqhashtbl; dqh--) { + for (dq = LIST_FIRST(dqh); dq; dq = nextdq) { + nextdq = LIST_NEXT(dq, dq_hash); + if (dq->dq_ump->um_quotas[dq->dq_type] != vp) + continue; + + /* + * Softupdates could keep additional + * references on the dquot structure. + * Wait for them to finish with dquot + * before exiting the flush. Dquot + * without a reference is removed from + * the chain immediately. + */ + if (dq->dq_cnt > 0) + refs = 1; + else { + LIST_REMOVE(dq, dq_hash); + dq->dq_ump = NULL; + } + } } + if (refs == 0) + break; + dqhashtbl_wakeup = 1; + msleep(&dqhashtbl_wakeup, &dqhlock, 0, "dqflsh", 0); } DQH_UNLOCK(); } @@ -1651,7 +1677,7 @@ quotaref(vp, qrp) */ found = 0; ip = VTOI(vp); - mtx_lock(&dqhlock); + DQH_LOCK(); for (i = 0; i < MAXQUOTAS; i++) { if ((dq = ip->i_dquot[i]) == NODQUOT) continue; @@ -1659,7 +1685,7 @@ quotaref(vp, qrp) qrp[i] = dq; found++; } - mtx_unlock(&dqhlock); + DQH_UNLOCK(); return (found); } Index: /usr/src/sys/ufs/ffs/ffs_alloc.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_alloc.c (revision 247264) +++ /usr/src/sys/ufs/ffs/ffs_alloc.c (working copy) @@ -1790,6 +1790,17 @@ fail: return (0); } +static inline struct buf * +getinobuf(struct inode *ip, u_int cg, u_int32_t cginoblk, int gbflags) +{ + struct fs *fs; + + fs = ip->i_fs; + return (getblk(ip->i_devvp, fsbtodb(fs, ino_to_fsba(fs, + cg * fs->fs_ipg + cginoblk)), (int)fs->fs_bsize, 0, 0, + gbflags)); +} + /* * Determine whether an inode can be allocated. * @@ -1814,9 +1825,11 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) u_int8_t *inosused, *loc; struct ufs2_dinode *dp2; int error, start, len, i; + u_int32_t old_initediblk; fs = ip->i_fs; ump = ip->i_ump; +check_nifree: if (fs->fs_cs(fs, cg).cs_nifree == 0) return (0); UFS_UNLOCK(ump); @@ -1828,13 +1841,13 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) return (0); } cgp = (struct cg *)bp->b_data; +restart: if (!cg_chkmagic(cgp) || cgp->cg_cs.cs_nifree == 0) { brelse(bp); UFS_LOCK(ump); return (0); } bp->b_xflags |= BX_BKGRDWRITE; - cgp->cg_old_time = cgp->cg_time = time_second; inosused = cg_inosused(cgp); if (ipref) { ipref %= fs->fs_ipg; @@ -1856,7 +1869,6 @@ ffs_nodealloccg(ip, cg, ipref, mode, unused) } } ipref = (loc - inosused) * NBBY + ffs(~*loc) - 1; - cgp->cg_irotor = ipref; gotit: /* * Check to see if we need to initialize more inodes. @@ -1864,9 +1876,37 @@ gotit: if (fs->fs_magic == FS_UFS2_MAGIC && ipref + INOPB(fs) > cgp->cg_initediblk && cgp->cg_initediblk < cgp->cg_niblk) { - ibp = getblk(ip->i_devvp, fsbtodb(fs, - ino_to_fsba(fs, cg * fs->fs_ipg + cgp->cg_initediblk)), - (int)fs->fs_bsize, 0, 0, 0); + old_initediblk = cgp->cg_initediblk; + + /* + * Free the cylinder group lock before writing the + * initialized inode block. Entering the + * babarrierwrite() with the cylinder group lock + * causes lock order violation between the lock and + * snaplk. + * + * Another thread can decide to initialize the same + * inode block, but whichever thread first gets the + * cylinder group lock after writing the newly + * allocated inode block will update it and the other + * will realize that it has lost and leave the + * cylinder group unchanged. + */ + ibp = getinobuf(ip, cg, old_initediblk, GB_LOCK_NOWAIT); + brelse(bp); + if (ibp == NULL) { + /* + * The inode block buffer is already owned by + * another thread, which must initialize it. + * Wait on the buffer to allow another thread + * to finish the updates, with dropped cg + * buffer lock, then retry. + */ + ibp = getinobuf(ip, cg, old_initediblk, 0); + brelse(ibp); + UFS_LOCK(ump); + goto check_nifree; + } bzero(ibp->b_data, (int)fs->fs_bsize); dp2 = (struct ufs2_dinode *)(ibp->b_data); for (i = 0; i < INOPB(fs); i++) { @@ -1883,8 +1923,29 @@ gotit: * loading of newly created filesystems. */ babarrierwrite(ibp); - cgp->cg_initediblk += INOPB(fs); + + /* + * After the inode block is written, try to update the + * cg initediblk pointer. If another thread beat us + * to it, then leave it unchanged as the other thread + * has already set it correctly. + */ + error = bread(ip->i_devvp, fsbtodb(fs, cgtod(fs, cg)), + (int)fs->fs_cgsize, NOCRED, &bp); + UFS_LOCK(ump); + ACTIVECLEAR(fs, cg); + UFS_UNLOCK(ump); + if (error != 0) { + brelse(bp); + return (error); + } + cgp = (struct cg *)bp->b_data; + if (cgp->cg_initediblk == old_initediblk) + cgp->cg_initediblk += INOPB(fs); + goto restart; } + cgp->cg_old_time = cgp->cg_time = time_second; + cgp->cg_irotor = ipref; UFS_LOCK(ump); ACTIVECLEAR(fs, cg); setbit(inosused, ipref); -- Test scenario: quota8.sh