GENERIC HEAD from 2013-06-05 01:22:59 UTC, r251406M, vmcore.18 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r251406M: Wed Jun 5 22:54:14 CEST 2013 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 0x10 Model = 0x2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 7831900160 (7469 MB) : Trying to mount root from ufs:/dev/ufs/root [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ufs/root: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/root: clean, 433305 free (905 frags, 54050 blocks, 0.1% fragmentation) /dev/ufs/home: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/home: clean, 797843 free (8347 frags, 98687 blocks, 0.8% fragmentation) /dev/ufs/usr: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/usr: clean, 4515813 free (207133 frags, 538585 blocks, 2.0% fragmentation) /dev/label/tmp: clean, 47941391 free (7559 frags, 5991729 blocks, 0.0% fragmentation) /dev/ufs/var: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/var: clean, 6412089 free (63609 frags, 793560 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat/pkg /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. savecore: unable to read from bounds, using 0 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device Jun 5 23:12:14 x4 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device savecore: unable to read from bounds, using 0 No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xffffff81e8e7c5d8 bufwait (bufwait) @ kern/vfs_bio.c:3028 2nd 0xfffffe001a77d600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82328623d0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232862490 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82328624b0 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232862550 _sx_xlock() at _sx_xlock+0x74/frame 0xffffff8232862580 ufsdirhash_acquire() at ufsdirhash_acquire+0x44/frame 0xffffff82328625a0 ufsdirhash_add() at ufsdirhash_add+0x19/frame 0xffffff82328625d0 ufs_direnter() at ufs_direnter+0x6c1/frame 0xffffff82328626a0 ufs_mkdir() at ufs_mkdir+0x50e/frame 0xffffff8232862890 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc9/frame 0xffffff82328628c0 kern_mkdirat() at kern_mkdirat+0x230/frame 0xffffff8232862ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232862bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232862bf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092679a, rsp = 0x7fffffffd788, rbp = 0x801006050 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting default moused. Expensive timeout(9) function: 0xffffffff80749f90(0xffffffff815f73c0) 0.011481213 s Starting inetd. Wed Jun 5 23:12:17 CEST 2013 FreeBSD/amd64 (x4.osted.lan) (console) login: Jun 5 23:12:42 x4 su: pho to root on /dev/pts/1 20130605 23:12:56 all (1/5): msync.sh 20130605 23:14:50 all (2/5): mlockall.sh 20130605 23:15:01 all (3/5): mlockall2.sh 20130605 23:20:36 all (4/5): mlockall3.sh 20130605 23:23:38 all (5/5): wire_no_page.sh 20130605 23:23:48 all (1/5): msync.sh Expensive timeout(9) function: 0xffffffff807971b0(0xffffff8000944320) 0.018336617 s 20130605 23:25:40 all (2/5): mlockall.sh 20130605 23:25:51 all (3/5): mlockall2.sh 20130605 23:30:56 all (4/5): mlockall3.sh 20130605 23:33:57 all (5/5): wire_no_page.sh 20130605 23:34:07 all (1/5): msync.sh 20130605 23:35:59 all (2/5): mlockall.sh 20130605 23:36:11 all (3/5): mlockall2.sh 20130605 23:41:42 all (4/5): mlockall3.sh 20130605 23:44:43 all (5/5): wire_no_page.sh 20130605 23:44:53 all (1/5): msync.sh 20130605 23:46:45 all (2/5): mlockall.sh 20130605 23:46:56 all (3/5): mlockall2.sh 20130605 23:52:32 all (4/5): mlockall3.sh 20130605 23:55:33 all (5/5): wire_no_page.sh 20130605 23:55:43 all (1/5): msync.sh 20130605 23:57:35 all (2/5): mlockall.sh 20130605 23:57:47 all (3/5): mlockall2.sh 20130606 00:03:21 all (4/5): mlockall3.sh 20130606 00:06:22 all (5/5): wire_no_page.sh 20130606 00:06:32 all (1/5): msync.sh 20130606 00:08:25 all (2/5): mlockall.sh 20130606 00:08:36 all (3/5): mlockall2.sh 20130606 00:14:10 all (4/5): mlockall3.sh 20130606 00:17:11 all (5/5): wire_no_page.sh 20130606 00:17:21 all (1/5): msync.sh 20130606 00:19:13 all (2/5): mlockall.sh 20130606 00:19:25 all (3/5): mlockall2.sh 20130606 00:24:41 all (4/5): mlockall3.sh 20130606 00:27:43 all (5/5): wire_no_page.sh 20130606 00:27:53 all (1/5): msync.sh 20130606 00:29:45 all (2/5): mlockall.sh 20130606 00:29:56 all (3/5): mlockall2.sh 20130606 00:35:24 all (4/5): mlockall3.sh 20130606 00:38:25 all (5/5): wire_no_page.sh 20130606 00:38:35 all (1/5): msync.sh 20130606 00:40:27 all (2/5): mlockall.sh 20130606 00:40:38 all (3/5): mlockall2.sh 20130606 00:46:29 all (4/5): mlockall3.sh 20130606 00:49:30 all (5/5): wire_no_page.sh 20130606 00:49:40 all (1/5): msync.sh 20130606 00:51:32 all (2/5): mlockall.sh 20130606 00:51:43 all (3/5): mlockall2.sh 20130606 00:57:18 all (4/5): mlockall3.sh 20130606 01:00:19 all (5/5): wire_no_page.sh 20130606 01:00:29 all (1/5): msync.sh 20130606 01:02:21 all (2/5): mlockall.sh 20130606 01:02:33 all (3/5): mlockall2.sh 20130606 01:08:10 all (4/5): mlockall3.sh 20130606 01:11:11 all (5/5): wire_no_page.sh 20130606 01:11:21 all (1/5): msync.sh 20130606 01:13:13 all (2/5): mlockall.sh 20130606 01:13:25 all (3/5): mlockall2.sh 20130606 01:18:19 all (4/5): mlockall3.sh 20130606 01:21:20 all (5/5): wire_no_page.sh 20130606 01:21:30 all (1/5): msync.sh 20130606 01:23:22 all (2/5): mlockall.sh 20130606 01:23:33 all (3/5): mlockall2.sh 20130606 01:29:06 all (4/5): mlockall3.sh 20130606 01:32:07 all (5/5): wire_no_page.sh 20130606 01:32:17 all (1/5): msync.sh 20130606 01:34:09 all (2/5): mlockall.sh 20130606 01:34:20 all (3/5): mlockall2.sh 20130606 01:39:28 all (4/5): mlockall3.sh 20130606 01:42:30 all (5/5): wire_no_page.sh 20130606 01:42:39 all (1/5): msync.sh 20130606 01:44:32 all (2/5): mlockall.sh 20130606 01:44:43 all (3/5): mlockall2.sh 20130606 01:50:38 all (4/5): mlockall3.sh 20130606 01:53:39 all (5/5): wire_no_page.sh 20130606 01:53:49 all (1/5): msync.sh 20130606 01:55:42 all (2/5): mlockall.sh 20130606 01:55:53 all (3/5): mlockall2.sh 20130606 02:01:12 all (4/5): mlockall3.sh 20130606 02:04:14 all (5/5): wire_no_page.sh 20130606 02:04:24 all (1/5): msync.sh 20130606 02:06:16 all (2/5): mlockall.sh 20130606 02:06:27 all (3/5): mlockall2.sh 20130606 02:12:05 all (4/5): mlockall3.sh 20130606 02:15:06 all (5/5): wire_no_page.sh 20130606 02:15:16 all (1/5): msync.sh 20130606 02:17:08 all (2/5): mlockall.sh 20130606 02:17:19 all (3/5): mlockall2.sh 20130606 02:22:49 all (4/5): mlockall3.sh 20130606 02:25:50 all (5/5): wire_no_page.sh 20130606 02:26:00 all (1/5): msync.sh 20130606 02:27:52 all (2/5): mlockall.sh 20130606 02:28:03 all (3/5): mlockall2.sh 20130606 02:33:34 all (4/5): mlockall3.sh 20130606 02:36:35 all (5/5): wire_no_page.sh 20130606 02:36:45 all (1/5): msync.sh 20130606 02:38:38 all (2/5): mlockall.sh 20130606 02:38:49 all (3/5): mlockall2.sh 20130606 02:44:34 all (4/5): mlockall3.sh 20130606 02:47:35 all (5/5): wire_no_page.sh 20130606 02:47:45 all (1/5): msync.sh 20130606 02:49:37 all (2/5): mlockall.sh 20130606 02:49:48 all (3/5): mlockall2.sh 20130606 02:55:13 all (4/5): mlockall3.sh 20130606 02:58:14 all (5/5): wire_no_page.sh 20130606 02:58:24 all (1/5): msync.sh 20130606 03:00:16 all (2/5): mlockall.sh 20130606 03:00:28 all (3/5): mlockall2.sh 20130606 03:06:27 all (4/5): mlockall3.sh 20130606 03:09:32 all (5/5): wire_no_page.sh 20130606 03:09:44 all (1/5): msync.sh 20130606 03:11:37 all (2/5): mlockall.sh 20130606 03:11:50 all (3/5): mlockall2.sh 20130606 03:17:15 all (4/5): mlockall3.sh 20130606 03:20:20 all (5/5): wire_no_page.sh 20130606 03:20:30 all (1/5): msync.sh 20130606 03:22:23 all (2/5): mlockall.sh 20130606 03:22:35 all (3/5): mlockall2.sh 20130606 03:27:59 all (4/5): mlockall3.sh 20130606 03:31:01 all (5/5): wire_no_page.sh 20130606 03:31:11 all (1/5): msync.sh 20130606 03:33:04 all (2/5): mlockall.sh 20130606 03:33:15 all (3/5): mlockall2.sh 20130606 03:38:42 all (4/5): mlockall3.sh 20130606 03:41:44 all (5/5): wire_no_page.sh 20130606 03:41:54 all (1/5): msync.sh 20130606 03:43:47 all (2/5): mlockall.sh 20130606 03:43:58 all (3/5): mlockall2.sh 20130606 03:49:32 all (4/5): mlockall3.sh 20130606 03:52:33 all (5/5): wire_no_page.sh 20130606 03:52:43 all (1/5): msync.sh 20130606 03:54:36 all (2/5): mlockall.sh 20130606 03:54:48 all (3/5): mlockall2.sh 20130606 04:00:30 all (4/5): mlockall3.sh 20130606 04:03:31 all (5/5): wire_no_page.sh 20130606 04:03:41 all (1/5): msync.sh 20130606 04:05:34 all (2/5): mlockall.sh 20130606 04:05:46 all (3/5): mlockall2.sh 20130606 04:11:18 all (4/5): mlockall3.sh 20130606 04:14:20 all (5/5): wire_no_page.sh 20130606 04:14:30 all (1/5): msync.sh 20130606 04:16:23 all (2/5): mlockall.sh 20130606 04:16:34 all (3/5): mlockall2.sh 20130606 04:22:09 all (4/5): mlockall3.sh 20130606 04:25:10 all (5/5): wire_no_page.sh 20130606 04:25:20 all (1/5): msync.sh 20130606 04:27:13 all (2/5): mlockall.sh 20130606 04:27:25 all (3/5): mlockall2.sh 20130606 04:33:11 all (4/5): mlockall3.sh 20130606 04:36:12 all (5/5): wire_no_page.sh 20130606 04:36:23 all (1/5): msync.sh 20130606 04:38:15 all (2/5): mlockall.sh 20130606 04:38:27 all (3/5): mlockall2.sh 20130606 04:43:56 all (4/5): mlockall3.sh 20130606 04:46:58 all (5/5): wire_no_page.sh 20130606 04:47:08 all (1/5): msync.sh 20130606 04:49:01 all (2/5): mlockall.sh 20130606 04:49:12 all (3/5): mlockall2.sh 20130606 04:54:57 all (4/5): mlockall3.sh 20130606 04:57:59 all (5/5): wire_no_page.sh 20130606 04:58:09 all (1/5): msync.sh 20130606 05:00:02 all (2/5): mlockall.sh 20130606 05:00:13 all (3/5): mlockall2.sh 20130606 05:05:57 all (4/5): mlockall3.sh 20130606 05:08:58 all (5/5): wire_no_page.sh 20130606 05:09:08 all (1/5): msync.sh 20130606 05:11:01 all (2/5): mlockall.sh 20130606 05:11:13 all (3/5): mlockall2.sh 20130606 05:17:01 all (4/5): mlockall3.sh 20130606 05:20:03 all (5/5): wire_no_page.sh 20130606 05:20:13 all (1/5): msync.sh 20130606 05:22:05 all (2/5): mlockall.sh 20130606 05:22:17 all (3/5): mlockall2.sh 20130606 05:27:38 all (4/5): mlockall3.sh 20130606 05:30:40 all (5/5): wire_no_page.sh 20130606 05:30:50 all (1/5): msync.sh 20130606 05:32:43 all (2/5): mlockall.sh 20130606 05:32:54 all (3/5): mlockall2.sh 20130606 05:38:33 all (4/5): mlockall3.sh 20130606 05:41:35 all (5/5): wire_no_page.sh 20130606 05:41:45 all (1/5): msync.sh 20130606 05:43:38 all (2/5): mlockall.sh 20130606 05:43:49 all (3/5): mlockall2.sh 20130606 05:49:29 all (4/5): mlockall3.sh 20130606 05:52:31 all (5/5): wire_no_page.sh 20130606 05:52:41 all (1/5): msync.sh 20130606 05:54:34 all (2/5): mlockall.sh 20130606 05:54:45 all (3/5): mlockall2.sh 20130606 05:59:55 all (4/5): mlockall3.sh 20130606 06:02:56 all (5/5): wire_no_page.sh 20130606 06:03:07 all (1/5): msync.sh 20130606 06:04:59 all (2/5): mlockall.sh 20130606 06:05:11 all (3/5): mlockall2.sh 20130606 06:10:23 all (4/5): mlockall3.sh 20130606 06:13:25 all (5/5): wire_no_page.sh 20130606 06:13:35 all (1/5): msync.sh 20130606 06:15:28 all (2/5): mlockall.sh 20130606 06:15:39 all (3/5): mlockall2.sh 20130606 06:21:04 all (4/5): mlockall3.sh 20130606 06:24:06 all (5/5): wire_no_page.sh 20130606 06:24:16 all (1/5): msync.sh 20130606 06:26:09 all (2/5): mlockall.sh 20130606 06:26:20 all (3/5): mlockall2.sh 20130606 06:32:02 all (4/5): mlockall3.sh 20130606 06:35:04 all (5/5): wire_no_page.sh 20130606 06:35:14 all (1/5): msync.sh 20130606 06:37:07 all (2/5): mlockall.sh 20130606 06:37:18 all (3/5): mlockall2.sh 20130606 06:42:40 all (4/5): mlockall3.sh 20130606 06:45:41 all (5/5): wire_no_page.sh 20130606 06:45:51 all (1/5): msync.sh 20130606 06:47:44 all (2/5): mlockall.sh 20130606 06:47:56 all (3/5): mlockall2.sh 20130606 06:53:25 all (4/5): mlockall3.sh 20130606 06:56:26 all (5/5): wire_no_page.sh 20130606 06:56:36 all (1/5): msync.sh 20130606 06:58:29 all (2/5): mlockall.sh 20130606 06:58:40 all (3/5): mlockall2.sh 20130606 07:03:57 all (4/5): mlockall3.sh 20130606 07:06:59 all (5/5): wire_no_page.sh 20130606 07:07:09 all (1/5): msync.sh 20130606 07:09:02 all (2/5): mlockall.sh 20130606 07:09:13 all (3/5): mlockall2.sh 20130606 07:14:33 all (4/5): mlockall3.sh 20130606 07:17:34 all (5/5): wire_no_page.sh 20130606 07:17:45 all (1/5): msync.sh 20130606 07:19:37 all (2/5): mlockall.sh 20130606 07:19:49 all (3/5): mlockall2.sh 20130606 07:25:26 all (4/5): mlockall3.sh 20130606 07:28:27 all (5/5): wire_no_page.sh 20130606 07:28:37 all (1/5): msync.sh 20130606 07:30:30 all (2/5): mlockall.sh 20130606 07:30:42 all (3/5): mlockall2.sh lock order reversal: 1st 0xfffffe001ad4b2d8 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe01842e57b8 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1411 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232b914b0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232b91570 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232b91590 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232b91630 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff8232b91720 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff8232b91740 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8232b91770 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8232b917f0 ffs_flushfiles() at ffs_flushfiles+0x114/frame 0xffffff8232b91870 softdep_flushfiles() at softdep_flushfiles+0x54/frame 0xffffff8232b918e0 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff8232b91950 dounmount() at dounmount+0x372/frame 0xffffff8232b919d0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff8232b91ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232b91bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232b91bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x80088523a, rsp = 0x7fffffffceb8, rbp = 0x801006ce8 --- Jun 6 07:41:14 x4 su: pho to root on /dev/pts/2 lock order reversal: 1st 0xfffffe0160cf8a28 ufs (ufs) @ kern/vfs_subr.c:2099 2nd 0xffffff81e9fc2c58 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 3rd 0xfffffe01f4d4c548 ufs (ufs) @ kern/vfs_subr.c:2099 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232efbd70 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232efbe30 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232efbe50 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232efbef0 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff8232efbfe0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff8232efc030 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8232efc060 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8232efc0e0 vget() at vget+0x63/frame 0xffffff8232efc130 vfs_hash_get() at vfs_hash_get+0xd5/frame 0xffffff8232efc180 ffs_vgetf() at ffs_vgetf+0x48/frame 0xffffff8232efc210 softdep_sync_buf() at softdep_sync_buf+0x3ab/frame 0xffffff8232efc300 ffs_syncvnode() at ffs_syncvnode+0x2fc/frame 0xffffff8232efc380 ffs_truncate() at ffs_truncate+0x10ec/frame 0xffffff8232efc5d0 ufs_direnter() at ufs_direnter+0x538/frame 0xffffff8232efc6a0 ufs_mkdir() at ufs_mkdir+0x50e/frame 0xffffff8232efc890 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc9/frame 0xffffff8232efc8c0 kern_mkdirat() at kern_mkdirat+0x230/frame 0xffffff8232efcad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232efcbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232efcbf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092679a, rsp = 0x7fffffffd418, rbp = 0x7fffffffd909 --- 20130606 08:26:32 all (1/226): ldt2.sh 20130606 08:26:35 all (2/226): rename8.sh 20130606 08:27:44 all (3/226): fullpath.sh 20130606 08:29:41 all (4/226): msdos2.sh 20130606 08:29:51 all (5/226): suj33.sh lock order reversal: 1st 0xffffffff815b0280 buf queue lock (buf queue lock) @ kern/vfs_bio.c:2585 2nd 0xfffffe001ab2b000 struct mount mtx (struct mount mtx) @ kern/vfs_mount.c:433 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232835850 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232835910 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232835930 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff82328359d0 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xffffff8232835a00 vfs_ref() at vfs_ref+0x2e/frame 0xffffff8232835a20 vop_stdgetwritemount() at vop_stdgetwritemount+0x29/frame 0xffffff8232835a40 VOP_GETWRITEMOUNT_APV() at VOP_GETWRITEMOUNT_APV+0xb8/frame 0xffffff8232835a70 vn_start_write() at vn_start_write+0x3f/frame 0xffffff8232835ac0 flushbufqueues() at flushbufqueues+0x2e0/frame 0xffffff8232835b60 buf_do_flush() at buf_do_flush+0x20/frame 0xffffff8232835b80 buf_daemon() at buf_daemon+0xe4/frame 0xffffff8232835ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232835bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232835bf0 --- trap 0, rip = 0, rsp = 0xffffff8232835cb0, rbp = 0 --- 20130606 08:30:35 all (6/226): snap2.sh lock order reversal: 1st 0xffffff81e9513688 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 2nd 0xfffffe007f4eb130 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2297 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232f29350 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232f29410 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232f29430 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232f294d0 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff8232f295c0 ffs_copyonwrite() at ffs_copyonwrite+0x199/frame 0xffffff8232f29660 ffs_geom_strategy() at ffs_geom_strategy+0x1ba/frame 0xffffff8232f29690 ufs_strategy() at ufs_strategy+0x45/frame 0xffffff8232f296c0 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0xd1/frame 0xffffff8232f296f0 bufstrategy() at bufstrategy+0x4b/frame 0xffffff8232f29730 bufwrite() at bufwrite+0x11a/frame 0xffffff8232f29760 ffs_syncvnode() at ffs_syncvnode+0x39b/frame 0xffffff8232f297e0 ffs_fsync() at ffs_fsync+0x52/frame 0xffffff8232f29820 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff8232f29850 ufs_remove() at ufs_remove+0xf3/frame 0xffffff8232f298c0 VOP_REMOVE_APV() at VOP_REMOVE_APV+0xeb/frame 0xffffff8232f298f0 kern_unlinkat() at kern_unlinkat+0x28b/frame 0xffffff8232f29ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232f29bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232f29bf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x800925e4a, rsp = 0x7fffffffd558, rbp = 0 --- lock order reversal: 1st 0xfffffe007f4eb130 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:968 2nd 0xfffffe01ca2faa28 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1628 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232f29280 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232f29340 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232f29360 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232f29400 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff8232f294f0 ffs_snapremove() at ffs_snapremove+0xe2/frame 0xffffff8232f29570 ffs_truncate() at ffs_truncate+0xd34/frame 0xffffff8232f297c0 ufs_inactive() at ufs_inactive+0x28c/frame 0xffffff8232f29800 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xc9/frame 0xffffff8232f29830 vinactive() at vinactive+0xb2/frame 0xffffff8232f29890 vputx() at vputx+0x365/frame 0xffffff8232f298f0 kern_unlinkat() at kern_unlinkat+0x1a1/frame 0xffffff8232f29ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232f29bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232f29bf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x800925e4a, rsp = 0x7fffffffd558, rbp = 0 --- 20130606 08:31:31 all (7/226): lookup_shared.sh 20130606 08:54:25 all (8/226): md4.sh 20130606 08:54:30 all (9/226): nfs.sh 20130606 09:06:29 all (10/226): msdos.sh 20130606 09:16:56 all (11/226): linger.sh 20130606 09:36:33 all (12/226): suj14.sh Jun 6 09:38:35 x4 kernel: pid 49609 (cp), uid 0 inumber 50994 on /mnt: filesystem full Jun 6 09:38:36 x4 kernel: pid 49684 (cp), uid 0 inumber 51007 on /mnt: filesystem full Jun 6 09:38:37 x4 kernel: pid 49773 (cp), uid 0 inumber 51007 on /mnt: filesystem full Jun 6 09:38:38 x4 kernel: pid 49866 (cp), uid 0 inumber 51007 on /mnt: filesystem full Jun 6 09:38:39 x4 kernel: pid 49895 (cp), uid 0 inumber 51007 on /mnt: filesystem full Jun 6 09:38:40 x4 kernel: pid 49925 (mkdir), uid 0 inumber 51008 on /mnt: filesystem full 20130606 09:38:47 all (13/226): md2.sh lock order reversal: 1st 0xfffffe019639c238 ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xfffffe009da0cc98 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1710 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823369f1c0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823369f280 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823369f2a0 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff823369f340 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff823369f430 ffs_vgetf() at ffs_vgetf+0x163/frame 0xffffff823369f4c0 ufs_root() at ufs_root+0x42/frame 0xffffff823369f500 ufs_extattr_autostart() at ufs_extattr_autostart+0x89/frame 0xffffff823369f560 ffs_mount() at ffs_mount+0x21b4/frame 0xffffff823369f770 vfs_donmount() at vfs_donmount+0x9ca/frame 0xffffff823369fa90 sys_nmount() at sys_nmount+0x66/frame 0xffffff823369fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823369fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823369fbf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a98e3a, rsp = 0x7fffffffc6b8, rbp = 0x7fffffffc6d0 --- 20130606 09:59:05 all (14/226): ldt.sh kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled kernel trap 9 with interrupts disabled 20130606 09:59:06 all (15/226): mmap.sh 20130606 09:59:07 all (16/226): snap4.sh 20130607 07:28:47 all (1/1): mlockall2.sh 20130607 07:34:23 all (1/1): mlockall2.sh 20130607 07:39:59 all (1/1): mlockall2.sh 20130607 07:45:30 all (1/1): mlockall2.sh 20130607 07:51:06 all (1/1): mlockall2.sh 20130607 07:56:36 all (1/1): mlockall2.sh 20130607 08:02:30 all (1/1): mlockall2.sh 20130607 08:07:21 all (1/1): mlockall2.sh 20130607 08:10:49 all (1/16): nfs12.sh 20130607 08:11:33 all (2/16): wire_no_page.sh 20130607 08:11:43 all (3/16): mlockall.sh 20130607 08:11:55 all (4/16): kinfo2.sh 20130607 08:41:57 all (5/16): kinfo.sh 20130607 09:11:59 all (6/16): mountu.sh lock order reversal: 1st 0xfffffe01174ac2d8 msdosfs (msdosfs) @ kern/vfs_subr.c:2099 2nd 0xfffffe01645fa068 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:882 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8232d44150 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8232d44210 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8232d44230 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff8232d442d0 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff8232d443c0 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff8232d443e0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8232d44410 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8232d44490 msdosfs_fsync() at msdosfs_fsync+0x7d/frame 0xffffff8232d444e0 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff8232d44510 msdosfs_sync() at msdosfs_sync+0x227/frame 0xffffff8232d445b0 msdosfs_mount() at msdosfs_mount+0x797/frame 0xffffff8232d44770 vfs_donmount() at vfs_donmount+0xee4/frame 0xffffff8232d44a90 sys_nmount() at sys_nmount+0x66/frame 0xffffff8232d44ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232d44bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232d44bf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a85e3a, rsp = 0x7fffffffd048, rbp = 0x7fffffffd601 --- 20130607 09:12:16 all (7/16): mmap3.sh Jun 7 09:19:23 x4 kernel: Failed to write core file for process mmap3 (error 14) lock order reversal: 1st 0xfffffe001ab127b8 syncer (syncer) @ kern/vfs_subr.c:1718 2nd 0xfffffe00358802d8 ufs (ufs) @ kern/vfs_subr.c:2099 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823283f690 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823283f750 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823283f770 witness_checkorder() at witness_checkorder+0x8a1/frame 0xffffff823283f810 __lockmgr_args() at __lockmgr_args+0x38e/frame 0xffffff823283f900 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823283f950 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823283f980 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823283fa00 vget() at vget+0x63/frame 0xffffff823283fa50 vfs_msync() at vfs_msync+0x72/frame 0xffffff823283fab0 sync_fsync() at sync_fsync+0x10d/frame 0xffffff823283fae0 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff823283fb10 sched_sync() at sched_sync+0x335/frame 0xffffff823283fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823283fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823283fbf0 --- trap 0, rip = 0, rsp = 0xffffff823283fcb0, rbp = 0 --- Jun 7 09:19:54 x4 kernel: Failed to write core file for process mmap3 (error 14) Jun 7 09:21:24 x4 last message repeated 17 times 20130607 09:29:52 all (8/16): kinfo3.sh Jun 7 09:29:50 x4 last message repeated 33 times 20130607 09:45:06 all (9/16): rot.sh 20130607 10:07:52 all (10/16): mmap4.sh vnode_pager_putpages: I/O error 28 vnode_pager_putpages: residual I/O 32768 at 9776 Jun 7 10:07:53 x4 kernel: pid 58383 (mmap4), uid 0 inumber 4 on /mnt: filesystem full 20130607 10:07:55 all (11/16): mlockall2.sh 20130607 10:13:57 all (12/16): rdwr.sh 20130607 10:13:59 all (13/16): mmap.sh 20130607 10:13:59 all (14/16): mmap2.sh 20130607 10:19:23 all (15/16): msync.sh 20130607 10:21:16 all (16/16): mlockall3.sh 20130607 10:24:17 all (1/16): rot.sh 20130607 10:48:05 all (2/16): kinfo.sh Jun 7 11:07:20 x4 su: pho to root on /dev/pts/2 Expensive timeout(9) function: 0xffffffff80749f90(0xffffffff815f73c0) 0.066095419 s Expensive timeout(9) function: 0xffffffff80ccb580(0xffffffff815e5560) 0.073232134 s Expensive timeout(9) function: 0xffffffff80ac45a0(0xfffffe016a932800) 0.110186781 s Expensive timeout(9) function: 0xffffffff80749f90(0xffffffff815f73c0) 0.177078924 s Expensive timeout(9) function: 0xffffffff80ac45a0(0xfffffe00cdb30000) 0.298901733 s Expensive timeout(9) function: 0xffffffff80ac45a0(0xfffffe0056134800) 0.887811614 s Jun 7 11:18:35 x4 kernel: pid 90667 (symlink), uid 1004 inumber 65602 on /mnt: out of inodes Jun 7 11:29:38 x4 kernel: pid 94520 (creat), uid 1004 inumber 67687 on /mnt: out of inodes 20130607 11:47:12 all (1/226): devfs2.sh 20130607 11:47:13 all (2/226): tmpfs7.sh 20130607 11:47:14 all (3/226): tcp.sh witness_lock_list_get: witness exhausted 20130607 12:04:17 all (4/226): pread.sh 20130607 12:04:27 all (5/226): linger4.sh Jun 7 12:06:48 x4 kernel: pid 20364 (linger4), uid 1004 inumber 229251 on /mnt: out of inodes 20130607 12:06:53 all (6/226): syscall5.sh 20130607 12:37:19 all (7/226): md.sh Jun 7 12:39:21 x4 kernel: pid 25641 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:39:25 x4 kernel: pid 25640 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:41:24 x4 kernel: pid 25834 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:41:30 x4 kernel: pid 25835 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:45:20 x4 kernel: pid 26249 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:45:22 x4 kernel: pid 26250 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:47:23 x4 kernel: pid 26430 (rw), uid 0 inumber 133 on /mnt: filesystem full Jun 7 12:47:24 x4 kernel: pid 26428 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:47:25 x4 kernel: pid 26429 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:47:28 x4 kernel: pid 26426 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:47:29 x4 kernel: pid 26427 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:49:22 x4 kernel: pid 26806 (rw), uid 0 inumber 134 on /mnt: filesystem full Jun 7 12:49:25 x4 kernel: pid 26802 (rw), uid 0 inumber 131 on /mnt: filesystem full Jun 7 12:49:26 x4 kernel: pid 26805 (rw), uid 0 inumber 131 on /mnt: filesystem full Jun 7 12:49:28 x4 kernel: pid 26803 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:51:22 x4 kernel: pid 27187 (rw), uid 0 inumber 136 on /mnt: filesystem full Jun 7 12:51:22 x4 kernel: pid 27182 (rw), uid 0 inumber 137 on /mnt: filesystem full Jun 7 12:51:27 x4 kernel: pid 27185 (rw), uid 0 inumber 130 on /mnt: filesystem full Jun 7 12:51:28 x4 kernel: pid 27181 (rw), uid 0 inumber 131 on /mnt: filesystem full Jun 7 12:53:22 x4 kernel: pid 27550 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:53:28 x4 kernel: pid 27549 (rw), uid 0 inumber 132 on /mnt: filesystem full Jun 7 12:53:29 x4 kernel: pid 27551 (rw), uid 0 inumber 131 on /mnt: filesystem full Jun 7 12:55:22 x4 kernel: pid 27936 (rw), uid 0 inumber 133 on /mnt: filesystem full Jun 7 12:55:25 x4 kernel: pid 27940 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:55:27 x4 kernel: pid 27938 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:55:30 x4 kernel: pid 27939 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:57:26 x4 kernel: pid 28308 (rw), uid 0 inumber 140 on /mnt: filesystem full Jun 7 12:57:30 x4 kernel: pid 28297 (rw), uid 0 inumber 131 on /mnt: filesystem full Jun 7 12:57:32 x4 kernel: pid 28300 (rw), uid 0 inumber 8 on /mnt: filesystem full Jun 7 12:59:23 x4 kernel: pid 28690 (rw), uid 0 inumber 5 on /mnt: filesystem full Jun 7 12:59:25 x4 kernel: pid 28692 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:59:28 x4 kernel: pid 28691 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 12:59:30 x4 kernel: pid 28688 (rw), uid 0 inumber 129 on /mnt: filesystem full Jun 7 13:01:29 x4 kernel: pid 29068 (rw), uid 0 inumber 11 on /mnt: filesystem full Jun 7 13:01:30 x4 kernel: pid 29072 (rw), uid 0 inumber 139 on /mnt: filesystem full Jun 7 13:01:32 x4 kernel: pid 29066 (rw), uid 0 inumber 139 on /mnt: filesystem full Jun 7 13:03:26 x4 kernel: pid 29465 (rw), uid 0 inumber 15 on /mnt: filesystem full Jun 7 13:03:29 x4 kernel: pid 29467 (rw), uid 0 inumber 141 on /mnt: filesystem full Jun 7 13:05:27 x4 kernel: pid 29865 (rw), uid 0 inumber 133 on /mnt: filesystem full Jun 7 13:05:28 x4 kernel: pid 29863 (rw), uid 0 inumber 132 on /mnt: filesystem full Jun 7 13:05:30 x4 kernel: pid 29866 (rw), uid 0 inumber 5 on /mnt: filesystem full 20130607 13:07:20 all (8/226): graid3.sh GEOM_RAID3: Device raid3/data launched (3/3). GEOM_RAID3: Device data: provider raid3/data destroyed. GEOM_RAID3: Device data destroyed. 20130607 13:28:42 all (9/226): tmpfs2.sh Jun 7 13:30:18 x4 last message repeated 2 times Jun 7 13:30:55 x4 last message repeated 2 times 20130607 13:31:33 all (10/226): truncate4.sh 20130607 13:31:34 all (11/226): umount.sh 20130607 13:35:47 all (12/226): rename3.sh 20130607 13:37:36 all (13/226): newfs5.sh 20130607 13:38:04 all (14/226): suj9.sh Jun 7 13:39:04 x4 kernel: pid 59449 (mkdir), uid 1004 inumber 16367 on /mnt: out of inodes 20130607 14:08:59 all (15/226): rename7.sh 20130607 14:09:57 all (16/226): nullfs13.sh 20130607 14:09:58 all (17/226): extattr_set_fd.sh 20130607 14:09:58 all (18/226): ffs_syncvnode.sh Jun 7 14:10:30 x4 kernel: pid 70942 (creat), uid 1004 inumber 131075 on /mnt: out of inodes 20130607 14:40:36 all (19/226): suj12.sh GEOM_NOP: Device md5.nop created. GEOM_NOP: Device md5.nop removed. 20130607 15:02:52 all (20/226): sendfile.sh 20130607 15:09:35 all (21/226): symlink2.sh 20130607 15:09:43 all (22/226): kevent2.sh 20130607 15:09:51 all (23/226): namecache.sh 20130607 15:19:08 all (24/226): quota9.sh 20130607 15:19:11 all (25/226): nullfs9.sh 20130607 15:19:13 all (26/226): suj4.sh 20130607 15:49:18 all (27/226): nullfs6.sh 20130607 15:49:19 all (28/226): nfs2.sh 20130607 15:59:24 all (29/226): procfs.sh Jun 7 16:04:26 x4 last message repeated 2 times Jun 7 17:36:19 x4 kernel: Failed to write core file for process mmap3 (error 14) Jun 7 17:36:43 x4 last message repeated 2 times Jun 7 17:38:51 x4 last message repeated 8 times Jun 7 17:44:36 x4 last message repeated 48 times ~KDB: enter: Break to debugger [ thread pid 46113 tid 100822 ] Stopped at kdb_enter+0x3b: movq $0,0xac2c72(%rip) db> ps pid ppid pgrp uid state wmesg wchan cmd 49408 1083 1080 1001 S nanslp 0xffffffff813bd700 sleep 49407 26461 26459 1001 S nanslp 0xffffffff813bd700 sleep 46113 1 45936 65534 T+ (threaded) syscall4 106353 s syscall4 106340 s syscall4 106311 s syscall4 106292 s syscall4 106288 s syscall4 106116 s syscall4 105967 s syscall4 105635 s syscall4 105634 s syscall4 105499 s syscall4 105380 s syscall4 105353 s syscall4 105160 s syscall4 104806 s syscall4 104805 s syscall4 104350 s syscall4 db> bt 106353 Tracing pid 46113 tid 106353 td 0xfffffe0131d28490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390eab20 mi_switch() at mi_switch+0x238/frame 0xffffff82390eab70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390eabb0 ast() at ast+0x3d0/frame 0xffffff82390eabf0 doreti_ast() at doreti_ast+0x1f db> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x5e8200 curthread = 0xfffffe0015485490: pid 11 "idle: cpu0" curpcb = 0xffffff800025dcc0 fpcurthread = none idlethread = 0xfffffe0015485490: tid 100003 "idle: cpu0" curpmap = 0xffffffff813b60b8 tssp = 0xffffffff815c8f80 commontssp = 0xffffffff815c8f80 rsp0 = 0xffffff800025dcc0 gs32p = 0xffffffff815c70b8 ldt = 0xffffffff815c70f8 tss = 0xffffffff815c70e8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807eec3200 curthread = 0xfffffe010b774490: pid 46046 "syscall4" curpcb = 0xffffff82341bccc0 fpcurthread = none idlethread = 0xfffffe0015485000: tid 100004 "idle: cpu1" curpmap = 0xfffffe007049bdb8 tssp = 0xffffffff815c8fe8 commontssp = 0xffffffff815c8fe8 rsp0 = 0xffffff82341bccc0 gs32p = 0xffffffff815c7120 ldt = 0xffffffff815c7160 tss = 0xffffffff815c7150 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807eeca200 curthread = 0xfffffe0069896920: pid 46113 "syscall4" curpcb = 0xffffff8235b39cc0 fpcurthread = none idlethread = 0xfffffe0015491920: tid 100005 "idle: cpu2" curpmap = 0xfffffe01d55a1138 tssp = 0xffffffff815c9050 commontssp = 0xffffffff815c9050 rsp0 = 0xffffff8235b39cc0 gs32p = 0xffffffff815c7188 ldt = 0xffffffff815c71c8 tss = 0xffffffff815c71b8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807eed1200 curthread = 0xfffffe0015491490: pid 11 "idle: cpu3" curpcb = 0xffffff800026ccc0 fpcurthread = none idlethread = 0xfffffe0015491490: tid 100006 "idle: cpu3" curpmap = 0xffffffff813b60b8 tssp = 0xffffffff815c90b8 commontssp = 0xffffffff815c90b8 rsp0 = 0xffffff800026ccc0 gs32p = 0xffffffff815c71f0 ldt = 0xffffffff815c7230 tss = 0xffffffff815c7220 spin locks held: db> bt 100006 Tracing pid 11 tid 100006 td 0xfffffe0015491490 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800024cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800024cd20 trap() at trap+0x335/frame 0xffffff800024cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800024cf20 --- trap 0x13, rip = 0xffffffff80c9e386, rsp = 0xffffff800024cfe0, rbp = 0xffffff800026ca90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800026ca90 acpi_cpu_idle() at acpi_cpu_idle+0x2ba/frame 0xffffff800026cad0 cpu_idle_acpi() at cpu_idle_acpi+0x57/frame 0xffffff800026caf0 cpu_idle() at cpu_idle+0x6b/frame 0xffffff800026cb10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff800026cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026cbf0 --- trap 0, rip = 0, rsp = 0xffffff800026ccb0, rbp = 0 --- db> c Expensive timeout(9) function: 0xffffffff80956690(0xfffffe001a831000) 1.000039453 s ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x3b: movq $0,0xac2c72(%rip) db> bt 100006 Tracing pid 11 tid 100006 td 0xfffffe0015491490 sched_switch() at sched_switch+0x1b4/frame 0xffffff800026cac0 mi_switch() at mi_switch+0x238/frame 0xffffff800026cb10 sched_idletd() at sched_idletd+0x323/frame 0xffffff800026cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026cbf0 --- trap 0, rip = 0, rsp = 0xffffff800026ccb0, rbp = 0 --- db> c Expensive timeout(9) function: 0xffffffff80956690(0xfffffe0015921490) 3.000104965 s Expensive timeout(9) function: 0xffffffff80956690(0xfffffe001591f490) 3.000106083 s Stopping inetd. Waiting for PIDS: 1023. Stopping moused. Waiting for PIDS: 990. Shutting down local packages:. Stopping cron. Waiting for PIDS: 960. Stopping sshd. Waiting for PIDS: 941. Stopping watchdogd. Waiting for PIDS: 970. Stopping ntpd. Waiting for PIDS: 850. Stopping nfsd. Waiting for PIDS: 754 755. Stopping mountd. Waiting for PIDS: 745. rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered Stopping rpcbind. Waiting for PIDS: 640. Stopping devd. Waiting for PIDS: 433. Writing entropy file:. . Terminated Jun 7 18:55:16 x4 syslogd: exiting on signal 15 Jun 7 18:55:37 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 49858 1 0 20 0 16840 3184 wait Ss u0 0:00.01 -sh (sh) 0 49859 49858 0 20 0 16432 2136 - R+ u0 0:00.00 ps -l # ps -lUnobody UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 65534 46046 1 0 20 0 23556 3436 - T 1- 10:28.18 /tmp/syscall4 7 65534 46113 1 0 20 0 23684 3424 - T 1- 10:11.49 /tmp/syscall4 7 # ~KDB: enter: Break to debugger [ thread pid 46046 tid 100903 ] Stopped at kdb_enter+0x3b: movq $0,0xac2c72(%rip) db> bt 100006 Tracing pid 11 tid 100006 td 0xfffffe0015491490 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800024cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800024cd20 trap() at trap+0x335/frame 0xffffff800024cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800024cf20 --- trap 0x13, rip = 0xffffffff80c9e386, rsp = 0xffffff800024cfe0, rbp = 0xffffff800026ca90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff800026ca90 acpi_cpu_idle() at acpi_cpu_idle+0x2ba/frame 0xffffff800026cad0 cpu_idle_acpi() at cpu_idle_acpi+0x57/frame 0xffffff800026caf0 cpu_idle() at cpu_idle+0x6b/frame 0xffffff800026cb10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff800026cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026cbf0 --- trap 0, rip = 0, rsp = 0xffffff800026ccb0, rbp = 0 --- db> c Expensive timeout(9) function: 0xffffffff80980000(0) 4.000132207 s ~KDB: enter: Break to debugger [ thread pid 46046 tid 100903 ] Stopped at kdb_enter+0x3b: movq $0,0xac2c72(%rip) db> run pho db:0:pho> bt Tracing pid 46046 tid 100903 td 0xfffffe010b774490 kdb_enter() at kdb_enter+0x3b/frame 0xffffff82341bc460 kdb_break() at kdb_break+0x27/frame 0xffffff82341bc470 uart_intr() at uart_intr+0x35a/frame 0xffffff82341bc4b0 intr_event_handle() at intr_event_handle+0xb7/frame 0xffffff82341bc500 intr_execute_handlers() at intr_execute_handlers+0x5f/frame 0xffffff82341bc530 lapic_handle_intr() at lapic_handle_intr+0x32/frame 0xffffff82341bc560 Xapic_isr1() at Xapic_isr1+0xa0/frame 0xffffff82341bc560 --- interrupt, rip = 0xffffffff808faa17, rsp = 0xffffff82341bc620, rbp = 0xffffff82341bc660 --- __mtx_unlock_flags() at __mtx_unlock_flags+0xd7/frame 0xffffff82341bc660 trap_pfault() at trap_pfault+0x1eb/frame 0xffffff82341bc6f0 trap() at trap+0x41a/frame 0xffffff82341bc8f0 calltrap() at calltrap+0x8/frame 0xffffff82341bc8f0 --- trap 0xc, rip = 0xffffffff80cb78f9, rsp = 0xffffff82341bc9b0, rbp = 0xffffff82341bca80 --- casuword32() at casuword32+0x29/frame 0xffffff82341bca80 __umtx_op_rw_rdlock() at __umtx_op_rw_rdlock+0xb0/frame 0xffffff82341bcad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82341bcbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82341bcbf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083075c, rsp = 0x7fffff9fcde8, rbp = 0x801407400 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x5e8200 curthread = 0xfffffe0015486920: pid 12 "swi4: clock" curpcb = 0xffffff8000276cc0 fpcurthread = none idlethread = 0xfffffe0015485490: tid 100003 "idle: cpu0" curpmap = 0xffffffff813b60b8 tssp = 0xffffffff815c8f80 commontssp = 0xffffffff815c8f80 rsp0 = 0xffffff8000276cc0 gs32p = 0xffffffff815c70b8 ldt = 0xffffffff815c70f8 tss = 0xffffffff815c70e8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807eec3200 curthread = 0xfffffe0015485000: pid 11 "idle: cpu1" curpcb = 0xffffff8000262cc0 fpcurthread = none idlethread = 0xfffffe0015485000: tid 100004 "idle: cpu1" curpmap = 0xffffffff813b60b8 tssp = 0xffffffff815c8fe8 commontssp = 0xffffffff815c8fe8 rsp0 = 0xffffff8000262cc0 gs32p = 0xffffffff815c7120 ldt = 0xffffffff815c7160 tss = 0xffffffff815c7150 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807eeca200 curthread = 0xfffffe010b774490: pid 46046 "syscall4" curpcb = 0xffffff82341bccc0 fpcurthread = none idlethread = 0xfffffe0015491920: tid 100005 "idle: cpu2" curpmap = 0xfffffe007049bdb8 tssp = 0xffffffff815c9050 commontssp = 0xffffffff815c9050 rsp0 = 0xffffff82341bccc0 gs32p = 0xffffffff815c7188 ldt = 0xffffffff815c71c8 tss = 0xffffffff815c71b8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807eed1200 curthread = 0xfffffe0069896920: pid 46113 "syscall4" curpcb = 0xffffff8235b39cc0 fpcurthread = none idlethread = 0xfffffe0015491490: tid 100006 "idle: cpu3" curpmap = 0xfffffe01d55a1138 tssp = 0xffffffff815c90b8 commontssp = 0xffffffff815c90b8 rsp0 = 0xffffff8235b39cc0 gs32p = 0xffffffff815c71f0 ldt = 0xffffffff815c7230 tss = 0xffffffff815c7220 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xfffffe001a958b58 /dev/ufs/root on / (ufs) 0xfffffe001a959000 devfs on /dev (devfs) 0xfffffe001ab2b790 /dev/ufs/home on /home (ufs) 0xfffffe001ab2b3c8 /dev/label/tmp on /tmp (ufs) 0xfffffe001ab2b000 /dev/ufs/usr on /usr (ufs) 0xfffffe001a959b58 /dev/ufs/var on /var (ufs) 0xfffffe001a959790 procfs on /proc (procfs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 49858 1 49858 0 Ss+ ttyin 0xfffffe001556e8a8 sh 46113 1 45936 65534 T+ (threaded) syscall4 106353 s syscall4 106340 s syscall4 106311 s syscall4 106292 s syscall4 106288 s syscall4 106116 s syscall4 105967 s syscall4 105635 s syscall4 105634 s syscall4 105499 s syscall4 105380 s syscall4 105353 s syscall4 105160 s syscall4 104806 s syscall4 104805 s syscall4 104350 s syscall4 104309 s syscall4 103738 s syscall4 102213 s syscall4 101639 s syscall4 101506 s syscall4 101376 s syscall4 101374 s syscall4 101349 s syscall4 101335 s syscall4 101237 s syscall4 101229 s syscall4 101222 s syscall4 101203 s syscall4 101194 s syscall4 101176 s syscall4 101052 s syscall4 101017 s syscall4 100965 s syscall4 100939 s syscall4 100936 s syscall4 100822 Run CPU 3 syscall4 100770 s syscall4 100493 s syscall4 103980 s syscall4 46046 1 45936 65534 T+ (threaded) syscall4 101602 s syscall4 101601 s syscall4 101594 s syscall4 101535 s syscall4 101502 s syscall4 101491 s syscall4 101489 s syscall4 101484 s syscall4 101483 s syscall4 101481 s syscall4 101476 s syscall4 101474 s syscall4 101473 s syscall4 101469 s syscall4 101467 s syscall4 101456 s syscall4 101453 s syscall4 101445 s syscall4 101441 s syscall4 101440 s syscall4 101438 s syscall4 101437 s syscall4 101434 s syscall4 101425 s syscall4 101421 s syscall4 101417 s syscall4 101412 s syscall4 101406 s syscall4 101402 s syscall4 101399 s syscall4 101397 s syscall4 101365 s syscall4 101364 s syscall4 101361 s syscall4 101346 s syscall4 101341 s syscall4 101276 s syscall4 100903 Run CPU 2 syscall4 100588 s syscall4 103855 s syscall4 17 0 0 0 DL sdflush 0xffffffff815bc180 [softdepflush] 16 0 0 0 DL syncer 0xffffffff815b0780 [syncer] 9 0 0 0 DL vlruwt 0xfffffe001a768000 [vnlru] 8 0 0 0 DL psleep 0xffffffff815b01a0 [bufdaemon] 7 0 0 0 DL pgzero 0xffffffff815c5f34 [pagezero] 6 0 0 0 DL psleep 0xffffffff815c50f0 [vmdaemon] 5 0 0 0 DL psleep 0xffffffff815c50cc [pagedaemon] 4 0 0 0 DL ccb_scan 0xffffffff8137dc60 [xpt_thrd] 3 0 0 0 DL waiting_ 0xffffffff815b64c0 [sctp_iterator] 2 0 0 0 DL - 0xfffffe0015900048 [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xffffff8000952e18 [usbus5] 100057 D - 0xffffff8000952dc0 [usbus5] 100056 D - 0xffffff8000952d68 [usbus5] 100055 D - 0xffffff8000952d10 [usbus5] 100053 D - 0xffffff800094a460 [usbus4] 100052 D - 0xffffff800094a408 [usbus4] 100051 D - 0xffffff800094a3b0 [usbus4] 100050 D - 0xffffff800094a358 [usbus4] 100049 D - 0xffffff8000947460 [usbus3] 100048 D - 0xffffff8000947408 [usbus3] 100047 D - 0xffffff80009473b0 [usbus3] 100046 D - 0xffffff8000947358 [usbus3] 100045 D - 0xffffff8000944460 [usbus2] 100044 D - 0xffffff8000944408 [usbus2] 100043 D - 0xffffff80009443b0 [usbus2] 100042 D - 0xffffff8000944358 [usbus2] 100040 D - 0xffffff8000941460 [usbus1] 100039 D - 0xffffff8000941408 [usbus1] 100038 D - 0xffffff80009413b0 [usbus1] 100037 D - 0xffffff8000941358 [usbus1] 100035 D - 0xffffff800093e460 [usbus0] 100034 D - 0xffffff800093e408 [usbus0] 100033 D - 0xffffff800093e3b0 [usbus0] 100032 D - 0xffffff800093e358 [usbus0] 14 0 0 0 DL - 0xffffffff8139a5b0 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xffffffff813b5310 [g_down] 100014 D - 0xffffffff813b5308 [g_up] 100013 D - 0xffffffff813b52f8 [g_event] 12 0 0 0 RL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100023 I [swi6: task queue] 100022 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100018 I [swi2: cambio] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 Run CPU 0 [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe0015483970 [init] 10 0 0 0 DL audit_wo 0xffffffff815bb0f0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100065 D - 0xffffffff813bc302 [deadlkres] 100064 D - 0xfffffe001546ed00 [mca taskq] 100027 D - 0xfffffe001546d100 [kqueue taskq] 100026 D - 0xfffffe001546d200 [acpi_task_2] 100025 D - 0xfffffe001546d200 [acpi_task_1] 100024 D - 0xfffffe001546d200 [acpi_task_0] 100021 D - 0xfffffe001546d500 [thread taskq] 100020 D - 0xfffffe001546d600 [ffs_trim taskq] 100016 D - 0xfffffe001546db00 [firmware taskq] 100000 D sched 0xffffffff813b5640 [swapper] db:0:ps> allt Tracing command sh pid 49858 tid 101508 td 0xfffffe0108bb5490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823777c710 mi_switch() at mi_switch+0x238/frame 0xffffff823777c760 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823777c7a0 sleepq_catch_signals() at sleepq_catch_signals+0x157/frame 0xffffff823777c800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823777c820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823777c880 tty_wait() at tty_wait+0x4c/frame 0xffffff823777c8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823777c950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823777c980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823777c9f0 dofileread() at dofileread+0xa1/frame 0xffffff823777ca40 kern_readv() at kern_readv+0x6c/frame 0xffffff823777ca80 sys_read() at sys_read+0x64/frame 0xffffff823777cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823777cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823777cbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800dccd8a, rsp = 0x7fffffffdbc8, rbp = 0x80180a100 --- Tracing command syscall4 pid 46113 tid 106353 td 0xfffffe0131d28490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390eab20 mi_switch() at mi_switch+0x238/frame 0xffffff82390eab70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390eabb0 ast() at ast+0x3d0/frame 0xffffff82390eabf0 doreti_ast() at doreti_ast+0x1f Tracing command syscall4 pid 46113 tid 106340 td 0xfffffe00ae7e2920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390e5b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390e5b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390e5bb0 ast() at ast+0x3d0/frame 0xffffff82390e5bf0 doreti_ast() at doreti_ast+0x1f Tracing command syscall4 pid 46113 tid 106311 td 0xfffffe003b1e9000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390e0b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390e0b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390e0bb0 ast() at ast+0x3d0/frame 0xffffff82390e0bf0 doreti_ast() at doreti_ast+0x1f Tracing command syscall4 pid 46113 tid 106292 td 0xfffffe0135461920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390dbb20 mi_switch() at mi_switch+0x238/frame 0xffffff82390dbb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390dbbb0 ast() at ast+0x3d0/frame 0xffffff82390dbbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140fc08 Tracing command syscall4 pid 46113 tid 106288 td 0xfffffe01d3188000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390d6b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390d6b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390d6bb0 ast() at ast+0x3d0/frame 0xffffff82390d6bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f808 Tracing command syscall4 pid 46113 tid 106116 td 0xfffffe01aa681920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390d1b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390d1b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390d1bb0 ast() at ast+0x3d0/frame 0xffffff82390d1bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f408 Tracing command syscall4 pid 46113 tid 105967 td 0xfffffe0030757000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390ccb20 mi_switch() at mi_switch+0x238/frame 0xffffff82390ccb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390ccbb0 ast() at ast+0x3d0/frame 0xffffff82390ccbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f008 Tracing command syscall4 pid 46113 tid 105635 td 0xfffffe00a36ed920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390c7b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390c7b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390c7bb0 ast() at ast+0x3d0/frame 0xffffff82390c7bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140ec08 Tracing command syscall4 pid 46113 tid 105634 td 0xfffffe01fed37490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390c2b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390c2b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390c2bb0 ast() at ast+0x3d0/frame 0xffffff82390c2bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e808 Tracing command syscall4 pid 46113 tid 105499 td 0xfffffe006e5a2490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390bdb20 mi_switch() at mi_switch+0x238/frame 0xffffff82390bdb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390bdbb0 ast() at ast+0x3d0/frame 0xffffff82390bdbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e408 Tracing command syscall4 pid 46113 tid 105380 td 0xfffffe00a36f3490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390b8b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390b8b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390b8bb0 ast() at ast+0x3d0/frame 0xffffff82390b8bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e008 Tracing command syscall4 pid 46113 tid 105353 td 0xfffffe008d702000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390b3b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390b3b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390b3bb0 ast() at ast+0x3d0/frame 0xffffff82390b3bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140dc08 Tracing command syscall4 pid 46113 tid 105160 td 0xfffffe0149a31000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390aeb20 mi_switch() at mi_switch+0x238/frame 0xffffff82390aeb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390aebb0 ast() at ast+0x3d0/frame 0xffffff82390aebf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d808 Tracing command syscall4 pid 46113 tid 104806 td 0xfffffe010b777000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390a9b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390a9b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390a9bb0 ast() at ast+0x3d0/frame 0xffffff82390a9bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d408 Tracing command syscall4 pid 46113 tid 104805 td 0xfffffe010b776920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82390a4b20 mi_switch() at mi_switch+0x238/frame 0xffffff82390a4b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82390a4bb0 ast() at ast+0x3d0/frame 0xffffff82390a4bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d008 Tracing command syscall4 pid 46113 tid 104350 td 0xfffffe006989e920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823909fb20 mi_switch() at mi_switch+0x238/frame 0xffffff823909fb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823909fbb0 ast() at ast+0x3d0/frame 0xffffff823909fbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140cc08 Tracing command syscall4 pid 46113 tid 104309 td 0xfffffe0080eca000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239095b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239095b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239095bb0 ast() at ast+0x3d0/frame 0xffffff8239095bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c808 Tracing command syscall4 pid 46113 tid 103738 td 0xfffffe006ed78490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239090b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239090b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239090bb0 ast() at ast+0x3d0/frame 0xffffff8239090bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c408 Tracing command syscall4 pid 46113 tid 102213 td 0xfffffe018d4c8920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823908bb20 mi_switch() at mi_switch+0x238/frame 0xffffff823908bb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823908bbb0 ast() at ast+0x3d0/frame 0xffffff823908bbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c008 Tracing command syscall4 pid 46113 tid 101639 td 0xfffffe0149a4f920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239086b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239086b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239086bb0 ast() at ast+0x3d0/frame 0xffffff8239086bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140bc08 Tracing command syscall4 pid 46113 tid 101506 td 0xfffffe008f83d000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239081b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239081b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239081bb0 ast() at ast+0x3d0/frame 0xffffff8239081bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b808 Tracing command syscall4 pid 46113 tid 101376 td 0xfffffe0149a38000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823907cb20 mi_switch() at mi_switch+0x238/frame 0xffffff823907cb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823907cbb0 ast() at ast+0x3d0/frame 0xffffff823907cbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b408 Tracing command syscall4 pid 46113 tid 101374 td 0xfffffe018d4c3920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239077b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239077b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239077bb0 ast() at ast+0x3d0/frame 0xffffff8239077bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b008 Tracing command syscall4 pid 46113 tid 101349 td 0xfffffe0149a4b920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239072b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239072b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239072bb0 ast() at ast+0x3d0/frame 0xffffff8239072bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140ac08 Tracing command syscall4 pid 46113 tid 101335 td 0xfffffe020797e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823906db20 mi_switch() at mi_switch+0x238/frame 0xffffff823906db70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823906dbb0 ast() at ast+0x3d0/frame 0xffffff823906dbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a808 Tracing command syscall4 pid 46113 tid 101237 td 0xfffffe01bd7a3920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239068b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239068b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239068bb0 ast() at ast+0x3d0/frame 0xffffff8239068bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a408 Tracing command syscall4 pid 46113 tid 101229 td 0xfffffe0149a2d920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239054b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239054b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239054bb0 ast() at ast+0x3d0/frame 0xffffff8239054bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a008 Tracing command syscall4 pid 46113 tid 101222 td 0xfffffe0149a50490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239031b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239031b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239031bb0 ast() at ast+0x3d0/frame 0xffffff8239031bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409c08 Tracing command syscall4 pid 46113 tid 101203 td 0xfffffe0069896000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8239027b20 mi_switch() at mi_switch+0x238/frame 0xffffff8239027b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8239027bb0 ast() at ast+0x3d0/frame 0xffffff8239027bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409808 Tracing command syscall4 pid 46113 tid 101194 td 0xfffffe008f83a000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238ff0b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238ff0b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238ff0bb0 ast() at ast+0x3d0/frame 0xffffff8238ff0bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409408 Tracing command syscall4 pid 46113 tid 101176 td 0xfffffe01118e2920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238fc3b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238fc3b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238fc3bb0 ast() at ast+0x3d0/frame 0xffffff8238fc3bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409008 Tracing command syscall4 pid 46113 tid 101052 td 0xfffffe012de46920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238fbeb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238fbeb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238fbebb0 ast() at ast+0x3d0/frame 0xffffff8238fbebf0 doreti_ast() at doreti_ast+0x1f/frame 0x801408c08 Tracing command syscall4 pid 46113 tid 101017 td 0xfffffe003b118920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238eecb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238eecb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238eecbb0 ast() at ast+0x3d0/frame 0xffffff8238eecbf0 doreti_ast() at doreti_ast+0x1f/frame 0x18a99 Tracing command syscall4 pid 46113 tid 100965 td 0xfffffe01d318f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238e65b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238e65b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238e65bb0 ast() at ast+0x3d0/frame 0xffffff8238e65bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801408408 Tracing command syscall4 pid 46113 tid 100939 td 0xfffffe0080eb8920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823762d720 mi_switch() at mi_switch+0x238/frame 0xffffff823762d770 thread_suspend_switch() at thread_suspend_switch+0x103/frame 0xffffff823762d7c0 thread_single() at thread_single+0x222/frame 0xffffff823762d800 sigexit() at sigexit+0x97/frame 0xffffff823762dac0 postsig() at postsig+0x417/frame 0xffffff823762dbb0 ast() at ast+0x247/frame 0xffffff823762dbf0 doreti_ast() at doreti_ast+0x1f/frame 0xa Tracing command syscall4 pid 46113 tid 100936 td 0xfffffe006989a920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82375c46e0 mi_switch() at mi_switch+0x238/frame 0xffffff82375c4730 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82375c4770 exit1() at exit1+0x82/frame 0xffffff82375c4800 sigexit() at sigexit+0x8f/frame 0xffffff82375c4ac0 postsig() at postsig+0x417/frame 0xffffff82375c4bb0 ast() at ast+0x247/frame 0xffffff82375c4bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801407c00 Tracing command syscall4 pid 46113 tid 100822 td 0xfffffe0069896920 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800024cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800024cd20 trap() at trap+0x335/frame 0xffffff800024cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800024cf20 --- trap 0x13, rip = 0xffffffff808fa202, rsp = 0xffffff800024cfe0, rbp = 0xffffff8235b399c0 --- __mtx_assert() at __mtx_assert+0x12/frame 0xffffff8235b399c0 do_rw_rdlock() at do_rw_rdlock+0x291/frame 0xffffff8235b39a80 __umtx_op_rw_rdlock() at __umtx_op_rw_rdlock+0xb0/frame 0xffffff8235b39ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8235b39bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8235b39bf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083075c, rsp = 0x7fffff7fbde8, rbp = 0x801407800 --- Tracing command syscall4 pid 46113 tid 100770 td 0xfffffe0030750920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82353b96e0 mi_switch() at mi_switch+0x238/frame 0xffffff82353b9730 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82353b9770 exit1() at exit1+0x82/frame 0xffffff82353b9800 sigexit() at sigexit+0x8f/frame 0xffffff82353b9ac0 postsig() at postsig+0x417/frame 0xffffff82353b9bb0 ast() at ast+0x247/frame 0xffffff82353b9bf0 doreti_ast() at doreti_ast+0x1f/frame 0xa Tracing command syscall4 pid 46113 tid 100493 td 0xfffffe0049f0e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8233e33b20 mi_switch() at mi_switch+0x238/frame 0xffffff8233e33b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8233e33bb0 ast() at ast+0x3d0/frame 0xffffff8233e33bf0 doreti_ast() at doreti_ast+0x1f/frame 0x1888d Tracing command syscall4 pid 46113 tid 103980 td 0xfffffe00a36f0000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8236093b20 mi_switch() at mi_switch+0x238/frame 0xffffff8236093b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8236093bb0 ast() at ast+0x3d0/frame 0xffffff8236093bf0 doreti_ast() at doreti_ast+0x1f/frame 0x200000 Tracing command syscall4 pid 46046 tid 101602 td 0xfffffe018d4b7920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238eceb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238eceb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238ecebb0 ast() at ast+0x3d0/frame 0xffffff8238ecebf0 doreti_ast() at doreti_ast+0x1f Tracing command syscall4 pid 46046 tid 101601 td 0xfffffe016bc44490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238ec4b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238ec4b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238ec4bb0 ast() at ast+0x3d0/frame 0xffffff8238ec4bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801410408 Tracing command syscall4 pid 46046 tid 101594 td 0xfffffe01118ee490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238eabb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238eabb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238eabbb0 ast() at ast+0x3d0/frame 0xffffff8238eabbf0 doreti_ast() at doreti_ast+0x1f/frame 0x801410008 Tracing command syscall4 pid 46046 tid 101535 td 0xfffffe01118f7920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238e88b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238e88b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238e88bb0 ast() at ast+0x3d0/frame 0xffffff8238e88bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140fc08 Tracing command syscall4 pid 46046 tid 101502 td 0xfffffe01d3185490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238de8b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238de8b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238de8bb0 ast() at ast+0x3d0/frame 0xffffff8238de8bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f808 Tracing command syscall4 pid 46046 tid 101491 td 0xfffffe0149a2d000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238dbbb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238dbbb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238dbbbb0 ast() at ast+0x3d0/frame 0xffffff8238dbbbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f408 Tracing command syscall4 pid 46046 tid 101489 td 0xfffffe00384fb490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238b04b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238b04b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238b04bb0 ast() at ast+0x3d0/frame 0xffffff8238b04bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140f008 Tracing command syscall4 pid 46046 tid 101484 td 0xfffffe0160498000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238a50b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238a50b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238a50bb0 ast() at ast+0x3d0/frame 0xffffff8238a50bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140ec08 Tracing command syscall4 pid 46046 tid 101483 td 0xfffffe003075e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238a4bb20 mi_switch() at mi_switch+0x238/frame 0xffffff8238a4bb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238a4bbb0 ast() at ast+0x3d0/frame 0xffffff8238a4bbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e808 Tracing command syscall4 pid 46046 tid 101481 td 0xfffffe0043776490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238857b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238857b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238857bb0 ast() at ast+0x3d0/frame 0xffffff8238857bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e408 Tracing command syscall4 pid 46046 tid 101476 td 0xfffffe0149a39490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238852b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238852b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238852bb0 ast() at ast+0x3d0/frame 0xffffff8238852bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140e008 Tracing command syscall4 pid 46046 tid 101474 td 0xfffffe01118eb920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823884db20 mi_switch() at mi_switch+0x238/frame 0xffffff823884db70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823884dbb0 ast() at ast+0x3d0/frame 0xffffff823884dbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140dc08 Tracing command syscall4 pid 46046 tid 101473 td 0xfffffe00a36ee490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238834b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238834b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238834bb0 ast() at ast+0x3d0/frame 0xffffff8238834bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d808 Tracing command syscall4 pid 46046 tid 101469 td 0xfffffe0175eda000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238807b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238807b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238807bb0 ast() at ast+0x3d0/frame 0xffffff8238807bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d408 Tracing command syscall4 pid 46046 tid 101467 td 0xfffffe0175ed8490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82387dfb20 mi_switch() at mi_switch+0x238/frame 0xffffff82387dfb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82387dfbb0 ast() at ast+0x3d0/frame 0xffffff82387dfbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140d008 Tracing command syscall4 pid 46046 tid 101456 td 0xfffffe01a2f05920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82387b2b20 mi_switch() at mi_switch+0x238/frame 0xffffff82387b2b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82387b2bb0 ast() at ast+0x3d0/frame 0xffffff82387b2bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140cc08 Tracing command syscall4 pid 46046 tid 101453 td 0xfffffe010b775490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238762b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238762b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238762bb0 ast() at ast+0x3d0/frame 0xffffff8238762bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c808 Tracing command syscall4 pid 46046 tid 101445 td 0xfffffe008742e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823873fb20 mi_switch() at mi_switch+0x238/frame 0xffffff823873fb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823873fbb0 ast() at ast+0x3d0/frame 0xffffff823873fbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c408 Tracing command syscall4 pid 46046 tid 101441 td 0xfffffe009c9c2920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823873ab20 mi_switch() at mi_switch+0x238/frame 0xffffff823873ab70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823873abb0 ast() at ast+0x3d0/frame 0xffffff823873abf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140c008 Tracing command syscall4 pid 46046 tid 101440 td 0xfffffe019d1ba920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823871cb20 mi_switch() at mi_switch+0x238/frame 0xffffff823871cb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823871cbb0 ast() at ast+0x3d0/frame 0xffffff823871cbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140bc08 Tracing command syscall4 pid 46046 tid 101438 td 0xfffffe00bf679490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823849cb20 mi_switch() at mi_switch+0x238/frame 0xffffff823849cb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823849cbb0 ast() at ast+0x3d0/frame 0xffffff823849cbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b808 Tracing command syscall4 pid 46046 tid 101437 td 0xfffffe00437d3920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238479b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238479b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238479bb0 ast() at ast+0x3d0/frame 0xffffff8238479bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b408 Tracing command syscall4 pid 46046 tid 101434 td 0xfffffe0149a48000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8238406b20 mi_switch() at mi_switch+0x238/frame 0xffffff8238406b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8238406bb0 ast() at ast+0x3d0/frame 0xffffff8238406bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140b008 Tracing command syscall4 pid 46046 tid 101425 td 0xfffffe00437d7920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82383b6b20 mi_switch() at mi_switch+0x238/frame 0xffffff82383b6b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82383b6bb0 ast() at ast+0x3d0/frame 0xffffff82383b6bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140ac08 Tracing command syscall4 pid 46046 tid 101421 td 0xfffffe01118f5920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823823fb20 mi_switch() at mi_switch+0x238/frame 0xffffff823823fb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff823823fbb0 ast() at ast+0x3d0/frame 0xffffff823823fbf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a808 Tracing command syscall4 pid 46046 tid 101417 td 0xfffffe016bc44000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82381e0b20 mi_switch() at mi_switch+0x238/frame 0xffffff82381e0b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82381e0bb0 ast() at ast+0x3d0/frame 0xffffff82381e0bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a408 Tracing command syscall4 pid 46046 tid 101412 td 0xfffffe0046506000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82381d6b20 mi_switch() at mi_switch+0x238/frame 0xffffff82381d6b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82381d6bb0 ast() at ast+0x3d0/frame 0xffffff82381d6bf0 doreti_ast() at doreti_ast+0x1f/frame 0x80140a008 Tracing command syscall4 pid 46046 tid 101406 td 0xfffffe0201d34490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82381c7b20 mi_switch() at mi_switch+0x238/frame 0xffffff82381c7b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82381c7bb0 ast() at ast+0x3d0/frame 0xffffff82381c7bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409c08 Tracing command syscall4 pid 46046 tid 101402 td 0xfffffe0198834920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82380e1b20 mi_switch() at mi_switch+0x238/frame 0xffffff82380e1b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82380e1bb0 ast() at ast+0x3d0/frame 0xffffff82380e1bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409808 Tracing command syscall4 pid 46046 tid 101399 td 0xfffffe0049f09000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237ee3b20 mi_switch() at mi_switch+0x238/frame 0xffffff8237ee3b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237ee3bb0 ast() at ast+0x3d0/frame 0xffffff8237ee3bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409408 Tracing command syscall4 pid 46046 tid 101397 td 0xfffffe008f8aa920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237e3eb20 mi_switch() at mi_switch+0x238/frame 0xffffff8237e3eb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237e3ebb0 ast() at ast+0x3d0/frame 0xffffff8237e3ebf0 doreti_ast() at doreti_ast+0x1f/frame 0x801409008 Tracing command syscall4 pid 46046 tid 101365 td 0xfffffe0149a37920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237deeb20 mi_switch() at mi_switch+0x238/frame 0xffffff8237deeb70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237deebb0 ast() at ast+0x3d0/frame 0xffffff8237deebf0 doreti_ast() at doreti_ast+0x1f/frame 0x801408c08 Tracing command syscall4 pid 46046 tid 101364 td 0xfffffe006989f920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237dd5110 mi_switch() at mi_switch+0x238/frame 0xffffff8237dd5160 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237dd51a0 exit1() at exit1+0x82/frame 0xffffff8237dd5230 sigexit() at sigexit+0x8f/frame 0xffffff8237dd54f0 sendsig() at sendsig+0x403/frame 0xffffff8237dd5960 trapsignal() at trapsignal+0x1e0/frame 0xffffff8237dd59f0 trap() at trap+0x200/frame 0xffffff8237dd5bf0 calltrap() at calltrap+0x8/frame 0xffffff8237dd5bf0 --- trap 0xc, rip = 0x401402, rsp = 0x7ffffeff7f70, rbp = 0x4a --- Tracing command syscall4 pid 46046 tid 101361 td 0xfffffe010b779920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237795b20 mi_switch() at mi_switch+0x238/frame 0xffffff8237795b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237795bb0 ast() at ast+0x3d0/frame 0xffffff8237795bf0 doreti_ast() at doreti_ast+0x1f/frame 0x801408408 Tracing command syscall4 pid 46046 tid 101346 td 0xfffffe014f136920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8237736b20 mi_switch() at mi_switch+0x238/frame 0xffffff8237736b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8237736bb0 ast() at ast+0x3d0/frame 0xffffff8237736bf0 doreti_ast() at doreti_ast+0x1f/frame 0x18be2 Tracing command syscall4 pid 46046 tid 101341 td 0xfffffe006989e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8236b5b110 mi_switch() at mi_switch+0x238/frame 0xffffff8236b5b160 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8236b5b1a0 exit1() at exit1+0x82/frame 0xffffff8236b5b230 sigexit() at sigexit+0x8f/frame 0xffffff8236b5b4f0 sendsig() at sendsig+0x403/frame 0xffffff8236b5b960 trapsignal() at trapsignal+0x1e0/frame 0xffffff8236b5b9f0 trap() at trap+0x200/frame 0xffffff8236b5bbf0 calltrap() at calltrap+0x8/frame 0xffffff8236b5bbf0 --- trap 0xc, rip = 0x800b4aa88, rsp = 0x7fffff5faf20, rbp = 0x4a --- Tracing command syscall4 pid 46046 tid 101276 td 0xfffffe008f83d490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8235d73b20 mi_switch() at mi_switch+0x238/frame 0xffffff8235d73b70 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff8235d73bb0 ast() at ast+0x3d0/frame 0xffffff8235d73bf0 doreti_ast() at doreti_ast+0x1f/frame 0x18b9c Tracing command syscall4 pid 46046 tid 100903 td 0xfffffe010b774490 kdb_enter() at kdb_enter+0x3b/frame 0xffffff82341bc460 kdb_break() at kdb_break+0x27/frame 0xffffff82341bc470 uart_intr() at uart_intr+0x35a/frame 0xffffff82341bc4b0 intr_event_handle() at intr_event_handle+0xb7/frame 0xffffff82341bc500 intr_execute_handlers() at intr_execute_handlers+0x5f/frame 0xffffff82341bc530 lapic_handle_intr() at lapic_handle_intr+0x32/frame 0xffffff82341bc560 Xapic_isr1() at Xapic_isr1+0xa0/frame 0xffffff82341bc560 --- interrupt, rip = 0xffffffff808faa17, rsp = 0xffffff82341bc620, rbp = 0xffffff82341bc660 --- __mtx_unlock_flags() at __mtx_unlock_flags+0xd7/frame 0xffffff82341bc660 trap_pfault() at trap_pfault+0x1eb/frame 0xffffff82341bc6f0 trap() at trap+0x41a/frame 0xffffff82341bc8f0 calltrap() at calltrap+0x8/frame 0xffffff82341bc8f0 --- trap 0xc, rip = 0xffffffff80cb78f9, rsp = 0xffffff82341bc9b0, rbp = 0xffffff82341bca80 --- casuword32() at casuword32+0x29/frame 0xffffff82341bca80 __umtx_op_rw_rdlock() at __umtx_op_rw_rdlock+0xb0/frame 0xffffff82341bcad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82341bcbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82341bcbf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083075c, rsp = 0x7fffff9fcde8, rbp = 0x801407400 --- Tracing command syscall4 pid 46046 tid 100588 td 0xfffffe01d3180000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8233302110 mi_switch() at mi_switch+0x238/frame 0xffffff8233302160 thread_suspend_check() at thread_suspend_check+0x1ae/frame 0xffffff82333021a0 exit1() at exit1+0x82/frame 0xffffff8233302230 sigexit() at sigexit+0x8f/frame 0xffffff82333024f0 sendsig() at sendsig+0x403/frame 0xffffff8233302960 trapsignal() at trapsignal+0x1e0/frame 0xffffff82333029f0 trap() at trap+0x200/frame 0xffffff8233302bf0 calltrap() at calltrap+0x8/frame 0xffffff8233302bf0 --- trap 0xc, rip = 0x4015b3, rsp = 0x7fffffbfdf60, rbp = 0x801407008 --- Tracing command syscall4 pid 46046 tid 103855 td 0xfffffe018d4a9490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8235e22150 mi_switch() at mi_switch+0x238/frame 0xffffff8235e221a0 thread_suspend_switch() at thread_suspend_switch+0x103/frame 0xffffff8235e221f0 thread_single() at thread_single+0x222/frame 0xffffff8235e22230 sigexit() at sigexit+0x97/frame 0xffffff8235e224f0 sendsig() at sendsig+0x403/frame 0xffffff8235e22960 trapsignal() at trapsignal+0x1e0/frame 0xffffff8235e229f0 trap() at trap+0x200/frame 0xffffff8235e22bf0 calltrap() at calltrap+0x8/frame 0xffffff8235e22bf0 --- trap 0xc, rip = 0x80082c966, rsp = 0x7fffffffd3f0, rbp = 0 --- Tracing command softdepflush pid 17 tid 100074 td 0xfffffe001a830920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232844a10 mi_switch() at mi_switch+0x238/frame 0xffffff8232844a60 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232844aa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8232844ad0 _sleep() at _sleep+0x28d/frame 0xffffff8232844b60 softdep_flush() at softdep_flush+0x213/frame 0xffffff8232844ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232844bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232844bf0 --- trap 0, rip = 0, rsp = 0xffffff8232844cb0, rbp = 0 --- Tracing command syncer pid 16 tid 100073 td 0xfffffe001a831000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823283f9e0 mi_switch() at mi_switch+0x238/frame 0xffffff823283fa30 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823283fa70 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff823283faa0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x1a4/frame 0xffffff823283fb10 sched_sync() at sched_sync+0x4fc/frame 0xffffff823283fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823283fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823283fbf0 --- trap 0, rip = 0, rsp = 0xffffff823283fcb0, rbp = 0 --- Tracing command vnlru pid 9 tid 100072 td 0xfffffe001591c000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823283a9c0 mi_switch() at mi_switch+0x238/frame 0xffffff823283aa10 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823283aa50 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff823283aa80 _sleep() at _sleep+0x28d/frame 0xffffff823283ab10 vnlru_proc() at vnlru_proc+0x56a/frame 0xffffff823283aba0 fork_exit() at fork_exit+0x139/frame 0xffffff823283abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823283abf0 --- trap 0, rip = 0, rsp = 0xffffff823283acb0, rbp = 0 --- Tracing command bufdaemon pid 8 tid 100071 td 0xfffffe001591c490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232835a30 mi_switch() at mi_switch+0x238/frame 0xffffff8232835a80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232835ac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8232835af0 _sleep() at _sleep+0x28d/frame 0xffffff8232835b80 buf_daemon() at buf_daemon+0x1bc/frame 0xffffff8232835ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232835bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232835bf0 --- trap 0, rip = 0, rsp = 0xffffff8232835cb0, rbp = 0 --- Tracing command pagezero pid 7 tid 100070 td 0xfffffe001591c920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232830a30 mi_switch() at mi_switch+0x238/frame 0xffffff8232830a80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232830ac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8232830af0 _sleep() at _sleep+0x28d/frame 0xffffff8232830b80 vm_pagezero() at vm_pagezero+0x88/frame 0xffffff8232830ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232830bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232830bf0 --- trap 0, rip = 0, rsp = 0xffffff8232830cb0, rbp = 0 --- Tracing command vmdaemon pid 6 tid 100069 td 0xfffffe001591f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823282b9d0 mi_switch() at mi_switch+0x238/frame 0xffffff823282ba20 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823282ba60 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823282ba90 _sleep() at _sleep+0x3fb/frame 0xffffff823282bb20 vm_daemon() at vm_daemon+0x57/frame 0xffffff823282bba0 fork_exit() at fork_exit+0x139/frame 0xffffff823282bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823282bbf0 --- trap 0, rip = 0, rsp = 0xffffff823282bcb0, rbp = 0 --- Tracing command pagedaemon pid 5 tid 100068 td 0xfffffe001591f490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232826940 mi_switch() at mi_switch+0x238/frame 0xffffff8232826990 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82328269d0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8232826a00 _sleep() at _sleep+0x28d/frame 0xffffff8232826a90 vm_pageout() at vm_pageout+0xbb7/frame 0xffffff8232826ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232826bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232826bf0 --- trap 0, rip = 0, rsp = 0xffffff8232826cb0, rbp = 0 --- Tracing command xpt_thrd pid 4 tid 100067 td 0xfffffe001591f920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82326a1a20 mi_switch() at mi_switch+0x238/frame 0xffffff82326a1a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82326a1ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82326a1ae0 _sleep() at _sleep+0x3fb/frame 0xffffff82326a1b70 xpt_scanner_thread() at xpt_scanner_thread+0xf2/frame 0xffffff82326a1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82326a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82326a1bf0 --- trap 0, rip = 0, rsp = 0xffffff82326a1cb0, rbp = 0 --- Tracing command sctp_iterator pid 3 tid 100066 td 0xfffffe0015921000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823269ca30 mi_switch() at mi_switch+0x238/frame 0xffffff823269ca80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823269cac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823269caf0 _sleep() at _sleep+0x3fb/frame 0xffffff823269cb80 sctp_iterator_thread() at sctp_iterator_thread+0x4b/frame 0xffffff823269cba0 fork_exit() at fork_exit+0x139/frame 0xffffff823269cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823269cbf0 --- trap 0, rip = 0, rsp = 0xffffff823269ccb0, rbp = 0 --- Tracing command fdc0 pid 2 tid 100061 td 0xfffffe0015922920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823267e990 mi_switch() at mi_switch+0x238/frame 0xffffff823267e9e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823267ea20 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff823267ea50 _sleep() at _sleep+0x28d/frame 0xffffff823267eae0 fdc_thread() at fdc_thread+0x846/frame 0xffffff823267eba0 fork_exit() at fork_exit+0x139/frame 0xffffff823267ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823267ebf0 --- trap 0, rip = 0, rsp = 0xffffff823267ecb0, rbp = 0 --- Tracing command usb pid 15 tid 100058 td 0xfffffe00158ec000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232638a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232638aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232638ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232638b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232638b70 usb_process() at usb_process+0x172/frame 0xffffff8232638ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232638bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232638bf0 --- trap 0, rip = 0, rsp = 0xffffff8232638cb0, rbp = 0 --- Tracing command usb pid 15 tid 100057 td 0xfffffe00158ec490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232633a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232633aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232633ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232633b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232633b70 usb_process() at usb_process+0x172/frame 0xffffff8232633ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232633bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232633bf0 --- trap 0, rip = 0, rsp = 0xffffff8232633cb0, rbp = 0 --- Tracing command usb pid 15 tid 100056 td 0xfffffe00158ec920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823262ea50 mi_switch() at mi_switch+0x238/frame 0xffffff823262eaa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823262eae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823262eb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823262eb70 usb_process() at usb_process+0x172/frame 0xffffff823262eba0 fork_exit() at fork_exit+0x139/frame 0xffffff823262ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823262ebf0 --- trap 0, rip = 0, rsp = 0xffffff823262ecb0, rbp = 0 --- Tracing command usb pid 15 tid 100055 td 0xfffffe00158ed000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232629a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232629aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232629ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232629b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232629b70 usb_process() at usb_process+0x172/frame 0xffffff8232629ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232629bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232629bf0 --- trap 0, rip = 0, rsp = 0xffffff8232629cb0, rbp = 0 --- Tracing command usb pid 15 tid 100053 td 0xfffffe00158ed920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823249ca50 mi_switch() at mi_switch+0x238/frame 0xffffff823249caa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823249cae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823249cb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823249cb70 usb_process() at usb_process+0x172/frame 0xffffff823249cba0 fork_exit() at fork_exit+0x139/frame 0xffffff823249cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823249cbf0 --- trap 0, rip = 0, rsp = 0xffffff823249ccb0, rbp = 0 --- Tracing command usb pid 15 tid 100052 td 0xfffffe00158ef000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232497a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232497aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232497ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232497b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232497b70 usb_process() at usb_process+0x172/frame 0xffffff8232497ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232497bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232497bf0 --- trap 0, rip = 0, rsp = 0xffffff8232497cb0, rbp = 0 --- Tracing command usb pid 15 tid 100051 td 0xfffffe00158ef490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232492a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232492aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232492ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232492b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232492b70 usb_process() at usb_process+0x172/frame 0xffffff8232492ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232492bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232492bf0 --- trap 0, rip = 0, rsp = 0xffffff8232492cb0, rbp = 0 --- Tracing command usb pid 15 tid 100050 td 0xfffffe00158ef920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823248da50 mi_switch() at mi_switch+0x238/frame 0xffffff823248daa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823248dae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823248db10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823248db70 usb_process() at usb_process+0x172/frame 0xffffff823248dba0 fork_exit() at fork_exit+0x139/frame 0xffffff823248dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823248dbf0 --- trap 0, rip = 0, rsp = 0xffffff823248dcb0, rbp = 0 --- Tracing command usb pid 15 tid 100049 td 0xfffffe00158dd000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232444a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232444aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232444ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232444b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232444b70 usb_process() at usb_process+0x172/frame 0xffffff8232444ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232444bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232444bf0 --- trap 0, rip = 0, rsp = 0xffffff8232444cb0, rbp = 0 --- Tracing command usb pid 15 tid 100048 td 0xfffffe00158dd490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823243fa50 mi_switch() at mi_switch+0x238/frame 0xffffff823243faa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823243fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823243fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823243fb70 usb_process() at usb_process+0x172/frame 0xffffff823243fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823243fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823243fbf0 --- trap 0, rip = 0, rsp = 0xffffff823243fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100047 td 0xfffffe00158dd920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823243aa50 mi_switch() at mi_switch+0x238/frame 0xffffff823243aaa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823243aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823243ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823243ab70 usb_process() at usb_process+0x172/frame 0xffffff823243aba0 fork_exit() at fork_exit+0x139/frame 0xffffff823243abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823243abf0 --- trap 0, rip = 0, rsp = 0xffffff823243acb0, rbp = 0 --- Tracing command usb pid 15 tid 100046 td 0xfffffe00158de000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232435a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232435aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232435ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232435b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232435b70 usb_process() at usb_process+0x172/frame 0xffffff8232435ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232435bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232435bf0 --- trap 0, rip = 0, rsp = 0xffffff8232435cb0, rbp = 0 --- Tracing command usb pid 15 tid 100045 td 0xfffffe00158de490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323eca50 mi_switch() at mi_switch+0x238/frame 0xffffff82323ecaa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82323ecae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82323ecb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82323ecb70 usb_process() at usb_process+0x172/frame 0xffffff82323ecba0 fork_exit() at fork_exit+0x139/frame 0xffffff82323ecbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82323ecbf0 --- trap 0, rip = 0, rsp = 0xffffff82323eccb0, rbp = 0 --- Tracing command usb pid 15 tid 100044 td 0xfffffe00158de920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323e7a50 mi_switch() at mi_switch+0x238/frame 0xffffff82323e7aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82323e7ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82323e7b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82323e7b70 usb_process() at usb_process+0x172/frame 0xffffff82323e7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82323e7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82323e7bf0 --- trap 0, rip = 0, rsp = 0xffffff82323e7cb0, rbp = 0 --- Tracing command usb pid 15 tid 100043 td 0xfffffe00158e1000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323e2a50 mi_switch() at mi_switch+0x238/frame 0xffffff82323e2aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82323e2ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82323e2b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82323e2b70 usb_process() at usb_process+0x172/frame 0xffffff82323e2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82323e2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82323e2bf0 --- trap 0, rip = 0, rsp = 0xffffff82323e2cb0, rbp = 0 --- Tracing command usb pid 15 tid 100042 td 0xfffffe00158e1490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323dda50 mi_switch() at mi_switch+0x238/frame 0xffffff82323ddaa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff82323ddae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82323ddb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82323ddb70 usb_process() at usb_process+0x172/frame 0xffffff82323ddba0 fork_exit() at fork_exit+0x139/frame 0xffffff82323ddbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82323ddbf0 --- trap 0, rip = 0, rsp = 0xffffff82323ddcb0, rbp = 0 --- Tracing command usb pid 15 tid 100040 td 0xfffffe00158e2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823238fa50 mi_switch() at mi_switch+0x238/frame 0xffffff823238faa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823238fae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823238fb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823238fb70 usb_process() at usb_process+0x172/frame 0xffffff823238fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823238fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823238fbf0 --- trap 0, rip = 0, rsp = 0xffffff823238fcb0, rbp = 0 --- Tracing command usb pid 15 tid 100039 td 0xfffffe00158d6000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823238aa50 mi_switch() at mi_switch+0x238/frame 0xffffff823238aaa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823238aae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823238ab10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823238ab70 usb_process() at usb_process+0x172/frame 0xffffff823238aba0 fork_exit() at fork_exit+0x139/frame 0xffffff823238abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823238abf0 --- trap 0, rip = 0, rsp = 0xffffff823238acb0, rbp = 0 --- Tracing command usb pid 15 tid 100038 td 0xfffffe00158d6490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232385a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232385aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232385ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232385b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232385b70 usb_process() at usb_process+0x172/frame 0xffffff8232385ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232385bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232385bf0 --- trap 0, rip = 0, rsp = 0xffffff8232385cb0, rbp = 0 --- Tracing command usb pid 15 tid 100037 td 0xfffffe00158d6920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232380a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232380aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232380ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232380b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232380b70 usb_process() at usb_process+0x172/frame 0xffffff8232380ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232380bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232380bf0 --- trap 0, rip = 0, rsp = 0xffffff8232380cb0, rbp = 0 --- Tracing command usb pid 15 tid 100035 td 0xfffffe00158d7490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232332a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232332aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232332ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232332b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232332b70 usb_process() at usb_process+0x172/frame 0xffffff8232332ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232332bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232332bf0 --- trap 0, rip = 0, rsp = 0xffffff8232332cb0, rbp = 0 --- Tracing command usb pid 15 tid 100034 td 0xfffffe00158d7920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823232da50 mi_switch() at mi_switch+0x238/frame 0xffffff823232daa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff823232dae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823232db10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823232db70 usb_process() at usb_process+0x172/frame 0xffffff823232dba0 fork_exit() at fork_exit+0x139/frame 0xffffff823232dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823232dbf0 --- trap 0, rip = 0, rsp = 0xffffff823232dcb0, rbp = 0 --- Tracing command usb pid 15 tid 100033 td 0xfffffe00158d8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232328a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232328aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232328ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232328b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232328b70 usb_process() at usb_process+0x172/frame 0xffffff8232328ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232328bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232328bf0 --- trap 0, rip = 0, rsp = 0xffffff8232328cb0, rbp = 0 --- Tracing command usb pid 15 tid 100032 td 0xfffffe00158d8490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232323a50 mi_switch() at mi_switch+0x238/frame 0xffffff8232323aa0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232323ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232323b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8232323b70 usb_process() at usb_process+0x172/frame 0xffffff8232323ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232323bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232323bf0 --- trap 0, rip = 0, rsp = 0xffffff8232323cb0, rbp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xfffffe00154bb920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a3a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002a3a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002a3ab0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff80002a3ae0 msleep_spin_sbt() at msleep_spin_sbt+0x17f/frame 0xffffff80002a3b60 random_kthread() at random_kthread+0x190/frame 0xffffff80002a3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a3bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a3cb0, rbp = 0 --- Tracing command geom pid 13 tid 100015 td 0xfffffe00154a3000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002999e0 mi_switch() at mi_switch+0x238/frame 0xffffff8000299a30 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8000299a70 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000299aa0 _sleep() at _sleep+0x3fb/frame 0xffffff8000299b30 g_io_schedule_down() at g_io_schedule_down+0x296/frame 0xffffff8000299b90 g_down_procbody() at g_down_procbody+0x7c/frame 0xffffff8000299ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000299bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000299bf0 --- trap 0, rip = 0, rsp = 0xffffff8000299cb0, rbp = 0 --- Tracing command geom pid 13 tid 100014 td 0xfffffe00154a3490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000294a20 mi_switch() at mi_switch+0x238/frame 0xffffff8000294a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8000294ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000294ae0 _sleep() at _sleep+0x3fb/frame 0xffffff8000294b70 g_io_schedule_up() at g_io_schedule_up+0xec/frame 0xffffff8000294b90 g_up_procbody() at g_up_procbody+0x7c/frame 0xffffff8000294ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000294bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000294bf0 --- trap 0, rip = 0, rsp = 0xffffff8000294cb0, rbp = 0 --- Tracing command geom pid 13 tid 100013 td 0xfffffe00154a3920 sched_switch() at sched_switch+0x1b4/frame 0xffffff800028fa10 mi_switch() at mi_switch+0x238/frame 0xffffff800028fa60 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff800028faa0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800028fad0 _sleep() at _sleep+0x3fb/frame 0xffffff800028fb60 g_run_events() at g_run_events+0x45f/frame 0xffffff800028fba0 fork_exit() at fork_exit+0x139/frame 0xffffff800028fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800028fbf0 --- trap 0, rip = 0, rsp = 0xffffff800028fcb0, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xfffffe0015922000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xfffffe0015922490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100060 td 0xfffffe00158e2490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823266fb00 mi_switch() at mi_switch+0x238/frame 0xffffff823266fb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff823266fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823266fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823266fbf0 --- trap 0, rip = 0, rsp = 0xffffff823266fcb0, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffffe00158e2920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232660b00 mi_switch() at mi_switch+0x238/frame 0xffffff8232660b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8232660ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232660bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232660bf0 --- trap 0, rip = 0, rsp = 0xffffff8232660cb0, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffffe00158ed490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffffe00158e1920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323d8b00 mi_switch() at mi_switch+0x238/frame 0xffffff82323d8b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff82323d8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82323d8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82323d8bf0 --- trap 0, rip = 0, rsp = 0xffffff82323d8cb0, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xfffffe00158d7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823237bb00 mi_switch() at mi_switch+0x238/frame 0xffffff823237bb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff823237bba0 fork_exit() at fork_exit+0x139/frame 0xffffff823237bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823237bbf0 --- trap 0, rip = 0, rsp = 0xffffff823237bcb0, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xfffffe00158d8920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823231eb00 mi_switch() at mi_switch+0x238/frame 0xffffff823231eb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff823231eba0 fork_exit() at fork_exit+0x139/frame 0xffffff823231ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823231ebf0 --- trap 0, rip = 0, rsp = 0xffffff823231ecb0, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xfffffe00154bd490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003ecb00 mi_switch() at mi_switch+0x238/frame 0xffffff80003ecb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80003ecba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003ecbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003ecbf0 --- trap 0, rip = 0, rsp = 0xffffff80003eccb0, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xfffffe00154bd920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003e7b00 mi_switch() at mi_switch+0x238/frame 0xffffff80003e7b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80003e7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003e7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003e7bf0 --- trap 0, rip = 0, rsp = 0xffffff80003e7cb0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffffe001565c000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002dbb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002dbb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002dbba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002dbbf0 --- trap 0, rip = 0, rsp = 0xffffff80002dbcb0, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xfffffe001565d920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002c2b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002c2b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002c2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002c2bf0 --- trap 0, rip = 0, rsp = 0xffffff80002c2cb0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xfffffe00154ba000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002bdb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002bdb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002bdba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002bdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bdbf0 --- trap 0, rip = 0, rsp = 0xffffff80002bdcb0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffffe00154bb000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100018 td 0xfffffe00154bb490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a9b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002a9b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002a9ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a9bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a9cb0, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xfffffe00154a4000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xfffffe00154a4490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xfffffe00154a4920 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xfffffe0015486490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xfffffe0015486920 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffffff815dc760 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff815dc780 trap() at trap+0x335/frame 0xffffffff815dc980 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff815dc980 --- trap 0x13, rip = 0xffffffff8036772b, rsp = 0xffffffff815dca40, rbp = 0xffffff8000276a00 --- acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x1b/frame 0xffffff8000276a00 binuptime() at binuptime+0x3b/frame 0xffffff8000276a40 softclock_call_cc() at softclock_call_cc+0x17b/frame 0xffffff8000276b00 softclock() at softclock+0x4a/frame 0xffffff8000276b20 intr_event_execute_handlers() at intr_event_execute_handlers+0x6a/frame 0xffffff8000276b50 ithread_loop() at ithread_loop+0x9a/frame 0xffffff8000276ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000276bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000276bf0 --- trap 0, rip = 0, rsp = 0xffffff8000276cb0, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xfffffe0015491000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000271b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000271b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8000271ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000271bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000271bf0 --- trap 0, rip = 0, rsp = 0xffffff8000271cb0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe0015491490 sched_switch() at sched_switch+0x1b4/frame 0xffffff800026cac0 mi_switch() at mi_switch+0x238/frame 0xffffff800026cb10 sched_idletd() at sched_idletd+0x323/frame 0xffffff800026cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026cbf0 --- trap 0, rip = 0, rsp = 0xffffff800026ccb0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe0015491920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000267ac0 mi_switch() at mi_switch+0x238/frame 0xffffff8000267b10 sched_idletd() at sched_idletd+0x323/frame 0xffffff8000267ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000267bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000267bf0 --- trap 0, rip = 0, rsp = 0xffffff8000267cb0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe0015485000 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800023ed00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800023ed20 trap() at trap+0x335/frame 0xffffff800023ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800023ef20 --- trap 0x13, rip = 0xffffffff80c9e386, rsp = 0xffffff800023efe0, rbp = 0xffffff8000262a90 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xffffff8000262a90 acpi_cpu_idle() at acpi_cpu_idle+0x2ba/frame 0xffffff8000262ad0 cpu_idle_acpi() at cpu_idle_acpi+0x57/frame 0xffffff8000262af0 cpu_idle() at cpu_idle+0x6b/frame 0xffffff8000262b10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff8000262ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000262bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000262bf0 --- trap 0, rip = 0, rsp = 0xffffff8000262cb0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe0015485490 sched_switch() at sched_switch+0x1b4/frame 0xffffff800025daa0 mi_switch() at mi_switch+0x238/frame 0xffffff800025daf0 critical_exit() at critical_exit+0xb0/frame 0xffffff800025db10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff800025dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800025dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800025dbf0 --- trap 0, rip = 0, rsp = 0xffffff800025dcb0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe0015485920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000258670 mi_switch() at mi_switch+0x238/frame 0xffffff80002586c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8000258700 sleepq_catch_signals() at sleepq_catch_signals+0x157/frame 0xffffff8000258760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8000258780 _sleep() at _sleep+0x385/frame 0xffffff8000258810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff80002588b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8000258a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8000258ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8000258bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8000258bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41242a, rsp = 0x7fffffffd6c8, rbp = 0xc2c2 --- Tracing command audit pid 10 tid 100001 td 0xfffffe0015486000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000253a00 mi_switch() at mi_switch+0x238/frame 0xffffff8000253a50 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8000253a90 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000253ac0 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8000253b20 audit_worker() at audit_worker+0x77/frame 0xffffff8000253ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000253bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000253bf0 --- trap 0, rip = 0, rsp = 0xffffff8000253cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe0015921490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82326969c0 mi_switch() at mi_switch+0x238/frame 0xffffff8232696a10 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232696a50 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8232696a80 _sleep() at _sleep+0x28d/frame 0xffffff8232696b10 pause_sbt() at pause_sbt+0xc3/frame 0xffffff8232696b60 deadlkres() at deadlkres+0x2e5/frame 0xffffff8232696ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232696bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232696bf0 --- trap 0, rip = 0, rsp = 0xffffff8232696cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffffe0015921920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8232691a30 mi_switch() at mi_switch+0x238/frame 0xffffff8232691a80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff8232691ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232691af0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff8232691b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x9c/frame 0xffffff8232691ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8232691bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8232691bf0 --- trap 0, rip = 0, rsp = 0xffffff8232691cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe001565c490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002d6a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002d6a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002d6ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002d6ae0 _sleep() at _sleep+0x3fb/frame 0xffffff80002d6b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x166/frame 0xffffff80002d6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002d6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d6bf0 --- trap 0, rip = 0, rsp = 0xffffff80002d6cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe001565c920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002d1a30 mi_switch() at mi_switch+0x238/frame 0xffffff80002d1a80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002d1ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002d1af0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002d1b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x9c/frame 0xffffff80002d1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002d1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d1bf0 --- trap 0, rip = 0, rsp = 0xffffff80002d1cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe001565d000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002cca30 mi_switch() at mi_switch+0x238/frame 0xffffff80002cca80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002ccac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002ccaf0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002ccb70 taskqueue_thread_loop() at taskqueue_thread_loop+0x9c/frame 0xffffff80002ccba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002ccbf0 --- trap 0, rip = 0, rsp = 0xffffff80002cccb0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe001565d490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002c7a30 mi_switch() at mi_switch+0x238/frame 0xffffff80002c7a80 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002c7ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002c7af0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002c7b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x9c/frame 0xffffff80002c7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002c7bf0 --- trap 0, rip = 0, rsp = 0xffffff80002c7cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffffe00154ba490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b8a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002b8a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002b8ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b8ae0 _sleep() at _sleep+0x3fb/frame 0xffffff80002b8b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x166/frame 0xffffff80002b8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b8bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b8cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe00154ba920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b3a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002b3a70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff80002b3ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b3ae0 _sleep() at _sleep+0x3fb/frame 0xffffff80002b3b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x166/frame 0xffffff80002b3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b3bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b3cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe00154bd000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800029ea20 mi_switch() at mi_switch+0x238/frame 0xffffff800029ea70 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffff800029eab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800029eae0 _sleep() at _sleep+0x3fb/frame 0xffffff800029eb70 taskqueue_thread_loop() at taskqueue_thread_loop+0x166/frame 0xffffff800029eba0 fork_exit() at fork_exit+0x139/frame 0xffffff800029ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800029ebf0 --- trap 0, rip = 0, rsp = 0xffffff800029ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff813b5b00 sched_switch() at sched_switch+0x1b4/frame 0xffffffff81963af0 mi_switch() at mi_switch+0x238/frame 0xffffffff81963b40 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffffff81963b80 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffffff81963bb0 _sleep() at _sleep+0x28d/frame 0xffffffff81963c40 scheduler() at scheduler+0x2c4/frame 0xffffffff81963c90 mi_startup() at mi_startup+0x77/frame 0xffffffff81963cb0 btext() at btext+0x2c db:0:allt> call doadump Dumping 1433 out of 8039 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) bt #0 doadump (textdump=0x15484000) at ../../../kern/kern_shutdown.c:264 #1 0xffffffff8033806c in db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:578 #2 0xffffffff8033831d in db_command (last_cmdp=0xffffffff8137f3a0, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:449 #3 0xffffffff8033ceb3 in db_script_exec (scriptname=0xffffffff8137fc40 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff803383a1 in db_command (last_cmdp=0xffffffff8137f3a0, cmd_table=, dopager=0x1) at ../../../ddb/db_command.c:449 #5 0xffffffff803385f0 in db_command_loop () at ../../../ddb/db_command.c:502 #6 0xffffffff8033a769 in db_trap (type=, code=) at ../../../ddb/db_main.c:231 #7 0xffffffff8094a508 in kdb_trap (type=0x3, code=0x0, tf=0xffffff82341bc390) at ../../../kern/subr_kdb.c:654 #8 0xffffffff80cba46d in trap (frame=0xffffff82341bc390) at ../../../amd64/amd64/trap.c:579 #9 0xffffffff80ca30d3 in calltrap () at ../../../amd64/amd64/exception.S:228 #10 0xffffffff80949f8b in kdb_enter (why=0xffffffff80d6d655 "break", msg=0x80
) at cpufunc.h:63 #11 0xffffffff80949fc7 in kdb_break () at ../../../kern/subr_kdb.c:273 #12 0xffffffff807799ea in uart_intr (arg=) at ../../../dev/uart/uart_core.c:125 #13 0xffffffff808e1627 in intr_event_handle (ie=0xfffffe0015473b00, frame=0xffffff82341bc570) at ../../../kern/kern_intr.c:1436 #14 0xffffffff80d492cf in intr_execute_handlers (isrc=0xfffffe001549f168, frame=0xffffff82341bc570) at ../../../x86/x86/intr_machdep.c:269 #15 0xffffffff80d4d062 in lapic_handle_intr (vector=, frame=0xffffff82341bc570) at ../../../x86/x86/local_apic.c:774 #16 0xffffffff80ca3760 in Xapic_isr1 () at apic_vector.S:86 #17 0xffffffff808faa17 in __mtx_unlock_flags (c=0xfffffe00b30675d0, opts=0x0, file=0xffffffff80fd744e "../../../amd64/amd64/trap.c", line=0x2f9) at atomic.h:160 #18 0xffffffff80cb9f3b in trap_pfault (frame=0xffffff82341bc900, usermode=0x0) at ../../../amd64/amd64/trap.c:761 #19 0xffffffff80cba51a in trap (frame=0xffffff82341bc900) at ../../../amd64/amd64/trap.c:463 #20 0xffffffff80ca30d3 in calltrap () at ../../../amd64/amd64/exception.S:228 #21 0xffffffff80cb78f9 in casuword32 () at ../../../amd64/amd64/support.S:326 #22 0xffffffff8092b02e in do_rw_rdlock (td=, rwlock=0x800a3ef00, fflag=, timeout=0x0) at ../../../kern/kern_umtx.c:2665 #23 0xffffffff8092be90 in __umtx_op_rw_rdlock (td=0xfffffe010b774490, uap=0xffffff82341bcb70) at ../../../kern/kern_umtx.c:3293 #24 0xffffffff80cb9233 in amd64_syscall (td=0xfffffe010b774490, traced=0x0) at subr_syscall.c:134 #25 0xffffffff80ca33b7 in Xfast_syscall () at ../../../amd64/amd64/exception.S:387 #26 0x000000080083075c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 23 #23 0xffffffff8092be90 in __umtx_op_rw_rdlock (td=0xfffffe010b774490, uap=0xffffff82341bcb70) at ../../../kern/kern_umtx.c:3293 3293 error = do_rw_rdlock(td, uap->obj, uap->val, 0); (kgdb) p *td $1 = {td_lock = 0xffffffff813d9500, td_proc = 0xfffffe00b30674b8, td_plist = {tqe_next = 0xfffffe01d3180000, tqe_prev = 0xfffffe008f83d4a0}, td_runq = {tqe_next = 0x0, tqe_prev = 0xffffffff813d9a40}, td_slpq = {tqe_next = 0x0, tqe_prev = 0xfffffe006ede7780}, td_lockq = {tqe_next = 0x0, tqe_prev = 0xffffff8235b56bb0}, td_hash = {le_next = 0x0, le_prev = 0xffffff8000859138}, td_cpuset = 0xfffffe001546bdc8, td_sel = 0x0, td_sleepqueue = 0xfffffe006ede7780, td_turnstile = 0xfffffe007c68db40, td_rlqe = 0xfffffe008f8521e0, td_umtxq = 0xfffffe01aa68fd00, td_tid = 0x18a27, td_sigqueue = {sq_signals = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xfffffe010b774548}, sq_proc = 0xfffffe00b30674b8, sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x4018804, td_inhibitors = 0x0, td_pflags = 0x0, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0x0, td_wmesg = 0x0, td_lastcpu = 0x1, td_oncpu = 0x2, td_owepreempt = 0x0, td_tsqueue = 0xff, td_locks = 0x1, td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_stopsched = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0x0}, td_sleeplocks = 0x0, td_intr_nesting_level = 0x1, td_pinned = 0x1, td_ucred = 0xfffffe0022c9e600, td_estcpu = 0x0, td_slptick = 0x0, td_blktick = 0x0, td_swvoltick = 0x9612680, td_cow = 0x0, td_ru = { ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0xaec, ru_ixrss = 0x95a28, ru_idrss = 0x9efca8, ru_isrss = 0x95a280, ru_minflt = 0x1, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0xa, ru_nivcsw = 0x1b48}, td_rux = {rux_runtime = 0x107440257f6, rux_uticks = 0x0, rux_sticks = 0x12550, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x1b6304fd, rux_tu = 0x1b6304fd}, td_incruntime = 0x6853959d3, td_runtime = 0x10dc93bb1c9, td_pticks = 0x12b45, td_sticks = 0x5f5, td_iticks = 0x0, td_uticks = 0x0, td_intrval = 0x0, td_oldsigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_generation = 0x1b52, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x0}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = "syscall4", '\0' , td_fpop = 0x0, td_dbgflags = 0x0, td_dbgksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = { si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = {_timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}, ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0, td_dbg_forked = 0x0, td_vp_reserv = 0x0, td_no_sleeping = 0x0, td_dom_rr_idx = 0x0, td_sigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_rqindex = 0x9, td_base_pri = 0x98, td_priority = 0xcb, td_pri_class = 0x3, td_user_pri = 0xcb, td_base_user_pri = 0xcb, td_pcb = 0xffffff82341bccc0, td_state = TDS_RUNNING, td_retval = {0x0, 0x0}, td_slpcallout = {c_links = {le = { le_next = 0xffffff80009560c8, le_prev = 0xfffffe006989e820}, sle = {sle_next = 0xffffff80009560c8}, tqe = {tqe_next = 0xffffff80009560c8, tqe_prev = 0xfffffe006989e820}}, c_time = 0x264b011024930, c_precision = 0x41893, c_arg = 0xfffffe010b774490, c_func = 0xffffffff80956690 , c_lock = 0x0, c_flags = 0x112, c_cpu = 0x0}, td_frame = 0xffffff82341bcc00, td_kstack_obj = 0xfffffe019ad64c30, td_kstack = 0xffffff82341b9000, td_kstack_pages = 0x4, td_critnest = 0x1, td_md = {md_spinlock_count = 0x0, md_saved_flags = 0x46, md_spurflt_addr = 0x0}, td_sched = 0xfffffe010b7748f8, td_ar = 0x0, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, td_dtrace = 0xfffffe01b8251000, td_errno = 0x0, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0xffffff82341bc570, td_rfppwait_p = 0x0, td_ma = 0x0, td_ma_cnt = 0x0} (kgdb) p *uap $2 = {obj_l_ = 0xffffff82341bcb70 "", obj = 0x800a3ef00, obj_r_ = 0xffffff82341bcb78 "\f", op_l_ = 0xffffff82341bcb78 "\f", op = 0xc, op_r_ = "\000\000\000", val_l_ = 0xffffff82341bcb80 "", val = 0x0, val_r_ = 0xffffff82341bcb88 "", uaddr1_l_ = 0xffffff82341bcb88 "", uaddr1 = 0x0, uaddr1_r_ = 0xffffff82341bcb90 "", uaddr2_l_ = 0xffffff82341bcb90 "", uaddr2 = 0x0, uaddr2_r_ = 0xffffff82341bcb98 "BÑÎÛb@cC¸t\006³"} (kgdb) f 22 #22 0xffffffff8092b02e in do_rw_rdlock (td=, rwlock=0x800a3ef00, fflag=, timeout=0x0) at ../../../kern/kern_umtx.c:2665 2665 oldstate = casuword32(&rwlock->rw_state, state, state + 1); Current language: auto; currently c (kgdb) l 2660 while (!(state & wrflags)) { 2661 if (__predict_false(URWLOCK_READER_COUNT(state) == URWLOCK_MAX_READERS)) { 2662 umtx_key_release(&uq->uq_key); 2663 return (EAGAIN); 2664 } 2665 oldstate = casuword32(&rwlock->rw_state, state, state + 1); 2666 if (oldstate == state) { 2667 umtx_key_release(&uq->uq_key); 2668 return (0); 2669 } (kgdb) info loc timo = {clockid = 0x264b0, cur = {tv_sec = 0x1044f, tv_nsec = 0xffffffff80f8321e}, end = {tv_sec = 0xffffff82341bcb70, tv_nsec = 0xffffff82341bca90}} uq = (struct umtx_q *) 0xfffffe01aa68fd00 flags = wrflags = 0x80000000 state = 0x20000000 blocked_readers = error = 0x0 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/vm/vm_map.c =================================================================== --- /usr/src/sys/vm/vm_map.c (revision 251406) +++ /usr/src/sys/vm/vm_map.c (working copy) @@ -2225,6 +2225,7 @@ vm_map_unwire(vm_map_t map, vm_offset_t start, vm_ return (KERN_INVALID_ADDRESS); } } + vm_map_busy(map); last_timestamp = map->timestamp; entry = first_entry; while (entry != &map->header && entry->start < end) { @@ -2257,6 +2258,7 @@ vm_map_unwire(vm_map_t map, vm_offset_t start, vm_ /* * First_entry has been deleted. */ + vm_map_unbusy(map); vm_map_unlock(map); return (KERN_INVALID_ADDRESS); } @@ -2281,6 +2283,7 @@ vm_map_unwire(vm_map_t map, vm_offset_t start, vm_ * above.) */ entry->eflags |= MAP_ENTRY_IN_TRANSITION; + entry->wire_thread = curthread; /* * Check the map for holes in the specified region. * If VM_MAP_WIRE_HOLESOK was specified, skip this check. @@ -2315,6 +2318,9 @@ done: } entry = first_entry; while (entry != &map->header && entry->start < end) { + if ((entry->eflags & MAP_ENTRY_IN_TRANSITION) == 0 || + entry->wire_thread != curthread) + goto next; if (rv == KERN_SUCCESS && (!user_unwire || (entry->eflags & MAP_ENTRY_USER_WIRED))) { if (user_unwire) @@ -2330,16 +2336,19 @@ done: OBJ_FICTITIOUS) != 0); } } - KASSERT(entry->eflags & MAP_ENTRY_IN_TRANSITION, - ("vm_map_unwire: in-transition flag missing")); + KASSERT((entry->eflags & MAP_ENTRY_IN_TRANSITION) != 0, + ("vm_map_unwire: in-transition flag missing")); entry->eflags &= ~MAP_ENTRY_IN_TRANSITION; + entry->wire_thread = NULL; if (entry->eflags & MAP_ENTRY_NEEDS_WAKEUP) { entry->eflags &= ~MAP_ENTRY_NEEDS_WAKEUP; need_wakeup = TRUE; } vm_map_simplify_entry(map, entry); +next: entry = entry->next; } + vm_map_unbusy(map); vm_map_unlock(map); if (need_wakeup) vm_map_wakeup(map); @@ -2376,6 +2385,7 @@ vm_map_wire(vm_map_t map, vm_offset_t start, vm_of return (KERN_INVALID_ADDRESS); } } + vm_map_busy(map); last_timestamp = map->timestamp; entry = first_entry; while (entry != &map->header && entry->start < end) { @@ -2408,6 +2418,7 @@ vm_map_wire(vm_map_t map, vm_offset_t start, vm_of /* * first_entry has been deleted. */ + vm_map_unbusy(map); vm_map_unlock(map); return (KERN_INVALID_ADDRESS); } @@ -2432,6 +2443,7 @@ vm_map_wire(vm_map_t map, vm_offset_t start, vm_of * above.) */ entry->eflags |= MAP_ENTRY_IN_TRANSITION; + entry->wire_thread = curthread; if ((entry->protection & (VM_PROT_READ | VM_PROT_EXECUTE)) == 0 || (entry->protection & prot) != prot) { entry->eflags |= MAP_ENTRY_WIRE_SKIPPED; @@ -2451,14 +2463,12 @@ vm_map_wire(vm_map_t map, vm_offset_t start, vm_of OBJ_FICTITIOUS) != 0; /* * Release the map lock, relying on the in-transition - * mark. Mark the map busy for fork. + * mark. */ - vm_map_busy(map); vm_map_unlock(map); rv = vm_fault_wire(map, saved_start, saved_end, fictitious); vm_map_lock(map); - vm_map_unbusy(map); if (last_timestamp + 1 != map->timestamp) { /* * Look again for the entry because the map was @@ -2527,6 +2537,12 @@ done: while (entry != &map->header && entry->start < end) { if ((entry->eflags & MAP_ENTRY_WIRE_SKIPPED) != 0) goto next_entry_done; + if ((entry->eflags & MAP_ENTRY_IN_TRANSITION) == 0 || + entry->wire_thread != curthread) { + KASSERT((flags & VM_MAP_WIRE_HOLESOK) != 0, + ("vm_map_wire: alien entry")); + goto next; + } if (rv == KERN_SUCCESS) { if (user_wire) entry->eflags |= MAP_ENTRY_USER_WIRED; @@ -2551,16 +2567,22 @@ done: } } next_entry_done: - KASSERT(entry->eflags & MAP_ENTRY_IN_TRANSITION, - ("vm_map_wire: in-transition flag missing")); - entry->eflags &= ~(MAP_ENTRY_IN_TRANSITION|MAP_ENTRY_WIRE_SKIPPED); + KASSERT((entry->eflags & MAP_ENTRY_IN_TRANSITION) != 0, + ("vm_map_wire: in-transition flag missing %p", entry)); + KASSERT(entry->wire_thread == curthread, + ("vm_map_wire: alien wire %p", entry)); + entry->eflags &= ~(MAP_ENTRY_IN_TRANSITION | + MAP_ENTRY_WIRE_SKIPPED); + entry->wire_thread = NULL; if (entry->eflags & MAP_ENTRY_NEEDS_WAKEUP) { entry->eflags &= ~MAP_ENTRY_NEEDS_WAKEUP; need_wakeup = TRUE; } vm_map_simplify_entry(map, entry); + next: entry = entry->next; } + vm_map_unbusy(map); vm_map_unlock(map); if (need_wakeup) vm_map_wakeup(map); @@ -3193,6 +3215,7 @@ vmspace_fork(struct vmspace *vm1, vm_ooffset_t *fo *new_entry = *old_entry; new_entry->eflags &= ~(MAP_ENTRY_USER_WIRED | MAP_ENTRY_IN_TRANSITION); + new_entry->wire_thread = NULL; new_entry->wired_count = 0; if (new_entry->eflags & MAP_ENTRY_VN_WRITECNT) { vnode_pager_update_writecount(object, @@ -3227,6 +3250,7 @@ vmspace_fork(struct vmspace *vm1, vm_ooffset_t *fo */ new_entry->eflags &= ~(MAP_ENTRY_USER_WIRED | MAP_ENTRY_IN_TRANSITION | MAP_ENTRY_VN_WRITECNT); + new_entry->wire_thread = NULL; new_entry->wired_count = 0; new_entry->object.vm_object = NULL; new_entry->cred = NULL; Index: /usr/src/sys/vm/vm_map.h =================================================================== --- /usr/src/sys/vm/vm_map.h (revision 251406) +++ /usr/src/sys/vm/vm_map.h (working copy) @@ -116,6 +116,7 @@ struct vm_map_entry { int wired_count; /* can be paged if = 0 */ vm_pindex_t next_read; /* index of the next sequential read */ struct ucred *cred; /* tmp storage for creator ref */ + struct thread *wire_thread; }; #define MAP_ENTRY_NOSYNC 0x0001 Index: /usr/src/sys/vm/vm_object.c =================================================================== --- /usr/src/sys/vm/vm_object.c (revision 251406) +++ /usr/src/sys/vm/vm_object.c (working copy) @@ -1054,9 +1054,9 @@ vm_object_sync(vm_object_t object, vm_ooffset_t of */ flags = OBJPR_NOTMAPPED; else if (old_msync) - flags = 0; + flags = OBJPR_NOTWIRED; else - flags = OBJPR_CLEANONLY; + flags = OBJPR_CLEANONLY | OBJPR_NOTWIRED; vm_object_page_remove(object, OFF_TO_IDX(offset), OFF_TO_IDX(offset + size + PAGE_MASK), flags); } @@ -1912,7 +1912,8 @@ again: vm_page_lock(p); if ((wirings = p->wire_count) != 0 && (wirings = pmap_page_wired_mappings(p)) != p->wire_count) { - if ((options & OBJPR_NOTMAPPED) == 0) { + if ((options & (OBJPR_NOTWIRED | OBJPR_NOTMAPPED)) == + 0) { pmap_remove_all(p); /* Account for removal of wired mappings. */ if (wirings != 0) @@ -1922,8 +1923,7 @@ again: p->valid = 0; vm_page_undirty(p); } - vm_page_unlock(p); - continue; + goto next; } if (vm_page_sleep_if_busy(p, TRUE, "vmopar")) goto again; @@ -1932,12 +1932,12 @@ again: if ((options & OBJPR_CLEANONLY) != 0 && p->valid != 0) { if ((options & OBJPR_NOTMAPPED) == 0) pmap_remove_write(p); - if (p->dirty) { - vm_page_unlock(p); - continue; - } + if (p->dirty) + goto next; } if ((options & OBJPR_NOTMAPPED) == 0) { + if ((options & OBJPR_NOTWIRED) != 0 && wirings != 0) + goto next; pmap_remove_all(p); /* Account for removal of wired mappings. */ if (wirings != 0) { @@ -1949,6 +1949,7 @@ again: } } vm_page_free(p); +next: vm_page_unlock(p); } vm_object_pip_wakeup(object); Index: /usr/src/sys/vm/vm_fault.c =================================================================== --- /usr/src/sys/vm/vm_fault.c (revision 251406) +++ /usr/src/sys/vm/vm_fault.c (working copy) @@ -280,6 +280,19 @@ RetryFault:; (u_long)vaddr); } + if (fs.entry->eflags & MAP_ENTRY_IN_TRANSITION && + fs.entry->wire_thread != curthread) { + vm_map_unlock_read(fs.map); + vm_map_lock(fs.map); + if (vm_map_lookup_entry(fs.map, vaddr, &fs.entry) && + (fs.entry->eflags & MAP_ENTRY_IN_TRANSITION)) { + fs.entry->eflags |= MAP_ENTRY_NEEDS_WAKEUP; + vm_map_unlock_and_wait(fs.map, 0); + } else + vm_map_unlock(fs.map); + goto RetryFault; + } + /* * Make a reference to this object to prevent its disposal while we * are messing with it. Once we have the reference, the map is free Index: /usr/src/sys/vm/vm_object.h =================================================================== --- /usr/src/sys/vm/vm_object.h (revision 251406) +++ /usr/src/sys/vm/vm_object.h (working copy) @@ -205,6 +205,7 @@ struct vm_object { */ #define OBJPR_CLEANONLY 0x1 /* Don't remove dirty pages. */ #define OBJPR_NOTMAPPED 0x2 /* Don't unmap pages. */ +#define OBJPR_NOTWIRED 0x4 /* Don't remove wired pages. */ TAILQ_HEAD(object_q, vm_object); Index: /usr/src/sys/vm/vm_page.c =================================================================== --- /usr/src/sys/vm/vm_page.c (revision 251406) +++ /usr/src/sys/vm/vm_page.c (working copy) @@ -2611,8 +2611,6 @@ vm_page_set_invalid(vm_page_t m, int base, int siz vm_page_bits_t bits; VM_OBJECT_ASSERT_WLOCKED(m->object); - KASSERT((m->oflags & VPO_BUSY) == 0, - ("vm_page_set_invalid: page %p is busy", m)); bits = vm_page_bits(base, size); if (m->valid == VM_PAGE_BITS_ALL && bits != 0) pmap_remove_all(m); Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 251406) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -1567,7 +1567,8 @@ brelse(struct buf *bp) KASSERT(presid >= 0, ("brelse: extra page")); VM_OBJECT_WLOCK(obj); - vm_page_set_invalid(m, poffset, presid); + if (pmap_page_wired_mappings(m) == 0) + vm_page_set_invalid(m, poffset, presid); VM_OBJECT_WUNLOCK(obj); if (had_bogus) printf("avoided corruption bug in bogus_page/brelse code\n"); -- Test scenario: syscall4.sh 74 (mprotect)