GENERIC HEAD from 2016-10-17 08:38:24 UTC, r307513M, vmcore.609 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #1 r307513M: Mon Oct 17 20:12:10 CEST 2016 pho@t1.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.23-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66601549824 (63516 MB) : Trying to mount root from ufs:/dev/gpt/root [rw]... uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered Setting hostuuid: 3141f787-07e3-e111-b968-001e6756c168. Setting hostid: 0x564ac5b6. Starting ddb. Starting file system checks: /dev/gpt/root: Fugen2.2: at usbus2 uhub3 numa-domain 0 on uhub0 uhub3: on usbus2 ILE SYSTEM CLEANugen0.2: at usbus0 uhub4 numa-domain 0 on uhub2 uhub4: on usbus0 ; SKIPPING CHECKS /dev/gpt/root: clean, 38152548 free (983764 frags, 4646098 blocks, 1.0% fragmentation) /dev/ada0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0p2: clean, 31203091 free (14243 frags, 3898606 blocks, 0.0% fragmentation) /dev/gpt/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/gpt/tmp: clean, 25257851 free (4795 frags, 3156632 blocks, 0.0% fragmentation) Mounting local filesystems:. uhub4: 6 ports with 6 removable, self powered ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat /usr/local/lib/gcc48 /usr/local/lib/gcc49 /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/llvm37/lib 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: t1.osted.lan. uhub3: 8 ports with 8 removable, self powered Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . ugen2.3: at usbus2 uhub5 numa-domain 0 on uhub3 uhub5: on usbus2 uhub5: MTT enabled uhub5: 4 ports with 4 removable, self powered ugen2.4: at usbus2 ukbd0 numa-domain 0 on uhub3 ukbd0: on usbus2 kbd2 at ukbd0 Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo igb0: flags=8843 metric 0 mtu 8192 options=6403bb ether 00:1e:67:56:c1:68 inet 192.168.1.108 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:c168%igb0 prefixlen 64 scopeid 0x1 nd6 options=23 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8c02 metric 0 mtu 1500 options=6403bb ether 00:1e:67:56:c1:69 nd6 options=29 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: igb1. igb1: flags=8c02 metric 0 mtu 1500 options=6403bb ether 00:1e:67:56:c1:69 nd6 options=29 media: Ethernet autoselect status: no carrier Configuring vt: keymap. ums0 numa-domain 0 on uhub3 ums0: on usbus2 ums0: 3 buttons and [Z] coordinates ID=0 add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0xfffffe0f8f8c5db0 bufwait (bufwait) @ kern/vfs_bio.c:3505 2nd 0xfffff8011dffc800 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:281 stack backtrace: #0 0xffffffff80ac0d30 at witness_debugger+0x70 #1 0xffffffff80ac0c24 at witness_checkorder+0xe54 #2 0xffffffff80a68552 at _sx_xlock+0x72 #3 0xffffffff80d3a02d at ufsdirhash_add+0x4d #4 0xffffffff80d3ea12 at ufs_direnter+0x4f2 #5 0xffffffff80d46ffe at ufs_mkdir+0x83e #6 0xffffffff810514a6 at VOP_MKDIR_APV+0x236 #7 0xffffffff80b3612c at kern_mkdirat+0x1fc #8 0xffffffff80ee8dcf at amd64_syscall+0x32f #9 0xffffffff80ec7f9b at Xfast_syscall+0xfb Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting saslauthd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. sysctl: unknown oid 'debug.mnowait_failure.zalloc_whitelist' at line 13 sysctl: unknown oid 'debug.fail_point.uma_zalloc_arg' at line 14 Starting inetd. Mon Oct 17 20:24 FreeBSD/amd64 (t1.osted.lan) (ttyu1) login: root Password: Oct 17 20:25:54 t1 login: ROOT LOGIN (root) ON ttyu1 Last login: Wed Oct 5 08:39:51 on ttyu1 FreeBSD 12.0-CURRENT (PHO) #1 r307513M: Mon Oct 17 20:12:10 CEST 2016 Welcome to FreeBSD! Release Notes, Errata: https://www.FreeBSD.org/releases/ Security Advisories: https://www.FreeBSD.org/security/ FreeBSD Handbook: https://www.FreeBSD.org/handbook/ FreeBSD FAQ: https://www.FreeBSD.org/faq/ Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/ FreeBSD Forums: https://forums.FreeBSD.org/ Documents installed with the system are in the /usr/local/share/doc/freebsd/ directory, or can be installed later with: pkg install en-freebsd-doc For other languages, replace "en" with a language code like de or fr. Show the version of FreeBSD installed: freebsd-version ; uname -a Please include that output and any error messages when posting questions. Introduction to manual pages: man man FreeBSD directory layout: man hier Edit /etc/motd to change this login announcement. You have new mail. root@t1:~ # bash cd ~$ cd ~pho/stress2/misc $ cat datamove.sh #!/bin/sh # # Copyright (c) 2009 Peter Holm # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD: user/pho/stress2/misc/datamove.sh 280389 2015-03-23 18:54:37Z pho $ # # There is a well-known problem in FreeBSD, caused by allowing page faults # while doing filesystem data move to or from userspace during read(2) and # write(2). The issue is that if the userspace address being read or write # from/to is backed by the mapping of the same file we are doing i/o to, # we deadlock. # Test scenario by ups [ `id -u ` -ne 0 ] && echo "Must be root!" && exit 1 . ../default.cfg here=`pwd` cd /tmp sed '1,/^EOF/d' < $here/$0 > datamove.c mycc -o datamove -Wall datamove.c rm -f datamove.c n=5 old=`sysctl vm.old_msync | awk '{print $NF}'` sysctl vm.old_msync=1 for i in `jot $n`; do mkdir -p /tmp/datamove.dir.$i cd /tmp/datamove.dir.$i /tmp/datamove & done cd /tmp for i in `jot $n`; do wait done for i in `jot $n`; do rm -rf /tmp/datamove.dir.$i done sysctl vm.old_msync=$old rm -rf /tmp/datamove exit 0 EOF /*- * Copyright (c) 2006, Stephan Uphoff * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice unmodified, this list of conditions, and the following * disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include #include #include #include #include #include #include int prepareFile(char* filename,int* fdp); int mapBuffer(char** bufferp,int fd1,int fd2); int startIO(int fd,char *buffer); int pagesize; #define FILESIZE (32*1024) char wbuffer[FILESIZE]; /* Create a FILESIZE sized file - then remove file data from the cache*/ int prepareFile(char* filename,int* fdp) { int fd; int len; int status; void *addr; fd = open(filename,O_CREAT | O_TRUNC | O_RDWR,S_IRWXU); if (fd == -1) { perror("Creating file"); return fd; } len = write(fd,wbuffer,FILESIZE); if (len < 0) { perror("Write failed"); return 1; } status = fsync(fd); if (status != 0) { perror("fsync failed"); return 1; } addr = mmap(NULL,FILESIZE, PROT_READ | PROT_WRITE , MAP_SHARED, fd, 0); if (addr == MAP_FAILED) { perror("Mmap failed"); return 1; } status = msync(addr,FILESIZE,MS_INVALIDATE | MS_SYNC); if (status != 0) { perror("Msync failed"); return 1; } munmap(addr,FILESIZE); *fdp = fd; return 0; } /* mmap a 2 page buffer - first page is from fd1, second page from fd2 */ int mapBuffer(char** bufferp,int fd1,int fd2) { void* addr; char *buffer; addr = mmap(NULL,pagesize*2, PROT_READ | PROT_WRITE , MAP_SHARED, fd1, 0); if (addr == MAP_FAILED) { perror("Mmap failed"); return 1; } buffer = addr; addr = mmap(buffer + pagesize,pagesize, PROT_READ | PROT_WRITE , MAP_FIXED | MAP_SHARED, fd2, 0); if (addr == MAP_FAILED) { perror("Mmap2 failed"); return 1; } *bufferp = buffer; return 0; } int startIO(int fd,char *buffer) { ssize_t len; len = write(fd,buffer,2*pagesize); if (len == -1) { perror("write failed"); return 1; } return 0; } int main(int argc,char *argv[],char *envp[]) { int fdA,fdB,fdDelayA,fdDelayB; int status; char *bufferA,*bufferB; pid_t pid; pagesize = getpagesize(); if ((prepareFile("A",&fdA)) || (prepareFile("B",&fdB)) || (prepareFile("DelayA",&fdDelayA)) || (prepareFile("DelayB",&fdDelayB)) || (mapBuffer(&bufferA,fdDelayA,fdB)) || (mapBuffer(&bufferB,fdDelayB,fdA))) exit(1); pid = fork(); if (pid == 0) { status = startIO(fdA,bufferA); exit(status); } if (pid == -1) { exit(1); } status = startIO(fdB,bufferB); exit(status); } $ ./datamove.sh & [1] 37835 $ export BLASTHOST=t2 vm.old_msync: 0 -> 1 $ pkill -9 datamove $ pkill -9 datamove $ pkill -9 datamove $ pgrep datamove | xargs ps -lp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 37863 37835 0 20 0 6332 2076 - R u1 1:37.08 /tmp/datamove 0 37865 37835 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove 0 37867 37835 0 20 0 6332 2076 - R u1 0:19.94 /tmp/datamove 0 37869 37835 0 20 0 6332 2076 - R u1 0:17.74 /tmp/datamove 0 37871 37835 0 20 0 6332 2076 - R u1 1:37.02 /tmp/datamove 0 37873 37863 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove 0 37874 37865 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove 0 37875 37867 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove 0 37876 37869 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove 0 37877 37871 0 20 0 6332 2076 - R u1 0:00.00 /tmp/datamove $ procstat -k 37863 PID TID COMM TDNAME KSTACK 37863 100500 datamove - __lockmgr_args _lockmgr_args_rw getblk breadn_flags ffs_getpages VOP_GETPAGES_APV vnode_pager_getpages vm_pager_get_pages vm_fault_hold vm_fault_quick_hold_pages vn_io_fault1 vn_io_fault dofilewrite kern_writev sys_write amd64_syscall Xfast_syscall $ procstat -k 37863 PID TID COMM TDNAME KSTACK 37863 100500 datamove - __mtx_lock_sleep __mtx_lock_flags binsfree brelse ffs_getpages VOP_GETPAGES_APV vnode_pager_getpages vm_pager_get_pages vm_fault_hold vm_fault_quick_hold_pages vn_io_fault1 vn_io_fault dofilewrite kern_writev sys_write amd64_syscall Xfast_syscall $ procstat -k 37863 PID TID COMM TDNAME KSTACK 37863 100500 datamove - binsfree brelse ffs_getpages VOP_GETPAGES_APV vnode_pager_getpages vm_pager_get_pages vm_fault_hold vm_fault_quick_hold_pages vn_io_fault1 vn_io_fault dofilewrite kern_writev sys_write amd64_syscall Xfast_syscall $ procstat -k 37863 PID TID COMM TDNAME KSTACK 37863 100500 datamove - bremfreel binsfree brelse ffs_getpages VOP_GETPAGES_APV vnode_pager_getpages vm_pager_get_pages vm_fault_hold vm_fault_quick_hold_pages vn_io_fault1 vn_io_fault dofilewrite kern_writev sys_write amd64_syscall Xfast_syscall $ init 1 $ Stopping inetd. Waiting for PIDS: 25258. Shutting down local packages:. Stopping cron. Waiting for PIDS: 831. Stopping sshd. Waiting for PIDS: 821. Stopping saslauthd. Waiting for PIDS: 763. Stopping powerd. Waiting for PIDS: 739. Stopping ntpd. Waiting for PIDS: 736, 736. Stopping lockd. Waiting for PIDS: 699, 699. Stopping statd. Waiting for PIDS: 696, 696. Stopping nfsd. Waiting for PIDS: 692 695. Stopping mountd. Waiting for PIDS: 686, 686. Stopping nfsuserd. Waiting for PIDS: 675 676 677 678 680. Stopping watchdogd. Waiting for PIDS: 640. Stopping rpcbind. Waiting for PIDS: 562. Stopping rtsold. Waiting for PIDS: 522. Stopping devd. Waiting for PIDS: 457. Writing entropy file:. Writing early boot entropy file:. . Terminated Oct 17 20:30:30 t1 syslogd: exiting on signal 15 Oct 17 20:30:50 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 37863 1 0 103 0 6332 2076 - R u1- 2:34.81 /tmp/datamove 0 37865 1 0 20 0 6332 2076 - R u1- 2:09.14 /tmp/datamove 0 37867 1 0 20 0 6332 2076 - R u1- 2:31.42 /tmp/datamove 0 37869 1 0 103 0 6332 2076 - R u1- 2:34.52 /tmp/datamove 0 37871 1 0 20 0 6332 2076 - R u1- 2:30.73 /tmp/datamove 0 37873 37863 0 20 0 6332 2076 - R u1- 2:04.19 /tmp/datamove 0 37874 37865 0 20 0 6332 2076 - R u1- 2:08.15 /tmp/datamove 0 37875 37867 0 20 0 6332 2076 - R u1- 2:04.18 /tmp/datamove 0 37876 37869 0 20 0 6332 2076 - R u1- 2:09.10 /tmp/datamove 0 37877 37871 0 20 0 6332 2076 - R u1- 2:04.11 /tmp/datamove 0 44760 1 0 20 0 13172 3488 wait Ss u1 0:00.01 -sh (sh) 0 44761 44760 0 20 0 21188 2824 - R+ u1 0:00.00 ps -l # # KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_alt_break_internal+0x18e: movq $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xfffff8000b6e7500 kdb_alt_break_internal() at kdb_alt_break_internal+0x18e/frame 0xfffffe0f93f686f0 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0f93f68700 uart_intr_rxready() at uart_intr_rxready+0x98/frame 0xfffffe0f93f68730 uart_intr() at uart_intr+0x121/frame 0xfffffe0f93f68770 intr_event_handle() at intr_event_handle+0x9c/frame 0xfffffe0f93f687c0 intr_execute_handlers() at intr_execute_handlers+0x48/frame 0xfffffe0f93f687f0 lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfffffe0f93f68830 Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfffffe0f93f68830 --- interrupt, rip = 0xffffffff803b9a3b, rsp = 0xfffffe0f93f68900, rbp = 0xfffffe0f93f68940 --- acpi_cpu_idle() at acpi_cpu_idle+0x2bb/frame 0xfffffe0f93f68940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f68960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f68980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f68a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f68ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f68ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x853780 curthread = 0xfffff8000b6e8000: pid 11 tid 100003 "idle: cpu0" curpcb = 0xfffffe0f93f5eb80 fpcurthread = none idlethread = 0xfffff8000b6e8000: tid 100003 "idle: cpu0" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a010 commontssp = 0xffffffff81e5a010 rsp0 = 0xfffffe0f93f5eb80 gs32p = 0xffffffff81e60868 ldt = 0xffffffff81e608a8 tss = 0xffffffff81e60898 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe1083da2780 curthread = 0xfffff8000b6e7a00: pid 11 tid 100004 "idle: cpu1" curpcb = 0xfffffe0f93f63b80 fpcurthread = none idlethread = 0xfffff8000b6e7a00: tid 100004 "idle: cpu1" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a078 commontssp = 0xffffffff81e5a078 rsp0 = 0xfffffe0f93f63b80 gs32p = 0xffffffff81e608d0 ldt = 0xffffffff81e60910 tss = 0xffffffff81e60900 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe1083daa780 curthread = 0xfffff8000b6e7500: pid 11 tid 100005 "idle: cpu2" curpcb = 0xfffffe0f93f68b80 fpcurthread = none idlethread = 0xfffff8000b6e7500: tid 100005 "idle: cpu2" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a0e0 commontssp = 0xffffffff81e5a0e0 rsp0 = 0xfffffe0f93f68b80 gs32p = 0xffffffff81e60938 ldt = 0xffffffff81e60978 tss = 0xffffffff81e60968 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe1083db2780 curthread = 0xfffff8000b6e7000: pid 11 tid 100006 "idle: cpu3" curpcb = 0xfffffe0f93f6db80 fpcurthread = none idlethread = 0xfffff8000b6e7000: tid 100006 "idle: cpu3" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a148 commontssp = 0xffffffff81e5a148 rsp0 = 0xfffffe0f93f6db80 gs32p = 0xffffffff81e609a0 ldt = 0xffffffff81e609e0 tss = 0xffffffff81e609d0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe1083dba780 curthread = 0xfffff8000b6e6a00: pid 11 tid 100007 "idle: cpu4" curpcb = 0xfffffe0f93f72b80 fpcurthread = none idlethread = 0xfffff8000b6e6a00: tid 100007 "idle: cpu4" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a1b0 commontssp = 0xffffffff81e5a1b0 rsp0 = 0xfffffe0f93f72b80 gs32p = 0xffffffff81e60a08 ldt = 0xffffffff81e60a48 tss = 0xffffffff81e60a38 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe1083dc2780 curthread = 0xfffff8000b6e6500: pid 11 tid 100008 "idle: cpu5" curpcb = 0xfffffe0f93f77b80 fpcurthread = none idlethread = 0xfffff8000b6e6500: tid 100008 "idle: cpu5" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a218 commontssp = 0xffffffff81e5a218 rsp0 = 0xfffffe0f93f77b80 gs32p = 0xffffffff81e60a70 ldt = 0xffffffff81e60ab0 tss = 0xffffffff81e60aa0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe1083dca780 curthread = 0xfffff8000b6e6000: pid 11 tid 100009 "idle: cpu6" curpcb = 0xfffffe0f93f7cb80 fpcurthread = none idlethread = 0xfffff8000b6e6000: tid 100009 "idle: cpu6" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a280 commontssp = 0xffffffff81e5a280 rsp0 = 0xfffffe0f93f7cb80 gs32p = 0xffffffff81e60ad8 ldt = 0xffffffff81e60b18 tss = 0xffffffff81e60b08 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe1083dd2780 curthread = 0xfffff80131260a00: pid 37869 tid 100446 "datamove" curpcb = 0xfffffe104ea40b80 fpcurthread = none idlethread = 0xfffff8000b6faa00: tid 100010 "idle: cpu7" curpmap = 0xfffff80121fd4138 tssp = 0xffffffff81e5a2e8 commontssp = 0xffffffff81e5a2e8 rsp0 = 0xfffffe104ea40b80 gs32p = 0xffffffff81e60b40 ldt = 0xffffffff81e60b80 tss = 0xffffffff81e60b70 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe1083dda780 curthread = 0xfffff80131119a00: pid 37863 tid 100500 "datamove" curpcb = 0xfffffe104ed71b80 fpcurthread = none idlethread = 0xfffff8000b6fa500: tid 100011 "idle: cpu8" curpmap = 0xfffff801311d9138 tssp = 0xffffffff81e5a350 commontssp = 0xffffffff81e5a350 rsp0 = 0xfffffe104ed71b80 gs32p = 0xffffffff81e60ba8 ldt = 0xffffffff81e60be8 tss = 0xffffffff81e60bd8 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe1083de2780 curthread = 0xfffff8000b6fa000: pid 11 tid 100012 "idle: cpu9" curpcb = 0xfffffe0f93f8bb80 fpcurthread = none idlethread = 0xfffff8000b6fa000: tid 100012 "idle: cpu9" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a3b8 commontssp = 0xffffffff81e5a3b8 rsp0 = 0xfffffe0f93f8bb80 gs32p = 0xffffffff81e60c10 ldt = 0xffffffff81e60c50 tss = 0xffffffff81e60c40 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe1083dea780 curthread = 0xfffff8000b6f9a00: pid 11 tid 100013 "idle: cpu10" curpcb = 0xfffffe0f93f90b80 fpcurthread = none idlethread = 0xfffff8000b6f9a00: tid 100013 "idle: cpu10" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a420 commontssp = 0xffffffff81e5a420 rsp0 = 0xfffffe0f93f90b80 gs32p = 0xffffffff81e60c78 ldt = 0xffffffff81e60cb8 tss = 0xffffffff81e60ca8 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe1083df2780 curthread = 0xfffff801a4ed1000: pid 37871 tid 100647 "datamove" curpcb = 0xfffffe104f054b80 fpcurthread = none idlethread = 0xfffff8000b6f9500: tid 100014 "idle: cpu11" curpmap = 0xfffff80932f24138 tssp = 0xffffffff81e5a488 commontssp = 0xffffffff81e5a488 rsp0 = 0xfffffe104f054b80 gs32p = 0xffffffff81e60ce0 ldt = 0xffffffff81e60d20 tss = 0xffffffff81e60d10 spin locks held: cpuid = 12 dynamic pcpu = 0xfffffe1083dfa780 curthread = 0xfffff8000b6f9000: pid 11 tid 100015 "idle: cpu12" curpcb = 0xfffffe0f93f9ab80 fpcurthread = none idlethread = 0xfffff8000b6f9000: tid 100015 "idle: cpu12" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a4f0 commontssp = 0xffffffff81e5a4f0 rsp0 = 0xfffffe0f93f9ab80 gs32p = 0xffffffff81e60d48 ldt = 0xffffffff81e60d88 tss = 0xffffffff81e60d78 spin locks held: cpuid = 13 dynamic pcpu = 0xfffffe1083e02780 curthread = 0xfffff80121c9f000: pid 37875 tid 100233 "datamove" curpcb = 0xfffffe104e5ecb80 fpcurthread = none idlethread = 0xfffff8000b6f8a00: tid 100016 "idle: cpu13" curpmap = 0xfffff80121f17138 tssp = 0xffffffff81e5a558 commontssp = 0xffffffff81e5a558 rsp0 = 0xfffffe104e5ecb80 gs32p = 0xffffffff81e60db0 ldt = 0xffffffff81e60df0 tss = 0xffffffff81e60de0 spin locks held: cpuid = 14 dynamic pcpu = 0xfffffe1083e0a780 curthread = 0xfffff8000b6f8500: pid 11 tid 100017 "idle: cpu14" curpcb = 0xfffffe0f93fa4b80 fpcurthread = none idlethread = 0xfffff8000b6f8500: tid 100017 "idle: cpu14" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a5c0 commontssp = 0xffffffff81e5a5c0 rsp0 = 0xfffffe0f93fa4b80 gs32p = 0xffffffff81e60e18 ldt = 0xffffffff81e60e58 tss = 0xffffffff81e60e48 spin locks held: cpuid = 15 dynamic pcpu = 0xfffffe1083e12780 curthread = 0xfffff80153066500: pid 37877 tid 100241 "datamove" curpcb = 0xfffffe104e630b80 fpcurthread = none idlethread = 0xfffff8000b6f8000: tid 100018 "idle: cpu15" curpmap = 0xfffff801218bd138 tssp = 0xffffffff81e5a628 commontssp = 0xffffffff81e5a628 rsp0 = 0xfffffe104e630b80 gs32p = 0xffffffff81e60e80 ldt = 0xffffffff81e60ec0 tss = 0xffffffff81e60eb0 spin locks held: cpuid = 16 dynamic pcpu = 0xfffffe1083e1a780 curthread = 0xfffff80153b36a00: pid 37874 tid 100497 "datamove" curpcb = 0xfffffe104ed62b80 fpcurthread = none idlethread = 0xfffff8000b6fda00: tid 100019 "idle: cpu16" curpmap = 0xfffff80932f8b138 tssp = 0xffffffff81e5a690 commontssp = 0xffffffff81e5a690 rsp0 = 0xfffffe104ed62b80 gs32p = 0xffffffff81e60ee8 ldt = 0xffffffff81e60f28 tss = 0xffffffff81e60f18 spin locks held: cpuid = 17 dynamic pcpu = 0xfffffe1083e22780 curthread = 0xfffff8000b6fd500: pid 11 tid 100020 "idle: cpu17" curpcb = 0xfffffe0f93fb3b80 fpcurthread = none idlethread = 0xfffff8000b6fd500: tid 100020 "idle: cpu17" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a6f8 commontssp = 0xffffffff81e5a6f8 rsp0 = 0xfffffe0f93fb3b80 gs32p = 0xffffffff81e60f50 ldt = 0xffffffff81e60f90 tss = 0xffffffff81e60f80 spin locks held: cpuid = 18 dynamic pcpu = 0xfffffe1083e2a780 curthread = 0xfffff8000b6fd000: pid 11 tid 100021 "idle: cpu18" curpcb = 0xfffffe0f93fb8b80 fpcurthread = none idlethread = 0xfffff8000b6fd000: tid 100021 "idle: cpu18" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a760 commontssp = 0xffffffff81e5a760 rsp0 = 0xfffffe0f93fb8b80 gs32p = 0xffffffff81e60fb8 ldt = 0xffffffff81e60ff8 tss = 0xffffffff81e60fe8 spin locks held: cpuid = 19 dynamic pcpu = 0xfffffe1083e32780 curthread = 0xfffff80121cf3a00: pid 37867 tid 100521 "datamove" curpcb = 0xfffffe104eddab80 fpcurthread = none idlethread = 0xfffff8000b6fca00: tid 100022 "idle: cpu19" curpmap = 0xfffff8012169d138 tssp = 0xffffffff81e5a7c8 commontssp = 0xffffffff81e5a7c8 rsp0 = 0xfffffe104eddab80 gs32p = 0xffffffff81e61020 ldt = 0xffffffff81e61060 tss = 0xffffffff81e61050 spin locks held: cpuid = 20 dynamic pcpu = 0xfffffe1083e3a780 curthread = 0xfffff8000b6fc500: pid 11 tid 100023 "idle: cpu20" curpcb = 0xfffffe0f93fc2b80 fpcurthread = none idlethread = 0xfffff8000b6fc500: tid 100023 "idle: cpu20" curpmap = 0xffffffff81e232f8 tssp = 0xffffffff81e5a830 commontssp = 0xffffffff81e5a830 rsp0 = 0xfffffe0f93fc2b80 gs32p = 0xffffffff81e61088 ldt = 0xffffffff81e610c8 tss = 0xffffffff81e610b8 spin locks held: cpuid = 21 dynamic pcpu = 0xfffffe1083e42780 curthread = 0xfffff801a4cb6500: pid 37876 tid 100665 "datamove" curpcb = 0xfffffe104f0aeb80 fpcurthread = none idlethread = 0xfffff8000b6fc000: tid 100024 "idle: cpu21" curpmap = 0xfffff8000b6f1138 tssp = 0xffffffff81e5a898 commontssp = 0xffffffff81e5a898 rsp0 = 0xfffffe104f0aeb80 gs32p = 0xffffffff81e610f0 ldt = 0xffffffff81e61130 tss = 0xffffffff81e61120 spin locks held: cpuid = 22 dynamic pcpu = 0xfffffe1083e4a780 curthread = 0xfffff801a4c6ea00: pid 37873 tid 100519 "datamove" curpcb = 0xfffffe104edd0b80 fpcurthread = none idlethread = 0xfffff8000b6fba00: tid 100025 "idle: cpu22" curpmap = 0xfffff80121b47138 tssp = 0xffffffff81e5a900 commontssp = 0xffffffff81e5a900 rsp0 = 0xfffffe104edd0b80 gs32p = 0xffffffff81e61158 ldt = 0xffffffff81e61198 tss = 0xffffffff81e61188 spin locks held: cpuid = 23 dynamic pcpu = 0xfffffe101259d780 curthread = 0xfffff801a4908500: pid 37865 tid 100642 "datamove" curpcb = 0xfffffe104f03bb80 fpcurthread = none idlethread = 0xfffff8000b6fb500: tid 100026 "idle: cpu23" curpmap = 0xfffff801310e4138 tssp = 0xffffffff81e5a968 commontssp = 0xffffffff81e5a968 rsp0 = 0xfffffe104f03bb80 gs32p = 0xffffffff81e611c0 ldt = 0xffffffff81e61200 tss = 0xffffffff81e611f0 spin locks held: db:0:allpcpu> show alllocks Process 37877 (datamove) thread 0xfffff80153066500 (100241) exclusive rw vm object (vm object) r = 0 (0xfffff80cba4bc000) locked @ ufs/ffs/ffs_vnops.c:1915 shared lockmgr ufs (ufs) r = 0 (0xfffff80cbaf5e548) locked @ kern/vfs_subr.c:2592 Process 37876 (datamove) thread 0xfffff801a4cb6500 (100665) shared rw bufobj interlock (bufobj interlock) r = 0 (0xfffff80fd20f33d8) locked @ kern/vfs_bio.c:3491 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd20f32d8) locked @ kern/vfs_subr.c:2592 Process 37875 (datamove) thread 0xfffff80121c9f000 (100233) shared lockmgr ufs (ufs) r = 0 (0xfffff80fd2298c98) locked @ kern/vfs_subr.c:2592 Process 37874 (datamove) thread 0xfffff80153b36a00 (100497) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0f91b05b70) locked @ kern/vfs_bio.c:3505 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd21c7c98) locked @ kern/vfs_subr.c:2592 Process 37873 (datamove) thread 0xfffff801a4c6ea00 (100519) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0f91a5f830) locked @ kern/vfs_bio.c:3505 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd20cdc98) locked @ kern/vfs_subr.c:2592 Process 37871 (datamove) thread 0xfffff801a4ed1000 (100647) shared rw bufobj interlock (bufobj interlock) r = 0 (0xfffff80cbaf5e3d8) locked @ kern/vfs_bio.c:3491 shared lockmgr ufs (ufs) r = 0 (0xfffff80cbaf5e2d8) locked @ kern/vfs_subr.c:2592 Process 37869 (datamove) thread 0xfffff80131260a00 (100446) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0f900413b0) locked @ kern/vfs_bio.c:3505 shared rw bufobj interlock (bufobj interlock) r = 0 (0xfffff80fd22bb8b8) locked @ kern/vfs_bio.c:3491 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd22bb7b8) locked @ kern/vfs_subr.c:2592 Process 37867 (datamove) thread 0xfffff80121cf3a00 (100521) exclusive sleep mutex bufq clean lock (bufq clean lock) r = 0 (0xffffffff81da3080) locked @ kern/vfs_bio.c:1337 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0f914e6bb0) locked @ kern/vfs_bio.c:3505 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd2298a28) locked @ kern/vfs_subr.c:2592 Process 37865 (datamove) thread 0xfffff801a4908500 (100642) shared lockmgr ufs (ufs) r = 0 (0xfffff80fd21c7a28) locked @ kern/vfs_subr.c:2592 Process 37863 (datamove) thread 0xfffff80131119a00 (100500) exclusive sleep mutex bufq clean lock (bufq clean lock) r = 0 (0xffffffff81da2f80) locked @ kern/vfs_bio.c:1337 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0f91791130) locked @ kern/vfs_bio.c:3505 shared lockmgr ufs (ufs) r = 0 (0xfffff80fd20cda28) locked @ kern/vfs_subr.c:2592 db:0:alllocks> show lockedvnods Locked vnodes vnode 0xfffff80fd20cdc30: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80cba3f8a50 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 3290499, on dev gpt/tmp vnode 0xfffff80fd21c7c30: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80153b88000 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12359429, on dev gpt/tmp vnode 0xfffff80fd2298c30: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80cbaffd948 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12760707, on dev gpt/tmp vnode 0xfffff80fd20f3270: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff809326f4420 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 240771, on dev gpt/tmp vnode 0xfffff80fd21c79c0: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80bb15bd108 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12359430, on dev gpt/tmp vnode 0xfffff80fd20cd9c0: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80932cf9630 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 3290500, on dev gpt/tmp vnode 0xfffff80fd22989c0: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80cbaf0e000 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12760708, on dev gpt/tmp vnode 0xfffff80fd22bb750: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80fd2080e70 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 240772, on dev gpt/tmp vnode 0xfffff80cbaf5e4e0: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80cba4bc000 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12359432, on dev gpt/tmp vnode 0xfffff80cbaf5e270: tag ufs, type VREG usecount 5, writecount 2, refcount 7 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80932d92528 ref 3 pages 8 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xffffffff80a3705d at __lockmgr_args+0xc0d #1 0xffffffff80d3525f at _lockmgr_args+0x8f #2 0xffffffff80d32fa2 at ffs_lock+0x112 #3 0xffffffff81052d4a at VOP_LOCK1_APV+0x21a #4 0xffffffff80b3b973 at VOP_LOCK1+0x53 #5 0xffffffff80b39811 at _vn_lock+0xc1 #6 0xffffffff80b2891d at vget+0x8d #7 0xffffffff80d55627 at vm_fault_hold+0x1097 #8 0xffffffff80d57b63 at vm_fault_quick_hold_pages+0x203 #9 0xffffffff80b3a6a5 at vn_io_fault1+0x385 #10 0xffffffff80b376c6 at vn_io_fault+0x1e6 #11 0xffffffff80ac5d04 at dofilewrite+0xa4 #12 0xffffffff80ac59a8 at kern_writev+0x68 #13 0xffffffff80ac5934 at sys_write+0x84 #14 0xffffffff80ee8dcf at amd64_syscall+0x32f #15 0xffffffff80ec7f9b at Xfast_syscall+0xfb ino 12359433, on dev gpt/tmp db:0:lockedvnods> show mount 0xfffff80121662400 /dev/gpt/root on / (ufs) 0xfffff8012158f000 devfs on /dev (devfs) 0xfffff80121fcb800 /dev/gpt/tmp on /tmp (ufs) 0xfffff80121fcb400 procfs on /proc (procfs) 0xfffff80121fcb000 /dev/ada0p2 on /media (ufs) 0xfffff80121fcac00 fdescfs on /dev/fd (fdescfs) More info: show mount db:0:mount> show page vm_cnt.v_free_count: 15016221 vm_cnt.v_cache_count: 0 vm_cnt.v_inactive_count: 987623 vm_cnt.v_active_count: 1974 vm_cnt.v_wire_count: 309294 vm_cnt.v_free_reserved: 21293 vm_cnt.v_free_min: 102867 vm_cnt.v_free_target: 347589 vm_cnt.v_inactive_target: 521383 db:0:page> ps pid ppid pgrp uid state wmesg wchan cmd 44760 1 44760 0 Ss+ ttyin 0xfffff8000dd92ca8 sh 37877 37871 37835 0 R+ CPU 15 datamove 37876 37869 37835 0 R+ CPU 21 datamove 37875 37867 37835 0 R+ CPU 13 datamove 37874 37865 37835 0 R+ CPU 16 datamove 37873 37863 37835 0 R+ CPU 22 datamove 37871 1 37835 0 R+ CPU 11 datamove 37869 1 37835 0 R+ CPU 7 datamove 37867 1 37835 0 R+ CPU 19 datamove 37865 1 37835 0 R+ CPU 23 datamove 37863 1 37835 0 R+ CPU 8 datamove 22 0 0 0 DL vlruwt 0xfffff80121144a50 [vnlru] 21 0 0 0 DL syncer 0xffffffff81da4620 [syncer] 20 0 0 0 DL (threaded) [bufdaemon] 100144 D psleep 0xffffffff81da2d04 [bufdaemon] 100188 D sdflush 0xfffff80121ec72e8 [/ worker] 100194 D sdflush 0xfffff80121ec70e8 [/tmp worker] 100196 D sdflush 0xfffff80121ec6ae8 [/media worker] 19 0 0 0 DL - 0xffffffff81da3a04 [bufspacedaemon] 18 0 0 0 DL psleep 0xffffffff81dad18c [vmdaemon] 17 0 0 0 DL (threaded) [pagedaemon] 100141 D psleep 0xffffffff81e59405 [pagedaemon] 100150 D umarcl 0xffffffff81dacab8 [uma] 16 0 0 0 DL idle 0xfffff8011dfeca50 [enc_daemon0] 15 0 0 0 DL - 0xffffffff81da256c [soaiod4] 9 0 0 0 DL - 0xffffffff81da256c [soaiod3] 8 0 0 0 DL - 0xffffffff81da256c [soaiod2] 7 0 0 0 DL - 0xffffffff81da256c [soaiod1] 6 0 0 0 DL - 0xffffffff81adec58 [rand_harvestq] 5 0 0 0 DL waiting 0xffffffff81e4a4a0 [sctp_iterator] 14 0 0 0 DL (threaded) [usb] 100076 D - 0xfffffe000147ad10 [usbus0] 100077 D - 0xfffffe000147ad68 [usbus0] 100078 D - 0xfffffe000147adc0 [usbus0] 100079 D - 0xfffffe000147ae18 [usbus0] 100080 D - 0xfffffe000147ae70 [usbus0] 100117 D - 0xfffffe000187f460 [usbus1] 100118 D - 0xfffffe000187f4b8 [usbus1] 100119 D - 0xfffffe000187f510 [usbus1] 100120 D - 0xfffffe000187f568 [usbus1] 100121 D - 0xfffffe000187f5c0 [usbus1] 100123 D - 0xfffffe00016e9d10 [usbus2] 100124 D - 0xfffffe00016e9d68 [usbus2] 100125 D - 0xfffffe00016e9dc0 [usbus2] 100126 D - 0xfffffe00016e9e18 [usbus2] 100127 D - 0xfffffe00016e9e70 [usbus2] 4 0 0 0 DL (threaded) [cam] 100065 D - 0xffffffff819b1c00 [doneq0] 100066 D - 0xffffffff819b1d00 [doneq1] 100067 D - 0xffffffff819b1e00 [doneq2] 100068 D - 0xffffffff819b1f00 [doneq3] 100069 D - 0xffffffff819b2000 [doneq4] 100135 D - 0xffffffff819b1a48 [scanner] 3 0 0 0 DL crypto_ 0xffffffff81dab5d8 [crypto returns] 2 0 0 0 DL crypto_ 0xffffffff81dab490 [crypto] 13 0 0 0 DL (threaded) [geom] 100060 D - 0xffffffff81e22730 [g_event] 100061 D - 0xffffffff81e22738 [g_up] 100062 D - 0xffffffff81e22740 [g_down] 12 0 0 0 WL (threaded) [intr] 100027 I [swi4: clock (0)] 100028 I [swi4: clock (1)] 100029 I [swi4: clock (2)] 100030 I [swi4: clock (3)] 100031 I [swi4: clock (4)] 100032 I [swi4: clock (5)] 100033 I [swi4: clock (6)] 100034 I [swi4: clock (7)] 100035 I [swi4: clock (8)] 100036 I [swi4: clock (9)] 100037 I [swi4: clock (10)] 100038 I [swi4: clock (11)] 100039 I [swi4: clock (12)] 100040 I [swi4: clock (13)] 100041 I [swi4: clock (14)] 100042 I [swi4: clock (15)] 100043 I [swi4: clock (16)] 100044 I [swi4: clock (17)] 100045 I [swi4: clock (18)] 100046 I [swi4: clock (19)] 100047 I [swi4: clock (20)] 100048 I [swi4: clock (21)] 100049 I [swi4: clock (22)] 100050 I [swi4: clock (23)] 100051 I [swi1: netisr 0] 100052 I [swi3: vm] 100053 I [swi5: fast taskq] 100055 I [swi6: task queue] 100056 I [swi6: Giant taskq] 100073 I [irq264: isci0] 100074 I [irq265: isci0] 100075 I [irq22: ehci0] 100081 I [irq266: hdac0] 100082 I [irq267: igb0:que 0] 100084 I [irq268: igb0:que 1] 100086 I [irq269: igb0:que 2] 100088 I [irq270: igb0:que 3] 100090 I [irq271: igb0:que 4] 100092 I [irq272: igb0:que 5] 100094 I [irq273: igb0:que 6] 100096 I [irq274: igb0:que 7] 100098 I [irq275: igb0:link] 100099 I [irq276: igb1:que 0] 100101 I [irq277: igb1:que 1] 100103 I [irq278: igb1:que 2] 100105 I [irq279: igb1:que 3] 100107 I [irq280: igb1:que 4] 100109 I [irq281: igb1:que 5] 100111 I [irq282: igb1:que 6] 100113 I [irq283: igb1:que 7] 100115 I [irq284: igb1:link] 100116 I [irq285: xhci0] 100122 I [irq20: ehci1] 100128 I [irq286: ahci0] 100129 I [irq287: hdac1] 100130 I [swi0: uart] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 CanRun [idle: cpu7] 100011 CanRun [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 CanRun [idle: cpu11] 100015 Run CPU 12 [idle: cpu12] 100016 CanRun [idle: cpu13] 100017 Run CPU 14 [idle: cpu14] 100018 CanRun [idle: cpu15] 100019 CanRun [idle: cpu16] 100020 Run CPU 17 [idle: cpu17] 100021 Run CPU 18 [idle: cpu18] 100022 CanRun [idle: cpu19] 100023 Run CPU 20 [idle: cpu20] 100024 CanRun [idle: cpu21] 100025 CanRun [idle: cpu22] 100026 CanRun [idle: cpu23] 1 0 1 0 SLs wait 0xfffff8000b6e5528 [init] 10 0 0 0 DL audit_w 0xffffffff81e4fb50 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81e22768 [swapper] 100054 D - 0xfffff8000b745e00 [kqueue_ctx taskq] 100057 D - 0xfffff8000b745b00 [thread taskq] 100058 D - 0xfffff8000b745a00 [aiod_kick taskq] 100059 D - 0xfffff8000b745800 [firmware taskq] 100070 D - 0xfffff8000b744700 [acpi_task_0] 100071 D - 0xfffff8000b744700 [acpi_task_1] 100072 D - 0xfffff8000b744700 [acpi_task_2] 100083 D - 0xfffff8000dcb5900 [igb0 que (qid 0)] 100085 D - 0xfffff8000dcb5800 [igb0 que (qid 1)] 100087 D - 0xfffff8000dcb5700 [igb0 que (qid 2)] 100089 D - 0xfffff8000dcb5600 [igb0 que (qid 3)] 100091 D - 0xfffff8000dcb5500 [igb0 que (qid 4)] 100093 D - 0xfffff8000dcb5400 [igb0 que (qid 5)] 100095 D - 0xfffff8000ddbd500 [igb0 que (qid 6)] 100097 D - 0xfffff8000ddbd400 [igb0 que (qid 7)] 100100 D - 0xfffff8000ddba700 [igb1 que (qid 8)] 100102 D - 0xfffff8000ddba600 [igb1 que (qid 9)] 100104 D - 0xfffff8000ddba500 [igb1 que (qid 10)] 100106 D - 0xfffff8000ddba400 [igb1 que (qid 11)] 100108 D - 0xfffff8000ddba300 [igb1 que (qid 12)] 100110 D - 0xfffff8000ddba200 [igb1 que (qid 13)] 100112 D - 0xfffff8000ddba100 [igb1 que (qid 14)] 100114 D - 0xfffff8000ddba000 [igb1 que (qid 15)] 100131 D - 0xfffff8000e182a00 [mca taskq] 100134 D - 0xfffff8000b744800 [CAM taskq] 100147 D - 0xfffff8000b744a00 [if_config_tqg_0] 100148 D - 0xfffff8000b744b00 [if_io_tqg_0] 100149 D - 0xfffff8000b744c00 [if_io_tqg_1] 100151 D - 0xfffff80121441e00 [if_io_tqg_2] 100152 D - 0xfffff80121441d00 [if_io_tqg_3] 100153 D - 0xfffff80121441c00 [if_io_tqg_4] 100154 D - 0xfffff80121441b00 [if_io_tqg_5] 100155 D - 0xfffff80121441a00 [if_io_tqg_6] 100156 D - 0xfffff80121441900 [if_io_tqg_7] 100157 D - 0xfffff80121441800 [if_io_tqg_8] 100158 D - 0xfffff80121441700 [if_io_tqg_9] 100159 D - 0xfffff80121441600 [if_io_tqg_10] 100160 D - 0xfffff80121441500 [if_io_tqg_11] 100161 D - 0xfffff80121441400 [if_io_tqg_12] 100162 D - 0xfffff80121441300 [if_io_tqg_13] 100163 D - 0xfffff80121441200 [if_io_tqg_14] 100164 D - 0xfffff80121441100 [if_io_tqg_15] 100165 D - 0xfffff80121441000 [if_io_tqg_16] 100166 D - 0xfffff80121440e00 [if_io_tqg_17] 100167 D - 0xfffff80121440d00 [if_io_tqg_18] 100168 D - 0xfffff80121440c00 [if_io_tqg_19] 100169 D - 0xfffff80121440b00 [if_io_tqg_20] 100170 D - 0xfffff80121440a00 [if_io_tqg_21] 100171 D - 0xfffff80121440900 [if_io_tqg_22] 100172 D - 0xfffff80121440800 [if_io_tqg_23] 100195 D - 0xfffff80121e7b200 [/media trim] db:0:ps> allt Tracing command sh pid 44760 tid 100554 td 0xfffff801536f0500 sched_switch() at sched_switch+0x48a/frame 0xfffffe104ee815c0 mi_switch() at mi_switch+0x167/frame 0xfffffe104ee815f0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104ee81630 sleepq_catch_signals() at sleepq_catch_signals+0x32c/frame 0xfffffe104ee81690 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe104ee816c0 _cv_wait_sig() at _cv_wait_sig+0x1ec/frame 0xfffffe104ee81730 tty_wait() at tty_wait+0x42/frame 0xfffffe104ee81760 ttydisc_read() at ttydisc_read+0x3ae/frame 0xfffffe104ee817c0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe104ee817f0 devfs_read_f() at devfs_read_f+0xe7/frame 0xfffffe104ee81850 dofileread() at dofileread+0xcf/frame 0xfffffe104ee81890 kern_readv() at kern_readv+0x68/frame 0xfffffe104ee818e0 sys_read() at sys_read+0x84/frame 0xfffffe104ee81930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104ee81ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104ee81ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bd90da, rsp = 0x7fffffffea68, rbp = 0x7fffffffeac0 --- Tracing command datamove pid 37877 tid 100241 td 0xfffff80153066500 (CPU 15) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057c1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057c1d10 trap() at trap+0x3a/frame 0xfffffe10057c1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057c1f20 --- trap 0x13, rip = 0xffffffff80ac2674, rsp = 0xfffffe104e62fe30, rbp = 0xfffffe104e62fe80 --- witness_assert() at witness_assert+0xc4/frame 0xfffffe104e62fe80 vm_page_grab() at vm_page_grab+0x109/frame 0xfffffe104e62fee0 ffs_getpages() at ffs_getpages+0x595/frame 0xfffffe104e62ffd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104e630050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104e6300c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104e630130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104e630540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104e6305d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104e630790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104e630840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104e630890 kern_writev() at kern_writev+0x68/frame 0xfffffe104e6308e0 sys_write() at sys_write+0x84/frame 0xfffffe104e630930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104e630ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104e630ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37876 tid 100665 td 0xfffff801a4cb6500 (CPU 21) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057f1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057f1d10 trap() at trap+0x3a/frame 0xfffffe10057f1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057f1f20 --- trap 0x13, rip = 0xffffffff80a368a8, rsp = 0xfffffe104f0adc80, rbp = 0xfffffe104f0add30 --- __lockmgr_args() at __lockmgr_args+0x458/frame 0xfffffe104f0add30 _lockmgr_args_rw() at _lockmgr_args_rw+0x8f/frame 0xfffffe104f0adda0 getblk() at getblk+0x161/frame 0xfffffe104f0ade50 breadn_flags() at breadn_flags+0x68/frame 0xfffffe104f0adee0 ffs_getpages() at ffs_getpages+0x403/frame 0xfffffe104f0adfd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104f0ae050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104f0ae0c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104f0ae130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104f0ae540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104f0ae5d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104f0ae790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104f0ae840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104f0ae890 kern_writev() at kern_writev+0x68/frame 0xfffffe104f0ae8e0 sys_write() at sys_write+0x84/frame 0xfffffe104f0ae930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104f0aeab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104f0aeab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37875 tid 100233 td 0xfffff80121c9f000 (CPU 13) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057b1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057b1d10 trap() at trap+0x3a/frame 0xfffffe10057b1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057b1f20 --- trap 0x13, rip = 0xffffffff80a5c490, rsp = 0xfffffe104e5ebd10, rbp = 0xfffffe104e5ebda0 --- __rw_rlock() at __rw_rlock+0x80/frame 0xfffffe104e5ebda0 getblk() at getblk+0xd4/frame 0xfffffe104e5ebe50 breadn_flags() at breadn_flags+0x68/frame 0xfffffe104e5ebee0 ffs_getpages() at ffs_getpages+0x403/frame 0xfffffe104e5ebfd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104e5ec050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104e5ec0c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104e5ec130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104e5ec540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104e5ec5d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104e5ec790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104e5ec840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104e5ec890 kern_writev() at kern_writev+0x68/frame 0xfffffe104e5ec8e0 sys_write() at sys_write+0x84/frame 0xfffffe104e5ec930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104e5ecab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104e5ecab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37874 tid 100497 td 0xfffff80153b36a00 (CPU 16) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057c9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057c9d10 trap() at trap+0x3a/frame 0xfffffe10057c9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057c9f20 --- trap 0x13, rip = 0xffffffff80aa31d2, rsp = 0xfffffe104ed61b60, rbp = 0xfffffe104ed61b80 --- lock_delay() at lock_delay+0x52/frame 0xfffffe104ed61b80 __mtx_lock_sleep() at __mtx_lock_sleep+0x4ee/frame 0xfffffe104ed61d40 __mtx_lock_flags() at __mtx_lock_flags+0x22e/frame 0xfffffe104ed61de0 binsfree() at binsfree+0xdb/frame 0xfffffe104ed61e50 brelse() at brelse+0x647/frame 0xfffffe104ed61ee0 ffs_getpages() at ffs_getpages+0x4fe/frame 0xfffffe104ed61fd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104ed62050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104ed620c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104ed62130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104ed62540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104ed625d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104ed62790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104ed62840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104ed62890 kern_writev() at kern_writev+0x68/frame 0xfffffe104ed628e0 sys_write() at sys_write+0x84/frame 0xfffffe104ed62930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104ed62ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104ed62ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37873 tid 100519 td 0xfffff801a4c6ea00 (CPU 22) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057f9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057f9d10 trap() at trap+0x3a/frame 0xfffffe10057f9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057f9f20 --- trap 0x13, rip = 0xffffffff80a402f6, rsp = 0xfffffe104edcfd58, rbp = 0xfffffe104edcfd60 --- __curthread() at __curthread+0x16/frame 0xfffffe104edcfd60 __mtx_unlock_flags() at __mtx_unlock_flags+0x17e/frame 0xfffffe104edcfde0 binsfree() at binsfree+0x3f7/frame 0xfffffe104edcfe50 brelse() at brelse+0x647/frame 0xfffffe104edcfee0 ffs_getpages() at ffs_getpages+0x4fe/frame 0xfffffe104edcffd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104edd0050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104edd00c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104edd0130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104edd0540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104edd05d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104edd0790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104edd0840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104edd0890 kern_writev() at kern_writev+0x68/frame 0xfffffe104edd08e0 sys_write() at sys_write+0x84/frame 0xfffffe104edd0930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104edd0ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104edd0ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe708, rbp = 0x7fffffffe730 --- Tracing command datamove pid 37871 tid 100647 td 0xfffff801a4ed1000 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057a1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057a1d10 trap() at trap+0x3a/frame 0xfffffe10057a1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057a1f20 --- trap 0x13, rip = 0xffffffff80a3691d, rsp = 0xfffffe104f053c80, rbp = 0xfffffe104f053d30 --- __lockmgr_args() at __lockmgr_args+0x4cd/frame 0xfffffe104f053d30 _lockmgr_args_rw() at _lockmgr_args_rw+0x8f/frame 0xfffffe104f053da0 getblk() at getblk+0x161/frame 0xfffffe104f053e50 breadn_flags() at breadn_flags+0x68/frame 0xfffffe104f053ee0 ffs_getpages() at ffs_getpages+0x403/frame 0xfffffe104f053fd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104f054050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104f0540c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104f054130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104f054540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104f0545d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104f054790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104f054840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104f054890 kern_writev() at kern_writev+0x68/frame 0xfffffe104f0548e0 sys_write() at sys_write+0x84/frame 0xfffffe104f054930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104f054ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104f054ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37869 tid 100446 td 0xfffff80131260a00 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005781cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005781d10 trap() at trap+0x3a/frame 0xfffffe1005781f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005781f20 --- trap 0x13, rip = 0xffffffff81046656, rsp = 0xfffffe104ea3fc68, rbp = 0xfffffe104ea3fc70 --- stack_save() at stack_save+0x6/frame 0xfffffe104ea3fc70 __lockmgr_args() at __lockmgr_args+0xb5a/frame 0xfffffe104ea3fd30 _lockmgr_args_rw() at _lockmgr_args_rw+0x8f/frame 0xfffffe104ea3fda0 getblk() at getblk+0x161/frame 0xfffffe104ea3fe50 breadn_flags() at breadn_flags+0x68/frame 0xfffffe104ea3fee0 ffs_getpages() at ffs_getpages+0x403/frame 0xfffffe104ea3ffd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104ea40050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104ea400c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104ea40130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104ea40540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104ea405d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104ea40790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104ea40840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104ea40890 kern_writev() at kern_writev+0x68/frame 0xfffffe104ea408e0 sys_write() at sys_write+0x84/frame 0xfffffe104ea40930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104ea40ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104ea40ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37867 tid 100521 td 0xfffff80121cf3a00 (CPU 19) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057e1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057e1d10 trap() at trap+0x3a/frame 0xfffffe10057e1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057e1f20 --- trap 0x13, rip = 0xffffffff80a40bca, rsp = 0xfffffe104edd9d70, rbp = 0xfffffe104edd9de0 --- __mtx_unlock_flags() at __mtx_unlock_flags+0xca/frame 0xfffffe104edd9de0 binsfree() at binsfree+0x3f7/frame 0xfffffe104edd9e50 brelse() at brelse+0x647/frame 0xfffffe104edd9ee0 ffs_getpages() at ffs_getpages+0x4fe/frame 0xfffffe104edd9fd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104edda050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104edda0c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104edda130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104edda540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104edda5d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104edda790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104edda840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104edda890 kern_writev() at kern_writev+0x68/frame 0xfffffe104edda8e0 sys_write() at sys_write+0x84/frame 0xfffffe104edda930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104eddaab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104eddaab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37865 tid 100642 td 0xfffff801a4908500 (CPU 23) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe0f93f4ccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe0f93f4cd10 trap() at trap+0x3a/frame 0xfffffe0f93f4cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0f93f4cf20 --- trap 0x13, rip = 0xffffffff80a5ba88, rsp = 0xfffffe104f03aeb0, rbp = 0xfffffe104f03aee0 --- _rw_wlock_cookie() at _rw_wlock_cookie+0x58/frame 0xfffffe104f03aee0 ffs_getpages() at ffs_getpages+0x543/frame 0xfffffe104f03afd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104f03b050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104f03b0c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104f03b130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104f03b540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104f03b5d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104f03b790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104f03b840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104f03b890 kern_writev() at kern_writev+0x68/frame 0xfffffe104f03b8e0 sys_write() at sys_write+0x84/frame 0xfffffe104f03b930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104f03bab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104f03bab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe720 --- Tracing command datamove pid 37863 tid 100500 td 0xfffff80131119a00 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005789cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005789d10 trap() at trap+0x3a/frame 0xfffffe1005789f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005789f20 --- trap 0x13, rip = 0xffffffff80b050c4, rsp = 0xfffffe104ed70de0, rbp = 0xfffffe104ed70de0 --- bremfreel() at bremfreel+0x4/frame 0xfffffe104ed70de0 binsfree() at binsfree+0xe4/frame 0xfffffe104ed70e50 brelse() at brelse+0x647/frame 0xfffffe104ed70ee0 ffs_getpages() at ffs_getpages+0x4fe/frame 0xfffffe104ed70fd0 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0x236/frame 0xfffffe104ed71050 vnode_pager_getpages() at vnode_pager_getpages+0x6d/frame 0xfffffe104ed710c0 vm_pager_get_pages() at vm_pager_get_pages+0x76/frame 0xfffffe104ed71130 vm_fault_hold() at vm_fault_hold+0x1349/frame 0xfffffe104ed71540 vm_fault_quick_hold_pages() at vm_fault_quick_hold_pages+0x203/frame 0xfffffe104ed715d0 vn_io_fault1() at vn_io_fault1+0x385/frame 0xfffffe104ed71790 vn_io_fault() at vn_io_fault+0x1e6/frame 0xfffffe104ed71840 dofilewrite() at dofilewrite+0xa4/frame 0xfffffe104ed71890 kern_writev() at kern_writev+0x68/frame 0xfffffe104ed718e0 sys_write() at sys_write+0x84/frame 0xfffffe104ed71930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe104ed71ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe104ed71ab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80097c09a, rsp = 0x7fffffffe708, rbp = 0x7fffffffe730 --- Tracing command vnlru pid 22 tid 100146 td 0xfffff8012114a500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482a28a0 mi_switch() at mi_switch+0x167/frame 0xfffffe10482a28d0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482a2910 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10482a2940 _sleep() at _sleep+0x318/frame 0xfffffe10482a29e0 vnlru_proc() at vnlru_proc+0xa4/frame 0xfffffe10482a2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482a2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482a2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 21 tid 100145 td 0xfffff8012114aa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104829d8c0 mi_switch() at mi_switch+0x167/frame 0xfffffe104829d8f0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104829d930 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe104829d960 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x21f/frame 0xfffffe104829d9e0 sched_sync() at sched_sync+0x60c/frame 0xfffffe104829da70 fork_exit() at fork_exit+0x84/frame 0xfffffe104829dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104829dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100144 td 0xfffff8012109a000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482988c0 mi_switch() at mi_switch+0x167/frame 0xfffffe10482988f0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048298930 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe1048298960 _sleep() at _sleep+0x318/frame 0xfffffe1048298a00 buf_daemon() at buf_daemon+0x1ef/frame 0xfffffe1048298a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048298ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048298ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100188 td 0xfffff80121b89500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048444900 mi_switch() at mi_switch+0x167/frame 0xfffffe1048444930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048444970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10484449a0 _sleep() at _sleep+0x318/frame 0xfffffe1048444a40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe1048444a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048444ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048444ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100194 td 0xfffff80121b88a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048462900 mi_switch() at mi_switch+0x167/frame 0xfffffe1048462930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048462970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10484629a0 _sleep() at _sleep+0x318/frame 0xfffffe1048462a40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe1048462a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048462ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048462ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100196 td 0xfffff80121b88000 sched_switch() at sched_switch+0x48a/frame 0xfffffe104846c900 mi_switch() at mi_switch+0x167/frame 0xfffffe104846c930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104846c970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe104846c9a0 _sleep() at _sleep+0x318/frame 0xfffffe104846ca40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe104846ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe104846cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104846cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufspacedaemon pid 19 tid 100143 td 0xfffff8012109a500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482938f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1048293920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048293960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe1048293990 _sleep() at _sleep+0x318/frame 0xfffffe1048293a30 bufspace_daemon() at bufspace_daemon+0x1ee/frame 0xfffffe1048293a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048293ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048293ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 18 tid 100142 td 0xfffff8012109aa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104828e8c0 mi_switch() at mi_switch+0x167/frame 0xfffffe104828e8f0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104828e930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe104828e960 _sleep() at _sleep+0x338/frame 0xfffffe104828ea00 vm_daemon() at vm_daemon+0x96/frame 0xfffffe104828ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe104828eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104828eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 17 tid 100141 td 0xfffff8012109b000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048289820 mi_switch() at mi_switch+0x167/frame 0xfffffe1048289850 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048289890 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10482898c0 _sleep() at _sleep+0x318/frame 0xfffffe1048289960 vm_pageout() at vm_pageout+0x26a/frame 0xfffffe1048289a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048289ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048289ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 17 tid 100150 td 0xfffff80121430a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482b6910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482b6940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482b6980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482b69b0 _sleep() at _sleep+0x338/frame 0xfffffe10482b6a50 uma_reclaim_worker() at uma_reclaim_worker+0x71/frame 0xfffffe10482b6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482b6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482b6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 16 tid 100140 td 0xfffff8012109b500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482848c0 mi_switch() at mi_switch+0x167/frame 0xfffffe10482848f0 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048284930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1048284960 _sleep() at _sleep+0x338/frame 0xfffffe1048284a00 enc_daemon() at enc_daemon+0x26f/frame 0xfffffe1048284a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048284ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048284ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 15 tid 100139 td 0xfffff8012109ba00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10481fd8f0 mi_switch() at mi_switch+0x167/frame 0xfffffe10481fd920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10481fd960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10481fd990 _sleep() at _sleep+0x318/frame 0xfffffe10481fda30 soaio_kproc_loop() at soaio_kproc_loop+0x13e/frame 0xfffffe10481fda70 fork_exit() at fork_exit+0x84/frame 0xfffffe10481fdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10481fdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 9 tid 100138 td 0xfffff8012109c000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10481f88f0 mi_switch() at mi_switch+0x167/frame 0xfffffe10481f8920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10481f8960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10481f8990 _sleep() at _sleep+0x318/frame 0xfffffe10481f8a30 soaio_kproc_loop() at soaio_kproc_loop+0x13e/frame 0xfffffe10481f8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10481f8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10481f8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 8 tid 100137 td 0xfffff8012109c500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10481f38f0 mi_switch() at mi_switch+0x167/frame 0xfffffe10481f3920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10481f3960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10481f3990 _sleep() at _sleep+0x318/frame 0xfffffe10481f3a30 soaio_kproc_loop() at soaio_kproc_loop+0x13e/frame 0xfffffe10481f3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10481f3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10481f3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 7 tid 100136 td 0xfffff8012109ca00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10481ee8f0 mi_switch() at mi_switch+0x167/frame 0xfffffe10481ee920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10481ee960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe10481ee990 _sleep() at _sleep+0x318/frame 0xfffffe10481eea30 soaio_kproc_loop() at soaio_kproc_loop+0x13e/frame 0xfffffe10481eea70 fork_exit() at fork_exit+0x84/frame 0xfffffe10481eeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10481eeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 6 tid 100133 td 0xfffff8000e00da00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104809f8d0 mi_switch() at mi_switch+0x167/frame 0xfffffe104809f900 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104809f940 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe104809f970 _sleep() at _sleep+0x318/frame 0xfffffe104809fa10 random_kthread() at random_kthread+0x230/frame 0xfffffe104809fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe104809fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104809fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sctp_iterator pid 5 tid 100132 td 0xfffff8000e00e000 sched_switch() at sched_switch+0x48a/frame 0xfffffe104809a910 mi_switch() at mi_switch+0x167/frame 0xfffffe104809a940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104809a980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe104809a9b0 _sleep() at _sleep+0x338/frame 0xfffffe104809aa50 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe104809aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe104809aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104809aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100076 td 0xfffff8000dd19a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045cf4940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045cf4970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045cf49b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045cf49e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045cf4a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045cf4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045cf4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045cf4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100077 td 0xfffff8000dd19500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045cf9940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045cf9970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045cf99b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045cf99e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045cf9a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045cf9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045cf9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045cf9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100078 td 0xfffff8000dd19000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045cfe940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045cfe970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045cfe9b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045cfe9e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045cfea40 usb_process() at usb_process+0x13b/frame 0xfffffe1045cfea70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045cfeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045cfeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100079 td 0xfffff8000dd18a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d03940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d03970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d039b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d039e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045d03a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045d03a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d03ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d03ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100080 td 0xfffff8000dd18500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d08940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d08970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d089b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d089e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045d08a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045d08a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d08ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d08ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100117 td 0xfffff8000ddd3000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e4f940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e4f970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e4f9b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e4f9e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045e4fa40 usb_process() at usb_process+0x13b/frame 0xfffffe1045e4fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e4fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e4fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100118 td 0xfffff8000ddf4a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e54940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e54970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e549b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e549e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045e54a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045e54a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e54ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e54ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100119 td 0xfffff8000ddf4500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e59940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e59970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e599b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e599e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045e59a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045e59a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e59ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e59ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100120 td 0xfffff8000ddf4000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e5e940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e5e970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e5e9b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e5e9e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045e5ea40 usb_process() at usb_process+0x13b/frame 0xfffffe1045e5ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e5eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e5eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100121 td 0xfffff8000ddf0a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e63940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e63970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e639b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e639e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045e63a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045e63a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e63ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e63ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100123 td 0xfffff8000ddf0000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045fee940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045fee970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045fee9b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045fee9e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045feea40 usb_process() at usb_process+0x13b/frame 0xfffffe1045feea70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045feeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045feeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100124 td 0xfffff8000ddeba00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045ff3940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045ff3970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045ff39b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045ff39e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045ff3a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045ff3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045ff3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045ff3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100125 td 0xfffff8000ddeb500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045ff8940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045ff8970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045ff89b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045ff89e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045ff8a40 usb_process() at usb_process+0x13b/frame 0xfffffe1045ff8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045ff8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045ff8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100126 td 0xfffff8000ddeb000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045ffd940 mi_switch() at mi_switch+0x167/frame 0xfffffe1045ffd970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045ffd9b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045ffd9e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1045ffda40 usb_process() at usb_process+0x13b/frame 0xfffffe1045ffda70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045ffdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045ffdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100127 td 0xfffff8000e00fa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1046004940 mi_switch() at mi_switch+0x167/frame 0xfffffe1046004970 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10460049b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10460049e0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe1046004a40 usb_process() at usb_process+0x13b/frame 0xfffffe1046004a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1046004ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1046004ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100065 td 0xfffff8000b7fc500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f77e8e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f77e910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f77e950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f77e980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f77ea20 xpt_done_td() at xpt_done_td+0x13f/frame 0xfffffe0f8f77ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f77eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f77eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100066 td 0xfffff8000b7fc000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7838e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f783910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f783950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f783980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f783a20 xpt_done_td() at xpt_done_td+0x13f/frame 0xfffffe0f8f783a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f783ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f783ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100067 td 0xfffff8000b7fba00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7888e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f788910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f788950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f788980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f788a20 xpt_done_td() at xpt_done_td+0x13f/frame 0xfffffe0f8f788a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f788ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f788ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100068 td 0xfffff8000b7fb500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f78d8e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f78d910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f78d950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f78d980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f78da20 xpt_done_td() at xpt_done_td+0x13f/frame 0xfffffe0f8f78da70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f78dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f78dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100069 td 0xfffff8000b7fb000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7928e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f792910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f792950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f792980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f792a20 xpt_done_td() at xpt_done_td+0x13f/frame 0xfffffe0f8f792a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f792ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f792ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100135 td 0xfffff8000e00d000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10480a98e0 mi_switch() at mi_switch+0x167/frame 0xfffffe10480a9910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10480a9950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10480a9980 _sleep() at _sleep+0x338/frame 0xfffffe10480a9a20 xpt_scanner_thread() at xpt_scanner_thread+0xa4/frame 0xfffffe10480a9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10480a9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10480a9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns pid 3 tid 100064 td 0xfffff8000b7fca00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7798f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f779920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f779960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f779990 _sleep() at _sleep+0x338/frame 0xfffffe0f8f779a30 crypto_ret_proc() at crypto_ret_proc+0x1da/frame 0xfffffe0f8f779a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f779ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f779ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100063 td 0xfffff8000b747000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7748f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f774920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f774960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f774990 _sleep() at _sleep+0x338/frame 0xfffffe0f8f774a30 crypto_proc() at crypto_proc+0x394/frame 0xfffffe0f8f774a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f774ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f774ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100060 td 0xfffff8000b748000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7658f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f765920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f765960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f765990 _sleep() at _sleep+0x338/frame 0xfffffe0f8f765a30 g_run_events() at g_run_events+0x50/frame 0xfffffe0f8f765a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f765ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f765ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100061 td 0xfffff8000b747a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f76a8f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f76a920 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f76a960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f76a990 _sleep() at _sleep+0x338/frame 0xfffffe0f8f76aa30 g_io_schedule_up() at g_io_schedule_up+0xa3/frame 0xfffffe0f8f76aa60 g_up_procbody() at g_up_procbody+0x6d/frame 0xfffffe0f8f76aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f76aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f76aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100062 td 0xfffff8000b747500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f76f8e0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f76f910 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f76f950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f76f980 _sleep() at _sleep+0x338/frame 0xfffffe0f8f76fa20 g_io_schedule_down() at g_io_schedule_down+0x12c/frame 0xfffffe0f8f76fa60 g_down_procbody() at g_down_procbody+0x6d/frame 0xfffffe0f8f76fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f76fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f76fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xfffff8000b6fb000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fd69f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fd6a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe0f93fd6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fd6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fd6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffff8000b72ba00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100029 td 0xfffff8000b72b500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100030 td 0xfffff8000b72b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xfffff8000b726a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100032 td 0xfffff8000b726500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xfffff8000b726000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100034 td 0xfffff8000b725a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100035 td 0xfffff8000b725500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xfffff8000b725000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100037 td 0xfffff8000b732a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xfffff8000b732500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100039 td 0xfffff8000b732000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100040 td 0xfffff8000b731a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffff8000b731500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100042 td 0xfffff8000b731000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100043 td 0xfffff8000b730a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100044 td 0xfffff8000b730500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100045 td 0xfffff8000b730000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100046 td 0xfffff8000b739a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100047 td 0xfffff8000b739500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100048 td 0xfffff8000b739000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100049 td 0xfffff8000b738a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100050 td 0xfffff8000b738500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xfffff8000b738000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7389f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f738a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe0f8f738a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f738ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f738ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100052 td 0xfffff8000b737a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100053 td 0xfffff8000b737500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100055 td 0xfffff8000b749a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f74c9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f74ca20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe0f8f74ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f74cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f74cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100056 td 0xfffff8000b749500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7519f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f751a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe0f8f751a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f751ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f751ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100073 td 0xfffff8000dd1aa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7e89f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f7e8a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe0f8f7e8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f7e8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f7e8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100074 td 0xfffff8000dd1a500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100075 td 0xfffff8000dd1a000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045cef9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045cefa20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045cefa70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045cefab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045cefab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100081 td 0xfffff8000dd18000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d0d9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d0da20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d0da70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d0dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d0dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100082 td 0xfffff8000ddb4a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d5c9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d5ca20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d5ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d5cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d5cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100084 td 0xfffff8000ddb4000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d669f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d66a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d66a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d66ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d66ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100086 td 0xfffff8000ddb3500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d709f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d70a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d70a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d70ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d70ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100088 td 0xfffff8000ddb2a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d7a9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d7aa20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d7aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d7aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d7aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100090 td 0xfffff8000ddb2000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d849f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d84a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d84a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d84ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d84ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100092 td 0xfffff8000ddb9500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d8e9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d8ea20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d8ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d8eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d8eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100094 td 0xfffff8000ddb8a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d989f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d98a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045d98a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d98ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d98ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100096 td 0xfffff8000ddb8000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045da29f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045da2a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045da2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045da2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045da2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100098 td 0xfffff8000ddb7500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045dac9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045daca20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045daca70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045dacab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045dacab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100099 td 0xfffff8000ddb7000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100101 td 0xfffff8000ddd1500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100103 td 0xfffff8000ddd0a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100105 td 0xfffff8000ddd0000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100107 td 0xfffff8000ddcf500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100109 td 0xfffff8000ddd5a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100111 td 0xfffff8000ddd5000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100113 td 0xfffff8000ddd4500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100115 td 0xfffff8000ddd3a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100116 td 0xfffff8000ddd3500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100122 td 0xfffff8000ddf0500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045fe99f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1045fe9a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1045fe9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045fe9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045fe9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100128 td 0xfffff8000e00f500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10460099f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1046009a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1046009a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1046009ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1046009ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100129 td 0xfffff8000e00f000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10480839f0 mi_switch() at mi_switch+0x167/frame 0xfffffe1048083a20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe1048083a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048083ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048083ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100130 td 0xfffff8000e00ea00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104808b9f0 mi_switch() at mi_switch+0x167/frame 0xfffffe104808ba20 ithread_loop() at ithread_loop+0x1e1/frame 0xfffffe104808ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe104808bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104808bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff8000b6e8000 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81db4f10 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff81db4f30 trap() at trap+0x3a/frame 0xffffffff81db5140 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81db5140 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f5e8e0, rbp = 0xfffffe0f93f5e8f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f5e8f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f5e940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f5e960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f5e980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f5ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f5eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f5eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff8000b6e7a00 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005751cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005751d10 trap() at trap+0x3a/frame 0xfffffe1005751f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005751f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f638e0, rbp = 0xfffffe0f93f638f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f638f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f63940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f63960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f63980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f63a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f63ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f63ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff8000b6e7500 (CPU 2) kdb_alt_break_internal() at kdb_alt_break_internal+0x18e/frame 0xfffffe0f93f686f0 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0f93f68700 uart_intr_rxready() at uart_intr_rxready+0x98/frame 0xfffffe0f93f68730 uart_intr() at uart_intr+0x121/frame 0xfffffe0f93f68770 intr_event_handle() at intr_event_handle+0x9c/frame 0xfffffe0f93f687c0 intr_execute_handlers() at intr_execute_handlers+0x48/frame 0xfffffe0f93f687f0 lapic_handle_intr() at lapic_handle_intr+0x68/frame 0xfffffe0f93f68830 Xapic_isr1() at Xapic_isr1+0xb7/frame 0xfffffe0f93f68830 --- interrupt, rip = 0xffffffff803b9a3b, rsp = 0xfffffe0f93f68900, rbp = 0xfffffe0f93f68940 --- acpi_cpu_idle() at acpi_cpu_idle+0x2bb/frame 0xfffffe0f93f68940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f68960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f68980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f68a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f68ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f68ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff8000b6e7000 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005761cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005761d10 trap() at trap+0x3a/frame 0xfffffe1005761f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005761f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f6d8e0, rbp = 0xfffffe0f93f6d8f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f6d8f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f6d940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f6d960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f6d980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f6da70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f6dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f6dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff8000b6e6a00 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005769cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005769d10 trap() at trap+0x3a/frame 0xfffffe1005769f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005769f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f728e0, rbp = 0xfffffe0f93f728f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f728f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f72940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f72960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f72980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f72a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f72ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f72ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff8000b6e6500 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005771cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005771d10 trap() at trap+0x3a/frame 0xfffffe1005771f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005771f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f778e0, rbp = 0xfffffe0f93f778f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f778f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f77940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f77960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f77980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f77a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f77ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f77ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff8000b6e6000 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005779cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005779d10 trap() at trap+0x3a/frame 0xfffffe1005779f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005779f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f7c8e0, rbp = 0xfffffe0f93f7c8f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f7c8f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f7c940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f7c960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f7c980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f7ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f7cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f7cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff8000b6faa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f81950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f81980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93f81a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f81ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f81ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff8000b6fa500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f86950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f86980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93f86a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f86ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f86ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff8000b6fa000 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005791cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005791d10 trap() at trap+0x3a/frame 0xfffffe1005791f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005791f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f8b8e0, rbp = 0xfffffe0f93f8b8f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f8b8f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f8b940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f8b960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f8b980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f8ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f8bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f8bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff8000b6f9a00 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe1005799cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe1005799d10 trap() at trap+0x3a/frame 0xfffffe1005799f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe1005799f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f908e0, rbp = 0xfffffe0f93f908f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f908f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f90940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f90960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f90980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f90a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f90ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f90ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff8000b6f9500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f95930 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f95960 critical_exit() at critical_exit+0x8b/frame 0xfffffe0f93f95980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f95a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f95ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f95ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100015 td 0xfffff8000b6f9000 (CPU 12) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057a9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057a9d10 trap() at trap+0x3a/frame 0xfffffe10057a9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057a9f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93f9a8e0, rbp = 0xfffffe0f93f9a8f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93f9a8f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93f9a940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93f9a960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93f9a980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93f9aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f9aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f9aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100016 td 0xfffff8000b6f8a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f9f950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f9f980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93f9fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f9fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f9fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100017 td 0xfffff8000b6f8500 (CPU 14) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057b9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057b9d10 trap() at trap+0x3a/frame 0xfffffe10057b9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057b9f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93fa48e0, rbp = 0xfffffe0f93fa48f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93fa48f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93fa4940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93fa4960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93fa4980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93fa4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fa4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fa4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100018 td 0xfffff8000b6f8000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fa9950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fa9980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93fa9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fa9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fa9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100019 td 0xfffff8000b6fda00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fae950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fae980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93faea70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93faeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93faeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100020 td 0xfffff8000b6fd500 (CPU 17) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057d1cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057d1d10 trap() at trap+0x3a/frame 0xfffffe10057d1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057d1f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93fb38e0, rbp = 0xfffffe0f93fb38f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93fb38f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93fb3940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93fb3960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93fb3980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93fb3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fb3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fb3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100021 td 0xfffff8000b6fd000 (CPU 18) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057d9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057d9d10 trap() at trap+0x3a/frame 0xfffffe10057d9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057d9f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93fb88e0, rbp = 0xfffffe0f93fb88f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93fb88f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93fb8940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93fb8960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93fb8980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93fb8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fb8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fb8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100022 td 0xfffff8000b6fca00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fbd930 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fbd960 critical_exit() at critical_exit+0x8b/frame 0xfffffe0f93fbd980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93fbda70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fbdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fbdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100023 td 0xfffff8000b6fc500 (CPU 20) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe10057e9cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe10057e9d10 trap() at trap+0x3a/frame 0xfffffe10057e9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe10057e9f20 --- trap 0x13, rip = 0xffffffff81037b7f, rsp = 0xfffffe0f93fc28e0, rbp = 0xfffffe0f93fc28f0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x3f/frame 0xfffffe0f93fc28f0 acpi_cpu_idle() at acpi_cpu_idle+0x1ee/frame 0xfffffe0f93fc2940 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0f93fc2960 cpu_idle() at cpu_idle+0x95/frame 0xfffffe0f93fc2980 sched_idletd() at sched_idletd+0x4a5/frame 0xfffffe0f93fc2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fc2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fc2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100024 td 0xfffff8000b6fc000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fc7950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fc7980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93fc7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fc7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fc7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100025 td 0xfffff8000b6fba00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fcc950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fcc980 sched_idletd() at sched_idletd+0xf9/frame 0xfffffe0f93fcca70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fccab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fccab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100026 td 0xfffff8000b6fb500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93fd1950 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93fd1980 sched_idletd() at sched_idletd+0x391/frame 0xfffffe0f93fd1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93fd1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93fd1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff8000b6e8500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f594f0 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f59520 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f93f59560 sleepq_catch_signals() at sleepq_catch_signals+0x32c/frame 0xfffffe0f93f595c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0f93f595f0 _sleep() at _sleep+0x32b/frame 0xfffffe0f93f59690 kern_wait6() at kern_wait6+0x422/frame 0xfffffe0f93f59740 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe0f93f59930 amd64_syscall() at amd64_syscall+0x32f/frame 0xfffffe0f93f59ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0f93f59ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41d3aa, rsp = 0x7fffffffe728, rbp = 0x7fffffffe810 --- Tracing command audit pid 10 tid 100001 td 0xfffff8000b6e8a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f93f54910 mi_switch() at mi_switch+0x167/frame 0xfffffe0f93f54940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f93f54980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f93f549b0 _cv_wait() at _cv_wait+0x1e4/frame 0xfffffe0f93f54a10 audit_worker() at audit_worker+0xa3/frame 0xfffffe0f93f54a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f93f54ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f93f54ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81e22c90 sched_switch() at sched_switch+0x48a/frame 0xffffffff822029e0 mi_switch() at mi_switch+0x167/frame 0xffffffff82202a10 sleepq_switch() at sleepq_switch+0xe7/frame 0xffffffff82202a50 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xffffffff82202a80 _sleep() at _sleep+0x318/frame 0xffffffff82202b20 swapper() at swapper+0x248/frame 0xffffffff82202b70 btext() at btext+0x2c Tracing command kernel pid 0 tid 100054 td 0xfffff8000b737000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f747900 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f747930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f747970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f7479a0 _sleep() at _sleep+0x338/frame 0xfffffe0f8f747a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0f8f747a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f747ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f747ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100057 td 0xfffff8000b749000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f756900 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f756930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f756970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f7569a0 _sleep() at _sleep+0x338/frame 0xfffffe0f8f756a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0f8f756a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f756ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f756ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xfffff8000b748a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f75b900 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f75b930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f75b970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f75b9a0 _sleep() at _sleep+0x338/frame 0xfffffe0f8f75ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0f8f75ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f75bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f75bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100059 td 0xfffff8000b748500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f760900 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f760930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f760970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f7609a0 _sleep() at _sleep+0x338/frame 0xfffffe0f8f760a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0f8f760a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f760ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f760ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffff8000b7faa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f797910 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f797940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f797980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f7979b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe0f8f797a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0f8f797a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f797ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f797ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffff8000b7fa500 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f79c910 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f79c940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f79c980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f79c9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe0f8f79ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0f8f79ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f79cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f79cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffff8000b7fa000 sched_switch() at sched_switch+0x48a/frame 0xfffffe0f8f7a1910 mi_switch() at mi_switch+0x167/frame 0xfffffe0f8f7a1940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe0f8f7a1980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0f8f7a19b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe0f8f7a1a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0f8f7a1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0f8f7a1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f8f7a1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100083 td 0xfffff8000ddb4500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d61900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d61930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d61970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d619a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d61a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d61a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d61ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d61ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100085 td 0xfffff8000ddb3a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d6b900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d6b930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d6b970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d6b9a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d6ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d6ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d6bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d6bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100087 td 0xfffff8000ddb3000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d75900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d75930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d75970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d759a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d75a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d75a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d75ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d75ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xfffff8000ddb2500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d7f900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d7f930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d7f970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d7f9a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d7fa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d7fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d7fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d7fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100091 td 0xfffff8000ddb9a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d89900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d89930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d89970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d899a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d89a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d89a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d89ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d89ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100093 td 0xfffff8000ddb9000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d93900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d93930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d93970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d939a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d93a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d93a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d93ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d93ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xfffff8000ddb8500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045d9d900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045d9d930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045d9d970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045d9d9a0 _sleep() at _sleep+0x338/frame 0xfffffe1045d9da40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045d9da70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045d9dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045d9dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xfffff8000ddb7a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045da7900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045da7930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045da7970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045da79a0 _sleep() at _sleep+0x338/frame 0xfffffe1045da7a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045da7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045da7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045da7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100100 td 0xfffff8000ddd1a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045df6900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045df6930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045df6970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045df69a0 _sleep() at _sleep+0x338/frame 0xfffffe1045df6a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045df6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045df6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045df6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100102 td 0xfffff8000ddd1000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e04900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e04930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e04970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e049a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e04a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e04a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e04ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e04ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100104 td 0xfffff8000ddd0500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e0e900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e0e930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e0e970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e0e9a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e0ea40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e0ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e0eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e0eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xfffff8000ddcfa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e18900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e18930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e18970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e189a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e18a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e18a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e18ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e18ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100108 td 0xfffff8000ddcf000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e22900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e22930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e22970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e229a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e22a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e22a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e22ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e22ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xfffff8000ddd5500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e2c900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e2c930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e2c970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e2c9a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e2ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e2ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e2cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e2cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xfffff8000ddd4a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e36900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e36930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e36970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e369a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e36a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e36a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e36ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e36ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100114 td 0xfffff8000ddd4000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1045e40900 mi_switch() at mi_switch+0x167/frame 0xfffffe1045e40930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1045e40970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe1045e409a0 _sleep() at _sleep+0x338/frame 0xfffffe1045e40a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1045e40a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1045e40ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1045e40ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100131 td 0xfffff8000e00e500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048094910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048094940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048094980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10480949b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048094a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe1048094a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048094ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048094ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100134 td 0xfffff8000e00d500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10480a4900 mi_switch() at mi_switch+0x167/frame 0xfffffe10480a4930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10480a4970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10480a49a0 _sleep() at _sleep+0x338/frame 0xfffffe10480a4a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe10480a4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10480a4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10480a4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100147 td 0xfffff8012114a000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482a7910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482a7940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482a7980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482a79b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482a7a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482a7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482a7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482a7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100148 td 0xfffff80121149a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482ac910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482ac940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482ac980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482ac9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482aca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482aca70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482acab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482acab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100149 td 0xfffff80121149500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482b1910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482b1940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482b1980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482b19b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482b1a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482b1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482b1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482b1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100151 td 0xfffff80121149000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482bb910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482bb940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482bb980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482bb9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482bba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482bba70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482bbab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482bbab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100152 td 0xfffff80121148a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482c0910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482c0940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482c0980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482c09b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482c0a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482c0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482c0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100153 td 0xfffff80121148500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482c5910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482c5940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482c5980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482c59b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482c5a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482c5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482c5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482c5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100154 td 0xfffff80121148000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482ca910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482ca940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482ca980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482ca9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482caa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482caa70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482caab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482caab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100155 td 0xfffff80121473a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482cf910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482cf940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482cf980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482cf9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482cfa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482cfa70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482cfab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482cfab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100156 td 0xfffff80121473500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482d4910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482d4940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482d4980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482d49b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482d4a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482d4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482d4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482d4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100157 td 0xfffff80121473000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482d9910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482d9940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482d9980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482d99b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482d9a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482d9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482d9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482d9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100158 td 0xfffff80121472a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482de910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482de940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482de980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482de9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482dea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482dea70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482deab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482deab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100159 td 0xfffff80121472500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482e3910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482e3940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482e3980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482e39b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482e3a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482e3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482e3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482e3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100160 td 0xfffff80121472000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482e8910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482e8940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482e8980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482e89b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482e8a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482e8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482e8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482e8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100161 td 0xfffff80121471a00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482ed910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482ed940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482ed980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482ed9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482eda40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482eda70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482edab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482edab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100162 td 0xfffff80121471500 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482f2910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482f2940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482f2980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482f29b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482f2a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482f2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482f2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482f2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100163 td 0xfffff80121471000 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482f7910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482f7940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482f7980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482f79b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482f7a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482f7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482f7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482f7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100164 td 0xfffff801214cca00 sched_switch() at sched_switch+0x48a/frame 0xfffffe10482fc910 mi_switch() at mi_switch+0x167/frame 0xfffffe10482fc940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe10482fc980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10482fc9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe10482fca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe10482fca70 fork_exit() at fork_exit+0x84/frame 0xfffffe10482fcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe10482fcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100165 td 0xfffff801214cc500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048301910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048301940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048301980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10483019b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048301a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe1048301a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048301ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048301ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100166 td 0xfffff801214cc000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048306910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048306940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048306980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10483069b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048306a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe1048306a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048306ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048306ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100167 td 0xfffff801214cba00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104830b910 mi_switch() at mi_switch+0x167/frame 0xfffffe104830b940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104830b980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe104830b9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe104830ba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe104830ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe104830bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104830bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100168 td 0xfffff801214cb500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048310910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048310940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048310980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10483109b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048310a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe1048310a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048310ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048310ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100169 td 0xfffff801214cb000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048315910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048315940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048315980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10483159b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048315a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe1048315a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048315ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048315ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100170 td 0xfffff801214caa00 sched_switch() at sched_switch+0x48a/frame 0xfffffe104831a910 mi_switch() at mi_switch+0x167/frame 0xfffffe104831a940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104831a980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe104831a9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe104831aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe104831aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe104831aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104831aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100171 td 0xfffff801214ca500 sched_switch() at sched_switch+0x48a/frame 0xfffffe104831f910 mi_switch() at mi_switch+0x167/frame 0xfffffe104831f940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe104831f980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe104831f9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe104831fa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe104831fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe104831fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe104831fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100172 td 0xfffff801214ca000 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048324910 mi_switch() at mi_switch+0x167/frame 0xfffffe1048324940 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048324980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10483249b0 msleep_spin_sbt() at msleep_spin_sbt+0x1ec/frame 0xfffffe1048324a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe1048324a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048324ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048324ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100195 td 0xfffff80121b88500 sched_switch() at sched_switch+0x48a/frame 0xfffffe1048467900 mi_switch() at mi_switch+0x167/frame 0xfffffe1048467930 sleepq_switch() at sleepq_switch+0xe7/frame 0xfffffe1048467970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe10484679a0 _sleep() at _sleep+0x338/frame 0xfffffe1048467a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe1048467a70 fork_exit() at fork_exit+0x84/frame 0xfffffe1048467ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1048467ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:allt> call doadump Dumping 2644 out of 65427 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 37863 [Switching to thread 178 (Thread 100500)]#0 cpustop_handler () at ../../../x86/x86/mp_x86.c:1228 1228 CPU_SET_ATOMIC(cpu, &stopped_cpus); (kgdb) bt #0 cpustop_handler () at ../../../x86/x86/mp_x86.c:1228 #1 0xffffffff810438ba in ipi_nmi_handler () at ../../../x86/x86/mp_x86.c:1210 #2 0xffffffff80ee7a8a in trap (frame=0xfffffe1005789f30) at ../../../amd64/amd64/trap.c:193 #3 0xffffffff80ec80f3 in nmi_calltrap () at ../../../amd64/amd64/exception.S:510 #4 0xffffffff80b050c4 in bremfreel (bp=0xffffffff81da2f98) at ../../../kern/vfs_bio.c:1661 #5 0xffffffff80b08704 in binsfree (bp=0xfffffe0f917910c0, qindex=0x4) at ../../../kern/vfs_bio.c:1338 #6 0xffffffff80b06917 in brelse (bp=0xfffffe0f917910c0) at ../../../kern/vfs_bio.c:2407 #7 0xffffffff80d335ae in ffs_getpages (ap=0xfffffe104ed71068) at ../../../ufs/ffs/ffs_vnops.c:1907 #8 0xffffffff81055126 in VOP_GETPAGES_APV (vop=0xffffffff819897c0, a=0xfffffe104ed71068) at vnode_if.c:2798 #9 0xffffffff80d7b23d in vnode_pager_getpages (object=, m=0xfffffe104ed71478, count=0x1, rbehind=0xfffffe104ed71450, rahead=0xfffffe104ed71454) at vnode_if.h:1154 #10 0xffffffff80d73116 in vm_pager_get_pages (object=0xfffff80932cf9630, m=0xfffffe104ed71478, count=0x1, rbehind=0xfffffe104ed71450, rahead=0xfffffe104ed71454) at ../../../vm/vm_pager.c:291 #11 0xffffffff80d558d9 in vm_fault_hold (map=0xfffff801311d9000, vaddr=0x800647000, fault_type=0x1, fault_flags=0x0, m_hold=0xfffffe104ed716f0) at ../../../vm/vm_fault.c:707 #12 0xffffffff80d57b63 in vm_fault_quick_hold_pages (map=0xfffff801311d9000, addr=0x800647000, len=0x2000, prot=0x1, ma=0xfffffe104ed716f0, max_count=0x12) at ../../../vm/vm_fault.c:1279 #13 0xffffffff80b3a6a5 in vn_io_fault1 (vp=0xfffff80fd2105000, uio=0xfffffe104ed71900, args=0xfffffe104ed717d8, td=0xfffff80131119a00) at ../../../kern/vfs_vnops.c:1096 #14 0xffffffff80b376c6 in vn_io_fault (fp=0xfffff80121828f00, uio=0xfffffe104ed71900, active_cred=0xfffff80153b26500, flags=0x0, td=0xfffff80131119a00) at ../../../kern/vfs_vnops.c:1165 #15 0xffffffff80ac5d04 in dofilewrite (td=0xfffff80131119a00, fd=0x4, fp=0xfffff80121828f00, auio=0xfffffe104ed71900, offset=, flags=0x0) at file.h:311 #16 0xffffffff80ac59a8 in kern_writev (td=0xfffff80131119a00, fd=0x4, auio=0xfffffe104ed71900) at ../../../kern/sys_generic.c:508 #17 0xffffffff80ac5934 in sys_write (td=0xfffffe0f917910c0, uap=) at ../../../kern/sys_generic.c:421 #18 0xffffffff80ee8dcf in amd64_syscall (td=0xfffff80131119a00, traced=) at subr_syscall.c:135 #19 0xffffffff80ec7f9b in Xfast_syscall () at ../../../amd64/amd64/exception.S:396 #20 0x000000080097c09a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 4 #4 0xffffffff80b050c4 in bremfreel (bp=0xffffffff81da2f98) at ../../../kern/vfs_bio.c:1661 1661 { (kgdb) l 1656 * Removes a buffer from the free list, must be called with the 1657 * correct qlock held. 1658 */ 1659 static void 1660 bremfreel(struct buf *bp) 1661 { 1662 1663 CTR3(KTR_BUF, "bremfreel(%p) vp %p flags %X", 1664 bp, bp->b_vp, bp->b_flags); 1665 KASSERT(bp->b_qindex != QUEUE_NONE, (kgdb) p *bp $1 = {b_bufobj = 0xfffff80131119a00, b_bcount = 0x0, b_caller1 = 0x0, b_data = 0x0, b_error = 0x0, b_iocmd = 0x0, b_ioflags = 0x0, b_iooffset = 0x0, b_resid = 0x0, b_iodone = 0, b_blkno = 0x0, b_offset = 0x0, b_bobufs = {tqe_next = 0x0, tqe_prev = 0x0}, b_vflags = 0x0, b_qindex = 0x0, b_flags = 0x81434974, b_xflags = 0xff, b_lock = {lock_object = {lo_name = 0x1030000
, lo_flags = 0xbdd600, lo_data = 0xfffffe00, lo_witness = 0x4}, lk_lock = 0x0, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x0, lk_stack = {depth = 0x0, pcs = 0xffffffff81da3040}}, b_bufsize = 0x0, b_runningbufspace = 0x0, b_kvasize = 0x0, b_dirtyoff = 0x0, b_dirtyend = 0x0, b_kvabase = 0x0, b_lblkno = 0x0, b_vp = 0x0, b_rcred = 0xffffffff81434974, b_wcred = 0x1030000, {b_freelist = { tqe_next = 0xfffffe0000bdd600, tqe_prev = 0x4}, {b_pgiodone = 0xfffffe0000bdd600, b_pgbefore = 0x4, b_pgafter = 0x0}}, b_cluster = {cluster_head = { tqh_first = 0x0, tqh_last = 0x0}, cluster_entry = {tqe_next = 0x0, tqe_prev = 0x0}}, b_pages = 0xffffffff81da3130, b_npages = 0x0, b_dep = {lh_first = 0x0}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0} (kgdb) f 5 #5 0xffffffff80b08704 in binsfree (bp=0xfffffe0f917910c0, qindex=0x4) at ../../../kern/vfs_bio.c:1338 1338 bremfreel(bp); (kgdb) l 1333 */ 1334 nlock = bqlock(qindex); 1335 if (bp->b_flags & B_REMFREE) { 1336 olock = bqlock(bp->b_qindex); 1337 mtx_lock(olock); 1338 bremfreel(bp); 1339 if (olock != nlock) { 1340 mtx_unlock(olock); 1341 mtx_lock(nlock); 1342 } (kgdb) info loc olock = (struct mtx *) 0xffffffff81da2f80 nlock = (struct mtx *) 0xffffffff81da2f80 (kgdb) f 10 #10 0xffffffff80d73116 in vm_pager_get_pages (object=0xfffff80932cf9630, m=0xfffffe104ed71478, count=0x1, rbehind=0xfffffe104ed71450, rahead=0xfffffe104ed71454) at ../../../vm/vm_pager.c:291 291 r = (*pagertab[object->type]->pgo_getpages)(object, m, count, rbehind, (kgdb) p *object $2 = {lock = {lock_object = {lo_name = 0xffffffff81475386 "vm object", lo_flags = 0x25630000, lo_data = 0x0, lo_witness = 0xfffffe0000bd8700}, rw_lock = 0x1}, object_list = {tqe_next = 0xfffff80932cf9738, tqe_prev = 0xfffff80932cf9548}, shadow_head = {lh_first = 0x0}, shadow_list = {le_next = 0xfffff801a4411420, le_prev = 0xfffff801a3ce2030}, memq = {tqh_first = 0xfffff80fec3f0fe8, tqh_last = 0xfffff80fec3ea7f8}, rtree = {rt_root = 0xfffff8053c9cb240, rt_flags = 0x0}, size = 0x8, generation = 0x1, ref_count = 0x3, shadow_count = 0x0, memattr = 0x6, type = 0x2, flags = 0x1000, pg_color = 0x0, paging_in_progress = 0x1, resident_page_count = 0x8, backing_object = 0x0, backing_object_offset = 0x0, pager_object_list = {tqe_next = 0x0, tqe_prev = 0x0}, rvq = {lh_first = 0x0}, cache = { rt_root = 0x0, rt_flags = 0x0}, handle = 0xfffff80fd20cd9c0, un_pager = {vnp = {vnp_size = 0x8000, writemappings = 0x2000}, devp = {devp_pglist = { tqh_first = 0x8000, tqh_last = 0x2000}, ops = 0x0, dev = 0x0}, sgp = {sgp_pglist = {tqh_first = 0x8000, tqh_last = 0x2000}}, swp = {swp_tmpfs = 0x8000, swp_bcount = 0x2000}}, cred = 0x0, charge = 0x0, umtx_data = 0x0} (kgdb) f 11 #11 0xffffffff80d558d9 in vm_fault_hold (map=0xfffff801311d9000, vaddr=0x800647000, fault_type=0x1, fault_flags=0x0, m_hold=0xfffffe104ed716f0) at ../../../vm/vm_fault.c:707 707 rv = vm_pager_get_pages(fs.object, &fs.m, 1, (kgdb) p *map $3 = {header = {prev = 0xfffff80121ae7380, next = 0xfffff80443059080, left = 0x0, right = 0x0, start = 0x1000, end = 0x800000000000, avail_ssize = 0x0, next_read = 0x0, adj_free = 0x0, max_free = 0x0, object = {vm_object = 0x0, sub_map = 0x0}, offset = 0x0, eflags = 0x0, protection = 0x0, max_protection = 0x0, inheritance = 0x0, read_ahead = 0x0, wired_count = 0x0, cred = 0x0, wiring_thread = 0x0}, lock = {lock_object = {lo_name = 0xffffffff8142e790 "vm map (user)", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xfffffe0000bd8680}, sx_lock = 0x1}, system_mtx = {lock_object = { lo_name = 0xffffffff8142e762 "vm map (system)", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xfffffe0000bd8480}, mtx_lock = 0x4}, nentries = 0x10, size = 0x62f000, timestamp = 0x35, needs_wakeup = 0x0, system_map = 0x0, flags = 0x0, root = 0xfffff80121fa9d80, pmap = 0xfffff801311d9138, busy = 0x0} (kgdb) $ svnlite diff -x -p /usr/src/sys Index: /usr/src/sys/ufs/ffs/ffs_vnops.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_vnops.c (revision 307513) +++ /usr/src/sys/ufs/ffs/ffs_vnops.c (working copy) @@ -77,6 +77,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -86,6 +87,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -102,8 +104,9 @@ __FBSDID("$FreeBSD$"); #ifdef DIRECTIO extern int ffs_rawread(struct vnode *vp, struct uio *uio, int *workdone); #endif +static vop_fdatasync_t ffs_fdatasync; static vop_fsync_t ffs_fsync; -static vop_fdatasync_t ffs_fdatasync; +static vop_getpages_t ffs_getpages; static vop_lock1_t ffs_lock; static vop_read_t ffs_read; static vop_write_t ffs_write; @@ -119,13 +122,12 @@ static vop_openextattr_t ffs_openextattr; static vop_setextattr_t ffs_setextattr; static vop_vptofh_t ffs_vptofh; - /* Global vfs data structures for ufs. */ struct vop_vector ffs_vnodeops1 = { .vop_default = &ufs_vnodeops, .vop_fsync = ffs_fsync, .vop_fdatasync = ffs_fdatasync, - .vop_getpages = vnode_pager_local_getpages, + .vop_getpages = ffs_getpages, .vop_getpages_async = vnode_pager_local_getpages_async, .vop_lock1 = ffs_lock, .vop_read = ffs_read, @@ -147,7 +149,7 @@ struct vop_vector ffs_vnodeops2 = { .vop_default = &ufs_vnodeops, .vop_fsync = ffs_fsync, .vop_fdatasync = ffs_fdatasync, - .vop_getpages = vnode_pager_local_getpages, + .vop_getpages = ffs_getpages, .vop_getpages_async = vnode_pager_local_getpages_async, .vop_lock1 = ffs_lock, .vop_read = ffs_read, @@ -1784,3 +1786,156 @@ vop_vptofh { ufhp->ufid_gen = ip->i_gen; return (0); } + +SYSCTL_DECL(_vfs_ffs); +static int use_buf_pager = 1; +SYSCTL_INT(_vfs_ffs, OID_AUTO, use_buf_pager, CTLFLAG_RWTUN, &use_buf_pager, 0, + "Always use buffer pager instead of bmap"); +static int buf_pager_relbuf; +SYSCTL_INT(_vfs_ffs, OID_AUTO, buf_pager_relbuf, CTLFLAG_RWTUN, + &buf_pager_relbuf, 0, + "Make buffer pager release buffers after reading"); + +/* + * The FFS pager. It uses buffer reads to validate pages. + * + * In contrast to the generic local pager from vm/vnode_pager.c, this + * pager correctly and easily handles volumes where the underlying + * device block size is greater than the machine page size. The + * buffer cache transparently extends the requested page run to be + * aligned at the block boundary, and does the necessary bogus page + * replacements in the addends to avoid obliterating already valid + * pages. + * + * The only non-trivial issue is that the exclusive busy state for + * pages, which is assumed by the vm_pager_getpages() interface, is + * incompatible with the VMIO buffer cache's desire to share-busy the + * pages. This function performs a trivial downgrade of the pages' + * state before reading buffers, and a less trivial upgrade from the + * shared-busy to excl-busy state after the read. + */ +static int +ffs_getpages(struct vop_getpages_args *ap) +{ + struct vnode *vp; + vm_page_t *ma, m; + vm_object_t object; + struct buf *bp; + struct ufsmount *um; + ufs_lbn_t lbn, lbnp; + vm_ooffset_t la, lb; + long bsize; + int bo_bs, count, error, i; + bool redo, lpart; + + vp = ap->a_vp; + ma = ap->a_m; + count = ap->a_count; + + um = VFSTOUFS(ap->a_vp->v_mount); + bo_bs = um->um_devvp->v_bufobj.bo_bsize; + if (!use_buf_pager && bo_bs <= PAGE_SIZE) + return (vnode_pager_generic_getpages(vp, ma, count, + ap->a_rbehind, ap->a_rahead, NULL, NULL)); + + object = vp->v_object; + la = IDX_TO_OFF(ma[count - 1]->pindex); + if (la >= object->un_pager.vnp.vnp_size) + return (VM_PAGER_BAD); + lpart = la + PAGE_SIZE > object->un_pager.vnp.vnp_size; + if (ap->a_rbehind != NULL) { + lb = IDX_TO_OFF(ma[0]->pindex); + *ap->a_rbehind = OFF_TO_IDX(lb - rounddown2(lb, bo_bs)); + } + if (ap->a_rahead != NULL) { + *ap->a_rahead = OFF_TO_IDX(roundup2(la, bo_bs) - la); + if (la + IDX_TO_OFF(*ap->a_rahead) >= + object->un_pager.vnp.vnp_size) { + *ap->a_rahead = OFF_TO_IDX(roundup2(object->un_pager. + vnp.vnp_size, PAGE_SIZE) - la); + } + } + VM_OBJECT_WLOCK(object); +again: + for (i = 0; i < count; i++) + vm_page_busy_downgrade(ma[i]); + VM_OBJECT_WUNLOCK(object); + + lbnp = -1; + for (i = 0; i < count; i++) { + m = ma[i]; + + /* + * Pages are shared busy and the object lock is not + * owned, which together allow for the pages' + * invalidation. The racy test for validity avoids + * useless creation of the buffer for the most typical + * case when invalidation is not used in redo or for + * parallel read. The shared->excl upgrade loop at + * the end of the function catches the race in a + * reliable way (protected by the object lock). + */ + if (m->valid == VM_PAGE_BITS_ALL) + continue; + + lbn = lblkno(um->um_fs, IDX_TO_OFF(m->pindex)); + if (lbn != lbnp) { + bsize = blksize(um->um_fs, VTOI(vp), lbn); + error = bread_gb(vp, lbn, bsize, NOCRED, GB_UNMAPPED, + &bp); + if (error != 0) + break; + KASSERT(1 /* racy, enable for debugging */ || + m->valid == VM_PAGE_BITS_ALL || i == count - 1, + ("buf %d %p invalid", i, m)); + if (i == count - 1 && lpart) { + VM_OBJECT_WLOCK(object); + if (m->valid != 0 && + m->valid != VM_PAGE_BITS_ALL) + vm_page_zero_invalid(m, TRUE); + VM_OBJECT_WUNLOCK(object); + } + if (LIST_EMPTY(&bp->b_dep)) { + /* + * Not strictly necessary, enable + * to reduce buf cache pressure. + */ + if (buf_pager_relbuf) + bp->b_flags |= B_RELBUF; + + bp->b_flags &= ~B_NOCACHE; + brelse(bp); + } else { + bqrelse(bp); + } + lbnp = lbn; + } + } + + VM_OBJECT_WLOCK(object); + redo = false; + for (i = 0; i < count; i++) { + vm_page_sunbusy(ma[i]); + ma[i] = vm_page_grab(object, ma[i]->pindex, VM_ALLOC_NORMAL); + + /* + * Since the pages were only sbusy while neither the + * buffer nor the object lock was held by us, or + * reallocated while vm_page_grab() slept for busy + * relinguish, they could have been invalidated. + * Recheck the valid bits and re-read as needed. + * + * Note that the last page is made fully valid in the + * read loop, and partial validity for the page at + * index count - 1 could mean that the page was + * invalidated or removed, so we must restart for + * safety as well. + */ + if (ma[i]->valid != VM_PAGE_BITS_ALL) + redo = true; + } + if (redo && error == 0) + goto again; + VM_OBJECT_WUNLOCK(object); + return (error != 0 ? VM_PAGER_ERROR : VM_PAGER_OK); +} -- Test scenario: misc/datamove.sh