KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r322496+703d159(nofault)-dirty: Tue Aug 15 08:32:59 UTC 2017 pho@zoo.freebsd.org:/tank/users/pho/mercat1/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33117904896 (31583 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads arc4random: no preloaded entropy cache ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #10 Launched! Timecounter "TSC-low" frequency 1750034768 Hz quality 1000 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff80f50900, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 netmap: loaded module nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 6 queue msgs: 9 admincnt: 1 igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: using 6 rx queues 6 tx queues iugen2.2: at usbus2 uhub3 on uhub1 uhub3: on usbus2 ugen1.2: at usbus1 uhub4 on uhub2 uhub4: on usbus1 ugen0.2: at usbus0 uhub5 on uhub0 uhub5: on usbus0 uhub4: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered uhub5: 4 ports with 3 removable, self powered ugen0.3: at usbus0 ukbd0 on uhub5 ukbd0: on usbus0 kbd2 at ukbd0 NFS ROOT: 192.168.5.1:/zoo/mercat1 random: unblocking device. Expensive timeout(9) function: 0xffffffff808d1760(0xffffffff81926060) 0.036316931 s arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache Link state changed to up igb0: link state changed to UP Interface igb0 IP-Address 192.168.5.171 Broadcast 192.168.5.255 Setting hostuuid: 00000000-0000-0000-0000-002590c99976. Setting hostid: 0x90b83241. Entropy harvesting:sysctl: unknown oid 'kern.random.sys.harvest.interrupt' interruptssysctl: unknown oid 'kern.random.sys.harvest.ethernet' ethernetsysctl: unknown oid 'kern.random.sys.harvest.point_to_point' point_to_pointsysctl: unknown oid 'kern.random.sys.harvest.swi' swi. Starting file system checks: mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/mercat1 Mounting local file systems:WARNING: /work was not properly dismounted . Writing entropy file:. Setting hostname: mercat1.netperf.freebsd.org. Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo igb0: flags=8843 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:da inet 192.168.5.171 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 nd6 options=21 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier ums0 on uhub5 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. Cannot open or create pidfile: Operation not supported can't open and lock /var/db/dhclient.leases.igb0: Operation not supported exiting. /etc/rc.d/dhclient: WARNING: failed to start dhclient devd: cannot open pid file: Operation not supported Starting pflogd: add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Mounting NFS file systems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc48 /usr/local/lib/perl5/5.20/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: cannot open pid file: Operation not supported Setting date via ntp. 16 Aug 20:52:14 ntpdate[551]: step time server 192.168.5.1 offset 0.268975 sec Starting rpcbind. NFS access cache time=60 savecore: reboot after panic: Assertion (x & ~LK_EXCLUSIVE_SPINNERS) == LK_SHARERS_LOCK(1) failed at ../../../kern/kern_lock.c:534 Aug 16 20:52:14 mercat1 savecore: reboot after panic: Assertion (x & ~LK_EXCLUSIVE_SPINNERS) == LK_SHARERS_LOCK(1) failed at ../../../kern/kern_lock.c:534 savecore: writing core to /var/crash/vmcore.1 Writing crash summary to /var/crash/core.txt.1. Starting casperd. Clearing /tmp (X related). Starting statd. Starting lockd. Updating motd:. Mounting late file systems:. Configuring syscons: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Wed Aug 16 20:52 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: root Aug 16 20:55:23 mercat1 login: ROOT LOGIN (root) ON ttyu1 Last login: Wed Aug 16 04:51:17 on ttyu1 FreeBSD 12.0-CURRENT (PHO) #0 r322496+703d159(nofault)-dirty: Tue Aug 15 08:32:59 UTC 2017 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc package has been installed (or fetched via pkg install lang-freebsd-doc, where lang is the 2-letter language code, e.g. en), they are also available formatted in /usr/local/share/doc/freebsd. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. Edit /etc/motd to change this login announcement. root@mercat1:~ # tput reset [!p[?3;4l>root@mercat1:~ # mount 192.168.5.1:/zoo/mercat1 on / (nfs) devfs on /dev (devfs, local, multilabel) /dev/ada0p2 on /work (ufs, local, soft-updates) 192.168.5.1:/zoo/pho on /zoo/pho (nfs) 192.168.5.1:/tank/users/pho on /tank/users/pho (nfs) root@mercat1:~ # lock order reversal: 1st 0xfffff80046a15908 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:283 2nd 0xfffff80014422040 filedesc structure (filedesc structure) @ kern/uipc_usrreq.c:1947 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a54e3b at _sx_slock+0x6b #3 0xffffffff80ae84d7 at uipc_send+0x167 #4 0xffffffff80ade6f9 at sosend_generic+0x449 #5 0xffffffff80ae5012 at kern_sendit+0x242 #6 0xffffffff80ae538e at sendit+0x18e #7 0xffffffff80ae5451 at sys_sendmsg+0x61 #8 0xffffffff80ed3aeb at amd64_syscall+0x2eb #9 0xffffffff80eb3b5b at Xfast_syscall+0xfb Aug 16 20:57:31 mercat1 su: pho to root on /dev/pts/0 20170816 20:57:48 all (1/254): nullfs18.sh lock order reversal: 1st 0xfffffe07c1e89970 bufwait (bufwait) @ kern/vfs_bio.c:3537 2nd 0xfffff8000f1b4000 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:281 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a53f8b at _sx_xlock+0x6b #3 0xffffffff80d2834a at ufsdirhash_add+0x4a #4 0xffffffff80d2cbde at ufs_direnter+0x4de #5 0xffffffff80d35ec3 at ufs_makeinode+0x5a3 #6 0xffffffff80d31d64 at ufs_create+0x34 #7 0xffffffff8104d0ce at VOP_CREATE_APV+0xfe #8 0xffffffff824261f2 at null_bypass+0x102 #9 0xffffffff8104d0d3 at VOP_CREATE_APV+0x103 #10 0xffffffff80b24c81 at vn_open_cred+0x2d1 #11 0xffffffff80b1d30f at kern_openat+0x26f #12 0xffffffff80ed3aeb at amd64_syscall+0x2eb #13 0xffffffff80eb3b5b at Xfast_syscall+0xfb lock order reversal: 1st 0xfffff80046fb1a28 nfs (nfs) @ kern/vfs_mount.c:1271 2nd 0xfffff801bc02e068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1410 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a22a63 at lockmgr_lock_fast_path+0x153 #3 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #4 0xffffffff80b2547a at _vn_lock+0xba #5 0xffffffff80d1ef60 at ffs_flushfiles+0x120 #6 0xffffffff80d2173d at ffs_unmount+0xed #7 0xffffffff80b0babb at dounmount+0x64b #8 0xffffffff80b0b446 at sys_unmount+0x3c6 #9 0xffffffff80ed3aeb at amd64_syscall+0x2eb #10 0xffffffff80eb3b5b at Xfast_syscall+0xfb 20170816 20:59:59 all (2/254): nullfs19.sh lock order reversal: 1st 0xfffff80605f83068 ufs (ufs) @ /tank/users/pho/mercat1/usr/src/sys/fs/nullfs/null_vnops.c:670 2nd 0xfffffe07c491eef0 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:280 3rd 0xfffff805ae6767b8 ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a23621 at __lockmgr_args+0xa71 #3 0xffffffff80d23482 at ffs_lock+0x92 #4 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #5 0xffffffff80b2547a at _vn_lock+0xba #6 0xffffffff80b1390a at vget+0xaa #7 0xffffffff80b056dc at vfs_hash_get+0xcc #8 0xffffffff80d1f021 at ffs_vgetf+0x41 #9 0xffffffff80d153a5 at softdep_sync_buf+0xa25 #10 0xffffffff80d240d6 at ffs_syncvnode+0x2a6 #11 0xffffffff80cfb070 at ffs_truncate+0x760 #12 0xffffffff80d2cd7e at ufs_direnter+0x67e #13 0xffffffff80d35ec3 at ufs_makeinode+0x5a3 #14 0xffffffff80d31d64 at ufs_create+0x34 #15 0xffffffff8104d0ce at VOP_CREATE_APV+0xfe #16 0xffffffff824261f2 at null_bypass+0x102 #17 0xffffffff8104d0d3 at VOP_CREATE_APV+0x103 lock order reversal: 1st 0xfffff801f86597b8 ufs (ufs) @ /tank/users/pho/mercat1/usr/src/sys/fs/nullfs/null_vnops.c:670 2nd 0xffffffff81e10328 proctree (proctree) @ kern/subr_prf.c:165 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a54e3b at _sx_slock+0x6b #3 0xffffffff80a921a7 at uprintf+0x67 #4 0xffffffff80cf33bb at ffs_valloc+0x4eb #5 0xffffffff80d359d6 at ufs_makeinode+0xb6 #6 0xffffffff80d31d64 at ufs_create+0x34 #7 0xffffffff8104d0ce at VOP_CREATE_APV+0xfe #8 0xffffffff824261f2 at null_bypass+0x102 #9 0xffffffff8104d0d3 at VOP_CREATE_APV+0x103 #10 0xffffffff80b24c81 at vn_open_cred+0x2d1 #11 0xffffffff80b1d30f at kern_openat+0x26f #12 0xffffffff80ed3aeb at amd64_syscall+0x2eb #13 0xffffffff80eb3b5b at Xfast_syscall+0xfb Aug 16 21:00:18 mercat1 kernel: pid 3608 (creat), uid 2007 inumber 65548 on /mnt: out of inodes 20170816 21:02:29 all (3/254): nullfs2.sh 20170816 21:14:06 all (4/254): nullfs20.sh 20170816 21:24:34 all (5/254): nullfs21.sh 20170816 21:24:35 all (6/254): nullfs22.sh 20170816 21:25:20 all (7/254): nullfs3.sh 20170816 21:25:21 all (8/254): nullfs4.sh lock order reversal: 1st 0xfffff803f250a2d8 pseudofs (pseudofs) @ kern/vfs_lookup.c:669 2nd 0xffffffff81e103a0 allproc (allproc) @ kern/kern_proc.c:334 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a54e3b at _sx_slock+0x6b #3 0xffffffff80a31f92 at pfind+0x22 #4 0xffffffff8098cd7a at pfs_lookup+0x49a #5 0xffffffff8104cefe at VOP_CACHEDLOOKUP_APV+0xfe #6 0xffffffff80afddd6 at vfs_cache_lookup+0xd6 #7 0xffffffff8104cd2e at VOP_LOOKUP_APV+0xfe #8 0xffffffff80b075e2 at lookup+0x732 #9 0xffffffff80b06aec at namei+0x52c #10 0xffffffff80b1f28e at kern_statat+0xae #11 0xffffffff80b1fa9f at sys_fstatat+0x2f #12 0xffffffff80ed3aeb at amd64_syscall+0x2eb #13 0xffffffff80eb3b5b at Xfast_syscall+0xfb 20170816 21:25:22 all (9/254): nullfs5.sh freework leak: 12/13. newblk leak: 8/10. 20170816 21:37:19 all (10/254): nullfs6.sh 20170816 21:37:20 all (11/254): nullfs7.sh lock order reversal: 1st 0xfffff803756f7a28 ufs (ufs) @ /tank/users/pho/mercat1/usr/src/sys/fs/nullfs/null_vnops.c:670 2nd 0xfffff8000ff4e068 nfs (nfs) @ /tank/users/pho/mercat1/usr/src/sys/fs/nullfs/null_vnops.c:670 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a22a63 at lockmgr_lock_fast_path+0x153 #3 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #4 0xffffffff82426a25 at null_lock+0xe5 #5 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #6 0xffffffff80b2547a at _vn_lock+0xba #7 0xffffffff82425c20 at nullfs_root+0x50 #8 0xffffffff80b0a1de at vfs_donmount+0x117e #9 0xffffffff80b09031 at sys_nmount+0x71 #10 0xffffffff80ed3aeb at amd64_syscall+0x2eb #11 0xffffffff80eb3b5b at Xfast_syscall+0xfb 20170816 21:37:22 all (12/254): nullfs9.sh 20170816 21:37:22 all (13/254): oom.sh 20170816 21:37:23 all (14/254): oovm.sh lock order reversal: 1st 0xfffff800141a62d8 syncer (syncer) @ kern/vfs_subr.c:2152 2nd 0xfffff8070b181c98 ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a23621 at __lockmgr_args+0xa71 #3 0xffffffff80d23482 at ffs_lock+0x92 #4 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #5 0xffffffff80b2547a at _vn_lock+0xba #6 0xffffffff80b1390a at vget+0xaa #7 0xffffffff80b15c89 at vfs_msync+0xd9 #8 0xffffffff80b180d8 at sync_fsync+0xc8 #9 0xffffffff8104ecc4 at VOP_FSYNC_APV+0x104 #10 0xffffffff80b19f96 at sched_sync+0x346 #11 0xffffffff80a0e9c4 at fork_exit+0x84 #12 0xffffffff80eb3dee at fork_trampoline+0xe Aug 16 21:38:10 mercat1 su: pho to root on /dev/pts/1 freework leak: 16/29. pts leak: 1/2 20170816 21:48:03 all (15/254): oovm2.sh Aug 16 21:48:51 mercat1 su: pho to root on /dev/pts/2 freework leak: 28/57. pts leak: 1/3 20170816 21:58:29 all (16/254): open.sh 20170816 21:58:30 all (17/254): openlock.sh 20170816 21:58:32 all (18/254): overcommit.sh Aug 16 22:02:46 mercat1 kernel: pid 33630 (kgdb), uid 0, was killed: out of swap space 20170816 22:10:19 all (19/254): overcommit2.sh Aug 16 22:12:26 mercat1 kernel: pid 36233 (dd), uid 0 inumber 9 on /mnt: filesystem full 20170816 22:12:33 all (20/254): overlap.sh 20170816 22:12:35 all (21/254): pageout.sh lock order reversal: 1st 0xfffffe07c36e2090 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:280 2nd 0xfffff80046ae8030 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2293 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a23621 at __lockmgr_args+0xa71 #3 0xffffffff80d01d49 at ffs_copyonwrite+0x179 #4 0xffffffff80d22669 at ffs_geom_strategy+0xe9 #5 0xffffffff80d35653 at ufs_strategy+0x83 #6 0xffffffff810504cc at VOP_STRATEGY_APV+0x10c #7 0xffffffff80af1206 at bufstrategy+0x46 #8 0xffffffff80af1005 at bufwrite+0x205 #9 0xffffffff80d24182 at ffs_syncvnode+0x352 #10 0xffffffff80d23330 at ffs_fsync+0x20 #11 0xffffffff8104ecc4 at VOP_FSYNC_APV+0x104 #12 0xffffffff80d331bf at ufs_remove+0xdf #13 0xffffffff8104eea2 at VOP_REMOVE_APV+0x132 #14 0xffffffff80b1e8b9 at kern_unlinkat+0x209 #15 0xffffffff80ed3aeb at amd64_syscall+0x2eb #16 0xffffffff80eb3b5b at Xfast_syscall+0xfb lock order reversal: 1st 0xfffff80046ae8030 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:939 2nd 0xfffff80276d50c98 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1620 stack backtrace: #0 0xffffffff80aacdc3 at witness_debugger+0x73 #1 0xffffffff80aaccc9 at witness_checkorder+0xea9 #2 0xffffffff80a23621 at __lockmgr_args+0xa71 #3 0xffffffff80d002fe at ffs_snapremove+0xfe #4 0xffffffff80cfabb7 at ffs_truncate+0x2a7 #5 0xffffffff80d2b244 at ufs_inactive+0x174 #6 0xffffffff8104fcd4 at VOP_INACTIVE_APV+0x104 #7 0xffffffff80b13d8e at vinactive+0xde #8 0xffffffff80b14407 at vputx+0x317 #9 0xffffffff80b1e8f3 at kern_unlinkat+0x243 #10 0xffffffff80ed3aeb at amd64_syscall+0x2eb #11 0xffffffff80eb3b5b at Xfast_syscall+0xfb 20170816 22:22:36 all (22/254): parallelmount.sh 20170816 22:32:36 all (23/254): parallelmount2.sh 20170816 22:32:37 all (24/254): pathconf.sh 20170816 22:32:38 all (25/254): pathconf2.sh 20170816 22:32:38 all (26/254): pcatch.sh 20170816 22:32:39 all (27/254): pcatch2.sh 20170816 22:32:40 all (28/254): pfl.sh fsync: giving up on dirty 0xfffff8068dcb8000: tag devfs, type VCHR usecount 1, writecount 0, refcount 1893 mountedhere 0xfffff80738c5ca00 flags (VI_ACTIVE) v_object 0xfffff8068385c3c0 ref 0 pages 15110 cleanbuf 1889 dirtybuf 2 lock type devfs: EXCL by thread 0xfffff80014393000 (pid 71506, umount, tid 100161) #0 0xffffffff80a22ac6 at lockmgr_lock_fast_path+0x1b6 #1 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #2 0xffffffff80b2547a at _vn_lock+0xba #3 0xffffffff80d02e50 at softdep_flushworklist+0x70 #4 0xffffffff80d21e33 at ffs_sync+0x2a3 #5 0xffffffff80b0ba70 at dounmount+0x600 #6 0xffffffff80b0b446 at sys_unmount+0x3c6 #7 0xffffffff80ed3aeb at amd64_syscall+0x2eb #8 0xffffffff80eb3b5b at Xfast_syscall+0xfb dev md11a 20170816 22:36:44 all (29/254): pfl2.sh 20170816 22:48:18 all (30/254): pfl4.sh 20170816 22:58:45 all (31/254): pipe.sh 20170816 23:03:48 all (32/254): pipe2.sh 20170816 23:04:00 all (33/254): posix_fadvise.sh 20170816 23:04:02 all (34/254): posix_fadvise2.sh 20170816 23:04:02 all (35/254): posix_fadvise3.sh 20170816 23:06:51 all (36/254): posix_openpt.sh 20170816 23:07:41 all (37/254): posix_openpt2.sh pts leak: 58/61 20170816 23:14:04 all (38/254): pread.sh 20170816 23:14:16 all (39/254): procfs.sh 20170816 23:14:31 all (40/254): procfs2.sh 20170816 23:14:36 all (41/254): procfs3.sh 20170816 23:15:50 all (42/254): procfs4.sh 20170816 23:26:45 all (43/254): procfs5.sh 20170816 23:26:46 all (44/254): pshared.sh 20170816 23:26:52 all (45/254): pthread.sh 20170816 23:26:58 all (46/254): pthread2.sh 20170816 23:29:24 all (47/254): pthread3.sh 20170816 23:37:49 all (48/254): pthread4.sh 20170816 23:41:27 all (49/254): pthread5.sh witness_lock_list_get: witness exhausted 20170816 23:42:36 all (50/254): pthread6.sh 20170816 23:46:39 all (51/254): pthread7.sh 20170816 23:48:55 all (52/254): pthread8.sh 20170816 23:51:56 all (53/254): pthread9.sh 20170816 23:51:56 all (54/254): ptrace.sh 20170816 23:51:57 all (55/254): ptrace10.sh 20170816 23:53:45 all (56/254): ptrace11.sh 20170816 23:53:56 all (57/254): ptrace2.sh 20170816 23:54:00 all (58/254): ptrace3.sh 20170816 23:54:00 all (59/254): ptrace4.sh 20170816 23:58:24 all (60/254): ptrace5.sh 20170817 00:03:25 all (61/254): ptrace6.sh 20170817 00:07:27 all (62/254): ptrace7.sh 20170817 00:11:27 all (63/254): ptrace8.sh 20170817 00:11:29 all (64/254): ptrace9.sh 20170817 00:11:35 all (65/254): pts.sh pts leak: 47/108 20170817 00:22:04 all (66/254): pts2.sh 20170817 00:22:05 all (67/254): pty.sh pts leak: 27/135 20170817 00:27:05 all (68/254): pty2.sh 20170817 00:30:06 all (69/254): quota1.sh 20170817 00:41:20 all (70/254): quota10.sh freework leak: 139/196. 20170817 00:42:18 all (71/254): quota11.sh 20170817 00:42:30 all (72/254): quota4.sh 20170817 00:43:38 all (73/254): quota5.sh 20170817 00:43:38 all (74/254): quota8.sh newblk leak: 114/124. 20170817 00:44:15 all (75/254): quota9.sh 20170817 00:44:18 all (76/254): radix.sh 20170817 00:50:27 all (77/254): rdgsbase.sh 20170817 00:50:38 all (78/254): rdwr.sh 20170817 00:50:56 all (79/254): readdir.sh 20170817 01:03:01 all (80/254): recursiveflushes.sh 20170817 01:03:06 all (81/254): rename.sh 20170817 01:08:06 all (82/254): rename10.sh 20170817 01:09:15 all (83/254): rename11.sh 20170817 01:09:20 all (84/254): rename12.sh 20170817 01:12:21 all (85/254): rename13.sh 20170817 01:12:22 all (86/254): rename2.sh 20170817 01:32:22 all (87/254): rename3.sh 20170817 01:40:57 all (88/254): rename4.sh 20170817 01:40:58 all (89/254): rename5.sh 20170817 01:46:01 all (90/254): rename6.sh 20170817 01:46:03 all (91/254): rename7.sh 20170817 01:46:28 all (92/254): rename8.sh 20170817 01:46:38 all (93/254): rename9.sh 20170817 01:46:50 all (94/254): revoke.sh 20170817 01:46:51 all (95/254): rot.sh 20170817 02:16:54 all (96/254): routetbl.sh 20170817 02:16:54 all (97/254): ruby.sh 20170817 02:16:55 all (98/254): rw.sh Aug 17 02:21:59 mercat1 kernel: pid 88231 (rw), uid 2007 inumber 5 on /mnt: filesystem full panic: Assertion reclaimable == delta failed at ../../../net/iflib.c:1947 cpuid = 2 time = 1502929500 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c781a760 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c781a810 vpanic() at vpanic+0x19f/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 0 tid 100043 ] Stopped at kdb_enter+0x3e: movq $0,kdb_why db> set $lines 0 db> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x9ebb00 curthread = 0xfffff80046659560: pid 88260 tid 100170 "swap" curpcb = 0xfffffe085ce04b80 fpcurthread = 0xfffff80046659560: pid 88260 "swap" idlethread = 0xfffff80007793000: tid 100003 "idle: cpu0" curpmap = 0xfffff80397541130 tssp = 0xffffffff81e3ae10 commontssp = 0xffffffff81e3ae10 rsp0 = 0xfffffe085ce04b80 gs32p = 0xffffffff81e41668 ldt = 0xffffffff81e416a8 tss = 0xffffffff81e41698 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe083fbfeb00 curthread = 0xfffff800144d2560: pid 88262 tid 100137 "swap" curpcb = 0xfffffe085cd5eb80 fpcurthread = 0xfffff800144d2560: pid 88262 "swap" idlethread = 0xfffff80007792560: tid 100004 "idle: cpu1" curpmap = 0xfffff806252bb130 tssp = 0xffffffff81e3ae78 commontssp = 0xffffffff81e3ae78 rsp0 = 0xfffffe085cd5eb80 gs32p = 0xffffffff81e416d0 ldt = 0xffffffff81e41710 tss = 0xffffffff81e41700 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe083fc06b00 curthread = 0xfffff80007872000: pid 0 tid 100043 "if_io_tqg_2" curpcb = 0xfffffe07c781ab80 fpcurthread = none idlethread = 0xfffff80007792000: tid 100005 "idle: cpu2" curpmap = 0xffffffff81e035b0 tssp = 0xffffffff81e3aee0 commontssp = 0xffffffff81e3aee0 rsp0 = 0xfffffe07c781ab80 gs32p = 0xffffffff81e41738 ldt = 0xffffffff81e41778 tss = 0xffffffff81e41768 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe083fc0eb00 curthread = 0xfffff8043a936000: pid 88259 tid 100356 "swap" curpcb = 0xfffffe085d1a8b80 fpcurthread = 0xfffff8043a936000: pid 88259 "swap" idlethread = 0xfffff8000778d560: tid 100006 "idle: cpu3" curpmap = 0xfffff8001dc56130 tssp = 0xffffffff81e3af48 commontssp = 0xffffffff81e3af48 rsp0 = 0xfffffe085d1a8b80 gs32p = 0xffffffff81e417a0 ldt = 0xffffffff81e417e0 tss = 0xffffffff81e417d0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe083fc16b00 curthread = 0xfffff800078aa560: pid 0 tid 100055 "thread taskq" curpcb = 0xfffffe07c7856b80 fpcurthread = none idlethread = 0xfffff8000778d000: tid 100007 "idle: cpu4" curpmap = 0xffffffff81e035b0 tssp = 0xffffffff81e3afb0 commontssp = 0xffffffff81e3afb0 rsp0 = 0xfffffe07c7856b80 gs32p = 0xffffffff81e41808 ldt = 0xffffffff81e41848 tss = 0xffffffff81e41838 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe083fc1eb00 curthread = 0xfffff8000778c560: pid 11 tid 100008 "idle: cpu5" curpcb = 0xfffffe0839163b80 fpcurthread = none idlethread = 0xfffff8000778c560: tid 100008 "idle: cpu5" curpmap = 0xffffffff81e035b0 tssp = 0xffffffff81e3b018 commontssp = 0xffffffff81e3b018 rsp0 = 0xfffffe0839163b80 gs32p = 0xffffffff81e41870 ldt = 0xffffffff81e418b0 tss = 0xffffffff81e418a0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe083fc26b00 curthread = 0xfffff805aeeca000: pid 88269 tid 100393 "swap" curpcb = 0xfffffe085d263b80 fpcurthread = 0xfffff805aeeca000: pid 88269 "swap" idlethread = 0xfffff8000778c000: tid 100009 "idle: cpu6" curpmap = 0xfffff803a447e130 tssp = 0xffffffff81e3b080 commontssp = 0xffffffff81e3b080 rsp0 = 0xfffffe085d263b80 gs32p = 0xffffffff81e418d8 ldt = 0xffffffff81e41918 tss = 0xffffffff81e41908 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe083fc2eb00 curthread = 0xfffff8043a936560: pid 88258 tid 100355 "swap" curpcb = 0xfffffe085d1a3b80 fpcurthread = 0xfffff8043a936560: pid 88258 "swap" idlethread = 0xfffff800077a7560: tid 100010 "idle: cpu7" curpmap = 0xfffff801bcaaa130 tssp = 0xffffffff81e3b0e8 commontssp = 0xffffffff81e3b0e8 rsp0 = 0xfffffe085d1a3b80 gs32p = 0xffffffff81e41940 ldt = 0xffffffff81e41980 tss = 0xffffffff81e41970 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe083fc36b00 curthread = 0xfffff800077a7000: pid 11 tid 100011 "idle: cpu8" curpcb = 0xfffffe0839172b80 fpcurthread = none idlethread = 0xfffff800077a7000: tid 100011 "idle: cpu8" curpmap = 0xffffffff81e035b0 tssp = 0xffffffff81e3b150 commontssp = 0xffffffff81e3b150 rsp0 = 0xfffffe0839172b80 gs32p = 0xffffffff81e419a8 ldt = 0xffffffff81e419e8 tss = 0xffffffff81e419d8 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe083fc3eb00 curthread = 0xfffff8000f1c1000: pid 18 tid 100103 "pagedaemon" curpcb = 0xfffffe085cbdeb80 fpcurthread = none idlethread = 0xfffff800077a6560: tid 100012 "idle: cpu9" curpmap = 0xffffffff81e035b0 tssp = 0xffffffff81e3b1b8 commontssp = 0xffffffff81e3b1b8 rsp0 = 0xfffffe085cbdeb80 gs32p = 0xffffffff81e41a10 ldt = 0xffffffff81e41a50 tss = 0xffffffff81e41a40 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe08b7772b00 curthread = 0xfffff80046a63560: pid 88265 tid 100182 "swap" curpcb = 0xfffffe085ce40b80 fpcurthread = 0xfffff80046a63560: pid 88265 "swap" idlethread = 0xfffff800077a6000: tid 100013 "idle: cpu10" curpmap = 0xfffff802a8205130 tssp = 0xffffffff81e3b220 commontssp = 0xffffffff81e3b220 rsp0 = 0xfffffe085ce40b80 gs32p = 0xffffffff81e41a78 ldt = 0xffffffff81e41ab8 tss = 0xffffffff81e41aa8 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe08b777ab00 curthread = 0xfffff8082eb09560: pid 88257 tid 101926 "swap" curpcb = 0xfffffe085dc63b80 fpcurthread = 0xfffff8082eb09560: pid 88257 "swap" idlethread = 0xfffff800077a5560: tid 100014 "idle: cpu11" curpmap = 0xfffff80343041130 tssp = 0xffffffff81e3b288 commontssp = 0xffffffff81e3b288 rsp0 = 0xfffffe085dc63b80 gs32p = 0xffffffff81e41ae0 ldt = 0xffffffff81e41b20 tss = 0xffffffff81e41b10 spin locks held: db> show alllocks db> show allchains chain 1: thread 100303 (pid 88268, rw) sleeping on 0xffffffff819939ac "vmwait" chain 2: thread 100122 (pid 88267, swap) sleeping on 0xffffffff819939ac "pfault" chain 3: thread 103429 (pid 88266, rw) sleeping on 0xffffffff819939ac "vmwait" chain 4: thread 100148 (pid 88264, swap) sleeping on 0xffffffff819939ac "pfault" chain 5: thread 102342 (pid 88263, rw) sleeping on 0xfffff8044bd06a80 "wait" chain 6: thread 100452 (pid 88256, swap) sleeping on 0xfffff800078d5a80 "wait" chain 7: thread 103678 (pid 88255, swap) sleeping on 0xffffffff81c1a4b5 "nanslp" chain 8: thread 100128 (pid 88254, rw) sleeping on 0xffffffff81c1a4b0 "nanslp" chain 9: thread 100505 (pid 88237, newnfs 0) sleeping on 0xffffffff81df88c0 "-" chain 10: thread 103427 (pid 88128, run) sleeping on 0xfffff8025dad0a80 "wait" chain 11: thread 100204 (pid 88127, run) sleeping on 0xfffff8004695b000 "wait" chain 12: thread 103674 (pid 88126, run) sleeping on 0xffffffff81c1a4b5 "nanslp" chain 13: thread 102344 (pid 88125, su) blocked on lockmgr @CEXCL thread 0 (pid -512, ) can run chain 14: thread 103364 (pid 88120, md10) sleeping on 0xfffff80145acd800 "mdwait" chain 15: thread 101927 (pid 88110, sh) sleeping on 0xfffff80443031540 "wait" chain 16: thread 101698 (pid 88109, tee) sleeping on 0xfffff803cdd805f0 "piperd" chain 17: thread 100206 (pid 88108, sh) sleeping on 0xfffff80046b6c000 "wait" chain 18: thread 100389 (pid 33902, hwpmc: proc(33901)) sleeping on 0xfffff803120fe2f8 "pipewr" chain 19: thread 100167 (pid 33901, pmcstat) sleeping on 0xffffffff819939ac "pfault" chain 20: thread 100396 (pid 33899, bash) sleeping on 0xfffff80046514000 "wait" chain 21: thread 100133 (pid 33897, csh) sleeping on 0xfffff8001429a5e8 "pause" chain 22: thread 100360 (pid 33896, su) sleeping on 0xfffff8039c6cd000 "wait" chain 23: thread 100401 (pid 33895, bash) sleeping on 0xfffff8026b91ba80 "wait" chain 24: thread 100357 (pid 33894, sshd) sleeping on 0xfffff803e1ed6140 "select" chain 25: thread 100351 (pid 33891, sshd) sleeping on 0xfffff80489c9ee40 "select" chain 26: thread 100266 (pid 33626, bash) sleeping on 0xfffff80007afacb0 "ttyin" chain 27: thread 100438 (pid 33624, csh) sleeping on 0xfffff801b44925e8 "pause" chain 28: thread 100265 (pid 33623, su) sleeping on 0xfffff801bcb4aa80 "wait" chain 29: thread 100440 (pid 33622, bash) sleeping on 0xfffff801b448da80 "wait" chain 30: thread 100304 (pid 33621, sshd) sleeping on 0xfffff8064a524440 "select" chain 31: thread 100145 (pid 33618, sshd) sleeping on 0xfffff80535ac6cc0 "select" chain 32: thread 100126 (pid 832, sh) sleeping on 0xfffff800143d4540 "wait" chain 33: thread 100198 (pid 831, bash) sleeping on 0xfffff80046b6ca80 "wait" chain 34: thread 100181 (pid 829, csh) sleeping on 0xfffff800145fa0a8 "pause" chain 35: thread 100197 (pid 828, su) sleeping on 0xfffff80014299000 "wait" chain 36: thread 100196 (pid 827, bash) sleeping on 0xfffff80014299540 "wait" chain 37: thread 100160 (pid 826, sshd) sleeping on 0xfffff8000774d0c0 "select" chain 38: thread 100171 (pid 823, sshd) sleeping on 0xfffff8000f3454c0 "select" chain 39: thread 100119 (pid 818, csh) sleeping on 0xfffff80007e224b0 "ttyin" chain 40: thread 100195 (pid 811, login) sleeping on 0xfffff80046b6d000 "wait" chain 41: thread 100194 (pid 810, getty) sleeping on 0xfffff80007e230b0 "ttyin" chain 42: thread 100193 (pid 809, getty) sleeping on 0xfffff8000f3638b0 "ttyin" chain 43: thread 100192 (pid 808, getty) sleeping on 0xfffff8000f363cb0 "ttyin" chain 44: thread 100191 (pid 807, getty) sleeping on 0xfffff8000f3640b0 "ttyin" chain 45: thread 100190 (pid 806, getty) sleeping on 0xfffff8000f3644b0 "ttyin" chain 46: thread 100189 (pid 805, getty) sleeping on 0xfffff8000f3648b0 "ttyin" chain 47: thread 100188 (pid 804, getty) sleeping on 0xfffff8000f364cb0 "ttyin" chain 48: thread 100180 (pid 803, getty) sleeping on 0xfffff8000f3650b0 "ttyin" chain 49: thread 100111 (pid 802, getty) sleeping on 0xfffff8000f3654b0 "ttyin" chain 50: thread 100172 (pid 766, cron) sleeping on 0xfffff8000f4fac00 "nfsreq" chain 51: thread 100183 (pid 762, sshd) sleeping on 0xfffff80007d82cc0 "select" chain 52: thread 100178 (pid 691, NLM: master) sleeping on 0xfffff8000f9910a0 "rpcsvc" chain 53: thread 100146 (pid 687, rpc.statd) sleeping on 0xfffff80007d82d40 "select" chain 54: thread 100147 (pid 662, casperd) sleeping on 0xfffff8000774d140 "select" chain 55: thread 100151 (pid 661, casperd) sleeping on 0xfffff8000774d1c0 "select" chain 56: thread 100125 (pid 554, rpcbind) sleeping on 0xfffff8000f4c7640 "select" chain 57: thread 100118 (pid 549, syslogd) sleeping on 0xfffff80007d82e40 "select" chain 58: thread 100152 (pid 462, devd) sleeping on 0xfffff8000f2cd140 "select" chain 59: thread 100108 (pid 23, vnlru) sleeping on 0xfffff8000f1ae540 "vlruwt" chain 60: thread 100107 (pid 22, syncer) sleeping on 0xffffffff81db3980 "syncer" chain 61: thread 100106 (pid 21, bufspacedaemon) sleeping on 0xffffffff81db2dd8 "-" chain 62: thread 100105 (pid 20, bufdaemon) sleeping on 0xffffffff81db2104 "psleep" chain 63: thread 100127 (pid 20, /work worker) sleeping on 0xfffff800142ce0e8 "sdflush" chain 64: thread 100592 (pid 20, /mnt worker) sleeping on 0xfffff803114886e8 "sdflush" chain 65: thread 100104 (pid 19, vmdaemon) sleeping on 0xffffffff81dbd238 "psleep" chain 66: thread 100109 (pid 18, laundry: dom0) sleeping on 0xffffffff81dbd248 "launds" chain 67: thread 100110 (pid 18, uma) sleeping on 0xffffffff81dbcbe8 "umarcl" chain 68: thread 100102 (pid 17, enc_daemon1) sleeping on 0xfffff8000f0a9000 "idle" chain 69: thread 100101 (pid 16, enc_daemon0) sleeping on 0xffffffff819939ac "vmwait" chain 70: thread 100098 (pid 15, rand_harvestq) sleeping on 0xffffffff81aed4d8 "-" chain 71: thread 100094 (pid 9, sctp_iterator) sleeping on 0xffffffff81e2a790 "waiting_for_work" chain 72: thread 100093 (pid 8, soaiod4) sleeping on 0xffffffff81db18f4 "-" chain 73: thread 100092 (pid 7, soaiod3) sleeping on 0xffffffff81db18f4 "-" chain 74: thread 100091 (pid 6, soaiod2) sleeping on 0xffffffff81db18f4 "-" chain 75: thread 100090 (pid 5, soaiod1) sleeping on 0xffffffff81db18f4 "-" chain 76: thread 100071 (pid 14, usbus0) sleeping on 0xfffffe00012f1460 "-" chain 77: thread 100072 (pid 14, usbus0) sleeping on 0xfffffe00012f14b8 "-" chain 78: thread 100073 (pid 14, usbus0) sleeping on 0xfffffe00012f1510 "-" chain 79: thread 100074 (pid 14, usbus0) sleeping on 0xfffffe00012f1568 "-" chain 80: thread 100075 (pid 14, usbus0) sleeping on 0xfffffe00012f15c0 "-" chain 81: thread 100077 (pid 14, usbus1) sleeping on 0xfffffe000114dd10 "-" chain 82: thread 100078 (pid 14, usbus1) sleeping on 0xfffffe000114dd68 "-" chain 83: thread 100079 (pid 14, usbus1) sleeping on 0xfffffe000114ddc0 "-" chain 84: thread 100080 (pid 14, usbus1) sleeping on 0xfffffe000114de18 "-" chain 85: thread 100081 (pid 14, usbus1) sleeping on 0xfffffe000114de70 "-" chain 86: thread 100082 (pid 14, usbus2) sleeping on 0xfffffe00012f9d10 "-" chain 87: thread 100083 (pid 14, usbus2) sleeping on 0xfffffe00012f9d68 "-" chain 88: thread 100084 (pid 14, usbus2) sleeping on 0xfffffe00012f9dc0 "-" chain 89: thread 100085 (pid 14, usbus2) sleeping on 0xfffffe00012f9e18 "-" chain 90: thread 100086 (pid 14, usbus2) sleeping on 0xfffffe00012f9e70 "-" chain 91: thread 100066 (pid 4, doneq0) sleeping on 0xffffffff819c0880 "-" chain 92: thread 100067 (pid 4, doneq1) sleeping on 0xffffffff819c0980 "-" chain 93: thread 100068 (pid 4, doneq2) sleeping on 0xffffffff819c0a80 "-" chain 94: thread 100100 (pid 4, scanner) sleeping on 0xffffffff819c06c8 "-" chain 95: thread 100065 (pid 3, crypto returns) sleeping on 0xffffffff81dbb010 "crypto_ret_wait" chain 96: thread 100064 (pid 2, crypto) sleeping on 0xffffffff81dbaec8 "crypto_wait" chain 97: thread 100060 (pid 13, g_event) sleeping on 0xffffffff81e02970 "-" chain 98: thread 100061 (pid 13, g_up) sleeping on 0xffffffff81e02978 "-" chain 99: thread 100062 (pid 13, g_down) sleeping on 0xffffffff81e02980 "-" chain 100: thread 100002 (pid 1, init) sleeping on 0xfffff8000778b540 "wait" chain 101: thread 100001 (pid 10, audit) sleeping on 0xffffffff81e2fe58 "audit_worker_cv" chain 102: thread 100000 (pid 0, swapper) sleeping on 0xffffffff819939ac "vmwait" chain 103: thread 100029 (pid 0, softirq_0) sleeping on 0xfffff800077e1000 "-" chain 104: thread 100030 (pid 0, softirq_1) sleeping on 0xfffff800077e0e00 "-" chain 105: thread 100031 (pid 0, softirq_2) sleeping on 0xfffff800077e0d00 "-" chain 106: thread 100032 (pid 0, softirq_3) sleeping on 0xfffff800077e0c00 "-" chain 107: thread 100033 (pid 0, softirq_4) sleeping on 0xfffff800077e0b00 "-" chain 108: thread 100034 (pid 0, softirq_5) sleeping on 0xfffff800077e0a00 "-" chain 109: thread 100035 (pid 0, softirq_6) sleeping on 0xfffff800077e0900 "-" chain 110: thread 100036 (pid 0, softirq_7) sleeping on 0xfffff800077e0800 "-" chain 111: thread 100037 (pid 0, softirq_8) sleeping on 0xfffff800077e0700 "-" chain 112: thread 100038 (pid 0, softirq_9) sleeping on 0xfffff800077e0600 "-" chain 113: thread 100039 (pid 0, softirq_10) sleeping on 0xfffff800077e0500 "-" chain 114: thread 100040 (pid 0, softirq_11) sleeping on 0xfffff800077e0400 "-" chain 115: thread 100041 (pid 0, if_io_tqg_0) sleeping on 0xfffff800077e0300 "-" chain 116: thread 100042 (pid 0, if_io_tqg_1) sleeping on 0xfffff80007857100 "-" chain 117: thread 100044 (pid 0, if_io_tqg_3) sleeping on 0xfffff80007856e00 "-" chain 118: thread 100045 (pid 0, if_io_tqg_4) sleeping on 0xfffff80007856d00 "-" chain 119: thread 100046 (pid 0, if_io_tqg_5) sleeping on 0xfffff80007856c00 "-" chain 120: thread 100047 (pid 0, if_io_tqg_6) sleeping on 0xfffff80007856b00 "-" chain 121: thread 100048 (pid 0, if_io_tqg_7) sleeping on 0xfffff80007856a00 "-" chain 122: thread 100049 (pid 0, if_io_tqg_8) sleeping on 0xfffff80007856900 "-" chain 123: thread 100050 (pid 0, if_io_tqg_9) sleeping on 0xfffff80007856800 "-" chain 124: thread 100051 (pid 0, if_io_tqg_10) sleeping on 0xfffff80007856700 "-" chain 125: thread 100052 (pid 0, if_io_tqg_11) sleeping on 0xfffff80007856600 "-" chain 126: thread 100053 (pid 0, if_config_tqg_0) sleeping on 0xfffff80007898200 "-" chain 127: thread 100056 (pid 0, aiod_kick taskq) sleeping on 0xfffff80007897e00 "-" chain 128: thread 100058 (pid 0, kqueue_ctx taskq) sleeping on 0xfffff80007897c00 "-" chain 129: thread 100063 (pid 0, firmware taskq) sleeping on 0xfffff80007897400 "-" chain 130: thread 100089 (pid 0, mca taskq) sleeping on 0xfffff8000a0d9300 "-" chain 131: thread 100095 (pid 0, acpi_task_0) sleeping on 0xfffff8000f1fa400 "-" chain 132: thread 100096 (pid 0, acpi_task_1) sleeping on 0xfffff8000f1fa400 "-" chain 133: thread 100097 (pid 0, acpi_task_2) sleeping on 0xfffff8000f1fa400 "-" chain 134: thread 100099 (pid 0, CAM taskq) sleeping on 0xfffff80007896a00 "-" db> show lockedvnods Locked vnodes vnode 0xfffff8000f97fc30: tag nfs, type VDIR usecount 102, writecount 0, refcount 103 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) v_object 0xfffff8000f15b1e0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type nfs: SHARED (count 1) #0 0xffffffff80a22ac6 at lockmgr_lock_fast_path+0x1b6 #1 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #2 0xffffffff80b2547a at _vn_lock+0xba #3 0xffffffff80b06fbc at lookup+0x10c #4 0xffffffff80b06aec at namei+0x52c #5 0xffffffff80b1f28e at kern_statat+0xae #6 0xffffffff80b1fa9f at sys_fstatat+0x2f #7 0xffffffff80ed3aeb at amd64_syscall+0x2eb #8 0xffffffff80eb3b5b at Xfast_syscall+0xfb fileid 39 fsid 0x3a3a000001 vnode 0xfffff80032bb39c0: tag ufs, type VREG usecount 1, writecount 1, refcount 6718 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80201410c30 ref 0 pages 53727 cleanbuf 6713 dirtybuf 3 lock type ufs: EXCL by thread 0xfffff801bc603560 (pid 88268, rw, tid 100303) #0 0xffffffff80a23f26 at __lockmgr_args+0x1376 #1 0xffffffff80d23482 at ffs_lock+0x92 #2 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #3 0xffffffff80b2547a at _vn_lock+0xba #4 0xffffffff80b27ee2 at vn_write+0x112 #5 0xffffffff80b28162 at vn_io_fault_doio+0x22 #6 0xffffffff80b25b5b at vn_io_fault1+0x1bb #7 0xffffffff80b23d78 at vn_io_fault+0x188 #8 0xffffffff80ab20cb at dofilewrite+0xab #9 0xffffffff80ab1cb8 at kern_writev+0x68 #10 0xffffffff80ab1c3b at sys_write+0x8b #11 0xffffffff80ed3aeb at amd64_syscall+0x2eb #12 0xffffffff80eb3b5b at Xfast_syscall+0xfb ino 65539, on dev md10a vnode 0xfffff8058a9614e0: tag ufs, type VREG usecount 1, writecount 1, refcount 5682 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff803052f5d20 ref 0 pages 45432 cleanbuf 5675 dirtybuf 5 lock type ufs: EXCL by thread 0xfffff80517c66560 (pid 88266, rw, tid 103429) #0 0xffffffff80a23f26 at __lockmgr_args+0x1376 #1 0xffffffff80d23482 at ffs_lock+0x92 #2 0xffffffff8105002a at VOP_LOCK1_APV+0xfa #3 0xffffffff80b2547a at _vn_lock+0xba #4 0xffffffff80b27ee2 at vn_write+0x112 #5 0xffffffff80b28162 at vn_io_fault_doio+0x22 #6 0xffffffff80b25b5b at vn_io_fault1+0x1bb #7 0xffffffff80b23d78 at vn_io_fault+0x188 #8 0xffffffff80ab20cb at dofilewrite+0xab #9 0xffffffff80ab1cb8 at kern_writev+0x68 #10 0xffffffff80ab1c3b at sys_write+0x8b #11 0xffffffff80ed3aeb at amd64_syscall+0x2eb #12 0xffffffff80eb3b5b at Xfast_syscall+0xfb ino 65540, on dev md10a db> show mount 0xfffff8000f97d000 192.168.5.1:/zoo/mercat1 on / (nfs) 0xfffff8000f97e000 devfs on /dev (devfs) 0xfffff800143bf000 /dev/ada0p2 on /work (ufs) 0xfffff800149ef000 192.168.5.1:/zoo/pho on /zoo/pho (nfs) 0xfffff80014ac7000 192.168.5.1:/tank/users/pho on /tank/users/pho (nfs) 0xfffff801bc4a2000 procfs on /proc (procfs) 0xfffff8013dcbb000 /dev/md10a on /mnt (ufs) More info: show mount db> show page vm_cnt.v_free_count: 2 vm_cnt.v_inactive_count: 163 vm_cnt.v_active_count: 7650898 vm_cnt.v_laundry_count: 16719 vm_cnt.v_wire_count: 463184 vm_cnt.v_free_reserved: 10653 vm_cnt.v_free_min: 51306 vm_cnt.v_free_target: 173265 vm_cnt.v_inactive_target: 259897 db> ps pid ppid pgrp uid state wmesg wchan cmd 88269 88256 88126 2007 R+ CPU 6 swap 88268 88263 88126 2007 D+ vmwait 0xffffffff819939ac rw 88267 88256 88126 2007 D+ pfault 0xffffffff819939ac swap 88266 88263 88126 2007 D+ vmwait 0xffffffff819939ac rw 88265 88256 88126 2007 R+ CPU 10 swap 88264 88256 88126 2007 D+ pfault 0xffffffff819939ac swap 88263 88254 88126 2007 S+ wait 0xfffff8044bd06a80 rw 88262 88256 88126 2007 R+ CPU 1 swap 88261 88256 88126 2007 R+ swap 88260 88256 88126 2007 R+ CPU 0 swap 88259 88256 88126 2007 R+ CPU 3 swap 88258 88256 88126 2007 R+ CPU 7 swap 88257 88256 88126 2007 R+ CPU 11 swap 88256 88255 88126 2007 S+ wait 0xfffff800078d5a80 swap 88255 88128 88126 2007 S+ nanslp 0xffffffff81c1a4b5 swap 88254 88128 88126 2007 S+ nanslp 0xffffffff81c1a4b0 rw 88237 0 0 0 SL - 0xffffffff81df88c0 [newnfs 0] 88128 88127 88126 2007 S+ wait 0xfffff8025dad0a80 run 88127 88126 88126 2007 SW+ wait 0xfffff8004695b000 run 88126 88125 88126 2007 S+ nanslp 0xffffffff81c1a4b5 run 88125 88110 832 0 SW+ wait 0xfffff8044bd06000 su 88120 0 0 0 DL mdwait 0xfffff80145acd800 [md10] 88110 88108 832 0 SW+ wait 0xfffff80443031540 sh 88109 832 832 0 S+ piperd 0xfffff803cdd805f0 tee 88108 832 832 0 SW+ wait 0xfffff80046b6c000 sh 33902 0 0 0 SL pipewr 0xfffff803120fe2f8 [hwpmc: proc(33901)] 33901 33899 33901 0 D+ pfault 0xffffffff819939ac pmcstat 33899 33897 33899 0 SW+ wait 0xfffff80046514000 bash 33897 33896 33897 0 SW+ pause 0xfffff8001429a5e8 csh 33896 33895 33896 2006 SW+ wait 0xfffff8039c6cd000 su 33895 33894 33895 2006 SWs+ wait 0xfffff8026b91ba80 bash 33894 33891 33891 2006 S select 0xfffff803e1ed6140 sshd 33891 762 33891 0 Ss select 0xfffff80489c9ee40 sshd 33626 33624 33626 0 S+ ttyin 0xfffff80007afacb0 bash 33624 33623 33624 0 SW+ pause 0xfffff801b44925e8 csh 33623 33622 33623 2006 SW+ wait 0xfffff801bcb4aa80 su 33622 33621 33622 2006 SWs+ wait 0xfffff801b448da80 bash 33621 33618 33618 2006 S select 0xfffff8064a524440 sshd 33618 762 33618 0 Ss select 0xfffff80535ac6cc0 sshd 832 831 832 0 SW+ wait 0xfffff800143d4540 sh 831 829 831 0 SW+ wait 0xfffff80046b6ca80 bash 829 828 829 0 SW+ pause 0xfffff800145fa0a8 csh 828 827 828 2006 SW+ wait 0xfffff80014299000 su 827 826 827 2006 SWs+ wait 0xfffff80014299540 bash 826 823 823 2006 S select 0xfffff8000774d0c0 sshd 823 762 823 0 Ss select 0xfffff8000f3454c0 sshd 818 811 818 0 S+ ttyin 0xfffff80007e224b0 csh 811 1 811 0 SWs+ wait 0xfffff80046b6d000 login 810 1 810 0 Ss+ ttyin 0xfffff80007e230b0 getty 809 1 809 0 Ss+ ttyin 0xfffff8000f3638b0 getty 808 1 808 0 Ss+ ttyin 0xfffff8000f363cb0 getty 807 1 807 0 Ss+ ttyin 0xfffff8000f3640b0 getty 806 1 806 0 Ss+ ttyin 0xfffff8000f3644b0 getty 805 1 805 0 Ss+ ttyin 0xfffff8000f3648b0 getty 804 1 804 0 Ss+ ttyin 0xfffff8000f364cb0 getty 803 1 803 0 Ss+ ttyin 0xfffff8000f3650b0 getty 802 1 802 0 Ss+ ttyin 0xfffff8000f3654b0 getty 766 1 766 0 Ds nfsreq 0xfffff8000f4fac00 cron 762 1 762 0 Ss select 0xfffff80007d82cc0 sshd 691 1 691 0 Ss rpcsvc 0xfffff8000f9910a0 NLM: master 687 1 687 0 Ss select 0xfffff80007d82d40 rpc.statd 662 1 662 0 Ss select 0xfffff8000774d140 casperd 661 1 661 0 Ss select 0xfffff8000774d1c0 casperd 554 1 554 0 Ss select 0xfffff8000f4c7640 rpcbind 549 1 549 0 Ss select 0xfffff80007d82e40 syslogd 462 1 462 0 Ss select 0xfffff8000f2cd140 devd 23 0 0 0 DL vlruwt 0xfffff8000f1ae540 [vnlru] 22 0 0 0 DL syncer 0xffffffff81db3980 [syncer] 21 0 0 0 DL - 0xffffffff81db2dd8 [bufspacedaemon] 20 0 0 0 DL (threaded) [bufdaemon] 100105 D psleep 0xffffffff81db2104 [bufdaemon] 100127 D sdflush 0xfffff800142ce0e8 [/work worker] 100592 D sdflush 0xfffff803114886e8 [/mnt worker] 19 0 0 0 DL psleep 0xffffffff81dbd238 [vmdaemon] 18 0 0 0 RL (threaded) [pagedaemon] 100103 Run CPU 9 [pagedaemon] 100109 D launds 0xffffffff81dbd248 [laundry: dom0] 100110 D umarcl 0xffffffff81dbcbe8 [uma] 17 0 0 0 DL idle 0xfffff8000f0a9000 [enc_daemon1] 16 0 0 0 DL vmwait 0xffffffff819939ac [enc_daemon0] 15 0 0 0 DL - 0xffffffff81aed4d8 [rand_harvestq] 9 0 0 0 DL waiting 0xffffffff81e2a790 [sctp_iterator] 8 0 0 0 DL - 0xffffffff81db18f4 [soaiod4] 7 0 0 0 DL - 0xffffffff81db18f4 [soaiod3] 6 0 0 0 DL - 0xffffffff81db18f4 [soaiod2] 5 0 0 0 DL - 0xffffffff81db18f4 [soaiod1] 14 0 0 0 DL (threaded) [usb] 100071 D - 0xfffffe00012f1460 [usbus0] 100072 D - 0xfffffe00012f14b8 [usbus0] 100073 D - 0xfffffe00012f1510 [usbus0] 100074 D - 0xfffffe00012f1568 [usbus0] 100075 D - 0xfffffe00012f15c0 [usbus0] 100077 D - 0xfffffe000114dd10 [usbus1] 100078 D - 0xfffffe000114dd68 [usbus1] 100079 D - 0xfffffe000114ddc0 [usbus1] 100080 D - 0xfffffe000114de18 [usbus1] 100081 D - 0xfffffe000114de70 [usbus1] 100082 D - 0xfffffe00012f9d10 [usbus2] 100083 D - 0xfffffe00012f9d68 [usbus2] 100084 D - 0xfffffe00012f9dc0 [usbus2] 100085 D - 0xfffffe00012f9e18 [usbus2] 100086 D - 0xfffffe00012f9e70 [usbus2] 4 0 0 0 DL (threaded) [cam] 100066 D - 0xffffffff819c0880 [doneq0] 100067 D - 0xffffffff819c0980 [doneq1] 100068 D - 0xffffffff819c0a80 [doneq2] 100100 D - 0xffffffff819c06c8 [scanner] 3 0 0 0 DL crypto_ 0xffffffff81dbb010 [crypto returns] 2 0 0 0 DL crypto_ 0xffffffff81dbaec8 [crypto] 13 0 0 0 DL (threaded) [geom] 100060 D - 0xffffffff81e02970 [g_event] 100061 D - 0xffffffff81e02978 [g_up] 100062 D - 0xffffffff81e02980 [g_down] 12 0 0 0 WL (threaded) [intr] 100015 I [swi3: vm] 100016 I [swi4: clock (0)] 100017 I [swi4: clock (1)] 100018 I [swi4: clock (2)] 100019 I [swi4: clock (3)] 100020 I [swi4: clock (4)] 100021 I [swi4: clock (5)] 100022 I [swi4: clock (6)] 100023 I [swi4: clock (7)] 100024 I [swi4: clock (8)] 100025 I [swi4: clock (9)] 100026 I [swi4: clock (10)] 100027 I [swi4: clock (11)] 100028 I [swi1: netisr 0] 100054 I [swi6: Giant taskq] 100057 I [swi5: fast taskq] 100059 I [swi6: task queue] 100069 I [irq264: ahci0] 100070 I [irq19: xhci0] 100076 I [irq18: ehci0 ehci1] 100087 I [irq279: ahci1] 100088 I [swi0: uart uart] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 100007 CanRun [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 CanRun [idle: cpu6] 100010 CanRun [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 CanRun [idle: cpu9] 100013 CanRun [idle: cpu10] 100014 CanRun [idle: cpu11] 1 0 1 0 SLs wait 0xfffff8000778b540 [init] 10 0 0 0 DL audit_w 0xffffffff81e2fe58 [audit] 0 0 0 0 RLs (threaded) [kernel] 100000 D vmwait 0xffffffff819939ac [swapper] 100029 D - 0xfffff800077e1000 [softirq_0] 100030 D - 0xfffff800077e0e00 [softirq_1] 100031 D - 0xfffff800077e0d00 [softirq_2] 100032 D - 0xfffff800077e0c00 [softirq_3] 100033 D - 0xfffff800077e0b00 [softirq_4] 100034 D - 0xfffff800077e0a00 [softirq_5] 100035 D - 0xfffff800077e0900 [softirq_6] 100036 D - 0xfffff800077e0800 [softirq_7] 100037 D - 0xfffff800077e0700 [softirq_8] 100038 D - 0xfffff800077e0600 [softirq_9] 100039 D - 0xfffff800077e0500 [softirq_10] 100040 D - 0xfffff800077e0400 [softirq_11] 100041 D - 0xfffff800077e0300 [if_io_tqg_0] 100042 D - 0xfffff80007857100 [if_io_tqg_1] 100043 Run CPU 2 [if_io_tqg_2] 100044 D - 0xfffff80007856e00 [if_io_tqg_3] 100045 D - 0xfffff80007856d00 [if_io_tqg_4] 100046 D - 0xfffff80007856c00 [if_io_tqg_5] 100047 D - 0xfffff80007856b00 [if_io_tqg_6] 100048 D - 0xfffff80007856a00 [if_io_tqg_7] 100049 D - 0xfffff80007856900 [if_io_tqg_8] 100050 D - 0xfffff80007856800 [if_io_tqg_9] 100051 D - 0xfffff80007856700 [if_io_tqg_10] 100052 D - 0xfffff80007856600 [if_io_tqg_11] 100053 D - 0xfffff80007898200 [if_config_tqg_0] 100055 Run CPU 4 [thread taskq] 100056 D - 0xfffff80007897e00 [aiod_kick taskq] 100058 D - 0xfffff80007897c00 [kqueue_ctx taskq] 100063 D - 0xfffff80007897400 [firmware taskq] 100089 D - 0xfffff8000a0d9300 [mca taskq] 100095 D - 0xfffff8000f1fa400 [acpi_task_0] 100096 D - 0xfffff8000f1fa400 [acpi_task_1] 100097 D - 0xfffff8000f1fa400 [acpi_task_2] 100099 D - 0xfffff80007896a00 [CAM taskq] db> allt Tracing command swap pid 88269 tid 100393 td 0xfffff805aeeca000 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15e4cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15e4d10 trap() at trap+0x49/frame 0xfffffe07c15e4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15e4f20 --- trap 0x13, rip = 0x401d5e, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command rw pid 88268 tid 100303 td 0xfffff801bc603560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d09ef60 mi_switch() at mi_switch+0x195/frame 0xfffffe085d09efa0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d09efe0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085d09f010 _sleep() at _sleep+0x30b/frame 0xfffffe085d09f0b0 vm_wait() at vm_wait+0xc5/frame 0xfffffe085d09f0d0 vm_page_grab_pages() at vm_page_grab_pages+0x1b4/frame 0xfffffe085d09f150 allocbuf() at allocbuf+0x21d/frame 0xfffffe085d09f1c0 getblk() at getblk+0x7cc/frame 0xfffffe085d09f270 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1c06/frame 0xfffffe085d09f440 ffs_write() at ffs_write+0x218/frame 0xfffffe085d09f4e0 VOP_WRITE_APV() at VOP_WRITE_APV+0x17f/frame 0xfffffe085d09f600 vn_write() at vn_write+0x22d/frame 0xfffffe085d09f680 vn_io_fault_doio() at vn_io_fault_doio+0x22/frame 0xfffffe085d09f6c0 vn_io_fault1() at vn_io_fault1+0x1bb/frame 0xfffffe085d09f810 vn_io_fault() at vn_io_fault+0x188/frame 0xfffffe085d09f890 dofilewrite() at dofilewrite+0xab/frame 0xfffffe085d09f8e0 kern_writev() at kern_writev+0x68/frame 0xfffffe085d09f930 sys_write() at sys_write+0x8b/frame 0xfffffe085d09f980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d09fab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d09fab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800bc544a, rsp = 0x7fffffffd5e8, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88267 tid 100122 td 0xfffff80014286000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd13510 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd13550 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd13590 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cd135c0 _sleep() at _sleep+0x30b/frame 0xfffffe085cd13660 vm_waitpfault() at vm_waitpfault+0x74/frame 0xfffffe085cd13680 vm_fault_hold() at vm_fault_hold+0x1213/frame 0xfffffe085cd137e0 vm_fault() at vm_fault+0x77/frame 0xfffffe085cd13820 trap_pfault() at trap_pfault+0x11a/frame 0xfffffe085cd138a0 trap() at trap+0x58f/frame 0xfffffe085cd13ab0 calltrap() at calltrap+0x8/frame 0xfffffe085cd13ab0 --- trap 0xc, rip = 0x800bc59d4, rsp = 0x7fffffffe328, rbp = 0x7fffffffe370 --- Tracing command rw pid 88266 tid 103429 td 0xfffff80517c66560 sched_switch() at sched_switch+0x486/frame 0xfffffe085f9d9f60 mi_switch() at mi_switch+0x195/frame 0xfffffe085f9d9fa0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085f9d9fe0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085f9da010 _sleep() at _sleep+0x30b/frame 0xfffffe085f9da0b0 vm_wait() at vm_wait+0xc5/frame 0xfffffe085f9da0d0 vm_page_grab_pages() at vm_page_grab_pages+0x1b4/frame 0xfffffe085f9da150 allocbuf() at allocbuf+0x21d/frame 0xfffffe085f9da1c0 getblk() at getblk+0x7cc/frame 0xfffffe085f9da270 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1c06/frame 0xfffffe085f9da440 ffs_write() at ffs_write+0x218/frame 0xfffffe085f9da4e0 VOP_WRITE_APV() at VOP_WRITE_APV+0x17f/frame 0xfffffe085f9da600 vn_write() at vn_write+0x22d/frame 0xfffffe085f9da680 vn_io_fault_doio() at vn_io_fault_doio+0x22/frame 0xfffffe085f9da6c0 vn_io_fault1() at vn_io_fault1+0x1bb/frame 0xfffffe085f9da810 vn_io_fault() at vn_io_fault+0x188/frame 0xfffffe085f9da890 dofilewrite() at dofilewrite+0xab/frame 0xfffffe085f9da8e0 kern_writev() at kern_writev+0x68/frame 0xfffffe085f9da930 sys_write() at sys_write+0x8b/frame 0xfffffe085f9da980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085f9daab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085f9daab0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800bc544a, rsp = 0x7fffffffd5e8, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88265 tid 100182 td 0xfffff80046a63560 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe0839130cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe0839130d10 trap() at trap+0x49/frame 0xfffffe0839130f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0839130f20 --- trap 0x13, rip = 0x401d5e, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88264 tid 100148 td 0xfffff800144d1560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd95510 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd95550 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd95590 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cd955c0 _sleep() at _sleep+0x30b/frame 0xfffffe085cd95660 vm_waitpfault() at vm_waitpfault+0x74/frame 0xfffffe085cd95680 vm_fault_hold() at vm_fault_hold+0x1213/frame 0xfffffe085cd957e0 vm_fault() at vm_fault+0x77/frame 0xfffffe085cd95820 trap_pfault() at trap_pfault+0x11a/frame 0xfffffe085cd958a0 trap() at trap+0x58f/frame 0xfffffe085cd95ab0 calltrap() at calltrap+0x8/frame 0xfffffe085cd95ab0 --- trap 0xc, rip = 0x800bc59d4, rsp = 0x7fffffffe328, rbp = 0x7fffffffe370 --- Tracing command rw pid 88263 tid 102342 td 0xfffff8015787b000 sched_switch() at sched_switch+0x486/frame 0xfffffe085e48b520 mi_switch() at mi_switch+0x195/frame 0xfffffe085e48b560 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085e48b5a0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085e48b610 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085e48b640 _sleep() at _sleep+0x301/frame 0xfffffe085e48b6e0 kern_wait6() at kern_wait6+0x467/frame 0xfffffe085e48b790 sys_wait4() at sys_wait4+0x7a/frame 0xfffffe085e48b980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085e48bab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085e48bab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5387a, rsp = 0x7fffffffe6a8, rbp = 0x7fffffffe7c0 --- Tracing command swap pid 88262 tid 100137 td 0xfffff800144d2560 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15bccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15bcd10 trap() at trap+0x49/frame 0xfffffe07c15bcf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15bcf20 --- trap 0x13, rip = 0x401d57, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88261 tid 100177 td 0xfffff80046659000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce27a10 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce27a50 critical_exit() at critical_exit+0x8b/frame 0xfffffe085ce27a70 ipi_bitmap_handler() at ipi_bitmap_handler+0x79/frame 0xfffffe085ce27ab0 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0xa9/frame 0xfffffe085ce27ab0 --- interrupt, rip = 0x401d59, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88260 tid 100170 td 0xfffff80046659560 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81dc4fb0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff81dc4fd0 trap() at trap+0x49/frame 0xffffffff81dc51e0 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81dc51e0 --- trap 0x13, rip = 0x401d57, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88259 tid 100356 td 0xfffff8043a936000 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15cccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15ccd10 trap() at trap+0x49/frame 0xfffffe07c15ccf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15ccf20 --- trap 0x13, rip = 0x401d57, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88258 tid 100355 td 0xfffff8043a936560 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15eccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15ecd10 trap() at trap+0x49/frame 0xfffffe07c15ecf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15ecf20 --- trap 0x13, rip = 0xffffffff810447e6, rsp = 0xfffffe085d1a3ab0, rbp = 0xfffffe085d1a3ab0 --- invlpg_handler() at invlpg_handler+0x16/frame 0xfffffe085d1a3ab0 Xinvlpg() at Xinvlpg+0xa4/frame 0x7fffffffe6a0 Tracing command swap pid 88257 tid 101926 td 0xfffff8082eb09560 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe0839138cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe0839138d10 trap() at trap+0x49/frame 0xfffffe0839138f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0839138f20 --- trap 0x13, rip = 0x401d59, rsp = 0x7fffffffe680, rbp = 0x7fffffffe6a0 --- Tracing command swap pid 88256 tid 100452 td 0xfffff8020416d560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d371520 mi_switch() at mi_switch+0x195/frame 0xfffffe085d371560 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d3715a0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d371610 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d371640 _sleep() at _sleep+0x301/frame 0xfffffe085d3716e0 kern_wait6() at kern_wait6+0x467/frame 0xfffffe085d371790 sys_wait4() at sys_wait4+0x7a/frame 0xfffffe085d371980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d371ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d371ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5387a, rsp = 0x7fffffffe6a8, rbp = 0x7fffffffe7c0 --- Tracing command swap pid 88255 tid 103678 td 0xfffff804f16b5000 sched_switch() at sched_switch+0x486/frame 0xfffffe085febd6d0 mi_switch() at mi_switch+0x195/frame 0xfffffe085febd710 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085febd750 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085febd7c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085febd7f0 _sleep() at _sleep+0x2e0/frame 0xfffffe085febd890 kern_clock_nanosleep() at kern_clock_nanosleep+0x1af/frame 0xfffffe085febd920 user_clock_nanosleep() at user_clock_nanosleep+0x72/frame 0xfffffe085febd980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085febdab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085febdab0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b539fa, rsp = 0x7fffffffe658, rbp = 0x7fffffffe6a0 --- Tracing command rw pid 88254 tid 100128 td 0xfffff800077e3560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd316d0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd31710 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd31750 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd317c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085cd317f0 _sleep() at _sleep+0x2e0/frame 0xfffffe085cd31890 kern_clock_nanosleep() at kern_clock_nanosleep+0x1af/frame 0xfffffe085cd31920 user_clock_nanosleep() at user_clock_nanosleep+0x72/frame 0xfffffe085cd31980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd31ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd31ab0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b539fa, rsp = 0x7fffffffe658, rbp = 0x7fffffffe6a0 --- Tracing command newnfs 0 pid 88237 tid 100505 td 0xfffff804b3a0d000 sched_switch() at sched_switch+0x486/frame 0xfffffe085f804870 mi_switch() at mi_switch+0x195/frame 0xfffffe085f8048b0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085f8048f0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085f804960 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085f804990 _sleep() at _sleep+0x2e0/frame 0xfffffe085f804a30 nfssvc_iod() at nfssvc_iod+0x101/frame 0xfffffe085f804a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085f804ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085f804ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command run pid 88128 tid 103427 td 0xfffff80030c8d560 sched_switch() at sched_switch+0x486/frame 0xfffffe085f9d0520 mi_switch() at mi_switch+0x195/frame 0xfffffe085f9d0560 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085f9d05a0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085f9d0610 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085f9d0640 _sleep() at _sleep+0x301/frame 0xfffffe085f9d06e0 kern_wait6() at kern_wait6+0x467/frame 0xfffffe085f9d0790 sys_wait4() at sys_wait4+0x7a/frame 0xfffffe085f9d0980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085f9d0ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085f9d0ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5287a, rsp = 0x7fffffffe138, rbp = 0x7fffffffe660 --- Tracing command run pid 88127 tid 100204 td 0xfffff80046a62560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command run pid 88126 tid 103674 td 0xfffff804f16b5560 sched_switch() at sched_switch+0x486/frame 0xfffffe085fea96d0 mi_switch() at mi_switch+0x195/frame 0xfffffe085fea9710 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085fea9750 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085fea97c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085fea97f0 _sleep() at _sleep+0x2e0/frame 0xfffffe085fea9890 kern_clock_nanosleep() at kern_clock_nanosleep+0x1af/frame 0xfffffe085fea9920 user_clock_nanosleep() at user_clock_nanosleep+0x72/frame 0xfffffe085fea9980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085fea9ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085fea9ab0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b529fa, rsp = 0x7fffffffe618, rbp = 0x7fffffffe660 --- Tracing command su pid 88125 tid 102344 td 0xfffff8021746d000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command md10 pid 88120 tid 103364 td 0xfffff804b1e1a000 sched_switch() at sched_switch+0x486/frame 0xfffffe085f8958d0 mi_switch() at mi_switch+0x195/frame 0xfffffe085f895910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085f895950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085f895980 _sleep() at _sleep+0x30b/frame 0xfffffe085f895a20 md_kthread() at md_kthread+0xe5/frame 0xfffffe085f895a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085f895ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085f895ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 88110 tid 101927 td 0xfffff8082eb09000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command tee pid 88109 tid 101698 td 0xfffff801ce0c7560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d7e9670 mi_switch() at mi_switch+0x195/frame 0xfffffe085d7e96b0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d7e96f0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d7e9760 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d7e9790 _sleep() at _sleep+0x301/frame 0xfffffe085d7e9830 pipe_read() at pipe_read+0x296/frame 0xfffffe085d7e9890 dofileread() at dofileread+0xb2/frame 0xfffffe085d7e98e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085d7e9930 sys_read() at sys_read+0x8b/frame 0xfffffe085d7e9980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d7e9ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d7e9ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8009ae48a, rsp = 0x7fffffffe718, rbp = 0x7fffffffe760 --- Tracing command sh pid 88108 tid 100206 td 0xfffff80046df4000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command hwpmc: proc(33901) pid 33902 tid 100389 td 0xfffff80228a93560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d24a750 mi_switch() at mi_switch+0x195/frame 0xfffffe085d24a790 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d24a7d0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d24a840 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d24a870 _sleep() at _sleep+0x301/frame 0xfffffe085d24a910 pipe_write() at pipe_write+0x809/frame 0xfffffe085d24a9c0 pmclog_loop() at pmclog_loop+0x203/frame 0xfffffe085d24aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe085d24aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085d24aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pmcstat pid 33901 tid 100167 td 0xfffff8000ffad560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cdf4510 mi_switch() at mi_switch+0x195/frame 0xfffffe085cdf4550 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cdf4590 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cdf45c0 _sleep() at _sleep+0x30b/frame 0xfffffe085cdf4660 vm_waitpfault() at vm_waitpfault+0x74/frame 0xfffffe085cdf4680 vm_fault_hold() at vm_fault_hold+0x1213/frame 0xfffffe085cdf47e0 vm_fault() at vm_fault+0x77/frame 0xfffffe085cdf4820 trap_pfault() at trap_pfault+0x11a/frame 0xfffffe085cdf48a0 trap() at trap+0x58f/frame 0xfffffe085cdf4ab0 calltrap() at calltrap+0x8/frame 0xfffffe085cdf4ab0 --- trap 0xc, rip = 0x8015159d4, rsp = 0x7fffffffe008, rbp = 0x7fffffffe1f0 --- Tracing command bash pid 33899 tid 100396 td 0xfffff801f8570560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command csh pid 33897 tid 100133 td 0xfffff80014285560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 33896 tid 100360 td 0xfffff8039c4ec000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 33895 tid 100401 td 0xfffff8020416d000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 33894 tid 100357 td 0xfffff8043a407560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d1ad560 mi_switch() at mi_switch+0x195/frame 0xfffffe085d1ad5a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d1ad5e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d1ad650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d1ad680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085d1ad6f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085d1ad730 kern_select() at kern_select+0x94f/frame 0xfffffe085d1ad940 sys_select() at sys_select+0x54/frame 0xfffffe085d1ad980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d1adab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d1adab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8021f846a, rsp = 0x7fffffffa608, rbp = 0x7fffffffa640 --- Tracing command sshd pid 33891 tid 100351 td 0xfffff8043a497560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d18f5f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085d18f630 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d18f670 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d18f6e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d18f710 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085d18f780 seltdwait() at seltdwait+0x8b/frame 0xfffffe085d18f7c0 kern_poll() at kern_poll+0x440/frame 0xfffffe085d18f960 sys_poll() at sys_poll+0x61/frame 0xfffffe085d18f980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d18fab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d18fab0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80219824a, rsp = 0x7fffffffe5e8, rbp = 0x7fffffffe620 --- Tracing command bash pid 33626 tid 100266 td 0xfffff801bcb4e560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cfe45e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cfe4620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cfe4660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cfe46d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cfe4700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cfe4770 tty_wait() at tty_wait+0x42/frame 0xfffffe085cfe47a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085cfe4800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085cfe4830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085cfe4890 dofileread() at dofileread+0xb2/frame 0xfffffe085cfe48e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085cfe4930 sys_read() at sys_read+0x8b/frame 0xfffffe085cfe4980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cfe4ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cfe4ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800efd48a, rsp = 0x7fffffffd6f8, rbp = 0x7fffffffd720 --- Tracing command csh pid 33624 tid 100438 td 0xfffff802248fd560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 33623 tid 100265 td 0xfffff801bcb4f000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 33622 tid 100440 td 0xfffff80204120560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 33621 tid 100304 td 0xfffff801bc603000 sched_switch() at sched_switch+0x486/frame 0xfffffe085d0a4560 mi_switch() at mi_switch+0x195/frame 0xfffffe085d0a45a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d0a45e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085d0a4650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085d0a4680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085d0a46f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085d0a4730 kern_select() at kern_select+0x94f/frame 0xfffffe085d0a4940 sys_select() at sys_select+0x54/frame 0xfffffe085d0a4980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085d0a4ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d0a4ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8021f846a, rsp = 0x7fffffffa608, rbp = 0x7fffffffa640 --- Tracing command sshd pid 33618 tid 100145 td 0xfffff8001458c000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd865f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd86630 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd86670 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd866e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cd86710 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cd86780 seltdwait() at seltdwait+0x8b/frame 0xfffffe085cd867c0 kern_poll() at kern_poll+0x440/frame 0xfffffe085cd86960 sys_poll() at sys_poll+0x61/frame 0xfffffe085cd86980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd86ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd86ab0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80219824a, rsp = 0x7fffffffe5e8, rbp = 0x7fffffffe620 --- Tracing command sh pid 832 tid 100126 td 0xfffff8000f95a000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 831 tid 100198 td 0xfffff80046a77000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command csh pid 829 tid 100181 td 0xfffff8001458a560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 828 tid 100197 td 0xfffff80014283560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 827 tid 100196 td 0xfffff80014284000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 826 tid 100160 td 0xfffff80014393560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cdd1560 mi_switch() at mi_switch+0x195/frame 0xfffffe085cdd15a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cdd15e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cdd1650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cdd1680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cdd16f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085cdd1730 kern_select() at kern_select+0x94f/frame 0xfffffe085cdd1940 sys_select() at sys_select+0x54/frame 0xfffffe085cdd1980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cdd1ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cdd1ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8021f846a, rsp = 0x7fffffffa608, rbp = 0x7fffffffa640 --- Tracing command sshd pid 823 tid 100171 td 0xfffff80014285000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce095f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce09630 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce09670 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce096e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce09710 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce09780 seltdwait() at seltdwait+0x8b/frame 0xfffffe085ce097c0 kern_poll() at kern_poll+0x440/frame 0xfffffe085ce09960 sys_poll() at sys_poll+0x61/frame 0xfffffe085ce09980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce09ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce09ab0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80219824a, rsp = 0x7fffffffe5e8, rbp = 0x7fffffffe620 --- Tracing command csh pid 818 tid 100119 td 0xfffff8000f0af560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd045e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd04620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd04660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd046d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cd04700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cd04770 tty_wait() at tty_wait+0x42/frame 0xfffffe085cd047a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085cd04800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085cd04830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085cd04890 dofileread() at dofileread+0xb2/frame 0xfffffe085cd048e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085cd04930 sys_read() at sys_read+0x8b/frame 0xfffffe085cd04980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd04ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd04ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800e8d48a, rsp = 0x7fffffffe248, rbp = 0x7fffffffe270 --- Tracing command login pid 811 tid 100195 td 0xfffff80046a77560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c7819eb0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c7819f60 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c7819f70 trap() at trap+0x63/frame 0xfffffe07c781a180 calltrap() at calltrap+0x8/frame 0xfffffe07c781a180 --- trap 0xc, rip = 0xffffffff80eb2a78, rsp = 0xfffffe07c781a250, rbp = 0xfffffe07c781a260 --- db_trace_thread() at db_trace_thread+0x28/frame 0xfffffe07c781a260 _db_stack_trace_all() at _db_stack_trace_all+0xe4/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command getty pid 810 tid 100194 td 0xfffff80046a6a000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce7c5e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce7c620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce7c660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce7c6d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce7c700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce7c770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce7c7a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce7c800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce7c830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce7c890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce7c8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce7c930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce7c980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce7cab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce7cab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 809 tid 100193 td 0xfffff80046a6a560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce775e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce77620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce77660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce776d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce77700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce77770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce777a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce77800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce77830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce77890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce778e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce77930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce77980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce77ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce77ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 808 tid 100192 td 0xfffff80046a6b000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce725e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce72620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce72660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce726d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce72700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce72770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce727a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce72800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce72830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce72890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce728e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce72930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce72980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce72ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce72ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 807 tid 100191 td 0xfffff80046a6b560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce6d5e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce6d620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce6d660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce6d6d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce6d700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce6d770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce6d7a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce6d800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce6d830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce6d890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce6d8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce6d930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce6d980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce6dab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce6dab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 806 tid 100190 td 0xfffff80046956000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce685e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce68620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce68660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce686d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce68700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce68770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce687a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce68800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce68830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce68890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce688e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce68930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce68980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce68ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce68ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 805 tid 100189 td 0xfffff80046956560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce635e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce63620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce63660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce636d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce63700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce63770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce637a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce63800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce63830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce63890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce638e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce63930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce63980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce63ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce63ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 804 tid 100188 td 0xfffff80046957000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce5e5e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce5e620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce5e660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce5e6d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce5e700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce5e770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce5e7a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce5e800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce5e830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce5e890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce5e8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce5e930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce5e980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce5eab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce5eab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 803 tid 100180 td 0xfffff8000a048560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce365e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce36620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce36660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce366d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce36700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce36770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ce367a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ce36800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ce36830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ce36890 dofileread() at dofileread+0xb2/frame 0xfffffe085ce368e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ce36930 sys_read() at sys_read+0x8b/frame 0xfffffe085ce36980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce36ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce36ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command getty pid 802 tid 100111 td 0xfffff8000f0c2000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ccdc5e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ccdc620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ccdc660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ccdc6d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ccdc700 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ccdc770 tty_wait() at tty_wait+0x42/frame 0xfffffe085ccdc7a0 ttydisc_read() at ttydisc_read+0x34f/frame 0xfffffe085ccdc800 ttydev_read() at ttydev_read+0x6d/frame 0xfffffe085ccdc830 devfs_read_f() at devfs_read_f+0xb3/frame 0xfffffe085ccdc890 dofileread() at dofileread+0xb2/frame 0xfffffe085ccdc8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe085ccdc930 sys_read() at sys_read+0x8b/frame 0xfffffe085ccdc980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ccdcab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ccdcab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc748a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 --- Tracing command cron pid 766 tid 100172 td 0xfffff80014284560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce0d8a0 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce0d8e0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce0d920 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085ce0d950 _sleep() at _sleep+0x2f2/frame 0xfffffe085ce0d9f0 clnt_vc_call() at clnt_vc_call+0x76f/frame 0xfffffe085ce0db50 clnt_reconnect_call() at clnt_reconnect_call+0x4a3/frame 0xfffffe085ce0dc00 newnfs_request() at newnfs_request+0x98b/frame 0xfffffe085ce0dda0 nfscl_request() at nfscl_request+0x72/frame 0xfffffe085ce0ddf0 nfsrpc_accessrpc() at nfsrpc_accessrpc+0x102/frame 0xfffffe085ce0df60 nfs34_access_otw() at nfs34_access_otw+0x50/frame 0xfffffe085ce0e0a0 nfs_access() at nfs_access+0x334/frame 0xfffffe085ce0e140 VOP_ACCESS_APV() at VOP_ACCESS_APV+0xfe/frame 0xfffffe085ce0e170 nfs_lookup() at nfs_lookup+0x157/frame 0xfffffe085ce0e4c0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xfe/frame 0xfffffe085ce0e4f0 lookup() at lookup+0x732/frame 0xfffffe085ce0e590 namei() at namei+0x52c/frame 0xfffffe085ce0e650 kern_statat() at kern_statat+0xae/frame 0xfffffe085ce0e880 sys_fstatat() at sys_fstatat+0x2f/frame 0xfffffe085ce0e980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce0eab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce0eab0 --- syscall (552, FreeBSD ELF64, sys_fstatat), rip = 0x800dd5a2a, rsp = 0x7fffffffe088, rbp = 0x7fffffffe130 --- Tracing command sshd pid 762 tid 100183 td 0xfffff80046a63000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce45560 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce455a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce455e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce45650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ce45680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ce456f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085ce45730 kern_select() at kern_select+0x94f/frame 0xfffffe085ce45940 sys_select() at sys_select+0x54/frame 0xfffffe085ce45980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce45ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce45ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8021f846a, rsp = 0x7fffffffe6b8, rbp = 0x7fffffffe6f0 --- Tracing command rpc.lockd pid 691 tid 100178 td 0xfffff80046658560 sched_switch() at sched_switch+0x486/frame 0xfffffe085ce2c420 mi_switch() at mi_switch+0x195/frame 0xfffffe085ce2c460 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ce2c4a0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ce2c510 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085ce2c540 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1f5/frame 0xfffffe085ce2c5c0 svc_run_internal() at svc_run_internal+0x43c/frame 0xfffffe085ce2c750 svc_run() at svc_run+0x226/frame 0xfffffe085ce2c7d0 sys_nlm_syscall() at sys_nlm_syscall+0x764/frame 0xfffffe085ce2c980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ce2cab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ce2cab0 --- syscall (154, FreeBSD ELF64, sys_nlm_syscall), rip = 0x800a889aa, rsp = 0x7fffffffec48, rbp = 0x7fffffffed00 --- Tracing command rpc.statd pid 687 tid 100146 td 0xfffff800144d2000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd8b550 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd8b590 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd8b5d0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd8b640 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085cd8b670 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1f5/frame 0xfffffe085cd8b6f0 seltdwait() at seltdwait+0x81/frame 0xfffffe085cd8b730 kern_select() at kern_select+0x94f/frame 0xfffffe085cd8b940 sys_select() at sys_select+0x54/frame 0xfffffe085cd8b980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd8bab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd8bab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800bbb46a, rsp = 0x7fffffffead8, rbp = 0x7fffffffec20 --- Tracing command casperd pid 662 tid 100147 td 0xfffff8001402b560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd90560 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd905a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd905e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd90650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cd90680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cd906f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085cd90730 kern_select() at kern_select+0x94f/frame 0xfffffe085cd90940 sys_select() at sys_select+0x54/frame 0xfffffe085cd90980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd90ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd90ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011e446a, rsp = 0x7fffffffe828, rbp = 0x7fffffffed10 --- Tracing command casperd pid 661 tid 100151 td 0xfffff8001458b560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cda4560 mi_switch() at mi_switch+0x195/frame 0xfffffe085cda45a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cda45e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cda4650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085cda4680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085cda46f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085cda4730 kern_select() at kern_select+0x94f/frame 0xfffffe085cda4940 sys_select() at sys_select+0x54/frame 0xfffffe085cda4980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cda4ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cda4ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011e446a, rsp = 0x7fffffffe688, rbp = 0x7fffffffe750 --- Tracing command rpcbind pid 554 tid 100125 td 0xfffff8000f95a560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd225e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd22620 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd22660 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cd226d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085cd22700 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1f5/frame 0xfffffe085cd22780 seltdwait() at seltdwait+0x81/frame 0xfffffe085cd227c0 kern_poll() at kern_poll+0x440/frame 0xfffffe085cd22960 sys_poll() at sys_poll+0x61/frame 0xfffffe085cd22980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cd22ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cd22ab0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800b6024a, rsp = 0x7fffffffca88, rbp = 0x7fffffffec90 --- Tracing command syslogd pid 549 tid 100118 td 0xfffff8000ffb7000 sched_switch() at sched_switch+0x486/frame 0xfffffe085ccff560 mi_switch() at mi_switch+0x195/frame 0xfffffe085ccff5a0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085ccff5e0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085ccff650 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe085ccff680 _cv_wait_sig() at _cv_wait_sig+0x1da/frame 0xfffffe085ccff6f0 seltdwait() at seltdwait+0x8b/frame 0xfffffe085ccff730 kern_select() at kern_select+0x94f/frame 0xfffffe085ccff940 sys_select() at sys_select+0x54/frame 0xfffffe085ccff980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085ccffab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085ccffab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800bcb46a, rsp = 0x7fffffffe6e8, rbp = 0x7fffffffed00 --- Tracing command devd pid 462 tid 100152 td 0xfffff8001458b000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cda9550 mi_switch() at mi_switch+0x195/frame 0xfffffe085cda9590 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cda95d0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe085cda9640 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe085cda9670 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1f5/frame 0xfffffe085cda96f0 seltdwait() at seltdwait+0x81/frame 0xfffffe085cda9730 kern_select() at kern_select+0x94f/frame 0xfffffe085cda9940 sys_select() at sys_select+0x54/frame 0xfffffe085cda9980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe085cda9ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085cda9ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x468b4a, rsp = 0x7fffffffcb88, rbp = 0x7fffffffed20 --- Tracing command vnlru pid 23 tid 100108 td 0xfffff8000f95b560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbf7890 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbf78d0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbf7910 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbf7940 _sleep() at _sleep+0x2f2/frame 0xfffffe085cbf79e0 vnlru_proc() at vnlru_proc+0x184/frame 0xfffffe085cbf7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbf7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbf7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 22 tid 100107 td 0xfffff8000f95c000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbf28b0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbf28f0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbf2930 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbf2960 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x1f5/frame 0xfffffe085cbf29e0 sched_sync() at sched_sync+0x5ee/frame 0xfffffe085cbf2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbf2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbf2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufspacedaemon pid 21 tid 100106 td 0xfffff8000f95c560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbed900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbed940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbed980 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbed9b0 _sleep() at _sleep+0x2f2/frame 0xfffffe085cbeda50 bufspace_daemon() at bufspace_daemon+0x1b7/frame 0xfffffe085cbeda70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbedab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbedab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100105 td 0xfffff8000f95d000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbe88f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbe8930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbe8970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbe89a0 _sleep() at _sleep+0x2f2/frame 0xfffffe085cbe8a40 buf_daemon() at buf_daemon+0x68/frame 0xfffffe085cbe8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbe8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbe8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100127 td 0xfffff8000f959560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cd2c8f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cd2c930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cd2c970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cd2c9a0 _sleep() at _sleep+0x2f2/frame 0xfffffe085cd2ca40 softdep_flush() at softdep_flush+0x182/frame 0xfffffe085cd2ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cd2cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cd2cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 20 tid 100592 td 0xfffff80713a14560 sched_switch() at sched_switch+0x486/frame 0xfffffe085d6ae8f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085d6ae930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085d6ae970 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085d6ae9a0 _sleep() at _sleep+0x2f2/frame 0xfffffe085d6aea40 softdep_flush() at softdep_flush+0x182/frame 0xfffffe085d6aea70 fork_exit() at fork_exit+0x84/frame 0xfffffe085d6aeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085d6aeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 19 tid 100104 td 0xfffff8000f0c6560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbe38b0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbe38f0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbe3930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbe3960 _sleep() at _sleep+0x30b/frame 0xfffffe085cbe3a00 vm_daemon() at vm_daemon+0x96/frame 0xfffffe085cbe3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbe3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbe3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 18 tid 100103 td 0xfffff8000f1c1000 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15fccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15fcd10 trap() at trap+0x49/frame 0xfffffe07c15fcf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15fcf20 --- trap 0x13, rip = 0xffffffff81044652, rsp = 0xfffffe085cbde870, rbp = 0xfffffe085cbde8e0 --- smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x332/frame 0xfffffe085cbde8e0 pmap_invalidate_page() at pmap_invalidate_page+0x189/frame 0xfffffe085cbde940 pmap_ts_referenced() at pmap_ts_referenced+0xa6f/frame 0xfffffe085cbde9f0 vm_pageout() at vm_pageout+0xd1d/frame 0xfffffe085cbdea70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbdeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbdeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 18 tid 100109 td 0xfffff8000f976560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbfc830 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbfc870 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbfc8b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbfc8e0 _sleep() at _sleep+0x30b/frame 0xfffffe085cbfc980 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xc33/frame 0xfffffe085cbfca70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbfcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbfcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 18 tid 100110 td 0xfffff8000f976000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cc048f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cc04930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cc04970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cc049a0 _sleep() at _sleep+0x30b/frame 0xfffffe085cc04a40 uma_reclaim_worker() at uma_reclaim_worker+0x80/frame 0xfffffe085cc04a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cc04ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cc04ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 17 tid 100102 td 0xfffff8000f0c2560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbd98b0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbd98f0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbd9930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbd9960 _sleep() at _sleep+0x30b/frame 0xfffffe085cbd9a00 enc_daemon() at enc_daemon+0x125/frame 0xfffffe085cbd9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbd9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbd9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 16 tid 100101 td 0xfffff8000f0c3000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbd4580 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbd45c0 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbd4600 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbd4630 _sleep() at _sleep+0x30b/frame 0xfffffe085cbd46d0 vm_wait() at vm_wait+0xc5/frame 0xfffffe085cbd46f0 kmem_back() at kmem_back+0x125/frame 0xfffffe085cbd4760 kmem_malloc() at kmem_malloc+0x62/frame 0xfffffe085cbd4790 keg_alloc_slab() at keg_alloc_slab+0xd1/frame 0xfffffe085cbd47e0 keg_fetch_slab() at keg_fetch_slab+0x18d/frame 0xfffffe085cbd4840 zone_fetch_slab() at zone_fetch_slab+0x6e/frame 0xfffffe085cbd4880 zone_import() at zone_import+0x3c/frame 0xfffffe085cbd48f0 zone_alloc_item() at zone_alloc_item+0x36/frame 0xfffffe085cbd4930 uma_zalloc_arg() at uma_zalloc_arg+0x668/frame 0xfffffe085cbd49a0 malloc() at malloc+0x1c2/frame 0xfffffe085cbd4a00 enc_daemon() at enc_daemon+0x1ab/frame 0xfffffe085cbd4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbd4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbd4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 15 tid 100098 td 0xfffff8000f1cd560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbc58d0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbc5910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbc5950 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbc5980 _sleep() at _sleep+0x2f2/frame 0xfffffe085cbc5a20 random_kthread() at random_kthread+0x236/frame 0xfffffe085cbc5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbc5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbc5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sctp_iterator pid 9 tid 100094 td 0xfffff8000f1cf560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbb1900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbb1940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbb1980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbb19b0 _sleep() at _sleep+0x30b/frame 0xfffffe085cbb1a50 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe085cbb1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbb1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbb1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 8 tid 100093 td 0xfffff8000f0c3560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbab8e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbab920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbab960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cbab990 _sleep() at _sleep+0x2f2/frame 0xfffffe085cbaba30 soaio_kproc_loop() at soaio_kproc_loop+0x141/frame 0xfffffe085cbaba70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbabab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbabab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 7 tid 100092 td 0xfffff8000f0c4000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cba68e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cba6920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cba6960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cba6990 _sleep() at _sleep+0x2f2/frame 0xfffffe085cba6a30 soaio_kproc_loop() at soaio_kproc_loop+0x141/frame 0xfffffe085cba6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cba6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cba6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 6 tid 100091 td 0xfffff8000f0c4560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cba18e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cba1920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cba1960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cba1990 _sleep() at _sleep+0x2f2/frame 0xfffffe085cba1a30 soaio_kproc_loop() at soaio_kproc_loop+0x141/frame 0xfffffe085cba1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cba1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cba1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 5 tid 100090 td 0xfffff8000f0c6000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb9c8e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb9c920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cb9c960 sleepq_timedwait() at sleepq_timedwait+0x43/frame 0xfffffe085cb9c990 _sleep() at _sleep+0x2f2/frame 0xfffffe085cb9ca30 soaio_kproc_loop() at soaio_kproc_loop+0x141/frame 0xfffffe085cb9ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb9cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb9cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100071 td 0xfffff80007ec2560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c75b920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c75b960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c75b9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c75b9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c75ba40 usb_process() at usb_process+0xd4/frame 0xfffffe085c75ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c75bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c75bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100072 td 0xfffff80007ec2000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c760920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c760960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c7609a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c7609d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c760a40 usb_process() at usb_process+0xd4/frame 0xfffffe085c760a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c760ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c760ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100073 td 0xfffff80007ec1560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c765920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c765960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c7659a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c7659d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c765a40 usb_process() at usb_process+0xd4/frame 0xfffffe085c765a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c765ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c765ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100074 td 0xfffff80007ec1000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c76a920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c76a960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c76a9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c76a9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c76aa40 usb_process() at usb_process+0xd4/frame 0xfffffe085c76aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c76aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c76aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100075 td 0xfffff80007eb5560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c76f920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c76f960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c76f9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c76f9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c76fa40 usb_process() at usb_process+0xd4/frame 0xfffffe085c76fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c76fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c76fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100077 td 0xfffff800078ec560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c8fc920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c8fc960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c8fc9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c8fc9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c8fca40 usb_process() at usb_process+0xd4/frame 0xfffffe085c8fca70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c8fcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c8fcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100078 td 0xfffff8000a048000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c901920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c901960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c9019a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c9019d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c901a40 usb_process() at usb_process+0xd4/frame 0xfffffe085c901a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c901ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c901ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100079 td 0xfffff8000a047560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c906920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c906960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c9069a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c9069d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c906a40 usb_process() at usb_process+0xd4/frame 0xfffffe085c906a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c906ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c906ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100080 td 0xfffff8000a047000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c90b920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c90b960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c90b9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c90b9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c90ba40 usb_process() at usb_process+0xd4/frame 0xfffffe085c90ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c90bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c90bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100081 td 0xfffff8000a046560 sched_switch() at sched_switch+0x486/frame 0xfffffe085c910920 mi_switch() at mi_switch+0x195/frame 0xfffffe085c910960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085c9109a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085c9109d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085c910a40 usb_process() at usb_process+0xd4/frame 0xfffffe085c910a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c910ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c910ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100082 td 0xfffff8000a046000 sched_switch() at sched_switch+0x486/frame 0xfffffe085caf8920 mi_switch() at mi_switch+0x195/frame 0xfffffe085caf8960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085caf89a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085caf89d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085caf8a40 usb_process() at usb_process+0xd4/frame 0xfffffe085caf8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085caf8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085caf8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100083 td 0xfffff8000a045560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cafd920 mi_switch() at mi_switch+0x195/frame 0xfffffe085cafd960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cafd9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cafd9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085cafda40 usb_process() at usb_process+0xd4/frame 0xfffffe085cafda70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cafdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cafdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100084 td 0xfffff8000a045000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb02920 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb02960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cb029a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cb029d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085cb02a40 usb_process() at usb_process+0xd4/frame 0xfffffe085cb02a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb02ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb02ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100085 td 0xfffff8000a044560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb07920 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb07960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cb079a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cb079d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085cb07a40 usb_process() at usb_process+0xd4/frame 0xfffffe085cb07a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb07ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb07ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 14 tid 100086 td 0xfffff8000a044000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb0c920 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb0c960 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cb0c9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cb0c9d0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe085cb0ca40 usb_process() at usb_process+0xd4/frame 0xfffffe085cb0ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb0cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb0cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100066 td 0xfffff800078ea000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c788d8d0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c788d910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c788d950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c788d980 _sleep() at _sleep+0x30b/frame 0xfffffe07c788da20 xpt_done_td() at xpt_done_td+0xa5/frame 0xfffffe07c788da70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c788dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c788dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100067 td 0xfffff800078e9560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78928d0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7892910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7892950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7892980 _sleep() at _sleep+0x30b/frame 0xfffffe07c7892a20 xpt_done_td() at xpt_done_td+0xa5/frame 0xfffffe07c7892a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7892ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7892ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100068 td 0xfffff800078e9000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78978d0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7897910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7897950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7897980 _sleep() at _sleep+0x30b/frame 0xfffffe07c7897a20 xpt_done_td() at xpt_done_td+0xa5/frame 0xfffffe07c7897a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7897ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7897ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100100 td 0xfffff8000f1c1560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbcf8c0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbcf900 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbcf940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbcf970 _sleep() at _sleep+0x30b/frame 0xfffffe085cbcfa10 xpt_scanner_thread() at xpt_scanner_thread+0x5c/frame 0xfffffe085cbcfa70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbcfab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbcfab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns pid 3 tid 100065 td 0xfffff800078ea560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78888e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7888920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7888960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7888990 _sleep() at _sleep+0x30b/frame 0xfffffe07c7888a30 crypto_ret_proc() at crypto_ret_proc+0x1b8/frame 0xfffffe07c7888a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7888ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7888ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100064 td 0xfffff800078eb000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78838e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7883920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7883960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7883990 _sleep() at _sleep+0x30b/frame 0xfffffe07c7883a30 crypto_proc() at crypto_proc+0x38d/frame 0xfffffe07c7883a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7883ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7883ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100060 td 0xfffff8000789d000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c786f8e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c786f920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c786f960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c786f990 _sleep() at _sleep+0x30b/frame 0xfffffe07c786fa30 g_run_events() at g_run_events+0x4e7/frame 0xfffffe07c786fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c786fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c786fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100061 td 0xfffff80007881560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78748e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7874920 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7874960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7874990 _sleep() at _sleep+0x30b/frame 0xfffffe07c7874a30 g_io_schedule_up() at g_io_schedule_up+0xa3/frame 0xfffffe07c7874a60 g_up_procbody() at g_up_procbody+0x6d/frame 0xfffffe07c7874a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7874ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7874ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100062 td 0xfffff800078ec000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78798d0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7879910 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7879950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c7879980 _sleep() at _sleep+0x30b/frame 0xfffffe07c7879a20 g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xfffffe07c7879a60 g_down_procbody() at g_down_procbody+0x6d/frame 0xfffffe07c7879a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7879ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7879ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100015 td 0xfffff800077a5000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100016 td 0xfffff800077a4560 sched_switch() at sched_switch+0x486/frame 0xfffffe083918b9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe083918ba20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe083918ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe083918bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083918bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100017 td 0xfffff800077a4000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391909e0 mi_switch() at mi_switch+0x195/frame 0xfffffe0839190a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe0839190a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839190ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839190ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xfffff80007794560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391959e0 mi_switch() at mi_switch+0x195/frame 0xfffffe0839195a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe0839195a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839195ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839195ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffff800077d6000 sched_switch() at sched_switch+0x486/frame 0xfffffe083919a9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe083919aa20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe083919aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe083919aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083919aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100020 td 0xfffff800077d5560 sched_switch() at sched_switch+0x486/frame 0xfffffe083919f9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe083919fa20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe083919fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe083919fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083919fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xfffff800077d5000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391a49e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391a4a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391a4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391a4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391a4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xfffff800077d4560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391a99e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391a9a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391a9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391a9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391a9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xfffff800077d4000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391ae9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391aea20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391aea70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391aeab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391aeab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100024 td 0xfffff800077d3560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391b39e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391b3a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391b3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391b3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391b3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100025 td 0xfffff800077d3000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391b89e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391b8a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391b8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391b8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391b8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xfffff800077d2560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391bd9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391bda20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391bda70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391bdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391bdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xfffff800077d2000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391c29e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391c2a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391c2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391c2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391c2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffff800077dc560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391c79e0 mi_switch() at mi_switch+0x195/frame 0xfffffe08391c7a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe08391c7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391c7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391c7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffff800078ab000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78519e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7851a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe07c7851a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7851ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7851ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100057 td 0xfffff800078a9560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xfffff8000789d560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c786a9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c786aa20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe07c786aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c786aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c786aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100069 td 0xfffff80007ec3560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c789c9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c789ca20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe07c789ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c789cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c789cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100070 td 0xfffff80007ec3000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c7569e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085c756a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe085c756a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c756ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c756ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100076 td 0xfffff80007eb5000 sched_switch() at sched_switch+0x486/frame 0xfffffe085c8f79e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085c8f7a20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe085c8f7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085c8f7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085c8f7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100087 td 0xfffff8000a0b5560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100088 td 0xfffff8000a0b5000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb8e9e0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb8ea20 ithread_loop() at ithread_loop+0x1ce/frame 0xfffffe085cb8ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb8eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb8eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff80007793000 sched_switch() at sched_switch+0x486/frame 0xfffffe083914a930 mi_switch() at mi_switch+0x195/frame 0xfffffe083914a970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe083914aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe083914aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083914aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff80007792560 sched_switch() at sched_switch+0x486/frame 0xfffffe083914f930 mi_switch() at mi_switch+0x195/frame 0xfffffe083914f970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe083914fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe083914fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083914fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff80007792000 sched_switch() at sched_switch+0x486/frame 0xfffffe0839154930 mi_switch() at mi_switch+0x195/frame 0xfffffe0839154970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe0839154a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839154ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839154ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff8000778d560 sched_switch() at sched_switch+0x486/frame 0xfffffe0839159930 mi_switch() at mi_switch+0x195/frame 0xfffffe0839159970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe0839159a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839159ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839159ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff8000778d000 sched_switch() at sched_switch+0x486/frame 0xfffffe083915e930 mi_switch() at mi_switch+0x195/frame 0xfffffe083915e970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe083915ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe083915eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083915eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff8000778c560 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15dccf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15dcd10 trap() at trap+0x49/frame 0xfffffe07c15dcf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15dcf20 --- trap 0x13, rip = 0xffffffff81035f16, rsp = 0xfffffe08391638e0, rbp = 0xfffffe08391638e0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe08391638e0 acpi_cpu_idle() at acpi_cpu_idle+0x308/frame 0xfffffe0839163930 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0839163950 cpu_idle() at cpu_idle+0x90/frame 0xfffffe0839163970 sched_idletd() at sched_idletd+0x4f0/frame 0xfffffe0839163a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839163ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839163ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff8000778c000 sched_switch() at sched_switch+0x486/frame 0xfffffe0839168930 mi_switch() at mi_switch+0x195/frame 0xfffffe0839168970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe0839168a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839168ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839168ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff800077a7560 sched_switch() at sched_switch+0x486/frame 0xfffffe083916d930 mi_switch() at mi_switch+0x195/frame 0xfffffe083916d970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe083916da70 fork_exit() at fork_exit+0x84/frame 0xfffffe083916dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083916dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff800077a7000 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15f4cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15f4d10 trap() at trap+0x49/frame 0xfffffe07c15f4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15f4f20 --- trap 0x13, rip = 0xffffffff81035f16, rsp = 0xfffffe08391728e0, rbp = 0xfffffe08391728e0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe08391728e0 acpi_cpu_idle() at acpi_cpu_idle+0x308/frame 0xfffffe0839172930 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0839172950 cpu_idle() at cpu_idle+0x90/frame 0xfffffe0839172970 sched_idletd() at sched_idletd+0x4f0/frame 0xfffffe0839172a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839172ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839172ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff800077a6560 sched_switch() at sched_switch+0x486/frame 0xfffffe0839177910 mi_switch() at mi_switch+0x195/frame 0xfffffe0839177950 critical_exit() at critical_exit+0x8b/frame 0xfffffe0839177970 sched_idletd() at sched_idletd+0x4f0/frame 0xfffffe0839177a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839177ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839177ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff800077a6000 sched_switch() at sched_switch+0x486/frame 0xfffffe083917c930 mi_switch() at mi_switch+0x195/frame 0xfffffe083917c970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe083917ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe083917cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe083917cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff800077a5560 sched_switch() at sched_switch+0x486/frame 0xfffffe0839181930 mi_switch() at mi_switch+0x195/frame 0xfffffe0839181970 sched_idletd() at sched_idletd+0x3d6/frame 0xfffffe0839181a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839181ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839181ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff80007793560 sched_switch() at sched_switch+0x486/frame 0xfffffe0839145520 mi_switch() at mi_switch+0x195/frame 0xfffffe0839145560 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391455a0 sleepq_catch_signals() at sleepq_catch_signals+0x466/frame 0xfffffe0839145610 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0839145640 _sleep() at _sleep+0x301/frame 0xfffffe08391456e0 kern_wait6() at kern_wait6+0x467/frame 0xfffffe0839145790 sys_wait4() at sys_wait4+0x7a/frame 0xfffffe0839145980 amd64_syscall() at amd64_syscall+0x2eb/frame 0xfffffe0839145ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0839145ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x42375a, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe810 --- Tracing command audit pid 10 tid 100001 td 0xfffff80007794000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391408f0 mi_switch() at mi_switch+0x195/frame 0xfffffe0839140930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe0839140970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391409a0 _cv_wait() at _cv_wait+0x1d3/frame 0xfffffe0839140a10 audit_worker() at audit_worker+0xa3/frame 0xfffffe0839140a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0839140ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0839140ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81e02f00 sched_switch() at sched_switch+0x486/frame 0xffffffff823a99b0 mi_switch() at mi_switch+0x195/frame 0xffffffff823a99f0 sleepq_switch() at sleepq_switch+0x183/frame 0xffffffff823a9a30 sleepq_wait() at sleepq_wait+0x43/frame 0xffffffff823a9a60 _sleep() at _sleep+0x30b/frame 0xffffffff823a9b00 vm_wait() at vm_wait+0xc5/frame 0xffffffff823a9b20 swapper() at swapper+0x25/frame 0xffffffff823a9b70 btext() at btext+0x2c Tracing command kernel pid 0 tid 100029 td 0xfffff800077dc000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391cc900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391cc940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391cc980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391cc9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391cca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391cca70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391ccab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391ccab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffff800077db560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391d1900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391d1940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391d1980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391d19b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391d1a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391d1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391d1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391d1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffff800077db000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391d6900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391d6940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391d6980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391d69b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391d6a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391d6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391d6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391d6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffff800077da560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391db900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391db940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391db980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391db9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391dba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391dba70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391dbab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391dbab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffff800077da000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391e0900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391e0940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391e0980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391e09b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391e0a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391e0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391e0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391e0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffff800077d9560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391e5900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391e5940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391e5980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391e59b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391e5a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391e5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391e5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391e5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffff800077d9000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391ea900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391ea940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391ea980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391ea9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391eaa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391eaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391eaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391eaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffff800077d6560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391ef900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391ef940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391ef980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391ef9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391efa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391efa70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391efab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391efab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffff800077e6000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391f4900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391f4940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391f4980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391f49b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391f4a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391f4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391f4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391f4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffff800077e5560 sched_switch() at sched_switch+0x486/frame 0xfffffe08391f9900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391f9940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391f9980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391f99b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391f9a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391f9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391f9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391f9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffff800077e5000 sched_switch() at sched_switch+0x486/frame 0xfffffe08391fe900 mi_switch() at mi_switch+0x195/frame 0xfffffe08391fe940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe08391fe980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe08391fe9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe08391fea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe08391fea70 fork_exit() at fork_exit+0x84/frame 0xfffffe08391feab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe08391feab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffff800077e4560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c780b900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c780b940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c780b980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c780b9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c780ba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c780ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c780bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c780bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffff800077e4000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7810900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7810940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7810980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78109b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7810a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7810a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7810ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7810ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffff80007872560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7815900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7815940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7815980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78159b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7815a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7815a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7815ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7815ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffff80007872000 (CPU 2) kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffff80007871560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c781f900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c781f940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c781f980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c781f9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c781fa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c781fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffff80007871000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7824900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7824940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7824980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78249b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7824a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7824a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7824ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7824ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xfffff80007870560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7829900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7829940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7829980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78299b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7829a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7829a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7829ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7829ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100047 td 0xfffff80007870000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c782e900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c782e940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c782e980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c782e9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c782ea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c782ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c782eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c782eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100048 td 0xfffff80007864560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7833900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7833940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7833980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78339b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7833a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7833a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7833ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7833ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100049 td 0xfffff80007864000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7838900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7838940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7838980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78389b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7838a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7838a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7838ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7838ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100050 td 0xfffff800077e6560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c783d900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c783d940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c783d980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c783d9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c783da40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c783da70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c783dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c783dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100051 td 0xfffff80007881000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7842900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7842940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7842980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78429b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7842a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7842a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7842ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7842ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100052 td 0xfffff80007880560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c7847900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7847940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7847980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78479b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c7847a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c7847a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7847ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7847ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100053 td 0xfffff800078ab560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c784c900 mi_switch() at mi_switch+0x195/frame 0xfffffe07c784c940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c784c980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c784c9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe07c784ca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x9d/frame 0xfffffe07c784ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c784cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c784cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100055 td 0xfffff800078aa560 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe07c15d4cf0 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe07c15d4d10 trap() at trap+0x49/frame 0xfffffe07c15d4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15d4f20 --- trap 0x13, rip = 0xffffffff80aaaec0, rsp = 0xfffffe07c7856990, rbp = 0xfffffe07c78569e0 --- vmem_periodic() at vmem_periodic+0x100/frame 0xfffffe07c78569e0 taskqueue_run_locked() at taskqueue_run_locked+0x12c/frame 0xfffffe07c7856a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xab/frame 0xfffffe07c7856a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7856ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7856ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100056 td 0xfffff800078aa000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c785b8f0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c785b930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c785b970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c785b9a0 _sleep() at _sleep+0x30b/frame 0xfffffe07c785ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xd8/frame 0xfffffe07c785ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c785bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c785bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xfffff800078a9000 sched_switch() at sched_switch+0x486/frame 0xfffffe07c78658f0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c7865930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c7865970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c78659a0 _sleep() at _sleep+0x30b/frame 0xfffffe07c7865a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xd8/frame 0xfffffe07c7865a70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c7865ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c7865ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffff800078eb560 sched_switch() at sched_switch+0x486/frame 0xfffffe07c787e8f0 mi_switch() at mi_switch+0x195/frame 0xfffffe07c787e930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe07c787e970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe07c787e9a0 _sleep() at _sleep+0x30b/frame 0xfffffe07c787ea40 taskqueue_thread_loop() at taskqueue_thread_loop+0xd8/frame 0xfffffe07c787ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c787eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c787eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xfffff8000a0b4560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cb97900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cb97940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cb97980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cb979b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe085cb97a40 taskqueue_thread_loop() at taskqueue_thread_loop+0x9d/frame 0xfffffe085cb97a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cb97ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cb97ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xfffff8000f1cf000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbb6900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbb6940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbb6980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbb69b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe085cbb6a40 taskqueue_thread_loop() at taskqueue_thread_loop+0x9d/frame 0xfffffe085cbb6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbb6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbb6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100096 td 0xfffff8000f1ce560 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbbb900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbbb940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbbb980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbbb9b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe085cbbba40 taskqueue_thread_loop() at taskqueue_thread_loop+0x9d/frame 0xfffffe085cbbba70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbbbab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbbbab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xfffff8000f1ce000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbc0900 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbc0940 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbc0980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbc09b0 msleep_spin_sbt() at msleep_spin_sbt+0x1e7/frame 0xfffffe085cbc0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0x9d/frame 0xfffffe085cbc0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbc0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbc0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100099 td 0xfffff8000f1cd000 sched_switch() at sched_switch+0x486/frame 0xfffffe085cbca8f0 mi_switch() at mi_switch+0x195/frame 0xfffffe085cbca930 sleepq_switch() at sleepq_switch+0x183/frame 0xfffffe085cbca970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe085cbca9a0 _sleep() at _sleep+0x30b/frame 0xfffffe085cbcaa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xd8/frame 0xfffffe085cbcaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe085cbcaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085cbcaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> dump Dumping 2271 out of 32627 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db> reset cpu_reset: Restarting BSP cpuK_reDB:set r_preenotxeyrin: Sg topped CPU 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07c15c4c50 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07c15c4d00 kdb_reenter() at kdb_reenter+0x33/frame 0xfffffe07c15c4d10 trap() at trap+0x63/frame 0xfffffe07c15c4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe07c15c4f20 --- trap 0x13, rip = 0xffffffff81043ff0, rsp = 0xfffffe07c781a1e0, rbp = 0xfffffe07c781a1f0 --- cpustop_handler() at cpustop_handler+0x60/frame 0xfffffe07c781a1f0 Xcpustop() at Xcpustop+0xa9/frame 0xfffffe07c781a1f0 --- interrupt, rip = 0xffffffff80ed5120, rsp = 0xfffffe07c781a2c0, rbp = 0xfffffe07c781a310 --- cpu_reset() at cpu_reset+0x1e0/frame 0xfffffe07c781a310 db_command() at db_command+0x252/frame 0xfffffe07c781a3e0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe07c781a3f0 db_trap() at db_trap+0xef/frame 0xfffffe07c781a480 kdb_trap() at kdb_trap+0x191/frame 0xfffffe07c781a520 trap() at trap+0x4df/frame 0xfffffe07c781a730 calltrap() at calltrap+0x8/frame 0xfffffe07c781a730 --- trap 0x3, rip = 0xffffffff80a8d9ee, rsp = 0xfffffe07c781a800, rbp = 0xfffffe07c781a810 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07c781a810 vpanic() at vpanic+0x1bf/frame 0xfffffe07c781a890 kassert_panic() at kassert_panic+0x139/frame 0xfffffe07c781a900 _task_fn_rx() at _task_fn_rx+0xa3c/frame 0xfffffe07c781a9f0 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x119/frame 0xfffffe07c781aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xab/frame 0xfffffe07c781aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe07c781aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07c781aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> ~. [terminated ipmitool] $ $ exit (kgdb) bt #0 doadump (textdump=0) at pcpu.h:232 #1 0xffffffff803a621e in db_dump (dummy=, dummy2=false, dummy3=0, dummy4=0x0) at ../../../ddb/db_command.c:572 #2 0xffffffff803a5bb2 in db_command (cmd_table=0x0) at ../../../ddb/db_command.c:479 #3 0xffffffff803a5944 in db_command_loop () at ../../../ddb/db_command.c:532 #4 0xffffffff803a8b2f in db_trap (type=, code=) at ../../../ddb/db_main.c:248 #5 0xffffffff80a8e2f1 in kdb_trap (type=3, code=-61456, tf=) at ../../../kern/subr_kdb.c:654 #6 0xffffffff80ed2b4f in trap (frame=0xfffffe07c781a740) at ../../../amd64/amd64/trap.c:543 #7 0xffffffff80eb3804 in calltrap () at ../../../amd64/amd64/exception.S:237 #8 0xffffffff80a8d9ee in kdb_enter (why=0xffffffff814215b2 "panic", msg=) at cpufunc.h:63 #9 0xffffffff80a4bfdf in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:772 #10 0xffffffff80a4be19 in kassert_panic (fmt=) at ../../../kern/kern_shutdown.c:669 #11 0xffffffff80b5738c in _task_fn_rx (context=0xfffff8000a069280) at ../../../net/iflib.c:1947 #12 0xffffffff80a8c739 in gtaskqueue_run_locked (queue=0xfffff80007857000) at ../../../kern/subr_gtaskqueue.c:329 #13 0xffffffff80a8c50b in gtaskqueue_thread_loop (arg=) at ../../../kern/subr_gtaskqueue.c:504 #14 0xffffffff80a0e9c4 in fork_exit (callout=0xffffffff80a8c460 , arg=0xfffffe00010ba038, frame=0xfffffe07c781aac0) at ../../../kern/kern_fork.c:1038 #15 0xffffffff80eb3dee in fork_trampoline () at ../../../amd64/amd64/exception.S:651 #16 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) f 11 #11 0xffffffff80b5738c in _task_fn_rx (context=0xfffff8000a069280) at ../../../net/iflib.c:1947 1947 MPASS(reclaimable == delta); (kgdb) l 1942 #ifdef INVARIANTS 1943 int32_t delta = fl->ifl_size - get_inuse(fl->ifl_size, fl->ifl_cidx, fl->ifl_pidx, fl->ifl_gen) - 1; 1944 #endif 1945 1946 MPASS(fl->ifl_credits <= fl->ifl_size); 1947 MPASS(reclaimable == delta); 1948 1949 if (reclaimable > 0) 1950 _iflib_fl_refill(ctx, fl, min(max, reclaimable)); 1951 } (kgdb) info loc No locals. (kgdb) p *context Attempt to dereference a generic pointer. (kgdb) f 12 #12 0xffffffff80a8c739 in gtaskqueue_run_locked (queue=0xfffff80007857000) at ../../../kern/subr_gtaskqueue.c:329 329 gtask->ta_func(gtask->ta_context); (kgdb) l 324 gtask->ta_flags &= ~TASK_ENQUEUED; 325 tb.tb_running = gtask; 326 TQ_UNLOCK(queue); 327 328 KASSERT(gtask->ta_func != NULL, ("task->ta_func is NULL")); 329 gtask->ta_func(gtask->ta_context); 330 331 TQ_LOCK(queue); 332 tb.tb_running = NULL; 333 wakeup(gtask); (kgdb) info loc tb = {tb_running = 0xfffff8000a069310, tb_link = {tqe_next = 0x0, tqe_prev = 0xfffff80007857028}} tb = {tb_running = 0xfffff8000a069310, tb_link = {tqe_next = 0x0, tqe_prev = 0xfffff80007857028}} gtask = (struct gtask *) 0xfffff8000a069310 tb_first = (kgdb) p *gtask $1 = {ta_link = {stqe_next = 0x0}, ta_flags = 2, ta_priority = 0, ta_func = 0xffffffff80b56950 <_task_fn_rx>, ta_context = 0xfffff8000a069280} (kgdb) p *queue $2 = {tq_queue = {stqh_first = 0x0, stqh_last = 0xfffff80007857000}, tq_enqueue = 0xffffffff80aa0c80 , tq_context = 0xfffffe00010ba038, tq_name = 0xfffff8000784a100 "taskqueue", tq_active = {tqh_first = 0xfffffe07c781aa00, tqh_last = 0xfffffe07c781aa08}, tq_mutex = {lock_object = {lo_name = 0xfffff8000784a100 "taskqueue", lo_flags = 196608, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, tq_threads = 0xfffff8000785d060, tq_tcount = 1, tq_spin = 1, tq_flags = 1, tq_callouts = 0, tq_callbacks = {0, 0}, tq_cb_contexts = {0x0, 0x0}} (kgdb) Script done on Thu Aug 17 06:34:51 2017