KDB: enter: watchdog timeout, vmcore.362 KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #1 main-n248036-be7c64704edf-dirty: Sun Jul 18 05:27:37 CEST 2021 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac564cd0e) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33174138880 (31637 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 8 11 2 4 7 9 6 5 10 3 Timecounter "TSC-low" frequency 1750037272 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020) vtvga0: smbios0: at iomem 0xf0560-0xf057e smbios0: Version: 2.8, BCD Revision: 2.7 aesni0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 apei0: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb433000-0xfb4333ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb432000-0xfb4327ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec Expensive timeout(9) function: 0xffffffff80510d60(0xfffffe0109f8a000) 0.004937846 s ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ses0: pass0 in 'Slot 01', SATA Slot: scbus1 target 0 ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> Expensive timeout(9) function: 0xffffffff80a75400(0xffffffff81aea0c8) 0.007387260 s mlx5_core0: mem 0xf8000000-0xf9ffffff irq 32 at device 0.0 on pci3 ugen2.1: at usbus2 ugen1.1: at usbus1 uhub0ugen0.1: <0x8086 XHCI root HUB> at usbus0 on usbus2 uhub0: on usbus2 uhub1 on usbus0 uhub1: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub2 on usbus1 uhub2: on usbus1 mlx5_core0: WARN: mlx5_vsc_set_space:124:(pid 0): Space 0x7 is not supported. mlx5_core0: WARN: mlx5_fwdump_prep:99:(pid 0): VSC scan space is not supported mlx5_core0: INFO: init_one:1613:(pid 0): cannot find SR-IOV PCIe cap mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 1, l2 table size(65536), per vport: max uc(1024) max mc(16384) mlx5_core0: Failed to initialize SR-IOV support, errouhub1: 21 ports with 21 removable, self powered uhub3: 4 ports with 3 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-002590c99976. Setting hostid: 0x90b83241. uhub5: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered Starting file system checks: WARNING: / was not properly dismounted Mounting local f. ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 fscklock order reversal: 1st 0xfffffe011bdec9d0 ufs (ufs, lockmgr) @ kern/vfs_mount.c:2050 2nd 0xfffffe014179e0d8 devfs (devfs, lockmgr) @ kern/vfs_subr.c:3251 lock order devfs -> ufs established at: #0 0xffffffff80c91d99 at witness_checkorder+0x459 #1 0xffffffff80bf1e78 at lockmgr_lock_flags+0x178 #2 0xffffffff80f5ae9d at ffs_lock+0x9d #3 0xffffffff811c113c at VOP_LOCK1_APV+0x3c #4 0xffffffff80d327b4 at _vn_lock+0x54 #5 0xffffffff80d0af70 at vfs_domount_first+0x910 #6 0xffffffff80d06032 at vfs_domount+0x482 #7 0xffffffff80d036ae at vfs_donmount+0x9ee #8 0xffffffff80d09aed at kernel_mount+0xfd #9 0xffffffff80d0e571 at parse_mount+0x4b1 #10 0xffffffff80d0cc53 at vfs_mountroot+0x7b3 #11 0xffffffff80bae6ff at start_init+0x1f #12 0xffffffff80bdb8a0 at fork_exit+0x80 #13 0xffffffff810c808e at fork_trampoline+0xe lock order ufs -> devfs attempted at: #0 0xffffffff80c926cf at witness_checkorder+0xd8f #1 0xffffffff80bf1e78 at lockmgr_lock_flags+0x178 #2 0xffffffff811c113c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d327b4 at _vn_lock+0x54 #4 0xffffffff80d155a3 at vput_final+0x1f3 #5 0xffffffff80d14a36 at vrele+0x46 #6 0xffffffff80f56c80 at ffs_unmount+0x4f0 #7 0xffffffff80d0753c at dounmount+0x95c #8 0xffffffff80d06b7d at kern_unmount+0x56d #9 0xffffffff80d06607 at sys_unmount+0x27 #10 0xffffffff810f79c7 at amd64_syscall+0x147 #11 0xffffffff810c792e at fast_syscall_common+0xf8 ** /dev/gpt/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 104 files, 252 used, 1265155 free (75 frags, 158135 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc10 /usr/local/lib/gcc8 /usr/local/lib/gcc9 /usr/local/lib/perl5/5.32/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: mercat1.netperf.freebsd.org. Setting up harvesting: PURE_RDRAND,[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . Autoloading module: acpi_wmi.ko Autoloading module: ichsmb.ko Autoloading module: ioat.ko ioat0: mem 0xfb42c000-0xfb42ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0xfb428000-0xfb42bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0xfb424000-0xfb427fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0xfb420000-0xfb423fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0xfb41c000-0xfb41ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0xfb418000-0xfb41bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0xfb414000-0xfb417fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0xfb410000-0xfb413fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 ichsmb0: port 0x580-0x59f mem 0xfb431000-0xfb4310ff irq 18 at device 31.3 on pci1 smbus0: on ichsmb0 acpi_wmi0: on acpi0 acpi_wmi0: cannot find EC device lo0: link state changed to UP Starting Network: lo0 igb0 igb1 mce0 mce1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8863 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect status: no carrier nd6 options=21 igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:3 status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Autoloading module: acpi_wmi.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: ichsmb.ko Starting Network: mce0. mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 Starting Network: mce1. mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid.ko Autoloading module: ums.ko Autoloading module: usbhid.ko ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192if_delmulti_locked: detaching ifnet instance 0xfffff80004396800 .168.5.1 bound to 192.168.5.171 -- renewal in 300 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Recovering vi editor sessions:. Updating motd:. Clearing /tmp (X related). Starting syslogd. Starting watchdogd. No core dumps found. Setting date via ntp. 18 Jul 05:32:03 ntpdate[2615]: step time server 192.168.5.1 offset -0.343435 sec Starting rpcbind. NFS access cache time=60 Starting statd. Starting lockd. Starting powerd. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Mounting late filesystems:. Starting inetd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. sysctl: unknown oid 'debug.fail_point.pmap_allocpte_page_alloc' at line 12 Sun Jul 18 05:32 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Jul 18 05:40:07 mercat1 su[2805]: pho to root on /dev/pts/0 20210718 05:40:47 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:41:32 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:41:50 all (1/1): force3.sh Jul 18 05:42:17 mercat1 kernel: pid 7393 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:42:30 all (1/1): force3.sh Jul 18 05:42:58 mercat1 kernel: pid 9055 (swap), jid 0, uid 2007, was killed: out of swap space g_vfs_done():md10[WRITE(offset=545161216, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:43:21 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:44:07 all (1/1): force3.sh Jul 18 05:44:17 mercat1 kernel: pid 12337 (rw), uid 2007 inumber 59329 on /mnt: filesystem full Jul 18 05:44:31 mercat1 kernel: pid 12353 (rw), uid 2007 inumber 49865 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:44:57 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:45:10 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:45:44 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:46:33 all (1/1): force3.sh Jul 18 05:46:40 mercat1 kernel: pid 22704 (rw), uid 2007 inumber 33967 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 05:47:02 mercat1 kernel: pid 22781 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 05:47:18 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:47:36 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:48:00 all (1/1): force3.sh Jul 18 05:48:18 mercat1 kernel: pid 29974 (swap), jid 0, uid 2007, was killed: out of swap space Jul 18 05:48:32 mercat1 kernel: pid 29908 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:48:52 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:49:24 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:50:11 all (1/1): force3.sh Jul 18 05:50:26 mercat1 kernel: pid 34544 (creat), uid 2007 inumber 32920 on /mnt: out of inodes UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:50:49 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:51:14 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:51:38 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:51:57 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:52:35 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:53:15 all (1/1): force3.sh Jul 18 05:53:25 mercat1 kernel: pid 48599 (rw), uid 2007 inumber 61143 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:53:51 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:54:46 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:55:09 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:55:42 all (1/1): force3.sh Jul 18 05:55:47 mercat1 kernel: pid 54626 (rw), uid 2007 inumber 55686 on /mnt: filesystem full Jul 18 05:56:02 mercat1 kernel: pid 54650 (rw), uid 2007 inumber 92141 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:56:20 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:56:43 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:57:17 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:57:46 all (1/1): force3.sh Jul 18 05:57:54 mercat1 kernel: pid 62354 (rw), uid 2007 inumber 87818 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:58:33 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 05:59:36 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:00:20 all (1/1): force3.sh Jul 18 06:00:29 mercat1 kernel: pid 66935 (rw), uid 2007 inumber 91861 on /mnt: filesystem full Jul 18 06:00:46 mercat1 kernel: pid 67009 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:01:14 all (1/1): force3.sh Jul 18 06:02:23 mercat1 kernel: pid 68407 (swap), jid 0, uid 2007, was killed: out of swap space Jul 18 06:02:29 mercat1 kernel: pid 68438 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:02:40 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:03:06 all (1/1): force3.sh Jul 18 06:03:27 mercat1 kernel: pid 72009 (openat), uid 2007 inumber 32951 on /mnt: out of inodes Jul 18 06:03:47 mercat1 kernel: pid 72052 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:03:56 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:04:35 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:05:21 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:05:41 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:06:17 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:07:06 all (1/1): force3.sh Jul 18 06:07:15 mercat1 kernel: pid 81547 (rw), uid 2007 inumber 97819 on /mnt: filesystem full g_vfs_done():md10[WRITE(offset=538050560, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:07:32 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:08:33 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:09:05 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=268599296, length=16384)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:09:28 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=537100288, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:10:44 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:11:06 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:11:50 all (1/1): force3.sh Jul 18 06:11:56 mercat1 kernel: pid 99023 (creat), uid 2007 inumber 32923 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:12:24 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:12:46 all (1/1): force3.sh Jul 18 06:12:57 mercat1 kernel: pid 2008 (rw), uid 2007 inumber 120709 on /mnt: filesystem full Jul 18 06:13:09 mercat1 kernel: pid 2033 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:13:32 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:13:53 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:14:49 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=16875008, length=512)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:15:22 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:16:00 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:16:33 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=805568512, length=32768)]error =UFS: forcibly unmounting /dev/md10 from /mnt 6 20210718 06:17:13 all (1/1): force3.sh Jul 18 06:17:43 mercat1 kernel: pid 15961 (swap), jid 0, uid 2007, was killed: out of swap space g_vfs_done():md10[WRITE(offset=273383424, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:17:54 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:18:48 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:19:49 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:20:17 all (1/1): force3.sh Jul 18 06:20:50 mercat1 kernel: pid 24856 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:21:04 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:21:48 all (1/1): force3.sh Jul 18 06:21:53 mercat1 kernel: pid 29431 (rw), uid 2007 inumber 79907 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:22:08 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:22:57 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:23:58 all (1/1): force3.sh Jul 18 06:24:08 mercat1 kernel: pid 34673 (rw), uid 2007 inumber 46568 on /mnt: filesystem full Jul 18 06:24:15 mercat1 kernel: pid 34613 (openat), uid 2007 inumber 32984 on /mnt: out of inodes UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:24:36 all (1/1): force3.sh Jul 18 06:25:07 mercat1 kernel: pid 36191 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:25:26 all (1/1): force3.sh Jul 18 06:25:33 mercat1 kernel: pid 38199 (rw), uid 2007 inumber 86606 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:26:39 all (1/1): force3.sh Jul 18 06:27:04 mercat1 kernel: pid 39745 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:27:20 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=273154048, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:28:52 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:29:18 all (1/1): force3.sh Jul 18 06:29:47 mercat1 kernel: pid 44701 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:30:01 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=270106624, length=32768)]error =UFS: forcibly unmounting /dev/md10 from /mnt 6 g_vfs_done():md10[WRITE(offset=814288896, length=4096)]error = 6 20210718 06:30:23 all (1/1): force3.sh Jul 18 06:30:29 mercat1 kernel: pid 47911 (rw), uid 2007 inumber 51155 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:30:54 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:31:27 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:33:07 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:33:46 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:34:04 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:34:50 all (1/1): force3.sh Jul 18 06:35:20 mercat1 kernel: pid 59246 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:35:38 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:36:23 all (1/1): force3.sh Jul 18 06:36:42 mercat1 kernel: pid 62678 (rw), uid 2007 inumber 38765 on /mnt: filesystem full Jul 18 06:36:49 mercat1 kernel: pid 62766 (mkdir), uid 2007 inumber 126839 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:37:02 all (1/1): force3.sh Jul 18 06:37:10 mercat1 kernel: pid 65549 (rw), uid 2007 inumber 57324 on /mnt: filesystem full Jul 18 06:37:17 mercat1 kernel: pid 65584 (rw), uid 2007 inumber 75715 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:37:27 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 06:37:58 mercat1 kernel: pid 67222 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 06:38:06 all (1/1): force3.sh Jul 18 06:38:50 mercat1 kernel: pid 68735 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:38:58 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:39:43 all (1/1): force3.sh Jul 18 06:39:50 mercat1 kernel: pid 71767 (rw), uid 2007 inumber 74386 on /mnt: filesystem full g_vfs_done():md10[WRITE(offset=7340032, length=32768)]error = 6UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:40:42 all (1/1): force3.sh Jul 18 06:40:46 mercat1 kernel: pid 73419 (mkdir), uid 2007 inumber 76499 on /mnt: filesystem full Jul 18 06:40:49 mercat1 kernel: pid 73335 (creat), uid 2007 inumber 32933 on /mnt: filesystem full Jul 18 06:40:51 mercat1 kernel: pid 73405 (rw), uid 2007 inumber 89056 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:41:06 all (1/1): force3.sh Jul 18 06:41:41 mercat1 kernel: pid 76077 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:41:59 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:42:44 all (1/1): force3.sh Jul 18 06:42:55 mercat1 kernel: pid 79211 (rw), uid 2007 inumber 90516 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:43:15 all (1/1): force3.sh Jul 18 06:43:22 mercat1 kernel: pid 81758 (symlink), uid 2007 inumber 32901 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:43:40 all (1/1): force3.sh Jul 18 06:44:06 mercat1 kernel: pid 83682 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:44:13 all (1/1): force3.sh Jul 18 06:44:31 mercat1 kernel: pid 85109 (mkdir), uid 2007 inumber 32111 on /mnt: out of inodes UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:44:38 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:44:56 all (1/1): force3.sh Jul 18 06:45:06 mercat1 kernel: pid 90442 (rw), uid 2007 inumber 42356 on /mnt: filesystem full Jul 18 06:45:24 mercat1 kernel: pid 90468 (swap), jid 0, uid 2007, was killed: out of swap space Jul 18 06:45:44 mercat1 kernel: pid 90538 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:45:56 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:46:23 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:47:17 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:47:50 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:48:34 all (1/1): force3.sh Jul 18 06:48:42 mercat1 kernel: pid 2871 (rw), uid 2007 inumber 117408 on /mnt: filesystem full Jul 18 06:48:54 mercat1 kernel: pid 2913 (rw), uid 2007 inumber 58915 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:49:05 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:49:51 all (1/1): force3.sh Jul 18 06:50:05 mercat1 kernel: pid 8651 (symlink), uid 2007 inumber 65796 on /mnt: out of inodes Jul 18 06:50:20 mercat1 kernel: pid 8671 (symlink), uid 2007 inumber 65812 on /mnt: out of inodes UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:50:26 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:50:41 all (1/1): force3.sh Jul 18 06:51:25 mercat1 kernel: pid 13989 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:51:39 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:52:15 all (1/1): force3.sh Jul 18 06:52:24 mercat1 kernel: pid 18384 (rw), uid 2007 inumber 27637 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:52:41 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 06:53:19 mercat1 kernel: pid 20444 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 06:53:26 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:53:53 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=271122432, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:54:23 all (1/1): force3.sh Jul 18 06:54:46 mercat1 kernel: pid 25024 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:55:57 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:56:20 all (1/1): force3.sh Jul 18 06:56:30 mercat1 kernel: pid 28010 (rw), uid 2007 inumber 119519 on /mnt: filesystem full g_vfs_done():md10[WRITE(offset=273416192, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:57:02 all (1/1): force3.sh Jul 18 06:57:23 mercat1 kernel: pid 29717 (swap), jid 0, uid 2007, was killed: out of swap space Jul 18 06:57:25 mercat1 kernel: pid 29650 (rw), uid 2007 inumber 38305 on /mnt: filesystem full Jul 18 06:57:35 mercat1 kernel: pid 29712 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:57:50 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:58:34 all (1/1): force3.sh Jul 18 06:58:43 mercat1 kernel: pid 34404 (link), uid 2007 inumber 32908 on /mnt: filesystem full Jul 18 06:58:50 mercat1 kernel: pid 34410 (rw), uid 2007 inumber 59591 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:59:11 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 06:59:59 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:01:13 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:01:59 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:02:20 all (1/1): force3.sh Jul 18 07:03:10 mercat1 kernel: pid 42258 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:03:38 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:04:16 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:04:36 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:05:19 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:06:00 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 07:06:37 mercat1 kernel: pid 51846 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 07:06:45 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=540966912, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:07:42 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:08:21 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:08:49 all (1/1): force3.sh Jul 18 07:09:00 mercat1 kernel: pid 59023 (rw), uid 2007 inumber 81638 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:09:39 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:11:07 all (1/1): force3.sh Jul 18 07:11:37 mercat1 kernel: pid 62206 (swap), jid 0, uid 2007, was killed: out of swap space g_vfs_done():md10[WRITE(offset=8585216, length=4096)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:12:31 all (1/1): force3.sh Jul 18 07:13:09 mercat1 kernel: pid 63724 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:13:22 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:14:17 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:15:16 all (1/1): force3.sh Jul 18 07:15:57 mercat1 kernel: pid 70392 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:16:07 all (1/1): force3.sh Jul 18 07:16:20 mercat1 kernel: pid 73054 (rw), uid 2007 inumber 91042 on /mnt: filesystem full g_vfs_done():md10[WRITE(offset=1900544, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:16:39 all (1/1): force3.sh Jul 18 07:17:15 mercat1 kernel: pid 74601 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:17:28 all (1/1): force3.sh Jul 18 07:18:07 mercat1 kernel: pid 76142 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:18:17 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:19:15 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:19:38 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 07:20:08 mercat1 kernel: pid 83939 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 07:20:17 all (1/1): force3.sh Jul 18 07:20:31 mercat1 kernel: pid 85508 (rw), uid 2007 inumber 74482 on /mnt: filesystem full Jul 18 07:20:40 mercat1 kernel: pid 85483 (openat), uid 2007 inumber 65816 on /mnt: out of inodes UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:21:08 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:21:30 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:22:38 all (1/1): force3.sh Jul 18 07:22:46 mercat1 kernel: pid 91359 (link), uid 2007 inumber 32992 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:23:12 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=540344320, length=32768)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:23:40 all (1/1): force3.sh Jul 18 07:23:52 mercat1 kernel: pid 94764 (rw), uid 2007 inumber 46050 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:24:06 all (1/1): force3.sh Jul 18 07:24:17 mercat1 kernel: pid 96379 (rw), uid 2007 inumber 60374 on /mnt: filesystem full g_vfs_done():md10[WRITE(offset=332496896, length=1048576)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:24:24 all (1/1): force3.sh Jul 18 07:25:01 mercat1 kernel: pid 98492 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt Jul 18 07:25:12 mercat1 kernel: pid 98472 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 07:25:21 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=65536, length=4096)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:25:59 all (1/1): force3.sh Jul 18 07:26:37 mercat1 kernel: pid 1449 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:26:44 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:26:58 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:27:24 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:28:07 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:28:27 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:29:15 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:30:04 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:30:45 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:31:07 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:31:41 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:32:07 all (1/1): force3.sh Jul 18 07:32:18 mercat1 kernel: pid 25205 (rw), uid 2007 inumber 41963 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:32:31 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:32:57 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:33:46 all (1/1): force3.sh Jul 18 07:33:56 mercat1 kernel: pid 31062 (rw), uid 2007 inumber 69724 on /mnt: filesystem full Jul 18 07:34:09 mercat1 kernel: pid 31085 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:34:34 all (1/1): force3.sh Jul 18 07:34:48 mercat1 kernel: pid 32549 (rw), uid 2007 inumber 73135 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:35:27 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:36:19 all (1/1): force3.sh Jul 18 07:37:27 mercat1 kernel: pid 35688 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:37:45 all (1/1): force3.sh Jul 18 07:37:55 mercat1 kernel: pid 37321 (rw), uid 2007 inumber 55211 on /mnt: filesystem full Jul 18 07:38:26 mercat1 kernel: pid 37365 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:38:47 all (1/1): force3.sh Jul 18 07:39:33 UFS: forcibly unmounting /dev/md10 from /mnt mercat1 kernel: pid 39016 (swap), jid 0, uid 2007, was killed: out of swap space 20210718 07:39:51 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:40:35 all (1/1): force3.sh Jul 18 07:41:03 mercat1 kernel: pid 42185 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:41:13 all (1/1): force3.sh Jul 18 07:41:42 mercat1 kernel: pid 43756 (rw), uid 2007 inumber 60393 on /mnt: filesystem full UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:41:56 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:42:16 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:42:58 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:43:14 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:43:49 all (1/1): force3.sh Jul 18 07:44:45 mercat1 kernel: pid 53109 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:44:58 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:46:07 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:46:52 all (1/1): force3.sh Jul 18 07:47:15 mercat1 kernel: pid 57856 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:47:31 all (1/1): force3.sh UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:48:23 all (1/1): force3.sh Jul 18 07:48:46 mercat1 kernel: pid 60914 (swap), jid 0, uid 2007, was killed: out of swap space UFS: forcibly unmounting /dev/md10 from /mnt 20210718 07:49:06 all (1/1): force3.sh g_vfs_done():md10[WRITE(offset=14176256, length=4096)]error = 6 UFS: forcibly unmounting /dev/md10 from /mnt stress2: mount leak: 13/44 stress2: GEOM leak: 4/77 interrupt total irq3: uart1 1662 irq4: uart0 5 irq9: acpi0 4 irq18: ehci0 ehci1+ 25144 irq19: xhci0 15563 cpu0:timer 7753642 cpu1:timer 7204112 cpu2:timer 7217666 cpu3:timer 7863203 cpu4:timer 7229666 cpu5:timer 7232432 cpu6:timer 7220441 cpu7:timer 7394725 cpu8:timer 7219257 cpu9:timer 7234440 cpu10:timer 10366405 cpu11:timer 7184009 irq56: ahci0 1271479 irq57: igb0:rxq0 6952 irq58: igb0:rxq1 41 irq59: igb0:rxq2 475 irq60: igb0:rxq3 384 irq61: igb0:rxq4 184 irq62: igb0:rxq5 41 irq63: igb0:aq 2 irq73: mlx5_core0 46 irq88: mlx5_core1 46 Total 92442041 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0145204830 hardclock() at hardclock+0x1bd/frame 0xfffffe0145204870 handleevents() at handleevents+0xb4/frame 0xfffffe01452048b0 timercb() at timercb+0x196/frame 0xfffffe0145204900 lapic_handle_timer() at lapic_handle_timer+0x9b/frame 0xfffffe0145204930 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe0145204930 --- interrupt, rip = 0xffffffff80c93348, rsp = 0xfffffe0145204a00, rbp = 0xfffffe0145204a30 --- witness_unlock() at witness_unlock+0x1b8/frame 0xfffffe0145204a30 __mtx_unlock_flags() at __mtx_unlock_flags+0x4d/frame 0xfffffe0145204a60 userret() at userret+0x19b/frame 0xfffffe0145204ac0 amd64_syscall() at amd64_syscall+0x205/frame 0xfffffe0145204bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145204bf0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8012094ca, rsp = 0x7fffffffdca8, rbp = 0x7fffffffdd30 --- KDB: enter: watchdog timeout [ thread pid 90887 tid 100423 ] Stopped at kdb_enter+0x37: movq $0,0x1279f7e(%rip) db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> dump Dumping 1622 out of 32624 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:1:pho1> bt Tracing pid 90887 tid 100423 td 0xfffffe0144221020 kdb_enter() at kdb_enter+0x37/frame 0xfffffe0145204830 hardclock() at hardclock+0x1d0/frame 0xfffffe0145204870 handleevents() at handleevents+0xb4/frame 0xfffffe01452048b0 timercb() at timercb+0x196/frame 0xfffffe0145204900 lapic_handle_timer() at lapic_handle_timer+0x9b/frame 0xfffffe0145204930 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe0145204930 --- interrupt, rip = 0xffffffff80c93348, rsp = 0xfffffe0145204a00, rbp = 0xfffffe0145204a30 --- witness_unlock() at witness_unlock+0x1b8/frame 0xfffffe0145204a30 __mtx_unlock_flags() at __mtx_unlock_flags+0x4d/frame 0xfffffe0145204a60 userret() at userret+0x19b/frame 0xfffffe0145204ac0 amd64_syscall() at amd64_syscall+0x205/frame 0xfffffe0145204bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145204bf0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8012094ca, rsp = 0x7fffffffdca8, rbp = 0x7fffffffdd30 --- db:1:pho1> show allpcpu Current CPU: 7 cpuid = 0 dynamic pcpu = 0x9b5bc0 curthread = 0xfffffe00e4c5b3a0: pid 11 tid 100003 critnest 1 "idle: cpu0" curpcb = 0xfffffe00e4c5b8b0 fpcurthread = none idlethread = 0xfffffe00e4c5b3a0: tid 100003 "idle: cpu0" self = 0xffffffff82610000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82610384 rsp0 = 0xfffffe0037790cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x7a3671b15 gs32p = 0xffffffff82610404 ldt = 0xffffffff82610444 tss = 0xffffffff82610434 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00bb288bc0 curthread = 0xfffffe00e4d3d3a0: pid 21 tid 100134 critnest 1 "doneq1" curpcb = 0xfffffe00e4d3d8b0 fpcurthread = none idlethread = 0xfffffe00e4c5ac80: tid 100004 "idle: cpu1" self = 0xffffffff82611000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82611384 rsp0 = 0xfffffe00e4e81cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x7195c4a2d gs32p = 0xffffffff82611404 ldt = 0xffffffff82611444 tss = 0xffffffff82611434 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00bb295bc0 curthread = 0xfffffe00e4c5a560: pid 11 tid 100005 critnest 1 "idle: cpu2" curpcb = 0xfffffe00e4c5aa70 fpcurthread = none idlethread = 0xfffffe00e4c5a560: tid 100005 "idle: cpu2" self = 0xffffffff82612000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82612384 rsp0 = 0xfffffe003779acc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x635e51dec gs32p = 0xffffffff82612404 ldt = 0xffffffff82612444 tss = 0xffffffff82612434 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00bb2a2bc0 curthread = 0xfffffe00e4ceac80: pid 0 tid 100048 critnest 0 "thread taskq" curpcb = 0xfffffe00e4ceb190 fpcurthread = none idlethread = 0xfffffe00e4c59e40: tid 100006 "idle: cpu3" self = 0xffffffff82613000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82613384 rsp0 = 0xfffffe00e4894cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x75efe488b gs32p = 0xffffffff82613404 ldt = 0xffffffff82613444 tss = 0xffffffff82613434 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00bb2afbc0 curthread = 0xfffffe00e4c59720: pid 11 tid 100007 critnest 1 "idle: cpu4" curpcb = 0xfffffe00e4c59c30 fpcurthread = none idlethread = 0xfffffe00e4c59720: tid 100007 "idle: cpu4" self = 0xffffffff82614000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82614384 rsp0 = 0xfffffe00377a4cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x65e26187c gs32p = 0xffffffff82614404 ldt = 0xffffffff82614444 tss = 0xffffffff82614434 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00bb2bcbc0 curthread = 0xfffffe0144128c80: pid 90885 tid 100341 critnest 0 "sh" curpcb = 0xfffffe0144129190 fpcurthread = 0xfffffe0144128c80: pid 90885 "sh" idlethread = 0xfffffe00e4c59000: tid 100008 "idle: cpu5" self = 0xffffffff82615000 curpmap = 0xfffffe0149f1c500 tssp = 0xffffffff82615384 rsp0 = 0xfffffe0144463cc0 kcr3 = 0x80000006e636c3c1 ucr3 = 0x800000001d738bc1 scr3 = 0x1d738bc1 gs32p = 0xffffffff82615404 ldt = 0xffffffff82615444 tss = 0xffffffff82615434 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00bb2c9bc0 curthread = 0xfffffe00e4c5d740: pid 11 tid 100009 critnest 1 "idle: cpu6" curpcb = 0xfffffe00e4c5dc50 fpcurthread = none idlethread = 0xfffffe00e4c5d740: tid 100009 "idle: cpu6" self = 0xffffffff82616000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82616384 rsp0 = 0xfffffe00377aecc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x6a17c6e0f gs32p = 0xffffffff82616404 ldt = 0xffffffff82616444 tss = 0xffffffff82616434 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00bb2d6bc0 curthread = 0xfffffe0144221020: pid 90887 tid 100423 critnest 1 "sh" curpcb = 0xfffffe0144221530 fpcurthread = 0xfffffe0144221020: pid 90887 "sh" idlethread = 0xfffffe00e4c5d020: tid 100010 "idle: cpu7" self = 0xffffffff82617000 curpmap = 0xfffffe0144216500 tssp = 0xffffffff82617384 rsp0 = 0xfffffe0145204cc0 kcr3 = 0x80000001e283d4c9 ucr3 = 0x80000008411e5cc9 scr3 = 0x8411e5cc9 gs32p = 0xffffffff82617404 ldt = 0xffffffff82617444 tss = 0xffffffff82617434 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00bb2e3bc0 curthread = 0xfffffe00e4c5c900: pid 11 tid 100011 critnest 1 "idle: cpu8" curpcb = 0xfffffe00e4c5ce10 fpcurthread = none idlethread = 0xfffffe00e4c5c900: tid 100011 "idle: cpu8" self = 0xffffffff82618000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82618384 rsp0 = 0xfffffe00377b8cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x11bf97b86 gs32p = 0xffffffff82618404 ldt = 0xffffffff82618444 tss = 0xffffffff82618434 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00bb2f0bc0 curthread = 0xfffffe00e4c611e0: pid 11 tid 100012 critnest 1 "idle: cpu9" curpcb = 0xfffffe00e4c616f0 fpcurthread = none idlethread = 0xfffffe00e4c611e0: tid 100012 "idle: cpu9" self = 0xffffffff82619000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff82619384 rsp0 = 0xfffffe00377b3cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x3f0cc0e4d gs32p = 0xffffffff82619404 ldt = 0xffffffff82619444 tss = 0xffffffff82619434 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00bb2fdbc0 curthread = 0xfffffe00e4c60ac0: pid 11 tid 100013 critnest 1 "idle: cpu10" curpcb = 0xfffffe00e4c60fd0 fpcurthread = none idlethread = 0xfffffe00e4c60ac0: tid 100013 "idle: cpu10" self = 0xffffffff8261a000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff8261a384 rsp0 = 0xfffffe00377c2cc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x1be5fad94 gs32p = 0xffffffff8261a404 ldt = 0xffffffff8261a444 tss = 0xffffffff8261a434 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00bb30abc0 curthread = 0xfffffe00e4c603a0: pid 11 tid 100014 critnest 1 "idle: cpu11" curpcb = 0xfffffe00e4c608b0 fpcurthread = none idlethread = 0xfffffe00e4c603a0: tid 100014 "idle: cpu11" self = 0xffffffff8261b000 curpmap = 0xffffffff81e9cc50 tssp = 0xffffffff8261b384 rsp0 = 0xfffffe00377bdcc0 kcr3 = 0x8000000002556002 ucr3 = 0xffffffffffffffff scr3 = 0x5ee237fa5 gs32p = 0xffffffff8261b404 ldt = 0xffffffff8261b444 tss = 0xffffffff8261b434 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 90885 (sh) thread 0xfffffe0144128c80 (100341) exclusive sx proctree (proctree) r = 0 (0xffffffff81a02a40) locked @ kern/kern_exit.c:1280 Process 78677 (ls) thread 0xfffffe0149677900 (100504) shared lockmgr ufs (ufs) r = 0 (0xfffffe010e1570d8) locked @ kern/vfs_lookup.c:903 Process 63634 (ls) thread 0xfffffe014423d740 (100414) shared lockmgr ufs (ufs) r = 0 (0xfffffe010e1570d8) locked @ kern/vfs_lookup.c:903 Process 2595 (syslogd) thread 0xfffffe01434c3900 (100282) exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0143f04070) locked @ kern/vfs_syscalls.c:3528 Process 0 (kernel) thread 0xfffffe00e4ceac80 (100048) exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0143ef09d0) locked @ kern/vfs_mount.c:2050 db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffffe010e157068: type VDIR usecount 73, writecount 0, refcount 4 seqc users 0 mountedhere 0 hold count flags () flags (VV_ROOT|VMP_LAZYLIST) v_object 0xfffff800031aab58 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 2) #0 0xffffffff80bf1ee7 at lockmgr_lock_flags+0x1e7 #1 0xffffffff80f5ae9d at ffs_lock+0x9d #2 0xffffffff811c113c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d327b4 at _vn_lock+0x54 #4 0xffffffff80d00812 at lookup+0xc2 #5 0xffffffff80cffa19 at namei+0x379 #6 0xffffffff80d31a6e at vn_open_cred+0x11e #7 0xffffffff80d28aac at kern_openat+0x28c #8 0xffffffff810f79c7 at amd64_syscall+0x147 #9 0xffffffff810c792e at fast_syscall_common+0xf8 nlink=44, effnlink=44, size=2048, extsize 0 generation=56909bb9, uid=0, gid=0, flags=0x0 ino 2, on dev gpt/work vnode 0xfffffe0143f04000: type VREG usecount 1, writecount 1, refcount 3 seqc users 0 hold count flags () flags (VIRF_PGREAD|VMP_LAZYLIST) v_object 0xfffff801d14e6b58 ref 0 pages 2 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe01434c3900 (pid 2595, syslogd, tid 100282) #0 0xffffffff80bf1ee7 at lockmgr_lock_flags+0x1e7 #1 0xffffffff80f5ae9d at ffs_lock+0x9d #2 0xffffffff811c113c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d327b4 at _vn_lock+0x54 #4 0xffffffff80d2dfc9 at kern_fsync+0x129 #5 0xffffffff810f79c7 at amd64_syscall+0x147 #6 0xffffffff810c792e at fast_syscall_common+0xf8 nlink=1, effnlink=1, size=300182, extsize 0 generation=dbdaa1b7, uid=0, gid=0, flags=0x0 ino 9951869, on dev gpt/work vnode 0xfffffe0143ef0960: type VDIR usecount 3, writecount 0, refcount 2 seqc users 1 mountedhere 0xfffffe010e153000 hold count flags () flags (VIRF_MOUNTPOINT) v_object 0xfffff801d14e6c60 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0xfffffe00e4ceac80 (pid 0, kernel, tid 100048) with shared waiters pending #0 0xffffffff80bf1ee7 at lockmgr_lock_flags+0x1e7 #1 0xffffffff80f5ae9d at ffs_lock+0x9d #2 0xffffffff811c113c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d327b4 at _vn_lock+0x54 #4 0xffffffff80d070e3 at dounmount+0x503 #5 0xffffffff80f52866 at ffs_fsfail_unmount+0x56 #6 0xffffffff80c8461a at taskqueue_run_locked+0xaa #7 0xffffffff80c856a2 at taskqueue_thread_loop+0xc2 #8 0xffffffff80bdb8a0 at fork_exit+0x80 #9 0xffffffff810c808e at fork_trampoline+0xe nlink=2, effnlink=2, size=1024, extsize 0 generation=74d5cf41, uid=0, gid=0, flags=0x0 ino 37078272, on dev gpt/work db:1:pho1> show mount 0xfffffe010e154700 /dev/gpt/work on / (ufs) 0xfffffe010e155280 devfs on /dev (devfs) 0xfffffe014285b280 /dev/gpt/tmp on /tmp (ufs) 0xfffffe010e153000 /dev/md10 on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 103672 Buf domain 0 freebufs 14947 lofreebufs 219 hifreebufs 329 bufspace 2883584 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 10 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 144 (1871872) dirtyq count 10 (327680) wakeup 0 lim 25 CPU 0, 8, 0, 1, 11, 0, 0, 1, 1, 7, 1, 1, Locked buffers: 1 space 32768 Total buffers: 36004 space 2883584 Buf domain 1 freebufs 14949 lofreebufs 219 hifreebufs 329 bufspace 2285568 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 171 (2232320) dirtyq count 0 (0) wakeup 0 lim 25 CPU 0, 0, 1, 0, 10, 0, 0, 0, 1, 1, 0, 0, Locked buffers: 0 space 0 Total buffers: 15220 space 2285568 Buf domain 2 freebufs 14983 lofreebufs 219 hifreebufs 329 bufspace 1839104 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 111 (1466368) dirtyq count 0 (0) wakeup 0 lim 25 CPU 0, 1, 4, 2, 19, 0, 9, 0, 0, 3, 1, 0, Locked buffers: 0 space 0 Total buffers: 8555 space 1839104 Buf domain 3 freebufs 14286 lofreebufs 219 hifreebufs 329 bufspace 24494080 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 742 (21540864) dirtyq count 0 (0) wakeup 0 lim 25 CPU 3, 1, 7, 4, 12, 1, 4, 22, 0, 20, 22, 9, Locked buffers: 0 space 0 Total buffers: 11759 space 24494080 Buf domain 4 freebufs 14706 lofreebufs 219 hifreebufs 329 bufspace 10452992 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 354 (8339456) dirtyq count 0 (0) wakeup 0 lim 25 CPU 24, 0, 4, 6, 3, 5, 5, 1, 16, 3, 3, 2, Locked buffers: 1 space 32768 Total buffers: 14168 space 10452992 Buf domain 5 freebufs 14979 lofreebufs 219 hifreebufs 329 bufspace 1978368 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 116 (1679360) dirtyq count 0 (0) wakeup 0 lim 25 CPU 2, 0, 2, 0, 7, 1, 1, 2, 0, 23, 0, 0, Locked buffers: 0 space 0 Total buffers: 9006 space 1978368 Buf domain 6 freebufs 14965 lofreebufs 219 hifreebufs 329 bufspace 2428928 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 1 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 107 (1785856) dirtyq count 1 (8192) wakeup 0 lim 25 CPU 1, 15, 1, 1, 18, 0, 7, 0, 2, 15, 0, 0, Locked buffers: 0 space 0 Total buffers: 11219 space 2428928 db:1:pho1> show page vm_cnt.v_free_count: 7932432 vm_cnt.v_inactive_count: 1069 vm_cnt.v_active_count: 1879 vm_cnt.v_laundry_count: 28 vm_cnt.v_wire_count: 186923 vm_cnt.v_free_reserved: 10643 vm_cnt.v_free_min: 51261 vm_cnt.v_free_target: 173115 vm_cnt.v_inactive_target: 259672 db:1:pho1> show pageq pq_free 7932432 dom 0 page_cnt 8123602 free 7932432 pq_act 1879 pq_inact 1069 pq_laund 28 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000145 | 000000 11 (008192K) | 000563 | 000000 10 (004096K) | 001492 | 000000 09 (002048K) | 002323 | 000000 08 (001024K) | 003251 | 000000 07 (000512K) | 007051 | 000004 06 (000256K) | 009274 | 000007 05 (000128K) | 013920 | 000017 04 (000064K) | 017408 | 000063 03 (000032K) | 021950 | 000277 02 (000016K) | 028406 | 000271 01 (000008K) | 031079 | 001005 00 (000004K) | 030260 | 000067 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000000 | 000000 05 (000128K) | 000002 | 000000 04 (000064K) | 000003 | 000000 03 (000032K) | 000001 | 000000 02 (000016K) | 000002 | 000000 01 (000008K) | 000001 | 000000 00 (000004K) | 000001 | 000000 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Total Mem XFree malloc-4096 4096 34128 0 132713 0 2 139788288 0 RADIX NODE 144 8059 527830 106296560 0 64 77168016 0 FFS inode 1160 13146 28609 41036005 0 8 48435800 0 VNODE 600 13272 39118 41036691 0 16 31434000 0 VM OBJECT 264 13150 92300 38722343 0 30 27838800 0 mbuf_cluster 2048 12706 16 53233 0 8 26054656 0 BUF TRIE 144 227 105725 12819465 0 62 15257088 0 swblk 136 1886 68033 2073203 0 62 9508984 0 FFS2 dinode 256 13146 9834 41035851 0 62 5882880 0 malloc-128 128 40461 428 11986152 0 126 5233792 0 UMA Slabs 0 112 42042 1335 247811 0 126 4858224 0 mbuf 256 16419 2046 41870 0 254 4727040 0 S VFS Cache 104 7521 20988 25626530 0 126 2964936 0 malloc-128 128 14127 5589 93283930 0 126 2523648 0 pbuf 2776 0 788 0 0 0 2187488 0 vmem btag 56 25046 10438 793176 0 254 1987104 0 malloc-32768 32768 10 37 5157 0 1 1540096 0 malloc-4096 4096 332 0 7227 0 2 1359872 0 THREAD 1808 576 40 762 0 8 1113728 0 swpctrie 144 307 7225 208652 0 62 1084608 0 malloc-64 64 9613 4499 1600487 0 254 903168 0 VMSPACE 2544 102 237 789916 0 4 862416 0 malloc-64 64 9761 3217 1440660 0 254 830592 0 malloc-65536 65536 3 9 4133 0 1 786432 0 malloc-4096 4096 139 43 790695 0 2 745472 0 malloc-8192 8192 90 0 276 0 1 737280 0 256 Bucket 2048 266 36 12827198 0 8 618496 0 malloc-65536 65536 1 7 157 0 1 524288 0 MAP ENTRY 96 2845 2447 34508900 0 126 508032 0 PROC 1336 136 239 790134 0 8 501000 0 UMA Zones 2048 228 7 228 0 8 481280 0 malloc-384 384 390 830 47734935 0 30 468480 0 malloc-32768 32768 13 0 13 0 1 425984 0 128 Bucket 1024 263 152 14055559 0 16 424960 0 malloc-32768 32768 12 0 12 0 1 393216 0 lkpimm 160 2 2323 2 0 62 372000 0 lkpicurr 160 9 2316 9 0 62 372000 0 malloc-32 32 8624 1960 40370 0 254 338688 0 filedesc0 1072 136 158 790135 0 8 315168 0 malloc-2048 2048 123 27 543 0 8 307200 0 64 Bucket 512 223 369 21423616 0 30 303104 0 malloc-256 256 732 423 139334829 0 62 295680 0 malloc-64 64 4147 452 5009 0 254 294336 0 malloc-256 256 895 215 6926 0 62 284160 0 ksiginfo 112 350 2062 41179 0 126 270144 0 malloc-16 16 15319 1431 15502 0 254 268000 0 malloc-65536 65536 4 0 9 0 1 262144 0 malloc-1024 1024 202 54 5594 0 16 262144 0 pipe 744 20 325 439926 0 16 256680 0 32 Bucket 256 169 731 19852842 0 62 230400 0 malloc-16384 16384 14 0 4834 0 1 229376 0 malloc-8192 8192 28 0 452 0 1 229376 0 g_bio 408 1 509 35795465 0 30 208080 0 Files 80 170 2380 124504418 0 126 204000 0 malloc-128 128 870 649 100534917 0 126 194432 0 socket 944 52 140 7614 0 16 181248 0 malloc-8192 8192 19 0 3069 0 1 155648 0 4 Bucket 48 30 3078 3164029 0 254 149184 0 PWD 32 54 4356 10809306 0 254 141120 0 8 Bucket 80 110 1540 15905213 0 126 132000 0 malloc-65536 65536 2 0 2 0 1 131072 0 malloc-65536 65536 2 0 2986 0 1 131072 0 malloc-1024 1024 66 62 753509 0 16 131072 0 fakepg 104 0 1170 2976 0 126 121680 0 malloc-32 32 458 3322 193391 0 254 120960 0 16 Bucket 144 204 608 14854232 0 62 116928 0 mbuf_packet 256 8 412 11703 0 254 107520 0 malloc-256 256 181 224 1521 0 62 103680 0 ada_ccb 272 0 378 1275821 0 30 102816 0 DEVCTL 1024 0 100 3361 0 0 102400 0 TURNSTILE 136 617 118 617 0 62 99960 0 malloc-8192 8192 12 0 12 0 1 98304 0 malloc-128 128 435 247 5750246 0 126 87296 0 rl_entry 40 340 1781 340 0 254 84840 0 malloc-128 128 69 582 1455250397 0 126 83328 0 malloc-16384 16384 5 0 5 0 1 81920 0 PGRP 88 33 887 496 0 126 80960 0 malloc-64 64 36 1224 1861110658 0 254 80640 0 UMA Kegs 384 197 6 197 0 30 77952 0 malloc-4096 4096 19 0 2932 0 2 77824 0 malloc-384 384 135 65 17590 0 30 76800 0 ttyinq 160 330 145 540 0 62 76000 0 NAMEI 1024 13 59 708754736 0 16 73728 0 udp_inpcb 496 22 122 617 0 30 71424 0 Mountpoints 2944 4 20 190 0 4 70656 0 SLEEPQUEUE 88 617 183 617 0 126 70400 0 malloc-2048 2048 10 24 8236 0 8 69632 0 swrbuf 2776 0 24 123369 0 2 66624 0 malloc-65536 65536 1 0 1 0 1 65536 0 malloc-32768 32768 2 0 33 0 1 65536 0 malloc-16384 16384 4 0 12966 0 1 65536 0 malloc-16384 16384 4 0 5 0 1 65536 0 malloc-1024 1024 10 54 1988576 0 16 65536 0 ttyoutq 256 173 82 282 0 62 65280 0 malloc-256 256 33 222 13780 0 62 65280 0 malloc-256 256 37 218 29509259 0 62 65280 0 unpcb 256 16 224 6929 0 62 61440 0 malloc-4096 4096 15 0 26 0 2 61440 0 malloc-1024 1024 17 43 1811 0 16 61440 0 malloc-32 32 613 1277 27098 0 254 60480 0 malloc-16 16 829 2421 70636 0 254 52000 0 malloc-128 128 52 351 1019414 0 126 51584 0 malloc-384 384 25 105 10877 0 30 49920 0 malloc-4096 4096 7 5 39 0 2 49152 0 tcpcb 1080 11 31 16 0 8 45360 0 pcpu-8 8 3502 2130 3509 0 254 45056 0 malloc-32 32 129 1257 182785 0 254 44352 0 2 Bucket 32 107 1279 2729337 0 254 44352 0 malloc-16 16 43 2707 5789 0 254 44000 0 malloc-16 16 568 2182 1094325 0 254 44000 0 malloc-256 256 0 165 234194178 0 62 42240 0 malloc-512 512 42 38 772009 0 30 40960 0 cpuset 104 27 345 27 0 126 38688 0 malloc-1024 1024 19 17 38 0 16 36864 0 malloc-512 512 31 41 229 0 30 36864 0 pcpu-64 64 451 125 451 0 254 36864 0 udpcb 32 22 1112 617 0 254 36288 0 malloc-32 32 9 1125 2236 0 254 36288 0 malloc-384 384 52 38 52 0 30 34560 0 malloc-32768 32768 1 0 1 0 1 32768 0 malloc-128 128 1 216 345437 0 126 27776 0 malloc-1024 1024 15 9 27 0 16 24576 0 rtentry 176 12 126 15 0 62 24288 0 malloc-64 64 74 304 133207 0 254 24192 0 malloc-32 32 134 622 349 0 254 24192 0 tcp_inpcb 496 11 37 16 0 30 23808 0 malloc-16 16 33 1217 47 0 254 20000 0 malloc-16 16 31 1219 60 0 254 20000 0 malloc-128 128 42 113 52 0 126 19840 0 malloc-384 384 13 37 14 0 30 19200 0 malloc-16384 16384 1 0 1 0 1 16384 0 malloc-8192 8192 2 0 2 0 1 16384 0 malloc-4096 4096 4 0 5 0 2 16384 0 malloc-2048 2048 5 3 460 0 8 16384 0 malloc-512 512 6 26 52 0 30 16384 0 pcpu-16 16 24 1000 24 0 254 16384 0 malloc-64 64 28 224 25010 0 254 16128 0 malloc-32 32 24 480 234 0 254 16128 0 routing nhops 256 9 51 15 0 62 15360 0 malloc-384 384 31 9 7888 0 30 15360 0 malloc-256 256 20 40 20 0 62 15360 0 malloc-64 64 4 185 8 0 254 12096 0 malloc-256 256 18 27 30 0 62 11520 0 malloc-1024 1024 3 5 3136 0 16 8192 0 kenv 258 18 12 21840 0 30 7740 0 FPU_save_area 832 1 8 1 0 16 7488 0 DIRHASH 1024 2 2 7795166 0 16 4096 0 epoch_record pcpu 256 4 12 4 0 62 4096 0 malloc-4096 4096 1 0 410 0 2 4096 0 malloc-2048 2048 2 0 2 0 8 4096 0 malloc-2048 2048 2 0 2 0 8 4096 0 malloc-1024 1024 4 0 4 0 16 4096 0 malloc-512 512 1 7 1895 0 30 4096 0 malloc-64 64 8 55 9 0 254 4032 0 vtnet_tx_hdr 24 0 167 0 0 254 4008 0 malloc-16 16 15 235 170282 0 254 4000 0 malloc-16 16 2 248 3 0 254 4000 0 ripcb 496 1 7 3 0 30 3968 0 UMA Slabs 1 176 10 12 10 0 62 3872 0 malloc-384 384 1 9 33 0 30 3840 0 KMAP ENTRY 96 33 6 33 0 0 3744 0 vmem 1856 1 1 1 0 8 3712 0 SMR CPU 32 3 92 3 0 254 3040 0 SMR SHARED 24 3 92 3 0 254 2280 0 FFS1 dinode 128 0 0 0 0 126 0 0 da_ccb 544 0 0 0 0 16 0 0 udplite_inpcb 496 0 0 0 0 30 0 0 tcp_log_node 120 0 0 0 0 126 0 0 tcp_log_bucket 176 0 0 0 0 62 0 0 tcp_log 416 0 0 0 0 30 0 0 tcpreass 48 0 0 0 0 254 0 0 tfo_ccache_entries 80 0 0 0 0 126 0 0 tfo 4 0 0 0 0 254 0 0 sackhole 32 0 0 0 0 254 0 0 hostcache 64 0 0 0 0 254 0 0 syncache 168 0 0 1 0 62 0 0 tcptw 88 0 0 0 0 126 0 0 ipq 56 0 0 0 0 254 0 0 IPsec SA lft_c 16 0 0 0 0 254 0 0 KNOTE 160 0 0 0 0 62 0 0 itimer 352 0 0 0 0 30 0 0 AIOLIO 272 0 0 0 0 30 0 0 AIOCB 552 0 0 0 0 16 0 0 AIOP 32 0 0 0 0 254 0 0 AIO 208 0 0 0 0 62 0 0 NCLNODE 736 0 0 0 0 16 0 0 TMPFS node 224 0 0 0 0 62 0 0 LTS VFS Cache 360 0 0 0 0 30 0 0 L VFS Cache 320 0 0 0 0 30 0 0 STS VFS Cache 144 0 0 0 0 62 0 0 cryptop 280 0 0 0 0 30 0 0 linux_dma_object 24 0 0 0 0 254 0 0 linux_dma_pctrie 144 0 0 0 0 62 0 0 IOMMU_MAP_ENTRY 120 0 0 0 0 126 0 0 ktls_session 192 0 0 0 0 62 0 0 mbuf_jumbo_16k 16384 0 0 0 0 1 0 0 mbuf_jumbo_9k 9216 0 0 0 0 1 0 0 mbuf_jumbo_page 4096 0 0 2 0 2 0 0 audit_record 1280 0 0 0 0 8 0 0 domainset 40 0 0 0 0 254 0 0 MAC labels 40 0 0 0 0 254 0 0 vnpbuf 2776 0 0 0 0 4 0 0 clpbuf 2776 0 0 455532 0 2 0 0 mdpbuf 2776 0 0 0 0 1 0 0 nfspbuf 2776 0 0 0 0 2 0 0 swwbuf 2776 0 0 1109747 0 1 0 0 umtx_shm 88 0 0 0 0 126 0 0 umtx pi 96 0 0 0 0 126 0 0 rangeset pctrie nodes 144 0 0 0 0 62 0 0 malloc-65536 65536 0 0 0 0 1 0 0 malloc-65536 65536 0 0 0 0 1 0 0 malloc-32768 32768 0 0 8 0 1 0 0 malloc-32768 32768 0 0 1015 0 1 0 0 malloc-32768 32768 0 0 0 0 1 0 0 malloc-16384 16384 0 0 12 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-8192 8192 0 0 2632 0 1 0 0 malloc-8192 8192 0 0 0 0 1 0 0 malloc-8192 8192 0 0 0 0 1 0 0 malloc-2048 2048 0 0 760 0 8 0 0 malloc-2048 2048 0 0 0 0 8 0 0 malloc-2048 2048 0 0 3 0 8 0 0 malloc-512 512 0 0 9358 0 30 0 0 malloc-512 512 0 0 0 0 30 0 0 malloc-512 512 0 0 0 0 30 0 0 malloc-512 512 0 0 0 0 30 0 0 malloc-384 384 0 0 0 0 30 0 0 malloc-32 32 0 0 2 0 254 0 0 pcpu-32 32 0 0 0 0 254 0 0 pcpu-4 4 0 0 0 0 254 0 0 UMA Hash 256 0 0 0 0 62 0 0 db:0:pho> show umacache Zone Size Used Free Requests Bucket debugnet_mbuf_packet 2048 0 0 0 0 debugnet_mbuf_cluster 2048 -6144 0 0 0 debugnet_mbuf 256 -10240 0 0 0 ktls_buffers 16384 0 0 0 1 vm pgcache 4096 80897 328 43748414 12 vm pgcache 4096 32109 670 457511582 18 buf free cache 848 2116 143 20511704 16 buffer arena-65 65536 0 0 0 1 buffer arena-61 61440 0 0 0 1 buffer arena-57 57344 0 0 0 1 buffer arena-53 53248 0 0 0 1 buffer arena-49 49152 0 0 0 1 buffer arena-45 45056 0 0 0 1 buffer arena-40 40960 0 0 0 1 buffer arena-36 36864 0 0 0 1 buffer arena-32 32768 1022 19 7700060 1 buffer arena-28 28672 0 0 0 1 buffer arena-24 24576 0 0 0 1 buffer arena-20 20480 0 0 0 1 buffer arena-16 16384 9 0 155819 1 buffer arena-12 12288 0 0 0 1 buffer arena-81 8192 0 0 0 1 buffer arena-40 4096 0 0 0 2 kstack_cache 16384 576 0 762 1 db:0:pho> acttrace Tracing command kernel pid 0 tid 100048 td 0xfffffe00e4ceac80 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce3ae00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce3ae10 trap() at trap+0x56/frame 0xfffffe003ce3af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3af20 --- trap 0x13, rip = 0xffffffff80c91baa, rsp = 0xfffffe00e4894660, rbp = 0xfffffe00e4894820 --- witness_checkorder() at witness_checkorder+0x26a/frame 0xfffffe00e4894820 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe00e4894870 __mnt_vnode_next_all() at __mnt_vnode_next_all+0x49/frame 0xfffffe00e48948b0 ffs_sync() at ffs_sync+0x304/frame 0xfffffe00e4894940 vfs_write_suspend() at vfs_write_suspend+0x165/frame 0xfffffe00e4894990 vfs_write_suspend_umnt() at vfs_write_suspend_umnt+0x35/frame 0xfffffe00e48949c0 ffs_unmount() at ffs_unmount+0xe9/frame 0xfffffe00e4894a10 dounmount() at dounmount+0x95c/frame 0xfffffe00e4894ac0 ffs_fsfail_unmount() at ffs_fsfail_unmount+0x56/frame 0xfffffe00e4894b00 taskqueue_run_locked() at taskqueue_run_locked+0xaa/frame 0xfffffe00e4894b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe00e4894bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4894bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4894bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00e4c5b3a0 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81ff9220 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81ff9230 trap() at trap+0x56/frame 0xffffffff81ff9340 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81ff9340 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe0037790a60, rbp = 0xfffffe0037790a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0037790a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe0037790aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe0037790ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe0037790ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe0037790bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037790bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037790bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00e4c5a560 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce2de00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce2de10 trap() at trap+0x56/frame 0xfffffe003ce2df20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce2df20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe003779aa60, rbp = 0xfffffe003779aa60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe003779aa60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe003779aaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe003779aac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe003779aae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe003779abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003779abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003779abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00e4c59720 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce47e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce47e10 trap() at trap+0x56/frame 0xfffffe003ce47f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce47f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377a4a60, rbp = 0xfffffe00377a4a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377a4a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377a4aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377a4ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377a4ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377a4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377a4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377a4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00e4c5d740 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce61e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce61e10 trap() at trap+0x56/frame 0xfffffe003ce61f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce61f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377aea60, rbp = 0xfffffe00377aea60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377aea60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377aeaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377aeac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377aeae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377aebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377aebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377aebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00e4c5c900 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce7be00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce7be10 trap() at trap+0x56/frame 0xfffffe003ce7bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce7bf20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377b8a60, rbp = 0xfffffe00377b8a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377b8a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377b8aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377b8ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377b8ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377b8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffffe00e4c611e0 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce88e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce88e10 trap() at trap+0x56/frame 0xfffffe003ce88f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce88f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377b3a60, rbp = 0xfffffe00377b3a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377b3a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377b3aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377b3ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377b3ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377b3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffffe00e4c60ac0 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce95e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce95e10 trap() at trap+0x56/frame 0xfffffe003ce95f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce95f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377c2a60, rbp = 0xfffffe00377c2a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377c2a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377c2aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377c2ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377c2ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377c2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377c2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00e4c603a0 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003cea2e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003cea2e10 trap() at trap+0x56/frame 0xfffffe003cea2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea2f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377bda60, rbp = 0xfffffe00377bda60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377bda60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377bdaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377bdac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377bdae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377bdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377bdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377bdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100134 td 0xfffffe00e4d3d3a0 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce20e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce20e10 trap() at trap+0x56/frame 0xfffffe003ce20f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce20f20 --- trap 0x13, rip = 0xffffffff810d4ff0, rsp = 0xfffffe00e4e81740, rbp = 0xfffffe00e4e817b0 --- smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x3b0/frame 0xfffffe00e4e817b0 pmap_invalidate_range() at pmap_invalidate_range+0x16f/frame 0xfffffe00e4e81820 allocbuf() at allocbuf+0xfd/frame 0xfffffe00e4e81880 brelse() at brelse+0x1e6/frame 0xfffffe00e4e818e0 bufdone() at bufdone+0x3b3/frame 0xfffffe00e4e81950 ffs_backgroundwritedone() at ffs_backgroundwritedone+0x170/frame 0xfffffe00e4e81980 bufdone() at bufdone+0x6c/frame 0xfffffe00e4e819f0 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81a40 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81a90 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81ae0 g_disk_done() at g_disk_done+0xed/frame 0xfffffe00e4e81b20 xpt_done_process() at xpt_done_process+0x3dd/frame 0xfffffe00e4e81b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe00e4e81bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e81bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e81bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 90885 tid 100341 td 0xfffffe0144128c80 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce54e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce54e10 trap() at trap+0x56/frame 0xfffffe003ce54f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce54f20 --- trap 0x13, rip = 0xffffffff80c6ffd0, rsp = 0xfffffe01444636e0, rbp = 0xfffffe01444636f0 --- lock_delay() at lock_delay+0x20/frame 0xfffffe01444636f0 __mtx_lock_sleep() at __mtx_lock_sleep+0xe8/frame 0xfffffe0144463780 __mtx_lock_flags() at __mtx_lock_flags+0xef/frame 0xfffffe01444637d0 proc_to_reap() at proc_to_reap+0x63/frame 0xfffffe0144463820 kern_wait6() at kern_wait6+0x1bb/frame 0xfffffe01444638c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0144463ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0144463bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0144463bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffdbb8, rbp = 0x7fffffffdc60 --- Tracing command sh pid 90887 tid 100423 td 0xfffffe0144221020 (CPU 7) kdb_enter() at kdb_enter+0x37/frame 0xfffffe0145204830 hardclock() at hardclock+0x1d0/frame 0xfffffe0145204870 handleevents() at handleevents+0xb4/frame 0xfffffe01452048b0 timercb() at timercb+0x196/frame 0xfffffe0145204900 lapic_handle_timer() at lapic_handle_timer+0x9b/frame 0xfffffe0145204930 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe0145204930 --- interrupt, rip = 0xffffffff80c93348, rsp = 0xfffffe0145204a00, rbp = 0xfffffe0145204a30 --- witness_unlock() at witness_unlock+0x1b8/frame 0xfffffe0145204a30 __mtx_unlock_flags() at __mtx_unlock_flags+0x4d/frame 0xfffffe0145204a60 userret() at userret+0x19b/frame 0xfffffe0145204ac0 amd64_syscall() at amd64_syscall+0x205/frame 0xfffffe0145204bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145204bf0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8012094ca, rsp = 0x7fffffffdca8, rbp = 0x7fffffffdd30 --- db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 90887 90885 2893 0 R CPU 7 sh 90885 90674 2893 0 R CPU 5 sh 90674 2942 2893 0 S piperd 0xfffff802aee3e5d0 sh 78678 78676 2808 0 S+ piperd 0xfffff807f19d5000 grep 78677 78676 2808 0 D+ ufs 0xfffffe0143ef09d0 ls 78676 78666 2808 0 S+ wait 0xfffff800096d7538 sh 78666 2808 2808 0 S+ piperd 0xfffff801b8d0d2e8 sh 63634 63632 2603 0 D ufs 0xfffffe0143ef09d0 ls 63632 2603 2603 0 S wait 0xfffff801adb88a70 sh 62490 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62489 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62488 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62487 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62486 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62485 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62484 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62483 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62482 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62481 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62480 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62479 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62478 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62477 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62476 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62474 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62472 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62471 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62469 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62466 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62465 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62464 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkfifo 62463 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62461 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62460 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62457 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62456 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62455 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62454 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkfifo 62452 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62451 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62450 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62449 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkfifo 62447 1 62392 2007 DW+ suspfs 0xfffffe010e153008 mkdir 62446 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62444 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62442 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rename 62440 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62437 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62436 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62435 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62434 1 62392 2007 DW+ suspfs 0xfffffe010e153008 link 62433 1 62392 2007 DW+ suspfs 0xfffffe010e153008 openat 62432 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62431 1 62392 2007 DW+ suspfs 0xfffffe010e153008 openat 62429 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rw 62428 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62427 1 62392 2007 DW+ suspfs 0xfffffe010e153008 openat 62426 1 62392 2007 DW+ suspfs 0xfffffe010e153008 rw 62425 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62424 1 62392 2007 DW+ suspfs 0xfffffe010e153008 openat 62423 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62422 1 62392 2007 DW+ suspfs 0xfffffe010e153008 openat 62420 1 62392 2007 DW+ suspfs 0xfffffe010e153008 symlink 62419 1 62392 2007 DW+ suspfs 0xfffffe010e153008 lockf2 62417 1 62392 2007 DW+ suspfs 0xfffffe010e153008 lockf2 62414 1 62392 2007 DW+ suspfs 0xfffffe010e153008 lockf2 2945 2893 2893 0 S piperd 0xfffff801d10b7ba0 sh 2943 2893 2893 0 S piperd 0xfffff8000417b5d0 awk 2942 2893 2893 0 S wait 0xfffff80272a34538 sh 2893 1 2893 0 SWs wait 0xfffff80272beca70 sh 2808 2807 2808 0 S+ wait 0xfffff80009837538 sh 2807 2805 2807 0 S+ select 0xfffff801b88b0440 bash 2805 2804 2805 0 SW+ pause 0xfffff8000611b0b0 csh 2804 2799 2804 2006 SW+ wait 0xfffff801ada9a000 su 2799 2798 2799 2006 SWs+ wait 0xfffff8000970ea70 bash 2798 2795 2795 2006 S select 0xfffff801adbc0cc0 sshd 2795 2729 2795 0 Ss select 0xfffff8000973e740 sshd 2756 1 2756 0 Ss+ ttyin 0xfffff80003c558b0 getty 2755 1 2755 0 Ss+ ttyin 0xfffff80003c55cb0 getty 2754 1 2754 0 Ss+ ttyin 0xfffff800097d28b0 getty 2753 1 2753 0 Ss+ ttyin 0xfffff800097d2cb0 getty 2752 1 2752 0 Ss+ ttyin 0xfffff800097d90b0 getty 2751 1 2751 0 Ss+ ttyin 0xfffff800097d94b0 getty 2750 1 2750 0 Ss+ ttyin 0xfffff800097d98b0 getty 2749 1 2749 0 Ss+ ttyin 0xfffff800097d9cb0 getty 2748 1 2748 0 Ss+ ttyin 0xfffff80003d4c0b0 getty 2747 1 2747 0 Ss+ ttyin 0xfffff80003d4c4b0 getty 2733 1 2733 0 Ss nanslp 0xffffffff81ebc0a0 cron 2729 1 2729 0 Ss select 0xfffff8000993f940 sshd 2710 1 2710 0 Ss select 0xfffff8000993f840 inetd 2684 1 2684 123 Ss select 0xfffff801b8ebfdc0 ntpd 2658 1 2658 0 Ss select 0xfffff8000973e4c0 powerd 2653 1 2653 0 Ss rpcsvc 0xfffff800095f8d60 NLM: master 2650 1 2650 0 Ss select 0xfffff801b8ebfe40 rpc.statd 2625 1 2625 0 Ss select 0xfffff801b8ebff40 rpcbind 2603 1 2603 0 Ss wait 0xfffff802728cba70 watchdogd 2595 1 2595 0 Ds getbuf 0xfffffe003c7b8e6c syslogd 2406 1 2406 0 Ss select 0xfffff801b8ebfec0 devd 2401 1 2401 65 Ss select 0xfffff801b88b05c0 dhclient 2342 1 2342 0 Ss select 0xfffff8000993f9c0 dhclient 2339 1 2339 0 Ss select 0xfffff8000993fa40 dhclient 34 0 0 0 DL syncer 0xffffffff81fab480 [syncer] 33 0 0 0 DL vlruwt 0xfffff8000436d538 [vnlru] 32 0 0 0 DL (threaded) [bufdaemon] 100213 D qsleep 0xffffffff81faa520 [bufdaemon] 100214 D - 0xffffffff81a0af40 [bufspacedaemon-0] 100215 D - 0xffffffff81a13200 [bufspacedaemon-1] 100216 D - 0xffffffff81a1b4c0 [bufspacedaemon-2] 100217 D - 0xffffffff81a23780 [bufspacedaemon-3] 100218 D - 0xffffffff81a2ba40 [bufspacedaemon-4] 100219 D - 0xffffffff81a33d00 [bufspacedaemon-5] 100220 D - 0xffffffff81a3bfc0 [bufspacedaemon-6] 100236 D sdflush 0xfffff8000983dce8 [/ worker] 100241 D sdflush 0xfffff80003cb78e8 [/tmp worker] 100762 D sdflush 0xfffff80343830ce8 [/mnt worker] 31 0 0 0 DL psleep 0xffffffff81fd0208 [vmdaemon] 30 0 0 0 DL (threaded) [pagedaemon] 100211 D psleep 0xffffffff81fc4678 [dom0] 100221 D launds 0xffffffff81fc4684 [laundry: dom0] 100222 D umarcl 0xffffffff80f7d410 [uma] 29 0 0 0 DL idle 0xfffff800099cf538 [enc_daemon1] 28 0 0 0 DL idle 0xfffff800099cfa70 [enc_daemon0] 27 0 0 0 DL - 0xffffffff81cf03d8 [rand_harvestq] 26 0 0 0 DL - 0xffffffff81fa7bdc [soaiod4] 25 0 0 0 DL - 0xffffffff81fa7bdc [soaiod3] 24 0 0 0 DL - 0xffffffff81fa7bdc [soaiod2] 23 0 0 0 DL - 0xffffffff81fa7bdc [soaiod1] 22 0 0 0 DL (threaded) [usb] 100140 D - 0xfffffe010a188440 [usbus0] 100141 D - 0xfffffe010a188498 [usbus0] 100142 D - 0xfffffe010a1884f0 [usbus0] 100143 D - 0xfffffe010a188548 [usbus0] 100144 D - 0xfffffe010a1885a0 [usbus0] 100146 D - 0xfffffe00e4d66cf0 [usbus1] 100147 D - 0xfffffe00e4d66d48 [usbus1] 100148 D - 0xfffffe00e4d66da0 [usbus1] 100149 D - 0xfffffe00e4d66df8 [usbus1] 100150 D - 0xfffffe00e4d66e50 [usbus1] 100151 D - 0xfffffe010a548cf0 [usbus2] 100152 D - 0xfffffe010a548d48 [usbus2] 100153 D - 0xfffffe010a548da0 [usbus2] 100154 D - 0xfffffe010a548df8 [usbus2] 100155 D - 0xfffffe010a548e50 [usbus2] 21 0 0 0 RL (threaded) [cam] 100133 D - 0xffffffff81bc7ac0 [doneq0] 100134 Run CPU 1 [doneq1] 100135 D - 0xffffffff81bc7bc0 [doneq2] 100136 D - 0xffffffff81bc7a40 [async] 100168 D - 0xffffffff81bc7910 [scanner] 20 0 0 0 DL seqstat 0xfffff80003d6b488 [sequencer 00] 19 0 0 0 DL crypto_ 0xfffff80003d4cfa0 [crypto returns 11] 18 0 0 0 DL crypto_ 0xfffff80003d4cf50 [crypto returns 10] 17 0 0 0 DL crypto_ 0xfffff80003d4cf00 [crypto returns 9] 16 0 0 0 DL crypto_ 0xfffff80003d4ceb0 [crypto returns 8] 15 0 0 0 DL crypto_ 0xfffff80003d4ce60 [crypto returns 7] 14 0 0 0 DL crypto_ 0xfffff80003d4ce10 [crypto returns 6] 9 0 0 0 DL crypto_ 0xfffff80003d4cdc0 [crypto returns 5] 8 0 0 0 DL crypto_ 0xfffff80003d4cd70 [crypto returns 4] 7 0 0 0 DL crypto_ 0xfffff80003d4cd20 [crypto returns 3] 6 0 0 0 DL crypto_ 0xfffff80003d4ccd0 [crypto returns 2] 5 0 0 0 DL crypto_ 0xfffff80003d4cc80 [crypto returns 1] 4 0 0 0 DL crypto_ 0xfffff80003d4cc30 [crypto returns 0] 3 0 0 0 DL crypto_ 0xffffffff81fc1ba0 [crypto] 13 0 0 0 DL (threaded) [geom] 100104 D - 0xffffffff81e9b920 [g_event] 100105 D - 0xffffffff81e9b928 [g_up] 100106 D - 0xffffffff81e9b930 [g_down] 2 0 0 0 DL (threaded) [KTLS] 100077 D - 0xfffff80003c22800 [thr_0] 100078 D - 0xfffff80003c22880 [thr_1] 100079 D - 0xfffff80003c22900 [thr_2] 100080 D - 0xfffff80003c22980 [thr_3] 100081 D - 0xfffff80003c22a00 [thr_4] 100082 D - 0xfffff80003c22a80 [thr_5] 100083 D - 0xfffff80003c22b00 [thr_6] 100084 D - 0xfffff80003c22b80 [thr_7] 100085 D - 0xfffff80003c22c00 [thr_8] 100086 D - 0xfffff80003c22c80 [thr_9] 100087 D - 0xfffff80003c22d00 [thr_10] 100088 D - 0xfffff80003c22d80 [thr_11] 12 0 0 0 WL (threaded) [intr] 100046 I [swi6: task queue] 100047 I [swi6: Giant taskq] 100049 I [swi5: fast taskq] 100089 I [swi4: clock (0)] 100090 I [swi4: clock (1)] 100091 I [swi4: clock (2)] 100092 I [swi4: clock (3)] 100093 I [swi4: clock (4)] 100094 I [swi4: clock (5)] 100095 I [swi4: clock (6)] 100096 I [swi4: clock (7)] 100097 I [swi4: clock (8)] 100098 I [swi4: clock (9)] 100099 I [swi4: clock (10)] 100100 I [swi4: clock (11)] 100101 I [swi1: netisr 0] 100102 I [swi3: vm] 100137 I [swi4: apei] 100138 I [irq56: ahci0] 100139 I [irq19: xhci0] 100145 I [irq18: ehci0 ehci1+] 100156 I [irq71: ahci1] 100157 I [swi0: uart uart] 100175 I [irq73: mlx5_core0] 100176 I [irq74: mlx5_core0] 100177 I [irq72: mlx5_core0] 100178 I [irq75: mlx5_core0] 100179 I [irq76: mlx5_core0] 100180 I [irq77: mlx5_core0] 100181 I [irq78: mlx5_core0] 100182 I [irq79: mlx5_core0] 100183 I [irq80: mlx5_core0] 100184 I [irq81: mlx5_core0] 100185 I [irq82: mlx5_core0] 100186 I [irq83: mlx5_core0] 100187 I [irq84: mlx5_core0] 100188 I [irq85: mlx5_core0] 100189 I [irq86: mlx5_core0] 100195 I [irq88: mlx5_core1] 100196 I [irq89: mlx5_core1] 100197 I [irq87: mlx5_core1] 100198 I [irq90: mlx5_core1] 100199 I [irq91: mlx5_core1] 100200 I [irq92: mlx5_core1] 100201 I [irq93: mlx5_core1] 100202 I [irq94: mlx5_core1] 100203 I [irq95: mlx5_core1] 100204 I [irq96: mlx5_core1] 100205 I [irq97: mlx5_core1] 100206 I [irq98: mlx5_core1] 100207 I [irq99: mlx5_core1] 100208 I [irq100: mlx5_core1] 100209 I [irq101: mlx5_core1] 100266 I [irq102: ioat0] 100267 I [irq103: ioat1] 100268 I [irq104: ioat2] 100269 I [irq105: ioat3] 100270 I [irq106: ioat4] 100271 I [irq107: ioat5] 100272 I [irq108: ioat6] 100273 I [irq109: ioat7] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 CanRun [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 CanRun [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 CanRun [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffff80003b43538 [init] 10 0 0 0 DL audit_w 0xffffffff81fc20b0 [audit] 0 0 0 0 RLs (threaded) [kernel] 100000 D swapin 0xffffffff81e9beb0 [swapper] 100015 D - 0xfffff80003965c00 [softirq_0] 100016 D - 0xfffff80003965b00 [softirq_1] 100017 D - 0xfffff80003965a00 [softirq_2] 100018 D - 0xfffff80003965900 [softirq_3] 100019 D - 0xfffff80003965800 [softirq_4] 100020 D - 0xfffff80003965700 [softirq_5] 100021 D - 0xfffff80003965600 [softirq_6] 100022 D - 0xfffff80003965500 [softirq_7] 100023 D - 0xfffff80003965400 [softirq_8] 100024 D - 0xfffff80003965300 [softirq_9] 100025 D - 0xfffff80003965200 [softirq_10] 100026 D - 0xfffff80003965100 [softirq_11] 100027 D - 0xfffff80003965000 [if_io_tqg_0] 100028 D - 0xfffff80003b1ce00 [if_io_tqg_1] 100029 D - 0xfffff80003b1cd00 [if_io_tqg_2] 100030 D - 0xfffff80003b1cc00 [if_io_tqg_3] 100031 D - 0xfffff80003b1cb00 [if_io_tqg_4] 100032 D - 0xfffff80003b1ca00 [if_io_tqg_5] 100033 D - 0xfffff80003b1c900 [if_io_tqg_6] 100034 D - 0xfffff80003b1c800 [if_io_tqg_7] 100035 D - 0xfffff80003b1c700 [if_io_tqg_8] 100036 D - 0xfffff80003b1c600 [if_io_tqg_9] 100037 D - 0xfffff80003b1c500 [if_io_tqg_10] 100038 D - 0xfffff80003b1c400 [if_io_tqg_11] 100039 D - 0xfffff80003b1c300 [if_config_tqg_0] 100040 D - 0xfffff80003b1fb00 [pci_hp taskq] 100041 D - 0xfffff80003b1fa00 [aiod_kick taskq] 100042 D - 0xfffff80003b1f900 [kqueue_ctx taskq] 100043 D - 0xfffff80003b1f800 [inm_free taskq] 100044 D - 0xfffff80003b1f700 [deferred_unmount ta] 100045 D - 0xfffff80003b1f600 [linuxkpi_irq_wq] 100048 Run CPU 3 [thread taskq] 100050 D - 0xfffff80003b12d00 [in6m_free taskq] 100051 D - 0xfffff80003b12c00 [linuxkpi_short_wq_0] 100052 D - 0xfffff80003b12c00 [linuxkpi_short_wq_1] 100053 D - 0xfffff80003b12c00 [linuxkpi_short_wq_2] 100054 D - 0xfffff80003b12c00 [linuxkpi_short_wq_3] 100055 D - 0xfffff80003b12c00 [linuxkpi_short_wq_4] 100056 D - 0xfffff80003b12c00 [linuxkpi_short_wq_5] 100057 D - 0xfffff80003b12c00 [linuxkpi_short_wq_6] 100058 D - 0xfffff80003b12c00 [linuxkpi_short_wq_7] 100059 D - 0xfffff80003b12c00 [linuxkpi_short_wq_8] 100060 D - 0xfffff80003b12c00 [linuxkpi_short_wq_9] 100061 D - 0xfffff80003b12c00 [linuxkpi_short_wq_1] 100062 D - 0xfffff80003b12c00 [linuxkpi_short_wq_1] 100063 D - 0xfffff80003b12c00 [linuxkpi_short_wq_1] 100064 D - 0xfffff80003b12b00 [linuxkpi_long_wq_0] 100065 D - 0xfffff80003b12b00 [linuxkpi_long_wq_1] 100066 D - 0xfffff80003b12b00 [linuxkpi_long_wq_2] 100067 D - 0xfffff80003b12b00 [linuxkpi_long_wq_3] 100068 D - 0xfffff80003b12b00 [linuxkpi_long_wq_4] 100069 D - 0xfffff80003b12b00 [linuxkpi_long_wq_5] 100070 D - 0xfffff80003b12b00 [linuxkpi_long_wq_6] 100071 D - 0xfffff80003b12b00 [linuxkpi_long_wq_7] 100072 D - 0xfffff80003b12b00 [linuxkpi_long_wq_8] 100073 D - 0xfffff80003b12b00 [linuxkpi_long_wq_9] 100074 D - 0xfffff80003b12b00 [linuxkpi_long_wq_10] 100075 D - 0xfffff80003b12b00 [linuxkpi_long_wq_11] 100076 D - 0xfffff80003b12b00 [linuxkpi_long_wq_12] 100103 D - 0xfffff80003bd0e00 [firmware taskq] 100107 D - 0xfffff80003bd0d00 [crypto_0] 100108 D - 0xfffff80003bd0d00 [crypto_1] 100109 D - 0xfffff80003bd0d00 [crypto_2] 100110 D - 0xfffff80003bd0d00 [crypto_3] 100111 D - 0xfffff80003bd0d00 [crypto_4] 100112 D - 0xfffff80003bd0d00 [crypto_5] 100113 D - 0xfffff80003bd0d00 [crypto_6] 100114 D - 0xfffff80003bd0d00 [crypto_7] 100115 D - 0xfffff80003bd0d00 [crypto_8] 100116 D - 0xfffff80003bd0d00 [crypto_9] 100117 D - 0xfffff80003bd0d00 [crypto_10] 100118 D - 0xfffff80003bd0d00 [crypto_11] 100158 D - 0xfffff80003c8bc00 [mca taskq] 100164 D - 0xfffff80003bd0b00 [acpi_task_0] 100165 D - 0xfffff80003bd0b00 [acpi_task_1] 100166 D - 0xfffff80003bd0b00 [acpi_task_2] 100167 D - 0xfffff80003bd0c00 [CAM taskq] 100171 D - 0xfffff80003bd0a00 [mlx5_core0-rec] 100172 D - 0xfffff80003bd0900 [mlx5_core0-wdg] 100173 D - 0xfffff80003bd0800 [mlx5_core0-cmd] 100174 D - 0xfffff80003b12a00 [mlx5_page_allocator] 100190 D - 0xfffff801b10e4900 [mlx5_esw_wq] 100191 D - 0xfffff801b10e4800 [mlx5_core1-rec] 100192 D - 0xfffff801b10e4700 [mlx5_core1-wdg] 100193 D - 0xfffff801b10e4600 [mlx5_core1-cmd] 100194 D - 0xfffff801b10e4500 [mlx5_page_allocator] 100210 D - 0xfffff801b88ba400 [mlx5_esw_wq] 90886 90885 2893 0 Z sh db:0:pho> allt Tracing command kernel pid 0 tid 100000 td 0xffffffff81e9c400 sched_switch() at sched_switch+0x5af/frame 0xffffffff8254eac0 mi_switch() at mi_switch+0x155/frame 0xffffffff8254eae0 sleepq_switch() at sleepq_switch+0x119/frame 0xffffffff8254eb20 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xffffffff8254eb60 _sleep() at _sleep+0x298/frame 0xffffffff8254ec00 swapper() at swapper+0x75/frame 0xffffffff8254ec60 mi_startup() at mi_startup+0x2f3/frame 0xffffffff8254ecb0 btext() at btext+0x22 Tracing command kernel pid 0 tid 100015 td 0xfffffe00e4c5fc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377ccab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377ccad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377ccb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377ccb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377ccbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377ccbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe00e4c5f560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377c7ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377c7ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377c7b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377c7b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377c7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377c7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xfffffe00e4c5ee40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377d6ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377d6ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377d6b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377d6b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377d6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377d6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377d6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe00e4c5e720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377d1ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377d1ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377d1b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377d1b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377d1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377d1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377d1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffffe00e4c5e000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377e0ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377e0ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377e0b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377e0b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377e0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377e0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377e0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe00e4c62740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377dbab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377dbad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377dbb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377dbb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffffe00e4c62020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377eaab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377eaad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377eab10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377eab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377eabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377eabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377eabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffffe00e4c61900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377e5ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377e5ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377e5b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377e5b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377e5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377e5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377e5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffffe00e4cde1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377f4ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377f4ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377f4b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377f4b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377f4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00e4cddac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377efab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377efad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377efb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377efb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377efbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377efbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377efbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00e4cdd3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377feab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377fead0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377feb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377feb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377febb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377febf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377febf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00e4cdcc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377f9ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377f9ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00377f9b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00377f9b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377f9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe00e4cdc560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4835ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4835ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4835b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4835b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4835bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4835bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4835bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffffe00e4cdbe40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4830ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4830ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4830b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4830b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4830bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4830bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4830bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffffe00e4cdb720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e483fab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e483fad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e483fb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e483fb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e483fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e483fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e483fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffffe00e4cdb000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e483aab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e483aad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e483ab10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e483ab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e483abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e483abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e483abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffffe00e4cdf740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4849ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4849ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4849b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4849b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4849bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4849bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4849bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffffe00e4cdf020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4844ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4844ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4844b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4844b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4844bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4844bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4844bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffffe00e4cde900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4853ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4853ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4853b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4853b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4853bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4853bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4853bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffffe00e4ce51e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e484eab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e484ead0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e484eb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e484eb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e484ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e484ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e484ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffffe00e4ce4ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e485dab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e485dad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e485db10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e485db80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e485dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e485dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e485dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffffe00e4ce43a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4858ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4858ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4858b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4858b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4858bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4858bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4858bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffffe00e4ce3c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4867ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4867ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4867b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4867b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4867bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4867bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4867bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffffe00e4ce3560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4862ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4862ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4862b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4862b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4862bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4862bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4862bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffffe00e4ce2e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4871ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4871ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4871b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4871b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4871bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4871bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4871bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffffe00e4ce2720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e486ca80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e486caa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e486cae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e486cb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e486cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e486cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e486cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffffe00e4ce2000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e487ba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e487baa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e487bae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e487bb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e487bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e487bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e487bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffffe00e4ce6740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4876a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4876aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4876ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4876b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4876bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4876bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4876bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffffe00e4ce6020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4885a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4885aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4885ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4885b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4885bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4885bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4885bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffffe00e4ce5900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4880a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4880aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4880ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4880b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4880bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4880bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4880bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffffe00e4cec1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e488fab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e488fad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e488fb10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e488fb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e488fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e488fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e488fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100048 td 0xfffffe00e4ceac80 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce3ae00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce3ae10 trap() at trap+0x56/frame 0xfffffe003ce3af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3af20 --- trap 0x13, rip = 0xffffffff80c91baa, rsp = 0xfffffe00e4894660, rbp = 0xfffffe00e4894820 --- witness_checkorder() at witness_checkorder+0x26a/frame 0xfffffe00e4894820 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe00e4894870 __mnt_vnode_next_all() at __mnt_vnode_next_all+0x49/frame 0xfffffe00e48948b0 ffs_sync() at ffs_sync+0x304/frame 0xfffffe00e4894940 vfs_write_suspend() at vfs_write_suspend+0x165/frame 0xfffffe00e4894990 vfs_write_suspend_umnt() at vfs_write_suspend_umnt+0x35/frame 0xfffffe00e48949c0 ffs_unmount() at ffs_unmount+0xe9/frame 0xfffffe00e4894a10 dounmount() at dounmount+0x95c/frame 0xfffffe00e4894ac0 ffs_fsfail_unmount() at ffs_fsfail_unmount+0x56/frame 0xfffffe00e4894b00 taskqueue_run_locked() at taskqueue_run_locked+0xaa/frame 0xfffffe00e4894b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe00e4894bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4894bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4894bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100050 td 0xfffffe00e4ce9e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e489ea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e489eaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e489eae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e489eb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e489ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e489ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e489ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100051 td 0xfffffe00e4ce9720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48ada80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48adaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48adae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48adb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48adbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48adbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48adbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100052 td 0xfffffe00e4ce9000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48a8a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48a8aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48a8ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48a8b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48a8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48a8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48a8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100053 td 0xfffffe00e4ced740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48b7a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48b7aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48b7ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48b7b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48b7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48b7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48b7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100054 td 0xfffffe00e4ced020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48b2a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48b2aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48b2ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48b2b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48b2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48b2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48b2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100055 td 0xfffffe00e4cec900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48c1a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48c1aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48c1ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48c1b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48c1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48c1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48c1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100056 td 0xfffffe00e4cf11e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48bca80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48bcaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48bcae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48bcb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48bcbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48bcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48bcbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100057 td 0xfffffe00e4cf0ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48cba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48cbaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48cbae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48cbb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48cbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48cbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48cbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xfffffe00e4cf03a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48c6a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48c6aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48c6ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48c6b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48c6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48c6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48c6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100059 td 0xfffffe00e4cefc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48d5a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48d5aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48d5ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48d5b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48d5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48d5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48d5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100060 td 0xfffffe00e4cef560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48d0a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48d0aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48d0ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48d0b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48d0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48d0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48d0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xfffffe00e4ceee40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48dfa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48dfaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48dfae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48dfb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48dfbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48dfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48dfbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100062 td 0xfffffe00e4cee720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48daa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48daaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48daae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48dab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48dabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48dabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48dabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffffe00e4cee000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48e9a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48e9aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48e9ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48e9b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48e9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48e9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48e9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffffe00e4cf2740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48e4a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48e4aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48e4ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48e4b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48e4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48e4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48e4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe00e4cf2020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48f3a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48f3aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48f3ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48f3b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48f3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48f3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48f3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffffe00e4cf1900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48eea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48eeaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48eeae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48eeb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48eebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48eebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48eebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffffe00e4cf61e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48fda80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48fdaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48fdae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48fdb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48fdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48fdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48fdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffffe00e4cf5ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e48f8a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e48f8aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e48f8ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e48f8b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48f8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48f8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48f8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffffe00e4cf53a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4907a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4907aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4907ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4907b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4907bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4907bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4907bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffffe00e4cf4c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4902a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4902aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4902ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4902b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4902bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4902bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4902bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffffe00e4cf4560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4911a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4911aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4911ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4911b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4911bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4911bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4911bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffffe00e4cf3e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e490ca80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e490caa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e490cae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e490cb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e490cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e490cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e490cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffffe00e4cf3720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e491ba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e491baa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e491bae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e491bb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e491bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e491bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e491bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffffe00e4cf3000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4916a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4916aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4916ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4916b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4916bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4916bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4916bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffffe00e4cf7740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4925a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4925aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4925ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4925b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4925bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4925bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4925bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100076 td 0xfffffe00e4cf7020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4920a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4920aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4920ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4920b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4920bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4920bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4920bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100103 td 0xfffffe00e4d08560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49eda80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49edaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e49edae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e49edb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e49edbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49edbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49edbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100107 td 0xfffffe00e4d0b740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e04a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e04aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e04ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e04b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e04bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e04bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e04bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100108 td 0xfffffe00e4d0b020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49fca80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49fcaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e49fcae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e49fcb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e49fcbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49fcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49fcbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100109 td 0xfffffe00e4d0a900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e0ea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e0eaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e0eae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e0eb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e0ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e0ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e0ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xfffffe00e4d241e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e09a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e09aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e09ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e09b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e09bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e09bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e09bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100111 td 0xfffffe00e4d23ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e18a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e18aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e18ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e18b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e18bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e18bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e18bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xfffffe00e4d233a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e13a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e13aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e13ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e13b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e13bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e13bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e13bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100113 td 0xfffffe00e4d22c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e22a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e22aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e22ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e22b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e22bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e22bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e22bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100114 td 0xfffffe00e4d22560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e1da80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e1daa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e1dae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e1db80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e1dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e1dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e1dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100115 td 0xfffffe00e4d21e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e2ca80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e2caa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e2cae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e2cb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e2cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e2cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e2cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100116 td 0xfffffe00e4d21720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e27a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e27aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e27ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e27b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e27bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e27bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e27bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100117 td 0xfffffe00e4d21000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e36a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e36aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e36ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e36b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e36bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e36bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e36bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100118 td 0xfffffe00e4d25740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e31a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e31aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e31ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e31b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e31bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e31bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e31bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100158 td 0xfffffe010a6d7000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f05ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f05ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f05b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4f05b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e4f05bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f05bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f05bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100164 td 0xfffffe00e4d3b720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f47ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f47ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f47b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4f47b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e4f47bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f47bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f47bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100165 td 0xfffffe00e4d3b000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f56ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f56ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f56b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4f56b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e4f56bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f56bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f56bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100166 td 0xfffffe010d676740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f51ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f51ad0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f51b10 msleep_spin_sbt() at msleep_spin_sbt+0x151/frame 0xfffffe00e4f51b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e4f51bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f51bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f51bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100167 td 0xfffffe010d676020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f60a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f60aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f60ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f60b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4f60bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f60bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f60bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100171 td 0xfffffe010dbae740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f80a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f80aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f80ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f80b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4f80bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f80bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f80bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100172 td 0xfffffe010dbae020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f7ba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f7baa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f7bae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f7bb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4f7bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f7bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f7bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100173 td 0xfffffe010dbad900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f8aa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f8aaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f8aae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f8ab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4f8abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f8abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f8abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100174 td 0xfffffe00e4cf6900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f94a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f94aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f94ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f94b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4f94bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f94bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f94bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100190 td 0xfffffe010dd3d560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4fe4a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4fe4aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4fe4ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4fe4b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4fe4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4fe4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4fe4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100191 td 0xfffffe010dd3ce40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4fdfa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4fdfaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4fdfae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4fdfb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4fdfbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4fdfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4fdfbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100192 td 0xfffffe010dd3c720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4feea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4feeaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4feeae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4feeb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4feebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4feebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4feebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100193 td 0xfffffe010dd3c000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4fe9a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4fe9aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4fe9ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4fe9b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4fe9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4fe9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4fe9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100194 td 0xfffffe010dd40740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ff8a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ff8aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ff8ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4ff8b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4ff8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ff8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ff8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100210 td 0xfffffe010e0ea3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e24aa80 mi_switch() at mi_switch+0x155/frame 0xfffffe010e24aaa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e24aae0 _sleep() at _sleep+0x32f/frame 0xfffffe010e24ab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010e24abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e24abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e24abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00e4c5bac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377816d0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377816f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0037781730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0037781780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0037781790 _sleep() at _sleep+0x325/frame 0xfffffe0037781830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe00377818c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0037781ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0037781bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0037781bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x28df8a, rsp = 0x7fffffffe718, rbp = 0x7fffffffe780 --- Tracing command KTLS pid 2 tid 100077 td 0xfffffe00e4d001e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49479b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49479d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4947a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4947ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4947bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4947bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4947bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100078 td 0xfffffe00e4cffac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49429b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49429d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4942a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4942ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4942bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4942bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4942bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100079 td 0xfffffe00e4cff3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49519b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49519d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4951a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4951ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4951bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4951bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4951bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100080 td 0xfffffe00e4cfec80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e494c9b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e494c9d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e494ca10 _sleep() at _sleep+0x32f/frame 0xfffffe00e494cab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e494cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e494cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e494cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100081 td 0xfffffe00e4cfe560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e495b9b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e495b9d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e495ba10 _sleep() at _sleep+0x32f/frame 0xfffffe00e495bab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e495bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e495bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e495bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100082 td 0xfffffe00e4cfde40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49569b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49569d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4956a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4956ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4956bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4956bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4956bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100083 td 0xfffffe00e4cfd720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49659b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49659d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4965a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4965ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4965bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4965bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4965bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100084 td 0xfffffe00e4cfd000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49609b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49609d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4960a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4960ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4960bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4960bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4960bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100085 td 0xfffffe00e4d01740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e496f9b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e496f9d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e496fa10 _sleep() at _sleep+0x32f/frame 0xfffffe00e496fab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e496fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e496fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e496fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100086 td 0xfffffe00e4d01020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e496a9b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e496a9d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e496aa10 _sleep() at _sleep+0x32f/frame 0xfffffe00e496aab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e496abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e496abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e496abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100087 td 0xfffffe00e4d00900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49799b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49799d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4979a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4979ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4979bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4979bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4979bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command KTLS pid 2 tid 100088 td 0xfffffe00e4d051e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49749b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49749d0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4974a10 _sleep() at _sleep+0x32f/frame 0xfffffe00e4974ab0 ktls_work_thread() at ktls_work_thread+0xdc/frame 0xfffffe00e4974bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4974bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4974bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100119 td 0xfffffe00e4d25020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e40a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e40a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e40ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e40b70 crypto_proc() at crypto_proc+0xbf/frame 0xfffffe00e4e40bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e40bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e40bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 0 pid 4 tid 100120 td 0xfffffe00e4d24900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e3ba70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e3ba90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e3bad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e3bb70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e3bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e3bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e3bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 1 pid 5 tid 100121 td 0xfffffe00e4d291e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e4aa70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e4aa90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e4aad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e4ab70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e4abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e4abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e4abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 2 pid 6 tid 100122 td 0xfffffe00e4d28ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e45a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e45a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e45ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e45b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e45bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e45bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e45bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 3 pid 7 tid 100123 td 0xfffffe00e4d283a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e54a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e54a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e54ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e54b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e54bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e54bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e54bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 4 pid 8 tid 100124 td 0xfffffe00e4d27c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e4fa70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e4fa90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e4fad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e4fb70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e4fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e4fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e4fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 5 pid 9 tid 100125 td 0xfffffe00e4d27560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e5ea70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e5ea90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e5ead0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e5eb70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e5ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e5ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e5ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command audit pid 10 tid 100001 td 0xfffffe00e4c5c1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe0037786a90 mi_switch() at mi_switch+0x155/frame 0xfffffe0037786ab0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0037786af0 _cv_wait() at _cv_wait+0x166/frame 0xfffffe0037786b50 audit_worker() at audit_worker+0x9f/frame 0xfffffe0037786bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037786bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037786bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00e4c5b3a0 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81ff9220 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81ff9230 trap() at trap+0x56/frame 0xffffffff81ff9340 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81ff9340 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe0037790a60, rbp = 0xfffffe0037790a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0037790a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe0037790aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe0037790ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe0037790ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe0037790bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037790bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037790bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00e4c5ac80 sched_switch() at sched_switch+0x5af/frame 0xfffffe003778bac0 mi_switch() at mi_switch+0x155/frame 0xfffffe003778bae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe003778bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003778bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003778bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00e4c5a560 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce2de00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce2de10 trap() at trap+0x56/frame 0xfffffe003ce2df20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce2df20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe003779aa60, rbp = 0xfffffe003779aa60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe003779aa60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe003779aaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe003779aac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe003779aae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe003779abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003779abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003779abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00e4c59e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe0037795ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe0037795ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe0037795bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037795bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037795bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00e4c59720 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce47e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce47e10 trap() at trap+0x56/frame 0xfffffe003ce47f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce47f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377a4a60, rbp = 0xfffffe00377a4a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377a4a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377a4aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377a4ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377a4ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377a4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377a4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377a4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffffe00e4c59000 sched_switch() at sched_switch+0x5af/frame 0xfffffe003779fac0 mi_switch() at mi_switch+0x155/frame 0xfffffe003779fae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe003779fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003779fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003779fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00e4c5d740 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce61e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce61e10 trap() at trap+0x56/frame 0xfffffe003ce61f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce61f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377aea60, rbp = 0xfffffe00377aea60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377aea60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377aeaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377aeac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377aeae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377aebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377aebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377aebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffffe00e4c5d020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00377a9ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00377a9ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377a9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377a9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377a9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00e4c5c900 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce7be00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce7be10 trap() at trap+0x56/frame 0xfffffe003ce7bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce7bf20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377b8a60, rbp = 0xfffffe00377b8a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377b8a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377b8aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377b8ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377b8ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377b8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffffe00e4c611e0 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce88e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce88e10 trap() at trap+0x56/frame 0xfffffe003ce88f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce88f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377b3a60, rbp = 0xfffffe00377b3a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377b3a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377b3aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377b3ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377b3ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377b3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffffe00e4c60ac0 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce95e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce95e10 trap() at trap+0x56/frame 0xfffffe003ce95f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce95f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377c2a60, rbp = 0xfffffe00377c2a60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377c2a60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377c2aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377c2ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377c2ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377c2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377c2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00e4c603a0 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003cea2e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003cea2e10 trap() at trap+0x56/frame 0xfffffe003cea2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea2f20 --- trap 0x13, rip = 0xffffffff810b1c36, rsp = 0xfffffe00377bda60, rbp = 0xfffffe00377bda60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377bda60 acpi_cpu_idle() at acpi_cpu_idle+0x288/frame 0xfffffe00377bdaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377bdac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377bdae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00377bdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377bdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377bdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100046 td 0xfffffe00e4cebac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e488ab00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e488ab20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e488abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e488abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e488abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100047 td 0xfffffe00e4ceb3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4899b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4899b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4899bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4899bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4899bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100049 td 0xfffffe00e4cea560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100089 td 0xfffffe00e4d04ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4983b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4983b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4983bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4983bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4983bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100090 td 0xfffffe00e4d043a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e497eb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e497eb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e497ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e497ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e497ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100091 td 0xfffffe00e4d03c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e498db00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e498db20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e498dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e498dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e498dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100092 td 0xfffffe00e4d03560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4988b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4988b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4988bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4988bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4988bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100093 td 0xfffffe00e4d02e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4997b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4997b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4997bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4997bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4997bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100094 td 0xfffffe00e4d02720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4992b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4992b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4992bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4992bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4992bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100095 td 0xfffffe00e4d02000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49a1b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49a1b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49a1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49a1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100096 td 0xfffffe00e4d06740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e499cb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e499cb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e499cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e499cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e499cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100097 td 0xfffffe00e4d06020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49abb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49abb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49abbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49abbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49abbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100098 td 0xfffffe00e4d05900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49a6b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49a6b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49a6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49a6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49a6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100099 td 0xfffffe00e4d0a1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49b5b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49b5b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49b5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100100 td 0xfffffe00e4d09ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49b0b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49b0b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49b0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100101 td 0xfffffe00e4d093a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49bfb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49bfb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49bfbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49bfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49bfbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100102 td 0xfffffe00e4d08c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100137 td 0xfffffe00e4d3f740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100138 td 0xfffffe00e4d3f020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e95b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e95b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4e95bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e95bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e95bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100139 td 0xfffffe00e4d3e900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ea4b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ea4b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4ea4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ea4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ea4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100145 td 0xfffffe010a18ae40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ec2b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ec2b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4ec2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ec2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ec2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100156 td 0xfffffe010a6d7e40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100157 td 0xfffffe010a6d7720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f0ab00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f0ab20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4f0abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f0abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f0abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100175 td 0xfffffe010dd0e1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f8fb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f8fb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4f8fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f8fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f8fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100176 td 0xfffffe010dd0dac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100177 td 0xfffffe010dd0d3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100178 td 0xfffffe010dd0cc80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100179 td 0xfffffe010dd0c560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100180 td 0xfffffe010dd0be40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100181 td 0xfffffe010dd0b720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100182 td 0xfffffe010dd0b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100183 td 0xfffffe010dd0f740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100184 td 0xfffffe010dd0f020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100185 td 0xfffffe010dd0e900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100186 td 0xfffffe010dd3f1e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100187 td 0xfffffe010dd3eac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100188 td 0xfffffe010dd3e3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100189 td 0xfffffe010dd3dc80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100195 td 0xfffffe010dd40020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ff3b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ff3b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4ff3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ff3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ff3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100196 td 0xfffffe010dd3f900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100197 td 0xfffffe010df861e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100198 td 0xfffffe010df85ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100199 td 0xfffffe010df853a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100200 td 0xfffffe010df84c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100201 td 0xfffffe010df84560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100202 td 0xfffffe010df83e40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100203 td 0xfffffe010df83720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100204 td 0xfffffe010df83000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100205 td 0xfffffe010df87740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100206 td 0xfffffe010df87020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100207 td 0xfffffe010df86900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100208 td 0xfffffe010e0eb1e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100209 td 0xfffffe010e0eaac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100266 td 0xfffffe0143628740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100267 td 0xfffffe0143628020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100268 td 0xfffffe0143627900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100269 td 0xfffffe0143a1a1e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100270 td 0xfffffe0143a19ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100271 td 0xfffffe0143a193a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100272 td 0xfffffe0143a18c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100273 td 0xfffffe0143a18560 fork_trampoline() at fork_trampoline Tracing command geom pid 13 tid 100104 td 0xfffffe00e4d07e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49e8a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49e8aa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e49e8ae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e49e8b80 g_run_events() at g_run_events+0x474/frame 0xfffffe00e49e8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49e8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49e8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100105 td 0xfffffe00e4d07720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49f7a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49f7a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e49f7ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e49f7b70 g_io_schedule_up() at g_io_schedule_up+0x3c/frame 0xfffffe00e49f7ba0 g_up_procbody() at g_up_procbody+0x5c/frame 0xfffffe00e49f7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49f7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49f7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100106 td 0xfffffe00e4d07000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e49f2a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e49f2a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e49f2ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e49f2b70 g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xfffffe00e49f2ba0 g_down_procbody() at g_down_procbody+0x5c/frame 0xfffffe00e49f2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49f2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49f2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 6 pid 14 tid 100126 td 0xfffffe00e4d26e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e59a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e59a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e59ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e59b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e59bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e59bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e59bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 7 pid 15 tid 100127 td 0xfffffe00e4d26720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e68a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e68a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e68ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e68b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e68bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e68bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e68bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 8 pid 16 tid 100128 td 0xfffffe00e4d26000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e63a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e63a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e63ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e63b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e63bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e63bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e63bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 9 pid 17 tid 100129 td 0xfffffe00e4d2a740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e72a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e72a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e72ad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e72b70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e72bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e72bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e72bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 10 pid 18 tid 100130 td 0xfffffe00e4d2a020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e6da70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e6da90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e6dad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e6db70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e6dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e6dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e6dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 11 pid 19 tid 100131 td 0xfffffe00e4d29900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e7ca70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e7ca90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e7cad0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e7cb70 crypto_ret_proc() at crypto_ret_proc+0x144/frame 0xfffffe00e4e7cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e7cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e7cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sequencer 00 pid 20 tid 100132 td 0xfffffe00e4d3e1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e77a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e77a90 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e77ad0 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4e77b30 seq_eventthread() at seq_eventthread+0x8c/frame 0xfffffe00e4e77bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e77bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e77bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100133 td 0xfffffe00e4d3dac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e86a60 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e86a80 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e86ac0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e86b60 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00e4e86bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e86bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e86bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100134 td 0xfffffe00e4d3d3a0 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce20e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce20e10 trap() at trap+0x56/frame 0xfffffe003ce20f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce20f20 --- trap 0x13, rip = 0xffffffff810d4ff0, rsp = 0xfffffe00e4e81740, rbp = 0xfffffe00e4e817b0 --- smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x3b0/frame 0xfffffe00e4e817b0 pmap_invalidate_range() at pmap_invalidate_range+0x16f/frame 0xfffffe00e4e81820 allocbuf() at allocbuf+0xfd/frame 0xfffffe00e4e81880 brelse() at brelse+0x1e6/frame 0xfffffe00e4e818e0 bufdone() at bufdone+0x3b3/frame 0xfffffe00e4e81950 ffs_backgroundwritedone() at ffs_backgroundwritedone+0x170/frame 0xfffffe00e4e81980 bufdone() at bufdone+0x6c/frame 0xfffffe00e4e819f0 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81a40 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81a90 g_io_deliver() at g_io_deliver+0x255/frame 0xfffffe00e4e81ae0 g_disk_done() at g_disk_done+0xed/frame 0xfffffe00e4e81b20 xpt_done_process() at xpt_done_process+0x3dd/frame 0xfffffe00e4e81b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe00e4e81bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e81bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e81bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100135 td 0xfffffe00e4d3cc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e90a60 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e90a80 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e90ac0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e90b60 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00e4e90bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e90bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e90bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100136 td 0xfffffe00e4d3c560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e8ba60 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e8ba80 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e8bac0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4e8bb60 xpt_async_td() at xpt_async_td+0x74/frame 0xfffffe00e4e8bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e8bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e8bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 21 tid 100168 td 0xfffffe010d675900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f5ba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f5baa0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f5bae0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f5bb80 xpt_scanner_thread() at xpt_scanner_thread+0x158/frame 0xfffffe00e4f5bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f5bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f5bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100140 td 0xfffffe010a18d1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4e9fac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4e9fae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4e9fb20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4e9fb80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4e9fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e9fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e9fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100141 td 0xfffffe010a18cac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4eaeac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4eaeae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4eaeb20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eaeb80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eaebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eaebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eaebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100142 td 0xfffffe010a18c3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ea9ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ea9ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ea9b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ea9b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ea9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ea9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ea9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100143 td 0xfffffe010a18bc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4eb8ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4eb8ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4eb8b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eb8b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eb8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eb8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eb8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100144 td 0xfffffe010a18b560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4eb3ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4eb3ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4eb3b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eb3b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eb3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eb3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eb3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100146 td 0xfffffe010a18a720 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ebdac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ebdae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ebdb20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ebdb80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ebdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ebdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ebdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100147 td 0xfffffe010a18a000 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4eccac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4eccae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4eccb20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eccb80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eccbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eccbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100148 td 0xfffffe010a18e740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ec7ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ec7ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ec7b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ec7b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ec7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ec7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ec7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100149 td 0xfffffe010a18e020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ed6ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ed6ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ed6b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ed6b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ed6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ed6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ed6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100150 td 0xfffffe010a18d900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ed1ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ed1ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ed1b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ed1b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ed1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ed1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ed1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100151 td 0xfffffe010a6da1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4eecac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4eecae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4eecb20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eecb80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eecbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eecbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eecbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100152 td 0xfffffe010a6d9ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ee7ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ee7ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ee7b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ee7b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ee7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ee7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ee7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100153 td 0xfffffe010a6d93a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ef6ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ef6ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ef6b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ef6b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ef6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ef6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ef6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100154 td 0xfffffe010a6d8c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4ef1ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4ef1ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4ef1b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ef1b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ef1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ef1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ef1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 22 tid 100155 td 0xfffffe010a6d8560 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f00ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f00ae0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f00b20 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f00b80 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f00bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f00bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f00bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 23 tid 100159 td 0xfffffe010a6db740 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f20a30 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f20a50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f20a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e4f20ad0 _sleep() at _sleep+0x298/frame 0xfffffe00e4f20b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00e4f20bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f20bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f20bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 24 tid 100160 td 0xfffffe010a6db020 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f1ba30 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f1ba50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f1ba90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e4f1bad0 _sleep() at _sleep+0x298/frame 0xfffffe00e4f1bb70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00e4f1bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f1bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f1bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 25 tid 100161 td 0xfffffe010a6da900 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f2aa30 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f2aa50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f2aa90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e4f2aad0 _sleep() at _sleep+0x298/frame 0xfffffe00e4f2ab70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00e4f2abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f2abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f2abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 26 tid 100162 td 0xfffffe010d6751e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f25a30 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f25a50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f25a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e4f25ad0 _sleep() at _sleep+0x298/frame 0xfffffe00e4f25b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00e4f25bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f25bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f25bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 27 tid 100163 td 0xfffffe00e4d3be40 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f4c9e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f4ca00 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f4ca40 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e4f4ca80 _sleep() at _sleep+0x298/frame 0xfffffe00e4f4cb20 random_kthread() at random_kthread+0x87/frame 0xfffffe00e4f4cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f4cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f4cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 28 tid 100169 td 0xfffffe010dbad1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f6aa40 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f6aa60 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f6aaa0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f6ab40 enc_daemon() at enc_daemon+0xf6/frame 0xfffffe00e4f6abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f6abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f6abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 29 tid 100170 td 0xfffffe010dbacac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe00e4f65a40 mi_switch() at mi_switch+0x155/frame 0xfffffe00e4f65a60 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00e4f65aa0 _sleep() at _sleep+0x32f/frame 0xfffffe00e4f65b40 enc_daemon() at enc_daemon+0xf6/frame 0xfffffe00e4f65bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f65bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f65bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100211 td 0xfffffe010e0ec740 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e260770 mi_switch() at mi_switch+0x155/frame 0xfffffe010e260790 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e2607d0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010e260810 _sleep() at _sleep+0x298/frame 0xfffffe010e2608b0 vm_pageout_worker() at vm_pageout_worker+0x29c/frame 0xfffffe010e260b70 vm_pageout() at vm_pageout+0x1d7/frame 0xfffffe010e260bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e260bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e260bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100221 td 0xfffffe010e15f740 sched_switch() at sched_switch+0x5af/frame 0xfffffe01123e9830 mi_switch() at mi_switch+0x155/frame 0xfffffe01123e9850 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01123e9890 _sleep() at _sleep+0x32f/frame 0xfffffe01123e9930 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xb9c/frame 0xfffffe01123e9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe01123e9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01123e9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100222 td 0xfffffe010e15f020 sched_switch() at sched_switch+0x5af/frame 0xfffffe01120e1a40 mi_switch() at mi_switch+0x155/frame 0xfffffe01120e1a60 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01120e1aa0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01120e1ae0 _sleep() at _sleep+0x298/frame 0xfffffe01120e1b80 uma_reclaim_worker() at uma_reclaim_worker+0xc8/frame 0xfffffe01120e1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe01120e1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01120e1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 31 tid 100212 td 0xfffffe010e0ec020 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e25ba40 mi_switch() at mi_switch+0x155/frame 0xfffffe010e25ba60 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e25baa0 _sleep() at _sleep+0x32f/frame 0xfffffe010e25bb40 vm_daemon() at vm_daemon+0x82/frame 0xfffffe010e25bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e25bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e25bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100213 td 0xfffffe010e0eb900 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e26aa20 mi_switch() at mi_switch+0x155/frame 0xfffffe010e26aa40 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e26aa80 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010e26aac0 _sleep() at _sleep+0x298/frame 0xfffffe010e26ab60 buf_daemon() at buf_daemon+0x116/frame 0xfffffe010e26abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e26abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e26abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100214 td 0xfffffe010e15e1e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe010ea7ca10 mi_switch() at mi_switch+0x155/frame 0xfffffe010ea7ca30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010ea7ca70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010ea7cab0 _sleep() at _sleep+0x298/frame 0xfffffe010ea7cb50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010ea7cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010ea7cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010ea7cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100215 td 0xfffffe010e15dac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e774a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010e774a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e774a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010e774ab0 _sleep() at _sleep+0x298/frame 0xfffffe010e774b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010e774bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e774bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e774bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100216 td 0xfffffe010e15d3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe010f18da10 mi_switch() at mi_switch+0x155/frame 0xfffffe010f18da30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010f18da70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010f18dab0 _sleep() at _sleep+0x298/frame 0xfffffe010f18db50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010f18dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010f18dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010f18dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100217 td 0xfffffe010e15cc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe010ef86a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010ef86a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010ef86a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010ef86ab0 _sleep() at _sleep+0x298/frame 0xfffffe010ef86b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010ef86bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010ef86bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010ef86bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100218 td 0xfffffe010e15c560 sched_switch() at sched_switch+0x5af/frame 0xfffffe010f6a8a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010f6a8a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010f6a8a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010f6a8ab0 _sleep() at _sleep+0x298/frame 0xfffffe010f6a8b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010f6a8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010f6a8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010f6a8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100219 td 0xfffffe010e15be40 sched_switch() at sched_switch+0x5af/frame 0xfffffe010f4a1a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010f4a1a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010f4a1a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010f4a1ab0 _sleep() at _sleep+0x298/frame 0xfffffe010f4a1b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010f4a1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010f4a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010f4a1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100220 td 0xfffffe010e15b720 sched_switch() at sched_switch+0x5af/frame 0xfffffe010fbb7a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010fbb7a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010fbb7a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010fbb7ab0 _sleep() at _sleep+0x298/frame 0xfffffe010fbb7b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe010fbb7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010fbb7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010fbb7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100236 td 0xfffffe014179a020 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bbd0a30 mi_switch() at mi_switch+0x155/frame 0xfffffe011bbd0a50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bbd0a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe011bbd0ad0 _sleep() at _sleep+0x298/frame 0xfffffe011bbd0b70 softdep_flush() at softdep_flush+0x188/frame 0xfffffe011bbd0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe011bbd0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011bbd0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100241 td 0xfffffe01417b7ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bbb7a30 mi_switch() at mi_switch+0x155/frame 0xfffffe011bbb7a50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bbb7a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe011bbb7ad0 _sleep() at _sleep+0x298/frame 0xfffffe011bbb7b70 softdep_flush() at softdep_flush+0x188/frame 0xfffffe011bbb7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe011bbb7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011bbb7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100762 td 0xfffffe01434c0000 sched_switch() at sched_switch+0x5af/frame 0xfffffe01453ffa30 mi_switch() at mi_switch+0x155/frame 0xfffffe01453ffa50 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01453ffa90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01453ffad0 _sleep() at _sleep+0x298/frame 0xfffffe01453ffb70 softdep_flush() at softdep_flush+0x188/frame 0xfffffe01453ffbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe01453ffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01453ffbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vnlru pid 33 tid 100223 td 0xfffffe011bdc21e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e265a10 mi_switch() at mi_switch+0x155/frame 0xfffffe010e265a30 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e265a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010e265ab0 _sleep() at _sleep+0x298/frame 0xfffffe010e265b50 vnlru_proc() at vnlru_proc+0x16d/frame 0xfffffe010e265bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe010e265bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010e265bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 34 tid 100224 td 0xfffffe011bdc1ac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bb8aa20 mi_switch() at mi_switch+0x155/frame 0xfffffe011bb8aa40 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bb8aa80 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe011bb8aac0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe011bb8ab40 sched_sync() at sched_sync+0x5cb/frame 0xfffffe011bb8abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe011bb8abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011bb8abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 90674 tid 100336 td 0xfffffe014401d000 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415e2840 mi_switch() at mi_switch+0x155/frame 0xfffffe01415e2860 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415e28a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415e28f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415e2900 _sleep() at _sleep+0x325/frame 0xfffffe01415e29a0 pipe_read() at pipe_read+0x4c6/frame 0xfffffe01415e2a00 dofileread() at dofileread+0x81/frame 0xfffffe01415e2a50 sys_read() at sys_read+0xc0/frame 0xfffffe01415e2ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415e2bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415e2bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8012096aa, rsp = 0x7fffffffde08, rbp = 0x7fffffffdf50 --- Tracing command sh pid 90885 tid 100341 td 0xfffffe0144128c80 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce54e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce54e10 trap() at trap+0x56/frame 0xfffffe003ce54f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce54f20 --- trap 0x13, rip = 0xffffffff80c6ffd0, rsp = 0xfffffe01444636e0, rbp = 0xfffffe01444636f0 --- lock_delay() at lock_delay+0x20/frame 0xfffffe01444636f0 __mtx_lock_sleep() at __mtx_lock_sleep+0xe8/frame 0xfffffe0144463780 __mtx_lock_flags() at __mtx_lock_flags+0xef/frame 0xfffffe01444637d0 proc_to_reap() at proc_to_reap+0x63/frame 0xfffffe0144463820 kern_wait6() at kern_wait6+0x1bb/frame 0xfffffe01444638c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0144463ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0144463bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0144463bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffdbb8, rbp = 0x7fffffffdc60 --- Tracing command sh pid 90886 tid 100365 td 0xfffffe014412a900 thread_exit() at thread_exit+0x203/frame 0xfffffe01444e7a40 exit1() at exit1+0xdde/frame 0xfffffe01444e7ab0 sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe01444e7ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01444e7bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01444e7bf0 --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x80120930a, rsp = 0x7fffffffe4a8, rbp = 0x7fffffffe5a0 --- Tracing command sh pid 90887 tid 100423 td 0xfffffe0144221020 (CPU 7) kdb_enter() at kdb_enter+0x37/frame 0xfffffe0145204830 hardclock() at hardclock+0x1d0/frame 0xfffffe0145204870 handleevents() at handleevents+0xb4/frame 0xfffffe01452048b0 timercb() at timercb+0x196/frame 0xfffffe0145204900 lapic_handle_timer() at lapic_handle_timer+0x9b/frame 0xfffffe0145204930 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe0145204930 --- interrupt, rip = 0xffffffff80c93348, rsp = 0xfffffe0145204a00, rbp = 0xfffffe0145204a30 --- witness_unlock() at witness_unlock+0x1b8/frame 0xfffffe0145204a30 __mtx_unlock_flags() at __mtx_unlock_flags+0x4d/frame 0xfffffe0145204a60 userret() at userret+0x19b/frame 0xfffffe0145204ac0 amd64_syscall() at amd64_syscall+0x205/frame 0xfffffe0145204bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145204bf0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8012094ca, rsp = 0x7fffffffdca8, rbp = 0x7fffffffdd30 --- Tracing command dhclient pid 2339 tid 100246 td 0xfffffe010dbac3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141467720 mi_switch() at mi_switch+0x155/frame 0xfffffe0141467740 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141467780 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01414677d0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01414677e0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141467840 seltdwait() at seltdwait+0x8d/frame 0xfffffe0141467880 kern_select() at kern_select+0xa4e/frame 0xfffffe0141467a80 sys_select() at sys_select+0x56/frame 0xfffffe0141467ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141467bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141467bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011e672a, rsp = 0x7fffffffe938, rbp = 0x7fffffffe9f0 --- Tracing command dhclient pid 2342 tid 100264 td 0xfffffe01434c4740 sched_switch() at sched_switch+0x5af/frame 0xfffffe01414cb760 mi_switch() at mi_switch+0x155/frame 0xfffffe01414cb780 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01414cb7c0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01414cb810 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01414cb820 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01414cb880 seltdwait() at seltdwait+0x8d/frame 0xfffffe01414cb8c0 kern_poll_kfds() at kern_poll_kfds+0x41b/frame 0xfffffe01414cb950 kern_poll() at kern_poll+0x9f/frame 0xfffffe01414cbaa0 sys_poll() at sys_poll+0x50/frame 0xfffffe01414cbac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01414cbbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01414cbbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011e662a, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebb0 --- Tracing command dhclient pid 2401 tid 100279 td 0xfffffe011bdbfe40 sched_switch() at sched_switch+0x5af/frame 0xfffffe014151f710 mi_switch() at mi_switch+0x155/frame 0xfffffe014151f730 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014151f770 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014151f7c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe014151f800 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe014151f880 seltdwait() at seltdwait+0x7f/frame 0xfffffe014151f8c0 kern_poll_kfds() at kern_poll_kfds+0x41b/frame 0xfffffe014151f950 kern_poll() at kern_poll+0x9f/frame 0xfffffe014151faa0 sys_poll() at sys_poll+0x50/frame 0xfffffe014151fac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014151fbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014151fbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011e662a, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebb0 --- Tracing command devd pid 2406 tid 100277 td 0xfffffe01436263a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415046d0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415046f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141504730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141504780 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe01415047c0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe0141504840 seltdwait() at seltdwait+0x7f/frame 0xfffffe0141504880 kern_select() at kern_select+0xa4e/frame 0xfffffe0141504a80 sys_select() at sys_select+0x56/frame 0xfffffe0141504ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141504bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141504bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x28c62a, rsp = 0x7fffffffcab8, rbp = 0x7fffffffec50 --- Tracing command syslogd pid 2595 tid 100282 td 0xfffffe01434c3900 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141533770 mi_switch() at mi_switch+0x155/frame 0xfffffe0141533790 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415337d0 _sleep() at _sleep+0x32f/frame 0xfffffe0141533870 getdirtybuf() at getdirtybuf+0x28c/frame 0xfffffe01415338c0 flush_deplist() at flush_deplist+0x97/frame 0xfffffe0141533900 softdep_sync_metadata() at softdep_sync_metadata+0x2d2/frame 0xfffffe0141533960 ffs_syncvnode() at ffs_syncvnode+0x4c/frame 0xfffffe01415339f0 ffs_fsync() at ffs_fsync+0x1e/frame 0xfffffe0141533a20 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x59/frame 0xfffffe0141533a40 kern_fsync() at kern_fsync+0x1b3/frame 0xfffffe0141533ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141533bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141533bf0 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x8011cb54a, rsp = 0x7fffffffe6b8, rbp = 0x7fffffffec20 --- Tracing command watchdogd pid 2603 tid 100291 td 0xfffffe010dbaa720 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415656d0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415656f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141565730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141565780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141565790 _sleep() at _sleep+0x325/frame 0xfffffe0141565830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01415658c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0141565ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141565bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141565bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8011c188a, rsp = 0x7fffffffe968, rbp = 0x7fffffffea30 --- Tracing command rpcbind pid 2625 tid 100275 td 0xfffffe01417b9020 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bbd5710 mi_switch() at mi_switch+0x155/frame 0xfffffe011bbd5730 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bbd5770 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe011bbd57c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe011bbd5800 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe011bbd5880 seltdwait() at seltdwait+0x7f/frame 0xfffffe011bbd58c0 kern_poll_kfds() at kern_poll_kfds+0x41b/frame 0xfffffe011bbd5950 kern_poll() at kern_poll+0x9f/frame 0xfffffe011bbd5aa0 sys_poll() at sys_poll+0x50/frame 0xfffffe011bbd5ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe011bbd5bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe011bbd5bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011bc62a, rsp = 0x7fffffffc978, rbp = 0x7fffffffeba0 --- Tracing command rpc.statd pid 2650 tid 100258 td 0xfffffe011bdc0560 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bbdf6d0 mi_switch() at mi_switch+0x155/frame 0xfffffe011bbdf6f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bbdf730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe011bbdf780 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe011bbdf7c0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe011bbdf840 seltdwait() at seltdwait+0x7f/frame 0xfffffe011bbdf880 kern_select() at kern_select+0xa4e/frame 0xfffffe011bbdfa80 sys_select() at sys_select+0x56/frame 0xfffffe011bbdfac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe011bbdfbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe011bbdfbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011b672a, rsp = 0x7fffffffe9f8, rbp = 0x7fffffffeb40 --- Tracing command rpc.lockd pid 2653 tid 100253 td 0xfffffe01417b6c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141476640 mi_switch() at mi_switch+0x155/frame 0xfffffe0141476660 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01414766a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01414766f0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0141476730 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe01414767b0 svc_run_internal() at svc_run_internal+0xac1/frame 0xfffffe01414768e0 svc_run() at svc_run+0x250/frame 0xfffffe0141476940 sys_nlm_syscall() at sys_nlm_syscall+0x74a/frame 0xfffffe0141476ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141476bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141476bf0 --- syscall (154, FreeBSD ELF64, sys_nlm_syscall), rip = 0x8011bef0a, rsp = 0x7fffffffeb58, rbp = 0x7fffffffec20 --- Tracing command powerd pid 2658 tid 100256 td 0xfffffe01417b6560 sched_switch() at sched_switch+0x5af/frame 0xfffffe01414a36d0 mi_switch() at mi_switch+0x155/frame 0xfffffe01414a36f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01414a3730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01414a3780 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe01414a37c0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19b/frame 0xfffffe01414a3840 seltdwait() at seltdwait+0x7f/frame 0xfffffe01414a3880 kern_select() at kern_select+0xa4e/frame 0xfffffe01414a3a80 sys_select() at sys_select+0x56/frame 0xfffffe01414a3ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01414a3bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01414a3bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011c172a, rsp = 0x7fffffffeaf8, rbp = 0x7fffffffec30 --- Tracing command ntpd pid 2684 tid 100250 td 0xfffffe01434e43a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141458720 mi_switch() at mi_switch+0x155/frame 0xfffffe0141458740 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141458780 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01414587d0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01414587e0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141458840 seltdwait() at seltdwait+0x8d/frame 0xfffffe0141458880 kern_select() at kern_select+0xa4e/frame 0xfffffe0141458a80 sys_select() at sys_select+0x56/frame 0xfffffe0141458ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141458bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141458bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8015db72a, rsp = 0x7fffffffde58, rbp = 0x7fffffffde90 --- Tracing command inetd pid 2710 tid 100296 td 0xfffffe0143624e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141583720 mi_switch() at mi_switch+0x155/frame 0xfffffe0141583740 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141583780 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415837d0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415837e0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141583840 seltdwait() at seltdwait+0x8d/frame 0xfffffe0141583880 kern_select() at kern_select+0xa4e/frame 0xfffffe0141583a80 sys_select() at sys_select+0x56/frame 0xfffffe0141583ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141583bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141583bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8011e072a, rsp = 0x7fffffffdd78, rbp = 0x7fffffffec10 --- Tracing command sshd pid 2729 tid 100295 td 0xfffffe0143e58900 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141579720 mi_switch() at mi_switch+0x155/frame 0xfffffe0141579740 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141579780 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415797d0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415797e0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141579840 seltdwait() at seltdwait+0x8d/frame 0xfffffe0141579880 kern_select() at kern_select+0xa4e/frame 0xfffffe0141579a80 sys_select() at sys_select+0x56/frame 0xfffffe0141579ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141579bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141579bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80169972a, rsp = 0x7fffffffe5c8, rbp = 0x7fffffffe600 --- Tracing command cron pid 2733 tid 100298 td 0xfffffe01434e6020 sched_switch() at sched_switch+0x5af/frame 0xfffffe014148a870 mi_switch() at mi_switch+0x155/frame 0xfffffe014148a890 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014148a8d0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014148a920 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe014148a960 _sleep() at _sleep+0x285/frame 0xfffffe014148aa00 kern_clock_nanosleep() at kern_clock_nanosleep+0x1e1/frame 0xfffffe014148aa80 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe014148aac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014148abf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014148abf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8011d85ca, rsp = 0x7fffffffea48, rbp = 0x7fffffffec20 --- Tracing command getty pid 2747 tid 100225 td 0xfffffe010e0e9c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe010e2457c0 mi_switch() at mi_switch+0x155/frame 0xfffffe010e2457e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe010e245820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010e245870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010e245880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010e2458e0 tty_wait() at tty_wait+0x42/frame 0xfffffe010e245910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe010e245970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010e2459a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe010e245a00 dofileread() at dofileread+0x81/frame 0xfffffe010e245a50 sys_read() at sys_read+0xc0/frame 0xfffffe010e245ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe010e245bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010e245bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2748 tid 100299 td 0xfffffe011bdbf720 sched_switch() at sched_switch+0x5af/frame 0xfffffe014151a7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe014151a7e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014151a820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014151a870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014151a880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe014151a8e0 tty_wait() at tty_wait+0x42/frame 0xfffffe014151a910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe014151a970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe014151a9a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe014151aa00 dofileread() at dofileread+0x81/frame 0xfffffe014151aa50 sys_read() at sys_read+0xc0/frame 0xfffffe014151aac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014151abf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014151abf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2749 tid 100247 td 0xfffffe010e0e8720 sched_switch() at sched_switch+0x5af/frame 0xfffffe01414717c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01414717e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0141471820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141471870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141471880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01414718e0 tty_wait() at tty_wait+0x42/frame 0xfffffe0141471910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe0141471970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01414719a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141471a00 dofileread() at dofileread+0x81/frame 0xfffffe0141471a50 sys_read() at sys_read+0xc0/frame 0xfffffe0141471ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141471bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141471bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2750 tid 100302 td 0xfffffe011bdbf000 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415a17c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415a17e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415a1820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415a1870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415a1880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415a18e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415a1910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415a1970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415a19a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415a1a00 dofileread() at dofileread+0x81/frame 0xfffffe01415a1a50 sys_read() at sys_read+0xc0/frame 0xfffffe01415a1ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415a1bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415a1bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2751 tid 100303 td 0xfffffe014401c740 sched_switch() at sched_switch+0x5af/frame 0xfffffe014159c7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe014159c7e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014159c820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014159c870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014159c880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe014159c8e0 tty_wait() at tty_wait+0x42/frame 0xfffffe014159c910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe014159c970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe014159c9a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe014159ca00 dofileread() at dofileread+0x81/frame 0xfffffe014159ca50 sys_read() at sys_read+0xc0/frame 0xfffffe014159cac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014159cbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014159cbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2752 tid 100304 td 0xfffffe014401c020 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415ab7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415ab7e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415ab820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415ab870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415ab880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415ab8e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415ab910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415ab970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415ab9a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415aba00 dofileread() at dofileread+0x81/frame 0xfffffe01415aba50 sys_read() at sys_read+0xc0/frame 0xfffffe01415abac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415abbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415abbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2753 tid 100305 td 0xfffffe014401b900 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415a67c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415a67e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415a6820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415a6870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415a6880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415a68e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415a6910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415a6970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415a69a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415a6a00 dofileread() at dofileread+0x81/frame 0xfffffe01415a6a50 sys_read() at sys_read+0xc0/frame 0xfffffe01415a6ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415a6bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415a6bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2754 tid 100306 td 0xfffffe01440201e0 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415b57c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415b57e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415b5820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415b5870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415b5880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415b58e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415b5910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415b5970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415b59a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415b5a00 dofileread() at dofileread+0x81/frame 0xfffffe01415b5a50 sys_read() at sys_read+0xc0/frame 0xfffffe01415b5ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415b5bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415b5bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2755 tid 100307 td 0xfffffe014401fac0 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415b07c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415b07e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415b0820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415b0870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415b0880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415b08e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415b0910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415b0970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415b09a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415b0a00 dofileread() at dofileread+0x81/frame 0xfffffe01415b0a50 sys_read() at sys_read+0xc0/frame 0xfffffe01415b0ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415b0bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415b0bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command getty pid 2756 tid 100308 td 0xfffffe014401f3a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415bf7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe01415bf7e0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415bf820 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415bf870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415bf880 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415bf8e0 tty_wait() at tty_wait+0x42/frame 0xfffffe01415bf910 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe01415bf970 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01415bf9a0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe01415bfa00 dofileread() at dofileread+0x81/frame 0xfffffe01415bfa50 sys_read() at sys_read+0xc0/frame 0xfffffe01415bfac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415bfbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415bfbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c56aa, rsp = 0x7fffffffeb58, rbp = 0x7fffffffeba0 --- Tracing command sshd pid 2795 tid 100290 td 0xfffffe0143625560 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141538760 mi_switch() at mi_switch+0x155/frame 0xfffffe0141538780 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415387c0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141538810 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141538820 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141538880 seltdwait() at seltdwait+0x8d/frame 0xfffffe01415388c0 kern_poll_kfds() at kern_poll_kfds+0x41b/frame 0xfffffe0141538950 kern_poll() at kern_poll+0x9f/frame 0xfffffe0141538aa0 sys_poll() at sys_poll+0x50/frame 0xfffffe0141538ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141538bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141538bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80169962a, rsp = 0x7fffffffe538, rbp = 0x7fffffffe570 --- Tracing command sshd pid 2798 tid 100309 td 0xfffffe014401ec80 sched_switch() at sched_switch+0x5af/frame 0xfffffe01415ba720 mi_switch() at mi_switch+0x155/frame 0xfffffe01415ba740 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415ba780 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415ba7d0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01415ba7e0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01415ba840 seltdwait() at seltdwait+0x8d/frame 0xfffffe01415ba880 kern_select() at kern_select+0xa4e/frame 0xfffffe01415baa80 sys_select() at sys_select+0x56/frame 0xfffffe01415baac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01415babf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01415babf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80169972a, rsp = 0x7fffffffa4c8, rbp = 0x7fffffffa500 --- Tracing command bash pid 2799 tid 100245 td 0xfffffe01434e4ac0 --- swapped out Tracing command su pid 2804 tid 100301 td 0xfffffe014401b1e0 --- swapped out Tracing command csh pid 2805 tid 100288 td 0xfffffe01417b8900 --- swapped out Tracing command bash pid 2807 tid 100231 td 0xfffffe011bdc13a0 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bb856e0 mi_switch() at mi_switch+0x155/frame 0xfffffe011bb85700 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bb85740 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe011bb85790 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe011bb857a0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe011bb85800 seltdwait() at seltdwait+0x8d/frame 0xfffffe011bb85840 kern_select() at kern_select+0xa4e/frame 0xfffffe011bb85a40 sys_pselect() at sys_pselect+0x127/frame 0xfffffe011bb85ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe011bb85bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe011bb85bf0 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x80052a66a, rsp = 0x7fffffffd5a8, rbp = 0x7fffffffd670 --- Tracing command sh pid 2808 tid 100297 td 0xfffffe0143a1b020 sched_switch() at sched_switch+0x5af/frame 0xfffffe014158d6d0 mi_switch() at mi_switch+0x155/frame 0xfffffe014158d6f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014158d730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014158d780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014158d790 _sleep() at _sleep+0x325/frame 0xfffffe014158d830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe014158d8c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe014158dac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014158dbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014158dbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe350 --- Tracing command sh pid 2893 tid 100317 td 0xfffffe014401d720 --- swapped out Tracing command sh pid 2942 tid 100323 td 0xfffffe01440ad560 sched_switch() at sched_switch+0x5af/frame 0xfffffe01444046d0 mi_switch() at mi_switch+0x155/frame 0xfffffe01444046f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0144404730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0144404780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0144404790 _sleep() at _sleep+0x325/frame 0xfffffe0144404830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01444048c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0144404ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0144404bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0144404bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffe228, rbp = 0x7fffffffe2d0 --- Tracing command awk pid 2943 tid 100322 td 0xfffffe01440adc80 sched_switch() at sched_switch+0x5af/frame 0xfffffe0144409840 mi_switch() at mi_switch+0x155/frame 0xfffffe0144409860 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01444098a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01444098f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0144409900 _sleep() at _sleep+0x325/frame 0xfffffe01444099a0 pipe_read() at pipe_read+0x4c6/frame 0xfffffe0144409a00 dofileread() at dofileread+0x81/frame 0xfffffe0144409a50 sys_read() at sys_read+0xc0/frame 0xfffffe0144409ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0144409bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0144409bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8012166aa, rsp = 0x7fffffffe1f8, rbp = 0x7fffffffe210 --- Tracing command sh pid 2945 tid 100326 td 0xfffffe01440ace40 sched_switch() at sched_switch+0x5af/frame 0xfffffe0144427840 mi_switch() at mi_switch+0x155/frame 0xfffffe0144427860 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01444278a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01444278f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0144427900 _sleep() at _sleep+0x325/frame 0xfffffe01444279a0 pipe_read() at pipe_read+0x4c6/frame 0xfffffe0144427a00 dofileread() at dofileread+0x81/frame 0xfffffe0144427a50 sys_read() at sys_read+0xc0/frame 0xfffffe0144427ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0144427bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0144427bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8012096aa, rsp = 0x7fffffffdcc8, rbp = 0x7fffffffe1f0 --- Tracing command sh pid 78666 tid 100280 td 0xfffffe01417b5720 sched_switch() at sched_switch+0x5af/frame 0xfffffe0141529840 mi_switch() at mi_switch+0x155/frame 0xfffffe0141529860 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01415298a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01415298f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141529900 _sleep() at _sleep+0x325/frame 0xfffffe01415299a0 pipe_read() at pipe_read+0x4c6/frame 0xfffffe0141529a00 dofileread() at dofileread+0x81/frame 0xfffffe0141529a50 sys_read() at sys_read+0xc0/frame 0xfffffe0141529ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0141529bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141529bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8012096aa, rsp = 0x7fffffffe318, rbp = 0x7fffffffe460 --- Tracing command sh pid 78676 tid 100289 td 0xfffffe0143e56c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe014154c6d0 mi_switch() at mi_switch+0x155/frame 0xfffffe014154c6f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe014154c730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014154c780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014154c790 _sleep() at _sleep+0x325/frame 0xfffffe014154c830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe014154c8c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe014154cac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe014154cbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014154cbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffe0c8, rbp = 0x7fffffffe170 --- Tracing command ls pid 78677 tid 100504 td 0xfffffe0149677900 sched_switch() at sched_switch+0x5af/frame 0xfffffe01453993e0 mi_switch() at mi_switch+0x155/frame 0xfffffe0145399400 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0145399440 sleeplk() at sleeplk+0x100/frame 0xfffffe01453994a0 lockmgr_slock_hard() at lockmgr_slock_hard+0x3d2/frame 0xfffffe0145399540 ffs_lock() at ffs_lock+0x9d/frame 0xfffffe0145399580 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x3c/frame 0xfffffe01453995a0 _vn_lock() at _vn_lock+0x54/frame 0xfffffe0145399600 vget_finish() at vget_finish+0x221/frame 0xfffffe0145399650 cache_lookup() at cache_lookup+0x7fe/frame 0xfffffe01453996b0 vfs_cache_lookup() at vfs_cache_lookup+0x7b/frame 0xfffffe0145399700 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x5a/frame 0xfffffe0145399720 lookup() at lookup+0x4d1/frame 0xfffffe01453997c0 namei() at namei+0x379/frame 0xfffffe0145399870 kern_statat() at kern_statat+0x12d/frame 0xfffffe01453999c0 sys_fstatat() at sys_fstatat+0x2f/frame 0xfffffe0145399ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0145399bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145399bf0 --- syscall (552, FreeBSD ELF64, sys_fstatat), rip = 0x80123b3aa, rsp = 0x7fffffffdd18, rbp = 0x7fffffffddd0 --- Tracing command grep pid 78678 tid 100496 td 0xfffffe0144267c80 sched_switch() at sched_switch+0x5af/frame 0xfffffe0145371840 mi_switch() at mi_switch+0x155/frame 0xfffffe0145371860 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01453718a0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01453718f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0145371900 _sleep() at _sleep+0x325/frame 0xfffffe01453719a0 pipe_read() at pipe_read+0x4c6/frame 0xfffffe0145371a00 dofileread() at dofileread+0x81/frame 0xfffffe0145371a50 sys_read() at sys_read+0xc0/frame 0xfffffe0145371ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe0145371bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0145371bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8011c06aa, rsp = 0x7fffffffe008, rbp = 0x7fffffffe120 --- Tracing command lockf2 pid 62414 tid 100469 td 0xfffffe0144286020 --- swapped out Tracing command lockf2 pid 62417 tid 100432 td 0xfffffe01440b0020 --- swapped out Tracing command lockf2 pid 62419 tid 100465 td 0xfffffe0144286740 --- swapped out Tracing command symlink pid 62420 tid 100292 td 0xfffffe010dbaa000 --- swapped out Tracing command openat pid 62422 tid 100508 td 0xfffffe0144304900 --- swapped out Tracing command symlink pid 62423 tid 100293 td 0xfffffe0143e59740 --- swapped out Tracing command openat pid 62424 tid 100729 td 0xfffffe014d3c7560 --- swapped out Tracing command symlink pid 62425 tid 100274 td 0xfffffe01417b5e40 --- swapped out Tracing command rw pid 62426 tid 100728 td 0xfffffe014d3c7c80 --- swapped out Tracing command openat pid 62427 tid 100732 td 0xfffffe014d3c6e40 --- swapped out Tracing command symlink pid 62428 tid 100319 td 0xfffffe01440b0740 --- swapped out Tracing command rw pid 62429 tid 100509 td 0xfffffe0144127e40 --- swapped out Tracing command openat pid 62431 tid 100576 td 0xfffffe0144244000 --- swapped out Tracing command symlink pid 62432 tid 100294 td 0xfffffe0143e59020 --- swapped out Tracing command openat pid 62433 tid 100578 td 0xfffffe0144282e40 --- swapped out Tracing command link pid 62434 tid 100445 td 0xfffffe01442741e0 --- swapped out Tracing command symlink pid 62435 tid 100226 td 0xfffffe010d674ac0 --- swapped out Tracing command symlink pid 62436 tid 100724 td 0xfffffe014d3ca740 --- swapped out Tracing command symlink pid 62437 tid 100243 td 0xfffffe010e15b000 --- swapped out Tracing command symlink pid 62440 tid 100361 td 0xfffffe0143a17e40 --- swapped out Tracing command rename pid 62442 tid 100392 td 0xfffffe014421fac0 --- swapped out Tracing command symlink pid 62444 tid 100363 td 0xfffffe0143a17720 --- swapped out Tracing command rename pid 62446 tid 100394 td 0xfffffe014421f3a0 --- swapped out Tracing command mkdir pid 62447 tid 100315 td 0xfffffe014401e560 --- swapped out Tracing command mkfifo pid 62449 tid 100348 td 0xfffffe0143e55000 --- swapped out Tracing command symlink pid 62450 tid 100375 td 0xfffffe01441f7740 --- swapped out Tracing command rename pid 62451 tid 100396 td 0xfffffe014421ec80 --- swapped out Tracing command mkdir pid 62452 tid 100239 td 0xfffffe011bdc0c80 --- swapped out Tracing command mkfifo pid 62454 tid 100346 td 0xfffffe014412b020 --- swapped out Tracing command symlink pid 62455 tid 100377 td 0xfffffe01441f7020 --- swapped out Tracing command rename pid 62456 tid 100446 td 0xfffffe014425a560 --- swapped out Tracing command mkdir pid 62457 tid 100359 td 0xfffffe01441dd1e0 --- swapped out Tracing command symlink pid 62460 tid 100379 td 0xfffffe01441f6900 --- swapped out Tracing command rename pid 62461 tid 100449 td 0xfffffe0144259e40 --- swapped out Tracing command mkdir pid 62463 tid 100234 td 0xfffffe010e0e9560 --- swapped out Tracing command mkfifo pid 62464 tid 100374 td 0xfffffe01441da000 --- swapped out Tracing command symlink pid 62465 tid 100434 td 0xfffffe014425bac0 --- swapped out Tracing command rename pid 62466 tid 100466 td 0xfffffe0144259720 --- swapped out Tracing command mkdir pid 62469 tid 100265 td 0xfffffe01436271e0 --- swapped out Tracing command rename pid 62471 tid 100386 td 0xfffffe0144127720 --- swapped out Tracing command mkdir pid 62472 tid 100276 td 0xfffffe0143626ac0 --- swapped out Tracing command rename pid 62474 tid 100415 td 0xfffffe014423d020 --- swapped out Tracing command mkdir pid 62476 tid 100492 td 0xfffffe01496771e0 --- swapped out Tracing command rename pid 62477 tid 100416 td 0xfffffe014423c900 --- swapped out Tracing command mkdir pid 62478 tid 100717 td 0xfffffe014d51c740 --- swapped out Tracing command rename pid 62479 tid 100380 td 0xfffffe01442003a0 --- swapped out Tracing command mkdir pid 62480 tid 100720 td 0xfffffe014d51c020 --- swapped out Tracing command rename pid 62481 tid 100382 td 0xfffffe01441ffc80 --- swapped out Tracing command mkdir pid 62482 tid 100721 td 0xfffffe014d51b900 --- swapped out Tracing command rename pid 62483 tid 100400 td 0xfffffe01441ff560 --- swapped out Tracing command mkdir pid 62484 tid 100454 td 0xfffffe0144275020 --- swapped out Tracing command rename pid 62485 tid 100530 td 0xfffffe01441fee40 --- swapped out Tracing command rename pid 62486 tid 100535 td 0xfffffe01441fe720 --- swapped out Tracing command rename pid 62487 tid 100536 td 0xfffffe01441fe000 --- swapped out Tracing command rename pid 62488 tid 100544 td 0xfffffe0144327560 --- swapped out Tracing command rename pid 62489 tid 100545 td 0xfffffe0148a991e0 --- swapped out Tracing command rename pid 62490 tid 100546 td 0xfffffe0148a98ac0 --- swapped out Tracing command sh pid 63632 tid 100242 td 0xfffffe010e0e8e40 sched_switch() at sched_switch+0x5af/frame 0xfffffe011bbc16d0 mi_switch() at mi_switch+0x155/frame 0xfffffe011bbc16f0 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe011bbc1730 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe011bbc1780 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe011bbc1790 _sleep() at _sleep+0x325/frame 0xfffffe011bbc1830 kern_wait6() at kern_wait6+0x474/frame 0xfffffe011bbc18c0 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe011bbc1ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe011bbc1bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe011bbc1bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120988a, rsp = 0x7fffffffe778, rbp = 0x7fffffffe820 --- Tracing command ls pid 63634 tid 100414 td 0xfffffe014423d740 sched_switch() at sched_switch+0x5af/frame 0xfffffe01445d2360 mi_switch() at mi_switch+0x155/frame 0xfffffe01445d2380 sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe01445d23c0 sleeplk() at sleeplk+0x100/frame 0xfffffe01445d2420 lockmgr_slock_hard() at lockmgr_slock_hard+0x3d2/frame 0xfffffe01445d24c0 ffs_lock() at ffs_lock+0x9d/frame 0xfffffe01445d2500 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x3c/frame 0xfffffe01445d2520 _vn_lock() at _vn_lock+0x54/frame 0xfffffe01445d2580 vget_finish() at vget_finish+0x221/frame 0xfffffe01445d25d0 cache_lookup() at cache_lookup+0x7fe/frame 0xfffffe01445d2630 vfs_cache_lookup() at vfs_cache_lookup+0x7b/frame 0xfffffe01445d2680 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x5a/frame 0xfffffe01445d26a0 lookup() at lookup+0x4d1/frame 0xfffffe01445d2740 namei() at namei+0x379/frame 0xfffffe01445d27f0 vn_open_cred() at vn_open_cred+0x487/frame 0xfffffe01445d2960 kern_openat() at kern_openat+0x28c/frame 0xfffffe01445d2ac0 amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01445d2bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01445d2bf0 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x80123b5ea, rsp = 0x7fffffffe428, rbp = 0x7fffffffe450 --- db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 7 (kgdb) thread 33 [Switching to thread 33 (Thread 100048)] #0 cpustop_handler () at ../../../x86/x86/mp_x86.c:1479 1479 use_mwait = (stop_mwait && (cpu_feature2 & CPUID2_MON) != 0 && (kgdb) bt #0 cpustop_handler () at ../../../x86/x86/mp_x86.c:1479 #1 0xffffffff810bc1a9 in ipi_nmi_handler () at ../../../x86/x86/mp_x86.c:1436 #2 0xffffffff810f6426 in trap (frame=0xfffffe003ce3af30) at ../../../amd64/amd64/trap.c:238 #3 #4 witness_checkorder (lock=0xfffffe010e153040, flags=0x9, file=file@entry=0xffffffff81280913 "../../../kern/vfs_subr.c", line=line@entry=0x19cb, interlock=interlock@entry=0x0) at ../../../kern/subr_witness.c:1235 #5 0xffffffff80bfeb14 in __mtx_lock_flags (c=0xfffffe010e153058, opts=0x0, file=0xffffffff81280913 "../../../kern/vfs_subr.c", line=0x19cb) at ../../../kern/kern_mutex.c:274 #6 0xffffffff80d17989 in __mnt_vnode_next_all (mvp=0xfffffe00e4894920, mp=0xfffffe010e153000) at ../../../kern/vfs_subr.c:6603 #7 0xffffffff80f57354 in ffs_sync (mp=0xfffffe010e153000, waitfor=0x1) at ../../../ufs/ffs/ffs_vfsops.c:1863 #8 0xffffffff80d343e5 in vfs_write_suspend (mp=mp@entry=0xfffffe010e153000, flags=flags@entry=0x0) at ../../../kern/vfs_vnops.c:2097 #9 0xffffffff80d345a5 in vfs_write_suspend_umnt (mp=0xfffffe010e153000) at ../../../kern/vfs_vnops.c:2151 #10 0xffffffff80f56879 in ffs_unmount (mp=0xfffffe010e153000, mntflags=0x80000) at ../../../ufs/ffs/ffs_vfsops.c:1519 #11 0xffffffff80d0753c in dounmount (mp=0xfffffe010e153000, flags=0x100000080000, td=0xfffffe00e4ceac80) at ../../../kern/vfs_mount.c:2137 #12 0xffffffff80f52866 in ffs_fsfail_unmount (v=0xfffff806aec07500, pending=0x1) at ../../../ufs/ffs/ffs_vfsops.c:300 #13 0xffffffff80c8461a in taskqueue_run_locked (queue=queue@entry=0xfffff80003b1f100) at ../../../kern/subr_taskqueue.c:476 #14 0xffffffff80c856a2 in taskqueue_thread_loop (arg=arg@entry=0xffffffff81f04830 ) at ../../../kern/subr_taskqueue.c:793 #15 0xffffffff80bdb8a0 in fork_exit (callout=0xffffffff80c855e0 , arg=0xffffffff81f04830 , frame=0xfffffe00e4894c00) at ../../../kern/kern_fork.c:1083 #16 (kgdb) f 7 #7 0xffffffff80f57354 in ffs_sync (mp=0xfffffe010e153000, waitfor=0x1) at ../../../ufs/ffs/ffs_vfsops.c:1863 1863 MNT_VNODE_FOREACH_ALL(vp, mp, mvp) { (kgdb) l 1858 MNT_IUNLOCK(mp); 1859 1860 /* Grab snapshot of softdep dependency counts */ 1861 softdep_get_depcounts(mp, &softdep_deps, &softdep_accdeps); 1862 1863 MNT_VNODE_FOREACH_ALL(vp, mp, mvp) { 1864 /* 1865 * Depend on the vnode interlock to keep things stable enough 1866 * for a quick test. Since there might be hundreds of 1867 * thousands of vnodes, we cannot afford even a subroutine (kgdb) info loc mvp = 0xfffff80004397000 vp = 0xfffffe014657e4b0 devvp = 0xfffffe014c30f000 td = 0xfffffe00e4ceac80 ip = 0xfffffe014d22b220 ump = 0xfffff800097e6200 fs = 0xfffffe014621e000 error = 0x0 count = 0x0 lockreq = 0x80900 allerror = 0x0 suspend = 0x1 suspended = 0x0 secondary_writes = 0x0 secondary_accwrites = 0x7f643 softdep_deps = 0x1 softdep_accdeps = 0x1bbe79 bo = 0xfffffe014c30f178 (kgdb) p *ump $1 = {um_mountp = 0xfffffe010e153000, um_dev = 0xfffff80003b26000, um_cp = 0xfffff80009885b80, um_bo = 0xfffffe014c30f178, um_odevvp = 0xfffffe014d3b5770, um_devvp = 0xfffffe014c30f000, um_fstype = 0x2, um_fs = 0xfffffe014621e000, um_extattr = {uepm_lock = {lock_object = { lo_name = 0xffffffff811dc4f0 "ufs_extattr_sx", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xfffff8087fd83d80}, sx_lock = 0x1}, uepm_list = {lh_first = 0x0}, uepm_ucred = 0x0, uepm_flags = 0x1}, um_nindir = 0x1000, um_bptrtodb = 0x3, um_seqinc = 0x8, um_maxsymlinklen = 0x78, um_lock = {lock_object = {lo_name = 0xffffffff811f4299 "FFS", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd83d00}, mtx_lock = 0x0}, um_fsckpid = 0x0, um_softdep = 0xfffff80343830c00, um_quotas = {0x0, 0x0}, um_cred = {0x0, 0x0}, um_btime = {0x0, 0x0}, um_itime = {0x0, 0x0}, um_qflags = "\000", um_savedmaxfilesize = 0x0, um_flags = 0x8, um_last_fullmsg = {tv_sec = 0x0, tv_usec = 0x0}, um_secs_fullmsg = 0x0, um_last_integritymsg = {tv_sec = 0x0, tv_usec = 0x0}, um_secs_integritymsg = 0x0, um_trim_inflight = 0x0, um_trim_inflight_blks = 0x0, um_trim_total = 0x0, um_trim_total_blks = 0x0, um_trim_tq = 0x0, um_trimhash = 0x0, um_trimlisthashsize = 0x0, um_fsfail_task = 0x0, um_balloc = 0xffffffff80f29840 , um_blkatoff = 0xffffffff80f58b30 , um_truncate = 0xffffffff80f2b6b0 , um_update = 0xffffffff80f2b300 , um_valloc = 0xffffffff80f24060 , um_vfree = 0xffffffff80f24ee0 , um_ifree = 0xffffffff80f58cb0 , um_rdonly = 0xffffffff80f2d3a0 , um_snapgone = 0xffffffff80f30fd0 , um_check_blkno = 0x0} (kgdb) p *mp $2 = {mnt_vfs_ops = 0x2, mnt_kern_flag = 0x49846345, mnt_flag = 0x100201000, mnt_pcpu = 0xfffffe018bc3bfd0, mnt_rootvnode = 0x0, mnt_vnodecovered = 0xfffffe0143ef0960, mnt_op = 0xffffffff81b65150 , mnt_vfc = 0xffffffff81b65270 , mnt_mtx = {lock_object = { lo_name = 0xffffffff811cd7ec "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd75b80}, mtx_lock = 0x0}, mnt_gen = 0x26, mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffffe014285b2e8}, mnt_syncer = 0x0, mnt_ref = 0x2efc, mnt_nvnodelist = { tqh_first = 0xfffffe014ea34bb8, tqh_last = 0xfffffe01475a1c48}, mnt_nvnodelistsize = 0x2ec2, mnt_writeopcount = 0x0, mnt_opt = 0xfffff80009a7a200, mnt_optnew = 0x0, mnt_stat = {f_version = 0x20140518, f_type = 0x35, f_flags = 0x100201000, f_bsize = 0x1000, f_iosize = 0x8000, f_blocks = 0x3df87, f_bfree = 0x3d776, f_bavail = 0x38824, f_files = 0x201fe, f_ffree = 0x201fb, f_syncwrites = 0x12ba2, f_asyncwrites = 0x2929d, f_syncreads = 0x10, f_asyncreads = 0x0, f_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = {0x60f3c0d3, 0xf3e1c864}}, f_charspare = '\000' , f_fstypename = "ufs", '\000' , f_mntfromname = "/dev/md10", '\000' , f_mntonname = "/mnt", '\000' }, mnt_cred = 0xfffff80004376900, mnt_data = 0xfffff800097e6200, mnt_time = 0x0, mnt_iosize_max = 0x100000, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 0x912472f, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x7f643, mnt_susp_owner = 0xfffffe00e4ceac80, mnt_gjprovider = 0x0, mnt_listmtx = {lock_object = {lo_name = 0xffffffff81268bfe "struct mount vlist mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd83980}, mtx_lock = 0x0}, mnt_lazyvnodelist = {tqh_first = 0xfffffe014ea34bb8, tqh_last = 0xfffffe01457d7428}, mnt_lazyvnodelistsize = 0x287, mnt_upper_pending = 0x0, mnt_explock = {lock_object = {lo_name = 0xffffffff812a1dce "explock", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xfffff8087fd81500}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_pri = 0x60, lk_timo = 0x0, lk_stack = {depth = 0x0, pcs = {0x0 }}}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffffe010e153b20}, mnt_notify = {tqh_first = 0x0, tqh_last = 0xfffffe010e153b30}, taskqueue_link = {stqe_next = 0x0}, taskqueue_flags = 0x0} (kgdb) p *fs $3 = {fs_firstfield = 0x0, fs_unused_1 = 0x0, fs_sblkno = 0x18, fs_cblkno = 0x20, fs_iblkno = 0x28, fs_dblkno = 0x830, fs_old_cgoffset = 0x0, fs_old_cgmask = 0x0, fs_old_time = 0x0, fs_old_size = 0x0, fs_old_dsize = 0x0, fs_ncg = 0x4, fs_bsize = 0x8000, fs_fsize = 0x1000, fs_frag = 0x8, fs_minfree = 0x8, fs_old_rotdelay = 0x0, fs_old_rps = 0x0, fs_bmask = 0xffff8000, fs_fmask = 0xfffff000, fs_bshift = 0xf, fs_fshift = 0xc, fs_maxcontig = 0x20, fs_maxbpg = 0x1000, fs_fragshift = 0x3, fs_fsbtodb = 0x3, fs_sbsize = 0x1000, fs_spare1 = {0x0, 0x0}, fs_nindir = 0x1000, fs_inopb = 0x80, fs_old_nspf = 0x0, fs_optim = 0x0, fs_old_npsect = 0x0, fs_old_interleave = 0x0, fs_old_trackskew = 0x0, fs_id = {0x60f3c0d3, 0xf3e1c864}, fs_old_csaddr = 0x0, fs_cssize = 0x1000, fs_cgsize = 0x4000, fs_spare2 = 0x0, fs_old_nsect = 0x0, fs_old_spc = 0x0, fs_old_ncyl = 0x0, fs_old_cpg = 0x0, fs_ipg = 0x8080, fs_fpg = 0x10008, fs_old_cstotal = {cs_ndir = 0x0, cs_nbfree = 0x0, cs_nifree = 0x0, cs_nffree = 0x0}, fs_fmod = 0x1, fs_clean = 0x0, fs_ronly = 0x0, fs_old_flags = 0x80, fs_fsmnt = "/mnt", '\000' , fs_volname = '\000' , fs_swuid = 0x0, fs_pad = 0x0, fs_cgrotor = 0x2, fs_ocsp = {0x0 }, fs_si = 0xfffff804f0fbfa40, fs_old_cpc = 0x0, fs_maxbsize = 0x8000, fs_unrefs = 0x0, fs_providersize = 0x40000, fs_metaspace = 0xa38, fs_sparecon64 = { 0x0 }, fs_sblockactualloc = 0x10000, fs_sblockloc = 0x10000, fs_cstotal = {cs_ndir = 0x27d, cs_nbfree = 0x6149, cs_nifree = 0x1d33b, cs_nffree = 0xf9, cs_numclusters = 0x0, cs_spare = {0x0, 0x0, 0x0}}, fs_time = 0x60f3c0fe, fs_size = 0x40000, fs_dsize = 0x3df87, fs_csaddr = 0x830, fs_pendingblocks = 0x0, fs_pendinginodes = 0x0, fs_snapinum = {0x0 }, fs_avgfilesize = 0x4000, fs_avgfpdir = 0x40, fs_save_cgsize = 0x0, fs_mtime = 0x60f3c0d3, fs_sujfree = 0x0, fs_sparecon32 = {0x0 }, fs_ckhash = 0x162f196f, fs_metackhash = 0x7, fs_flags = 0x20a, fs_contigsumsize = 0x10, fs_maxsymlinklen = 0x78, fs_old_inodefmt = 0x0, fs_maxfilesize = 0x800800805ffff, fs_qbmask = 0x7fff, fs_qfmask = 0xfff, fs_state = 0x0, fs_old_postblformat = 0x0, fs_old_nrpos = 0x0, fs_spare5 = {0x0, 0x0}, fs_magic = 0x19540119} (kgdb) diff --git a/sys/fs/nullfs/null.h b/sys/fs/nullfs/null.h index 6fdac4b1006e..a41625536d65 100644 --- a/sys/fs/nullfs/null.h +++ b/sys/fs/nullfs/null.h @@ -45,6 +45,8 @@ struct null_mount { struct mount *nullm_vfs; struct vnode *nullm_lowerrootvp; /* Ref to lower root vnode */ uint64_t nullm_flags; + struct mount_upper_node upper_node; + struct mount_upper_node notify_node; }; #ifdef _KERNEL diff --git a/sys/fs/nullfs/null_vfsops.c b/sys/fs/nullfs/null_vfsops.c index 4914e5fc2dbf..73301c9275d2 100644 --- a/sys/fs/nullfs/null_vfsops.c +++ b/sys/fs/nullfs/null_vfsops.c @@ -163,7 +163,8 @@ nullfs_mount(struct mount *mp) * Save pointer to underlying FS and the reference to the * lower root vnode. */ - xmp->nullm_vfs = vfs_pin_from_vp(lowerrootvp); + xmp->nullm_vfs = vfs_register_upper_from_vp(lowerrootvp, mp, + &xmp->upper_node); if (xmp->nullm_vfs == NULL) { vput(lowerrootvp); free(xmp, M_NULLFSMNT); @@ -178,7 +179,7 @@ nullfs_mount(struct mount *mp) */ error = null_nodeget(mp, lowerrootvp, &nullm_rootvp); if (error != 0) { - vfs_unpin(xmp->nullm_vfs); + vfs_unregister_upper(xmp->nullm_vfs, &xmp->upper_node); vrele(lowerrootvp); free(xmp, M_NULLFSMNT); return (error); @@ -195,6 +196,11 @@ nullfs_mount(struct mount *mp) (xmp->nullm_vfs->mnt_kern_flag & MNTK_NULL_NOCACHE) != 0) xmp->nullm_flags &= ~NULLM_CACHE; + if ((xmp->nullm_flags & NULLM_CACHE) != 0) { + vfs_register_for_notification(xmp->nullm_vfs, mp, + &xmp->notify_node); + } + MNT_ILOCK(mp); if ((xmp->nullm_flags & NULLM_CACHE) != 0) { mp->mnt_kern_flag |= lowerrootvp->v_mount->mnt_kern_flag & @@ -206,13 +212,6 @@ nullfs_mount(struct mount *mp) (MNTK_USES_BCACHE | MNTK_NO_IOPF | MNTK_UNMAPPED_BUFS); MNT_IUNLOCK(mp); vfs_getnewfsid(mp); - if ((xmp->nullm_flags & NULLM_CACHE) != 0) { - MNT_ILOCK(xmp->nullm_vfs); - TAILQ_INSERT_TAIL(&xmp->nullm_vfs->mnt_uppers, mp, - mnt_upper_link); - MNT_IUNLOCK(xmp->nullm_vfs); - } - vfs_mountedfrom(mp, target); vput(nullm_rootvp); @@ -230,7 +229,6 @@ nullfs_unmount(mp, mntflags) int mntflags; { struct null_mount *mntdata; - struct mount *ump; int error, flags; NULLFSDEBUG("nullfs_unmount: mp = %p\n", (void *)mp); @@ -259,17 +257,11 @@ nullfs_unmount(mp, mntflags) * Finally, throw away the null_mount structure */ mntdata = mp->mnt_data; - ump = mntdata->nullm_vfs; if ((mntdata->nullm_flags & NULLM_CACHE) != 0) { - MNT_ILOCK(ump); - while ((ump->mnt_kern_flag & MNTK_VGONE_UPPER) != 0) { - ump->mnt_kern_flag |= MNTK_VGONE_WAITER; - msleep(&ump->mnt_uppers, &ump->mnt_mtx, 0, "vgnupw", 0); - } - TAILQ_REMOVE(&ump->mnt_uppers, mp, mnt_upper_link); - MNT_IUNLOCK(ump); + vfs_unregister_for_notification(mntdata->nullm_vfs, + &mntdata->notify_node); } - vfs_unpin(ump); + vfs_unregister_upper(mntdata->nullm_vfs, &mntdata->upper_node); vrele(mntdata->nullm_lowerrootvp); mp->mnt_data = NULL; free(mntdata, M_NULLFSMNT); diff --git a/sys/fs/unionfs/union.h b/sys/fs/unionfs/union.h index 64706b2b21a2..96180480dbec 100644 --- a/sys/fs/unionfs/union.h +++ b/sys/fs/unionfs/union.h @@ -57,6 +57,8 @@ struct unionfs_mount { struct vnode *um_lowervp; /* VREFed once */ struct vnode *um_uppervp; /* VREFed once */ struct vnode *um_rootvp; /* ROOT vnode */ + struct mount_upper_node um_lower_link; /* node in lower FS list of uppers */ + struct mount_upper_node um_upper_link; /* node in upper FS list of uppers */ unionfs_copymode um_copymode; unionfs_whitemode um_whitemode; uid_t um_uid; diff --git a/sys/fs/unionfs/union_vfsops.c b/sys/fs/unionfs/union_vfsops.c index 96a30f0ae8b5..c17650dedc63 100644 --- a/sys/fs/unionfs/union_vfsops.c +++ b/sys/fs/unionfs/union_vfsops.c @@ -292,14 +292,16 @@ unionfs_domount(struct mount *mp) return (error); } - lowermp = vfs_pin_from_vp(ump->um_lowervp); - uppermp = vfs_pin_from_vp(ump->um_uppervp); + lowermp = vfs_register_upper_from_vp(ump->um_lowervp, mp, + &ump->um_lower_link); + uppermp = vfs_register_upper_from_vp(ump->um_uppervp, mp, + &ump->um_upper_link); if (lowermp == NULL || uppermp == NULL) { if (lowermp != NULL) - vfs_unpin(lowermp); + vfs_unregister_upper(lowermp, &ump->um_lower_link); if (uppermp != NULL) - vfs_unpin(uppermp); + vfs_unregister_upper(uppermp, &ump->um_upper_link); free(ump, M_UNIONFSMNT); mp->mnt_data = NULL; return (ENOENT); @@ -357,8 +359,8 @@ unionfs_unmount(struct mount *mp, int mntflags) if (error) return (error); - vfs_unpin(ump->um_lowervp->v_mount); - vfs_unpin(ump->um_uppervp->v_mount); + vfs_unregister_upper(ump->um_lowervp->v_mount, &ump->um_lower_link); + vfs_unregister_upper(ump->um_uppervp->v_mount, &ump->um_upper_link); free(ump, M_UNIONFSMNT); mp->mnt_data = NULL; diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c index 8b4426209818..a52e9351bf50 100644 --- a/sys/kern/vfs_mount.c +++ b/sys/kern/vfs_mount.c @@ -65,6 +65,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -89,6 +90,11 @@ static bool default_autoro = false; SYSCTL_BOOL(_vfs, OID_AUTO, default_autoro, CTLFLAG_RW, &default_autoro, 0, "Retry failed r/w mount as r/o if no explicit ro/rw option is specified"); +static bool recursive_forced_unmount = false; +SYSCTL_BOOL(_vfs, OID_AUTO, recursive_forced_unmount, CTLFLAG_RW, + &recursive_forced_unmount, 0, "Recursively unmount stacked upper mounts" + " when a file system is forcibly unmounted"); + MALLOC_DEFINE(M_MOUNT, "mount", "vfs mount structure"); MALLOC_DEFINE(M_STATFS, "statfs", "statfs structure"); static uma_zone_t mount_zone; @@ -103,6 +109,16 @@ MTX_SYSINIT(mountlist, &mountlist_mtx, "mountlist", MTX_DEF); EVENTHANDLER_LIST_DEFINE(vfs_mounted); EVENTHANDLER_LIST_DEFINE(vfs_unmounted); +static void vfs_deferred_unmount(void *arg, int pending); +static struct task deferred_unmount_task = + TASK_INITIALIZER(0, vfs_deferred_unmount, NULL);; +static struct mtx deferred_unmount_lock; +MTX_SYSINIT(deferred_unmount, &deferred_unmount_lock, "deferred_unmount", + MTX_DEF); +static STAILQ_HEAD(, mount) deferred_unmount_list = + STAILQ_HEAD_INITIALIZER(deferred_unmount_list); +TASKQUEUE_DEFINE_THREAD(deferred_unmount); + static void mount_devctl_event(const char *type, struct mount *mp, bool donew); /* @@ -505,8 +521,21 @@ vfs_ref(struct mount *mp) MNT_IUNLOCK(mp); } +/* + * Register ump as an upper mount of the mount associated with + * vnode vp. This registration will be tracked through + * mount_upper_node upper, which should be allocated by the + * caller and stored in per-mount data associated with mp. + * + * If successful, this function will return the mount associated + * with vp, and will ensure that it cannot be unmounted until + * ump has been unregistered as one of its upper mounts. + * + * Upon failure this function will return NULL. + */ struct mount * -vfs_pin_from_vp(struct vnode *vp) +vfs_register_upper_from_vp(struct vnode *vp, struct mount *ump, + struct mount_upper_node *upper) { struct mount *mp; @@ -514,26 +543,81 @@ vfs_pin_from_vp(struct vnode *vp) if (mp == NULL) return (NULL); MNT_ILOCK(mp); - if (mp != vp->v_mount || (mp->mnt_kern_flag & MNTK_UNMOUNT) != 0) { + if (mp != vp->v_mount || + ((mp->mnt_kern_flag & (MNTK_UNMOUNT | MNTK_RECURSE)) != 0)) { MNT_IUNLOCK(mp); return (NULL); } + KASSERT(ump != mp, ("upper and lower mounts are identical")); + upper->mp = ump; MNT_REF(mp); - KASSERT(mp->mnt_pinned_count < INT_MAX, - ("mount pinned count overflow")); - ++mp->mnt_pinned_count; + TAILQ_INSERT_TAIL(&mp->mnt_uppers, upper, mnt_upper_link); MNT_IUNLOCK(mp); return (mp); } +/* + * Register upper mount ump to receive vnode unlink/reclaim + * notifications from lower mount mp. This registration will + * be tracked through mount_upper_node upper, which should be + * allocated by the caller and stored in per-mount data + * associated with mp. + * + * ump must already be registered as an upper mount of mp + * through a call to vfs_register_upper_from_vp(). + */ +void +vfs_register_for_notification(struct mount *mp, struct mount *ump, + struct mount_upper_node *upper) +{ + upper->mp = ump; + MNT_ILOCK(mp); + TAILQ_INSERT_TAIL(&mp->mnt_notify, upper, mnt_upper_link); + MNT_IUNLOCK(mp); +} + +static void +vfs_drain_upper_locked(struct mount *mp) +{ + mtx_assert(MNT_MTX(mp), MA_OWNED); + while (mp->mnt_upper_pending != 0) { + mp->mnt_kern_flag |= MNTK_UPPER_WAITER; + msleep(&mp->mnt_uppers, MNT_MTX(mp), 0, "mntupw", 0); + } +} + +/* + * Undo a previous call to vfs_register_for_notification(). + * The mount represented by upper must be currently registered + * as an upper mount for mp. + */ +void +vfs_unregister_for_notification(struct mount *mp, + struct mount_upper_node *upper) +{ + MNT_ILOCK(mp); + vfs_drain_upper_locked(mp); + TAILQ_REMOVE(&mp->mnt_notify, upper, mnt_upper_link); + MNT_IUNLOCK(mp); +} + +/* + * Undo a previous call to vfs_register_upper_from_vp(). + * This must be done before mp can be unmounted. + */ void -vfs_unpin(struct mount *mp) +vfs_unregister_upper(struct mount *mp, struct mount_upper_node *upper) { MNT_ILOCK(mp); - KASSERT(mp->mnt_pinned_count > 0, ("mount pinned count underflow")); KASSERT((mp->mnt_kern_flag & MNTK_UNMOUNT) == 0, - ("mount pinned with pending unmount")); - --mp->mnt_pinned_count; + ("registered upper with pending unmount")); + vfs_drain_upper_locked(mp); + TAILQ_REMOVE(&mp->mnt_uppers, upper, mnt_upper_link); + if ((mp->mnt_kern_flag & MNTK_TASKQUEUE_WAITER) != 0 && + TAILQ_EMPTY(&mp->mnt_uppers)) { + mp->mnt_kern_flag &= ~MNTK_TASKQUEUE_WAITER; + wakeup(&mp->taskqueue_link); + } MNT_REL(mp); MNT_IUNLOCK(mp); } @@ -600,8 +684,10 @@ vfs_mount_alloc(struct vnode *vp, struct vfsconf *vfsp, const char *fspath, mac_mount_create(cred, mp); #endif arc4rand(&mp->mnt_hashseed, sizeof mp->mnt_hashseed, 0); + mp->mnt_upper_pending = 0; TAILQ_INIT(&mp->mnt_uppers); - mp->mnt_pinned_count = 0; + TAILQ_INIT(&mp->mnt_notify); + mp->taskqueue_flags = 0; return (mp); } @@ -640,9 +726,9 @@ vfs_mount_destroy(struct mount *mp) vn_printf(vp, "dangling vnode "); panic("unmount: dangling vnode"); } - KASSERT(mp->mnt_pinned_count == 0, - ("mnt_pinned_count = %d", mp->mnt_pinned_count)); + KASSERT(mp->mnt_upper_pending == 0, ("mnt_upper_pending")); KASSERT(TAILQ_EMPTY(&mp->mnt_uppers), ("mnt_uppers")); + KASSERT(TAILQ_EMPTY(&mp->mnt_notify), ("mnt_notify")); if (mp->mnt_nvnodelistsize != 0) panic("vfs_mount_destroy: nonzero nvnodelistsize"); if (mp->mnt_lazyvnodelistsize != 0) @@ -1799,17 +1885,164 @@ vfs_mount_fetch_counter(struct mount *mp, enum mount_counter which) return (sum); } +static bool +deferred_unmount_enqueue(struct mount *mp, uint64_t flags, bool requeue) +{ + bool enqueued; + + enqueued = false; + mtx_lock(&deferred_unmount_lock); + if ((mp->taskqueue_flags & MNT_TASKQUEUE) == 0 || requeue) { + mp->taskqueue_flags = flags | MNT_TASKQUEUE; + STAILQ_INSERT_TAIL(&deferred_unmount_list, mp, taskqueue_link); + enqueued = true; + } + mtx_unlock(&deferred_unmount_lock); + + if (enqueued) { + taskqueue_enqueue(taskqueue_deferred_unmount, + &deferred_unmount_task); + } + + return (enqueued); +} + +/* + * Taskqueue handler for processing async/recursive unmounts + */ +static void +vfs_deferred_unmount(void *argi __unused, int pending __unused) +{ + STAILQ_HEAD(, mount) local_unmounts; + uint64_t flags; + struct mount *mp, *tmp; + bool unmounted; + + STAILQ_INIT(&local_unmounts); + mtx_lock(&deferred_unmount_lock); + STAILQ_CONCAT(&local_unmounts, &deferred_unmount_list); + mtx_unlock(&deferred_unmount_lock); + + STAILQ_FOREACH_SAFE(mp, &local_unmounts, taskqueue_link, tmp) { + flags = mp->taskqueue_flags; + KASSERT((flags & MNT_TASKQUEUE) != 0, + ("taskqueue unmount without MNT_TASKQUEUE")); + if (dounmount(mp, flags, curthread) != 0) { + MNT_ILOCK(mp); + unmounted = ((mp->mnt_kern_flag & MNTK_REFEXPIRE) != 0); + MNT_IUNLOCK(mp); + if (!unmounted) + deferred_unmount_enqueue(mp, flags, true); + else + vfs_rel(mp); + } + } +} + /* * Do the actual filesystem unmount. */ int -dounmount(struct mount *mp, int flags, struct thread *td) +dounmount(struct mount *mp, uint64_t flags, struct thread *td) { + struct mount_upper_node *upper; struct vnode *coveredvp, *rootvp; int error; uint64_t async_flag; int mnt_gen_r; + KASSERT((flags & MNT_TASKQUEUE) == 0 || + (flags & (MNT_RECURSE | MNT_FORCE)) == (MNT_RECURSE | MNT_FORCE), + ("MNT_TASKQUEUE requires MNT_RECURSE | MNT_FORCE")); + /* + * If the caller has explicitly requested the unmount to be handled by + * the taskqueue and we're not already in taskqueue context, queue + * up the unmount request and exit. This is done prior to any + * credential checks; MNT_TASKQUEUE should be used only for kernel- + * initiated unmounts and will therefore be processed with the + * (kernel) credentials of the taskqueue thread. Still, callers + * should be sure this is the behavior they want. + */ + if ((flags & MNT_TASKQUEUE) != 0 && + taskqueue_member(taskqueue_deferred_unmount, curthread) == 0) { + if (!deferred_unmount_enqueue(mp, flags, false)) + vfs_rel(mp); + return (EINPROGRESS); + } + + /* + * Only privileged root, or (if MNT_USER is set) the user that did the + * original mount is permitted to unmount this filesystem. + * This check should be made prior to queueing up any recursive + * unmounts of upper filesystems. Those unmounts will be executed + * with kernel thread credentials and are expected to succeed, so + * we must at least ensure the originating context has sufficient + * privilege to unmount the base filesystem before proceeding with + * the uppers. + */ + error = vfs_suser(mp, td); + if (error != 0) { + KASSERT((flags & MNT_TASKQUEUE) == 0, + ("taskqueue unmount with insufficient privilege")); + vfs_rel(mp); + return (error); + } + + if (recursive_forced_unmount && ((flags & MNT_FORCE) != 0)) + flags |= MNT_RECURSE; + + if ((flags & MNT_RECURSE) != 0) { + KASSERT((flags & MNT_FORCE) != 0, + ("MNT_RECURSE requires MNT_FORCE")); + + MNT_ILOCK(mp); + /* + * Set MNTK_RECURSE to prevent new upper mounts from being + * added, and note that an operation on the uppers list is in + * progress. This will ensure that unregistration from the + * uppers list, and therefore any pending unmount of the upper + * FS, can't complete until after we finish walking the list. + */ + mp->mnt_kern_flag |= MNTK_RECURSE; + mp->mnt_upper_pending++; + TAILQ_FOREACH(upper, &mp->mnt_uppers, mnt_upper_link) { + MNT_IUNLOCK(mp); + vfs_ref(upper->mp); + if (!deferred_unmount_enqueue(upper->mp, flags, false)) + vfs_rel(upper->mp); + MNT_ILOCK(mp); + } + mp->mnt_upper_pending--; + if ((mp->mnt_kern_flag & MNTK_UPPER_WAITER) != 0 && + mp->mnt_upper_pending == 0) { + mp->mnt_kern_flag &= ~MNTK_UPPER_WAITER; + wakeup(&mp->mnt_uppers); + } + /* + * If we're not on the taskqueue, wait until the uppers list + * is drained before proceeding with unmount. Otherwise, if + * we are on the taskqueue and there are still pending uppers, + * just re-enqueue on the end of the taskqueue. + */ + if ((flags & MNT_TASKQUEUE) == 0) { + while (!TAILQ_EMPTY(&mp->mnt_uppers)) { + mp->mnt_kern_flag |= MNTK_TASKQUEUE_WAITER; + msleep(&mp->taskqueue_link, MNT_MTX(mp), 0, + "umntqw", 0); + } + } else if (!TAILQ_EMPTY(&mp->mnt_uppers)) { + MNT_IUNLOCK(mp); + deferred_unmount_enqueue(mp, flags, true); + return (0); + } + MNT_IUNLOCK(mp); + KASSERT(TAILQ_EMPTY(&mp->mnt_uppers), ("mnt_uppers not empty")); + } + + /* Allow the taskqueue to safely re-enqueue on failure */ + if ((flags & MNT_TASKQUEUE) != 0) + vfs_ref(mp); + if ((coveredvp = mp->mnt_vnodecovered) != NULL) { mnt_gen_r = mp->mnt_gen; VI_LOCK(coveredvp); @@ -1828,27 +2061,13 @@ dounmount(struct mount *mp, int flags, struct thread *td) } } - /* - * Only privileged root, or (if MNT_USER is set) the user that did the - * original mount is permitted to unmount this filesystem. - */ - error = vfs_suser(mp, td); - if (error != 0) { - if (coveredvp != NULL) { - VOP_UNLOCK(coveredvp); - vdrop(coveredvp); - } - vfs_rel(mp); - return (error); - } - vfs_op_enter(mp); vn_start_write(NULL, &mp, V_WAIT | V_MNTREF); MNT_ILOCK(mp); if ((mp->mnt_kern_flag & MNTK_UNMOUNT) != 0 || (mp->mnt_flag & MNT_UPDATE) != 0 || - mp->mnt_pinned_count != 0) { + !TAILQ_EMPTY(&mp->mnt_uppers)) { dounmount_cleanup(mp, coveredvp, 0); return (EBUSY); } @@ -1952,6 +2171,7 @@ dounmount(struct mount *mp, int flags, struct thread *td) } return (error); } + mtx_lock(&mountlist_mtx); TAILQ_REMOVE(&mountlist, mp, mnt_list); mtx_unlock(&mountlist_mtx); @@ -1977,6 +2197,8 @@ dounmount(struct mount *mp, int flags, struct thread *td) } if (mp == rootdevmp) rootdevmp = NULL; + if ((flags & MNT_TASKQUEUE) != 0) + vfs_rel(mp); vfs_mount_destroy(mp); return (0); } diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c index a2f25bf78495..8add6951645f 100644 --- a/sys/kern/vfs_subr.c +++ b/sys/kern/vfs_subr.c @@ -831,9 +831,9 @@ vfs_busy(struct mount *mp, int flags) * valid. */ while (mp->mnt_kern_flag & MNTK_UNMOUNT) { - KASSERT(mp->mnt_pinned_count == 0, - ("%s: non-zero pinned count %d with pending unmount", - __func__, mp->mnt_pinned_count)); + KASSERT(TAILQ_EMPTY(&mp->mnt_uppers), + ("%s: non-empty upper mount list with pending unmount", + __func__)); if (flags & MBF_NOWAIT || mp->mnt_kern_flag & MNTK_REFEXPIRE) { MNT_REL(mp); MNT_IUNLOCK(mp); @@ -3897,6 +3897,11 @@ notify_lowervp_vfs_dummy(struct mount *mp __unused, { } +struct notify_mount { + struct mount mp; + struct mount_upper_node upper; +}; + /* * Notify upper mounts about reclaimed or unlinked vnode. */ @@ -3907,45 +3912,52 @@ vfs_notify_upper(struct vnode *vp, int event) .vfs_reclaim_lowervp = notify_lowervp_vfs_dummy, .vfs_unlink_lowervp = notify_lowervp_vfs_dummy, }; - struct mount *mp, *ump, *mmp; + struct mount *mp; + struct mount_upper_node *ump; + struct notify_mount *mmp; mp = vp->v_mount; if (mp == NULL) return; - if (TAILQ_EMPTY(&mp->mnt_uppers)) + if (TAILQ_EMPTY(&mp->mnt_notify)) return; - mmp = malloc(sizeof(struct mount), M_TEMP, M_WAITOK | M_ZERO); - mmp->mnt_op = &vgonel_vfsops; - mmp->mnt_kern_flag |= MNTK_MARKER; + mmp = malloc(sizeof(*mmp), M_TEMP, M_WAITOK | M_ZERO); + mmp->mp.mnt_op = &vgonel_vfsops; + mmp->mp.mnt_kern_flag |= MNTK_MARKER; + mmp->upper.mp = &mmp->mp; MNT_ILOCK(mp); - mp->mnt_kern_flag |= MNTK_VGONE_UPPER; - for (ump = TAILQ_FIRST(&mp->mnt_uppers); ump != NULL;) { - if ((ump->mnt_kern_flag & MNTK_MARKER) != 0) { + mp->mnt_upper_pending++; + KASSERT(mp->mnt_upper_pending > 0, + ("%s: mnt_upper_pending %d", __func__, mp->mnt_upper_pending)); + for (ump = TAILQ_FIRST(&mp->mnt_notify); ump != NULL;) { + if ((ump->mp->mnt_kern_flag & MNTK_MARKER) != 0) { ump = TAILQ_NEXT(ump, mnt_upper_link); continue; } - TAILQ_INSERT_AFTER(&mp->mnt_uppers, ump, mmp, mnt_upper_link); + TAILQ_INSERT_AFTER(&mp->mnt_notify, ump, &mmp->upper, + mnt_upper_link); MNT_IUNLOCK(mp); switch (event) { case VFS_NOTIFY_UPPER_RECLAIM: - VFS_RECLAIM_LOWERVP(ump, vp); + VFS_RECLAIM_LOWERVP(ump->mp, vp); break; case VFS_NOTIFY_UPPER_UNLINK: - VFS_UNLINK_LOWERVP(ump, vp); + VFS_UNLINK_LOWERVP(ump->mp, vp); break; default: KASSERT(0, ("invalid event %d", event)); break; } MNT_ILOCK(mp); - ump = TAILQ_NEXT(mmp, mnt_upper_link); - TAILQ_REMOVE(&mp->mnt_uppers, mmp, mnt_upper_link); + ump = TAILQ_NEXT(&mmp->upper, mnt_upper_link); + TAILQ_REMOVE(&mp->mnt_notify, &mmp->upper, mnt_upper_link); } free(mmp, M_TEMP); - mp->mnt_kern_flag &= ~MNTK_VGONE_UPPER; - if ((mp->mnt_kern_flag & MNTK_VGONE_WAITER) != 0) { - mp->mnt_kern_flag &= ~MNTK_VGONE_WAITER; + mp->mnt_upper_pending--; + if ((mp->mnt_kern_flag & MNTK_UPPER_WAITER) != 0 && + mp->mnt_upper_pending == 0) { + mp->mnt_kern_flag &= ~MNTK_UPPER_WAITER; wakeup(&mp->mnt_uppers); } MNT_IUNLOCK(mp); @@ -4376,12 +4388,13 @@ DB_SHOW_COMMAND(mount, db_show_mount) MNT_KERN_FLAG(MNTK_EXTENDED_SHARED); MNT_KERN_FLAG(MNTK_SHARED_WRITES); MNT_KERN_FLAG(MNTK_NO_IOPF); - MNT_KERN_FLAG(MNTK_VGONE_UPPER); - MNT_KERN_FLAG(MNTK_VGONE_WAITER); + MNT_KERN_FLAG(MNTK_RECURSE); + MNT_KERN_FLAG(MNTK_UPPER_WAITER); MNT_KERN_FLAG(MNTK_LOOKUP_EXCL_DOTDOT); MNT_KERN_FLAG(MNTK_MARKER); MNT_KERN_FLAG(MNTK_USES_BCACHE); MNT_KERN_FLAG(MNTK_FPLOOKUP); + MNT_KERN_FLAG(MNTK_TASKQUEUE_WAITER); MNT_KERN_FLAG(MNTK_NOASYNC); MNT_KERN_FLAG(MNTK_UNMOUNT); MNT_KERN_FLAG(MNTK_MWAIT); diff --git a/sys/sys/mount.h b/sys/sys/mount.h index 693293b12370..1941d58a28d9 100644 --- a/sys/sys/mount.h +++ b/sys/sys/mount.h @@ -190,6 +190,19 @@ struct mount_pcpu { _Static_assert(sizeof(struct mount_pcpu) == 16, "the struct is allocated from pcpu 16 zone"); +/* + * Structure for tracking a stacked filesystem mounted above another + * filesystem. This is expected to be stored in the upper FS' per-mount data. + * + * Lock reference: + * i - lower mount interlock + * c - constant from node initialization + */ +struct mount_upper_node { + struct mount *mp; /* (c) mount object for upper FS */ + TAILQ_ENTRY(mount_upper_node) mnt_upper_link; /* (i) position in uppers list */ +}; + /* * Structure per mounted filesystem. Each mounted filesystem has an * array of operations and an instance record. The filesystems are @@ -199,8 +212,8 @@ _Static_assert(sizeof(struct mount_pcpu) == 16, * l - mnt_listmtx * m - mountlist_mtx * i - interlock - * i* - interlock of uppers' list head * v - vnode freelist mutex + * d - deferred unmount list mutex * * Unmarked fields are considered stable as long as a ref is held. * @@ -242,10 +255,12 @@ struct mount { struct mtx mnt_listmtx; struct vnodelst mnt_lazyvnodelist; /* (l) list of lazy vnodes */ int mnt_lazyvnodelistsize; /* (l) # of lazy vnodes */ - int mnt_pinned_count; /* (i) unmount prevented */ + int mnt_upper_pending; /* (i) # of pending ops on mnt_uppers */ struct lock mnt_explock; /* vfs_export walkers lock */ - TAILQ_ENTRY(mount) mnt_upper_link; /* (i*) we in the all uppers */ - TAILQ_HEAD(, mount) mnt_uppers; /* (i) upper mounts over us */ + TAILQ_HEAD(, mount_upper_node) mnt_uppers; /* (i) upper mounts over us */ + TAILQ_HEAD(, mount_upper_node) mnt_notify; /* (i) upper mounts for notification */ + STAILQ_ENTRY(mount) taskqueue_link; /* (d) our place in deferred unmount list */ + uint64_t taskqueue_flags; /* (d) unmount flags passed from taskqueue */ }; #endif /* _WANT_MOUNT || _KERNEL */ @@ -438,9 +453,13 @@ struct mntoptnames { #define MNT_BYFSID 0x0000000008000000ULL /* specify filesystem by ID. */ #define MNT_NOCOVER 0x0000001000000000ULL /* Do not cover a mount point */ #define MNT_EMPTYDIR 0x0000002000000000ULL /* Only mount on empty dir */ -#define MNT_CMDFLAGS (MNT_UPDATE | MNT_DELEXPORT | MNT_RELOAD | \ +#define MNT_RECURSE 0x0000100000000000ULL /* recursively unmount uppers */ +#define MNT_TASKQUEUE 0x0000200000000000ULL /* unmount in taskqueue context */ +#define MNT_CMDFLAGS (MNT_UPDATE | MNT_DELEXPORT | MNT_RELOAD | \ MNT_FORCE | MNT_SNAPSHOT | MNT_NONBUSY | \ - MNT_BYFSID | MNT_NOCOVER | MNT_EMPTYDIR) + MNT_BYFSID | MNT_NOCOVER | MNT_EMPTYDIR | \ + MNT_RECURSE | MNT_TASKQUEUE) + /* * Internal filesystem control flags stored in mnt_kern_flag. * @@ -466,8 +485,8 @@ struct mntoptnames { #define MNTK_NO_IOPF 0x00000100 /* Disallow page faults during reads and writes. Filesystem shall properly handle i/o state on EFAULT. */ -#define MNTK_VGONE_UPPER 0x00000200 -#define MNTK_VGONE_WAITER 0x00000400 +#define MNTK_RECURSE 0x00000200 /* pending recursive unmount */ +#define MNTK_UPPER_WAITER 0x00000400 /* waiting to drain MNTK_UPPER_PENDING */ #define MNTK_LOOKUP_EXCL_DOTDOT 0x00000800 #define MNTK_MARKER 0x00001000 #define MNTK_UNMAPPED_BUFS 0x00002000 @@ -477,8 +496,9 @@ struct mntoptnames { #define MNTK_UNIONFS 0x00020000 /* A hack for F_ISUNIONSTACK */ #define MNTK_FPLOOKUP 0x00040000 /* fast path lookup is supported */ #define MNTK_SUSPEND_ALL 0x00080000 /* Suspended by all-fs suspension */ -#define MNTK_NOASYNC 0x00800000 /* disable async */ -#define MNTK_UNMOUNT 0x01000000 /* unmount in progress */ +#define MNTK_TASKQUEUE_WAITER 0x00100000 /* Waiting on unmount taskqueue */ +#define MNTK_NOASYNC 0x00800000 /* disable async */ +#define MNTK_UNMOUNT 0x01000000 /* unmount in progress */ #define MNTK_MWAIT 0x02000000 /* waiting for unmount to finish */ #define MNTK_SUSPEND 0x08000000 /* request write suspension */ #define MNTK_SUSPEND2 0x04000000 /* block secondary writes */ @@ -952,7 +972,7 @@ vfs_statfs_t __vfs_statfs; * exported vnode operations */ -int dounmount(struct mount *, int, struct thread *); +int dounmount(struct mount *, uint64_t, struct thread *); int kernel_mount(struct mntarg *ma, uint64_t flags); int kernel_vmount(int flags, ...); @@ -1012,8 +1032,13 @@ struct mount *vfs_mount_alloc(struct vnode *, struct vfsconf *, const char *, int vfs_suser(struct mount *, struct thread *); void vfs_unbusy(struct mount *); void vfs_unmountall(void); -struct mount *vfs_pin_from_vp(struct vnode *); -void vfs_unpin(struct mount *); +struct mount *vfs_register_upper_from_vp(struct vnode *, + struct mount *ump, struct mount_upper_node *); +void vfs_register_for_notification(struct mount *, struct mount *, + struct mount_upper_node *); +void vfs_unregister_for_notification(struct mount *, + struct mount_upper_node *); +void vfs_unregister_upper(struct mount *, struct mount_upper_node *); extern TAILQ_HEAD(mntlist, mount) mountlist; /* mounted filesystem list */ extern struct mtx_padalign mountlist_mtx; extern struct nfs_public nfs_pub; diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c index 6b7407eb88f9..689c85d7bb1f 100644 --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -297,7 +297,7 @@ ffs_fsfail_unmount(void *v, int pending) */ mp = vfs_getvfs(&etp->fsid); if (mp != NULL) - dounmount(mp, MNT_FORCE, curthread); + dounmount(mp, MNT_FORCE | MNT_RECURSE, curthread); free(etp, M_UFSMNT); } -- Test scenario: misc/force3.sh