panic: page fault, vmcore.487 KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2022 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #1 ufs-n252568-613a2d5355d-dirty: Fri Jan 21 11:17:24 CET 2022 pho@mercat1.netperf.freebsd.org:/var/tmp/deviant3/sys/amd64/compile/PHO amd64 FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.11-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33176580096 (31639 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 5 11 2 4 9 8 7 6 3 10 random: entropy device external interface kbd1 at kbdmux0 vtvga0: smbios0: at iomem 0xf0560-0xf057e smbios0: Version: 2.8, BCD Revision: 2.7 aesni0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 apei0: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: EEPROM V1.63-0 eTrack 0x800009fa igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: EEPROM V1.63-0 eTrack 0x800009fa igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb433000-0xfb4333ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb432000-0xfb4327ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounter "TSC-low" frequency 1749998685 Hz quality 1000 Timecounters tick every 1.000 msec Expensive timeout(9) function: 0xffffffff805086d0(0xfffffe0109fa4000) 0.004954909 s ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ses0: pass0 in 'Slot 01', SATA Slot: scbus1 target 0 ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> Expensive timeout(9) function: 0xffffffff80a51630(0xffffffff81ae9598) 0.007400547 s mlx5_core0: mem 0xf8000000-0xf9ffffff irq 32 at device 0.0 on pci3 mlx5: Mellanox Core driver 3.7.0 (July 2021)ugen2.1: at usbus2 ugen1.1: at usbus1 uhub0ugen0.1: <0x8086 XHCI root HUB> at usbus0 on usbus2 uhub0: on usbus2 uhub1 on usbus0 uhub1: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub2 on usbus1 uhub2: on usbus1 mlx5_core0: WARN: mlx5_vsc_set_space:124:(pid 0): Space 0x7 is not supported. mlx5_core0: WARN: mlx5_fwdump_prep:99:(pid 0): VSC scan space is not supported mlx5_core0: INFO: init_one:1614:(pid 0): cannot find SR-IOV PCIe cap mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 1, l2 table size(65536), per vport: m fsckugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 ** /dev/gpt/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 445 files, 23306 used, 1242101 free (53 frags, 155256 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc10 /usr/local/lib/gcc8 /usr/local/lib/gcc9 /usr/local/lib/perl5/5.32/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: mercat1.netperf.freebsd.org. Setting up harvesting: PURE_RDRAND,[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . Autoloading module: acpi_wmi.ko Autoloading module: ichsmb.ko Autoloading module: ioat.ko acpi_wmi0: on acpi0 acpi_wmi0: cannot find EC device ioat0: mem 0xfb42c000-0xfb42ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0xfb428000-0xfb42bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0xfb424000-0xfb427fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0xfb420000-0xfb423fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0xfb41c000-0xfb41ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0xfb418000-0xfb41bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0xfb414000-0xfb417fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0xfb410000-0xfb413fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 ichsmb0: port 0x580-0x59f mem 0xfb431000-0xfb4310ff irq 18 at device 31.3 on pci1 smbus0: on ichsmb0 lo0: link state changed to UP Starting Network: lo0 igb0 igb1 mce0 mce1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8863 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect status: no carrier nd6 options=21 igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Autoloading module: acpi_wmi.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: ichsmb.ko Starting Network: mce0. mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 Starting Network: mce1. mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid.ko Autoloading module: ums.ko Autoloading module: usbhid.ko ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192if_delmulti_locked: detaching ifnet instance 0xfffff8000495d800 .168.5.1 bound to 192.168.5.171 -- renewal in 300 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Recovering vi editor sessions:. Updating motd:. Clearing /tmp (X related). Starting syslogd. Starting watchdogd. No core dumps found. Setting date via ntp. 21 Jan 11:21:46 ntpdate[2587]: step time server 192.168.5.1 offset +0.847415 sec Starting rpcbind. NFS access cache time=60 Starting statd. Starting lockd. Starting powerd. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Mounting late filesystems:. Starting inetd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. sysctl: unknown oid 'debug.fail_point.pmap_allocpte_page_alloc' at line 12 Fri Jan 21 11:21 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Jan 21 11:21:58 mercat1 su[2739]: pho to root on /dev/pts/0 20220121 11:22:42 all (1/19): altbufferflushes.sh FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: stress2: newblk leak: 192549/192564. 20220121 11:36:08 all (2/19): flock_open_close.sh 20220121 11:39:18 all (3/19): quota6.sh lock order reversal: 1st 0xfffff80076fd6e98 FFS (FFS Lock, sleep mutex) @ ufs/ufs/ufs_quota.c:755 2nd 0xfffffe0143564740 struct mount mtx (struct mount mtx, sleep mutex) @ ufs/ufs/ufs_quota.c:761 lock order struct mount mtx -> FFS Lock established at: #0 0xffffffff80c7630e at witness_checkorder+0x31e #1 0xffffffff80be0b34 at __mtx_lock_flags+0x94 #2 0xffffffff80f23af5 at softdep_flushfiles+0x395 #3 0xffffffff80f4475b at ffs_unmount+0xab #4 0xffffffff80cef8ea at dounmount+0x7ea #5 0xffffffff80cef0b7 at kern_unmount+0x2d7 #6 0xffffffff810e1455 at amd64_syscall+0x145 #7 0xffffffff810b246b at fast_syscall_common+0xf8 lock order FFS Lock -> struct mount mtx attempted at: #0 0xffffffff80c76bcd at witness_checkorder+0xbdd #1 0xffffffff80be0b34 at __mtx_lock_flags+0x94 #2 0xffffffff80f58a8d at quotaoff_inchange+0xbd #3 0xffffffff80f42eba at ffs_flushfiles+0x10a #4 0xffffffff80f2380d at softdep_flushfiles+0xad #5 0xffffffff80f4475b at ffs_unmount+0xab #6 0xffffffff80cef8ea at dounmount+0x7ea #7 0xffffffff80cef0b7 at kern_unmount+0x2d7 #8 0xffffffff810e1455 at amd64_syscall+0x145 #9 0xffffffff810b246b at fast_syscall_common+0xf8 stress2: newblk leak: 138/142. 20220121 11:39:20 all (4/19): quota8.sh stress2: newblk leak: 28531/28674. 20220121 11:40:11 all (5/19): recursiveflushes.sh 20220121 11:40:14 all (6/19): snap.sh lock order reversal: 1st 0xfffff800040d8e30 snaplk (snaplk, lockmgr) @ kern/vfs_default.c:1217 2nd 0xfffffe00381d78a0 bufwait (bufwait, lockmgr) @ kern/vfs_subr.c:2190 lock order bufwait -> snaplk established at: #0 0xffffffff80c7630e at witness_checkorder+0x31e #1 0xffffffff80bd458c at lockmgr_xlock_hard+0x6c #2 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #3 0xffffffff80f1e7e5 at _lockmgr_args+0x85 #4 0xffffffff80f21ee6 at ffs_copyonwrite+0x2e6 #5 0xffffffff80f46847 at ffs_geom_strategy+0x107 #6 0xffffffff80f5df53 at ufs_strategy+0x83 #7 0xffffffff811adbd1 at VOP_STRATEGY_APV+0x61 #8 0xffffffff80ccf7f9 at VOP_STRATEGY+0x39 #9 0xffffffff80cc740f at bufstrategy+0xbf #10 0xffffffff80cca4bd at bstrategy+0xdd #11 0xffffffff80cc72c1 at bufwrite+0x291 #12 0xffffffff80ccc006 at bwrite+0xd6 #13 0xffffffff80ccb612 at bawrite+0x22 #14 0xffffffff80ce379f at cluster_wbuild+0x73f #15 0xffffffff80ccb59e at vfs_bio_awrite+0x28e #16 0xffffffff80cc76de at bufbdflush+0x22e #17 0xffffffff80ccb72c at bdwrite+0x10c lock order snaplk -> bufwait attempted at: #0 0xffffffff80c76bcd at witness_checkorder+0xbdd #1 0xffffffff80bd458c at lockmgr_xlock_hard+0x6c #2 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #3 0xffffffff80cf8760 at bnoreuselist+0xc0 #4 0xffffffff80ce4d3e at vop_stdadvise+0x14e #5 0xffffffff811b0149 at VOP_ADVISE_APV+0x59 #6 0xffffffff80702664 at mdstart_vnode+0x564 #7 0xffffffff8070005c at md_kthread+0x20c #8 0xffffffff80bbd110 at fork_exit+0x80 #9 0xffffffff810b2bce at fork_trampoline+0xe 20220121 11:40:16 all (7/19): snap10.sh 20220121 11:40:19 all (8/19): snap11.sh 20220121 11:40:36 all (9/19): snap12.sh 20220121 11:40:38 all (10/19): snap2-1.sh 20220121 11:40:39 all (11/19): snap2.sh 20220121 11:40:41 all (12/19): snap3.sh 2 20220121 12:00:43 all (13/19): snap4.sh 20220121 12:20:44 all (14/19): snap5-1.sh 20220121 12:30:47 all (15/19): snap5.sh stress2: pts leak: 1/2 20220121 12:51:31 all (16/19): snap6.sh 20220121 13:02:01 all (17/19): snap7.sh 20220121 13:22:03 all (18/19): snap9.sh Jan 21 13:22:05 mercat1 kernel: pid 97028 (dd), uid 0 inumber 8 on /mnt: filesystem full 20220121 13:22:06 all (19/19): trim5.sh 20220121 13:22:08 all.sh done, elapsed 0 day(s), 01:59.26 20220121 13:22:11 all (1/19): altbufferflushes.sh stress2: newblk leak: 258102/258112. 20220121 13:35:04 all (2/19): flock_open_close.sh 20220121 13:38:23 all (3/19): quota6.sh stress2: newblk leak: 137/141. 20220121 13:38:25 all (4/19): quota8.sh stress2: newblk leak: 28532/28674. 20220121 13:39:24 all (5/19): recursiveflushes.sh 20220121 13:39:27 all (6/19): snap.sh 20220121 13:39:29 all (7/19): snap10.sh 20220121 13:39:32 all (8/19): snap11.sh 20220121 13:41:34 all (9/19): snap12.sh 20220121 13:41:36 all (10/19): snap2-1.sh 20220121 13:41:38 all (11/19): snap2.sh 20220121 13:41:40 all (12/19): snap3.sh 20220121 14:01:42 all (13/19): snap4.sh 20220121 14:21:44 all (14/19): snap5-1.sh 20220121 14:32:18 all (15/19): snap5.sh 20220121 14:53:39 all (16/19): snap6.sh 20220121 15:04:19 all (17/19): snap7.sh 20220121 15:24:21 all (18/19): snap9.sh Jan 21 15:24:23 mercat1 kernel: pid 17672 (dd), uid 0 inumber 8 on /mnt: filesystem full 20220121 15:24:24 all (19/19): trim5.sh 20220121 15:24:26 all.sh done, elapsed 0 day(s), 02:02.15 login: root Jan 21 15:24:46 mercat1 login[2728]: ROOT LOGIN (root) ON ttyu1 Last login: Wed Jan 19 18:36:48 on ttyu1 FreeBSD 14.0-CURRENT (PHO) #1 ufs-n252568-613a2d5355d-dirty: Fri Jan 21 11:17:24 CET 2022 root@mercat1:~ # cd /usr/src/tools/test/stress2/misc root@mercat1:/usr/src/tools/test/stress2/misc # ./sbnap11.sh root@mercat1:/usr/src/tools/test/stress2/misc # ./snap11.sh root@mercat1:/usr/src/tools/test/stress2/misc # ./snap11.sh root@mercat1:/usr/src/tools/test/stress2/misc # Fatal trap 12: page fault while in kernel mode cpuid = 9; apic id = 09 fault virtual address = 0xf fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80f2023c stack pointer = 0x28:0xfffffe01439384b0 frame pointer = 0x28:0xfffffe01439385b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 81128 (mkdir) trap number = 12 panic: page fault cpuid = 9 time = 1642777290 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0143938270 vpanic() at vpanic+0x17f/frame 0xfffffe01439382c0 panic() at panic+0x43/frame 0xfffffe0143938320 trap_fatal() at trap_fatal+0x385/frame 0xfffffe0143938380 trap_pfault() at trap_pfault+0xab/frame 0xfffffe01439383e0 calltrap() at calltrap+0x8/frame 0xfffffe01439383e0 --- trap 0xc, rip = 0xffffffff80f2023c, rsp = 0xfffffe01439384b0, rbp = 0xfffffe01439385b0 --- ffs_snapblkfree() at ffs_snapblkfree+0x14c/frame 0xfffffe01439385b0 ffs_blkfree() at ffs_blkfree+0x62/frame 0xfffffe0143938610 freework_freeblock() at freework_freeblock+0x714/frame 0xfffffe0143938690 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x1eb/frame 0xfffffe01439386e0 softdep_setup_freeblocks() at softdep_setup_freeblocks+0x97c/frame 0xfffffe0143938750 ffs_truncate() at ffs_truncate+0xd8e/frame 0xfffufs_inactive() at ufs_inactive+0x1aa/frame 0xfffffe0143938bd0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x59/frame 0xfffffe0143938bf0 vinactivef() at vinactivef+0xda/frame 0xfffffe0143938c30 vput_final() at vput_final+0x2aa/frame 0xfffffe0143938c90 kern_frmdirat() at kern_frmdirat+0x2d0/frame 0xfffffe0143938e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143938f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143938f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c72a8, rbp = 0x42c76e0 --- KDB: enter: panic [ thread pid 81128 tid 100370 ] Stopped at kdb_enter+0x37: movq $0,0x128552e(%rip) db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> dump Dumping 1751 out of 32626 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:1:pho1> bt Tracing pid 81128 tid 100370 td 0xfffffe01437a0020 kdb_enter() at kdb_enter+0x37/frame 0xfffffe0143938270 vpanic() at vpanic+0x1b0/frame 0xfffffe01439382c0 panic() at panic+0x43/frame 0xfffffe0143938320 trap_fatal() at trap_fatal+0x385/frame 0xfffffe0143938380 trap_pfault() at trap_pfault+0xab/frame 0xfffffe01439383e0 calltrap() at calltrap+0x8/frame 0xfffffe01439383e0 --- trap 0xc, rip = 0xffffffff80f2023c, rsp = 0xfffffe01439384b0, rbp = 0xfffffe01439385b0 --- ffs_snapblkfree() at ffs_snapblkfree+0x14c/frame 0xfffffe01439385b0 ffs_blkfree() at ffs_blkfree+0x62/frame 0xfffffe0143938610 freework_freeblock() at freework_freeblock+0x714/frame 0xfffffe0143938690 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x1eb/frame 0xfffffe01439386e0 softdep_setup_freeblocks() at softdep_setup_freeblocks+0x97c/frame 0xfffffe0143938750 ffs_truncate() at ffs_truncate+0xd8e/frame 0xfffffe0143938b90 ufs_inactive() at ufs_inactive+0x1aa/frame 0xfffffe0143938bd0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x59/frame 0xfffffe0143938bf0 vinactivef() at vinactivef+0xda/frame 0xfffffe0143938c30 vput_final() at vput_final+0x2aa/frame 0xfffffe0143938c90 kern_frmdirat() at kern_frmdirat+0x2d0/frame 0xfffffe0143938e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143938f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143938f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c72a8, rbp = 0x2e1d642c76e0 --- db:1:pho1> show allpcpu Current CPU: 9 cpuid = 0 dynamic pcpu = 0x7c5600 curthread = 0xfffffe00e4c603a0: pid 11 tid 100003 critnest 1 "idle: cpu0" curpcb = 0xfffffe00e4c608b0 fpcurthread = none idlethread = 0xfffffe00e4c603a0: tid 100003 "idle: cpu0" self = 0xffffffff82410000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82410384 rsp0 = 0xfffffe00377f1000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x22a392951 gs32p = 0xffffffff82410404 ldt = 0xffffffff82410444 tss = 0xffffffff82410434 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00bb287600 curthread = 0xfffffe010e3e5e40: pid 84874 tid 100216 critnest 0 "egrep" curpcb = 0xfffffe010e3e6350 fpcurthread = 0xfffffe010e3e5e40: pid 84874 "egrep" idlethread = 0xfffffe00e4c5fc80: tid 100004 "idle: cpu1" self = 0xffffffff82411000 curpmap = 0xfffffe014ff00b20 tssp = 0xffffffff82411384 rsp0 = 0xfffffe0141b8b000 kcr3 = 0x80000000569bf3b4 ucr3 = 0x80000005ef0f5bb4 scr3 = 0x5ef0f5bb4 gs32p = 0xffffffff82411404 ldt = 0xffffffff82411444 tss = 0xffffffff82411434 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00bb294600 curthread = 0xfffffe014fe93020: pid 16 tid 100560 critnest 0 "/mnt worker" curpcb = 0xfffffe014fe93530 fpcurthread = none idlethread = 0xfffffe00e4c5f560: tid 100005 "idle: cpu2" self = 0xffffffff82412000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82412384 rsp0 = 0xfffffe0150156000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x64ed24d54 gs32p = 0xffffffff82412404 ldt = 0xffffffff82412444 tss = 0xffffffff82412434 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00bb2a1600 curthread = 0xfffffe00e4c5ee40: pid 11 tid 100006 critnest 1 "idle: cpu3" curpcb = 0xfffffe00e4c5f350 fpcurthread = none idlethread = 0xfffffe00e4c5ee40: tid 100006 "idle: cpu3" self = 0xffffffff82413000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82413384 rsp0 = 0xfffffe00377f6000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x6eb8af1e gs32p = 0xffffffff82413404 ldt = 0xffffffff82413444 tss = 0xffffffff82413434 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00bb2ae600 curthread = 0xfffffe00e4c5e720: pid 11 tid 100007 critnest 1 "idle: cpu4" curpcb = 0xfffffe00e4c5ec30 fpcurthread = none idlethread = 0xfffffe00e4c5e720: tid 100007 "idle: cpu4" self = 0xffffffff82414000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82414384 rsp0 = 0xfffffe00e4831000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x1c876889a gs32p = 0xffffffff82414404 ldt = 0xffffffff82414444 tss = 0xffffffff82414434 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00bb2bb600 curthread = 0xfffffe00e4c5e000: pid 11 tid 100008 critnest 1 "idle: cpu5" curpcb = 0xfffffe00e4c5e510 fpcurthread = none idlethread = 0xfffffe00e4c5e000: tid 100008 "idle: cpu5" self = 0xffffffff82415000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82415384 rsp0 = 0xfffffe0037800000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x6eb8ac9c gs32p = 0xffffffff82415404 ldt = 0xffffffff82415444 tss = 0xffffffff82415434 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00bb2c8600 curthread = 0xfffffe00e4c66c80: pid 11 tid 100009 critnest 1 "idle: cpu6" curpcb = 0xfffffe00e4c67190 fpcurthread = none idlethread = 0xfffffe00e4c66c80: tid 100009 "idle: cpu6" self = 0xffffffff82416000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82416384 rsp0 = 0xfffffe00e483b000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x802eeaa6d gs32p = 0xffffffff82416404 ldt = 0xffffffff82416444 tss = 0xffffffff82416434 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00bb2d5600 curthread = 0xfffffe00e4c66560: pid 11 tid 100010 critnest 1 "idle: cpu7" curpcb = 0xfffffe00e4c66a70 fpcurthread = none idlethread = 0xfffffe00e4c66560: tid 100010 "idle: cpu7" self = 0xffffffff82417000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82417384 rsp0 = 0xfffffe00e4836000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x64e7bb9c9 gs32p = 0xffffffff82417404 ldt = 0xffffffff82417444 tss = 0xffffffff82417434 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00bb2e2600 curthread = 0xfffffe00e4c65e40: pid 11 tid 100011 critnest 1 "idle: cpu8" curpcb = 0xfffffe00e4c66350 fpcurthread = none idlethread = 0xfffffe00e4c65e40: tid 100011 "idle: cpu8" self = 0xffffffff82418000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff82418384 rsp0 = 0xfffffe00e4845000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x6721e0b4a gs32p = 0xffffffff82418404 ldt = 0xffffffff82418444 tss = 0xffffffff82418434 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00bb2ef600 curthread = 0xfffffe01437a0020: pid 81128 tid 100370 critnest 1 "mkdir" curpcb = 0xfffffe01437a0530 fpcurthread = 0xfffffe01437a0020: pid 81128 "mkdir" idlethread = 0xfffffe00e4c65720: tid 100012 "idle: cpu9" self = 0xffffffff82419000 curpmap = 0xfffffe014feefb20 tssp = 0xffffffff82419384 rsp0 = 0xfffffe0143939000 kcr3 = 0x800000005694b2f4 ucr3 = 0x800000006e8fbaf4 scr3 = 0x67259fc9a gs32p = 0xffffffff82419404 ldt = 0xffffffff82419444 tss = 0xffffffff82419434 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00bb2fc600 curthread = 0xfffffe00e4c65000: pid 11 tid 100013 critnest 1 "idle: cpu10" curpcb = 0xfffffe00e4c65510 fpcurthread = none idlethread = 0xfffffe00e4c65000: tid 100013 "idle: cpu10" self = 0xffffffff8241a000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff8241a384 rsp0 = 0xfffffe00e484f000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x3882ddf92 gs32p = 0xffffffff8241a404 ldt = 0xffffffff8241a444 tss = 0xffffffff8241a434 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00bb309600 curthread = 0xfffffe00e4c62740: pid 11 tid 100014 critnest 1 "idle: cpu11" curpcb = 0xfffffe00e4c62c50 fpcurthread = none idlethread = 0xfffffe00e4c62740: tid 100014 "idle: cpu11" self = 0xffffffff8241b000 curpmap = 0xffffffff81e8c2d8 tssp = 0xffffffff8241b384 rsp0 = 0xfffffe00e484a000 kcr3 = 0x8000000002367002 ucr3 = 0xffffffffffffffff scr3 = 0x388610d6f gs32p = 0xffffffff8241b404 ldt = 0xffffffff8241b444 tss = 0xffffffff8241b434 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 81128 (mkdir) thread 0xfffffe01437a0020 (100370) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe003b5ad000) locked @ kern/vfs_bio.c:1721 exclusive lockmgr ufs (ufs) r = 0 (0xfffffe015022a9d0) locked @ ufs/ffs/ffs_snapshot.c:2748 exclusive lockmgr ufs (ufs) r = 0 (0xfffffe01522ea7e0) locked @ kern/vfs_subr.c:2993 exclusive lockmgr ufs (ufs) r = 0 (0xfffffe015236c0d8) locked @ kern/vfs_lookup.c:900 Process 16 (bufdaemon) thread 0xfffffe014fe93020 (100560) exclusive rw vm object (vm object) r = 0 (0xfffff8073e695e70) locked @ kern/vfs_bio.c:3029 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe003afc5ac0) locked @ kern/vfs_bio.c:1721 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe003ae95210) locked @ kern/vfs_bio.c:1721 db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffffe015022a960: type VREG usecount 0, writecount 0, refcount 5 seqc users 0 hold count flags () flags (VIRF_PGREAD|VV_SYSTEM|VMP_LAZYLIST) v_object 0xfffff80599f56108 ref 0 pages 24 cleanbuf 1 dirtybuf 2 lock type ufs: EXCL by thread 0xfffffe01437a0020 (pid 81128, mkdir, tid 100370) with exclusive waiters pending #0 0xffffffff80bd49cc at lockmgr_xlock_hard+0x4ac #1 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #2 0xffffffff80f1e7e5 at _lockmgr_args+0x85 #3 0xffffffff80f1fe26 at revert_snaplock+0x76 #4 0xffffffff80f1f66e at ffs_snapremove+0x18e #5 0xffffffff80f17702 at ffs_truncate+0x8f2 #6 0xffffffff80f533ea at ufs_inactive+0x1aa #7 0xffffffff811ad2e9 at VOP_INACTIVE_APV+0x59 #8 0xffffffff80cfb0ca at vinactivef+0xda #9 0xffffffff80cfa71a at vput_final+0x2aa #10 0xffffffff80f1eff3 at ffs_snap_relock+0xc3 #11 0xffffffff80f14e9d at ffs_balloc_ufs2+0x70d #12 0xffffffff80f2032f at ffs_snapblkfree+0x23f #13 0xffffffff80f0daa2 at ffs_blkfree+0x62 #14 0xffffffff80f3dcf4 at freework_freeblock+0x714 #15 0xffffffff80f2fc7b at handle_workitem_freeblocks+0x1eb #16 0xffffffff80f30ccc at softdep_setup_freeblocks+0x97c #17 0xffffffff80f17b9e at ffs_truncate+0xd8e nlink=1, effnlink=0, size=0, extsize 0 generation=701c32ec, uid=0, gid=5, flags=0x0 ino 90, on dev md10a vnode 0xfffffe015236c068: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff805134a4948 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe01437a0020 (pid 81128, mkdir, tid 100370) #0 0xffffffff80bd3ecf at lockmgr_lock_flags+0x1cf #1 0xffffffff80f4826b at ffs_lock+0x14b #2 0xffffffff811ad75c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d12674 at _vn_lock+0x54 #4 0xffffffff80cea625 at lookup+0xc5 #5 0xffffffff80ce98b8 at namei+0x3f8 #6 0xffffffff80d0e9d4 at kern_frmdirat+0x154 #7 0xffffffff810e1455 at amd64_syscall+0x145 #8 0xffffffff810b246b at fast_syscall_common+0xf8 nlink=3, effnlink=2, size=512, extsize 0 generation=afcb96ec, uid=0, gid=0, flags=0x0 ino 197195, on dev md10a vnode 0xfffffe01522ea770: type VDIR usecount 0, writecount 0, refcount 1 seqc users 0 mountedhere 0 hold count flags () flags (VI_DOINGINACT|VMP_LAZYLIST) v_object 0xfffff80599eb2b58 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0xfffffe01437a0020 (pid 81128, mkdir, tid 100370) #0 0xffffffff80bd3ecf at lockmgr_lock_flags+0x1cf #1 0xffffffff80f4826b at ffs_lock+0x14b #2 0xffffffff811ad75c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d12674 at _vn_lock+0x54 #4 0xffffffff80cf9e5a at vget_finish+0x7a #5 0xffffffff80ce7f1d at vfs_hash_get+0xcd #6 0xffffffff80f42f88 at ffs_vgetf+0x58 #7 0xffffffff80f5442d at ufs_lookup_ino+0xd6d #8 0xffffffff811aa14a at VOP_CACHEDLOOKUP_APV+0x5a #9 0xffffffff80cd95e6 at vfs_cache_lookup+0xa6 #10 0xffffffff811a9f8a at VOP_LOOKUP_APV+0x5a #11 0xffffffff80ceaa21 at lookup+0x4c1 #12 0xffffffff80ce98b8 at namei+0x3f8 #13 0xffffffff80d0e9d4 at kern_frmdirat+0x154 #14 0xffffffff810e1455 at amd64_syscall+0x145 #15 0xffffffff810b246b at fast_syscall_common+0xf8 nlink=1, effnlink=0, size=0, extsize 0 generation=2b809354, uid=0, gid=0, flags=0x0 ino 197197, on dev md10a db:1:pho1> show mount 0xfffffe010e3e1700 /dev/gpt/work on / (ufs) 0xfffffe010e3e2280 devfs on /dev (devfs) 0xfffffe0143565280 /dev/gpt/tmp on /tmp (ufs) 0xfffffe014289b280 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 79479 Buf domain 0 freebufs 9902 lofreebufs 219 hifreebufs 329 bufspace 169717760 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 48 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 4998 (163467264) dirtyq count 48 (253952) wakeup 0 lim 25 CPU 23, 18, 21, 19, 8, 23, 15, 11, 14, 19, 8, 6, Locked buffers: 0 space 0 Total buffers: 18227 space 169717760 Buf domain 1 freebufs 14352 lofreebufs 219 hifreebufs 329 bufspace 23957504 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 51 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 574 (18264064) dirtyq count 51 (581632) wakeup 0 lim 25 CPU 16, 11, 16, 16, 0, 18, 23, 3, 14, 5, 23, 11, Locked buffers: 0 space 0 Total buffers: 14424 space 23957504 Buf domain 2 freebufs 7723 lofreebufs 219 hifreebufs 329 bufspace 241463296 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 77 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 7137 (233865216) dirtyq count 77 (1175552) wakeup 0 lim 25 CPU 24, 22, 18, 13, 23, 9, 23, 12, 3, 21, 14, 14, Locked buffers: 0 space 0 Total buffers: 17681 space 241463296 Buf domain 3 freebufs 15089 lofreebufs 219 hifreebufs 329 bufspace 180224 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 41 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 41 (167936) wakeup 0 lim 25 CPU 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 10787 space 180224 Buf domain 4 freebufs 15052 lofreebufs 219 hifreebufs 329 bufspace 1519616 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 67 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 5 (77824) dirtyq count 67 (1146880) wakeup 0 lim 25 CPU 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, Locked buffers: 6 space 196608 Total buffers: 10627 space 1519616 Buf domain 5 freebufs 15088 lofreebufs 219 hifreebufs 329 bufspace 299008 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 41 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 3 (40960) dirtyq count 41 (225280) wakeup 0 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 1 space 32768 Total buffers: 11104 space 299008 Buf domain 6 freebufs 12461 lofreebufs 219 hifreebufs 329 bufspace 86061056 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 51 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 2509 (81952768) dirtyq count 51 (438272) wakeup 0 lim 25 CPU 12, 1, 4, 13, 7, 1, 6, 7, 15, 14, 22, 10, Locked buffers: 0 space 0 Total buffers: 9077 space 86061056 db:1:pho1> show lockedbufs buf at 0xfffffe003a508790 b_flags = 0x20000000, b_xflags=0x2 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe004de10000 , b_blkno = 1133952, b_lblkno = 1133952, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe004de10000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff8073e695e70, 0x229b0, 0x67ae3f000),(0xfffff8073e695e70, 0x229b1, 0x1f6b99000),(0xfffff8073e695e70, 0x229b2, 0x223119000),(0xfffff8073e695e70, 0x229b3, 0x223118000),(0xfffff8073e695e70, 0x229b4, 0x6d54e5000),(0xfffff8073e695e70, 0x229b5, 0x6d54e4000),(0xfffff8073e695e70, 0x229b6, 0x53c639000),(0xfffff8073e695e70, 0x229b7, 0x53c638000) lock type bufwait: EXCL by KERNPROC #0 0xffffffff80bd6070 at _lockmgr_disown+0xd0 #1 0xffffffff80f3cf61 at indirdep_lookup+0xa1 #2 0xffffffff80f2cd70 at setup_allocindir_phase2+0x80 #3 0xffffffff80f2cafc at softdep_setup_allocindir_page+0xfc #4 0xffffffff80f163ae at ffs_balloc_ufs2+0x1c1e #5 0xffffffff80f478c2 at ffs_write+0x402 #6 0xffffffff811abb3f at VOP_WRITE_APV+0xcf #7 0xffffffff80d176fc at vn_write+0x2fc #8 0xffffffff80d17003 at vn_io_fault_doio+0x43 #9 0xffffffff80d13023 at vn_io_fault1+0x163 #10 0xffffffff80d1099a at vn_io_fault+0x1aa #11 0xffffffff80c7c301 at dofilewrite+0x81 #12 0xffffffff80c7be30 at sys_write+0xc0 #13 0xffffffff810e1455 at amd64_syscall+0x145 #14 0xffffffff810b246b at fast_syscall_common+0xf8 b_io_tracking: b_io_tcnt = 5266 1: getblkx 2: brelse 3: bqrelse 4: bufdone 5: cluster_wbuild 6: bqrelse 7: bdwrite 8: getblkx 9: bqrelse 10: bdwrite 11: getblkx 12: bqrelse 13: bdwrite 14: getblkx 15: bqrelse 16: bdwrite 17: getblkx 18: bqrelse 19: bdwrite 20: getblkx 21: bqrelse 22: bdwrite 23: getblkx 24: bqrelse 25: bdwrite 26: getblkx 27: bqrelse 28: bdwrite 29: getblkx 30: brelse 31: bqrelse 32: bufdone buf at 0xfffffe003ae95190 b_flags = 0x20000220, b_xflags=0x6 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe003d480000 , b_blkno = 38208, b_lblkno = 38208, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe003d480000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff8073e695e70, 0x12a8, 0x3b3c9000),(0xfffff8073e695e70, 0x12a9, 0x3b3c8000),(0xfffff8073e695e70, 0x12aa, 0x1f5dc7000),(0xfffff8073e695e70, 0x12ab, 0x1f5dc6000),(0xfffff8073e695e70, 0x12ac, 0x1f5dc5000),(0xfffff8073e695e70, 0x12ad, 0x1f5dc4000),(0xfffff8073e695e70, 0x12ae, 0x1f5dc3000),(0xfffff8073e695e70, 0x12af, 0x1f5dc2000) lock type bufwait: EXCL by thread 0xfffffe014fe93020 (pid 16, bufdaemon, tid 100560) #0 0xffffffff80bd49cc at lockmgr_xlock_hard+0x4ac #1 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #2 0xffffffff80ccb225 at _lockmgr_args_rw+0x85 #3 0xffffffff80cd2fd4 at buf_alloc+0xe4 #4 0xffffffff80ccd1e8 at getnewbuf+0x148 #5 0xffffffff80cc9f23 at getblkx+0x683 #6 0xffffffff80cc9628 at breadn_flags+0x68 #7 0xffffffff80f4225a at ffs_breadz+0x5a #8 0xffffffff80f3e5c6 at indir_trunc+0x766 #9 0xffffffff80f2fcd0 at handle_workitem_freeblocks+0x240 #10 0xffffffff80f28ac3 at process_worklist_item+0x2a3 #11 0xffffffff80f23687 at softdep_process_worklist+0xb7 #12 0xffffffff80f272af at softdep_flush+0x10f #13 0xffffffff80bbd110 at fork_exit+0x80 #14 0xffffffff810b2bce at fork_trampoline+0xe b_io_tracking: b_io_tcnt = 1084 1: bufdone 2: biodone 3: g_io_deliver 4: biodone 5: g_io_deliver 6: g_io_check 7: g_io_request 8: g_part_start 9: g_io_check 10: g_io_request 11: g_vfs_strategy 12: getblkx 13: brelse 14: bqrelse 15: bufdone 16: cluster_wbuild 17: brelse 18: getblkx 19: bqrelse 20: bdwrite 21: getblkx 22: bqrelse 23: bdwrite 24: getblkx 25: bqrelse 26: bdwrite 27: getblkx 28: bqrelse 29: bdwrite 30: getblkx 31: bqrelse 32: bdwrite buf at 0xfffffe003afc5a40 b_flags = 0x2000a200, b_xflags=0x6 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe004bb98000 , b_blkno = 39040, b_lblkno = 39040, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe004bb98000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): ( ??? ),( ??? ),( ??? ),( ??? ),( ??? ),( ??? ),(0xfffff8073e695e70, 0x1316, 0x1ca27a000),(0xfffff8073e695e70, 0x1317, 0x1ca27b000) lock type bufwait: EXCL by thread 0xfffffe014fe93020 (pid 16, bufdaemon, tid 100560) #0 0xffffffff80bd49cc at lockmgr_xlock_hard+0x4ac #1 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #2 0xffffffff80ccb225 at _lockmgr_args_rw+0x85 #3 0xffffffff80cd2fd4 at buf_alloc+0xe4 #4 0xffffffff80ccd1e8 at getnewbuf+0x148 #5 0xffffffff80cc9f23 at getblkx+0x683 #6 0xffffffff80cc9628 at breadn_flags+0x68 #7 0xffffffff80f4225a at ffs_breadz+0x5a #8 0xffffffff80f3e5c6 at indir_trunc+0x766 #9 0xffffffff80f3e124 at indir_trunc+0x2c4 #10 0xffffffff80f2fcd0 at handle_workitem_freeblocks+0x240 #11 0xffffffff80f28ac3 at process_worklist_item+0x2a3 #12 0xffffffff80f23687 at softdep_process_worklist+0xb7 #13 0xffffffff80f272af at softdep_flush+0x10f #14 0xffffffff80bbd110 at fork_exit+0x80 #15 0xffffffff810b2bce at fork_trampoline+0xe b_io_tracking: b_io_tcnt = 9245 1: bufdone 2: biodone 3: g_io_deliver 4: biodone 5: g_io_deliver 6: g_io_check 7: g_io_request 8: g_part_start 9: g_io_check 10: g_io_request 11: g_vfs_strategy 12: getblkx 13: brelse 14: bufdone 15: biodone 16: g_io_deliver 17: biodone 18: g_io_deliver 19: g_io_check 20: g_io_request 21: g_part_start 22: g_io_check 23: g_io_request 24: g_vfs_strategy 25: getblkx 26: brelse 27: bufdone 28: biodone 29: g_io_deliver 30: biodone 31: g_io_deliver 32: g_io_check buf at 0xfffffe003b259090 b_flags = 0x20000000, b_xflags=0x2 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe003d6e0000 , b_blkno = 36480, b_lblkno = 36480, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe003d6e0000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff8073e695e70, 0x11d0, 0x175403000),(0xfffff8073e695e70, 0x11d1, 0x175402000),(0xfffff8073e695e70, 0x11d2, 0x175401000),(0xfffff8073e695e70, 0x11d3, 0x175400000),(0xfffff8073e695e70, 0x11d4, 0x843b7f000),(0xfffff8073e695e70, 0x11d5, 0x843b7e000),(0xfffff8073e695e70, 0x11d6, 0x843b7d000),(0xfffff8073e695e70, 0x11d7, 0x843b7c000) lock type bufwait: EXCL by KERNPROC #0 0xffffffff80bd6070 at _lockmgr_disown+0xd0 #1 0xffffffff80f3cf61 at indirdep_lookup+0xa1 #2 0xffffffff80f2cd70 at setup_allocindir_phase2+0x80 #3 0xffffffff80f2d070 at softdep_setup_allocindir_meta+0xb0 #4 0xffffffff80f15162 at ffs_balloc_ufs2+0x9d2 #5 0xffffffff80f20877 at ffs_snapblkfree+0x787 #6 0xffffffff80f0daa2 at ffs_blkfree+0x62 #7 0xffffffff80f3dcf4 at freework_freeblock+0x714 #8 0xffffffff80f2fc7b at handle_workitem_freeblocks+0x1eb #9 0xffffffff80f30ccc at softdep_setup_freeblocks+0x97c #10 0xffffffff80f17b9e at ffs_truncate+0xd8e #11 0xffffffff80f533ea at ufs_inactive+0x1aa #12 0xffffffff811ad2e9 at VOP_INACTIVE_APV+0x59 #13 0xffffffff80cfb0ca at vinactivef+0xda #14 0xffffffff80cfa71a at vput_final+0x2aa #15 0xffffffff80d0eb50 at kern_frmdirat+0x2d0 #16 0xffffffff810e1455 at amd64_syscall+0x145 #17 0xffffffff810b246b at fast_syscall_common+0xf8 b_io_tracking: b_io_tcnt = 775 1: getblkx 2: brelse 3: bqrelse 4: bufdone 5: biodone 6: g_io_deliver 7: biodone 8: g_io_deliver 9: g_io_check 10: g_io_request 11: g_part_start 12: g_io_check 13: g_io_request 14: g_vfs_strategy 15: bufwrite 16: bufdone 17: biodone 18: g_io_deliver 19: biodone 20: g_io_deliver 21: g_io_check 22: g_io_request 23: g_part_start 24: g_io_check 25: g_io_request 26: g_vfs_strategy 27: getblkx 28: brelse 29: bqrelse 30: bufdone 31: cluster_wbuild 32: bqrelse buf at 0xfffffe003b302630 b_flags = 0x20000000, b_xflags=0x2 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe00555a8000 , b_blkno = 36544, b_lblkno = 36544, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe00555a8000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff8073e695e70, 0x11d8, 0x7922ec000),(0xfffff8073e695e70, 0x11d9, 0x7922ee000),(0xfffff8073e695e70, 0x11da, 0x7922ed000),(0xfffff8073e695e70, 0x11db, 0x5ae5c4000),(0xfffff8073e695e70, 0x11dc, 0x7c1a25000),(0xfffff8073e695e70, 0x11dd, 0x5a0e3d000),(0xfffff8073e695e70, 0x11de, 0x5a0e3c000),(0xfffff8073e695e70, 0x11df, 0x662ade000) lock type bufwait: EXCL by KERNPROC #0 0xffffffff80bd6070 at _lockmgr_disown+0xd0 #1 0xffffffff80f3cf61 at indirdep_lookup+0xa1 #2 0xffffffff80f2cd70 at setup_allocindir_phase2+0x80 #3 0xffffffff80f2cafc at softdep_setup_allocindir_page+0xfc #4 0xffffffff80f163ae at ffs_balloc_ufs2+0x1c1e #5 0xffffffff80f20877 at ffs_snapblkfree+0x787 #6 0xffffffff80f0daa2 at ffs_blkfree+0x62 #7 0xffffffff80f3dcf4 at freework_freeblock+0x714 #8 0xffffffff80f2fc7b at handle_workitem_freeblocks+0x1eb #9 0xffffffff80f30ccc at softdep_setup_freeblocks+0x97c #10 0xffffffff80f17b9e at ffs_truncate+0xd8e #11 0xffffffff80f533ea at ufs_inactive+0x1aa #12 0xffffffff811ad2e9 at VOP_INACTIVE_APV+0x59 #13 0xffffffff80cfb0ca at vinactivef+0xda #14 0xffffffff80cfa71a at vput_final+0x2aa #15 0xffffffff80d0eb50 at kern_frmdirat+0x2d0 #16 0xffffffff810e1455 at amd64_syscall+0x145 #17 0xffffffff810b246b at fast_syscall_common+0xf8 b_io_tracking: b_io_tcnt = 371 1: getblkx 2: brelse 3: bqrelse 4: bufdone 5: biodone 6: g_io_deliver 7: biodone 8: g_io_deliver 9: g_io_check 10: g_io_request 11: g_part_start 12: g_io_check 13: g_io_request 14: g_vfs_strategy 15: bufwrite 16: bufdone 17: biodone 18: g_io_deliver 19: biodone 20: g_io_deliver 21: g_io_check 22: g_io_request 23: g_part_start 24: g_io_check 25: g_io_request 26: g_vfs_strategy 27: getblkx 28: brelse 29: bqrelse 30: getblkx 31: bqrelse 32: getblkx buf at 0xfffffe003b41c5d0 b_flags = 0x20000000, b_xflags=0x2 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe01504d7d98), b_data = 0xfffffe004d448000 , b_blkno = 1128704, b_lblkno = 1128704, b_vp = 0xfffffe01504d7c20, b_dep = 0 b_kvabase = 0xfffffe004d448000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff8073e695e70, 0x22720, 0x15caab000),(0xfffff8073e695e70, 0x22721, 0x36845c000),(0xfffff8073e695e70, 0x22722, 0x1cd451000),(0xfffff8073e695e70, 0x22723, 0x130931000),(0xfffff8073e695e70, 0x22724, 0x1cd458000),(0xfffff8073e695e70, 0x22725, 0x2294d8000),(0xfffff8073e695e70, 0x22726, 0x2294d1000),(0xfffff8073e695e70, 0x22727, 0x302737000) lock type bufwait: EXCL by KERNPROC #0 0xffffffff80bd6070 at _lockmgr_disown+0xd0 #1 0xffffffff80f3cf61 at indirdep_lookup+0xa1 #2 0xffffffff80f2cd70 at setup_allocindir_phase2+0x80 #3 0xffffffff80f2cafc at softdep_setup_allocindir_page+0xfc #4 0xffffffff80f163ae at ffs_balloc_ufs2+0x1c1e #5 0xffffffff80f478c2 at ffs_write+0x402 #6 0xffffffff811abb3f at VOP_WRITE_APV+0xcf #7 0xffffffff80d176fc at vn_write+0x2fc #8 0xffffffff80d17003 at vn_io_fault_doio+0x43 #9 0xffffffff80d13023 at vn_io_fault1+0x163 #10 0xffffffff80d1099a at vn_io_fault+0x1aa #11 0xffffffff80c7c301 at dofilewrite+0x81 #12 0xffffffff80c7be30 at sys_write+0xc0 #13 0xffffffff810e1455 at amd64_syscall+0x145 #14 0xffffffff810b246b at fast_syscall_common+0xf8 b_io_tracking: b_io_tcnt = 776 1: getblkx 2: brelse 3: bqrelse 4: bufdone 5: cluster_wbuild 6: bqrelse 7: bdwrite 8: getblkx 9: bqrelse 10: bdwrite 11: getblkx 12: bqrelse 13: bdwrite 14: getblkx 15: bqrelse 16: bdwrite 17: getblkx 18: bqrelse 19: bdwrite 20: getblkx 21: bqrelse 22: bdwrite 23: getblkx 24: bqrelse 25: bdwrite 26: getblkx 27: bqrelse 28: bdwrite 29: getblkx 30: brelse 31: bqrelse 32: bufdone buf at 0xfffffe003b5acf80 b_flags = 0x20000200, b_xflags=0x2 b_vflags=0x0 b_ioflags0x0 b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0 b_bufobj = (0xfffffe015022aad8), b_data = 0xfffffe0059310000 , b_blkno = 77824, b_lblkno = 49685, b_vp = 0xfffffe015022a960, b_dep = 0xfffff8004b432900 b_kvabase = 0xfffffe0059310000, b_kvasize = 32768 b_npages = 8, pages(OBJ, IDX, PA): (0xfffff80599f56108, 0x610a8, 0x56dc48000),(0xfffff80599f56108, 0x610a9, 0x56dc49000),(0xfffff80599f56108, 0x610aa, 0x4167b2000),(0xfffff80599f56108, 0x610ab, 0x4167b3000),(0xfffff80599f56108, 0x610ac, 0x3eac60000),(0xfffff80599f56108, 0x610ad, 0x3eac61000),(0xfffff80599f56108, 0x610ae, 0x109fa6000),(0xfffff80599f56108, 0x610af, 0x109fa7000) lock type bufwait: EXCL by thread 0xfffffe01437a0020 (pid 81128, mkdir, tid 100370) #0 0xffffffff80bd49cc at lockmgr_xlock_hard+0x4ac #1 0xffffffff80bd4e99 at __lockmgr_args+0x1f9 #2 0xffffffff80ccb225 at _lockmgr_args_rw+0x85 #3 0xffffffff80cd2fd4 at buf_alloc+0xe4 #4 0xffffffff80ccd1e8 at getnewbuf+0x148 #5 0xffffffff80cc9f23 at getblkx+0x683 #6 0xffffffff80f16734 at ffs_balloc_getblk+0x54 #7 0xffffffff80f16280 at ffs_balloc_ufs2+0x1af0 #8 0xffffffff80f20877 at ffs_snapblkfree+0x787 #9 0xffffffff80f0daa2 at ffs_blkfree+0x62 #10 0xffffffff80f3dcf4 at freework_freeblock+0x714 #11 0xffffffff80f2fc7b at handle_workitem_freeblocks+0x1eb #12 0xffffffff80f30ccc at softdep_setup_freeblocks+0x97c #13 0xffffffff80f17b9e at ffs_truncate+0xd8e #14 0xffffffff80f533ea at ufs_inactive+0x1aa #15 0xffffffff811ad2e9 at VOP_INACTIVE_APV+0x59 #16 0xffffffff80cfb0ca at vinactivef+0xda #17 0xffffffff80cfa71a at vput_final+0x2aa b_io_tracking: b_io_tcnt = 20717 1: bufdone 2: biodone 3: g_io_deliver 4: biodone 5: g_io_deliver 6: g_io_check 7: g_io_request 8: g_part_start 9: g_io_check 10: g_io_request 11: g_vfs_strategy 12: getblkx 13: brelse 14: bqrelse 15: bufdone 16: biodone 17: g_io_deliver 18: biodone 19: g_io_deliver 20: g_io_check 21: g_io_request 22: g_part_start 23: g_io_check 24: g_io_request 25: g_vfs_strategy 26: bufwrite 27: bufdone 28: biodone 29: g_io_deliver 30: biodone 31: g_io_deliver 32: g_io_check db:1:pho1> show page vm_cnt.v_free_count: 7333530 vm_cnt.v_inactive_count: 118906 vm_cnt.v_active_count: 319288 vm_cnt.v_laundry_count: 6911 vm_cnt.v_wire_count: 339967 vm_cnt.v_free_reserved: 10644 vm_cnt.v_free_min: 51265 vm_cnt.v_free_target: 173128 vm_cnt.v_inactive_target: 259692 db:1:pho1> show pageq pq_free 7333530 dom 0 page_cnt 8124374 free 7333530 pq_act 319288 pq_inact 118906 pq_laund 6911 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000817 | 000001 11 (008192K) | 000757 | 000002 10 (004096K) | 001065 | 000006 09 (002048K) | 000663 | 000013 08 (001024K) | 001365 | 000015 07 (000512K) | 001610 | 000029 06 (000256K) | 002274 | 000058 05 (000128K) | 000693 | 000136 04 (000064K) | 000554 | 000346 03 (000032K) | 003274 | 000747 02 (000016K) | 006761 | 002035 01 (000008K) | 000000 | 006470 00 (000004K) | 000000 | 008969 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000001 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000001 | 000000 05 (000128K) | 000001 | 000000 04 (000064K) | 000002 | 000000 03 (000032K) | 000001 | 000000 02 (000016K) | 000000 | 000000 01 (000008K) | 000000 | 000000 00 (000004K) | 000000 | 000000 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Total Mem XFree malloc-4096 4096 34133 3 34265 0 2 139821056 0 FFS inode 1160 100094 14167 13898396 0 45 132542760 0 VNODE 600 100223 14203 14772242 0 54 68655600 0 mbuf_cluster 2048 16098 978 16983 0 172 34971648 0 FFS2 dinode 256 100094 14311 13898393 0 65 29287680 0 RADIX NODE 144 49220 106205 158962437 0 119 22381200 0 BUF TRIE 144 1142 104810 16608111 0 64 15257088 0 S VFS Cache 104 100433 3775 4964282 0 126 10837632 0 malloc-256 256 2019 35616 21107037 0 63 9634560 0 VM OBJECT 264 13571 18724 122310626 0 39 8525880 0 malloc-384 384 2664 14876 18223925 0 42 6735360 0 UMA Slabs 0 112 47552 1258 59022 0 126 5466720 0 mbuf 256 16444 4391 12194131 0 254 5333760 0 malloc-128 128 26580 2808 45751209 0 126 3761664 0 malloc-256 256 1687 11153 12099500 0 66 3287040 0 mbuf_jumbo_page 4096 0 762 6 0 172 3121152 0 vmem btag 56 44813 7591 76354 0 254 2934624 0 socket 960 57 2991 7159 0 172 2926080 0 malloc-128 128 1401 18377 15251276 0 126 2531584 0 malloc-65536 65536 1 34 21844 0 1 2293760 0 pbuf 2776 0 760 0 0 0 2109760 0 malloc-128 128 14320 1304 1136990 0 126 1999872 0 tcpcb 1096 12 1766 17 0 172 1948688 0 malloc-4096 4096 421 51 5766936 0 2 1933312 0 256 Bucket 2048 814 122 1425553 0 8 1916928 0 malloc-128 128 13481 1151 4328831 0 126 1872896 0 malloc-32768 32768 9 43 5304 0 1 1703936 0 malloc-256 256 1185 5205 7169809 0 62 1635840 0 g_bio 408 0 3800 64563623 0 37 1550400 0 64 Bucket 512 2099 797 2340785 0 30 1482752 0 malloc-32768 32768 12 33 166590 0 1 1474560 0 128 Bucket 1024 848 563 2422927 0 16 1444864 0 malloc-32768 32768 2 36 825299 0 1 1245184 0 malloc-32768 32768 1 35 21347 0 1 1179648 0 THREAD 1808 492 69 560 0 8 1014288 0 malloc-128 128 722 6997 12109039 0 126 988032 0 malloc-65536 65536 3 12 4208 0 1 983040 0 mbuf_packet 256 6 3804 1640449 0 254 975360 0 MAP ENTRY 96 2693 7051 354923221 0 127 935424 0 malloc-64 64 9210 4776 29332387 0 254 895104 0 malloc-16384 16384 2 52 2257401 0 1 884736 0 VMSPACE 2552 65 262 5260476 0 6 834504 0 unpcb 256 20 3040 6288 0 172 783360 0 malloc-64 64 9296 2926 17756 0 254 782208 0 32 Bucket 256 305 2605 968431 0 62 744960 0 malloc-8192 8192 86 0 86 0 1 704512 0 malloc-256 256 318 2292 22637271 0 62 668160 0 malloc-128 128 524 4529 4784424 0 126 646784 0 malloc-32768 32768 12 4 20 0 1 524288 0 FPU_save_area 832 494 136 31083 0 16 524160 0 malloc-1024 1024 224 256 2548068 0 16 491520 0 UMA Zones 2048 232 3 234 0 8 481280 0 malloc-256 256 511 1349 2882084 0 62 476160 0 malloc-65536 65536 2 5 7 0 1 458752 0 filedesc0 1072 84 336 5281681 0 13 450240 0 malloc-8192 8192 14 39 85863 0 1 434176 0 malloc-128 128 1181 2105 219111 0 126 420608 0 malloc-64 64 4279 2084 4336 0 254 407232 0 DIRHASH 1024 37 359 267406 0 16 405504 0 lkpicurr 168 10 2390 10 0 62 403200 0 PROC 1352 83 214 5281680 0 8 401544 0 lkpimm 168 4 2348 4 0 62 395136 0 malloc-16384 16384 12 12 4912 0 1 393216 0 malloc-32 32 8285 3685 838967 0 254 383040 0 malloc-8192 8192 7 38 1536705 0 1 368640 0 malloc-256 256 566 874 140831 0 62 368640 0 16 Bucket 144 521 1887 7387180 0 62 346752 0 malloc-16384 16384 7 13 23 0 1 327680 0 malloc-2048 2048 118 42 138 0 8 327680 0 malloc-512 512 235 397 48042 0 30 323584 0 L VFS Cache 320 798 138 798 0 30 299520 0 ksiginfo 112 281 2383 44123 0 126 298368 0 malloc-8192 8192 23 13 47 0 1 294912 0 malloc-16 16 16111 2139 58894 0 254 292000 0 malloc-256 256 62 1078 718073 0 62 291840 0 NAMEI 1024 1 271 107098374 0 16 278528 0 malloc-64 64 523 3824 91113702 0 254 278208 0 malloc-65536 65536 2 2 5 0 1 262144 0 malloc-1024 1024 2 254 12810108 0 16 262144 0 malloc-128 128 35 2011 1018905 0 126 261888 0 8 Bucket 80 468 2782 15407874 0 126 260000 0 pipe 744 8 332 1019617 0 16 252960 0 malloc-64 64 6 3837 788831 0 254 245952 0 malloc-2048 2048 1 119 41815 0 8 245760 0 malloc-1024 1024 2 238 166547 0 16 245760 0 malloc-256 256 131 799 42542 0 62 238080 0 malloc-512 512 3 453 789544 0 30 233472 0 malloc-64 64 169 3422 10662650 0 254 229824 0 malloc-2048 2048 6 106 1802 0 8 229376 0 malloc-512 512 0 424 524733 0 30 217088 0 malloc-512 512 12 412 42391 0 30 217088 0 Files 80 148 2502 65987794 0 126 212000 0 Mountpoints 2944 4 68 831707 0 4 211968 0 malloc-64 64 62 3025 518 0 254 197568 0 4 Bucket 48 30 4086 243164 0 254 197568 0 malloc-65536 65536 3 0 3 0 1 196608 0 malloc-8192 8192 17 7 3067 0 1 196608 0 malloc-2048 2048 2 94 5683 0 8 196608 0 malloc-1024 1024 32 160 4689 0 16 196608 0 malloc-1024 1024 23 169 63 0 16 196608 0 malloc-384 384 323 187 10687 0 30 195840 0 malloc-1024 1024 21 155 356 0 16 180224 0 udp_inpcb 424 22 401 785 0 30 179352 0 ttyoutq 256 181 509 290 0 62 176640 0 malloc-384 384 152 298 157 0 30 172800 0 malloc-4096 4096 4 38 811232 0 2 172032 0 malloc-2048 2048 2 78 363 0 8 163840 0 fakepg 104 0 1521 312 0 126 158184 0 cpuset 104 27 1492 27 0 126 157976 0 ada_ccb 272 0 574 11641853 0 30 156128 0 malloc-4096 4096 1 37 42437 0 2 155648 0 malloc-384 384 3 387 3868 0 30 149760 0 PWD 32 40 4622 7651807 0 254 149184 0 malloc-64 64 23 2308 48 0 254 149184 0 malloc-32 32 367 4043 3484336 0 254 141120 0 2 Bucket 32 592 3818 1613264 0 254 141120 0 malloc-4096 4096 16 18 43 0 2 139264 0 SLEEPQUEUE 88 562 974 562 0 126 135168 0 ttyinq 160 345 480 555 0 62 132000 0 malloc-65536 65536 2 0 2 0 1 131072 0 malloc-8192 8192 4 12 26 0 1 131072 0 TURNSTILE 136 562 383 562 0 62 128520 0 PGRP 88 35 1391 228 0 126 125488 0 rl_entry 40 273 2858 273 0 254 125240 0 malloc-32 32 177 3477 950212 0 254 116928 0 malloc-32 32 519 3135 251480 0 254 116928 0 malloc-32 32 564 3090 25210 0 254 116928 0 malloc-384 384 53 247 55 0 30 115200 0 malloc-2048 2048 5 51 1511 0 8 114688 0 ertt_txseginfo 40 0 2828 766 0 254 113120 0 DEVCTL 1024 0 108 1725547 0 0 110592 0 malloc-512 512 5 211 1259 0 30 110592 0 malloc-4096 4096 1 25 818479 0 2 106496 0 udpcb 32 22 3128 785 0 254 100800 0 malloc-32 32 5 3145 103863 0 254 100800 0 malloc-32 32 67 3083 129 0 254 100800 0 clpbuf 2776 0 36 274023 0 2 99936 0 malloc-1024 1024 25 71 44 0 16 98304 0 routing nhops 256 9 366 15 0 62 96000 0 kenv 258 18 342 21842 0 30 92880 0 malloc-32 32 65 2833 165 0 254 92736 0 tcp_inpcb 424 12 204 17 0 30 91584 0 syncache 168 0 528 2 0 172 88704 0 malloc-16 16 428 5072 10219766 0 254 88000 0 malloc-16384 16384 2 3 14 0 1 81920 0 malloc-16384 16384 5 0 5 0 1 81920 0 malloc-1024 1024 9 71 13 0 16 81920 0 malloc-384 384 11 199 21 0 30 80640 0 UMA Kegs 384 202 1 204 0 30 77952 0 malloc-16 16 235 4265 15430721 0 254 72000 0 rtentry 176 12 379 15 0 62 68816 0 malloc-32768 32768 2 0 2 0 1 65536 0 malloc-4096 4096 14 2 25 0 2 65536 0 malloc-2048 2048 6 26 6 0 8 65536 0 pcpu-64 64 453 571 453 0 254 65536 0 ertt 72 12 884 17 0 126 64512 0 malloc-16 16 20 3980 915679 0 254 64000 0 malloc-16 16 468 3032 1094070 0 254 56000 0 udp_inpcb ports 32 10 1628 18 0 254 52416 0 malloc-8192 8192 4 2 6 0 1 49152 0 pcpu-16 16 72 2744 72 0 254 45056 0 pcpu-8 8 3526 2106 3541 0 254 45056 0 mdpbuf 2776 0 16 886112 0 1 44416 0 malloc-16 16 255 2245 257 0 254 40000 0 ripcb 424 1 89 5 0 30 38160 0 tcp_inpcb ports 32 6 1128 6 0 254 36288 0 malloc-16 16 25 2225 25 0 254 36000 0 malloc-512 512 1 63 2 0 30 32768 0 malloc-512 512 0 64 2 0 30 32768 0 malloc-16 16 58 1942 74 0 254 32000 0 SMR CPU 32 7 760 7 0 254 24544 0 malloc-384 384 3 57 16 0 30 23040 0 SMR SHARED 24 7 760 7 0 254 18408 0 epoch_record pcpu 256 4 60 4 0 62 16384 0 malloc-4096 4096 1 3 13 0 2 16384 0 malloc-2048 2048 1 7 1 0 8 16384 0 malloc-512 512 1 31 1 0 30 16384 0 vmem 1856 1 7 1 0 8 14848 0 vtnet_tx_hdr 24 0 167 0 0 254 4008 0 UMA Slabs 1 176 10 12 10 0 62 3872 0 KMAP ENTRY 96 33 6 33 0 0 3744 0 FFS1 dinode 128 0 0 0 0 126 0 0 swblk 136 0 0 0 0 62 0 0 swpctrie 144 0 0 0 0 62 0 0 da_ccb 544 0 0 0 0 16 0 0 tfo_ccache_entries 80 0 0 0 0 126 0 0 tfo 4 0 0 0 0 254 0 0 sackhole 32 0 0 0 0 254 0 0 hostcache 64 0 0 0 0 254 0 0 tcptw 72 0 0 0 0 172 0 0 ipq 56 0 0 0 0 254 0 0 tcp_log_node 120 0 0 0 0 126 0 0 tcp_log_bucket 176 0 0 0 0 62 0 0 tcp_log 416 0 0 0 0 172 0 0 tcpreass 48 0 0 0 0 254 0 0 ripcb ports 32 0 0 0 0 254 0 0 udplite_inpcb ports 32 0 0 0 0 254 0 0 udplite_inpcb 424 0 0 0 0 30 0 0 IPsec SA lft_c 16 0 0 0 0 254 0 0 KNOTE 160 0 0 0 0 62 0 0 itimer 352 0 0 0 0 30 0 0 AIOLIO 272 0 0 0 0 30 0 0 AIOCB 552 0 0 0 0 16 0 0 AIOP 32 0 0 0 0 254 0 0 AIO 208 0 0 0 0 62 0 0 TMPFS node 224 0 0 0 0 62 0 0 NCLNODE 760 0 0 0 0 16 0 0 LTS VFS Cache 360 0 0 0 0 30 0 0 STS VFS Cache 144 0 0 0 0 62 0 0 cryptop 280 0 0 0 0 30 0 0 linux_dma_object 32 0 0 0 0 254 0 0 linux_dma_pctrie 144 0 0 0 0 62 0 0 IOMMU_MAP_ENTRY 120 0 0 0 0 126 0 0 mbuf_jumbo_16k 16384 0 0 0 0 172 0 0 mbuf_jumbo_9k 9216 0 0 0 0 172 0 0 audit_record 1280 0 0 0 0 8 0 0 domainset 40 0 0 0 0 254 0 0 MAC labels 40 0 0 0 0 254 0 0 vnpbuf 2776 0 0 0 0 4 0 0 nfspbuf 2776 0 0 0 0 2 0 0 swwbuf 2776 0 0 0 0 1 0 0 swrbuf 2776 0 0 0 0 2 0 0 umtx_shm 88 0 0 0 0 126 0 0 umtx pi 96 0 0 0 0 126 0 0 rangeset pctrie nodes 144 0 0 0 0 62 0 0 malloc-65536 65536 0 0 0 0 1 0 0 malloc-65536 65536 0 0 0 0 1 0 0 malloc-32768 32768 0 0 0 0 1 0 0 malloc-32768 32768 0 0 0 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-8192 8192 0 0 0 0 1 0 0 malloc-384 384 0 0 0 0 30 0 0 pcpu-32 32 0 0 0 0 254 0 0 pcpu-4 4 0 0 0 0 254 0 0 UMA Hash 256 0 0 0 0 62 0 0 db:0:pho> show umacache Zone Size Used Free Requests Bucket debugnet_mbuf_packet 2048 0 0 0 0 debugnet_mbuf_cluster 2048 -6144 0 0 0 debugnet_mbuf 256 -10240 0 0 0 vm pgcache 4096 88625 4264 254542469 90 vm pgcache 4096 248318 1291 583660075 127 buf free cache 848 16264 10188 28317242 31 buffer arena-65 65536 0 0 0 1 buffer arena-61 61440 0 0 0 1 buffer arena-57 57344 0 0 0 1 buffer arena-53 53248 0 0 0 1 buffer arena-49 49152 0 0 0 1 buffer arena-45 45056 0 0 0 1 buffer arena-40 40960 0 0 0 1 buffer arena-36 36864 0 0 0 1 buffer arena-32 32768 446 5300 8930281 15 buffer arena-28 28672 0 0 0 1 buffer arena-24 24576 0 0 0 1 buffer arena-20 20480 0 0 0 1 buffer arena-16 16384 5 35 1538119 1 buffer arena-12 12288 0 0 0 1 buffer arena-81 8192 0 0 0 1 buffer arena-40 4096 0 0 0 2 kstack_cache 16384 492 12 560 1 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0xfffffe00e4c603a0 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81fea310 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81fea320 trap() at trap+0x56/frame 0xffffffff81fea430 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81fea430 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377f0dd0, rbp = 0xfffffe00377f0dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377f0dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377f0e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377f0e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377f0e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377f0ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00e4c5ee40 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce3ae00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce3ae10 trap() at trap+0x56/frame 0xfffffe003ce3af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3af20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377f5dd0, rbp = 0xfffffe00377f5dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377f5dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377f5e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377f5e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377f5e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377f5ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00e4c5e720 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce47e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce47e10 trap() at trap+0x56/frame 0xfffffe003ce47f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce47f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4830dd0, rbp = 0xfffffe00e4830dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4830dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4830e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4830e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4830e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4830ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4830f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4830f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffffe00e4c5e000 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce54e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce54e10 trap() at trap+0x56/frame 0xfffffe003ce54f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce54f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377ffdd0, rbp = 0xfffffe00377ffdd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377ffdd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377ffe10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377ffe30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377ffe50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377ffef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377fff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377fff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00e4c66c80 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce61e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce61e10 trap() at trap+0x56/frame 0xfffffe003ce61f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce61f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e483add0, rbp = 0xfffffe00e483add0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e483add0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e483ae10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e483ae30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e483ae50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e483aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e483af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e483af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffffe00e4c66560 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce6ee00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce6ee10 trap() at trap+0x56/frame 0xfffffe003ce6ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce6ef20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4835dd0, rbp = 0xfffffe00e4835dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4835dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4835e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4835e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4835e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4835ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4835f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4835f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00e4c65e40 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce7be00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce7be10 trap() at trap+0x56/frame 0xfffffe003ce7bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce7bf20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4844dd0, rbp = 0xfffffe00e4844dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4844dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4844e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4844e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4844e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4844ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4844f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4844f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffffe00e4c65000 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce95e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce95e10 trap() at trap+0x56/frame 0xfffffe003ce95f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce95f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e484edd0, rbp = 0xfffffe00e484edd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e484edd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e484ee10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e484ee30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e484ee50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e484eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e484ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e484ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00e4c62740 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003cea2e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003cea2e10 trap() at trap+0x56/frame 0xfffffe003cea2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea2f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4849dd0, rbp = 0xfffffe00e4849dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4849dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4849e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4849e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4849e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4849ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4849f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4849f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100560 td 0xfffffe014fe93020 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce2de00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce2de10 trap() at trap+0x56/frame 0xfffffe003ce2df20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce2df20 --- trap 0x13, rip = 0xffffffff80f9117a, rsp = 0xfffffe0150155b10, rbp = 0xfffffe0150155b30 --- vm_page_pqstate_commit() at vm_page_pqstate_commit+0x1a/frame 0xfffffe0150155b30 vm_page_release_locked() at vm_page_release_locked+0xff/frame 0xfffffe0150155b70 vfs_vmio_invalidate() at vfs_vmio_invalidate+0x1bb/frame 0xfffffe0150155bb0 brelse() at brelse+0x50b/frame 0xfffffe0150155bf0 indir_trunc() at indir_trunc+0x558/frame 0xfffffe0150155cc0 indir_trunc() at indir_trunc+0x2c4/frame 0xfffffe0150155d90 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x240/frame 0xfffffe0150155de0 process_worklist_item() at process_worklist_item+0x2a3/frame 0xfffffe0150155e70 softdep_process_worklist() at softdep_process_worklist+0xb7/frame 0xfffffe0150155eb0 softdep_flush() at softdep_flush+0x10f/frame 0xfffffe0150155ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0150155f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0150155f30 --- trap 0xa5a5a5a5, rip = 0xa5a5a5a5a5a5a5a5, rsp = 0xa5a5a5a5a5a5a5a5, rbp = 0 --- Tracing command egrep pid 84874 tid 100216 td 0xfffffe010e3e5e40 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce20e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce20e10 trap() at trap+0x56/frame 0xfffffe003ce20f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce20f20 --- trap 0x13, rip = 0xffffffff80f70d04, rsp = 0xfffffe0141b8abd0, rbp = 0xfffffe0141b8ac00 --- uma_dbg_free() at uma_dbg_free+0x44/frame 0xfffffe0141b8ac00 uma_zfree_arg() at uma_zfree_arg+0x153/frame 0xfffffe0141b8ac50 vm_map_process_deferred() at vm_map_process_deferred+0x41/frame 0xfffffe0141b8ac70 vm_map_remove() at vm_map_remove+0xee/frame 0xfffffe0141b8aca0 pipeclose() at pipeclose+0x2e0/frame 0xfffffe0141b8ace0 pipe_close() at pipe_close+0x60/frame 0xfffffe0141b8ad10 _fdrop() at _fdrop+0x1b/frame 0xfffffe0141b8ad30 closef() at closef+0x1db/frame 0xfffffe0141b8adc0 closefp_impl() at closefp_impl+0x77/frame 0xfffffe0141b8ae00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b8af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b8af30 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x1b13a9eca4fa, rsp = 0x1b13a883adb8, rbp = 0x1b13a883adc0 --- Tracing command mkdir pid 81128 tid 100370 td 0xfffffe01437a0020 (CPU 9) kdb_enter() at kdb_enter+0x37/frame 0xfffffe0143938270 vpanic() at vpanic+0x1b0/frame 0xfffffe01439382c0 panic() at panic+0x43/frame 0xfffffe0143938320 trap_fatal() at trap_fatal+0x385/frame 0xfffffe0143938380 trap_pfault() at trap_pfault+0xab/frame 0xfffffe01439383e0 calltrap() at calltrap+0x8/frame 0xfffffe01439383e0 --- trap 0xc, rip = 0xffffffff80f2023c, rsp = 0xfffffe01439384b0, rbp = 0xfffffe01439385b0 --- ffs_snapblkfree() at ffs_snapblkfree+0x14c/frame 0xfffffe01439385b0 ffs_blkfree() at ffs_blkfree+0x62/frame 0xfffffe0143938610 freework_freeblock() at freework_freeblock+0x714/frame 0xfffffe0143938690 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x1eb/frame 0xfffffe01439386e0 softdep_setup_freeblocks() at softdep_setup_freeblocks+0x97c/frame 0xfffffe0143938750 ffs_truncate() at ffs_truncate+0xd8e/frame 0xfffffe0143938b90 ufs_inactive() at ufs_inactive+0x1aa/frame 0xfffffe0143938bd0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x59/frame 0xfffffe0143938bf0 vinactivef() at vinactivef+0xda/frame 0xfffffe0143938c30 vput_final() at vput_final+0x2aa/frame 0xfffffe0143938c90 kern_frmdirat() at kern_frmdirat+0x2d0/frame 0xfffffe0143938e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143938f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143938f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c72a8, rbp = 0x2e1d642c76e0 --- db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 84874 84544 3222 0 R CPU 1 egrep 84846 81082 81082 0 D+ ufs 0xfffffe015022a9d0 mksnap_ffs 84544 3286 3222 0 S wait 0xfffff804e6c24000 sh 81138 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81137 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81136 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81135 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81134 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81133 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81132 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81131 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81130 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81129 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81128 81123 81082 0 R+ CPU 9 mkdir 81127 81125 81082 0 D+ suspfs 0xfffffe014289b288 rw 81126 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81125 81122 81082 0 S+ wait 0xfffff806d0604548 rw 81124 81123 81082 0 D+ suspfs 0xfffffe014289b288 mkdir 81123 81121 81082 0 S+ wait 0xfffff801c8293548 mkdir 81122 81118 81082 0 S+ nanslp 0xffffffff81eab829 rw 81121 81118 81082 0 S+ nanslp 0xffffffff81eab827 mkdir 81118 81117 81082 0 S+ wait 0xfffff8002d4a6a90 run 81117 81116 81082 0 S+ wait 0xfffff80076c2aa90 run 81116 81102 81082 0 S+ nanslp 0xffffffff81eab828 run 81102 81082 81082 0 S+ wait 0xfffff806d0068a90 sh 81092 0 0 0 DL mdwait 0xfffff802b4cdc000 [md10] 81082 2748 81082 0 S+ wait 0xfffff806d0fd2548 sh 20253 2728 20253 0 S+ ttyin 0xfffff8000e1928b0 csh 22874 22873 22874 2006 Ss+ select 0xfffff804e6ba2540 bash 22873 22870 22870 2006 S select 0xfffff80003da1840 sshd 22870 2701 22870 0 Ss select 0xfffff8002d34dac0 sshd 3289 3222 3222 0 S piperd 0xfffff800385fc2e8 sh 3287 3222 3222 0 S piperd 0xfffff80076a9a2e8 awk 3286 3222 3222 0 S wait 0xfffff80038a07a90 sh 3222 1 3222 0 Ss wait 0xfffff80003c24000 sh 2748 2739 2748 0 S+ wait 0xfffff80076f84a90 bash 2739 2738 2739 0 S+ pause 0xfffff80038a070b0 csh 2738 2733 2738 2006 S+ wait 0xfffff80156d68548 su 2733 2732 2733 2006 Ss+ wait 0xfffff8002d2ee000 bash 2732 2729 2729 2006 S select 0xfffff8002d34ca40 sshd 2729 2701 2729 0 Ss select 0xfffff8000423a0c0 sshd 2728 1 2728 0 Ss+ wait 0xfffff8002d4a2548 login 2727 1 2727 0 Ss+ ttyin 0xfffff8000e192cb0 getty 2726 1 2726 0 Ss+ ttyin 0xfffff8002d3c18b0 getty 2725 1 2725 0 Ss+ ttyin 0xfffff8002d3c1cb0 getty 2724 1 2724 0 Ss+ ttyin 0xfffff8002d3c00b0 getty 2723 1 2723 0 Ss+ ttyin 0xfffff8002d3c04b0 getty 2722 1 2722 0 Ss+ ttyin 0xfffff8002d3c08b0 getty 2721 1 2721 0 Ss+ ttyin 0xfffff8002d3c0cb0 getty 2720 1 2720 0 Ss+ ttyin 0xfffff8002d3bf0b0 getty 2719 1 2719 0 Ss+ ttyin 0xfffff8002d3bf4b0 getty 2705 1 2705 0 Ss nanslp 0xffffffff81eab822 cron 2701 1 2701 0 Ss select 0xfffff80038368dc0 sshd 2682 1 2682 0 Ss select 0xfffff8000496e8c0 inetd 2656 1 2656 123 Ss select 0xfffff80003da0ac0 ntpd 2630 1 2630 0 Ss select 0xfffff8004d4577c0 powerd 2625 1 2625 0 Ss rpcsvc 0xfffff80003e28360 NLM: master 2622 1 2622 0 Ss select 0xfffff80038368cc0 rpc.statd 2597 1 2597 0 Ss select 0xfffff8002d4baec0 rpcbind 2575 1 2575 0 Ss nanslp 0xffffffff81eab820 watchdogd 2567 1 2567 0 Ss select 0xfffff8000423a1c0 syslogd 2378 1 2378 0 Ss select 0xfffff8002d34cac0 devd 2373 1 2373 65 Ss select 0xfffff80003ce04c0 dhclient 2314 1 2314 0 Ss select 0xfffff80033ccea40 dhclient 2311 1 2311 0 Ss select 0xfffff80038368d40 dhclient 18 0 0 0 DL syncer 0xffffffff81f9c630 [syncer] 17 0 0 0 DL vlruwt 0xfffff80038361a90 [vnlru] 16 0 0 0 RL (threaded) [bufdaemon] 100197 D psleep 0xffffffff81f9b660 [bufdaemon] 100198 D - 0xffffffff81a0af40 [bufspacedaemon-0] 100199 D - 0xffffffff81a13200 [bufspacedaemon-1] 100200 D - 0xffffffff81a1b4c0 [bufspacedaemon-2] 100201 D - 0xffffffff81a23780 [bufspacedaemon-3] 100202 D - 0xffffffff81a2ba40 [bufspacedaemon-4] 100203 D - 0xffffffff81a33d00 [bufspacedaemon-5] 100204 D - 0xffffffff81a3bfc0 [bufspacedaemon-6] 100218 D sdflush 0xfffff800048468e8 [/ worker] 100225 D sdflush 0xfffff800044874e8 [/tmp worker] 100560 Run CPU 2 [/mnt worker] 9 0 0 0 DL psleep 0xffffffff81fc1390 [vmdaemon] 8 0 0 0 DL (threaded) [pagedaemon] 100195 D psleep 0xffffffff81fb57f8 [dom0] 100205 D launds 0xffffffff81fb5804 [laundry: dom0] 100206 D umarcl 0xffffffff80f6bf20 [uma] 7 0 0 0 DL idle 0xfffff80004206a90 [enc_daemon1] 6 0 0 0 DL idle 0xfffff80004205000 [enc_daemon0] 5 0 0 0 DL - 0xffffffff81cdf1d8 [rand_harvestq] 15 0 0 0 DL (threaded) [usb] 100128 D - 0xfffffe010a1a2440 [usbus0] 100129 D - 0xfffffe010a1a2498 [usbus0] 100130 D - 0xfffffe010a1a24f0 [usbus0] 100131 D - 0xfffffe010a1a2548 [usbus0] 100132 D - 0xfffffe010a1a25a0 [usbus0] 100134 D - 0xfffffe00e4d76cf0 [usbus1] 100135 D - 0xfffffe00e4d76d48 [usbus1] 100136 D - 0xfffffe00e4d76da0 [usbus1] 100137 D - 0xfffffe00e4d76df8 [usbus1] 100138 D - 0xfffffe00e4d76e50 [usbus1] 100139 D - 0xfffffe010a564cf0 [usbus2] 100140 D - 0xfffffe010a564d48 [usbus2] 100141 D - 0xfffffe010a564da0 [usbus2] 100142 D - 0xfffffe010a564df8 [usbus2] 100143 D - 0xfffffe010a564e50 [usbus2] 4 0 0 0 DL (threaded) [cam] 100121 D - 0xffffffff81bc8fc0 [doneq0] 100122 D - 0xffffffff81bc9040 [doneq1] 100123 D - 0xffffffff81bc90c0 [doneq2] 100124 D - 0xffffffff81bc8f40 [async] 100152 D - 0xffffffff81bc8e10 [scanner] 14 0 0 0 DL seqstat 0xfffff80004408888 [sequencer 00] 3 0 0 0 DL (threaded) [crypto] 100107 D crypto_ 0xffffffff81fb2d20 [crypto] 100108 D crypto_ 0xfffff80004325c30 [crypto returns 0] 100109 D crypto_ 0xfffff80004325c80 [crypto returns 1] 100110 D crypto_ 0xfffff80004325cd0 [crypto returns 2] 100111 D crypto_ 0xfffff80004325d20 [crypto returns 3] 100112 D crypto_ 0xfffff80004325d70 [crypto returns 4] 100113 D crypto_ 0xfffff80004325dc0 [crypto returns 5] 100114 D crypto_ 0xfffff80004325e10 [crypto returns 6] 100115 D crypto_ 0xfffff80004325e60 [crypto returns 7] 100116 D crypto_ 0xfffff80004325eb0 [crypto returns 8] 100117 D crypto_ 0xfffff80004325f00 [crypto returns 9] 100118 D crypto_ 0xfffff80004325f50 [crypto returns 10] 100119 D crypto_ 0xfffff80004325fa0 [crypto returns 11] 13 0 0 0 DL (threaded) [geom] 100092 D - 0xffffffff81e8af90 [g_event] 100093 D - 0xffffffff81e8af98 [g_up] 100094 D - 0xffffffff81e8afa0 [g_down] 2 0 0 0 WL (threaded) [clock] 100078 I [clock (0)] 100079 I [clock (1)] 100080 I [clock (2)] 100081 I [clock (3)] 100082 I [clock (4)] 100083 I [clock (5)] 100084 I [clock (6)] 100085 I [clock (7)] 100086 I [clock (8)] 100087 I [clock (9)] 100088 I [clock (10)] 100089 I [clock (11)] 12 0 0 0 WL (threaded) [intr] 100043 I [swi5: fast taskq] 100048 I [swi6: task queue] 100049 I [swi6: Giant taskq] 100077 I [swi1: netisr 0] 100090 I [swi3: busdma] 100125 I [swi4: apei] 100126 I [irq56: ahci0] 100127 I [irq19: xhci0] 100133 I [irq18: ehci0 ehci1+] 100144 I [irq71: ahci1] 100145 I [swi0: uart uart] 100159 I [irq73: mlx5_core0] 100160 I [irq74: mlx5_core0] 100161 I [irq72: mlx5_core0] 100162 I [irq75: mlx5_core0] 100163 I [irq76: mlx5_core0] 100164 I [irq77: mlx5_core0] 100165 I [irq78: mlx5_core0] 100166 I [irq79: mlx5_core0] 100167 I [irq80: mlx5_core0] 100168 I [irq81: mlx5_core0] 100169 I [irq82: mlx5_core0] 100170 I [irq83: mlx5_core0] 100171 I [irq84: mlx5_core0] 100172 I [irq85: mlx5_core0] 100173 I [irq86: mlx5_core0] 100179 I [irq88: mlx5_core1] 100180 I [irq89: mlx5_core1] 100181 I [irq87: mlx5_core1] 100182 I [irq90: mlx5_core1] 100183 I [irq91: mlx5_core1] 100184 I [irq92: mlx5_core1] 100185 I [irq93: mlx5_core1] 100186 I [irq94: mlx5_core1] 100187 I [irq95: mlx5_core1] 100188 I [irq96: mlx5_core1] 100189 I [irq97: mlx5_core1] 100190 I [irq98: mlx5_core1] 100191 I [irq99: mlx5_core1] 100192 I [irq100: mlx5_core1] 100193 I [irq101: mlx5_core1] 100253 I [irq102: ioat0] 100254 I [irq103: ioat1] 100255 I [irq104: ioat2] 100256 I [irq105: ioat3] 100257 I [irq106: ioat4] 100258 I [irq107: ioat5] 100259 I [irq108: ioat6] 100260 I [irq109: ioat7] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 CanRun [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffff80003c23000 [init] 10 0 0 0 DL audit_w 0xffffffff81fb3230 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81e8b530 [swapper] 100015 D - 0xfffff80003a0d000 [softirq_0] 100016 D - 0xfffff80003a0cd00 [softirq_1] 100017 D - 0xfffff80003a0cb00 [softirq_2] 100018 D - 0xfffff80003a0c900 [softirq_3] 100019 D - 0xfffff80003a0c700 [softirq_4] 100020 D - 0xfffff80003a0c500 [softirq_5] 100021 D - 0xfffff80003a0c300 [softirq_6] 100022 D - 0xfffff80003a0c100 [softirq_7] 100023 D - 0xfffff80003a0be00 [softirq_8] 100024 D - 0xfffff80003a0bc00 [softirq_9] 100025 D - 0xfffff80003a0ba00 [softirq_10] 100026 D - 0xfffff80003a0b800 [softirq_11] 100027 D - 0xfffff80003a0b600 [if_io_tqg_0] 100028 D - 0xfffff80003a0b400 [if_io_tqg_1] 100029 D - 0xfffff80003a0b200 [if_io_tqg_2] 100030 D - 0xfffff80003a0b000 [if_io_tqg_3] 100031 D - 0xfffff80003a0ad00 [if_io_tqg_4] 100032 D - 0xfffff80003a0ab00 [if_io_tqg_5] 100033 D - 0xfffff80003a0a900 [if_io_tqg_6] 100034 D - 0xfffff80003a0a700 [if_io_tqg_7] 100035 D - 0xfffff80003a0a500 [if_io_tqg_8] 100036 D - 0xfffff80003a0a300 [if_io_tqg_9] 100037 D - 0xfffff80003a0a100 [if_io_tqg_10] 100038 D - 0xfffff80003bd4300 [if_io_tqg_11] 100039 D - 0xfffff80003bd4100 [if_config_tqg_0] 100040 D - 0xfffff80003bd5e00 [inm_free taskq] 100041 D - 0xfffff80003bd5c00 [deferred_unmount ta] 100042 D - 0xfffff80003bd5a00 [thread taskq] 100044 D - 0xfffff80003bd5600 [pci_hp taskq] 100045 D - 0xfffff80003bd5400 [linuxkpi_irq_wq] 100046 D - 0xfffff80003bd5200 [kqueue_ctx taskq] 100047 D - 0xfffff80003bd5000 [in6m_free taskq] 100050 D - 0xfffff80003bd6900 [aiod_kick taskq] 100051 D - 0xfffff80003bd6700 [linuxkpi_short_wq_0] 100052 D - 0xfffff80003bd6700 [linuxkpi_short_wq_1] 100053 D - 0xfffff80003bd6700 [linuxkpi_short_wq_2] 100054 D - 0xfffff80003bd6700 [linuxkpi_short_wq_3] 100055 D - 0xfffff80003bd6700 [linuxkpi_short_wq_4] 100056 D - 0xfffff80003bd6700 [linuxkpi_short_wq_5] 100057 D - 0xfffff80003bd6700 [linuxkpi_short_wq_6] 100058 D - 0xfffff80003bd6700 [linuxkpi_short_wq_7] 100059 D - 0xfffff80003bd6700 [linuxkpi_short_wq_8] 100060 D - 0xfffff80003bd6700 [linuxkpi_short_wq_9] 100061 D - 0xfffff80003bd6700 [linuxkpi_short_wq_1] 100062 D - 0xfffff80003bd6700 [linuxkpi_short_wq_1] 100063 D - 0xfffff80003bd6700 [linuxkpi_short_wq_1] 100064 D - 0xfffff80003bd7800 [linuxkpi_long_wq_0] 100065 D - 0xfffff80003bd7800 [linuxkpi_long_wq_1] 100066 D - 0xfffff80003bd7800 [linuxkpi_long_wq_2] 100067 D - 0xfffff80003bd7800 [linuxkpi_long_wq_3] 100068 D - 0xfffff80003bd7800 [linuxkpi_long_wq_4] 100069 D - 0xfffff80003bd7800 [linuxkpi_long_wq_5] 100070 D - 0xfffff80003bd7800 [linuxkpi_long_wq_6] 100071 D - 0xfffff80003bd7800 [linuxkpi_long_wq_7] 100072 D - 0xfffff80003bd7800 [linuxkpi_long_wq_8] 100073 D - 0xfffff80003bd7800 [linuxkpi_long_wq_9] 100074 D - 0xfffff80003bd7800 [linuxkpi_long_wq_10] 100075 D - 0xfffff80003bd7800 [linuxkpi_long_wq_11] 100076 D - 0xfffff80003bd7800 [linuxkpi_long_wq_12] 100091 D - 0xfffff8000403f600 [firmware taskq] 100095 D - 0xfffff8000403f100 [crypto_0] 100096 D - 0xfffff8000403f100 [crypto_1] 100097 D - 0xfffff8000403f100 [crypto_2] 100098 D - 0xfffff8000403f100 [crypto_3] 100099 D - 0xfffff8000403f100 [crypto_4] 100100 D - 0xfffff8000403f100 [crypto_5] 100101 D - 0xfffff8000403f100 [crypto_6] 100102 D - 0xfffff8000403f100 [crypto_7] 100103 D - 0xfffff8000403f100 [crypto_8] 100104 D - 0xfffff8000403f100 [crypto_9] 100105 D - 0xfffff8000403f100 [crypto_10] 100106 D - 0xfffff8000403f100 [crypto_11] 100147 D - 0xfffff8002d4b0d00 [acpi_task_0] 100148 D - 0xfffff8002d4b0d00 [acpi_task_1] 100149 D - 0xfffff8002d4b0d00 [acpi_task_2] 100150 D - 0xfffff80003a0e100 [mca taskq] 100151 D - 0xfffff80004041400 [CAM taskq] 100155 D - 0xfffff8002d4b0600 [mlx5_core0-rec] 100156 D - 0xfffff8002d4b0400 [mlx5_core0-wdg] 100157 D - 0xfffff8002d4b0200 [mlx5_core0-cmd] 100158 D - 0xfffff8002d4b0000 [mlx5_page_allocator] 100174 D - 0xfffff8002d4b2d00 [mlx5_esw_wq] 100175 D - 0xfffff8002d4b2b00 [mlx5_core1-rec] 100176 D - 0xfffff8002d4b2900 [mlx5_core1-wdg] 100177 D - 0xfffff8002d4b2700 [mlx5_core1-cmd] 100178 D - 0xfffff8002d4b2500 [mlx5_page_allocator] 100194 D - 0xfffff80003dfb100 [mlx5_esw_wq] db:0:pho> allt Tracing command kernel pid 0 tid 100000 td 0xffffffff81e8ba80 sched_switch() at sched_switch+0x559/frame 0xffffffff8235fe00 mi_switch() at mi_switch+0x157/frame 0xffffffff8235fe20 sleepq_switch() at sleepq_switch+0x107/frame 0xffffffff8235fe60 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xffffffff8235fea0 _sleep() at _sleep+0x2ce/frame 0xffffffff8235ff40 swapper() at swapper+0x75/frame 0xffffffff8235ffa0 mi_startup() at mi_startup+0x2f3/frame 0xffffffff8235fff0 btext() at btext+0x22 Tracing command kernel pid 0 tid 100015 td 0xfffffe00e4c62020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4858de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4858e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4858e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4858ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4858ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4858f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4858f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe00e4c61900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4853de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4853e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4853e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4853ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4853ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4853f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4853f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xfffffe00e4ce6720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4862de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4862e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4862e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4862ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4862ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4862f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4862f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe00e4ce6000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e485dde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e485de00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e485de40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e485dec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e485def0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e485df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e485df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffffe00e4c69740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e486cde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e486ce00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e486ce40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e486cec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e486cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e486cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e486cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe00e4c69020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4867de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4867e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4867e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4867ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4867ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4867f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4867f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffffe00e4c68900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4876de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4876e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4876e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4876ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4876ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4876f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4876f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffffe00e4c681e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4871de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4871e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4871e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4871ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4871ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4871f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4871f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffffe00e4c67ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4880de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4880e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4880e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4880ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4880ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4880f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4880f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00e4c673a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e487bde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e487be00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e487be40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e487bec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e487bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e487bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e487bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00e4cea020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e488ade0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e488ae00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e488ae40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e488aec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e488aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e488af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e488af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00e4ce9900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4885de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4885e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4885e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4885ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4885ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4885f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4885f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe00e4ce91e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4894de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4894e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4894e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4894ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4894ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4894f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4894f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffffe00e4ce8ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e488fde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e488fe00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e488fe40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e488fec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e488fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e488ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e488ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffffe00e4ce83a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e489ede0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e489ee00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e489ee40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e489eec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e489eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e489ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e489ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffffe00e4ce7c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4899de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4899e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4899e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4899ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e4899ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4899f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4899f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffffe00e4ce7560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48a8de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48a8e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48a8e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48a8ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48a8ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48a8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48a8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffffe00e4ce6e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48a3de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48a3e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48a3e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48a3ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48a3ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48a3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48a3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffffe00e4cf3ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48b2de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48b2e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48b2e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48b2ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48b2ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48b2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48b2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffffe00e4cf33a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48adde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48ade00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48ade40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48adec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48adef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48adf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48adf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffffe00e4cf2c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48bcde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48bce00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48bce40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48bcec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48bcef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48bcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48bcf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffffe00e4cf2560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48b7de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48b7e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48b7e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48b7ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48b7ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48b7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48b7f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffffe00e4cf1e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48c6de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48c6e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48c6e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48c6ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48c6ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48c6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48c6f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffffe00e4cf1720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48c1de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48c1e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48c1e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48c1ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48c1ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48c1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48c1f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffffe00e4cf1000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48d0de0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48d0e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48d0e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48d0ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00e48d0ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48d0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48d0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffffe00e4cea740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48cbdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48cbde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48cbe20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48cbec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48cbef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48cbf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48cbf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffffe00e4cf9560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48dadc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48dade0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48dae20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48daec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48daef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48daf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48daf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffffe00e4cf8e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48d5dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48d5de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48d5e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48d5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48d5ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48d5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48d5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffffe00e4cf8000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48dfdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48dfde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48dfe20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48dfec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48dfef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48dff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48dff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffffe00e4cf5740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48eede0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48eee00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48eee40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e48eeec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e48eeef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48eef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48eef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xfffffe00e4cf5020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48e9dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48e9de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48e9e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48e9ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48e9ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48e9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48e9f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100047 td 0xfffffe00e4cf4900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48f8dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48f8de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48f8e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48f8ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48f8ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48f8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48f8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100050 td 0xfffffe00e4cfc740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48fddc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48fdde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e48fde20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e48fdec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e48fdef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48fdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48fdf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100051 td 0xfffffe00e4cfc020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e490cdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e490cde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e490ce20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e490cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e490cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e490cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e490cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100052 td 0xfffffe00e4cfb900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4907dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4907de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4907e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4907ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4907ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4907f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4907f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100053 td 0xfffffe00e4cfb1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4916dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4916de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4916e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4916ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4916ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4916f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4916f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100054 td 0xfffffe00e4cfaac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4911dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4911de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4911e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4911ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4911ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4911f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4911f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100055 td 0xfffffe00e4cfa3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4920dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4920de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4920e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4920ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4920ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4920f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4920f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100056 td 0xfffffe00e4cf9c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e491bdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e491bde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e491be20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e491bec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e491bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e491bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e491bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100057 td 0xfffffe00e4d04900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e492adc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e492ade0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e492ae20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e492aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e492aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e492af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e492af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xfffffe00e4d041e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4925dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4925de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4925e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4925ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4925ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4925f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4925f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100059 td 0xfffffe00e4d03ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4934dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4934de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4934e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4934ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4934ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4934f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4934f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100060 td 0xfffffe00e4d033a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e492fdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e492fde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e492fe20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e492fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e492fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e492ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e492ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xfffffe00e4d02c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e493edc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e493ede0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e493ee20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e493eec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e493eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e493ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e493ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100062 td 0xfffffe00e4d02560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4939dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4939de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4939e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4939ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4939ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4939f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4939f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffffe00e4d01e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4948dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4948de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4948e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4948ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4948ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4948f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4948f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffffe00e4d01720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4943dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4943de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4943e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4943ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4943ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4943f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4943f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe00e4d0a3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4952dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4952de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4952e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4952ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4952ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4952f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4952f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffffe00e4d09c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e494ddc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e494dde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e494de20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e494dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e494def0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e494df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e494df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffffe00e4d09560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e495cdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e495cde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e495ce20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e495cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e495cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e495cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e495cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffffe00e4d08e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4957dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4957de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4957e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4957ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4957ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4957f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4957f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffffe00e4d08720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4966dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4966de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4966e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4966ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4966ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4966f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4966f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffffe00e4d08000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4961dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4961de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4961e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4961ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4961ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4961f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4961f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffffe00e4d05740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4970dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4970de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4970e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4970ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4970ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4970f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4970f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffffe00e4d05020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e496bdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e496bde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e496be20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e496bec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e496bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e496bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e496bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffffe00e4d0de40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e497adc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e497ade0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e497ae20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e497aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e497aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e497af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e497af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffffe00e4d0d720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4975dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4975de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4975e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4975ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4975ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4975f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4975f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffffe00e4d0d000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4984dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4984de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4984e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4984ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4984ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4984f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4984f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100076 td 0xfffffe00e4d0c740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e497fdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e497fde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e497fe20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e497fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e497fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e497ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e497ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100091 td 0xfffffe00e4d17720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e34dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e34de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e34e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e34ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e34ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e34f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e34f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xfffffe00e4d2ce40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e48dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e48de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e48e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e48ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e48ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e48f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e48f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100096 td 0xfffffe00e4d2c720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e43dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e43de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e43e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e43ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e43ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e43f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e43f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xfffffe00e4d2c000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e52dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e52de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e52e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e52ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e52ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e52f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e52f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100098 td 0xfffffe00e4d1b740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e4ddc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e4dde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e4de20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e4dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e4def0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e4df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e4df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100099 td 0xfffffe00e4d1b020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e5cdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e5cde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e5ce20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e5cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e5cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e5cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e5cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100100 td 0xfffffe00e4d1a900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e57dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e57de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e57e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e57ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e57ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e57f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e57f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100101 td 0xfffffe00e4d36720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e66dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e66de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e66e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e66ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e66ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e66f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e66f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100102 td 0xfffffe00e4d36000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e61dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e61de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e61e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e61ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e61ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e61f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e61f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100103 td 0xfffffe00e4d30740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e70dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e70de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e70e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e70ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e70ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e70f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e70f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100104 td 0xfffffe00e4d30020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e6bdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e6bde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e6be20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e6bec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e6bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e6bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e6bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100105 td 0xfffffe00e4d2f900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e7adc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e7ade0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e7ae20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e7aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e7aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e7af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e7af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xfffffe00e4d2f1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e75dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e75de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e75e20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e75ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00e4e75ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e75f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e75f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100147 td 0xfffffe010d9eec80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ffcde0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ffce00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ffce40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe00e4ffcec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe00e4ffcef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ffcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ffcf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100148 td 0xfffffe010d9ee560 sched_switch() at sched_switch+0x559/frame 0xfffffe010da32de0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da32e00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da32e40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010da32ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe010da32ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da32f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da32f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100149 td 0xfffffe010d9ede40 sched_switch() at sched_switch+0x559/frame 0xfffffe010da2dde0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da2de00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da2de40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010da2dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe010da2def0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da2df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da2df30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100150 td 0xfffffe010d9ed720 sched_switch() at sched_switch+0x559/frame 0xfffffe010da3cde0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da3ce00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da3ce40 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010da3cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe010da3cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da3cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da3cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100151 td 0xfffffe010d9ed000 sched_switch() at sched_switch+0x559/frame 0xfffffe010da37dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da37de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da37e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da37ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010da37ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da37f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da37f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100155 td 0xfffffe010a32d020 sched_switch() at sched_switch+0x559/frame 0xfffffe010da41dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da41de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da41e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da41ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010da41ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da41f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da41f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100156 td 0xfffffe010d9fc740 sched_switch() at sched_switch+0x559/frame 0xfffffe010da66dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da66de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da66e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da66ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010da66ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da66f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da66f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100157 td 0xfffffe010d9fc020 sched_switch() at sched_switch+0x559/frame 0xfffffe010da61dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da61de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da61e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da61ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010da61ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da61f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da61f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100158 td 0xfffffe010d9fb900 sched_switch() at sched_switch+0x559/frame 0xfffffe010da70dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da70de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da70e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da70ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010da70ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da70f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da70f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100174 td 0xfffffe010e655e40 sched_switch() at sched_switch+0x559/frame 0xfffffe010dac0dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dac0de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dac0e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010dac0ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010dac0ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dac0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dac0f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100175 td 0xfffffe010e655720 sched_switch() at sched_switch+0x559/frame 0xfffffe010dabbdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dabbde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dabbe20 _sleep() at _sleep+0x2e7/frame 0xfffffe010dabbec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010dabbef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dabbf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dabbf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100176 td 0xfffffe010e655000 sched_switch() at sched_switch+0x559/frame 0xfffffe010dacadc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dacade0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dacae20 _sleep() at _sleep+0x2e7/frame 0xfffffe010dacaec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010dacaef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dacaf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dacaf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100177 td 0xfffffe010e463740 sched_switch() at sched_switch+0x559/frame 0xfffffe010dac5dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dac5de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dac5e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010dac5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010dac5ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dac5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dac5f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100178 td 0xfffffe010e463020 sched_switch() at sched_switch+0x559/frame 0xfffffe010dad4dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dad4de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dad4e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010dad4ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010dad4ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dad4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dad4f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100194 td 0xfffffe010e208560 sched_switch() at sched_switch+0x559/frame 0xfffffe010db24dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db24de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db24e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010db24ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe010db24ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db24f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db24f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command init pid 1 tid 100002 td 0xfffffe00e4c60ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00377e1a10 mi_switch() at mi_switch+0x157/frame 0xfffffe00377e1a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00377e1a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00377e1ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00377e1ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe00377e1b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe00377e1c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00377e1e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe00377e1f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00377e1f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x28e8da, rsp = 0x820cdb358, rbp = 0x820cdb3c0 --- Tracing command clock pid 2 tid 100078 td 0xfffffe00e4d11020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49b9ea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49b9ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49b9ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b9f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100079 td 0xfffffe00e4d10900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49c8ea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49c8ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49c8ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49c8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49c8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100080 td 0xfffffe00e4d101e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49c3ea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49c3ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49c3ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49c3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49c3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100081 td 0xfffffe00e4d0fac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49d2ea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49d2ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49d2ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49d2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49d2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100082 td 0xfffffe00e4d0f3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49cdea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49cdec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49cdef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49cdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49cdf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100083 td 0xfffffe00e4d0ec80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49dcea0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49dcec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe00e49dcef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49dcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49dcf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command clock pid 2 tid 100084 td 0xfffffe00e4d0e560 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100085 td 0xfffffe00e4d1a1e0 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100086 td 0xfffffe00e4d19ac0 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100087 td 0xfffffe00e4d193a0 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100088 td 0xfffffe00e4d18c80 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100089 td 0xfffffe00e4d18560 fork_trampoline() at fork_trampoline Tracing command crypto pid 3 tid 100107 td 0xfffffe00e4d2eac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e84db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e84dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e84e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e84eb0 crypto_dispatch_thread() at crypto_dispatch_thread+0xbf/frame 0xfffffe00e4e84ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e84f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e84f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100108 td 0xfffffe00e4d2e3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e7fdb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e7fdd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e7fe10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e7feb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e7fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e7ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e7ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100109 td 0xfffffe00e4d3a020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e8edb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e8edd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e8ee10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e8eeb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e8eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e8ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e8ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100110 td 0xfffffe00e4d39900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e89db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e89dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e89e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e89eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e89ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e89f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e89f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100111 td 0xfffffe00e4d391e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e98db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e98dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e98e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e98eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e98ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e98f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e98f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100112 td 0xfffffe00e4d38ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e93db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e93dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e93e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e93eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e93ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e93f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e93f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100113 td 0xfffffe00e4d383a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ea2db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ea2dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ea2e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ea2eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4ea2ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ea2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ea2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100114 td 0xfffffe00e4d37c80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e9ddb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e9ddd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e9de10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e9deb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4e9def0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e9df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e9df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100115 td 0xfffffe00e4d37560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4eacdb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4eacdd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4eace10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4eaceb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4eacef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eacf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eacf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100116 td 0xfffffe00e4d36e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ea7db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ea7dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ea7e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ea7eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4ea7ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ea7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ea7f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100117 td 0xfffffe00e4d3dac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4eb6db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4eb6dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4eb6e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4eb6eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4eb6ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eb6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eb6f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100118 td 0xfffffe00e4d3d3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4eb1db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4eb1dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4eb1e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4eb1eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4eb1ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eb1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eb1f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 3 tid 100119 td 0xfffffe00e4d3cc80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ec0db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ec0dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ec0e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ec0eb0 crypto_ret_thread() at crypto_ret_thread+0x134/frame 0xfffffe00e4ec0ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ec0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ec0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100121 td 0xfffffe00e4d3be40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ecada0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ecadc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ecae00 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ecaea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00e4ecaef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ecaf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ecaf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100122 td 0xfffffe00e4d3b720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ec5da0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ec5dc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ec5e00 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ec5ea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00e4ec5ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ec5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ec5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100123 td 0xfffffe00e4d3b000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ed4da0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ed4dc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ed4e00 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ed4ea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00e4ed4ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ed4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ed4f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100124 td 0xfffffe00e4d3a740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ecfda0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ecfdc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ecfe00 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4ecfea0 xpt_async_td() at xpt_async_td+0x74/frame 0xfffffe00e4ecfef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ecff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ecff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 4 tid 100152 td 0xfffffe010a32d740 sched_switch() at sched_switch+0x559/frame 0xfffffe010da46dc0 mi_switch() at mi_switch+0x157/frame 0xfffffe010da46de0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da46e20 _sleep() at _sleep+0x2e7/frame 0xfffffe010da46ec0 xpt_scanner_thread() at xpt_scanner_thread+0x154/frame 0xfffffe010da46ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da46f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da46f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command rand_harvestq pid 5 tid 100146 td 0xfffffe010d9ef3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe010da28d20 mi_switch() at mi_switch+0x157/frame 0xfffffe010da28d40 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da28d80 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010da28dc0 _sleep() at _sleep+0x2ce/frame 0xfffffe010da28e60 random_kthread() at random_kthread+0xa7/frame 0xfffffe010da28ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da28f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da28f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command enc_daemon0 pid 6 tid 100153 td 0xfffffe010d9f8e40 sched_switch() at sched_switch+0x559/frame 0xfffffe010da5cd80 mi_switch() at mi_switch+0x157/frame 0xfffffe010da5cda0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da5cde0 _sleep() at _sleep+0x2e7/frame 0xfffffe010da5ce80 enc_daemon() at enc_daemon+0xeb/frame 0xfffffe010da5cef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da5cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da5cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command enc_daemon1 pid 7 tid 100154 td 0xfffffe010d9f8720 sched_switch() at sched_switch+0x559/frame 0xfffffe010da57d80 mi_switch() at mi_switch+0x157/frame 0xfffffe010da57da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010da57de0 _sleep() at _sleep+0x2e7/frame 0xfffffe010da57e80 enc_daemon() at enc_daemon+0xeb/frame 0xfffffe010da57ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da57f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da57f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command pagedaemon pid 8 tid 100195 td 0xfffffe010e207e40 sched_switch() at sched_switch+0x559/frame 0xfffffe010db1fab0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db1fad0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db1fb10 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010db1fb50 _sleep() at _sleep+0x2ce/frame 0xfffffe010db1fbf0 vm_pageout_worker() at vm_pageout_worker+0x28c/frame 0xfffffe010db1feb0 vm_pageout() at vm_pageout+0x1d7/frame 0xfffffe010db1fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db1ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db1ff30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command pagedaemon pid 8 tid 100205 td 0xfffffe010a329720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f49b70 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f49b90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f49bd0 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4f49c70 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xb33/frame 0xfffffe00e4f49ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f49f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f49f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 8 tid 100206 td 0xfffffe010de97000 sched_switch() at sched_switch+0x559/frame 0xfffffe0115f95d80 mi_switch() at mi_switch+0x157/frame 0xfffffe0115f95da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0115f95de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0115f95e20 _sleep() at _sleep+0x2ce/frame 0xfffffe0115f95ec0 uma_reclaim_worker() at uma_reclaim_worker+0xc8/frame 0xfffffe0115f95ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0115f95f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0115f95f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 9 tid 100196 td 0xfffffe010e3e7ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe010db3ad80 mi_switch() at mi_switch+0x157/frame 0xfffffe010db3ada0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db3ade0 _sleep() at _sleep+0x2e7/frame 0xfffffe010db3ae80 vm_daemon() at vm_daemon+0x92/frame 0xfffffe010db3aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db3af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db3af30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command audit pid 10 tid 100001 td 0xfffffe00e4c611e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00377e6dd0 mi_switch() at mi_switch+0x157/frame 0xfffffe00377e6df0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00377e6e30 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00377e6e90 audit_worker() at audit_worker+0x9f/frame 0xfffffe00377e6ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377e6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377e6f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00e4c603a0 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81fea310 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81fea320 trap() at trap+0x56/frame 0xffffffff81fea430 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81fea430 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377f0dd0, rbp = 0xfffffe00377f0dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377f0dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377f0e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377f0e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377f0e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377f0ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00e4c5fc80 sched_switch() at sched_switch+0x559/frame 0xfffffe00377ebe30 mi_switch() at mi_switch+0x157/frame 0xfffffe00377ebe50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377ebef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377ebf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377ebf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00e4c5f560 sched_switch() at sched_switch+0x559/frame 0xfffffe00377fae30 mi_switch() at mi_switch+0x157/frame 0xfffffe00377fae50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377faef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377faf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377faf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00e4c5ee40 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce3ae00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce3ae10 trap() at trap+0x56/frame 0xfffffe003ce3af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3af20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377f5dd0, rbp = 0xfffffe00377f5dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377f5dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377f5e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377f5e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377f5e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377f5ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00e4c5e720 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce47e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce47e10 trap() at trap+0x56/frame 0xfffffe003ce47f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce47f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4830dd0, rbp = 0xfffffe00e4830dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4830dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4830e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4830e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4830e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4830ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4830f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4830f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffffe00e4c5e000 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce54e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce54e10 trap() at trap+0x56/frame 0xfffffe003ce54f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce54f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00377ffdd0, rbp = 0xfffffe00377ffdd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00377ffdd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00377ffe10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00377ffe30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00377ffe50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00377ffef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377fff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377fff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00e4c66c80 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce61e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce61e10 trap() at trap+0x56/frame 0xfffffe003ce61f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce61f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e483add0, rbp = 0xfffffe00e483add0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e483add0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e483ae10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e483ae30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e483ae50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e483aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e483af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e483af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffffe00e4c66560 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce6ee00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce6ee10 trap() at trap+0x56/frame 0xfffffe003ce6ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce6ef20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4835dd0, rbp = 0xfffffe00e4835dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4835dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4835e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4835e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4835e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4835ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4835f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4835f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00e4c65e40 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce7be00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce7be10 trap() at trap+0x56/frame 0xfffffe003ce7bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce7bf20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4844dd0, rbp = 0xfffffe00e4844dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4844dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4844e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4844e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4844e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4844ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4844f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4844f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffffe00e4c65720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e483fe30 mi_switch() at mi_switch+0x157/frame 0xfffffe00e483fe50 sched_idletd() at sched_idletd+0xa1/frame 0xfffffe00e483fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e483ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e483ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffffe00e4c65000 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce95e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce95e10 trap() at trap+0x56/frame 0xfffffe003ce95f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce95f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e484edd0, rbp = 0xfffffe00e484edd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e484edd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e484ee10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e484ee30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e484ee50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e484eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e484ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e484ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00e4c62740 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003cea2e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003cea2e10 trap() at trap+0x56/frame 0xfffffe003cea2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea2f20 --- trap 0x13, rip = 0xffffffff8109b946, rsp = 0xfffffe00e4849dd0, rbp = 0xfffffe00e4849dd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00e4849dd0 acpi_cpu_idle() at acpi_cpu_idle+0x278/frame 0xfffffe00e4849e10 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00e4849e30 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00e4849e50 sched_idletd() at sched_idletd+0x4b1/frame 0xfffffe00e4849ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4849f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4849f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100043 td 0xfffffe00e4cf8720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100048 td 0xfffffe00e4cf41e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e48f3e40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e48f3e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e48f3ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e48f3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48f3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100049 td 0xfffffe00e4d01000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4902e40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4902e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4902ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4902f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4902f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100077 td 0xfffffe00e4d11740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49bee40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49bee60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e49beef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49bef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49bef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100090 td 0xfffffe00e4d17e40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100125 td 0xfffffe00e4d89560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100126 td 0xfffffe00e4d88e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ed9e40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ed9e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4ed9ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ed9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ed9f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100127 td 0xfffffe00e4d88720 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ee8e40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ee8e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4ee8ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ee8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ee8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100133 td 0xfffffe010a329000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f06e40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f06e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4f06ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f06f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f06f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100144 td 0xfffffe010a32a560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100145 td 0xfffffe010a329e40 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f4ee40 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f4ee60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00e4f4eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f4ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f4ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100159 td 0xfffffe010d9fb1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe010da6be40 mi_switch() at mi_switch+0x157/frame 0xfffffe010da6be60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010da6bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010da6bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010da6bf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100160 td 0xfffffe010d9faac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100161 td 0xfffffe010d9fa3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100162 td 0xfffffe010d9f9c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100163 td 0xfffffe010d9f9560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100164 td 0xfffffe010e4621e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100165 td 0xfffffe010e461ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100166 td 0xfffffe010e4613a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100167 td 0xfffffe010e460c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100168 td 0xfffffe010e460560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100169 td 0xfffffe010e45fe40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100170 td 0xfffffe010e45f720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100171 td 0xfffffe010e45f000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100172 td 0xfffffe010e656c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100173 td 0xfffffe010e656560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100179 td 0xfffffe010e462900 sched_switch() at sched_switch+0x559/frame 0xfffffe010dacfe40 mi_switch() at mi_switch+0x157/frame 0xfffffe010dacfe60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010dacfef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dacff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dacff30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100180 td 0xfffffe010e207720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100181 td 0xfffffe010e207000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100182 td 0xfffffe010e659740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100183 td 0xfffffe010e659020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100184 td 0xfffffe010e658900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100185 td 0xfffffe010e6581e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100186 td 0xfffffe010e657ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100187 td 0xfffffe010e6573a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100188 td 0xfffffe010e20b020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100189 td 0xfffffe010e20a900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100190 td 0xfffffe010e20a1e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100191 td 0xfffffe010e209ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100192 td 0xfffffe010e2093a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100193 td 0xfffffe010e208c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100253 td 0xfffffe01428923a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100254 td 0xfffffe0142891c80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100255 td 0xfffffe0142891560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100256 td 0xfffffe0143613020 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100257 td 0xfffffe0143612900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100258 td 0xfffffe01436121e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100259 td 0xfffffe0143611ac0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100260 td 0xfffffe01436113a0 fork_trampoline() at fork_trampoline Tracing command geom pid 13 tid 100092 td 0xfffffe00e4d17000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e49ffdc0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e49ffde0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e49ffe20 _sleep() at _sleep+0x2e7/frame 0xfffffe00e49ffec0 g_run_events() at g_run_events+0x444/frame 0xfffffe00e49ffef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e49fff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49fff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100093 td 0xfffffe00e4d2dc80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e3edb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e3edd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e3ee10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e3eeb0 g_io_schedule_up() at g_io_schedule_up+0x3c/frame 0xfffffe00e4e3eee0 g_up_procbody() at g_up_procbody+0x5c/frame 0xfffffe00e4e3eef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e3ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e3ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100094 td 0xfffffe00e4d2d560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4e39db0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4e39dd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4e39e10 _sleep() at _sleep+0x2e7/frame 0xfffffe00e4e39eb0 g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xfffffe00e4e39ee0 g_down_procbody() at g_down_procbody+0x5c/frame 0xfffffe00e4e39ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4e39f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4e39f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sequencer 00 pid 14 tid 100120 td 0xfffffe00e4d3c560 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ebbdb0 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ebbdd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ebbe10 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ebbe70 seq_eventthread() at seq_eventthread+0x8c/frame 0xfffffe00e4ebbef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ebbf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ebbf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100128 td 0xfffffe00e4d88000 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ee3e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ee3e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ee3e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ee3ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ee3ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ee3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ee3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100129 td 0xfffffe00e4d3f740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ef2e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ef2e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ef2e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ef2ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ef2ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ef2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ef2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100130 td 0xfffffe00e4d3f020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4eede00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4eede20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4eede60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4eedec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4eedef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4eedf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4eedf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100131 td 0xfffffe00e4d3e900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4efce00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4efce20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4efce60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4efcec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4efcef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4efcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4efcf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100132 td 0xfffffe00e4d3e1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4ef7e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4ef7e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4ef7e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4ef7ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4ef7ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4ef7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4ef7f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100134 td 0xfffffe00e4d8c740 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f01e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f01e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f01e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f01ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f01ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f01f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f01f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100135 td 0xfffffe00e4d8c020 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f10e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f10e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f10e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f10ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f10ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f10f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f10f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100136 td 0xfffffe00e4d8b900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f0be00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f0be20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f0be60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f0bec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f0bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f0bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f0bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100137 td 0xfffffe00e4d8b1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f1ae00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f1ae20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f1ae60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f1aec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f1aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f1af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f1af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100138 td 0xfffffe00e4d8aac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f15e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f15e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f15e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f15ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f15ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f15f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f15f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100139 td 0xfffffe010a32c900 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f30e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f30e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f30e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f30ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f30ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f30f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f30f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100140 td 0xfffffe010a32c1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f2be00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f2be20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f2be60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f2bec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f2bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f2bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f2bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100141 td 0xfffffe010a32bac0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f3ae00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f3ae20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f3ae60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f3aec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f3aef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f3af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f3af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100142 td 0xfffffe010a32b3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f35e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f35e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f35e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f35ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f35ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f35f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f35f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100143 td 0xfffffe010a32ac80 sched_switch() at sched_switch+0x559/frame 0xfffffe00e4f44e00 mi_switch() at mi_switch+0x157/frame 0xfffffe00e4f44e20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00e4f44e60 _cv_wait() at _cv_wait+0x166/frame 0xfffffe00e4f44ec0 usb_process() at usb_process+0xb3/frame 0xfffffe00e4f44ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e4f44f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e4f44f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100197 td 0xfffffe010e3e73a0 sched_switch() at sched_switch+0x559/frame 0xfffffe010db35d60 mi_switch() at mi_switch+0x157/frame 0xfffffe010db35d80 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db35dc0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010db35e00 _sleep() at _sleep+0x2ce/frame 0xfffffe010db35ea0 buf_daemon() at buf_daemon+0x2f4/frame 0xfffffe010db35ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db35f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db35f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command bufdaemon pid 16 tid 100198 td 0xfffffe010e3f7560 sched_switch() at sched_switch+0x559/frame 0xfffffe0110926d80 mi_switch() at mi_switch+0x157/frame 0xfffffe0110926da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0110926de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0110926e20 _sleep() at _sleep+0x2ce/frame 0xfffffe0110926ec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe0110926ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0110926f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0110926f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100199 td 0xfffffe010e3f6e40 sched_switch() at sched_switch+0x559/frame 0xfffffe011051dd80 mi_switch() at mi_switch+0x157/frame 0xfffffe011051dda0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe011051dde0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe011051de20 _sleep() at _sleep+0x2ce/frame 0xfffffe011051dec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe011051def0 fork_exit() at fork_exit+0x80/frame 0xfffffe011051df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011051df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100200 td 0xfffffe010e3f6720 sched_switch() at sched_switch+0x559/frame 0xfffffe0111940d80 mi_switch() at mi_switch+0x157/frame 0xfffffe0111940da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0111940de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0111940e20 _sleep() at _sleep+0x2ce/frame 0xfffffe0111940ec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe0111940ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0111940f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0111940f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100201 td 0xfffffe010e3f6000 sched_switch() at sched_switch+0x559/frame 0xfffffe0111739d80 mi_switch() at mi_switch+0x157/frame 0xfffffe0111739da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0111739de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0111739e20 _sleep() at _sleep+0x2ce/frame 0xfffffe0111739ec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe0111739ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0111739f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0111739f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100202 td 0xfffffe010e3e9740 sched_switch() at sched_switch+0x559/frame 0xfffffe010db3fd80 mi_switch() at mi_switch+0x157/frame 0xfffffe010db3fda0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db3fde0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010db3fe20 _sleep() at _sleep+0x2ce/frame 0xfffffe010db3fec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe010db3fef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db3ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db3ff30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command bufdaemon pid 16 tid 100203 td 0xfffffe010e3e9020 sched_switch() at sched_switch+0x559/frame 0xfffffe0111f4bd80 mi_switch() at mi_switch+0x157/frame 0xfffffe0111f4bda0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0111f4bde0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0111f4be20 _sleep() at _sleep+0x2ce/frame 0xfffffe0111f4bec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe0111f4bef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0111f4bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0111f4bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100204 td 0xfffffe010e3e8900 sched_switch() at sched_switch+0x559/frame 0xfffffe010db44d80 mi_switch() at mi_switch+0x157/frame 0xfffffe010db44da0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db44de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010db44e20 _sleep() at _sleep+0x2ce/frame 0xfffffe010db44ec0 bufspace_daemon() at bufspace_daemon+0x296/frame 0xfffffe010db44ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010db44f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010db44f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command bufdaemon pid 16 tid 100218 td 0xfffffe00e4d89c80 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b71d70 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b71d90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b71dd0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0141b71e10 _sleep() at _sleep+0x2ce/frame 0xfffffe0141b71eb0 softdep_flush() at softdep_flush+0x188/frame 0xfffffe0141b71ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0141b71f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0141b71f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100225 td 0xfffffe01435f1c80 sched_switch() at sched_switch+0x559/frame 0xfffffe0141bded70 mi_switch() at mi_switch+0x157/frame 0xfffffe0141bded90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141bdedd0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0141bdee10 _sleep() at _sleep+0x2ce/frame 0xfffffe0141bdeeb0 softdep_flush() at softdep_flush+0x188/frame 0xfffffe0141bdeef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0141bdef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0141bdef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100560 td 0xfffffe014fe93020 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce2de00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce2de10 trap() at trap+0x56/frame 0xfffffe003ce2df20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce2df20 --- trap 0x13, rip = 0xffffffff80f9117a, rsp = 0xfffffe0150155b10, rbp = 0xfffffe0150155b30 --- vm_page_pqstate_commit() at vm_page_pqstate_commit+0x1a/frame 0xfffffe0150155b30 vm_page_release_locked() at vm_page_release_locked+0xff/frame 0xfffffe0150155b70 vfs_vmio_invalidate() at vfs_vmio_invalidate+0x1bb/frame 0xfffffe0150155bb0 brelse() at brelse+0x50b/frame 0xfffffe0150155bf0 indir_trunc() at indir_trunc+0x558/frame 0xfffffe0150155cc0 indir_trunc() at indir_trunc+0x2c4/frame 0xfffffe0150155d90 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x240/frame 0xfffffe0150155de0 process_worklist_item() at process_worklist_item+0x2a3/frame 0xfffffe0150155e70 softdep_process_worklist() at softdep_process_worklist+0xb7/frame 0xfffffe0150155eb0 softdep_flush() at softdep_flush+0x10f/frame 0xfffffe0150155ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0150155f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0150155f30 --- trap 0xa5a5a5a5, rip = 0xa5a5a5a5a5a5a5a5, rsp = 0xa5a5a5a5a5a5a5a5, rbp = 0 --- Tracing command vnlru pid 17 tid 100207 td 0xfffffe010e3e6c80 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b58d10 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b58d30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b58d70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0141b58db0 _sleep() at _sleep+0x2ce/frame 0xfffffe0141b58e50 vnlru_proc() at vnlru_proc+0x9fc/frame 0xfffffe0141b58ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0141b58f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0141b58f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 18 tid 100208 td 0xfffffe010e3e6560 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b53d40 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b53d60 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b53da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0141b53de0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe0141b53e60 sched_sync() at sched_sync+0x63d/frame 0xfffffe0141b53ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe0141b53f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0141b53f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command dhclient pid 2311 tid 100273 td 0xfffffe010de98c80 sched_switch() at sched_switch+0x559/frame 0xfffffe014388cab0 mi_switch() at mi_switch+0x157/frame 0xfffffe014388cad0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014388cb10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014388cb60 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014388cb70 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe014388cbd0 kern_select() at kern_select+0xa95/frame 0xfffffe014388cdc0 sys_select() at sys_select+0x56/frame 0xfffffe014388ce00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014388cf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014388cf30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x2cefcc7e177a, rsp = 0x2cefc9ea7c58, rbp = 0x2cefc9ea7d10 --- Tracing command dhclient pid 2314 tid 100231 td 0xfffffe01435f0720 sched_switch() at sched_switch+0x559/frame 0xfffffe010db6bae0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db6bb00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db6bb40 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db6bb90 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db6bba0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db6bc00 kern_poll_kfds() at kern_poll_kfds+0x4bc/frame 0xfffffe010db6bc90 kern_poll() at kern_poll+0x9f/frame 0xfffffe010db6bde0 sys_poll() at sys_poll+0x50/frame 0xfffffe010db6be00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db6bf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db6bf30 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x2cefcc7e167a, rsp = 0x2cefc9ea7e98, rbp = 0x2cefc9ea7ed0 --- Tracing command dhclient pid 2373 tid 100248 td 0xfffffe010e0e3720 sched_switch() at sched_switch+0x559/frame 0xfffffe010dbcaa90 mi_switch() at mi_switch+0x157/frame 0xfffffe010dbcaab0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dbcaaf0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010dbcab40 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010dbcab80 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe010dbcac00 kern_poll_kfds() at kern_poll_kfds+0x485/frame 0xfffffe010dbcac90 kern_poll() at kern_poll+0x9f/frame 0xfffffe010dbcade0 sys_poll() at sys_poll+0x50/frame 0xfffffe010dbcae00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010dbcaf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010dbcaf30 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x2cefcc7e167a, rsp = 0x2cefc9ea7e98, rbp = 0x2cefc9ea7ed0 --- Tracing command devd pid 2378 tid 100249 td 0xfffffe010e0e3000 sched_switch() at sched_switch+0x559/frame 0xfffffe010dbc5a60 mi_switch() at mi_switch+0x157/frame 0xfffffe010dbc5a80 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dbc5ac0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010dbc5b10 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010dbc5b50 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe010dbc5bd0 kern_select() at kern_select+0xa6f/frame 0xfffffe010dbc5dc0 sys_select() at sys_select+0x56/frame 0xfffffe010dbc5e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010dbc5f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010dbc5f30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x28b20a, rsp = 0x820ff5378, rbp = 0x820ff7510 --- Tracing command syslogd pid 2567 tid 100282 td 0xfffffe010de97e40 sched_switch() at sched_switch+0x559/frame 0xfffffe01438caab0 mi_switch() at mi_switch+0x157/frame 0xfffffe01438caad0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438cab10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438cab60 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438cab70 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01438cabd0 kern_select() at kern_select+0xa95/frame 0xfffffe01438cadc0 sys_select() at sys_select+0x56/frame 0xfffffe01438cae00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438caf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438caf30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0xa8f411177a, rsp = 0xa8f37835b8, rbp = 0xa8f3783b40 --- Tracing command watchdogd pid 2575 tid 100250 td 0xfffffe010de99ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe010dbd4bb0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dbd4bd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dbd4c10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010dbd4c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010dbd4ca0 _sleep() at _sleep+0x2bf/frame 0xfffffe010dbd4d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1d1/frame 0xfffffe010dbd4dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe010dbd4e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010dbd4f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010dbd4f30 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x258ff122a5fa, rsp = 0x258fee54dc28, rbp = 0x258fee54dc70 --- Tracing command rpcbind pid 2597 tid 100232 td 0xfffffe01435f0000 sched_switch() at sched_switch+0x559/frame 0xfffffe010db7aa90 mi_switch() at mi_switch+0x157/frame 0xfffffe010db7aab0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db7aaf0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db7ab40 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010db7ab80 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe010db7ac00 kern_poll_kfds() at kern_poll_kfds+0x485/frame 0xfffffe010db7ac90 kern_poll() at kern_poll+0x9f/frame 0xfffffe010db7ade0 sys_poll() at sys_poll+0x50/frame 0xfffffe010db7ae00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db7af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db7af30 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x10ba931567a, rsp = 0x10ba71327b8, rbp = 0x10ba71349e0 --- Tracing command rpc.statd pid 2622 tid 100271 td 0xfffffe014353bc80 sched_switch() at sched_switch+0x559/frame 0xfffffe0143849a60 mi_switch() at mi_switch+0x157/frame 0xfffffe0143849a80 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0143849ac0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0143849b10 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0143849b50 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe0143849bd0 kern_select() at kern_select+0xa6f/frame 0xfffffe0143849dc0 sys_select() at sys_select+0x56/frame 0xfffffe0143849e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143849f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143849f30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x15a56c06d77a, rsp = 0x15a56a7b4d78, rbp = 0x15a56a7b4ec0 --- Tracing command sh pid 84544 tid 100416 td 0xfffffe014451c000 sched_switch() at sched_switch+0x559/frame 0xfffffe014393da10 mi_switch() at mi_switch+0x157/frame 0xfffffe014393da30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014393da70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014393dac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014393dad0 _sleep() at _sleep+0x2dd/frame 0xfffffe014393db70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe014393dc00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe014393de00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014393df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014393df30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1e158cbdb8da, rsp = 0x1e15895ad048, rbp = 0x1e15895ad100 --- Tracing command rpc.lockd pid 2625 tid 100283 td 0xfffffe010de97720 sched_switch() at sched_switch+0x559/frame 0xfffffe01438c5990 mi_switch() at mi_switch+0x157/frame 0xfffffe01438c59b0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438c59f0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438c5a40 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe01438c5a80 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe01438c5b00 svc_run_internal() at svc_run_internal+0xada/frame 0xfffffe01438c5c30 svc_run() at svc_run+0x230/frame 0xfffffe01438c5c90 sys_nlm_syscall() at sys_nlm_syscall+0x6fa/frame 0xfffffe01438c5e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438c5f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438c5f30 --- syscall (154, FreeBSD ELF64, sys_nlm_syscall), rip = 0xe1e6a27deba, rsp = 0xe1e677c6308, rbp = 0xe1e677c63d0 --- Tracing command powerd pid 2630 tid 100265 td 0xfffffe010e3f8ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe014385fa60 mi_switch() at mi_switch+0x157/frame 0xfffffe014385fa80 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014385fac0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014385fb10 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe014385fb50 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe014385fbd0 kern_select() at kern_select+0xa6f/frame 0xfffffe014385fdc0 sys_select() at sys_select+0x56/frame 0xfffffe014385fe00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014385ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014385ff30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0xa89e377877a, rsp = 0xa89e10daa28, rbp = 0xa89e10dab60 --- Tracing command ntpd pid 2656 tid 100229 td 0xfffffe01435f0e40 sched_switch() at sched_switch+0x559/frame 0xfffffe010db70ab0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db70ad0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db70b10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db70b60 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db70b70 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db70bd0 kern_select() at kern_select+0xa95/frame 0xfffffe010db70dc0 sys_select() at sys_select+0x56/frame 0xfffffe010db70e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db70f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db70f30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x320a8e59d77a, rsp = 0x320a8c8e71a8, rbp = 0x320a8c8e71e0 --- Tracing command inetd pid 2682 tid 100276 td 0xfffffe010e3f7c80 sched_switch() at sched_switch+0x559/frame 0xfffffe014387dab0 mi_switch() at mi_switch+0x157/frame 0xfffffe014387dad0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014387db10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014387db60 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014387db70 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe014387dbd0 kern_select() at kern_select+0xa95/frame 0xfffffe014387ddc0 sys_select() at sys_select+0x56/frame 0xfffffe014387de00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014387df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014387df30 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0xda15536c77a, rsp = 0xda1528b1d88, rbp = 0xda1528b2c20 --- Tracing command sshd pid 2701 tid 100234 td 0xfffffe010e0e4560 sched_switch() at sched_switch+0x559/frame 0xfffffe010db8ea70 mi_switch() at mi_switch+0x157/frame 0xfffffe010db8ea90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db8ead0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db8eb20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db8eb30 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db8eb90 kern_select() at kern_select+0xa95/frame 0xfffffe010db8ed80 sys_pselect() at sys_pselect+0x127/frame 0xfffffe010db8ee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db8ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db8ef30 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x1c423b6da6ba, rsp = 0x1c42343b2a18, rbp = 0x1c42343b2a60 --- Tracing command cron pid 2705 tid 100215 td 0xfffffe010d9f1740 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b7bbb0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b7bbd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b7bc10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b7bc60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0141b7bca0 _sleep() at _sleep+0x2bf/frame 0xfffffe0141b7bd40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1d1/frame 0xfffffe0141b7bdc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe0141b7be00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b7bf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b7bf30 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x365eed715fa, rsp = 0x365ed104b28, rbp = 0x365ed104d10 --- Tracing command getty pid 2719 tid 100209 td 0xfffffe010de9a900 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b62af0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b62b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b62b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b62ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141b62bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141b62c10 tty_wait() at tty_wait+0x42/frame 0xfffffe0141b62c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0141b62cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0141b62ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141b62d40 dofileread() at dofileread+0x81/frame 0xfffffe0141b62d90 sys_read() at sys_read+0xc0/frame 0xfffffe0141b62e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b62f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b62f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x379203d296fa, rsp = 0x3792019f68d8, rbp = 0x3792019f6920 --- Tracing command getty pid 2720 tid 100238 td 0xfffffe010e0e3e40 sched_switch() at sched_switch+0x559/frame 0xfffffe010db89af0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db89b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db89b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db89ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db89bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db89c10 tty_wait() at tty_wait+0x42/frame 0xfffffe010db89c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe010db89cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010db89ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe010db89d40 dofileread() at dofileread+0x81/frame 0xfffffe010db89d90 sys_read() at sys_read+0xc0/frame 0xfffffe010db89e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db89f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db89f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x1e4a893f86fa, rsp = 0x1e4a86b1c9a8, rbp = 0x1e4a86b1c9f0 --- Tracing command getty pid 2721 tid 100237 td 0xfffffe0142890000 sched_switch() at sched_switch+0x559/frame 0xfffffe0141bbeaf0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141bbeb10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141bbeb50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141bbeba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141bbebb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141bbec10 tty_wait() at tty_wait+0x42/frame 0xfffffe0141bbec40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0141bbecb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0141bbece0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141bbed40 dofileread() at dofileread+0x81/frame 0xfffffe0141bbed90 sys_read() at sys_read+0xc0/frame 0xfffffe0141bbee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141bbef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141bbef30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x10d839f2e6fa, rsp = 0x10d837af7fd8, rbp = 0x10d837af8020 --- Tracing command getty pid 2722 tid 100236 td 0xfffffe014353e740 sched_switch() at sched_switch+0x559/frame 0xfffffe010db75af0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db75b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db75b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db75ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db75bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db75c10 tty_wait() at tty_wait+0x42/frame 0xfffffe010db75c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe010db75cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010db75ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe010db75d40 dofileread() at dofileread+0x81/frame 0xfffffe010db75d90 sys_read() at sys_read+0xc0/frame 0xfffffe010db75e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db75f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db75f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x35057d84e6fa, rsp = 0x35057baae478, rbp = 0x35057baae4c0 --- Tracing command getty pid 2723 tid 100212 td 0xfffffe00e4d8a3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b76af0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b76b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b76b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b76ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141b76bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141b76c10 tty_wait() at tty_wait+0x42/frame 0xfffffe0141b76c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0141b76cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0141b76ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141b76d40 dofileread() at dofileread+0x81/frame 0xfffffe0141b76d90 sys_read() at sys_read+0xc0/frame 0xfffffe0141b76e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b76f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b76f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x1a0fceddf6fa, rsp = 0x1a0fce3cbf18, rbp = 0x1a0fce3cbf60 --- Tracing command getty pid 2724 tid 100235 td 0xfffffe014360f000 sched_switch() at sched_switch+0x559/frame 0xfffffe010db7faf0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db7fb10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db7fb50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db7fba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db7fbb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db7fc10 tty_wait() at tty_wait+0x42/frame 0xfffffe010db7fc40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe010db7fcb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010db7fce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe010db7fd40 dofileread() at dofileread+0x81/frame 0xfffffe010db7fd90 sys_read() at sys_read+0xc0/frame 0xfffffe010db7fe00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db7ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db7ff30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8928e2216fa, rsp = 0x8928cde2638, rbp = 0x8928cde2680 --- Tracing command getty pid 2725 tid 100221 td 0xfffffe0142890720 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b99af0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b99b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b99b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b99ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141b99bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141b99c10 tty_wait() at tty_wait+0x42/frame 0xfffffe0141b99c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0141b99cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0141b99ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141b99d40 dofileread() at dofileread+0x81/frame 0xfffffe0141b99d90 sys_read() at sys_read+0xc0/frame 0xfffffe0141b99e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b99f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b99f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x259b36ff46fa, rsp = 0x259b34eb3ac8, rbp = 0x259b34eb3b10 --- Tracing command getty pid 2726 tid 100233 td 0xfffffe014360f720 sched_switch() at sched_switch+0x559/frame 0xfffffe010db84af0 mi_switch() at mi_switch+0x157/frame 0xfffffe010db84b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010db84b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010db84ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010db84bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010db84c10 tty_wait() at tty_wait+0x42/frame 0xfffffe010db84c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe010db84cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010db84ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe010db84d40 dofileread() at dofileread+0x81/frame 0xfffffe010db84d90 sys_read() at sys_read+0xc0/frame 0xfffffe010db84e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010db84f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010db84f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x13786917b6fa, rsp = 0x1378670f92a8, rbp = 0x1378670f92f0 --- Tracing command getty pid 2727 tid 100220 td 0xfffffe0142890e40 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b9eaf0 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b9eb10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b9eb50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b9eba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141b9ebb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0141b9ec10 tty_wait() at tty_wait+0x42/frame 0xfffffe0141b9ec40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0141b9ecb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0141b9ece0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0141b9ed40 dofileread() at dofileread+0x81/frame 0xfffffe0141b9ed90 sys_read() at sys_read+0xc0/frame 0xfffffe0141b9ee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b9ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b9ef30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x7b6946176fa, rsp = 0x7b693baae68, rbp = 0x7b693baaeb0 --- Tracing command login pid 2728 tid 100290 td 0xfffffe010e0e7740 sched_switch() at sched_switch+0x559/frame 0xfffffe0141bb9a10 mi_switch() at mi_switch+0x157/frame 0xfffffe0141bb9a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141bb9a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141bb9ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141bb9ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe0141bb9b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe0141bb9c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0141bb9e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141bb9f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141bb9f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x3ef9beeeb8da, rsp = 0x3ef9baf81c48, rbp = 0x3ef9baf81e20 --- Tracing command sshd pid 2729 tid 100268 td 0xfffffe010de9b740 sched_switch() at sched_switch+0x559/frame 0xfffffe0143878ae0 mi_switch() at mi_switch+0x157/frame 0xfffffe0143878b00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0143878b40 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0143878b90 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0143878ba0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0143878c00 kern_poll_kfds() at kern_poll_kfds+0x4bc/frame 0xfffffe0143878c90 kern_poll() at kern_poll+0x9f/frame 0xfffffe0143878de0 sys_poll() at sys_poll+0x50/frame 0xfffffe0143878e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143878f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143878f30 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x36999299b67a, rsp = 0x36998bb89d78, rbp = 0x36998bb89db0 --- Tracing command sshd pid 2732 tid 100287 td 0xfffffe0144153ac0 sched_switch() at sched_switch+0x559/frame 0xfffffe01438dea70 mi_switch() at mi_switch+0x157/frame 0xfffffe01438dea90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438dead0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438deb20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438deb30 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01438deb90 kern_select() at kern_select+0xa95/frame 0xfffffe01438ded80 sys_pselect() at sys_pselect+0x127/frame 0xfffffe01438dee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438def30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438def30 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x36999299b6ba, rsp = 0x36998bb85cf8, rbp = 0x36998bb85d40 --- Tracing command bash pid 2733 tid 100274 td 0xfffffe0143610560 sched_switch() at sched_switch+0x559/frame 0xfffffe01438a2a10 mi_switch() at mi_switch+0x157/frame 0xfffffe01438a2a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438a2a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438a2ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438a2ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe01438a2b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01438a2c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe01438a2e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438a2f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438a2f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x823af28da, rsp = 0x820bdba38, rbp = 0x820bdba90 --- Tracing command su pid 2738 tid 100291 td 0xfffffe014449f3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe01438e8a10 mi_switch() at mi_switch+0x157/frame 0xfffffe01438e8a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438e8a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438e8ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438e8ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe01438e8b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01438e8c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe01438e8e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438e8f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438e8f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8f189ecd8da, rsp = 0x8f18687a318, rbp = 0x8f18687a860 --- Tracing command csh pid 2739 tid 100245 td 0xfffffe01428931e0 sched_switch() at sched_switch+0x559/frame 0xfffffe010dba7ba0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dba7bc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dba7c00 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe010dba7c50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010dba7c60 _sleep() at _sleep+0x2dd/frame 0xfffffe010dba7d00 kern_sigsuspend() at kern_sigsuspend+0x1ec/frame 0xfffffe010dba7dd0 sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe010dba7e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010dba7f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010dba7f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x1363b982883a, rsp = 0x1363b7ac03b8, rbp = 0x1363b7ac0440 --- Tracing command bash pid 2748 tid 100286 td 0xfffffe01441541e0 sched_switch() at sched_switch+0x559/frame 0xfffffe01438cfa10 mi_switch() at mi_switch+0x157/frame 0xfffffe01438cfa30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438cfa70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438cfac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438cfad0 _sleep() at _sleep+0x2dd/frame 0xfffffe01438cfb70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01438cfc00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe01438cfe00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438cff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438cff30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x822ad08da, rsp = 0x820535068, rbp = 0x8205350c0 --- Tracing command mksnap_ffs pid 84846 tid 100444 td 0xfffffe014fcf3020 sched_switch() at sched_switch+0x559/frame 0xfffffe014ef6b0d0 mi_switch() at mi_switch+0x157/frame 0xfffffe014ef6b0f0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ef6b130 sleeplk() at sleeplk+0x107/frame 0xfffffe014ef6b190 lockmgr_xlock_hard() at lockmgr_xlock_hard+0x3ef/frame 0xfffffe014ef6b230 ffs_lock() at ffs_lock+0x14b/frame 0xfffffe014ef6b290 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x3c/frame 0xfffffe014ef6b2b0 _vn_lock() at _vn_lock+0x54/frame 0xfffffe014ef6b310 vget_finish() at vget_finish+0x7a/frame 0xfffffe014ef6b340 ffs_sync() at ffs_sync+0x193/frame 0xfffffe014ef6b3e0 vfs_write_suspend() at vfs_write_suspend+0x165/frame 0xfffffe014ef6b430 ffs_snapshot() at ffs_snapshot+0xf22/frame 0xfffffe014ef6b930 ffs_mount() at ffs_mount+0x60a/frame 0xfffffe014ef6ba80 vfs_domount_update() at vfs_domount_update+0x291/frame 0xfffffe014ef6bbf0 vfs_domount() at vfs_domount+0x26b/frame 0xfffffe014ef6bd20 vfs_donmount() at vfs_donmount+0x878/frame 0xfffffe014ef6bdc0 sys_nmount() at sys_nmount+0x69/frame 0xfffffe014ef6be00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ef6bf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ef6bf30 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x3c3a8c5f4e3a, rsp = 0x3c3a8b54dc78, rbp = 0x3c3a8b54f4d0 --- Tracing command egrep pid 84874 tid 100216 td 0xfffffe010e3e5e40 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe003ce20e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003ce20e10 trap() at trap+0x56/frame 0xfffffe003ce20f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce20f20 --- trap 0x13, rip = 0xffffffff80f70d04, rsp = 0xfffffe0141b8abd0, rbp = 0xfffffe0141b8ac00 --- uma_dbg_free() at uma_dbg_free+0x44/frame 0xfffffe0141b8ac00 uma_zfree_arg() at uma_zfree_arg+0x153/frame 0xfffffe0141b8ac50 vm_map_process_deferred() at vm_map_process_deferred+0x41/frame 0xfffffe0141b8ac70 vm_map_remove() at vm_map_remove+0xee/frame 0xfffffe0141b8aca0 pipeclose() at pipeclose+0x2e0/frame 0xfffffe0141b8ace0 pipe_close() at pipe_close+0x60/frame 0xfffffe0141b8ad10 _fdrop() at _fdrop+0x1b/frame 0xfffffe0141b8ad30 closef() at closef+0x1db/frame 0xfffffe0141b8adc0 closefp_impl() at closefp_impl+0x77/frame 0xfffffe0141b8ae00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b8af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b8af30 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x1b13a9eca4fa, rsp = 0x1b13a883adb8, rbp = 0x1b13a883adc0 --- Tracing command sh pid 3222 tid 100219 td 0xfffffe00e4d0c020 sched_switch() at sched_switch+0x559/frame 0xfffffe0141b94a10 mi_switch() at mi_switch+0x157/frame 0xfffffe0141b94a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141b94a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141b94ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141b94ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe0141b94b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe0141b94c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0141b94e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141b94f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141b94f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1e158cbdb8da, rsp = 0x1e15895ad438, rbp = 0x1e15895ad4f0 --- Tracing command sh pid 3286 tid 100240 td 0xfffffe0142894020 sched_switch() at sched_switch+0x559/frame 0xfffffe0141ba3a10 mi_switch() at mi_switch+0x157/frame 0xfffffe0141ba3a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0141ba3a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0141ba3ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0141ba3ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe0141ba3b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe0141ba3c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0141ba3e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0141ba3f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0141ba3f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1e158cbdb8da, rsp = 0x1e15895ad288, rbp = 0x1e15895ad340 --- Tracing command awk pid 3287 tid 100279 td 0xfffffe014413c740 sched_switch() at sched_switch+0x559/frame 0xfffffe01438b1b80 mi_switch() at mi_switch+0x157/frame 0xfffffe01438b1ba0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438b1be0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438b1c30 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438b1c40 _sleep() at _sleep+0x2dd/frame 0xfffffe01438b1ce0 pipe_read() at pipe_read+0x4b6/frame 0xfffffe01438b1d40 dofileread() at dofileread+0x81/frame 0xfffffe01438b1d90 sys_read() at sys_read+0xc0/frame 0xfffffe01438b1e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438b1f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438b1f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x28e85ec226fa, rsp = 0x28e85e5fdb68, rbp = 0x28e85e5fdb80 --- Tracing command sh pid 3289 tid 100266 td 0xfffffe010e3e5000 sched_switch() at sched_switch+0x559/frame 0xfffffe014386eb80 mi_switch() at mi_switch+0x157/frame 0xfffffe014386eba0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014386ebe0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014386ec30 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014386ec40 _sleep() at _sleep+0x2dd/frame 0xfffffe014386ece0 pipe_read() at pipe_read+0x4b6/frame 0xfffffe014386ed40 dofileread() at dofileread+0x81/frame 0xfffffe014386ed90 sys_read() at sys_read+0xc0/frame 0xfffffe014386ee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014386ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014386ef30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x1e158cbdb6fa, rsp = 0x1e15895acd38, rbp = 0x1e15895ad260 --- Tracing command csh pid 20253 tid 100520 td 0xfffffe01507f5c80 sched_switch() at sched_switch+0x559/frame 0xfffffe0150094af0 mi_switch() at mi_switch+0x157/frame 0xfffffe0150094b10 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0150094b50 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0150094ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0150094bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0150094c10 tty_wait() at tty_wait+0x42/frame 0xfffffe0150094c40 ttydisc_read() at ttydisc_read+0x21e/frame 0xfffffe0150094cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0150094ce0 devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0150094d40 dofileread() at dofileread+0x81/frame 0xfffffe0150094d90 sys_read() at sys_read+0xc0/frame 0xfffffe0150094e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150094f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150094f30 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x74c09e116fa, rsp = 0x74c0825bfc8, rbp = 0x74c0825bff0 --- Tracing command sshd pid 22870 tid 100418 td 0xfffffe01444a0900 sched_switch() at sched_switch+0x559/frame 0xfffffe014eef8ae0 mi_switch() at mi_switch+0x157/frame 0xfffffe014eef8b00 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014eef8b40 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014eef8b90 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014eef8ba0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe014eef8c00 kern_poll_kfds() at kern_poll_kfds+0x4bc/frame 0xfffffe014eef8c90 kern_poll() at kern_poll+0x9f/frame 0xfffffe014eef8de0 sys_poll() at sys_poll+0x50/frame 0xfffffe014eef8e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014eef8f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014eef8f30 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x3e1be0c2467a, rsp = 0x3e1bda03fc28, rbp = 0x3e1bda03fc60 --- Tracing command sshd pid 22873 tid 100498 td 0xfffffe014fd1f900 sched_switch() at sched_switch+0x559/frame 0xfffffe015000ea70 mi_switch() at mi_switch+0x157/frame 0xfffffe015000ea90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe015000ead0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe015000eb20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe015000eb30 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe015000eb90 kern_select() at kern_select+0xa95/frame 0xfffffe015000ed80 sys_pselect() at sys_pselect+0x127/frame 0xfffffe015000ee00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe015000ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe015000ef30 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x3e1be0c246ba, rsp = 0x3e1bda03bba8, rbp = 0x3e1bda03bbf0 --- Tracing command bash pid 22874 tid 100354 td 0xfffffe014ec0b000 sched_switch() at sched_switch+0x559/frame 0xfffffe01439d3a70 mi_switch() at mi_switch+0x157/frame 0xfffffe01439d3a90 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01439d3ad0 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01439d3b20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01439d3b30 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01439d3b90 kern_select() at kern_select+0xa95/frame 0xfffffe01439d3d80 sys_pselect() at sys_pselect+0x127/frame 0xfffffe01439d3e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01439d3f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01439d3f30 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x82446d6ba, rsp = 0x8212838b8, rbp = 0x821283980 --- Tracing command sh pid 81082 tid 100477 td 0xfffffe014fef6900 sched_switch() at sched_switch+0x559/frame 0xfffffe015001da10 mi_switch() at mi_switch+0x157/frame 0xfffffe015001da30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe015001da70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe015001dac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe015001dad0 _sleep() at _sleep+0x2dd/frame 0xfffffe015001db70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe015001dc00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe015001de00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe015001df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe015001df30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1e0c0f9aa8da, rsp = 0x1e0c0e465f28, rbp = 0x1e0c0e465fe0 --- Tracing command md10 pid 81092 tid 100242 td 0xfffffe01435f4740 sched_switch() at sched_switch+0x559/frame 0xfffffe010dba2da0 mi_switch() at mi_switch+0x157/frame 0xfffffe010dba2dc0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dba2e00 _sleep() at _sleep+0x2e7/frame 0xfffffe010dba2ea0 md_kthread() at md_kthread+0xd5/frame 0xfffffe010dba2ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe010dba2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010dba2f30 --- trap 0xc, rip = 0x2bf714aab31a, rsp = 0x2bf7127a1cb8, rbp = 0x2bf7127a1cd0 --- Tracing command sh pid 81102 tid 100497 td 0xfffffe01507f63a0 sched_switch() at sched_switch+0x559/frame 0xfffffe0150099a10 mi_switch() at mi_switch+0x157/frame 0xfffffe0150099a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0150099a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0150099ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0150099ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe0150099b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe0150099c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0150099e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150099f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150099f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x251011ec78da, rsp = 0x251010aa8208, rbp = 0x251010aa82c0 --- Tracing command run pid 81116 tid 100364 td 0xfffffe014eda3c80 sched_switch() at sched_switch+0x559/frame 0xfffffe014ee18bb0 mi_switch() at mi_switch+0x157/frame 0xfffffe014ee18bd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ee18c10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014ee18c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe014ee18ca0 _sleep() at _sleep+0x2bf/frame 0xfffffe014ee18d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1d1/frame 0xfffffe014ee18dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe014ee18e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ee18f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ee18f30 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x33fbc3e945fa, rsp = 0x33fbc23ab0f8, rbp = 0x33fbc23ab140 --- Tracing command run pid 81117 tid 100288 td 0xfffffe0144138e40 sched_switch() at sched_switch+0x559/frame 0xfffffe01438a7a10 mi_switch() at mi_switch+0x157/frame 0xfffffe01438a7a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438a7a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438a7ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438a7ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe01438a7b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01438a7c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe01438a7e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438a7f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438a7f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x33fbc3e948da, rsp = 0x33fbc23ab148, rbp = 0x33fbc23ab270 --- Tracing command run pid 81118 tid 100325 td 0xfffffe010e0e6900 sched_switch() at sched_switch+0x559/frame 0xfffffe01438eda10 mi_switch() at mi_switch+0x157/frame 0xfffffe01438eda30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01438eda70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe01438edac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01438edad0 _sleep() at _sleep+0x2dd/frame 0xfffffe01438edb70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe01438edc00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe01438ede00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01438edf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01438edf30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x33fbc3e948da, rsp = 0x33fbc23aac08, rbp = 0x33fbc23ab140 --- Tracing command mkdir pid 81121 tid 100267 td 0xfffffe010e20b740 sched_switch() at sched_switch+0x559/frame 0xfffffe0143869bb0 mi_switch() at mi_switch+0x157/frame 0xfffffe0143869bd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0143869c10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0143869c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0143869ca0 _sleep() at _sleep+0x2bf/frame 0xfffffe0143869d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1d1/frame 0xfffffe0143869dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe0143869e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143869f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143869f30 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x2e1d65c485fa, rsp = 0x2e1d642cb648, rbp = 0x2e1d642cb690 --- Tracing command rw pid 81122 tid 100474 td 0xfffffe014ec0f740 sched_switch() at sched_switch+0x559/frame 0xfffffe0150004bb0 mi_switch() at mi_switch+0x157/frame 0xfffffe0150004bd0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0150004c10 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0150004c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0150004ca0 _sleep() at _sleep+0x2bf/frame 0xfffffe0150004d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1d1/frame 0xfffffe0150004dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe0150004e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150004f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150004f30 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x34b4535015fa, rsp = 0x34b452450768, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81123 tid 100381 td 0xfffffe014edbb740 sched_switch() at sched_switch+0x559/frame 0xfffffe014ee45a10 mi_switch() at mi_switch+0x157/frame 0xfffffe014ee45a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ee45a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe014ee45ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe014ee45ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe014ee45b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe014ee45c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe014ee45e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ee45f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ee45f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2e1d65c488da, rsp = 0x2e1d642cb698, rbp = 0x2e1d642cb7c0 --- Tracing command mkdir pid 81124 tid 100481 td 0xfffffe014fef4c80 sched_switch() at sched_switch+0x559/frame 0xfffffe0150031b00 mi_switch() at mi_switch+0x157/frame 0xfffffe0150031b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0150031b60 _sleep() at _sleep+0x2e7/frame 0xfffffe0150031c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe0150031c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe0150031c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe0150031e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150031f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150031f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c2ea8, rbp = 0x2e1d642c32e0 --- Tracing command rw pid 81125 tid 100480 td 0xfffffe014fef53a0 sched_switch() at sched_switch+0x559/frame 0xfffffe0150022a10 mi_switch() at mi_switch+0x157/frame 0xfffffe0150022a30 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0150022a70 sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0150022ac0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0150022ad0 _sleep() at _sleep+0x2dd/frame 0xfffffe0150022b70 kern_wait6() at kern_wait6+0x474/frame 0xfffffe0150022c00 sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0150022e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150022f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150022f30 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x34b4535018da, rsp = 0x34b4524507b8, rbp = 0x34b4524508e0 --- Tracing command mkdir pid 81126 tid 100470 td 0xfffffe014f07dc80 sched_switch() at sched_switch+0x559/frame 0xfffffe014ee95b00 mi_switch() at mi_switch+0x157/frame 0xfffffe014ee95b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ee95b60 _sleep() at _sleep+0x2e7/frame 0xfffffe014ee95c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014ee95c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe014ee95c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe014ee95e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ee95f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ee95f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c1528, rbp = 0x2e1d642c1960 --- Tracing command rw pid 81127 tid 100518 td 0xfffffe014edb7720 sched_switch() at sched_switch+0x559/frame 0xfffffe014eea4940 mi_switch() at mi_switch+0x157/frame 0xfffffe014eea4960 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014eea49a0 _sleep() at _sleep+0x2e7/frame 0xfffffe014eea4a40 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014eea4a90 vn_write() at vn_write+0x105/frame 0xfffffe014eea4b20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe014eea4b80 vn_io_fault1() at vn_io_fault1+0x163/frame 0xfffffe014eea4cc0 vn_io_fault() at vn_io_fault+0x1aa/frame 0xfffffe014eea4d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe014eea4d90 sys_write() at sys_write+0xc0/frame 0xfffffe014eea4e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014eea4f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014eea4f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x34b45350191a, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81128 tid 100370 td 0xfffffe01437a0020 (CPU 9) kdb_enter() at kdb_enter+0x37/frame 0xfffffe0143938270 vpanic() at vpanic+0x1b0/frame 0xfffffe01439382c0 panic() at panic+0x43/frame 0xfffffe0143938320 trap_fatal() at trap_fatal+0x385/frame 0xfffffe0143938380 trap_pfault() at trap_pfault+0xab/frame 0xfffffe01439383e0 calltrap() at calltrap+0x8/frame 0xfffffe01439383e0 --- trap 0xc, rip = 0xffffffff80f2023c, rsp = 0xfffffe01439384b0, rbp = 0xfffffe01439385b0 --- ffs_snapblkfree() at ffs_snapblkfree+0x14c/frame 0xfffffe01439385b0 ffs_blkfree() at ffs_blkfree+0x62/frame 0xfffffe0143938610 freework_freeblock() at freework_freeblock+0x714/frame 0xfffffe0143938690 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x1eb/frame 0xfffffe01439386e0 softdep_setup_freeblocks() at softdep_setup_freeblocks+0x97c/frame 0xfffffe0143938750 ffs_truncate() at ffs_truncate+0xd8e/frame 0xfffffe0143938b90 ufs_inactive() at ufs_inactive+0x1aa/frame 0xfffffe0143938bd0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x59/frame 0xfffffe0143938bf0 vinactivef() at vinactivef+0xda/frame 0xfffffe0143938c30 vput_final() at vput_final+0x2aa/frame 0xfffffe0143938c90 kern_frmdirat() at kern_frmdirat+0x2d0/frame 0xfffffe0143938e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143938f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143938f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c72a8, rbp = 0x2e1d642c76e0 --- Tracing command rw pid 81129 tid 100431 td 0xfffffe014fce4e40 sched_switch() at sched_switch+0x559/frame 0xfffffe014ef2fad0 mi_switch() at mi_switch+0x157/frame 0xfffffe014ef2faf0 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ef2fb30 _sleep() at _sleep+0x2e7/frame 0xfffffe014ef2fbd0 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014ef2fc20 vn_close1() at vn_close1+0x8d/frame 0xfffffe014ef2fc90 vn_closefile() at vn_closefile+0x3d/frame 0xfffffe014ef2fd10 _fdrop() at _fdrop+0x1b/frame 0xfffffe014ef2fd30 closef() at closef+0x1db/frame 0xfffffe014ef2fdc0 closefp_impl() at closefp_impl+0x77/frame 0xfffffe014ef2fe00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ef2ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ef2ff30 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x34b4535014fa, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81130 tid 100360 td 0xfffffe01435f23a0 sched_switch() at sched_switch+0x559/frame 0xfffffe01439f1b00 mi_switch() at mi_switch+0x157/frame 0xfffffe01439f1b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01439f1b60 _sleep() at _sleep+0x2e7/frame 0xfffffe01439f1c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe01439f1c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe01439f1c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe01439f1e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01439f1f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01439f1f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642be228, rbp = 0x2e1d642be660 --- Tracing command rw pid 81131 tid 100475 td 0xfffffe014451e3a0 sched_switch() at sched_switch+0x559/frame 0xfffffe0143983940 mi_switch() at mi_switch+0x157/frame 0xfffffe0143983960 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01439839a0 _sleep() at _sleep+0x2e7/frame 0xfffffe0143983a40 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe0143983a90 vn_write() at vn_write+0x105/frame 0xfffffe0143983b20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0143983b80 vn_io_fault1() at vn_io_fault1+0x163/frame 0xfffffe0143983cc0 vn_io_fault() at vn_io_fault+0x1aa/frame 0xfffffe0143983d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0143983d90 sys_write() at sys_write+0xc0/frame 0xfffffe0143983e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143983f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143983f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x34b45350191a, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81132 tid 100264 td 0xfffffe010e3f91e0 sched_switch() at sched_switch+0x559/frame 0xfffffe0143864b00 mi_switch() at mi_switch+0x157/frame 0xfffffe0143864b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0143864b60 _sleep() at _sleep+0x2e7/frame 0xfffffe0143864c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe0143864c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe0143864c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe0143864e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0143864f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0143864f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c3fa8, rbp = 0x2e1d642c43e0 --- Tracing command mkdir pid 81133 tid 100425 td 0xfffffe014451f1e0 sched_switch() at sched_switch+0x559/frame 0xfffffe014ef11b00 mi_switch() at mi_switch+0x157/frame 0xfffffe014ef11b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014ef11b60 _sleep() at _sleep+0x2e7/frame 0xfffffe014ef11c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014ef11c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe014ef11c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe014ef11e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014ef11f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014ef11f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642beaa8, rbp = 0x2e1d642beee0 --- Tracing command rw pid 81134 tid 100490 td 0xfffffe014f09fac0 sched_switch() at sched_switch+0x559/frame 0xfffffe0150059940 mi_switch() at mi_switch+0x157/frame 0xfffffe0150059960 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01500599a0 _sleep() at _sleep+0x2e7/frame 0xfffffe0150059a40 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe0150059a90 vn_write() at vn_write+0x105/frame 0xfffffe0150059b20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0150059b80 vn_io_fault1() at vn_io_fault1+0x163/frame 0xfffffe0150059cc0 vn_io_fault() at vn_io_fault+0x1aa/frame 0xfffffe0150059d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0150059d90 sys_write() at sys_write+0xc0/frame 0xfffffe0150059e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe0150059f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0150059f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x34b45350191a, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81135 tid 100355 td 0xfffffe014ec0f020 sched_switch() at sched_switch+0x559/frame 0xfffffe01439e2b00 mi_switch() at mi_switch+0x157/frame 0xfffffe01439e2b20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe01439e2b60 _sleep() at _sleep+0x2e7/frame 0xfffffe01439e2c00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe01439e2c50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe01439e2c90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe01439e2e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe01439e2f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01439e2f30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642beee8, rbp = 0x2e1d642bf320 --- Tracing command rw pid 81136 tid 100462 td 0xfffffe014fd11720 sched_switch() at sched_switch+0x559/frame 0xfffffe014efc5940 mi_switch() at mi_switch+0x157/frame 0xfffffe014efc5960 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014efc59a0 _sleep() at _sleep+0x2e7/frame 0xfffffe014efc5a40 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014efc5a90 vn_write() at vn_write+0x105/frame 0xfffffe014efc5b20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe014efc5b80 vn_io_fault1() at vn_io_fault1+0x163/frame 0xfffffe014efc5cc0 vn_io_fault() at vn_io_fault+0x1aa/frame 0xfffffe014efc5d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe014efc5d90 sys_write() at sys_write+0xc0/frame 0xfffffe014efc5e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014efc5f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014efc5f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x34b45350191a, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- Tracing command mkdir pid 81137 tid 100407 td 0xfffffe014edcce40 sched_switch() at sched_switch+0x559/frame 0xfffffe014eebdb00 mi_switch() at mi_switch+0x157/frame 0xfffffe014eebdb20 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe014eebdb60 _sleep() at _sleep+0x2e7/frame 0xfffffe014eebdc00 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe014eebdc50 vn_start_write() at vn_start_write+0x6f/frame 0xfffffe014eebdc90 kern_frmdirat() at kern_frmdirat+0x3ab/frame 0xfffffe014eebde00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe014eebdf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe014eebdf30 --- syscall (137, FreeBSD ELF64, sys_rmdir), rip = 0x2e1d65c45e1a, rsp = 0x2e1d642c10e8, rbp = 0x2e1d642c1520 --- Tracing command rw pid 81138 tid 100289 td 0xfffffe010d9f0900 sched_switch() at sched_switch+0x559/frame 0xfffffe010dbd9940 mi_switch() at mi_switch+0x157/frame 0xfffffe010dbd9960 sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe010dbd99a0 _sleep() at _sleep+0x2e7/frame 0xfffffe010dbd9a40 vn_start_write_refed() at vn_start_write_refed+0x1ff/frame 0xfffffe010dbd9a90 vn_write() at vn_write+0x105/frame 0xfffffe010dbd9b20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe010dbd9b80 vn_io_fault1() at vn_io_fault1+0x163/frame 0xfffffe010dbd9cc0 vn_io_fault() at vn_io_fault+0x1aa/frame 0xfffffe010dbd9d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe010dbd9d90 sys_write() at sys_write+0xc0/frame 0xfffffe010dbd9e00 amd64_syscall() at amd64_syscall+0x145/frame 0xfffffe010dbd9f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010dbd9f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x34b45350191a, rsp = 0x34b45244f6f8, rbp = 0x34b4524507b0 --- db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 9 (kgdb) bt #0 __curthread () at /var/tmp/deviant3/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=textdump@entry=0x0) at ../../../kern/kern_shutdown.c:399 #2 0xffffffff804c657a in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:575 #3 0xffffffff804c6432 in db_command (last_cmdp=, cmd_table=, dopager=dopager@entry=0x0) at ../../../ddb/db_command.c:482 #4 0xffffffff804c64c6 in db_command_script (command=command@entry=0xffffffff81bd3cc0 "dump") at ../../../ddb/db_command.c:553 #5 0xffffffff804cb7f8 in db_script_exec (scriptname=0xffffffff81bd3630 "dump", warnifnotfound=) at ../../../ddb/db_script.c:304 #6 0xffffffff804c6432 in db_command (last_cmdp=, cmd_table=, dopager=dopager@entry=0x0) at ../../../ddb/db_command.c:482 #7 0xffffffff804c64c6 in db_command_script (command=command@entry=0xffffffff81bd3c52 "run pho1") at ../../../ddb/db_command.c:553 #8 0xffffffff804cb7f8 in db_script_exec (scriptname=0xffffffff81bd3630 "dump", warnifnotfound=) at ../../../ddb/db_script.c:304 #9 0xffffffff804c6432 in db_command (last_cmdp=, cmd_table=, dopager=dopager@entry=0x1) at ../../../ddb/db_command.c:482 #10 0xffffffff804c608d in db_command_loop () at ../../../ddb/db_command.c:535 #11 0xffffffff804c96c6 in db_trap (type=, code=) at ../../../ddb/db_main.c:270 #12 0xffffffff80c5345b in kdb_trap (type=type@entry=0x3, code=code@entry=0x0, tf=tf@entry=0xfffffe01439381a0) at ../../../kern/subr_kdb.c:733 #13 0xffffffff810e067a in trap (frame=0xfffffe01439381a0) at ../../../amd64/amd64/trap.c:609 #14 #15 kdb_enter (why=0xffffffff812e013a "panic", msg=) at ../../../kern/subr_kdb.c:506 #16 0xffffffff80c059f0 in vpanic (fmt=0xffffffff811efa77 "%s", ap=ap@entry=0xfffffe0143938300) at ../../../kern/kern_shutdown.c:908 #17 0xffffffff80c05783 in panic (fmt=0xffffffff81e8cf60 "d\303)\201\377\377\377\377") at ../../../kern/kern_shutdown.c:844 #18 0xffffffff810e0a85 in trap_fatal (frame=0xfffffe01439383f0, eva=0xf) at ../../../amd64/amd64/trap.c:946 #19 0xffffffff810e0b3b in trap_pfault (frame=0xfffffe01439383f0, usermode=0x0, signo=, ucode=) at ../../../amd64/amd64/trap.c:765 #20 #21 0xffffffff80f2023c in ffs_snapblkfree (fs=0xfffffe014fe77000, devvp=0xfffffe01504d7c20, bno=0x610ac, size=0x1000, inum=0x3024d, vtype=VDIR, wkhd=0xfffffe0143938648) at ../../../ufs/ffs/ffs_snapshot.c:1855 #22 0xffffffff80f0daa2 in ffs_blkfree (ump=ump@entry=0xfffff8004d473200, fs=fs@entry=0xfffffe014fe77000, devvp=0xfffffe01504d7c20, bno=0x610ac, size=size@entry=0x1000, inum=0x3024d, vtype=VDIR, dephd=0xfffffe0143938648, key=0x2) at ../../../ufs/ffs/ffs_alloc.c:2661 #23 0xffffffff80f3dcf4 in freework_freeblock (freework=, freework@entry=0xfffff8064e49ea00, key=key@entry=0x2) at ../../../ufs/ffs/ffs_softdep.c:8216 #24 0xffffffff80f2fc7b in handle_workitem_freeblocks (freeblks=freeblks@entry=0xfffff806d07e7300, flags=flags@entry=0x0) at ../../../ufs/ffs/ffs_softdep.c:8372 #25 0xffffffff80f30ccc in softdep_setup_freeblocks (ip=0xfffffe015258d910, length=, flags=0x800) at ../../../ufs/ffs/ffs_softdep.c:7415 #26 0xffffffff80f17b9e in ffs_truncate (vp=0xfffffe01522ea770, length=0x0, flags=0x800, cred=0x0) at ../../../ufs/ffs/ffs_inode.c:431 #27 0xffffffff80f533ea in ufs_inactive (ap=) at ../../../ufs/ufs/ufs_inode.c:168 #28 0xffffffff811ad2e9 in VOP_INACTIVE_APV (vop=0xffffffff81b66720 , a=a@entry=0xfffffe0143938c00) at vnode_if.c:2058 #29 0xffffffff80cfb0ca in VOP_INACTIVE (vp=0xfffffe01522ea770) at ./vnode_if.h:1035 #30 vinactivef (vp=0xfffffe01522ea770, vp@entry=0x1) at ../../../kern/vfs_subr.c:3663 #31 0xffffffff80cfafef in vinactive (vp=vp@entry=0xfffffe01522ea770) at ../../../kern/vfs_subr.c:3687 #32 0xffffffff80cfa71a in vput_final (vp=0xfffffe01522ea770, vp@entry=0xfffffe0143938cc8, func=, func@entry=VPUT) at ../../../kern/vfs_subr.c:3270 #33 0xffffffff80cfa840 in vput (vp=vp@entry=0xfffffe01522ea770) at ../../../kern/vfs_subr.c:3327 #34 0xffffffff80d0eb50 in kern_frmdirat (td=, dfd=0xffffff9c, path=0x2e1d642c76f0 , fd=, pathseg=UIO_USERSPACE, flag=) at ../../../kern/vfs_syscalls.c:3961 #35 0xffffffff810e1455 in syscallenter (td=0xfffffe01437a0020) at ../../../amd64/amd64/../../kern/subr_syscall.c:189 #36 amd64_syscall (td=0xfffffe01437a0020, traced=0x0) at ../../../amd64/amd64/trap.c:1191 #37 #38 0x00002e1d65c45e1a in ?? () Backtrace stopped: Cannot access memory at address 0x2e1d642c72a8 (kgdb) f 21 #21 0xffffffff80f2023c in ffs_snapblkfree (fs=0xfffffe014fe77000, devvp=0xfffffe01504d7c20, bno=0x610ac, size=0x1000, inum=0x3024d, vtype=VDIR, wkhd=0xfffffe0143938648) at ../../../ufs/ffs/ffs_snapshot.c:1855 1855 vp = ITOV(ip); (kgdb) l 1850 } 1851 if (lockmgr(&sn->sn_lock, LK_INTERLOCK | LK_EXCLUSIVE | LK_SLEEPFAIL, 1852 VI_MTX(devvp)) != 0) 1853 goto retry; 1854 TAILQ_FOREACH(ip, &sn->sn_head, i_nextsnap) { 1855 vp = ITOV(ip); 1856 if (DOINGSOFTDEP(vp)) 1857 softdep_prealloc(vp, MNT_WAIT); 1858 /* 1859 * Lookup block being written. (kgdb) info loc ibp = 0xfffffe003c210b00 cbp = 0xfffffe003b5acf80 savedcbp = 0xfffffe003b5acf80 td = 0xfffffe01437a0020 ip = 0xffffffffffffffff vp = 0xfffffe015022a960 lbn = 0xc215 blkno = 0x0 indiroff = 0x209 error = 0x0 claimedblk = 0x1 sn = 0xfffff800040d8e00 (kgdb) p *sn $1 = {sn_link = {le_next = 0xfffff8004dcea500, le_prev = 0xffffffff81fb44a8 }, sn_head = {tqh_first = 0x0, tqh_last = 0xfffff800040d8e10}, sn_listsize = 0x0, sn_blklist = 0x0, sn_lock = {lock_object = {lo_name = 0xffffffff81257e5a "snaplk", lo_flags = 0x67b0008, lo_data = 0x0, lo_witness = 0xfffff8087fd84c80}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_pri = 0x60, lk_timo = 0x33, lk_stack = {depth = 0x12, pcs = { 0xffffffff80bd54db, 0xffffffff80f1e7e5, 0xffffffff80f20048, 0xffffffff80f1f677, 0xffffffff80f17702, 0xffffffff80f533ea, 0xffffffff811ad2e9, 0xffffffff80cfb0ca, 0xffffffff80cfa71a, 0xffffffff80f1eff3, 0xffffffff80f14e9d, 0xffffffff80f2032f, 0xffffffff80f0daa2, 0xffffffff80f3dcf4, 0xffffffff80f2fc7b, 0xffffffff80f30ccc, 0xffffffff80f17b9e, 0xffffffff80f533ea}}}} (kgdb) p *vp $2 = {v_type = VREG, v_irflag = 0x2, v_seqc = 0x3c68, v_nchash = 0x53dd67b, v_hash = 0x5a, v_op = 0xffffffff81b66720 , v_data = 0xfffffe01503a4488, v_mount = 0xfffffe014289b280, v_nmntvnodes = {tqe_next = 0xfffff80513b65c00, tqe_prev = 0xfffffe01525e1730}, { v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0}, v_hashlist = {le_next = 0xfffffe0148b4c770, le_prev = 0xfffffe010c8b3400}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffffe015022a9b8}, v_cache_dd = 0x0, v_lock = {lock_object = { lo_name = 0xffffffff811dd38f "ufs", lo_flags = 0x6fb0000, lo_data = 0x0, lo_witness = 0xfffff8087fd84000}, lk_lock = 0xfffffe01437a0024, lk_exslpfail = 0x1, lk_pri = 0x60, lk_timo = 0x33, lk_stack = {depth = 0x12, pcs = {0xffffffff80bd49cc, 0xffffffff80bd4e99, 0xffffffff80f1e7e5, 0xffffffff80f1fe26, 0xffffffff80f1f66e, 0xffffffff80f17702, 0xffffffff80f533ea, 0xffffffff811ad2e9, 0xffffffff80cfb0ca, 0xffffffff80cfa71a, 0xffffffff80f1eff3, 0xffffffff80f14e9d, 0xffffffff80f2032f, 0xffffffff80f0daa2, 0xffffffff80f3dcf4, 0xffffffff80f2fc7b, 0xffffffff80f30ccc, 0xffffffff80f17b9e}}}, v_interlock = {lock_object = {lo_name = 0xffffffff8132e0bd "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd75500}, mtx_lock = 0x0}, v_vnlock = 0xfffffe015022a9d0, v_vnodelist = {tqe_next = 0xfffffe01524fe518, tqe_prev = 0xfffffe01522d3f68}, v_lazylist = {tqe_next = 0xfffffe0151739000, tqe_prev = 0xfffffe014fa23618}, v_bufobj = {bo_lock = {lock_object = { lo_name = 0xffffffff812e5fd6 "bufobj interlock", lo_flags = 0x5230000, lo_data = 0x0, lo_witness = 0xfffff8087fd80800}, rw_lock = 0x1}, bo_ops = 0xffffffff81b25f10 , bo_object = 0xfffff80599f56108, bo_synclist = {le_next = 0xfffffe0151a741e0, le_prev = 0xfffff8002d261348}, bo_private = 0xfffffe015022a960, bo_clean = {bv_hd = {tqh_first = 0xfffffe003b5acf80, tqh_last = 0xfffffe003b5acfe0}, bv_root = {pt_root = 0xfffffe003b5ad0e1}, bv_cnt = 0x1}, bo_dirty = {bv_hd = {tqh_first = 0xfffffe003c210b00, tqh_last = 0xfffffe003b9d0fe0}, bv_root = {pt_root = 0xfffff8002d27e900}, bv_cnt = 0x2}, bo_numoutput = 0x0, bo_flag = 0x1, bo_domain = 0x5, bo_bsize = 0x8000}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffffe015022ab88}, rl_currdep = 0x0}, v_holdcnt = 0x5, v_usecount = 0x0, v_iflag = 0x0, v_vflag = 0x80, v_mflag = 0x1, v_dbatchcpu = 0x1, v_writecount = 0x0, v_seqc_users = 0x0} (kgdb) diff --git a/sys/ufs/ffs/ffs_snapshot.c b/sys/ufs/ffs/ffs_snapshot.c index 1bc360b580c..8365773dc8c 100644 --- a/sys/ufs/ffs/ffs_snapshot.c +++ b/sys/ufs/ffs/ffs_snapshot.c @@ -181,6 +181,7 @@ static int mapacct_ufs2(struct vnode *, ufs2_daddr_t *, ufs2_daddr_t *, struct fs *, ufs_lbn_t, int); static int readblock(struct vnode *vp, struct buf *, ufs2_daddr_t); static void try_free_snapdata(struct vnode *devvp); +static void revert_snaplock(struct vnode *, struct vnode *, struct snapdata *); static struct snapdata *ffs_snapdata_acquire(struct vnode *devvp); static int ffs_bp_snapblk(struct vnode *, struct buf *); @@ -1699,7 +1700,7 @@ ffs_snapremove(vp) struct buf *ibp; struct fs *fs; ufs2_daddr_t numblks, blkno, dblk; - int error, i, last, loc; + int error, last, loc; struct snapdata *sn; ip = VTOI(vp); @@ -1717,20 +1718,10 @@ ffs_snapremove(vp) sn = devvp->v_rdev->si_snapdata; TAILQ_REMOVE(&sn->sn_head, ip, i_nextsnap); ip->i_nextsnap.tqe_prev = 0; - VI_UNLOCK(devvp); - lockmgr(&vp->v_lock, LK_EXCLUSIVE, NULL); - for (i = 0; i < sn->sn_lock.lk_recurse; i++) - lockmgr(&vp->v_lock, LK_EXCLUSIVE, NULL); - KASSERT(vp->v_vnlock == &sn->sn_lock, - ("ffs_snapremove: lost lock mutation")); - vp->v_vnlock = &vp->v_lock; - VI_LOCK(devvp); - while (sn->sn_lock.lk_recurse > 0) - lockmgr(&sn->sn_lock, LK_RELEASE, NULL); - lockmgr(&sn->sn_lock, LK_RELEASE, NULL); + revert_snaplock(vp, devvp, sn); try_free_snapdata(devvp); - } else - VI_UNLOCK(devvp); + } + VI_UNLOCK(devvp); /* * Clear all BLK_NOCOPY fields. Pass any block claims to other * snapshots that want them (see ffs_snapblkfree below). @@ -2200,27 +2191,18 @@ ffs_snapshot_unmount(mp) xp->i_nextsnap.tqe_prev = 0; lockmgr(&sn->sn_lock, LK_INTERLOCK | LK_EXCLUSIVE, VI_MTX(devvp)); - /* - * Avoid LOR with above snapshot lock. The LK_NOWAIT should - * never fail as the lock is currently unused. Rather than - * panic, we recover by doing the blocking lock. - */ - if (lockmgr(&vp->v_lock, LK_EXCLUSIVE | LK_NOWAIT, NULL) != 0) { - printf("ffs_snapshot_unmount: Unexpected LK_NOWAIT " - "failure\n"); - lockmgr(&vp->v_lock, LK_EXCLUSIVE, NULL); - } - KASSERT(vp->v_vnlock == &sn->sn_lock, - ("ffs_snapshot_unmount: lost lock mutation")); - vp->v_vnlock = &vp->v_lock; + VI_LOCK(devvp); + revert_snaplock(vp, devvp, sn); lockmgr(&vp->v_lock, LK_RELEASE, NULL); - lockmgr(&sn->sn_lock, LK_RELEASE, NULL); - if (xp->i_effnlink > 0) + if (xp->i_effnlink > 0) { + VI_UNLOCK(devvp); vrele(vp); - VI_LOCK(devvp); + VI_LOCK(devvp); + } sn = devvp->v_rdev->si_snapdata; } try_free_snapdata(devvp); + VI_UNLOCK(devvp); } /* @@ -2724,10 +2706,8 @@ try_free_snapdata(struct vnode *devvp) sn = devvp->v_rdev->si_snapdata; if (sn == NULL || TAILQ_FIRST(&sn->sn_head) != NULL || - (devvp->v_vflag & VV_COPYONWRITE) == 0) { - VI_UNLOCK(devvp); + (devvp->v_vflag & VV_COPYONWRITE) == 0) return; - } devvp->v_rdev->si_snapdata = NULL; devvp->v_vflag &= ~VV_COPYONWRITE; @@ -2739,6 +2719,46 @@ try_free_snapdata(struct vnode *devvp) if (snapblklist != NULL) free(snapblklist, M_UFSMNT); ffs_snapdata_free(sn); + VI_LOCK(devvp); +} + +/* + * Revert a vnode lock from using the snapshot lock back to its own lock. + * + * Aquire a lock on the vnode's own lock and release the lock on the + * snapshot lock. If there are any recursions on the snapshot lock + * get the same number of recursions on the vnode's own lock. + */ +static void +revert_snaplock(vp, devvp, sn) + struct vnode *vp; + struct vnode *devvp; + struct snapdata *sn; +{ + int i; + + ASSERT_VI_LOCKED(devvp, "revert_snaplock"); + /* + * Avoid LOR with snapshot lock. The LK_NOWAIT should + * never fail as the lock is currently unused. Rather than + * panic, we recover by doing the blocking lock. + */ + for (i = 0; i <= sn->sn_lock.lk_recurse; i++) { + if (lockmgr(&vp->v_lock, LK_EXCLUSIVE | LK_NOWAIT | + LK_INTERLOCK, VI_MTX(devvp)) != 0) { + printf("revert_snaplock: Unexpected LK_NOWAIT " + "failure\n"); + lockmgr(&vp->v_lock, LK_EXCLUSIVE | LK_INTERLOCK, + VI_MTX(devvp)); + } + VI_LOCK(devvp); + } + KASSERT(vp->v_vnlock == &sn->sn_lock, + ("revert_snaplock: lost lock mutation")); + vp->v_vnlock = &vp->v_lock; + while (sn->sn_lock.lk_recurse > 0) + lockmgr(&sn->sn_lock, LK_RELEASE, NULL); + lockmgr(&sn->sn_lock, LK_RELEASE, NULL); } static struct snapdata * -- Test scenario: misc/snap11.sh