vmcore.818 KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2024 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-CURRENT #0 main-n270716-f65d0b18d937: Tue Jun 11 09:49:59 CEST 2024 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.20-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33171128320 (31634 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 6 4 10 11 5 8 3 9 2 7 random: entropy device external interface kbd1 at kbdmux0 vtvga0: smbios0: at iomem 0xf05e0-0xf05fe smbios0: Version: 3.0, BCD Revision: 3.0 aesni0: acpi0: <> acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb416000-0xfb4167ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb412000-0xfb4123ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: EEPROM V1.63-0 eTrack 0x800009fa igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: EEPROM V1.63-0 eTrack 0x800009fa igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb411000-0xfb4113ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb410000-0xfb4107ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 apei0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounter "TSC-low" frequency 1749998366 Hz quality 1000 Timecounters tick every 1.000 msec Expensive callout(9) function: 0xffffffff804df950(0xfffffe010a7d1000) 0.004265063 s ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ses0: ada0,pass0 in 'Slot 01', SATA Slot: scbus1 target 0 Expensive callout(9) function: 0xffffffff8098ae20(0xffffffff818a8de8) 0.007386581 s mlx5_core0: mem 0xf8000000-0xf9ffffff irq 32 at device 0.0 on pci3 mlx5: Mellanox Core driver 3.7.1 (November 2021)ugen1.1: at usbus1 ugen0.1: at usbus0 ugen2.1: at usbus2 uhub0 on usbus1 uhub0: on usbus1 uhub1 on usbus0 uhub1: on usbus0 uhub2 on usbus2 uhub2: on usbus2 mlx5_core0: WARN: mlx5_vsc_set_space:125:(pid 0): Space 0x7 is not supported. mlx5_core0: WARN: mlx5_fwdump_prep:99:(pid 0): VSC scan space is not supported mlx5_core0: INFO: init_one:1708:(pid 0): cannot find SR-IOV PCIe cap mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 1, l2 table size(65536), per vport: max uc(1024) max mc(16384) mlx5_core0: Failed to initialize SR-IOV support, erroSetting hostuuid: 00000000-0000-0000-0000-002590c99976. Setting hostid: 0x90b83241. Starting file system checks: WARNING: / was not properly dismounted Mounting local filesystems:. Autoloading module: acpi_wmi Autoloading module: ichsmb Autoloading module: ioat ioat0: mem 0x383ffff1c000-0x383ffff1ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0x383ffff18000-0x383ffff1bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0x383ffff14000-0x383ffff17fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0x383ffff10000-0x383ffff13fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0x383ffff0c000-0x383ffff0ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0x383ffff08000-0x383ffff0bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0x383ffff04000-0x383ffff07fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0x383ffff00000-0x383ffff03fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 ichsmb0: port 0x580-0x59f mem 0x383ffff21000-0x383ffff210ff irq 18 at device 31.3 on pci1 smbus0: on ichsmb0 acpi_wmi0: on acpi0 acpi_wmi0: cannot find EC device fsck2+ fsck -fy /tmp ** /dev/gpt/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 1121 files, 64424 used, 1200983 free (135 frags, 150106 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** + set +x Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/e2fsprogs /usr/local/lib/perl5/5.36/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: mercat1.netperf.freebsd.org. Setting up harvesting: PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . igb0: link state changed to UP debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80005010800), ignoring. lo0: link state changed to UP Autoloading module: mlx5 Autoloading module: mlx5 Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Starting Network: mce0. mce0: flags=8802 metric 0 mtu 1500 options=7eef07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 Starting Network: mce1. mce1: flags=8802 metric 0 mtu 1500 options=7eef07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid Autoloading module: ums Autoloading module: usbhid ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192.168.5.1 bound to 192.168.5.171 -- renewal in 300 seconds. route: message indicates error: File exists add host 127.0.0.1: gateway lo0 fib 0: route already in table route: message indicates error: File exists add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. Starting watchdogd. Setting date via ntp. 11 Jun 12:53:43 ntpdate[3473]: step time server 192.168.5.1 offset -1.025576 sec Clearing /tmp (X related). Recovering vi editor sessions:. Updating motd:. Updating /var/run/os-release done. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Mounting late filesystems:. Starting powerd. Starting inetd. Starting cron. Performing sanity check on sshd configuration. Starting sshd. Tue Jun 11 12:53 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Jun 11 12:53:54 mercat1 su[3661]: pho to root on /dev/pts/0 20240611 12:54:19 all (1/18): unionfs.sh root Jun 11 12:54:46 mercat1 login[3633]: ROOT LOGIN (root) ON ttyu1 Last login: Sat May 25 09:42:52 on ttyu1 FreeBSD 15.0-CURRENT #0 main-n270716-f65d0b18d937: Tue Jun 11 09:49:59 CEST 2024 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO Welcome to FreeBSD! Release Notes, Errata: https://www.FreeBSD.org/releases/ Security Advisories: https://www.FreeBSD.org/security/ FreeBSD Handbook: https://www.FreeBSD.org/handbook/ FreeBSD FAQ: https://www.FreeBSD.org/faq/ Questions List: https://www.FreeBSD.org/lists/questions/ FreeBSD Forums: https://forums.FreeBSD.org/ Documents installed with the system are in the /usr/local/share/doc/freebsd/ directory, or can be installed later with: pkg install en-freebsd-doc For other languages, replace "en" with a language code like de or fr. Show the version of FreeBSD installed: freebsd-version ; uname -a Please include that output and any error messages when posting questions. Introduction to manual pages: man man FreeBSD directory layout: man hier To change this login announcement, see motd(5). You have new mail. root@mercat1:~ # ls -l /boot/kernel/union* -r--r--r-- 1 root wheel 130968 Jun 11 12:50 /boot/kernel/unionfs.ko root@mercat1:~ # 20240611 12:56:24 all (2/18): unionfs10.sh 20240611 12:56:27 all (3/18): unionfs11.sh 20240611 12:56:31 all (4/18): unionfs12.sh 20240611 12:56:34 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe0173256000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424d400 hold count flags () flags () v_object 0xfffff805380b0420 ref 0 pages 4159 cleanbuf 515 dirtybuf 7 lock type mntfs: EXCL by thread 0xfffff8004ee05740 (pid 90735, mkdir, tid 100425) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0173256000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424d400 hold count flags () flags () v_object 0xfffff805380b0420 ref 0 pages 4159 cleanbuf 515 dirtybuf 7 lock type mntfs: EXCL by thread 0xfffff8004ee05740 (pid 90735, mkdir, tid 100425) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 12:57:21 all (6/18): unionfs14.sh 20240611 13:02:26 all (7/18): unionfs15.sh 20240611 13:02:34 all (8/18): unionfs16.sh 20240611 13:05:13 all (9/18): unionfs17.sh 20240611 13:11:34 all (10/18): unionfs18.sh 20240611 13:11:37 all (11/18): unionfs2.sh stress2: freework leak: 687/693. stress2: newblk leak: 156102/156108. 20240611 13:22:53 all (12/18): unionfs3.sh 20240611 13:22:58 all (13/18): unionfs4.sh 20240611 13:25:08 all (14/18): unionfs5.sh 20240611 13:27:12 all (15/18): unionfs6.sh 20240611 13:34:12 all (16/18): unionfs7.sh 20240611 13:40:35 all (17/18): unionfs8.sh 20240611 13:46:45 all (18/18): unionfs9.sh 20240611 13:51:51 all (1/18): unionfs.sh 20240611 13:53:57 all (2/18): unionfs10.sh 20240611 13:54:00 all (3/18): unionfs11.sh 20240611 13:54:04 all (4/18): unionfs12.sh 20240611 13:54:07 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe017cc40960: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8005a54f000 hold count flags () flags () v_object 0xfffff8053872ed68 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff805491bb740 (pid 21751, mkdir, tid 100626) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017cc40960: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8005a54f000 hold count flags () flags () v_object 0xfffff8053872ed68 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff805491bb740 (pid 21751, mkdir, tid 100626) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 13:54:52 all (6/18): unionfs14.sh 20240611 13:59:57 all (7/18): unionfs15.sh 20240611 14:00:05 all (8/18): unionfs16.sh 20240611 14:02:21 all (9/18): unionfs17.sh 20240611 14:08:36 all (10/18): unionfs18.sh 20240611 14:08:39 all (11/18): unionfs2.sh stress2: freework leak: 300/306. stress2: newblk leak: 163861/163866. 20240611 14:18:44 all (12/18): unionfs3.sh 20240611 14:18:54 all (13/18): unionfs4.sh 20240611 14:21:14 all (14/18): unionfs5.sh fsync: giving up on dirty (error = 35) 0xfffffe0184315c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff8035b443800 hold count flags () flags () v_object 0xfffff8053da21000 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff805a1466000 (pid 73975, mkdir, tid 100723) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0184315c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff8035b443800 hold count flags () flags () v_object 0xfffff8053da21000 ref 0 pages 4251 cleanbuf 515 dirtybuf 18 lock type mntfs: EXCL by thread 0xfffff805a1466000 (pid 73975, mkdir, tid 100723) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0184315c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff8035b443800 hold count flags () flags () v_object 0xfffff8053da21000 ref 0 pages 4251 cleanbuf 515 dirtybuf 18 lock type mntfs: EXCL by thread 0xfffff805a1466000 (pid 73975, mkdir, tid 100723) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0184315c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff8035b443800 hold count flags () flags () v_object 0xfffff8053da21000 ref 0 pages 4251 cleanbuf 516 dirtybuf 17 lock type mntfs: EXCL by thread 0xfffff80579742740 (pid 73970, mkdir, tid 100761) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0184315c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff8035b443800 hold count flags () flags () v_object 0xfffff8053da21000 ref 0 pages 4251 cleanbuf 515 dirtybuf 18 lock type mntfs: EXCL by thread 0xfffff80579742740 (pid 73970, mkdir, tid 100761) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 14:23:17 all (15/18): unionfs6.sh 20240611 14:29:40 all (16/18): unionfs7.sh 20240611 14:34:48 all (17/18): unionfs8.sh Jun 11 14:39:48 mercat1 kernel: pid 5542 (swap), jid 0, uid 2007, was killed: failed to reclaim memory 20240611 14:41:07 all (18/18): unionfs9.sh 20240611 14:46:10 all (1/18): unionfs.sh 20240611 14:48:14 all (2/18): unionfs10.sh 20240611 14:48:17 all (3/18): unionfs11.sh 20240611 14:48:21 all (4/18): unionfs12.sh 20240611 14:48:24 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe017d735c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80121921800 hold count flags () flags () v_object 0xfffff8052f758840 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8005a374000 (pid 56943, mkdir, tid 100536) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017d735c20: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80121921800 hold count flags () flags () v_object 0xfffff8052f758840 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8005a374000 (pid 56943, mkdir, tid 100536) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 14:49:06 all (6/18): unionfs14.sh 20240611 14:54:12 all (7/18): unionfs15.sh 20240611 14:54:20 all (8/18): unionfs16.sh 20240611 14:56:43 all (9/18): unionfs17.sh 20240611 15:03:06 all (10/18): unionfs18.sh 20240611 15:03:09 all (11/18): unionfs2.sh stress2: freework leak: 144/150. stress2: newblk leak: 32854/32859. 20240611 15:13:45 all (12/18): unionfs3.sh 20240611 15:13:57 all (13/18): unionfs4.sh 20240611 15:16:13 all (14/18): unionfs5.sh 20240611 15:18:18 all (15/18): unionfs6.sh Jun 11 15:21:54 mercat1 kernel: pid 68006 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 15:25:24 all (16/18): unionfs7.sh Jun 11 15:30:51 mercat1 kernel: pid 24223 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 15:31:44 all (17/18): unionfs8.sh 20240611 15:38:03 all (18/18): unionfs9.sh 20240611 15:43:07 all (1/18): unionfs.sh 20240611 15:45:10 all (2/18): unionfs10.sh 20240611 15:45:14 all (3/18): unionfs11.sh 20240611 15:45:17 all (4/18): unionfs12.sh 20240611 15:45:21 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe01719bb518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff802561f3800 hold count flags () flags () v_object 0xfffff8064f6dfa50 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff80551acf000 (pid 20067, mkdir, tid 100541) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01719bb518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff802561f3800 hold count flags () flags () v_object 0xfffff8064f6dfa50 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff80551acf000 (pid 20067, mkdir, tid 100541) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 15:46:04 all (6/18): unionfs14.sh 20240611 15:51:19 all (7/18): unionfs15.sh 20240611 15:51:27 all (8/18): unionfs16.sh stress2: newblk leak: 510/517. 20240611 15:53:41 all (9/18): unionfs17.sh 20240611 15:59:57 all (10/18): unionfs18.sh 20240611 16:00:00 all (11/18): unionfs2.sh stress2: freework leak: 240/246. stress2: newblk leak: 131117/131122. 20240611 16:11:28 all (12/18): unionfs3.sh 20240611 16:11:31 all (13/18): unionfs4.sh 20240611 16:13:39 all (14/18): unionfs5.sh fsync: giving up on dirty (error = 35) 0xfffffe01817b4708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80708e73000 hold count flags () flags () v_object 0xfffff805d41b6420 ref 0 pages 4251 cleanbuf 520 dirtybuf 13 lock type mntfs: EXCL by thread 0xfffff80109098740 (pid 45097, mkdir, tid 100381) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01817b4708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80708e73000 hold count flags () flags () v_object 0xfffff805d41b6420 ref 0 pages 4251 cleanbuf 516 dirtybuf 17 lock type mntfs: EXCL by thread 0xfffff80109098740 (pid 45097, mkdir, tid 100381) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01817b4708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80708e73000 hold count flags () flags () v_object 0xfffff805d41b6420 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff80551371740 (pid 45099, mkdir, tid 100770) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01817b4708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80708e73000 hold count flags () flags () v_object 0xfffff805d41b6420 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff80551371740 (pid 45099, mkdir, tid 100770) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 16:15:42 all (15/18): unionfs6.sh 20240611 16:22:21 all (16/18): unionfs7.sh 20240611 16:28:40 all (17/18): unionfs8.sh 20240611 16:35:05 all (18/18): unionfs9.sh 20240611 16:40:10 all (1/18): unionfs.sh 20240611 16:42:14 all (2/18): unionfs10.sh 20240611 16:42:17 all (3/18): unionfs11.sh 20240611 16:42:21 all (4/18): unionfs12.sh 20240611 16:42:24 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff803b884b740 (pid 3061, mkdir, tid 100399) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80005024740 (pid 3060, mkdir, tid 100253) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8004e55c740 (pid 3066, mkdir, tid 100227) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8005a56b740 (pid 3482, mkdir, tid 100926) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80109412000 (pid 3071, mkdir, tid 100417) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff800043ff000 (pid 3062, mkdir, tid 100213) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8053da30000 (pid 3480, mkdir, tid 100553) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff803b884b740 (pid 3061, mkdir, tid 100399) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80005024740 (pid 3060, mkdir, tid 100253) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8004e55c740 (pid 3066, mkdir, tid 100227) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 519 dirtybuf 3 lock type mntfs: EXCL by thread 0xfffff800043ff000 (pid 3062, mkdir, tid 100213) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8053da30000 (pid 3480, mkdir, tid 100553) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8055d0c3740 (pid 3064, mkdir, tid 100637) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8055d0d2740 (pid 3481, mkdir, tid 100785) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80109412000 (pid 3071, mkdir, tid 100417) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8055d0c3740 (pid 3064, mkdir, tid 100637) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8005a56b740 (pid 3482, mkdir, tid 100926) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8004ee09740 (pid 3058, mkdir, tid 100260) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8004ee09740 (pid 3058, mkdir, tid 100260) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8055d0d2740 (pid 3481, mkdir, tid 100785) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80547611740 (pid 3478, mkdir, tid 100629) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018c4fb068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80126f59800 hold count flags () flags () v_object 0xfffff803b85f9528 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80547611740 (pid 3478, mkdir, tid 100629) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 16:44:44 all (6/18): unionfs14.sh 20240611 16:49:47 all (7/18): unionfs15.sh 20240611 16:49:55 all (8/18): unionfs16.sh stress2: newblk leak: 408/415. stress2: mount leak: 13/44 stress2: GEOM leak: 11/85 20240611 16:52:08 all (9/18): unionfs17.sh 20240611 16:58:30 all (10/18): unionfs18.sh 20240611 16:58:33 all (11/18): unionfs2.sh stress2: freework leak: 140/146. stress2: newblk leak: 49215/49220. 20240611 17:09:40 all (12/18): unionfs3.sh 20240611 17:09:51 all (13/18): unionfs4.sh 20240611 17:12:14 all (14/18): unionfs5.sh 20240611 17:15:00 all (15/18): unionfs6.sh Jun 11 17:17:50 mercat1 kernel: pid 54726 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 17:21:29 all (16/18): unionfs7.sh 20240611 17:27:52 all (17/18): unionfs8.sh Jun 11 17:34:07 mercat1 kernel: pid 22736 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 17:34:32 all (18/18): unionfs9.sh Jun 11 17:35:47 mercat1 kernel: pid 28504 (swap), jid 0, uid 0, was killed: a thread waited too long to allocate a page 20240611 17:39:38 all (1/18): unionfs.sh 20240611 17:41:43 all (2/18): unionfs10.sh 20240611 17:41:46 all (3/18): unionfs11.sh 20240611 17:41:50 all (4/18): unionfs12.sh 20240611 17:41:53 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe017229d2c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff801bea86c00 hold count flags () flags () v_object 0xfffff805e882d318 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff80003e6d740 (pid 50340, mkdir, tid 100424) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017229d2c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff801bea86c00 hold count flags () flags () v_object 0xfffff805e882d318 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff80003e6d740 (pid 50340, mkdir, tid 100424) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 17:42:37 all (6/18): unionfs14.sh 20240611 17:47:42 all (7/18): unionfs15.sh 20240611 17:47:50 all (8/18): unionfs16.sh stress2: newblk leak: 315/322. 20240611 17:50:02 all (9/18): unionfs17.sh 20240611 17:56:22 all (10/18): unionfs18.sh 20240611 17:56:25 all (11/18): unionfs2.sh stress2: freework leak: 105/111. stress2: newblk leak: 49144/49149. 20240611 18:08:56 all (12/18): unionfs3.sh 20240611 18:09:03 all (13/18): unionfs4.sh 20240611 18:11:13 all (14/18): unionfs5.sh 20240611 18:13:55 all (15/18): unionfs6.sh swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio 20240611 18:21:03 all (16/18): unionfs7.sh Jun 11 18:24:10 mercat1 kernel: pid 30222 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page Jun 11 18:25:59 mercat1 kernel: pid 35347 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 18:27:22 all (17/18): unionfs8.sh 20240611 18:32:45 all (18/18): unionfs9.sh 20240611 18:37:51 all (1/18): unionfs.sh 20240611 18:39:55 all (2/18): unionfs10.sh 20240611 18:39:58 all (3/18): unionfs11.sh 20240611 18:40:02 all (4/18): unionfs12.sh 20240611 18:40:05 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe019a5eae10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80130ae0000 hold count flags () flags () v_object 0xfffff807f070e000 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff80271a12000 (pid 56787, mkdir, tid 100641) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe019a5eae10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80130ae0000 hold count flags () flags () v_object 0xfffff807f070e000 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff80271a12000 (pid 56787, mkdir, tid 100641) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 18:40:48 all (6/18): unionfs14.sh 20240611 18:45:52 all (7/18): unionfs15.sh 20240611 18:46:00 all (8/18): unionfs16.sh 20240611 18:48:28 all (9/18): unionfs17.sh Jun 11 18:49:37 mercat1 kernel: pid 99157 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 18:54:58 all (10/18): unionfs18.sh 20240611 18:55:01 all (11/18): unionfs2.sh stress2: newblk leak: 32778/32782. 20240611 19:05:46 all (12/18): unionfs3.sh 20240611 19:05:49 all (13/18): unionfs4.sh 20240611 19:08:42 all (14/18): unionfs5.sh 20240611 19:11:25 all (15/18): unionfs6.sh 20240611 19:18:19 all (16/18): unionfs7.sh Jun 11 19:20:12 mercat1 kernel: pid 98869 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 19:25:38 all (17/18): unionfs8.sh 20240611 19:32:14 all (18/18): unionfs9.sh 20240611 19:37:19 all (1/18): unionfs.sh 20240611 19:39:24 all (2/18): unionfs10.sh 20240611 19:39:27 all (3/18): unionfs11.sh 20240611 19:39:31 all (4/18): unionfs12.sh 20240611 19:39:34 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8034a00f740 (pid 15620, mkdir, tid 100369) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8034a534000 (pid 15609, mkdir, tid 100402) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff80109568740 (pid 15614, mkdir, tid 100389) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8058c9fb000 (pid 15610, mkdir, tid 100505) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8054ece8740 (pid 16025, mkdir, tid 100447) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8034a06d740 (pid 16022, mkdir, tid 100397) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176292000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff806d624a800 hold count flags () flags () v_object 0xfffff80003ffa840 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8034a3e9740 (pid 15615, mkdir, tid 100405) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 19:41:03 all (6/18): unionfs14.sh 20240611 19:46:08 all (7/18): unionfs15.sh 20240611 19:46:16 all (8/18): unionfs16.sh Jun 11 19:48:06 mercat1 kernel: pid 84245 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 19:48:45 all (9/18): unionfs17.sh 20240611 19:55:19 all (10/18): unionfs18.sh 20240611 19:55:22 all (11/18): unionfs2.sh stress2: freework leak: 4073/4079. stress2: newblk leak: 20414/20418. 20240611 20:06:23 all (12/18): unionfs3.sh 20240611 20:06:26 all (13/18): unionfs4.sh Jun 11 20:07:42 mercat1 kernel: pid 94099 (swap), jid 0, uid 0, was killed: a thread waited too long to allocate a page 20240611 20:08:54 all (14/18): unionfs5.sh 20240611 20:11:17 all (15/18): unionfs6.sh Jun 11 20:12:24 mercat1 kernel: pid 51558 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 20:18:02 all (16/18): unionfs7.sh 20240611 20:24:50 all (17/18): unionfs8.sh 20240611 20:30:13 all (18/18): unionfs9.sh 20240611 20:35:18 all (1/18): unionfs.sh 20240611 20:37:24 all (2/18): unionfs10.sh 20240611 20:37:27 all (3/18): unionfs11.sh 20240611 20:37:31 all (4/18): unionfs12.sh 20240611 20:37:34 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe01725bf518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff803d9215c00 hold count flags () flags () v_object 0xfffff807f0db1840 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8004e55c740 (pid 31265, mkdir, tid 100227) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01725bf518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff803d9215c00 hold count flags () flags () v_object 0xfffff807f0db1840 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8004e55c740 (pid 31265, mkdir, tid 100227) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 20:38:17 all (6/18): unionfs14.sh 20240611 20:43:22 all (7/18): unionfs15.sh 20240611 20:43:30 all (8/18): unionfs16.sh 20240611 20:45:56 all (9/18): unionfs17.sh 20240611 20:52:11 all (10/18): unionfs18.sh 20240611 20:52:14 all (11/18): unionfs2.sh stress2: freework leak: 165/171. stress2: newblk leak: 90142/90147. 20240611 21:03:30 all (12/18): unionfs3.sh 20240611 21:03:33 all (13/18): unionfs4.sh 20240611 21:05:41 all (14/18): unionfs5.sh fsync: giving up on dirty (error = 35) 0xfffffe017e8432c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80433f95400 hold count flags () flags () v_object 0xfffff8053231de70 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff8059441b740 (pid 8107, mkdir, tid 100705) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017e8432c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80433f95400 hold count flags () flags () v_object 0xfffff8053231de70 ref 0 pages 4251 cleanbuf 519 dirtybuf 14 lock type mntfs: EXCL by thread 0xfffff8004ebdf740 (pid 8108, mkdir, tid 100282) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017e8432c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80433f95400 hold count flags () flags () v_object 0xfffff8053231de70 ref 0 pages 4251 cleanbuf 523 dirtybuf 10 lock type mntfs: EXCL by thread 0xfffff8004ebdf740 (pid 8108, mkdir, tid 100282) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017e8432c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80433f95400 hold count flags () flags () v_object 0xfffff8053231de70 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff8059441b740 (pid 8107, mkdir, tid 100705) with exclusive waiters pending #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017e8432c0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80433f95400 hold count flags () flags () v_object 0xfffff8053231de70 ref 0 pages 4251 cleanbuf 518 dirtybuf 15 lock type mntfs: EXCL by thread 0xfffff8059441b740 (pid 8107, mkdir, tid 100705) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 21:07:44 all (15/18): unionfs6.sh 20240611 21:14:03 all (16/18): unionfs7.sh Jun 11 21:20:12 mercat1 kernel: pid 64304 (rw), uid 2007 inumber 80389 on /mnt10: filesystem full 20240611 21:20:37 all (17/18): unionfs8.sh 20240611 21:27:20 all (18/18): unionfs9.sh 20240611 21:32:25 all (1/18): unionfs.sh 20240611 21:34:31 all (2/18): unionfs10.sh 20240611 21:34:34 all (3/18): unionfs11.sh 20240611 21:34:38 all (4/18): unionfs12.sh 20240611 21:34:41 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe018b5b4e10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424c400 hold count flags () flags () v_object 0xfffff80587464c60 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8034f19c000 (pid 60676, mkdir, tid 100749) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018b5b4e10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424c400 hold count flags () flags () v_object 0xfffff80587464c60 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8034f19c000 (pid 60676, mkdir, tid 100749) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018b5b4e10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424c400 hold count flags () flags () v_object 0xfffff80587464c60 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8005a057740 (pid 60677, mkdir, tid 100379) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe018b5b4e10: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8000424c400 hold count flags () flags () v_object 0xfffff80587464c60 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8005a057740 (pid 60677, mkdir, tid 100379) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 21:35:38 all (6/18): unionfs14.sh 20240611 21:40:43 all (7/18): unionfs15.sh 20240611 21:40:51 all (8/18): unionfs16.sh 20240611 21:43:06 all (9/18): unionfs17.sh 20240611 21:49:21 all (10/18): unionfs18.sh 20240611 21:49:24 all (11/18): unionfs2.sh stress2: newblk leak: 49168/49172. 20240611 22:00:45 all (12/18): unionfs3.sh 20240611 22:00:49 all (13/18): unionfs4.sh 20240611 22:03:14 all (14/18): unionfs5.sh fsync: giving up on dirty (error = 35) 0xfffffe0176fdd518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff805ae05e400 hold count flags () flags () v_object 0xfffff803b8ef7840 ref 0 pages 4251 cleanbuf 516 dirtybuf 17 lock type mntfs: EXCL by thread 0xfffff801091bc000 (pid 43936, mkdir, tid 100255) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176fdd518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff805ae05e400 hold count flags () flags () v_object 0xfffff803b8ef7840 ref 0 pages 4251 cleanbuf 520 dirtybuf 13 lock type mntfs: EXCL by thread 0xfffff801091bc000 (pid 43936, mkdir, tid 100255) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0176fdd518: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff805ae05e400 hold count flags () flags () v_object 0xfffff803b8ef7840 ref 0 pages 4251 cleanbuf 515 dirtybuf 18 lock type mntfs: EXCL by thread 0xfffff8005a04d000 (pid 43945, mkdir, tid 100325) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 22:05:17 all (15/18): unionfs6.sh 20240611 22:12:03 all (16/18): unionfs7.sh 20240611 22:18:57 all (17/18): unionfs8.sh Jun 11 22:23:10 mercat1 kernel: pid 53570 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 22:25:40 all (18/18): unionfs9.sh 20240611 22:30:53 all (1/18): unionfs.sh 20240611 22:32:56 all (2/18): unionfs10.sh 20240611 22:33:00 all (3/18): unionfs11.sh 20240611 22:33:03 all (4/18): unionfs12.sh 20240611 22:33:07 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe0173847068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff803fc039800 hold count flags () flags () v_object 0xfffff8060f48b000 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8010923d740 (pid 44771, mkdir, tid 100324) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0173847068: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff803fc039800 hold count flags () flags () v_object 0xfffff8060f48b000 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8010923d740 (pid 44771, mkdir, tid 100324) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 22:33:51 all (6/18): unionfs14.sh 20240611 22:38:55 all (7/18): unionfs15.sh 20240611 22:39:03 all (8/18): unionfs16.sh 20240611 22:41:32 all (9/18): unionfs17.sh 20240611 22:47:51 all (10/18): unionfs18.sh 20240611 22:47:54 all (11/18): unionfs2.sh stress2: freework leak: 165/171. stress2: newblk leak: 90107/90111. 20240611 23:00:22 all (12/18): unionfs3.sh 20240611 23:00:28 all (13/18): unionfs4.sh 20240611 23:03:03 all (14/18): unionfs5.sh 20240611 23:05:33 all (15/18): unionfs6.sh 20240611 23:12:40 all (16/18): unionfs7.sh 20240611 23:19:27 all (17/18): unionfs8.sh Jun 11 23:23:28 mercat1 kernel: pid 14535 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240611 23:26:28 all (18/18): unionfs9.sh 20240611 23:31:34 all (1/18): unionfs.sh Jun 11 23:32:11 mercat1 kernel: pid 78532 (mkdir), uid 0 inumber 124 on /mnt10: out of inodes 20240611 23:33:38 all (2/18): unionfs10.sh 20240611 23:33:41 all (3/18): unionfs11.sh 20240611 23:33:45 all (4/18): unionfs12.sh 20240611 23:33:48 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe017005c708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff800051fc800 hold count flags () flags () v_object 0xfffff803206a3420 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8054a0aa740 (pid 66550, mkdir, tid 100556) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017005c708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff800051fc800 hold count flags () flags () v_object 0xfffff803206a3420 ref 0 pages 4159 cleanbuf 517 dirtybuf 5 lock type mntfs: EXCL by thread 0xfffff8054a0aa740 (pid 66550, mkdir, tid 100556) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017005c708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff800051fc800 hold count flags () flags () v_object 0xfffff803206a3420 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8005a569000 (pid 66551, mkdir, tid 100428) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe017005c708: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff800051fc800 hold count flags () flags () v_object 0xfffff803206a3420 ref 0 pages 4159 cleanbuf 516 dirtybuf 6 lock type mntfs: EXCL by thread 0xfffff8005a569000 (pid 66551, mkdir, tid 100428) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240611 23:34:49 all (6/18): unionfs14.sh 20240611 23:39:55 all (7/18): unionfs15.sh 20240611 23:40:04 all (8/18): unionfs16.sh 20240611 23:42:18 all (9/18): unionfs17.sh 20240611 23:48:33 all (10/18): unionfs18.sh 20240611 23:48:36 all (11/18): unionfs2.sh stress2: freework leak: 196/202. stress2: newblk leak: 106535/106539. 20240612 00:00:35 all (12/18): unionfs3.sh 20240612 00:00:42 all (13/18): unionfs4.sh 20240612 00:02:55 all (14/18): unionfs5.sh 20240612 00:05:00 all (15/18): unionfs6.sh Jun 12 00:10:58 mercat1 kernel: pid 93226 (rw), uid 2007 inumber 206249 on /mnt10: filesystem full 20240612 00:11:21 all (16/18): unionfs7.sh Jun 12 00:13:22 mercat1 kernel: pid 16436 (openat), uid 2007 inumber 80195 on /mnt10: filesystem full 20240612 00:17:54 all (17/18): unionfs8.sh Jun 12 00:18:52 mercat1 kernel: pid 39832 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240612 00:24:15 all (18/18): unionfs9.sh 20240612 00:29:20 all (1/18): unionfs.sh 20240612 00:31:29 all (2/18): unionfs10.sh 20240612 00:31:33 all (3/18): unionfs11.sh 20240612 00:31:36 all (4/18): unionfs12.sh 20240612 00:31:40 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe0180910000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8079354ac00 hold count flags () flags () v_object 0xfffff806021d4948 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8034a534740 (pid 19933, mkdir, tid 100401) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0180910000: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff8079354ac00 hold count flags () flags () v_object 0xfffff806021d4948 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff8034a534740 (pid 19933, mkdir, tid 100401) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240612 00:32:24 all (6/18): unionfs14.sh 20240612 00:37:30 all (7/18): unionfs15.sh 20240612 00:37:38 all (8/18): unionfs16.sh 20240612 00:39:53 all (9/18): unionfs17.sh 20240612 00:46:05 all (10/18): unionfs18.sh 20240612 00:46:08 all (11/18): unionfs2.sh stress2: freework leak: 310/316. stress2: newblk leak: 118649/118653. 20240612 00:57:43 all (12/18): unionfs3.sh 20240612 00:57:46 all (13/18): unionfs4.sh 20240612 01:00:04 all (14/18): unionfs5.sh 20240612 01:02:42 all (15/18): unionfs6.sh 20240612 01:09:20 all (16/18): unionfs7.sh Jun 12 01:12:23 mercat1 kernel: pid 88861 (swap), jid 0, uid 2007, was killed: a thread waited too long to allocate a page 20240612 01:15:47 all (17/18): unionfs8.sh 20240612 01:22:08 all (18/18): unionfs9.sh 20240612 01:27:13 all (1/18): unionfs.sh 20240612 01:29:18 all (2/18): unionfs10.sh 20240612 01:29:21 all (3/18): unionfs11.sh 20240612 01:29:25 all (4/18): unionfs12.sh 20240612 01:29:28 all (5/18): unionfs13.sh fsync: giving up on dirty (error = 35) 0xfffffe0172ff9770: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80439914800 hold count flags () flags () v_object 0xfffff807596ab630 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80109551000 (pid 48227, mkdir, tid 100420) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe0172ff9770: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 524 seqc users 0 rdev 0xfffff80439914800 hold count flags () flags () v_object 0xfffff807596ab630 ref 0 pages 4159 cleanbuf 518 dirtybuf 4 lock type mntfs: EXCL by thread 0xfffff80109551000 (pid 48227, mkdir, tid 100420) #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 20240612 01:30:12 all (6/18): unionfs14.sh 20240612 01:35:17 all (7/18): unionfs15.sh 20240612 01:35:25 all (8/18): unionfs16.sh stress2: newblk leak: 126/132. 20240612 01:37:45 all (9/18): unionfs17.sh 20240612 01:43:59 all (10/18): unionfs18.sh 20240612 01:44:02 all (11/18): unionfs2.sh stress2: freework leak: 300/306. stress2: newblk leak: 163898/163902. 20240612 01:55:33 all (12/18): unionfs3.sh 20240612 01:55:36 all (13/18): unionfs4.sh 20240612 01:57:57 all (14/18): unionfs5.sh fsync: giving up on dirty (error = 35) 0xfffffe01820104b0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80167d12400 hold count flags () flags () v_object 0xfffff807f0a26420 ref 0 pages 4251 cleanbuf 515 dirtybuf 18 lock type mntfs: EXCL by thread 0xfffff8034f19c740 (pid 50925, mkdir, tid 100748) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01820104b0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80167d12400 hold count flags () flags () v_object 0xfffff807f0a26420 ref 0 pages 4251 cleanbuf 517 dirtybuf 16 lock type mntfs: EXCL by thread 0xfffff8034f19c740 (pid 50925, mkdir, tid 100748) with exclusive waiters pending #0 0xffffffff80b140c8 at lockmgr_lock_flags+0x1c8 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80ec482f at ffs_sync+0x2bf #4 0xffffffff80edc53d at ufs_sync_nlink+0xad #5 0xffffffff80eda730 at ufs_mkdir+0x40 #6 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #7 0xffffffff82852b9e at unionfs_mkdir+0xce #8 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #9 0xffffffff80c567bb at kern_mkdirat+0x20b #10 0xffffffff8106c6a8 at amd64_syscall+0x158 #11 0xffffffff8103debb at fast_syscall_common+0xf8 fsync: giving up on dirty (error = 35) 0xfffffe01820104b0: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff818addf0 usecount 1, writecount 0, refcount 535 seqc users 0 rdev 0xfffff80167d12400 hold count flags () flags () v_object 0xfffff807f0a26420 ref 0 pages 4251 cleanbuf 520 dirtybuf 13 lock type mntfs: EXCL by thread 0xfffff80109089000 (pid 50927, mkdir, tid 100370) #0 0xffffffff80b14bc7 at lockmgr_xlock_hard+0x487 #1 0xffffffff8113629a at VOP_LOCK1_APV+0x3a #2 0xffffffff80c5aa83 at _vn_lock+0x53 #3 0xffffffff80e9faed at softdep_flushworklist+0x5d #4 0xffffffff80ec4996 at ffs_sync+0x426 #5 0xffffffff80edc53d at ufs_sync_nlink+0xad #6 0xffffffff80eda730 at ufs_mkdir+0x40 #7 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #8 0xffffffff82852b9e at unionfs_mkdir+0xce #9 0xffffffff8113547a at VOP_MKDIR_APV+0x6a #10 0xffffffff80c567bb at kern_mkdirat+0x20b #11 0xffffffff8106c6a8 at amd64_syscall+0x158 #12 0xffffffff8103debb at fast_syscall_common+0xf8 20240612 02:00:00 all (15/18): unionfs6.sh 20240612 02:06:20 all (16/18): unionfs7.sh 20240612 02:12:56 all (17/18): unionfs8.sh 20240612 02:19:29 all (18/18): unionfs9.sh ps -lp55088 UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 55088 37241 6 68 0 13088 2324 unioncp I+ 0 0:00.01 find /mnt12 -type f -maxdepth 2 -ls You have new mail. root@mercat1:~ # procstat -k 55088 PID TID COMM TDNAME KSTACK 55088 100476 find - mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep unionfs_set_in_progress_flag unionfs_lookup VOP_CACHEDLOOKUP_APV vfs_cache_lookup VOP_LOOKUP_APV vfs_lookup namei kern_statat sys_fstatat amd64_syscall fast_syscall_common root@mercat1:~ # date Wed Jun 12 11:58:20 CEST 2024 root@mercat1:~ # root@mercat1:~ # KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_alt_break_internal+0x14d: movq $0,0x104e4e8(%rip) db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> dump Dumping 3121 out of 32620 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:1:pho1> bt Tracing pid 11 tid 100003 td 0xfffff80003bf4740 kdb_alt_break_internal() at kdb_alt_break_internal+0x14d/frame 0xfffffe0108014b90 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0108014ba0 uart_intr_rxready() at uart_intr_rxready+0xf4/frame 0xfffffe0108014bd0 uart_intr() at uart_intr+0x118/frame 0xfffffe0108014c10 intr_event_handle() at intr_event_handle+0xc8/frame 0xfffffe0108014c80 intr_execute_handlers() at intr_execute_handlers+0x63/frame 0xfffffe0108014cb0 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe0108014cb0 --- interrupt, rip = 0xffffffff804c551e, rsp = 0xfffffe0108014d80, rbp = 0xfffffe0108014db0 --- acpi_cpu_idle() at acpi_cpu_idle+0x27e/frame 0xfffffe0108014db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108014dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108014df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108014ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108014f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108014f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:1:pho1> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0xa05140 curthread = 0xfffff80003bf4740: pid 11 tid 100003 critnest 2 "idle: cpu0" curpcb = 0xfffff80003bf4c60 fpcurthread = none idlethread = 0xfffff80003bf4740: tid 100003 "idle: cpu0" self = 0xffffffff82410000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82410384 rsp0 = 0xfffffe0108015000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x1ae921f39 gs32p = 0xffffffff82410404 ldt = 0xffffffff82410444 tss = 0xffffffff82410434 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00bb4c5140 curthread = 0xfffff80003bf4000: pid 11 tid 100004 critnest 1 "idle: cpu1" curpcb = 0xfffff80003bf4520 fpcurthread = none idlethread = 0xfffff80003bf4000: tid 100004 "idle: cpu1" self = 0xffffffff82411000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82411384 rsp0 = 0xfffffe0108010000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x291ecfac4 gs32p = 0xffffffff82411404 ldt = 0xffffffff82411444 tss = 0xffffffff82411434 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00bb4d4140 curthread = 0xfffff80003bf5740: pid 11 tid 100005 critnest 1 "idle: cpu2" curpcb = 0xfffff80003bf5c60 fpcurthread = none idlethread = 0xfffff80003bf5740: tid 100005 "idle: cpu2" self = 0xffffffff82412000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82412384 rsp0 = 0xfffffe010801f000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x584c2c861 gs32p = 0xffffffff82412404 ldt = 0xffffffff82412444 tss = 0xffffffff82412434 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00bb4e3140 curthread = 0xfffff80003bf5000: pid 11 tid 100006 critnest 1 "idle: cpu3" curpcb = 0xfffff80003bf5520 fpcurthread = none idlethread = 0xfffff80003bf5000: tid 100006 "idle: cpu3" self = 0xffffffff82413000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82413384 rsp0 = 0xfffffe010801a000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x577623b74 gs32p = 0xffffffff82413404 ldt = 0xffffffff82413444 tss = 0xffffffff82413434 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00bb4f2140 curthread = 0xfffff80003bf6740: pid 11 tid 100007 critnest 1 "idle: cpu4" curpcb = 0xfffff80003bf6c60 fpcurthread = none idlethread = 0xfffff80003bf6740: tid 100007 "idle: cpu4" self = 0xffffffff82414000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82414384 rsp0 = 0xfffffe0108029000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x3b8193be8 gs32p = 0xffffffff82414404 ldt = 0xffffffff82414444 tss = 0xffffffff82414434 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00bb501140 curthread = 0xfffff80003bf6000: pid 11 tid 100008 critnest 1 "idle: cpu5" curpcb = 0xfffff80003bf6520 fpcurthread = none idlethread = 0xfffff80003bf6000: tid 100008 "idle: cpu5" self = 0xffffffff82415000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82415384 rsp0 = 0xfffffe0108024000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x44e5b9ca2 gs32p = 0xffffffff82415404 ldt = 0xffffffff82415444 tss = 0xffffffff82415434 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00bb510140 curthread = 0xfffff80003bdf740: pid 11 tid 100009 critnest 1 "idle: cpu6" curpcb = 0xfffff80003bdfc60 fpcurthread = none idlethread = 0xfffff80003bdf740: tid 100009 "idle: cpu6" self = 0xffffffff82416000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82416384 rsp0 = 0xfffffe0108033000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x1be68cbb9 gs32p = 0xffffffff82416404 ldt = 0xffffffff82416444 tss = 0xffffffff82416434 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00bb51f140 curthread = 0xfffff80003bdf000: pid 11 tid 100010 critnest 1 "idle: cpu7" curpcb = 0xfffff80003bdf520 fpcurthread = none idlethread = 0xfffff80003bdf000: tid 100010 "idle: cpu7" self = 0xffffffff82417000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82417384 rsp0 = 0xfffffe010802e000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x54dc4cab3 gs32p = 0xffffffff82417404 ldt = 0xffffffff82417444 tss = 0xffffffff82417434 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00bb52e140 curthread = 0xfffff80003be0740: pid 11 tid 100011 critnest 1 "idle: cpu8" curpcb = 0xfffff80003be0c60 fpcurthread = none idlethread = 0xfffff80003be0740: tid 100011 "idle: cpu8" self = 0xffffffff82418000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82418384 rsp0 = 0xfffffe010803d000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x5af1b2be1 gs32p = 0xffffffff82418404 ldt = 0xffffffff82418444 tss = 0xffffffff82418434 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00bb53d140 curthread = 0xfffff80003be0000: pid 11 tid 100012 critnest 1 "idle: cpu9" curpcb = 0xfffff80003be0520 fpcurthread = none idlethread = 0xfffff80003be0000: tid 100012 "idle: cpu9" self = 0xffffffff82419000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff82419384 rsp0 = 0xfffffe0108038000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x31d7c7e94 gs32p = 0xffffffff82419404 ldt = 0xffffffff82419444 tss = 0xffffffff82419434 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00bb54c140 curthread = 0xfffff80003be1740: pid 11 tid 100013 critnest 1 "idle: cpu10" curpcb = 0xfffff80003be1c60 fpcurthread = none idlethread = 0xfffff80003be1740: tid 100013 "idle: cpu10" self = 0xffffffff8241a000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff8241a384 rsp0 = 0xfffffe0108047000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x58c3cfaa7 gs32p = 0xffffffff8241a404 ldt = 0xffffffff8241a444 tss = 0xffffffff8241a434 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00bb55b140 curthread = 0xfffff80003be1000: pid 11 tid 100014 critnest 1 "idle: cpu11" curpcb = 0xfffff80003be1520 fpcurthread = none idlethread = 0xfffff80003be1000: tid 100014 "idle: cpu11" self = 0xffffffff8241b000 curpmap = 0xffffffff81b88ef0 tssp = 0xffffffff8241b384 rsp0 = 0xfffffe0108042000 kcr3 = 0x8000000002387002 ucr3 = 0xffffffffffffffff scr3 = 0x2a9161ea9 gs32p = 0xffffffff8241b404 ldt = 0xffffffff8241b444 tss = 0xffffffff8241b434 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes db:1:pho1> show mount 0xfffffe010ebd5700 /dev/gpt/work on / (ufs) 0xfffffe010ebd6280 devfs on /dev (devfs) 0xfffffe010ebd4b80 /dev/gpt/tmp on /tmp (ufs) 0xfffffe018354c280 /dev/md10 on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 10985 Buf domain 0 freebufs 3055 lofreebufs 219 hifreebufs 329 bufspace 235339776 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 11977 (233017344) dirtyq count 0 (0) wakeup 0 lim 25 CPU 9, 20, 2, 3, 17, 5, 11, 7, 5, 0, 3, 19, Locked buffers: 0 space 0 Total buffers: 14565 space 235339776 Buf domain 1 freebufs 3281 lofreebufs 219 hifreebufs 329 bufspace 235429888 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 11755 (232947712) dirtyq count 0 (0) wakeup 0 lim 25 CPU 21, 2, 0, 1, 12, 7, 2, 14, 13, 2, 6, 17, Locked buffers: 0 space 0 Total buffers: 14033 space 235429888 Buf domain 2 freebufs 329 lofreebufs 219 hifreebufs 329 bufspace 234868736 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 14674 (232128512) dirtyq count 0 (0) wakeup 0 lim 25 CPU 3, 11, 4, 6, 18, 16, 11, 3, 14, 0, 23, 21, Locked buffers: 0 space 0 Total buffers: 15527 space 234868736 Buf domain 3 freebufs 1613 lofreebufs 219 hifreebufs 329 bufspace 235438080 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 13370 (231866368) dirtyq count 0 (0) wakeup 0 lim 25 CPU 24, 7, 20, 24, 18, 0, 11, 10, 9, 19, 3, 5, Locked buffers: 0 space 0 Total buffers: 16252 space 235438080 Buf domain 4 freebufs 1463 lofreebufs 219 hifreebufs 329 bufspace 235450368 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 13541 (233029632) dirtyq count 0 (0) wakeup 0 lim 25 CPU 8, 7, 17, 15, 10, 5, 23, 23, 6, 0, 12, 3, Locked buffers: 0 space 0 Total buffers: 15598 space 235450368 Buf domain 5 freebufs 329 lofreebufs 219 hifreebufs 329 bufspace 171921408 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 14629 (168456192) dirtyq count 0 (0) wakeup 0 lim 25 CPU 19, 5, 14, 19, 12, 15, 23, 14, 8, 10, 23, 13, Locked buffers: 0 space 0 Total buffers: 15484 space 171921408 Buf domain 6 freebufs 1395 lofreebufs 219 hifreebufs 329 bufspace 235421696 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 14 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 13678 (234041344) dirtyq count 14 (458752) wakeup 0 lim 25 CPU 0, 2, 1, 1, 1, 0, 0, 3, 0, 1, 21, 16, Locked buffers: 0 space 0 Total buffers: 14472 space 235421696 db:1:pho1> show page vm_cnt.v_free_count: 6656320 vm_cnt.v_inactive_count: 554030 vm_cnt.v_active_count: 853 vm_cnt.v_laundry_count: 176 vm_cnt.v_wire_count: 925340 vm_cnt.v_free_reserved: 10642 vm_cnt.v_free_min: 51257 vm_cnt.v_free_target: 173102 vm_cnt.v_inactive_target: 259653 db:1:pho1> show pageq pq_free 6656320 dom 0 page_cnt 8123039 free 6656320 pq_act 853 pq_inact 554030 pq_laund 176 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000002 | 000001 10 (004096K) | 000169 | 000093 09 (002048K) | 001596 | 000377 08 (001024K) | 006611 | 000501 07 (000512K) | 011099 | 000734 06 (000256K) | 016914 | 000983 05 (000128K) | 023975 | 000057 04 (000064K) | 005976 | 000001 03 (000032K) | 000000 | 000001 02 (000016K) | 000000 | 000008 01 (000008K) | 000001 | 000163 00 (000004K) | 000001 | 001577 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000001 | 000000 06 (000256K) | 000000 | 000001 05 (000128K) | 000003 | 000000 04 (000064K) | 000003 | 000000 03 (000032K) | 000002 | 000000 02 (000016K) | 000005 | 000000 01 (000008K) | 000005 | 000000 00 (000004K) | 000004 | 000001 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Total Mem XFree FFS inode 1168 619285 201 285464701 0 8 723559648 0 VNODE 600 619417 969 2353294979 0 53 372231600 0 FFS2 dinode 256 619285 635 285464533 0 62 158699520 0 malloc-4096 4096 34123 0 163625 0 2 139767808 0 VM OBJECT 264 183004 261746 819065491 0 31 117414000 0 L VFS Cache 320 249954 906 3575179 0 30 80275200 0 S VFS Cache 104 371032 132536 1223125784 0 129 52371072 0 mbuf_cluster 2048 15600 148 345551 0 8 32251904 0 swblk 136 1866 224073 12873755 0 62 30727704 0 RADIX NODE 144 113482 94695 994210841 0 146 29977488 0 DIRHASH 1024 18273 63 1254562 0 16 18776064 0 UMA Slabs 0 112 138264 150 2686240 0 126 15502368 0 BUF TRIE 144 6347 99605 104969247 0 62 15257088 0 vmem btag 56 170152 33164 5105638 0 254 11385696 0 malloc-128 128 44548 1394 1643474 0 126 5880576 0 mbuf 256 16449 3320 3225224 0 254 5060864 0 pbuf 2776 0 1267 0 0 0 3517192 0 256 Bucket 2048 747 761 42056571 0 8 3088384 0 malloc-128 128 1928 21911 2248871015 0 126 3051392 0 TMPFS node 232 0 11645 195206057 0 62 2701640 0 swpctrie 144 309 15875 977081 0 62 2330496 0 malloc-32768 32768 14 43 36234 0 1 1867776 0 malloc-128 128 12610 1402 436655286 0 126 1793536 0 THREAD 1824 825 109 3390 0 8 1703616 0 malloc-4096 4096 332 24 22248 0 2 1458176 0 malloc-512 512 1810 198 133543 0 30 1028096 0 malloc-65536 65536 2 12 1386 0 1 917504 0 malloc-16384 16384 2 54 5077285 0 1 917504 0 32 Bucket 256 235 3200 48420425 0 62 879360 0 PROC 1376 64 563 33278926 0 8 862752 0 FPU_save_area 832 827 199 105372 0 16 853632 0 g_bio 408 0 2090 227682091 0 30 852720 0 mbuf_packet 256 10 3306 2963150 0 254 848896 0 malloc-8192 8192 99 0 111 0 1 811008 0 malloc-32768 32768 24 0 24 0 1 786432 0 malloc-64 64 11697 588 21425 0 254 786240 0 128 Bucket 1024 314 445 50806429 0 16 777216 0 malloc-64 64 9006 2838 24682 0 254 758016 0 MAP ENTRY 96 2075 5317 1842892773 0 128 709632 0 64 Bucket 512 439 705 63777846 0 30 585728 0 malloc-4096 4096 72 65 33283834 0 2 561152 0 lkpicurr 168 13 3083 13 0 62 520128 0 malloc-1024 1024 261 243 3296013 0 16 516096 0 malloc-64 64 4650 3351 573106693 0 254 512064 0 malloc-256 256 873 1062 1994733327 0 62 495360 0 UMA Zones 2048 227 8 227 0 8 481280 0 VMSPACE 616 42 648 33278391 0 16 425040 0 malloc-256 256 943 692 3508 0 62 418560 0 malloc-2048 2048 116 80 1798 0 8 401408 0 malloc-256 256 637 878 10767500 0 62 387840 0 filedesc0 1072 65 292 33278927 0 11 382704 0 4 Bucket 48 111 7533 4880211 0 254 366912 0 malloc-32 32 8136 3078 40085 0 254 358848 0 ksiginfo 112 610 2522 321983 0 126 350784 0 2 Bucket 32 345 10491 11059047 0 254 346752 0 malloc-65536 65536 3 2 3312 0 1 327680 0 malloc-128 128 1028 1390 354287291 0 126 309504 0 malloc-16 16 15732 2268 15916 0 254 288000 0 malloc-64 64 427 4046 38994819 0 254 286272 0 socket 960 39 241 25700 0 16 268800 0 NAMEI 1024 1 259 2784464378 0 16 266240 0 malloc-65536 65536 1 3 2498 0 1 262144 0 malloc-1024 1024 4 240 53251042 0 16 249856 0 malloc-384 384 628 22 630 0 30 249600 0 malloc-128 128 517 1405 27234513 0 126 246016 0 malloc-128 128 386 1536 6998643717 0 126 246016 0 malloc-16384 16384 15 0 4820 0 1 245760 0 malloc-64 64 25 3566 404874470 0 254 229824 0 16 Bucket 144 338 1230 142721455 0 62 225792 0 malloc-1024 1024 45 175 734795 0 16 225280 0 Mountpoints 2944 4 72 365752 0 4 223744 0 malloc-256 256 0 870 1012068 0 62 222720 0 malloc-256 256 177 693 1912 0 62 222720 0 malloc-64 64 102 3363 33664512 0 254 221760 0 malloc-1024 1024 17 195 1402 0 16 217088 0 malloc-2048 2048 16 88 8998 0 8 212992 0 udp_inpcb 416 11 493 3275 0 30 209664 0 malloc-64 64 186 3090 3568192385 0 254 209664 0 pipe 728 5 281 18081303 0 16 208208 0 malloc-256 256 45 750 714374873 0 62 203520 0 swrbuf 2776 0 73 1974840 0 4 202648 0 clpbuf 2776 0 72 5331228 0 4 199872 0 malloc-128 128 182 1368 82811353 0 126 198400 0 malloc-64 64 14 3073 269534457 0 254 197568 0 malloc-65536 65536 3 0 3 0 1 196608 0 malloc-2048 2048 89 7 3714 0 8 196608 0 Files 80 123 2327 500692872 0 126 196000 0 unpcb 256 19 746 22070 0 62 195840 0 malloc-256 256 14 751 4090726 0 62 195840 0 malloc-256 256 16 749 2687122 0 62 195840 0 malloc-128 128 107 1412 164941707 0 126 194432 0 PWD 40 18 4729 166530257 0 254 189880 0 malloc-8192 8192 23 0 8754 0 1 188416 0 PGRP 120 31 1487 911 0 126 182160 0 lkpimm 56 3 3093 3 0 254 173376 0 TURNSTILE 136 935 304 935 0 62 168504 0 malloc-384 384 39 391 316164809 0 30 165120 0 malloc-32768 32768 3 2 6 0 1 163840 0 fakepg 104 0 1521 2408 0 126 158184 0 rl_entry 40 676 3061 185898 0 254 149480 0 ada_ccb 272 0 546 10884618 0 30 148512 0 8 Bucket 80 229 1571 88301068 0 126 144000 0 malloc-384 384 27 333 1470 0 30 138240 0 malloc-65536 65536 2 0 2 0 1 131072 0 malloc-32768 32768 4 0 3609 0 1 131072 0 malloc-1024 1024 121 7 218 0 16 131072 0 malloc-32 32 112 3920 13933953 0 254 129024 0 SLEEPQUEUE 88 935 473 935 0 126 123904 0 malloc-4096 4096 6 24 374782 0 2 122880 0 swwbuf 2776 0 44 6819510 0 2 122144 0 malloc-32 32 544 3110 83929540 0 254 116928 0 malloc-8192 8192 14 0 479 0 1 114688 0 ertt_txseginfo 40 0 2828 41650 0 254 113120 0 udp_inpcb ports 32 3 3525 643 0 254 112896 0 DEVCTL 1024 0 104 739762 0 0 106496 0 malloc-32 32 708 2442 77857 0 254 100800 0 malloc-32 32 83 3067 3377 0 254 100800 0 malloc-32 32 38 3112 117845415 0 254 100800 0 ttyinq 160 330 295 825 0 62 100000 0 malloc-4096 4096 22 0 39 0 2 90112 0 ttyoutq 256 173 172 431 0 62 88320 0 malloc-384 384 164 56 8815 0 30 84480 0 malloc-16384 16384 5 0 5 0 1 81920 0 malloc-4096 4096 14 6 1095 0 2 81920 0 malloc-32 32 1151 1369 552987 0 254 80640 0 UMA Kegs 384 198 5 198 0 30 77952 0 malloc-4096 4096 19 0 21 0 2 77824 0 malloc-65536 65536 1 0 1 0 1 65536 0 malloc-65536 65536 1 0 1 0 1 65536 0 malloc-32768 32768 2 0 2 0 1 65536 0 malloc-32768 32768 2 0 188139 0 1 65536 0 malloc-16384 16384 4 0 9 0 1 65536 0 malloc-8192 8192 8 0 50 0 1 65536 0 malloc-2048 2048 28 4 40 0 8 65536 0 malloc-16 16 47 3953 54789464 0 254 64000 0 malloc-16 16 1874 2126 3004270 0 254 64000 0 malloc-16 16 783 2967 574098761 0 254 60000 0 malloc-32 32 380 1258 248983 0 254 52416 0 malloc-16 16 862 2388 1686499 0 254 52000 0 malloc-16 16 79 3171 1695608988 0 254 52000 0 malloc-16384 16384 3 0 7 0 1 49152 0 malloc-8192 8192 6 0 1056 0 1 49152 0 pcpu-8 8 4208 1936 4210 0 254 49152 0 tcp_inpcb 1304 6 30 17 0 8 46944 0 KNOTE 160 11 264 2909767 0 62 44000 0 pcpu-64 64 454 186 454 0 254 40960 0 pcpu-16 16 76 2484 76 0 254 40960 0 kenv 258 18 132 27413 0 30 38700 0 malloc-512 512 28 36 28080 0 30 32768 0 cpuset 200 27 133 27 0 62 32000 0 malloc-2048 2048 11 3 74197 0 8 28672 0 malloc-8192 8192 3 0 3 0 1 24576 0 malloc-1024 1024 5 19 6 0 16 24576 0 malloc-16 16 24 1476 25469 0 254 24000 0 malloc-16 16 4 1496 10 0 254 24000 0 malloc-384 384 11 49 166543 0 30 23040 0 malloc-384 384 27 33 490 0 30 23040 0 rtentry 168 12 108 14 0 62 20160 0 tcp_inpcb ports 32 2 628 4 0 254 20160 0 ertt 72 6 274 17 0 126 20160 0 routing nhops 256 9 66 14 0 62 19200 0 mbuf_jumbo_page 4096 0 4 6 0 2 16384 0 malloc-16384 16384 1 0 1 0 1 16384 0 ripcb 384 1 39 4 0 30 15360 0 malloc-2048 2048 3 3 3 0 8 12288 0 malloc-512 512 12 12 12 0 30 12288 0 malloc-512 512 3 21 3 0 30 12288 0 malloc-384 384 13 17 13 0 30 11520 0 malloc-8192 8192 1 0 1 0 1 8192 0 malloc-8192 8192 1 0 25 0 1 8192 0 malloc-512 512 3 13 466 0 30 8192 0 epoch_record pcpu 256 4 12 4 0 62 4096 0 malloc-2048 2048 1 1 127 0 8 4096 0 malloc-1024 1024 2 2 2 0 16 4096 0 malloc-512 512 1 7 57 0 30 4096 0 malloc-512 512 2 6 50 0 30 4096 0 SMR CPU 32 7 120 7 0 254 4064 0 vtnet_tx_hdr 24 0 167 0 0 254 4008 0 UMA Slabs 1 176 11 11 11 0 62 3872 0 KMAP ENTRY 96 36 3 36 0 0 3744 0 vmem 1856 2 0 2 0 8 3712 0 SMR SHARED 24 7 120 7 0 254 3048 0 FFS1 dinode 128 0 0 0 0 126 0 0 da_ccb 544 0 0 0 0 16 0 0 tfo_ccache_entries 80 0 0 0 0 126 0 0 tfo 4 0 0 0 0 254 0 0 sackhole 32 0 0 0 0 254 0 0 hostcache 64 0 0 0 0 254 0 0 syncache 168 0 0 2 0 62 0 0 ipq 56 0 0 0 0 254 0 0 tcp_log_id_node 120 0 0 0 0 126 0 0 tcp_log_id_bucket 176 0 0 0 0 62 0 0 tcp_log 416 0 0 0 0 30 0 0 tcpreass 48 0 0 0 0 254 0 0 udplite_inpcb ports 32 0 0 0 0 254 0 0 udplite_inpcb 416 0 0 0 0 30 0 0 ripcb ports 32 0 0 0 0 254 0 0 IPsec SA lft_c 16 0 0 0 0 254 0 0 itimer 352 0 0 0 0 30 0 0 AIOLIO 272 0 0 0 0 30 0 0 AIOCB 552 0 0 0 0 16 0 0 AIO 208 0 0 0 0 62 0 0 NCLNODE 760 0 0 0 0 16 0 0 LTS VFS Cache 360 0 0 0 0 30 0 0 STS VFS Cache 144 0 0 0 0 62 0 0 cryptop 280 0 0 0 0 30 0 0 linux_dma_object 32 0 0 0 0 254 0 0 linux_dma_pctrie 144 0 0 0 0 62 0 0 IOMMU_MAP_ENTRY 104 0 0 0 0 126 0 0 mbuf_jumbo_16k 16384 0 0 0 0 1 0 0 mbuf_jumbo_9k 9216 0 0 0 0 1 0 0 audit_record 1280 0 0 0 0 8 0 0 domainset 40 0 0 0 0 254 0 0 MAC labels 40 0 0 0 0 254 0 0 vnpbuf 2776 0 0 0 0 4 0 0 nfspbuf 2776 0 0 0 0 4 0 0 umtx_shm 88 0 0 0 0 126 0 0 umtx pi 96 0 0 0 0 126 0 0 rangeset pctrie nodes 144 0 0 0 0 62 0 0 malloc-65536 65536 0 0 0 0 1 0 0 malloc-32768 32768 0 0 0 0 1 0 0 malloc-32768 32768 0 0 0 0 1 0 0 malloc-16384 16384 0 0 0 0 1 0 0 malloc-16384 16384 0 0 603 0 1 0 0 malloc-4096 4096 0 0 0 0 2 0 0 malloc-2048 2048 0 0 0 0 8 0 0 malloc-1024 1024 0 0 0 0 16 0 0 malloc-512 512 0 0 0 0 30 0 0 malloc-384 384 0 0 1 0 30 0 0 pcpu-32 32 0 0 0 0 254 0 0 pcpu-4 4 0 0 0 0 254 0 0 UMA Hash 256 0 0 0 0 62 0 0 db:0:pho> show umacache Zone Size Used Free Requests Bucket debugnet_mbuf_packet 2048 0 0 0 0 debugnet_mbuf_cluster 2048 -6144 0 0 0 debugnet_mbuf 256 -10240 0 0 0 vm pgcache 4096 185410 3871 1399555235 66 vm pgcache 4096 1122534 2645 4375564385 169 buf free cache 848 94466 480 232161729 16 buffer arena-65 65536 0 0 0 1 buffer arena-61 61440 0 0 0 1 buffer arena-57 57344 0 0 0 1 buffer arena-53 53248 0 0 0 1 buffer arena-49 49152 0 0 0 1 buffer arena-45 45056 0 0 0 1 buffer arena-40 40960 0 0 0 1 buffer arena-36 36864 0 0 0 1 buffer arena-32 32768 11229 315 88817750 3 buffer arena-28 28672 0 0 0 1 buffer arena-24 24576 0 0 0 1 buffer arena-20 20480 0 0 0 1 buffer arena-16 16384 3 7 5457 1 buffer arena-12 12288 0 0 0 1 buffer arena-81 8192 0 0 0 1 buffer arena-40 4096 0 0 0 2 kstack_cache 16384 825 30 3390 1 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0xfffff80003bf4740 (CPU 0) kdb_alt_break_internal() at kdb_alt_break_internal+0x14d/frame 0xfffffe0108014b90 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0108014ba0 uart_intr_rxready() at uart_intr_rxready+0xf4/frame 0xfffffe0108014bd0 uart_intr() at uart_intr+0x118/frame 0xfffffe0108014c10 intr_event_handle() at intr_event_handle+0xc8/frame 0xfffffe0108014c80 intr_execute_handlers() at intr_execute_handlers+0x63/frame 0xfffffe0108014cb0 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe0108014cb0 --- interrupt, rip = 0xffffffff804c551e, rsp = 0xfffffe0108014d80, rbp = 0xfffffe0108014db0 --- acpi_cpu_idle() at acpi_cpu_idle+0x27e/frame 0xfffffe0108014db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108014dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108014df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108014ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108014f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108014f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff80003bf4000 (CPU 1) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce3cdf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce3ce00 trap() at trap+0x56/frame 0xfffffe003ce3cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3cf20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010800fd78, rbp = 0xfffffe010800fdb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010800fdb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010800fdd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010800fdf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010800fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010800ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010800ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff80003bf5740 (CPU 2) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce4bdf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce4be00 trap() at trap+0x56/frame 0xfffffe003ce4bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce4bf20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010801ed78, rbp = 0xfffffe010801edb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010801edb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010801edd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010801edf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010801eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010801ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010801ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff80003bf5000 (CPU 3) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce5adf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce5ae00 trap() at trap+0x56/frame 0xfffffe003ce5af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce5af20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108019d78, rbp = 0xfffffe0108019db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108019db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108019dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108019df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108019ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108019f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108019f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff80003bf6740 (CPU 4) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce69df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce69e00 trap() at trap+0x56/frame 0xfffffe003ce69f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce69f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108028d78, rbp = 0xfffffe0108028db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108028db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108028dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108028df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108028ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108028f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108028f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff80003bf6000 (CPU 5) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce78df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce78e00 trap() at trap+0x56/frame 0xfffffe003ce78f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce78f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108023d78, rbp = 0xfffffe0108023db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108023db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108023dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108023df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108023ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108023f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108023f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff80003bdf740 (CPU 6) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce87df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce87e00 trap() at trap+0x56/frame 0xfffffe003ce87f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce87f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108032d78, rbp = 0xfffffe0108032db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108032db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108032dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108032df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108032ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108032f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108032f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff80003bdf000 (CPU 7) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce96df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce96e00 trap() at trap+0x56/frame 0xfffffe003ce96f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce96f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010802dd78, rbp = 0xfffffe010802ddb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010802ddb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010802ddd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010802ddf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010802def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010802df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010802df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff80003be0740 (CPU 8) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003cea5df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003cea5e00 trap() at trap+0x56/frame 0xfffffe003cea5f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea5f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010803cd78, rbp = 0xfffffe010803cdb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010803cdb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010803cdd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010803cdf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010803cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010803cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010803cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff80003be0000 (CPU 9) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ceb4df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ceb4e00 trap() at trap+0x56/frame 0xfffffe003ceb4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ceb4f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108037d78, rbp = 0xfffffe0108037db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108037db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108037dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108037df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108037ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108037f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108037f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff80003be1740 (CPU 10) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003cec3df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003cec3e00 trap() at trap+0x56/frame 0xfffffe003cec3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cec3f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108046d78, rbp = 0xfffffe0108046db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108046db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108046dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108046df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108046ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108046f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108046f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff80003be1000 (CPU 11) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ced2df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ced2e00 trap() at trap+0x56/frame 0xfffffe003ced2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ced2f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108041d78, rbp = 0xfffffe0108041db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108041db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108041dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108041df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108041ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108041f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108041f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 818 3904 3854 0 S nanslp 0xffffffff81bcc1a9 sleep 55088 37241 3716 0 S+ unioncp 0xfffffe018e383770 find 37241 37206 3716 0 SW+ wait 0xfffffe016fdb5ae0 sh 37228 0 0 0 DL mdwait 0xfffff8044b384000 [md24] 37223 0 0 0 DL mdwait 0xfffff8070effc000 [md23] 37218 0 0 0 DL mdwait 0xfffff804c97be000 [md22] 37213 0 0 0 DL mdwait 0xfffff8071833d000 [md10] 37207 3716 3716 0 S+ piperd 0xfffffe017ca4e000 tee 37206 37205 3716 0 S+ pause 0xfffffe016fdb4b70 sh 37205 3716 3716 0 S+ wait 0xfffffe016fdb75c0 sh 14083 3633 14083 0 S+ ttyin 0xfffff800055838b0 csh 3907 3854 3854 0 S piperd 0xfffffe016e4e9888 sh 3905 3854 3854 0 S piperd 0xfffffe016ad96998 awk 3904 3854 3854 0 S wait 0xfffffe016512a5c0 sh 3885 3882 3885 2006 Ss+ ttyin 0xfffff8004effacb0 sh 3882 3781 3781 2006 S select 0xfffff80004442040 sshd 3854 1 3854 0 SWs wait 0xfffffe01651e9020 sh 3781 3611 3781 0 Ss select 0xfffff80003e18a40 sshd 3716 3663 3716 0 S+ wait 0xfffffe016d71b560 sh 3663 3661 3663 0 SW+ wait 0xfffffe003ce73ae0 bash 3661 3660 3661 0 SW+ pause 0xfffffe016d3370b0 csh 3660 3638 3660 2006 SW+ wait 0xfffffe016ad4d040 su 3638 3637 3638 2006 SWs+ wait 0xfffffe016d71c020 sh 3637 3634 3634 2006 S select 0xfffff800438682c0 sshd 3634 3611 3634 0 Ss select 0xfffff80004442240 sshd 3633 1 3633 0 SWs+ wait 0xfffffe016d6af000 login 3632 1 3632 0 Ss+ ttyin 0xfffff800437818b0 getty 3631 1 3631 0 Ss+ ttyin 0xfffff80043781cb0 getty 3630 1 3630 0 Ss+ ttyin 0xfffff800438d20b0 getty 3629 1 3629 0 Ss+ ttyin 0xfffff800438d24b0 getty 3628 1 3628 0 Ss+ ttyin 0xfffff800438d28b0 getty 3627 1 3627 0 Ss+ ttyin 0xfffff800438d2cb0 getty 3626 1 3626 0 Ss+ ttyin 0xfffff800438d10b0 getty 3625 1 3625 0 Ss+ ttyin 0xfffff800438d14b0 getty 3611 1 3611 0 Ss select 0xfffff8000421a740 sshd 3594 1 3594 0 Ss nanslp 0xffffffff81bcc1a9 cron 3589 1 3589 0 Ss select 0xfffff80003e19dc0 inetd 3568 1 3568 0 Ss select 0xfffff80005144dc0 powerd 3551 1 3551 123 Ss (threaded) ntpd 100344 S select 0xfffff80003e186c0 ntpd 103390 S usem 0xfffff806ff771c80 ntpd 3451 1 3451 0 Ss nanslp 0xffffffff81bcc1a2 watchdogd 3434 1 3434 0 Ss kqread 0xfffff80003d7c100 syslogd 3286 1 3286 0 Ss select 0xfffff800044421c0 devd 3271 1 3271 65 Ss select 0xfffff80003d2f3c0 dhclient 3212 1 3212 0 Ss select 0xfffff80003e170c0 dhclient 3209 1 3209 0 Ss select 0xfffff8005a240cc0 dhclient 19 0 0 0 DL syncer 0xffffffff81cb29a8 [syncer] 18 0 0 0 DL vlruwt 0xfffffe010ebd0000 [vnlru] 17 0 0 0 DL (threaded) [bufdaemon] 100201 D psleep 0xffffffff81cb19c0 [bufdaemon] 100202 D - 0xffffffff81802100 [bufspacedaemon-0] 100203 D - 0xffffffff81802380 [bufspacedaemon-1] 100204 D - 0xffffffff81802600 [bufspacedaemon-2] 100205 D - 0xffffffff81802880 [bufspacedaemon-3] 100206 D - 0xffffffff81802b00 [bufspacedaemon-4] 100207 D - 0xffffffff81802d80 [bufspacedaemon-5] 100208 D - 0xffffffff81803000 [bufspacedaemon-6] 100232 D sdflush 0xfffff8004e8afce8 [/ worker] 100248 D sdflush 0xfffff8004e8af8e8 [/tmp worker] 103138 D sdflush 0xfffff807298410e8 [/mnt worker] 16 0 0 0 DL psleep 0xffffffff81ce5510 [vmdaemon] 9 0 0 0 DL (threaded) [pagedaemon] 100197 D psleep 0xffffffff81ccd978 [dom0] 100198 D launds 0xffffffff81ccd984 [laundry: dom0] 100199 D umarcl 0xffffffff80ee9220 [uma] 8 0 0 0 DL idle 0xfffffe010e65dae0 [enc_daemon1] 7 0 0 0 DL idle 0xfffffe010e65e040 [enc_daemon0] 6 0 0 0 DL - 0xffffffff819cd9b0 [rand_harvestq] 15 0 0 0 DL (threaded) [usb] 100127 D - 0xfffffe010a9cf440 [usbus0] 100128 D - 0xfffffe010a9cf498 [usbus0] 100129 D - 0xfffffe010a9cf4f0 [usbus0] 100130 D - 0xfffffe010a9cf548 [usbus0] 100131 D - 0xfffffe010a9cf5a0 [usbus0] 100133 D - 0xfffffe00e4d9ccf0 [usbus1] 100134 D - 0xfffffe00e4d9cd48 [usbus1] 100135 D - 0xfffffe00e4d9cda0 [usbus1] 100136 D - 0xfffffe00e4d9cdf8 [usbus1] 100137 D - 0xfffffe00e4d9ce50 [usbus1] 100138 D - 0xfffffe010ad7ccf0 [usbus2] 100139 D - 0xfffffe010ad7cd48 [usbus2] 100140 D - 0xfffffe010ad7cda0 [usbus2] 100141 D - 0xfffffe010ad7cdf8 [usbus2] 100142 D - 0xfffffe010ad7ce50 [usbus2] 5 0 0 0 DL - 0xffffffff81cefba8 [busdma] 4 0 0 0 DL (threaded) [cam] 100120 D - 0xffffffff8199f240 [doneq0] 100121 D - 0xffffffff8199f2c0 [doneq1] 100122 D - 0xffffffff8199f340 [doneq2] 100123 D - 0xffffffff8199f1c0 [async] 100152 D - 0xffffffff8199f090 [scanner] 3 0 0 0 DL (threaded) [crypto] 100107 D crypto_ 0xffffffff81ccadb0 [crypto] 100108 D crypto_ 0xfffff8000429e830 [crypto returns 0] 100109 D crypto_ 0xfffff8000429e880 [crypto returns 1] 100110 D crypto_ 0xfffff8000429e8d0 [crypto returns 2] 100111 D crypto_ 0xfffff8000429e920 [crypto returns 3] 100112 D crypto_ 0xfffff8000429e970 [crypto returns 4] 100113 D crypto_ 0xfffff8000429e9c0 [crypto returns 5] 100114 D crypto_ 0xfffff8000429ea10 [crypto returns 6] 100115 D crypto_ 0xfffff8000429ea60 [crypto returns 7] 100116 D crypto_ 0xfffff8000429eab0 [crypto returns 8] 100117 D crypto_ 0xfffff8000429eb00 [crypto returns 9] 100118 D crypto_ 0xfffff8000429eb50 [crypto returns 10] 100119 D crypto_ 0xfffff8000429eba0 [crypto returns 11] 14 0 0 0 DL seqstat 0xfffff80004256488 [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100091 D - 0xffffffff81b87bb0 [g_event] 100092 D - 0xffffffff81b87bb8 [g_up] 100093 D - 0xffffffff81b87bc0 [g_down] 2 0 0 0 WL (threaded) [clock] 100077 I [clock (0)] 100078 I [clock (1)] 100079 I [clock (2)] 100080 I [clock (3)] 100081 I [clock (4)] 100082 I [clock (5)] 100083 I [clock (6)] 100084 I [clock (7)] 100085 I [clock (8)] 100086 I [clock (9)] 100087 I [clock (10)] 100088 I [clock (11)] 12 0 0 0 WL (threaded) [intr] 100042 I [swi6: task queue] 100043 I [swi6: Giant taskq] 100045 I [swi5: fast taskq] 100089 I [swi1: netisr 0] 100124 I [irq56: ahci0] 100126 I [irq19: xhci0] 100132 I [irq18: ehci0 ehci1+] 100143 I [irq71: ahci1] 100144 I [swi4: apei] 100145 I [swi0: uart uart] 100159 I [irq73: mlx5_core0] 100160 I [irq74: mlx5_core0] 100161 I [irq72: mlx5_core0] 100162 I [irq75: mlx5_core0] 100163 I [irq76: mlx5_core0] 100164 I [irq77: mlx5_core0] 100165 I [irq78: mlx5_core0] 100166 I [irq79: mlx5_core0] 100167 I [irq80: mlx5_core0] 100168 I [irq81: mlx5_core0] 100169 I [irq82: mlx5_core0] 100170 I [irq83: mlx5_core0] 100171 I [irq84: mlx5_core0] 100172 I [irq85: mlx5_core0] 100173 I [irq86: mlx5_core0] 100180 I [irq88: mlx5_core1] 100181 I [irq89: mlx5_core1] 100182 I [irq87: mlx5_core1] 100183 I [irq90: mlx5_core1] 100184 I [irq91: mlx5_core1] 100185 I [irq92: mlx5_core1] 100186 I [irq93: mlx5_core1] 100187 I [irq94: mlx5_core1] 100188 I [irq95: mlx5_core1] 100189 I [irq96: mlx5_core1] 100190 I [irq97: mlx5_core1] 100191 I [irq98: mlx5_core1] 100192 I [irq99: mlx5_core1] 100193 I [irq100: mlx5_core1] 100194 I [irq101: mlx5_core1] 100240 I [irq102: ioat0] 100241 I [irq103: ioat1] 100242 I [irq104: ioat2] 100243 I [irq105: ioat3] 100244 I [irq106: ioat4] 100245 I [irq107: ioat5] 100246 I [irq108: ioat6] 100247 I [irq109: ioat7] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffffe00e4c84040 [init] 10 0 0 0 DL audit_w 0xffffffff81ccb2b0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81b88120 [swapper] 100015 D - 0xfffff800039edd00 [softirq_0] 100016 D - 0xfffff800039edc00 [softirq_1] 100017 D - 0xfffff800039edb00 [softirq_2] 100018 D - 0xfffff800039eda00 [softirq_3] 100019 D - 0xfffff800039ed900 [softirq_4] 100020 D - 0xfffff800039ed800 [softirq_5] 100021 D - 0xfffff800039ed700 [softirq_6] 100022 D - 0xfffff800039ed600 [softirq_7] 100023 D - 0xfffff800039ed500 [softirq_8] 100024 D - 0xfffff800039ed400 [softirq_9] 100025 D - 0xfffff800039ed300 [softirq_10] 100026 D - 0xfffff800039ed200 [softirq_11] 100027 D - 0xfffff800039ed100 [if_io_tqg_0] 100028 D - 0xfffff800039ed000 [if_io_tqg_1] 100029 D - 0xfffff800039ece00 [if_io_tqg_2] 100030 D - 0xfffff800039ecd00 [if_io_tqg_3] 100031 D - 0xfffff800039ecc00 [if_io_tqg_4] 100032 D - 0xfffff800039ecb00 [if_io_tqg_5] 100033 D - 0xfffff800039eca00 [if_io_tqg_6] 100034 D - 0xfffff800039ec900 [if_io_tqg_7] 100035 D - 0xfffff800039ec800 [if_io_tqg_8] 100036 D - 0xfffff800039ec700 [if_io_tqg_9] 100037 D - 0xfffff800039ec600 [if_io_tqg_10] 100038 D - 0xfffff800039ec500 [if_io_tqg_11] 100039 D - 0xfffff800039ec400 [if_config_tqg_0] 100040 D - 0xfffff800039ec300 [pci_hp taskq] 100041 D - 0xfffff800039ec200 [kqueue_ctx taskq] 100044 D - 0xfffff80003b9c300 [thread taskq] 100046 D - 0xfffff80003b9c100 [aiod_kick taskq] 100047 D - 0xfffff80003b9c000 [deferred_unmount ta] 100048 D - 0xfffff80003b9de00 [inm_free taskq] 100049 D - 0xfffff80003b9dd00 [in6m_free taskq] 100050 D - 0xfffff80003b9dc00 [linuxkpi_irq_wq] 100051 D - 0xfffff80003b9db00 [linuxkpi_short_wq_0] 100052 D - 0xfffff80003b9db00 [linuxkpi_short_wq_1] 100053 D - 0xfffff80003b9db00 [linuxkpi_short_wq_2] 100054 D - 0xfffff80003b9db00 [linuxkpi_short_wq_3] 100055 D - 0xfffff80003b9db00 [linuxkpi_short_wq_4] 100056 D - 0xfffff80003b9db00 [linuxkpi_short_wq_5] 100057 D - 0xfffff80003b9db00 [linuxkpi_short_wq_6] 100058 D - 0xfffff80003b9db00 [linuxkpi_short_wq_7] 100059 D - 0xfffff80003b9db00 [linuxkpi_short_wq_8] 100060 D - 0xfffff80003b9db00 [linuxkpi_short_wq_9] 100061 D - 0xfffff80003b9db00 [linuxkpi_short_wq_1] 100062 D - 0xfffff80003b9db00 [linuxkpi_short_wq_1] 100063 D - 0xfffff80003b9db00 [linuxkpi_short_wq_1] 100064 D - 0xfffff80003b9da00 [linuxkpi_long_wq_0] 100065 D - 0xfffff80003b9da00 [linuxkpi_long_wq_1] 100066 D - 0xfffff80003b9da00 [linuxkpi_long_wq_2] 100067 D - 0xfffff80003b9da00 [linuxkpi_long_wq_3] 100068 D - 0xfffff80003b9da00 [linuxkpi_long_wq_4] 100069 D - 0xfffff80003b9da00 [linuxkpi_long_wq_5] 100070 D - 0xfffff80003b9da00 [linuxkpi_long_wq_6] 100071 D - 0xfffff80003b9da00 [linuxkpi_long_wq_7] 100072 D - 0xfffff80003b9da00 [linuxkpi_long_wq_8] 100073 D - 0xfffff80003b9da00 [linuxkpi_long_wq_9] 100074 D - 0xfffff80003b9da00 [linuxkpi_long_wq_10] 100075 D - 0xfffff80003b9da00 [linuxkpi_long_wq_11] 100076 D - 0xfffff80003b9da00 [linuxkpi_long_wq_12] 100090 D - 0xfffff80003b9d800 [firmware taskq] 100095 D - 0xfffff80003b9ec00 [crypto_0] 100096 D - 0xfffff80003b9ec00 [crypto_1] 100097 D - 0xfffff80003b9ec00 [crypto_2] 100098 D - 0xfffff80003b9ec00 [crypto_3] 100099 D - 0xfffff80003b9ec00 [crypto_4] 100100 D - 0xfffff80003b9ec00 [crypto_5] 100101 D - 0xfffff80003b9ec00 [crypto_6] 100102 D - 0xfffff80003b9ec00 [crypto_7] 100103 D - 0xfffff80003b9ec00 [crypto_8] 100104 D - 0xfffff80003b9ec00 [crypto_9] 100105 D - 0xfffff80003b9ec00 [crypto_10] 100106 D - 0xfffff80003b9ec00 [crypto_11] 100146 D - 0xfffff8000408a100 [acpi_task_0] 100147 D - 0xfffff8000408a100 [acpi_task_1] 100148 D - 0xfffff8000408a100 [acpi_task_2] 100150 D - 0xfffff800039ede00 [mca taskq] 100151 D - 0xfffff80003b9e500 [CAM taskq] 100155 D - 0xfffff8000408b400 [mlx5_core0-rec] 100156 D - 0xfffff8000408b300 [mlx5_core0-wdg] 100157 D - 0xfffff8000408b200 [mlx5_core0-cmd] 100158 D - 0xfffff8000408b100 [mlx5_page_allocator] 100174 D - 0xfffff8000408ce00 [mlx5_fc] 100175 D - 0xfffff8000408cd00 [mlx5_esw_wq] 100176 D - 0xfffff8000408cc00 [mlx5_core1-rec] 100177 D - 0xfffff8000408cb00 [mlx5_core1-wdg] 100178 D - 0xfffff8000408ca00 [mlx5_core1-cmd] 100179 D - 0xfffff8000408c900 [mlx5_page_allocator] 100195 D - 0xfffff80003d19e00 [mlx5_fc] 100196 D - 0xfffff80003d19d00 [mlx5_esw_wq] 100396 D - 0xfffff800046fd800 [unionfs_rele taskq] db:0:pho> allt Tracing command kernel pid 0 tid 100000 td 0xffffffff81b88680 sched_switch() at sched_switch+0x5d2/frame 0xffffffff8237de10 mi_switch() at mi_switch+0x170/frame 0xffffffff8237de30 sleepq_switch() at sleepq_switch+0x109/frame 0xffffffff8237de70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xffffffff8237deb0 _sleep() at _sleep+0x27e/frame 0xffffffff8237df50 swapper() at swapper+0x75/frame 0xffffffff8237dfa0 mi_startup() at mi_startup+0x226/frame 0xffffffff8237dff0 Tracing command kernel pid 0 tid 100015 td 0xfffff80003be2740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108050df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108050e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108050e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108050ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108050ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108050f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108050f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffff80003be2000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010804bdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010804be10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010804be50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010804bec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010804bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010804bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010804bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xfffff80003bb5740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010805adf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010805ae10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010805ae50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010805aec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010805aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010805af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010805af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffff80003bb5000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108055df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108055e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108055e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108055ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108055ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108055f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108055f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffff80003bb6740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108064df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108064e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108064e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108064ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108064ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108064f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108064f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffff80003bb6000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010805fdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010805fe10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010805fe50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010805fec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010805fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010805ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010805ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffff80003bb7740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010806edf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010806ee10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010806ee50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010806eec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010806eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010806ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010806ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffff80003bb7000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108069df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108069e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108069e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108069ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108069ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108069f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108069f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffff80003bb8740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108078df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108078e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108078e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108078ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108078ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108078f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108078f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffff80003bb8000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108073df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108073e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108073e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108073ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108073ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108073f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108073f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffff80003bb1740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108082df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108082e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108082e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108082ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108082ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108082f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108082f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffff80003bb1000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010807ddf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010807de10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010807de50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010807dec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010807def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010807df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010807df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffff80003bb2740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010808cdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010808ce10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010808ce50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010808cec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010808cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010808cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010808cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffff80003bb2000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108087df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108087e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108087e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108087ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108087ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108087f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108087f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffff80003bb3740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108096df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108096e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108096e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108096ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108096ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108096f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108096f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffff80003bb3000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108091df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108091e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108091e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe0108091ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe0108091ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108091f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108091f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffff80003bb4740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080a0df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080a0e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080a0e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080a0ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080a0ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080a0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080a0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffff80003bb4000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010809bdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010809be10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010809be50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe010809bec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe010809bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010809bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010809bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffff80003bab740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080aadf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080aae10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080aae50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080aaec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080aaef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080aaf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080aaf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffff80003bab000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080a5df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080a5e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080a5e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080a5ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080a5ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080a5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080a5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffff80003bac740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080b4df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080b4e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080b4e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080b4ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080b4ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080b4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080b4f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffff80003bac000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080afdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080afe10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080afe50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080afec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080afef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080aff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080aff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffff80003bad740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080bedf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080bee10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080bee50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080beec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080beef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080bef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080bef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffff80003bad000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080b9df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080b9e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080b9e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080b9ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080b9ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080b9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080b9f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffff80003bae740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080c8df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080c8e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080c8e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080c8ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x104/frame 0xfffffe01080c8ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080c8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080c8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffff80003bae000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080c3dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080c3de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080c3e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080c3ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080c3ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080c3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080c3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffff80003ba7740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080d2dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080d2de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080d2e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080d2ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080d2ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080d2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080d2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffff80003ba8000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080d7dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080d7de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080d7e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080d7ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080d7ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080d7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080d7f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xfffff80003ba9000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080e1dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080e1de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080e1e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080e1ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080e1ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080e1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080e1f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100047 td 0xfffff80003baa740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080f0dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080f0de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080f0e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080f0ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080f0ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080f0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080f0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100048 td 0xfffff80003baa000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080ebdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080ebde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080ebe20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080ebec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080ebef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080ebf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080ebf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100049 td 0xfffff80003b98740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080fadc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080fade0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080fae20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080faec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080faef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080faf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080faf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100050 td 0xfffff80003b98000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080f5df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080f5e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080f5e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01080f5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0x104/frame 0xfffffe01080f5ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080f5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080f5f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100051 td 0xfffff80003b99740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108104dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108104de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108104e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108104ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108104ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108104f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108104f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100052 td 0xfffff80003b99000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080ffdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01080ffde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01080ffe20 _sleep() at _sleep+0x2d8/frame 0xfffffe01080ffec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01080ffef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080fff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080fff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100053 td 0xfffff80003b9a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010810edc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010810ede0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010810ee20 _sleep() at _sleep+0x2d8/frame 0xfffffe010810eec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010810eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010810ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010810ef30 --- trap 0xdb562d60, rip = 0xd3833b52806a24d1, rsp = 0xc017ea3c0e849163, rbp = 0x4cbc887aeae97d87 --- Tracing command kernel pid 0 tid 100054 td 0xfffff80003b9a000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108109dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108109de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108109e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108109ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108109ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108109f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108109f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100055 td 0xfffff80003b9b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108118dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108118de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108118e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108118ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108118ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108118f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108118f30 --- trap 0x4f749a2a, rip = 0x56c451a51bbbdcb3, rsp = 0x231de2dc42a1381a, rbp = 0xa33f8752a0ebe3ae --- Tracing command kernel pid 0 tid 100056 td 0xfffff80003b9b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108113dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108113de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108113e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108113ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108113ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108113f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108113f30 --- trap 0x9ca4db72, rip = 0xd42ee0608e18736a, rsp = 0x819291fbd7b90688, rbp = 0xef795bbdcfa5fb4c --- Tracing command kernel pid 0 tid 100057 td 0xfffff80003b91740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108122dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108122de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108122e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108122ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108122ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108122f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108122f30 --- trap 0xeb1cc483, rip = 0x297cf6855959f08b, rsp = 0x1c75ff147cf685f0, rbp = 0xfffffeb3e8c68bf8 --- ??() at 0x297cf6855959f08b/frame 0xfffffeb3e8c68bf8 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0108014280 kdb_reenter() at kdb_reenter+0x2b/frame 0xfffffe0108014290 calltrap() at calltrap+0x8/frame 0xfffffe0108014290 --- trap 0xc, rip = 0xffffffff8103aec0, rsp = 0xfffffe0108014360, rbp = 0xfffffe01080143f0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe01080143f0 db_get_value() at db_get_value+0x31/frame 0xfffffe0108014430 db_backtrace() at db_backtrace+0x25a/frame 0xfffffe01080144b0 db_trace_thread() at db_trace_thread+0x3a/frame 0xfffffe01080144d0 _db_stack_trace_all() at _db_stack_trace_all+0xfd/frame 0xfffffe0108014570 db_command() at db_command+0x30d/frame 0xfffffe0108014630 db_script_exec() at db_script_exec+0xd8/frame 0xfffffe01080146e0 db_command() at db_command+0x30d/frame 0xfffffe01080147a0 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe01080147b0 db_trap() at db_trap+0xe6/frame 0xfffffe0108014840 kdb_trap() at kdb_trap+0x24f/frame 0xfffffe0108014990 trap() at trap+0x2c9/frame 0xfffffe0108014ab0 calltrap() at calltrap+0x8/frame 0xfffffe0108014ab0 --- trap 0x3, rip = 0xffffffff80b9679d, rsp = 0xfffffe0108014b80, rbp = 0xfffffe0108014b90 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x14d/frame 0xfffffe0108014b90 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0108014ba0 uart_intr_rxready() at uart_intr_rxready+0xf4/frame 0xfffffe0108014bd0 uart_intr() at uart_intr+0x118/frame 0xfffffe0108014c10 intr_event_handle() at intr_event_handle+0xc8/frame 0xfffffe0108014c80 intr_execute_handlers() at intr_execute_handlers+0x63/frame 0xfffffe0108014cb0 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe0108014cb0 --- interrupt, rip = 0xffffffff804c551e, rsp = 0xfffffe0108014d80, rbp = 0xfffffe0108014db0 --- acpi_cpu_idle() at acpi_cpu_idle+0x27e/frame 0xfffffe0108014db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108014dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108014df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108014ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108014f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108014f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffeb3e8c68c00 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01080143e0 kdb_reenter() at kdb_reenter+0x2b/frame 0xfffffe01080143f0 db_get_value() at db_get_value+0x4b/frame 0xfffffe0108014430 db_backtrace() at db_backtrace+0x25a/frame 0xfffffe01080144b0 db_trace_thread() at db_trace_thread+0x3a/frame 0xfffffe01080144d0 _db_stack_trace_all() at _db_stack_trace_all+0xfd/frame 0xfffffe0108014570 db_command() at db_command+0x30d/frame 0xfffffe0108014630 db_script_exec() at db_script_exec+0xd8/frame 0xfffffe01080146e0 db_command() at db_command+0x30d/frame 0xfffffe01080147a0 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe01080147b0 db_trap() at db_trap+0xe6/frame 0xfffffe0108014840 kdb_trap() at kdb_trap+0x24f/frame 0xfffffe0108014990 trap() at trap+0x2c9/frame 0xfffffe0108014ab0 calltrap() at calltrap+0x8/frame 0xfffffe0108014ab0 --- trap 0x3, rip = 0xffffffff80b9679d, rsp = 0xfffffe0108014b80, rbp = 0xfffffe0108014b90 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x14d/frame 0xfffffe0108014b90 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0108014ba0 uart_intr_rxready() at uart_intr_rxready+0xf4/frame 0xfffffe0108014bd0 uart_intr() at uart_intr+0x118/frame 0xfffffe0108014c10 intr_event_handle() at intr_event_handle+0xc8/frame 0xfffffe0108014c80 intr_execute_handlers() at intr_execute_handlers+0x63/frame 0xfffffe0108014cb0 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe0108014cb0 --- interrupt, rip = 0xffffffff804c551e, rsp = 0xfffffe0108014d80, rbp = 0xfffffe0108014db0 --- acpi_cpu_idle() at acpi_cpu_idle+0x27e/frame 0xfffffe0108014db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108014dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108014df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108014ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108014f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108014f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xfffff80003b91000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010811ddc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010811dde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010811de20 _sleep() at _sleep+0x2d8/frame 0xfffffe010811dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010811def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010811df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010811df30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100059 td 0xfffff80003b95740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010812cdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010812cde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010812ce20 _sleep() at _sleep+0x2d8/frame 0xfffffe010812cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010812cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010812cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010812cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100060 td 0xfffff80003b95000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108127dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108127de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108127e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108127ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108127ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108127f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108127f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100061 td 0xfffff80003b96740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108136dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108136de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108136e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108136ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108136ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108136f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108136f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xc8664 --- Tracing command kernel pid 0 tid 100062 td 0xfffff80003b96000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108131dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108131de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108131e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108131ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108131ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108131f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108131f30 --- trap 0x8803458a, rip = 0x2458a0745880376, rsp = 0x820f02037d800345, rbp = 0x76005d38085d8a00 --- Tracing command kernel pid 0 tid 100063 td 0xfffff80003b97740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108140dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108140de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108140e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108140ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108140ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108140f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108140f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xeb3096c48a93e6d6 --- Tracing command kernel pid 0 tid 100064 td 0xfffff80003b97000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010813bdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010813bde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010813be20 _sleep() at _sleep+0x2d8/frame 0xfffffe010813bec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010813bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010813bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010813bf30 --- trap 0xaefb331a, rip = 0xdef7467b3e49ee3a, rsp = 0xa12524c30b47a7a1, rbp = 0x966e70d4c7a4a507 --- Tracing command kernel pid 0 tid 100065 td 0xfffff80003b89740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010814adc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010814ade0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010814ae20 _sleep() at _sleep+0x2d8/frame 0xfffffe010814aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010814aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010814af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010814af30 --- trap 0xad6fff29, rip = 0x72d058b8857a22bd, rsp = 0x224b9b576cd043e, rbp = 0xe5fb780639307b8f --- Tracing command kernel pid 0 tid 100066 td 0xfffff80003b89000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108145dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108145de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108145e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108145ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108145ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108145f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108145f30 --- trap 0x1e8510c9, rip = 0xefb31a2da6765e70, rsp = 0xfe236f877ae5751b, rbp = 0x479532f003de01d --- Tracing command kernel pid 0 tid 100067 td 0xfffff80003b8a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108154dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108154de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108154e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108154ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108154ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108154f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108154f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100068 td 0xfffff80003b8a000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010814fdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010814fde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010814fe20 _sleep() at _sleep+0x2d8/frame 0xfffffe010814fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010814fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010814ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010814ff30 --- trap 0xe9003090, rip = 0xe900169000004100, rsp = 0xe9002c08e0000100, rbp = 0xbfffffff0 --- Tracing command kernel pid 0 tid 100069 td 0xfffff80003b8b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010815edc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010815ede0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010815ee20 _sleep() at _sleep+0x2d8/frame 0xfffffe010815eec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010815eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010815ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010815ef30 --- trap 0x2b35353f, rip = 0x707070707070707, rsp = 0x2c32393f002e0701, rbp = 0x2b35353f2b35353f --- Tracing command kernel pid 0 tid 100070 td 0xfffff80003b8b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108159dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108159de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108159e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108159ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108159ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108159f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108159f30 --- trap 0x1070103, rip = 0xfff57428fff57430, rsp = 0xfff573f8fff57400, rbp = 0x100010601060001 --- Tracing command kernel pid 0 tid 100071 td 0xfffff80003b8c740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108168dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108168de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108168e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108168ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108168ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108168f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108168f30 --- trap 0x1, rip = 0x16b5ad6bc0000000, rsp = 0x316b5ad6b, rbp = 0x120046ec00000004 --- Tracing command kernel pid 0 tid 100072 td 0xfffff80003b8c000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108163dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108163de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108163e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108163ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108163ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108163f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108163f30 --- trap 0xa0a0a00, rip = 0x2929292929290029, rsp = 0x2929292929292929, rbp = 0xb0b0b0b0b0b000b --- Tracing command kernel pid 0 tid 100073 td 0xfffff80003b85740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108172dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108172de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108172e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108172ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108172ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108172f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108172f30 --- trap 0x16000203, rip = 0x2525000303ff2525, rsp = 0x3ff494907000803, rbp = 0x3ff1d1d16010003 --- Tracing command kernel pid 0 tid 100074 td 0xfffff80003b85000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010816ddc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010816dde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010816de20 _sleep() at _sleep+0x2d8/frame 0xfffffe010816dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010816def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010816df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010816df30 --- trap 0x10272900, rip = 0x1b10150162a2b01, rsp = 0x56182c2d00f7122e, rbp = 0x1620e2122002c82 --- Tracing command kernel pid 0 tid 100075 td 0xfffff80003b86740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010817cdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010817cde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010817ce20 _sleep() at _sleep+0x2d8/frame 0xfffffe010817cec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010817cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010817cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010817cf30 --- trap 0x5988fc58, rip = 0x868ad9754a411259, rsp = 0x868a00000ce68788, rbp = 0x5d5b5e5fb9750c4d --- Tracing command kernel pid 0 tid 100076 td 0xfffff80003b86000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108177dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108177de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108177e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108177ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108177ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108177f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108177f30 --- trap 0x1516, rip = 0x14d3e8c78b52d3, rsp = 0xf000014dbe8c78b, rbp = 0x450908e0c1c0b60f --- Tracing command kernel pid 0 tid 100090 td 0xfffff80003c92000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081bddc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081bdde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081bde20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081bdec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081bdef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081bdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081bdf30 --- trap 0x75ff0002, rip = 0xcc483fffd4b76e8, rsp = 0xc033db335708758b, rbp = 0xc7400022099863a --- Tracing command kernel pid 0 tid 100095 td 0xfffff80003c95740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081e0dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081e0de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081e0e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081e0ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081e0ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081e0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081e0f30 --- trap 0x404d8ba0, rip = 0xbac083110166ba50, rsp = 0x42d2336775ff3045, rbp = 0x6375ff6b75ff52f9 --- Tracing command kernel pid 0 tid 100096 td 0xfffff80003c95000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081dbdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081dbde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081dbe20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081dbec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081dbef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081dbf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081dbf30 --- trap 0xc0b60fc0, rip = 0xf04707d8000001c, rsp = 0xe856576875ff6c75, rbp = 0x74c0843cc483fffd --- Tracing command kernel pid 0 tid 100097 td 0xfffff80004328740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081eadc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081eade0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081eae20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081eaec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081eaef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081eaf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081eaf30 --- trap 0x8d374688, rip = 0x9968500b45, rsp = 0x68500b458d394688, rbp = 0x9668500b458d --- Tracing command kernel pid 0 tid 100098 td 0xfffff80004328000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081e5dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081e5de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081e5e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081e5ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081e5ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081e5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081e5f30 --- trap 0x820f04f8, rip = 0xe9e8560875fff875, rsp = 0x48ffffff490784b6, rbp = 0x183830c483cc458b --- Tracing command kernel pid 0 tid 100099 td 0xfffff80004329740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081f4dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081f4de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081f4e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081f4ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081f4ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081f4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081f4f30 --- trap 0x72c83b5b, rip = 0x8bc3c9c0331cc483, rsp = 0x1e73085d8b530218, rbp = 0x81f845fffc758966 --- Tracing command kernel pid 0 tid 100100 td 0xfffff80004329000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081efdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081efde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081efe20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081efec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081efef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081eff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081eff30 --- trap 0x8a327400, rip = 0x330eeb0000009d97, rsp = 0xad75144dff000005, rbp = 0x2ae8000000a98789 --- Tracing command kernel pid 0 tid 100101 td 0xfffff8000432a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081fedc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081fede0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081fee20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081feec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081feef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081fef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081fef30 --- trap 0xd0ec8194, rip = 0x7c7d8bfffff71ce8, rsp = 0xe89c458d5659ffff, rbp = 0x45b60ffe45fe0cc4 --- Tracing command kernel pid 0 tid 100102 td 0xfffff8000432a000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081f9dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081f9de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081f9e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081f9ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01081f9ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081f9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081f9f30 --- trap 0x6e3c1d8, rip = 0x498a118ac803dc7d, rsp = 0x8b0a88d4558b1788, rbp = 0xe307c800146b60f --- Tracing command kernel pid 0 tid 100103 td 0xfffff8000432b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108208dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108208de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108208e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108208ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108208ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108208f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108208f30 --- trap 0x18c483d2, rip = 0xdb33ff03fb0bd923, rsp = 0xc70b04e7c1fa0bff, rbp = 0x4db60f187db60fff --- Tracing command kernel pid 0 tid 100104 td 0xfffff8000432b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108203dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108203de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108203e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108203ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108203ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108203f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108203f30 --- trap 0xec8b55c3, rip = 0xc75ff0875ff1875, rsp = 0x75ff560c408b0845, rbp = 0xc95bc0335e714688 --- Tracing command kernel pid 0 tid 100105 td 0xfffff80004324740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108212dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108212de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108212e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108212ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108212ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108212f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108212f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xfffff80004324000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010820ddc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010820dde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010820de20 _sleep() at _sleep+0x2d8/frame 0xfffffe010820dec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010820def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010820df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010820df30 --- trap 0x10244c8b, rip = 0xb60f02e8c034244c, rsp = 0xcab60fd984098a14, rbp = 0x968e8d000004198e --- Tracing command kernel pid 0 tid 100146 td 0xfffff80043897740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082e4df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082e4e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082e4e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01082e4ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0x104/frame 0xfffffe01082e4ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082e4f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082e4f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100147 td 0xfffff80043897000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082dfdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082dfe10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082dfe50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01082dfec0 taskqueue_thread_loop() at taskqueue_thread_loop+0x104/frame 0xfffffe01082dfef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082dff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082dff30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100148 td 0xfffff80043898740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082eedf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082eee10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082eee50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01082eeec0 taskqueue_thread_loop() at taskqueue_thread_loop+0x104/frame 0xfffffe01082eeef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082eef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082eef30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100150 td 0xfffff80043899740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082f8df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082f8e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082f8e50 msleep_spin_sbt() at msleep_spin_sbt+0x154/frame 0xfffffe01082f8ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0x104/frame 0xfffffe01082f8ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082f8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082f8f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100151 td 0xfffff80043899000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082f3dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082f3de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082f3e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01082f3ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01082f3ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082f3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082f3f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100155 td 0xfffff800438a1000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082fddc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082fdde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082fde20 _sleep() at _sleep+0x2d8/frame 0xfffffe01082fdec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01082fdef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082fdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082fdf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100156 td 0xfffff8004384f740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108316dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108316de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108316e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108316ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108316ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108316f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108316f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100157 td 0xfffff8004384f000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108311dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108311de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108311e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108311ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108311ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108311f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108311f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100158 td 0xfffff80043850740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108320dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108320de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108320e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108320ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108320ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108320f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108320f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100174 td 0xfffff80049e2c740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108370dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108370de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108370e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108370ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108370ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108370f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108370f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100175 td 0xfffff80049e2c000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010836bdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010836bde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010836be20 _sleep() at _sleep+0x2d8/frame 0xfffffe010836bec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010836bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010836bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010836bf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100176 td 0xfffff80049e2d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010837adc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010837ade0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010837ae20 _sleep() at _sleep+0x2d8/frame 0xfffffe010837aec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010837aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010837af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010837af30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100177 td 0xfffff80049e2d000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108375dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108375de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108375e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108375ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108375ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108375f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108375f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100178 td 0xfffff80049e2e740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108384dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108384de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108384e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108384ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe0108384ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108384f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108384f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100179 td 0xfffff80049e2e000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010837fdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010837fde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010837fe20 _sleep() at _sleep+0x2d8/frame 0xfffffe010837fec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe010837fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010837ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010837ff30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command kernel pid 0 tid 100195 td 0xfffff80003d22740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083dedc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01083dede0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083dee20 _sleep() at _sleep+0x2d8/frame 0xfffffe01083deec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01083deef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083def30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083def30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100196 td 0xfffff80003d22000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083d9dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01083d9de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083d9e20 _sleep() at _sleep+0x2d8/frame 0xfffffe01083d9ec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01083d9ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083d9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083d9f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100396 td 0xfffff803b884c740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01086ccdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01086ccde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01086cce20 _sleep() at _sleep+0x2d8/frame 0xfffffe01086ccec0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe01086ccef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01086ccf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01086ccf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff80003bf3000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108005a10 mi_switch() at mi_switch+0x170/frame 0xfffffe0108005a30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108005a70 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108005ad0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108005ae0 _sleep() at _sleep+0x2ce/frame 0xfffffe0108005b80 kern_wait6() at kern_wait6+0x471/frame 0xfffffe0108005c10 sys_wait4() at sys_wait4+0x6b/frame 0xfffffe0108005e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108005f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108005f30 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x2dd08a, rsp = 0x820b71ee8, rbp = 0x820b71f50 --- Tracing command clock pid 2 tid 100077 td 0xfffff80003b87740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108186ea0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108186ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe0108186ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108186f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108186f30 --- trap 0x1b0c35d, rip = 0x7d830cc483ffffed, rsp = 0x8bc2b60fd2320cc4, rbp = 0x33d88b008b10458b --- Tracing command clock pid 2 tid 100078 td 0xfffff80003b87000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108181ea0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108181ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe0108181ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108181f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108181f30 --- trap 0xf8728908, rip = 0x9ce0bcf0b0a8b1a, rsp = 0x8b9e75f04dff0cc2, rbp = 0xa0458dfc5d8918c4 --- Tracing command clock pid 2 tid 100079 td 0xfffff80003b88740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108190ea0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108190ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe0108190ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108190f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108190f30 --- trap 0x45c6d772, rip = 0x8300048e78e85653, rsp = 0x5750fde08300048e, rbp = 0x75c0840cc483ffff --- Tracing command clock pid 2 tid 100080 td 0xfffff80003b88000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010818bea0 mi_switch() at mi_switch+0x170/frame 0xfffffe010818bec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe010818bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010818bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010818bf30 --- trap 0x6a0a01, rip = 0xfe54820f04fa7d80, rsp = 0x65830cec83ec8b55, rbp = 0x331ee0c1d0f7c0b6 --- Tracing command clock pid 2 tid 100081 td 0xfffff80003c96740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010819aea0 mi_switch() at mi_switch+0x170/frame 0xfffffe010819aec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe010819aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010819af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010819af30 --- trap 0x276b93c, rip = 0xb8f8b70ffffffc4a, rsp = 0xff045af0f667f45, rbp = 0x975c08418eb0000 --- Tracing command clock pid 2 tid 100082 td 0xfffff80003c96000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108195ea0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108195ec0 softclock_thread() at softclock_thread+0x6e/frame 0xfffffe0108195ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108195f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108195f30 --- trap 0x750a3e44, rip = 0x5010c483ffffefa4, rsp = 0xdfffffd46581d045, rbp = 0xff1c75ff0d024240 --- Tracing command clock pid 2 tid 100083 td 0xfffff80003c97740 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100084 td 0xfffff80003c97000 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100085 td 0xfffff80003c98740 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100086 td 0xfffff80003c98000 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100087 td 0xfffff80003c99740 fork_trampoline() at fork_trampoline Tracing command clock pid 2 tid 100088 td 0xfffff80003c99000 fork_trampoline() at fork_trampoline Tracing command crypto pid 3 tid 100107 td 0xfffff80004325740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010821cdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010821cde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010821ce20 _sleep() at _sleep+0x2d8/frame 0xfffffe010821cec0 crypto_dispatch_thread() at crypto_dispatch_thread+0x193/frame 0xfffffe010821cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010821cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010821cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command crypto pid 3 tid 100108 td 0xfffff80004325000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108217db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108217dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108217e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108217eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108217ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108217f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108217f30 --- trap 0xef75d285, rip = 0x89fc752b0574c085, rsp = 0x66ee75c9840e8af8, rbp = 0xc0b60f0424c18b09 --- Tracing command crypto pid 3 tid 100109 td 0xfffff80004326740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108226db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108226dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108226e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108226eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108226ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108226f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108226f30 --- trap 0x54ee900, rip = 0x2df8832c24448910, rsp = 0x8300000124840f00, rbp = 0x1182444c7 --- Tracing command crypto pid 3 tid 100110 td 0xfffff80004326000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108221db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108221dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108221e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108221eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108221ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108221f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108221f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command crypto pid 3 tid 100111 td 0xfffff80004327740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108230db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108230dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108230e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108230eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108230ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108230f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108230f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0x8c82 --- Tracing command crypto pid 3 tid 100112 td 0xfffff80004327000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010822bdb0 mi_switch() at mi_switch+0x170/frame 0xfffffe010822bdd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010822be10 _sleep() at _sleep+0x2d8/frame 0xfffffe010822beb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe010822bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010822bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010822bf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command crypto pid 3 tid 100113 td 0xfffff8000431a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010823adb0 mi_switch() at mi_switch+0x170/frame 0xfffffe010823add0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010823ae10 _sleep() at _sleep+0x2d8/frame 0xfffffe010823aeb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe010823aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010823af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010823af30 --- trap 0x4d8bd00b, rip = 0x8d8b07e8c1ffffff, rsp = 0xffffff78b58b12e9, rbp = 0xfffffee88d89c803 --- Tracing command crypto pid 3 tid 100114 td 0xfffff8000431a000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108235db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108235dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108235e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108235eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108235ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108235f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108235f30 --- trap 0xff25, rip = 0xcb888808458b00e2, rsp = 0x45c708c483ffffe6, rbp = 0xc8828808558b --- Tracing command crypto pid 3 tid 100115 td 0xfffff8000431b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108244db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108244dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108244e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108244eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108244ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108244f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108244f30 --- trap 0x1, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0x30646870 --- Tracing command crypto pid 3 tid 100116 td 0xfffff8000431b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010823fdb0 mi_switch() at mi_switch+0x170/frame 0xfffffe010823fdd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010823fe10 _sleep() at _sleep+0x2d8/frame 0xfffffe010823feb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe010823fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010823ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010823ff30 --- trap 0x8ba84d8b, rip = 0x558b044a3318498b, rsp = 0xb0558b244189a84d, rbp = 0xe1c1b44d8b1be8c1 --- Tracing command crypto pid 3 tid 100117 td 0xfffff8000431c740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010824edb0 mi_switch() at mi_switch+0x170/frame 0xfffffe010824edd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010824ee10 _sleep() at _sleep+0x2d8/frame 0xfffffe010824eeb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe010824eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010824ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010824ef30 --- trap 0xb589ffff, rip = 0x8d89fffffde4b589, rsp = 0xfffdf4b589fffffd, rbp = 0xfec4b589fffffef0 --- Tracing command crypto pid 3 tid 100118 td 0xfffff8000431c000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108249db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108249dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108249e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108249eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108249ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108249f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108249f30 --- trap 0, rip = 0x2, rsp = 0x21e09, rbp = 0x21409 --- Tracing command crypto pid 3 tid 100119 td 0xfffff8000431d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108258db0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108258dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108258e10 _sleep() at _sleep+0x2d8/frame 0xfffffe0108258eb0 crypto_ret_thread() at crypto_ret_thread+0x144/frame 0xfffffe0108258ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108258f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108258f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command cam pid 4 tid 100120 td 0xfffff8000431d000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108253da0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108253dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108253e00 _sleep() at _sleep+0x2d8/frame 0xfffffe0108253ea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0108253ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108253f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108253f30 --- trap 0x29934c8d, rip = 0x4e0, rsp = 0xafafafafafafafaf, rbp = 0x66629d8b627c2b77 --- Tracing command cam pid 4 tid 100121 td 0xfffff800043fa740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108262da0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108262dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108262e00 _sleep() at _sleep+0x2d8/frame 0xfffffe0108262ea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0108262ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108262f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108262f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0x690062006d0053 --- Tracing command cam pid 4 tid 100122 td 0xfffff800043fa000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010825dda0 mi_switch() at mi_switch+0x170/frame 0xfffffe010825ddc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010825de00 _sleep() at _sleep+0x2d8/frame 0xfffffe010825dea0 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe010825def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010825df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010825df30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command cam pid 4 tid 100123 td 0xfffff800043ff740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010826cda0 mi_switch() at mi_switch+0x170/frame 0xfffffe010826cdc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010826ce00 _sleep() at _sleep+0x2d8/frame 0xfffffe010826cea0 xpt_async_td() at xpt_async_td+0x74/frame 0xfffffe010826cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010826cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010826cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafaf0000 --- Tracing command cam pid 4 tid 100152 td 0xfffff800438a1740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108302dc0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108302de0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108302e20 _sleep() at _sleep+0x2d8/frame 0xfffffe0108302ec0 xpt_scanner_thread() at xpt_scanner_thread+0x138/frame 0xfffffe0108302ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108302f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108302f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command busdma pid 5 tid 100125 td 0xfffff800048d8000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108271da0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108271dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108271e00 _sleep() at _sleep+0x2d8/frame 0xfffffe0108271ea0 busdma_thread() at busdma_thread+0xc5/frame 0xfffffe0108271ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108271f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108271f30 --- trap 0xabeb0810, rip = 0x8eb185d89fc438b, rsp = 0x8bde030a75ff85ff, rbp = 0x302444891455892c --- Tracing command rand_harvestq pid 6 tid 100149 td 0xfffff80043898000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082e9d20 mi_switch() at mi_switch+0x170/frame 0xfffffe01082e9d40 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082e9d80 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01082e9dc0 _sleep() at _sleep+0x27e/frame 0xfffffe01082e9e60 random_kthread() at random_kthread+0xb7/frame 0xfffffe01082e9ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082e9f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082e9f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command enc_daemon0 pid 7 tid 100153 td 0xfffff80043750740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010830cd80 mi_switch() at mi_switch+0x170/frame 0xfffffe010830cda0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010830cde0 _sleep() at _sleep+0x2d8/frame 0xfffffe010830ce80 enc_daemon() at enc_daemon+0xdd/frame 0xfffffe010830cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010830cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010830cf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command enc_daemon1 pid 8 tid 100154 td 0xfffff80043750000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108307d80 mi_switch() at mi_switch+0x170/frame 0xfffffe0108307da0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108307de0 _sleep() at _sleep+0x2d8/frame 0xfffffe0108307e80 enc_daemon() at enc_daemon+0xdd/frame 0xfffffe0108307ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108307f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108307f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command pagedaemon pid 9 tid 100197 td 0xfffff8004e5b8740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083e89b0 mi_switch() at mi_switch+0x170/frame 0xfffffe01083e89d0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083e8a10 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01083e8a50 _sleep() at _sleep+0x27e/frame 0xfffffe01083e8af0 vm_pageout_worker() at vm_pageout_worker+0x28c/frame 0xfffffe01083e8eb0 vm_pageout() at vm_pageout+0x1d7/frame 0xfffffe01083e8ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083e8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083e8f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100198 td 0xfffff8004e559000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108389a70 mi_switch() at mi_switch+0x170/frame 0xfffffe0108389a90 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108389ad0 _sleep() at _sleep+0x2d8/frame 0xfffffe0108389b70 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xb52/frame 0xfffffe0108389ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108389f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108389f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100199 td 0xfffff8004e55a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083f2d80 mi_switch() at mi_switch+0x170/frame 0xfffffe01083f2da0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083f2de0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01083f2e20 _sleep() at _sleep+0x27e/frame 0xfffffe01083f2ec0 uma_reclaim_worker() at uma_reclaim_worker+0xc8/frame 0xfffffe01083f2ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083f2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083f2f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command audit pid 10 tid 100001 td 0xfffff80003bf3740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010800adc0 mi_switch() at mi_switch+0x170/frame 0xfffffe010800ade0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010800ae20 _cv_wait() at _cv_wait+0x160/frame 0xfffffe010800ae90 audit_worker() at audit_worker+0x8f/frame 0xfffffe010800aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010800af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010800af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff80003bf4740 (CPU 0) kdb_alt_break_internal() at kdb_alt_break_internal+0x14d/frame 0xfffffe0108014b90 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe0108014ba0 uart_intr_rxready() at uart_intr_rxready+0xf4/frame 0xfffffe0108014bd0 uart_intr() at uart_intr+0x118/frame 0xfffffe0108014c10 intr_event_handle() at intr_event_handle+0xc8/frame 0xfffffe0108014c80 intr_execute_handlers() at intr_execute_handlers+0x63/frame 0xfffffe0108014cb0 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe0108014cb0 --- interrupt, rip = 0xffffffff804c551e, rsp = 0xfffffe0108014d80, rbp = 0xfffffe0108014db0 --- acpi_cpu_idle() at acpi_cpu_idle+0x27e/frame 0xfffffe0108014db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108014dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108014df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108014ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108014f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108014f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff80003bf4000 (CPU 1) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce3cdf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce3ce00 trap() at trap+0x56/frame 0xfffffe003ce3cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce3cf20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010800fd78, rbp = 0xfffffe010800fdb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010800fdb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010800fdd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010800fdf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010800fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010800ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010800ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff80003bf5740 (CPU 2) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce4bdf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce4be00 trap() at trap+0x56/frame 0xfffffe003ce4bf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce4bf20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010801ed78, rbp = 0xfffffe010801edb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010801edb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010801edd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010801edf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010801eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010801ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010801ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff80003bf5000 (CPU 3) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce5adf0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce5ae00 trap() at trap+0x56/frame 0xfffffe003ce5af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce5af20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108019d78, rbp = 0xfffffe0108019db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108019db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108019dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108019df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108019ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108019f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108019f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff80003bf6740 (CPU 4) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce69df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce69e00 trap() at trap+0x56/frame 0xfffffe003ce69f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce69f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108028d78, rbp = 0xfffffe0108028db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108028db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108028dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108028df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108028ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108028f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108028f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff80003bf6000 (CPU 5) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce78df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce78e00 trap() at trap+0x56/frame 0xfffffe003ce78f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce78f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108023d78, rbp = 0xfffffe0108023db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108023db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108023dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108023df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108023ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108023f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108023f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff80003bdf740 (CPU 6) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce87df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce87e00 trap() at trap+0x56/frame 0xfffffe003ce87f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce87f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108032d78, rbp = 0xfffffe0108032db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108032db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108032dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108032df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108032ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108032f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108032f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff80003bdf000 (CPU 7) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ce96df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ce96e00 trap() at trap+0x56/frame 0xfffffe003ce96f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ce96f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010802dd78, rbp = 0xfffffe010802ddb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010802ddb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010802ddd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010802ddf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010802def0 fork_exit() at fork_exit+0x82/frame 0xfffffe010802df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010802df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff80003be0740 (CPU 8) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003cea5df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003cea5e00 trap() at trap+0x56/frame 0xfffffe003cea5f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cea5f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe010803cd78, rbp = 0xfffffe010803cdb0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe010803cdb0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe010803cdd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe010803cdf0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe010803cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010803cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010803cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff80003be0000 (CPU 9) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ceb4df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ceb4e00 trap() at trap+0x56/frame 0xfffffe003ceb4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ceb4f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108037d78, rbp = 0xfffffe0108037db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108037db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108037dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108037df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108037ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108037f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108037f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff80003be1740 (CPU 10) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003cec3df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003cec3e00 trap() at trap+0x56/frame 0xfffffe003cec3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003cec3f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108046d78, rbp = 0xfffffe0108046db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108046db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108046dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108046df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108046ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108046f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108046f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff80003be1000 (CPU 11) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe003ced2df0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe003ced2e00 trap() at trap+0x56/frame 0xfffffe003ced2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003ced2f20 --- trap 0x13, rip = 0xffffffff81027dbe, rsp = 0xfffffe0108041d78, rbp = 0xfffffe0108041db0 --- acpi_cpu_idle_mwait() at acpi_cpu_idle_mwait+0x6e/frame 0xfffffe0108041db0 cpu_idle_acpi() at cpu_idle_acpi+0x50/frame 0xfffffe0108041dd0 cpu_idle() at cpu_idle+0xa6/frame 0xfffffe0108041df0 sched_idletd() at sched_idletd+0x534/frame 0xfffffe0108041ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108041f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108041f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100042 td 0xfffff80003ba7000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01080cde40 mi_switch() at mi_switch+0x170/frame 0xfffffe01080cde60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe01080cdef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01080cdf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01080cdf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100043 td 0xfffff80003ba8740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100045 td 0xfffff80003ba9740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100089 td 0xfffff80003c92740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081c2e40 mi_switch() at mi_switch+0x170/frame 0xfffffe01081c2e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe01081c2ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081c2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081c2f30 --- trap 0x483afc4d, rip = 0xbe087d8b5756145d, rsp = 0xe85753006a565040, rbp = 0x10000035abb8024 --- Tracing command intr pid 12 tid 100124 td 0xfffff800048d8740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108276e40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108276e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe0108276ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108276f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108276f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100126 td 0xfffff800048d9740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108280e40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108280e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe0108280ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108280f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108280f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100132 td 0xfffff80005021740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010829ee40 mi_switch() at mi_switch+0x170/frame 0xfffffe010829ee60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010829eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010829ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010829ef30 --- trap 0xa0003f8e, rip = 0x7894bfd0, rsp = 0xafafafafafafafaf, rbp = 0x30646870 --- Tracing command intr pid 12 tid 100143 td 0xfffff8000512f000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100144 td 0xfffff80005137740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100145 td 0xfffff80005137000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082d5e40 mi_switch() at mi_switch+0x170/frame 0xfffffe01082d5e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe01082d5ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082d5f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082d5f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100159 td 0xfffff80043850000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010831be40 mi_switch() at mi_switch+0x170/frame 0xfffffe010831be60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010831bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010831bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010831bf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100160 td 0xfffff80043851740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010832ae40 mi_switch() at mi_switch+0x170/frame 0xfffffe010832ae60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010832aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010832af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010832af30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command intr pid 12 tid 100161 td 0xfffff80043851000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100162 td 0xfffff80043852740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100163 td 0xfffff80043852000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100164 td 0xfffff80049e2f740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100165 td 0xfffff80049e2f000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100166 td 0xfffff80049e30740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100167 td 0xfffff80049e30000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100168 td 0xfffff80049e31740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100169 td 0xfffff80049e31000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100170 td 0xfffff80049e32740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100171 td 0xfffff80049e32000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100172 td 0xfffff80049e2b740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100173 td 0xfffff80049e2b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100180 td 0xfffff8004e559740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010838ee40 mi_switch() at mi_switch+0x170/frame 0xfffffe010838ee60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe010838eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010838ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010838ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100181 td 0xfffff80003d39740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108398e40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108398e60 ithread_loop() at ithread_loop+0x99/frame 0xfffffe0108398ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108398f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108398f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100182 td 0xfffff80003d39000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100183 td 0xfffff80003d3a740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100184 td 0xfffff80003d3a000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100185 td 0xfffff80003d3b740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100186 td 0xfffff80003d3b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100187 td 0xfffff80003d47740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100188 td 0xfffff80003d47000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100189 td 0xfffff80003d1f740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100190 td 0xfffff80003d1f000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100191 td 0xfffff80003d20740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100192 td 0xfffff80003d20000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100193 td 0xfffff80003d21740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100194 td 0xfffff80003d21000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100240 td 0xfffff8004ed21740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100241 td 0xfffff8004ed21000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100242 td 0xfffff8004ed22740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100243 td 0xfffff8004ed22000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100244 td 0xfffff8004ed2a740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100245 td 0xfffff8004ed2a000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100246 td 0xfffff8005a568740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100247 td 0xfffff8005a568000 fork_trampoline() at fork_trampoline Tracing command geom pid 13 tid 100091 td 0xfffff80003c93740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081ccdc0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081ccde0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081cce20 _sleep() at _sleep+0x2d8/frame 0xfffffe01081ccec0 g_run_events() at g_run_events+0x434/frame 0xfffffe01081ccef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081ccf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081ccf30 --- trap 0xc9834905, rip = 0x458bd00307e28399, rsp = 0x4905790000022d1a, rbp = 0x8b180c8dc2578a68 --- Tracing command geom pid 13 tid 100092 td 0xfffff80003c93000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081c7db0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081c7dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081c7e10 _sleep() at _sleep+0x2d8/frame 0xfffffe01081c7eb0 g_io_schedule_up() at g_io_schedule_up+0x3c/frame 0xfffffe01081c7ee0 g_up_procbody() at g_up_procbody+0x5c/frame 0xfffffe01081c7ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081c7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081c7f30 --- trap 0x3068b1c, rip = 0xe80875ff00ff45c6, rsp = 0x1075b60ffffd25d8, rbp = 0x68b14758bfff56f --- Tracing command geom pid 13 tid 100093 td 0xfffff80003c94740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081d6db0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081d6dd0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081d6e10 _sleep() at _sleep+0x2d8/frame 0xfffffe01081d6eb0 g_io_schedule_down() at g_io_schedule_down+0x3c/frame 0xfffffe01081d6ee0 g_down_procbody() at g_down_procbody+0x5c/frame 0xfffffe01081d6ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081d6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081d6f30 --- trap 0x7501f80d, rip = 0xf703e9c1000003e8, rsp = 0x5645b70f1c75544d, rbp = 0x50f83f6000001 --- Tracing command sequencer 00 pid 14 tid 100094 td 0xfffff80003c94000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01081d1da0 mi_switch() at mi_switch+0x170/frame 0xfffffe01081d1dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01081d1e00 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01081d1e70 seq_eventthread() at seq_eventthread+0x8c/frame 0xfffffe01081d1ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01081d1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01081d1f30 --- trap 0x15c06bc7, rip = 0x148800020646938a, rsp = 0x8400fe45c6000201, rbp = 0x2b76303b000220d4 --- Tracing command usb pid 15 tid 100127 td 0xfffff800048d9000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010827bdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010827be10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010827be50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe010827bec0 usb_process() at usb_process+0xb0/frame 0xfffffe010827bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010827bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010827bf30 --- trap 0x2fa000, rip = 0x2fc000, rsp = 0x1000, rbp = 0x1000 --- Tracing command usb pid 15 tid 100128 td 0xfffff800048da740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010828adf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010828ae10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010828ae50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe010828aec0 usb_process() at usb_process+0xb0/frame 0xfffffe010828aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010828af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010828af30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100129 td 0xfffff800048da000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108285df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108285e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108285e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe0108285ec0 usb_process() at usb_process+0xb0/frame 0xfffffe0108285ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108285f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108285f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100130 td 0xfffff800048e6740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108294df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108294e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108294e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe0108294ec0 usb_process() at usb_process+0xb0/frame 0xfffffe0108294ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108294f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108294f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 15 tid 100131 td 0xfffff800048e6000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010828fdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe010828fe10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010828fe50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe010828fec0 usb_process() at usb_process+0xb0/frame 0xfffffe010828fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010828ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010828ff30 --- trap 0x5dc01f4, rip = 0x201020000000100, rsp = 0x1001000001000d3, rbp = 0xb000600000b00 --- Tracing command usb pid 15 tid 100133 td 0xfffff80005021000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108299df0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108299e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108299e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe0108299ec0 usb_process() at usb_process+0xb0/frame 0xfffffe0108299ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108299f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108299f30 --- trap 0, rip = 0x4, rsp = 0, rbp = 0x300002 --- Tracing command usb pid 15 tid 100134 td 0xfffff80005022740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082a8df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082a8e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082a8e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082a8ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082a8ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082a8f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082a8f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100135 td 0xfffff80005022000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082a3df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082a3e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082a3e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082a3ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082a3ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082a3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082a3f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100136 td 0xfffff80005023740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082b2df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082b2e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082b2e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082b2ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082b2ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082b2f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082b2f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100137 td 0xfffff80005023000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082addf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082ade10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082ade50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082adec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082adef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082adf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082adf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100138 td 0xfffff8000512d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082bcdf0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082bce10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082bce50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082bcec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082bcef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082bcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082bcf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100139 td 0xfffff8000512d000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082b7df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082b7e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082b7e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082b7ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082b7ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082b7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082b7f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100140 td 0xfffff8000512e740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082c6df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082c6e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082c6e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082c6ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082c6ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082c6f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082c6f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100141 td 0xfffff8000512e000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082c1df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082c1e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082c1e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082c1ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082c1ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082c1f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082c1f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command usb pid 15 tid 100142 td 0xfffff8000512f740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01082d0df0 mi_switch() at mi_switch+0x170/frame 0xfffffe01082d0e10 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01082d0e50 _cv_wait() at _cv_wait+0x160/frame 0xfffffe01082d0ec0 usb_process() at usb_process+0xb0/frame 0xfffffe01082d0ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01082d0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01082d0f30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf --- Tracing command vmdaemon pid 16 tid 100200 td 0xfffff8004e5b8000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083e3d80 mi_switch() at mi_switch+0x170/frame 0xfffffe01083e3da0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083e3de0 _sleep() at _sleep+0x2d8/frame 0xfffffe01083e3e80 vm_daemon() at vm_daemon+0x82/frame 0xfffffe01083e3ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083e3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083e3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100201 td 0xfffff8004e5b9740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083fcd60 mi_switch() at mi_switch+0x170/frame 0xfffffe01083fcd80 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083fcdc0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01083fce00 _sleep() at _sleep+0x27e/frame 0xfffffe01083fcea0 buf_daemon() at buf_daemon+0x121/frame 0xfffffe01083fcef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083fcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083fcf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100202 td 0xfffff800041eb740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108406d40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108406d60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108406da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108406de0 _sleep() at _sleep+0x27e/frame 0xfffffe0108406e80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe0108406ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108406f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108406f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100203 td 0xfffff800041eb000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108401d40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108401d60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108401da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108401de0 _sleep() at _sleep+0x27e/frame 0xfffffe0108401e80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe0108401ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108401f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108401f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100204 td 0xfffff800041ec740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108410d40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108410d60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108410da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108410de0 _sleep() at _sleep+0x27e/frame 0xfffffe0108410e80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe0108410ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108410f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108410f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100205 td 0xfffff800041ec000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010840bd40 mi_switch() at mi_switch+0x170/frame 0xfffffe010840bd60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010840bda0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010840bde0 _sleep() at _sleep+0x27e/frame 0xfffffe010840be80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe010840bef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010840bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010840bf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100206 td 0xfffff800041ed740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010841ad40 mi_switch() at mi_switch+0x170/frame 0xfffffe010841ad60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010841ada0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010841ade0 _sleep() at _sleep+0x27e/frame 0xfffffe010841ae80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe010841aef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010841af30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010841af30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100207 td 0xfffff800041ed000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108415d40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108415d60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108415da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108415de0 _sleep() at _sleep+0x27e/frame 0xfffffe0108415e80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe0108415ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108415f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108415f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100208 td 0xfffff800041f9740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108424d40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108424d60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108424da0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108424de0 _sleep() at _sleep+0x27e/frame 0xfffffe0108424e80 bufspace_daemon() at bufspace_daemon+0x322/frame 0xfffffe0108424ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108424f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108424f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100232 td 0xfffff8004e55b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108456d70 mi_switch() at mi_switch+0x170/frame 0xfffffe0108456d90 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108456dd0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0108456e10 _sleep() at _sleep+0x27e/frame 0xfffffe0108456eb0 softdep_flush() at softdep_flush+0x188/frame 0xfffffe0108456ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108456f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108456f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 100248 td 0xfffff8004e55b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010844cd70 mi_switch() at mi_switch+0x170/frame 0xfffffe010844cd90 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010844cdd0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010844ce10 _sleep() at _sleep+0x27e/frame 0xfffffe010844ceb0 softdep_flush() at softdep_flush+0x188/frame 0xfffffe010844cef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010844cf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010844cf30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 17 tid 103138 td 0xfffff80579c3d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0183c0fd70 mi_switch() at mi_switch+0x170/frame 0xfffffe0183c0fd90 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0183c0fdd0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0183c0fe10 _sleep() at _sleep+0x27e/frame 0xfffffe0183c0feb0 softdep_flush() at softdep_flush+0x188/frame 0xfffffe0183c0fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0183c0ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0183c0ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vnlru pid 18 tid 100209 td 0xfffff8004e5b9000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01083f7d10 mi_switch() at mi_switch+0x170/frame 0xfffffe01083f7d30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01083f7d70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe01083f7db0 _sleep() at _sleep+0x27e/frame 0xfffffe01083f7e50 vnlru_proc() at vnlru_proc+0x18b/frame 0xfffffe01083f7ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01083f7f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01083f7f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 19 tid 100210 td 0xfffff8004e5ba740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010842ed30 mi_switch() at mi_switch+0x170/frame 0xfffffe010842ed50 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010842ed90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe010842edd0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x187/frame 0xfffffe010842ee60 sched_sync() at sched_sync+0x6ae/frame 0xfffffe010842eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010842ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010842ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sleep pid 818 tid 100227 td 0xfffff8004e55c740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010846aba0 mi_switch() at mi_switch+0x170/frame 0xfffffe010846abc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010846ac00 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010846ac60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010846aca0 _sleep() at _sleep+0x26c/frame 0xfffffe010846ad40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1cf/frame 0xfffffe010846adc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe010846ae00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010846af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010846af30 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x31d8863706ba, rsp = 0x31d8858ae2c8, rbp = 0x31d8858ae320 --- Tracing command dhclient pid 3209 tid 100228 td 0xfffff8004ed94740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108474a90 mi_switch() at mi_switch+0x170/frame 0xfffffe0108474ab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108474af0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108474b50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108474b60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0108474bd0 kern_select() at kern_select+0xa29/frame 0xfffffe0108474dc0 sys_select() at sys_select+0x57/frame 0xfffffe0108474e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108474f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108474f30 --- syscall (93, FreeBSD ELF64, select), rip = 0x2f763f6a083a, rsp = 0x2f763bfcb018, rbp = 0x2f763bfcb0d0 --- Tracing command dhclient pid 3212 tid 100261 td 0xfffff8004ed9d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010850aac0 mi_switch() at mi_switch+0x170/frame 0xfffffe010850aae0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010850ab20 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010850ab80 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010850ab90 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010850ac00 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe010850ac90 kern_poll() at kern_poll+0x9d/frame 0xfffffe010850ade0 sys_poll() at sys_poll+0x50/frame 0xfffffe010850ae00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010850af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010850af30 --- syscall (209, FreeBSD ELF64, poll), rip = 0x2f763f6a073a, rsp = 0x2f763bfcb258, rbp = 0x2f763bfcb290 --- Tracing command dhclient pid 3271 tid 100273 td 0xfffff801091cd740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010853ca70 mi_switch() at mi_switch+0x170/frame 0xfffffe010853ca90 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010853cad0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010853cb30 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010853cb70 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x187/frame 0xfffffe010853cc00 kern_poll_kfds() at kern_poll_kfds+0x493/frame 0xfffffe010853cc90 kern_poll() at kern_poll+0x9d/frame 0xfffffe010853cde0 sys_poll() at sys_poll+0x50/frame 0xfffffe010853ce00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010853cf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010853cf30 --- syscall (209, FreeBSD ELF64, poll), rip = 0x2f763f6a073a, rsp = 0x2f763bfcb258, rbp = 0x2f763bfcb290 --- Tracing command devd pid 3286 tid 100269 td 0xfffff8005a04f000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010852da40 mi_switch() at mi_switch+0x170/frame 0xfffffe010852da60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010852daa0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010852db00 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe010852db40 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x187/frame 0xfffffe010852dbd0 kern_select() at kern_select+0xa03/frame 0xfffffe010852ddc0 sys_select() at sys_select+0x57/frame 0xfffffe010852de00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010852df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010852df30 --- syscall (93, FreeBSD ELF64, select), rip = 0x38d56f0cb83a, rsp = 0x38d569af4d88, rbp = 0x38d569af6f20 --- Tracing command syslogd pid 3434 tid 100327 td 0xfffff80003e7b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01084f1800 mi_switch() at mi_switch+0x170/frame 0xfffffe01084f1820 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01084f1860 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01084f18c0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01084f18d0 _sleep() at _sleep+0x2ce/frame 0xfffffe01084f1970 kqueue_scan() at kqueue_scan+0x99d/frame 0xfffffe01084f1a30 kqueue_kevent() at kqueue_kevent+0x13b/frame 0xfffffe01084f1c90 kern_kevent_fp() at kern_kevent_fp+0x97/frame 0xfffffe01084f1ce0 kern_kevent() at kern_kevent+0x82/frame 0xfffffe01084f1d40 kern_kevent_generic() at kern_kevent_generic+0x73/frame 0xfffffe01084f1da0 sys_kevent() at sys_kevent+0x61/frame 0xfffffe01084f1e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01084f1f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01084f1f30 --- syscall (560, FreeBSD ELF64, kevent), rip = 0x15b8629a067a, rsp = 0x15b8616fb828, rbp = 0x15b8616fbd10 --- Tracing command watchdogd pid 3451 tid 100338 td 0xfffff8010923d000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085c3ba0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085c3bc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085c3c00 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085c3c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe01085c3ca0 _sleep() at _sleep+0x26c/frame 0xfffffe01085c3d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1cf/frame 0xfffffe01085c3dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe01085c3e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085c3f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085c3f30 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0xcf8a38ac6ba, rsp = 0xcf8a17ce8d8, rbp = 0xcf8a17ce920 --- Tracing command ntpd pid 3551 tid 100344 td 0xfffff80003e77740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085d2a90 mi_switch() at mi_switch+0x170/frame 0xfffffe01085d2ab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085d2af0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085d2b50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085d2b60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085d2bd0 kern_select() at kern_select+0xa29/frame 0xfffffe01085d2dc0 sys_select() at sys_select+0x57/frame 0xfffffe01085d2e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085d2f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085d2f30 --- syscall (93, FreeBSD ELF64, select), rip = 0x29a743c6883a, rsp = 0x29a73e9ed548, rbp = 0x29a73e9ed580 --- Tracing command ntpd pid 3551 tid 103390 td 0xfffff804e222d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0183c6eb10 mi_switch() at mi_switch+0x170/frame 0xfffffe0183c6eb30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0183c6eb70 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0183c6ebd0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0183c6ebe0 _sleep() at _sleep+0x2ce/frame 0xfffffe0183c6ec80 umtxq_sleep() at umtxq_sleep+0x302/frame 0xfffffe0183c6ed00 __umtx_op_sem2_wait() at __umtx_op_sem2_wait+0x3fb/frame 0xfffffe0183c6edc0 sys__umtx_op() at sys__umtx_op+0x7e/frame 0xfffffe0183c6ee00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0183c6ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0183c6ef30 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x29a743c675da, rsp = 0x29a7454e2ee8, rbp = 0x29a7454e2f60 --- Tracing command powerd pid 3568 tid 100296 td 0xfffff8005a525740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108537a40 mi_switch() at mi_switch+0x170/frame 0xfffffe0108537a60 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108537aa0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108537b00 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0108537b40 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x187/frame 0xfffffe0108537bd0 kern_select() at kern_select+0xa03/frame 0xfffffe0108537dc0 sys_select() at sys_select+0x57/frame 0xfffffe0108537e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108537f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108537f30 --- syscall (93, FreeBSD ELF64, select), rip = 0x38ec2143183a, rsp = 0x38ec1ff22a18, rbp = 0x38ec1ff22b50 --- Tracing command inetd pid 3589 tid 100225 td 0xfffff80003e6f740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108429a90 mi_switch() at mi_switch+0x170/frame 0xfffffe0108429ab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108429af0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108429b50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108429b60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0108429bd0 kern_select() at kern_select+0xa29/frame 0xfffffe0108429dc0 sys_select() at sys_select+0x57/frame 0xfffffe0108429e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108429f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108429f30 --- syscall (93, FreeBSD ELF64, select), rip = 0x39fb64cd83a, rsp = 0x39fb2c72c48, rbp = 0x39fb2c73ad0 --- Tracing command cron pid 3594 tid 100299 td 0xfffff8010913d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108582ba0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108582bc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108582c00 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108582c60 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe0108582ca0 _sleep() at _sleep+0x26c/frame 0xfffffe0108582d40 kern_clock_nanosleep() at kern_clock_nanosleep+0x1cf/frame 0xfffffe0108582dc0 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe0108582e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108582f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108582f30 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x192a870d66ba, rsp = 0x192a83fddb58, rbp = 0x192a83fddce0 --- Tracing command sshd pid 3611 tid 100343 td 0xfffff80043753740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108564a90 mi_switch() at mi_switch+0x170/frame 0xfffffe0108564ab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108564af0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108564b50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108564b60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0108564bd0 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe0108564c60 kern_poll() at kern_poll+0x9d/frame 0xfffffe0108564db0 sys_ppoll() at sys_ppoll+0x70/frame 0xfffffe0108564e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108564f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108564f30 --- syscall (545, FreeBSD ELF64, ppoll), rip = 0x176edc6ba75a, rsp = 0x176ece5a31c8, rbp = 0x176ece5a3200 --- Tracing command getty pid 3625 tid 100345 td 0xfffff8004ee68000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085b9ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085b9b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085b9b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085b9ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085b9bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085b9c20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085b9c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085b9cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085b9ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085b9d40 dofileread() at dofileread+0x80/frame 0xfffffe01085b9d90 sys_read() at sys_read+0xb7/frame 0xfffffe01085b9e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085b9f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085b9f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x20ecaf47a7ba, rsp = 0x20ecabcfcd68, rbp = 0x20ecabcfcdb0 --- Tracing command getty pid 3626 tid 100283 td 0xfffff80109413740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010855aae0 mi_switch() at mi_switch+0x170/frame 0xfffffe010855ab00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010855ab40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010855aba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010855abb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010855ac20 tty_wait() at tty_wait+0x42/frame 0xfffffe010855ac50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe010855acb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010855ace0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe010855ad40 dofileread() at dofileread+0x80/frame 0xfffffe010855ad90 sys_read() at sys_read+0xb7/frame 0xfffffe010855ae00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010855af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010855af30 --- syscall (3, FreeBSD ELF64, read), rip = 0x5e3a79af7ba, rsp = 0x5e3a6682348, rbp = 0x5e3a6682390 --- Tracing command getty pid 3627 tid 100348 td 0xfffff80109416740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085e6ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085e6b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085e6b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085e6ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085e6bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085e6c20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085e6c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085e6cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085e6ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085e6d40 dofileread() at dofileread+0x80/frame 0xfffffe01085e6d90 sys_read() at sys_read+0xb7/frame 0xfffffe01085e6e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085e6f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085e6f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x663378c87ba, rsp = 0x663356ee0e8, rbp = 0x663356ee130 --- Tracing command getty pid 3628 tid 100349 td 0xfffff80109416000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085e1ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085e1b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085e1b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085e1ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085e1bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085e1c20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085e1c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085e1cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085e1ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085e1d40 dofileread() at dofileread+0x80/frame 0xfffffe01085e1d90 sys_read() at sys_read+0xb7/frame 0xfffffe01085e1e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085e1f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085e1f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x11a3540347ba, rsp = 0x11a352db8ac8, rbp = 0x11a352db8b10 --- Tracing command getty pid 3629 tid 100350 td 0xfffff80160b3b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085f0ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085f0b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085f0b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085f0ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085f0bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085f0c20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085f0c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085f0cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085f0ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085f0d40 dofileread() at dofileread+0x80/frame 0xfffffe01085f0d90 sys_read() at sys_read+0xb7/frame 0xfffffe01085f0e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085f0f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085f0f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x1ae70333f7ba, rsp = 0x1ae7004ebde8, rbp = 0x1ae7004ebe30 --- Tracing command getty pid 3630 tid 100351 td 0xfffff80160b3b000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085ebae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085ebb00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085ebb40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085ebba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085ebbb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085ebc20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085ebc50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085ebcb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085ebce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085ebd40 dofileread() at dofileread+0x80/frame 0xfffffe01085ebd90 sys_read() at sys_read+0xb7/frame 0xfffffe01085ebe00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085ebf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085ebf30 --- syscall (3, FreeBSD ELF64, read), rip = 0x1704936407ba, rsp = 0x170490ffd388, rbp = 0x170490ffd3d0 --- Tracing command getty pid 3631 tid 100352 td 0xfffff80109a69740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085faae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085fab00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085fab40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085faba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085fabb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085fac20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085fac50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085facb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085face0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085fad40 dofileread() at dofileread+0x80/frame 0xfffffe01085fad90 sys_read() at sys_read+0xb7/frame 0xfffffe01085fae00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085faf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085faf30 --- syscall (3, FreeBSD ELF64, read), rip = 0x1c35357f87ba, rsp = 0x1c35333c3228, rbp = 0x1c35333c3270 --- Tracing command getty pid 3632 tid 100353 td 0xfffff80109a69000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085f5ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085f5b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085f5b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085f5ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085f5bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085f5c20 tty_wait() at tty_wait+0x42/frame 0xfffffe01085f5c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe01085f5cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe01085f5ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe01085f5d40 dofileread() at dofileread+0x80/frame 0xfffffe01085f5d90 sys_read() at sys_read+0xb7/frame 0xfffffe01085f5e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085f5f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085f5f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x3ce9fcb287ba, rsp = 0x3ce9fa582108, rbp = 0x3ce9fa582150 --- Tracing command login pid 3633 tid 100354 td 0xfffff8005a0ea740 --- swapped out Tracing command sshd pid 3634 tid 100356 td 0xfffff8005a0ea000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01085ffac0 mi_switch() at mi_switch+0x170/frame 0xfffffe01085ffae0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01085ffb20 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01085ffb80 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01085ffb90 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01085ffc00 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe01085ffc90 kern_poll() at kern_poll+0x9d/frame 0xfffffe01085ffde0 sys_poll() at sys_poll+0x50/frame 0xfffffe01085ffe00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01085fff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01085fff30 --- syscall (209, FreeBSD ELF64, poll), rip = 0x1779a377873a, rsp = 0x17799640c328, rbp = 0x17799640c350 --- Tracing command sshd pid 3637 tid 100212 td 0xfffff8004ecb3740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108438a90 mi_switch() at mi_switch+0x170/frame 0xfffffe0108438ab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108438af0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108438b50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108438b60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0108438bd0 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe0108438c60 kern_poll() at kern_poll+0x9d/frame 0xfffffe0108438db0 sys_ppoll() at sys_ppoll+0x70/frame 0xfffffe0108438e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108438f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108438f30 --- syscall (545, FreeBSD ELF64, ppoll), rip = 0x1779a377875a, rsp = 0x17799640c0d8, rbp = 0x17799640c110 --- Tracing command sh pid 3638 tid 100357 td 0xfffff80160dc4740 --- swapped out Tracing command su pid 3660 tid 100329 td 0xfffff80003e6d000 --- swapped out Tracing command csh pid 3661 tid 100271 td 0xfffff801091ce000 --- swapped out Tracing command bash pid 3663 tid 100270 td 0xfffff8004edf9000 --- swapped out Tracing command sh pid 3716 tid 100361 td 0xfffff8034aa07740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108622a10 mi_switch() at mi_switch+0x170/frame 0xfffffe0108622a30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108622a70 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108622ad0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108622ae0 _sleep() at _sleep+0x2ce/frame 0xfffffe0108622b80 kern_wait6() at kern_wait6+0x471/frame 0xfffffe0108622c10 sys_wait4() at sys_wait4+0x6b/frame 0xfffffe0108622e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108622f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108622f30 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x2de3bf6a199a, rsp = 0x2de3bc70bb28, rbp = 0x2de3bc70bbd0 --- Tracing command sshd pid 3781 tid 100256 td 0xfffff8005a569740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01084f6ac0 mi_switch() at mi_switch+0x170/frame 0xfffffe01084f6ae0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01084f6b20 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01084f6b80 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01084f6b90 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01084f6c00 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe01084f6c90 kern_poll() at kern_poll+0x9d/frame 0xfffffe01084f6de0 sys_poll() at sys_poll+0x50/frame 0xfffffe01084f6e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01084f6f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01084f6f30 --- syscall (209, FreeBSD ELF64, poll), rip = 0x193ff77a173a, rsp = 0x193fe8ceb188, rbp = 0x193fe8ceb1b0 --- Tracing command sh pid 3854 tid 100250 td 0xfffff8004eeed740 --- swapped out Tracing command sshd pid 3882 tid 100259 td 0xfffff80005024000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01084dda90 mi_switch() at mi_switch+0x170/frame 0xfffffe01084ddab0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01084ddaf0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01084ddb50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01084ddb60 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe01084ddbd0 kern_poll_kfds() at kern_poll_kfds+0x4cc/frame 0xfffffe01084ddc60 kern_poll() at kern_poll+0x9d/frame 0xfffffe01084dddb0 sys_ppoll() at sys_ppoll+0x70/frame 0xfffffe01084dde00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01084ddf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01084ddf30 --- syscall (545, FreeBSD ELF64, ppoll), rip = 0x193ff77a175a, rsp = 0x193fe8ceaf38, rbp = 0x193fe8ceaf70 --- Tracing command sh pid 3885 tid 100371 td 0xfffff8010908f740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108654ae0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108654b00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108654b40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108654ba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108654bb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0108654c20 tty_wait() at tty_wait+0x42/frame 0xfffffe0108654c50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe0108654cb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe0108654ce0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe0108654d40 dofileread() at dofileread+0x80/frame 0xfffffe0108654d90 sys_read() at sys_read+0xb7/frame 0xfffffe0108654e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108654f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108654f30 --- syscall (3, FreeBSD ELF64, read), rip = 0xbcff5a5c7ba, rsp = 0xbcff3b8c1f8, rbp = 0xbcff3b8c2d0 --- Tracing command sh pid 3904 tid 100298 td 0xfffff8005a51c000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010848da10 mi_switch() at mi_switch+0x170/frame 0xfffffe010848da30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010848da70 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010848dad0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010848dae0 _sleep() at _sleep+0x2ce/frame 0xfffffe010848db80 kern_wait6() at kern_wait6+0x471/frame 0xfffffe010848dc10 sys_wait4() at sys_wait4+0x6b/frame 0xfffffe010848de00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010848df30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010848df30 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x3a3dba49499a, rsp = 0x3a3db5f947c8, rbp = 0x3a3db5f94870 --- Tracing command awk pid 3905 tid 100332 td 0xfffff8004e5ba000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108465b60 mi_switch() at mi_switch+0x170/frame 0xfffffe0108465b80 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108465bc0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108465c20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108465c30 _sleep() at _sleep+0x2ce/frame 0xfffffe0108465cd0 pipe_read() at pipe_read+0x450/frame 0xfffffe0108465d40 dofileread() at dofileread+0x80/frame 0xfffffe0108465d90 sys_read() at sys_read+0xb7/frame 0xfffffe0108465e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108465f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108465f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x2c3d8cb77ba, rsp = 0x2c3d6272128, rbp = 0x2c3d6272140 --- Tracing command sh pid 3907 tid 100263 td 0xfffff8004eef0740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108514b60 mi_switch() at mi_switch+0x170/frame 0xfffffe0108514b80 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108514bc0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108514c20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108514c30 _sleep() at _sleep+0x2ce/frame 0xfffffe0108514cd0 pipe_read() at pipe_read+0x450/frame 0xfffffe0108514d40 dofileread() at dofileread+0x80/frame 0xfffffe0108514d90 sys_read() at sys_read+0xb7/frame 0xfffffe0108514e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108514f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108514f30 --- syscall (3, FreeBSD ELF64, read), rip = 0x3a3dba4947ba, rsp = 0x3a3db5f94398, rbp = 0x3a3db5f948b0 --- Tracing command sh pid 37205 tid 100512 td 0xfffff805516bf000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01088cfa10 mi_switch() at mi_switch+0x170/frame 0xfffffe01088cfa30 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01088cfa70 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe01088cfad0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe01088cfae0 _sleep() at _sleep+0x2ce/frame 0xfffffe01088cfb80 kern_wait6() at kern_wait6+0x471/frame 0xfffffe01088cfc10 sys_wait4() at sys_wait4+0x6b/frame 0xfffffe01088cfe00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe01088cff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01088cff30 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x2de3bf6a199a, rsp = 0x2de3bc70b7d8, rbp = 0x2de3bc70b880 --- Tracing command sh pid 37206 tid 100528 td 0xfffff8063983d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010892eb90 mi_switch() at mi_switch+0x170/frame 0xfffffe010892ebb0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010892ebf0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010892ec50 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010892ec60 _sleep() at _sleep+0x2ce/frame 0xfffffe010892ed00 kern_sigsuspend() at kern_sigsuspend+0x1f2/frame 0xfffffe010892edd0 sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe010892ee00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010892ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010892ef30 --- syscall (4, FreeBSD ELF64, write), rip = 0x2875929778fa, rsp = 0x287590e328e8, rbp = 0x287590e32990 --- Tracing command tee pid 37207 tid 100408 td 0xfffff803b8549000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108703b60 mi_switch() at mi_switch+0x170/frame 0xfffffe0108703b80 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108703bc0 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe0108703c20 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0108703c30 _sleep() at _sleep+0x2ce/frame 0xfffffe0108703cd0 pipe_read() at pipe_read+0x450/frame 0xfffffe0108703d40 dofileread() at dofileread+0x80/frame 0xfffffe0108703d90 sys_read() at sys_read+0xb7/frame 0xfffffe0108703e00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe0108703f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0108703f30 --- syscall (3, FreeBSD ELF64, read), rip = 0xd25679187ba, rsp = 0xd256620e798, rbp = 0xd256620e7f0 --- Tracing command md10 pid 37213 tid 100378 td 0xfffff80003e7b740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108672da0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108672dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108672e00 _sleep() at _sleep+0x2d8/frame 0xfffffe0108672ea0 md_kthread() at md_kthread+0xd5/frame 0xfffffe0108672ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108672f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108672f30 --- trap 0xc, rip = 0xd85e0bf14ba, rsp = 0xd85df978858, rbp = 0xd85df978870 --- Tracing command md22 pid 37218 tid 100671 td 0xfffff8027140a740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe01837dcda0 mi_switch() at mi_switch+0x170/frame 0xfffffe01837dcdc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe01837dce00 _sleep() at _sleep+0x2d8/frame 0xfffffe01837dcea0 md_kthread() at md_kthread+0xd5/frame 0xfffffe01837dcef0 fork_exit() at fork_exit+0x82/frame 0xfffffe01837dcf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01837dcf30 --- trap 0xc, rip = 0x19c0e466c4ba, rsp = 0x19c0e1ea0c78, rbp = 0x19c0e1ea0c90 --- Tracing command md23 pid 37223 tid 100444 td 0xfffff8054a38f740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010879eda0 mi_switch() at mi_switch+0x170/frame 0xfffffe010879edc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010879ee00 _sleep() at _sleep+0x2d8/frame 0xfffffe010879eea0 md_kthread() at md_kthread+0xd5/frame 0xfffffe010879eef0 fork_exit() at fork_exit+0x82/frame 0xfffffe010879ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010879ef30 --- trap 0xc, rip = 0x1f92078124ba, rsp = 0x1f920676f1b8, rbp = 0x1f920676f1d0 --- Tracing command md24 pid 37228 tid 100479 td 0xfffff8034a8f2740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe0108848da0 mi_switch() at mi_switch+0x170/frame 0xfffffe0108848dc0 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe0108848e00 _sleep() at _sleep+0x2d8/frame 0xfffffe0108848ea0 md_kthread() at md_kthread+0xd5/frame 0xfffffe0108848ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe0108848f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0108848f30 --- trap 0xc, rip = 0x19f829cb4ba, rsp = 0x19f806d51e8, rbp = 0x19f806d5200 --- Tracing command sh pid 37241 tid 100505 td 0xfffff8058c9fb000 --- swapped out Tracing command csh pid 14083 tid 100421 td 0xfffff8005a04d740 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010873aae0 mi_switch() at mi_switch+0x170/frame 0xfffffe010873ab00 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010873ab40 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010873aba0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010873abb0 _cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe010873ac20 tty_wait() at tty_wait+0x42/frame 0xfffffe010873ac50 ttydisc_read() at ttydisc_read+0x1f5/frame 0xfffffe010873acb0 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe010873ace0 devfs_read_f() at devfs_read_f+0xe4/frame 0xfffffe010873ad40 dofileread() at dofileread+0x80/frame 0xfffffe010873ad90 sys_read() at sys_read+0xb7/frame 0xfffffe010873ae00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010873af30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010873af30 --- syscall (3, FreeBSD ELF64, read), rip = 0x2e8e9d9587ba, rsp = 0x2e8e9c3c3318, rbp = 0x2e8e9c3c3340 --- Tracing command find pid 55088 tid 100476 td 0xfffff8034abd1000 sched_switch() at sched_switch+0x5d2/frame 0xfffffe010882f730 mi_switch() at mi_switch+0x170/frame 0xfffffe010882f750 sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe010882f790 sleepq_catch_signals() at sleepq_catch_signals+0x282/frame 0xfffffe010882f7f0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe010882f800 _sleep() at _sleep+0x2ce/frame 0xfffffe010882f8a0 unionfs_set_in_progress_flag() at unionfs_set_in_progress_flag+0xc4/frame 0xfffffe010882f8f0 unionfs_lookup() at unionfs_lookup+0xa5/frame 0xfffffe010882fa40 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x5f/frame 0xfffffe010882fa70 vfs_cache_lookup() at vfs_cache_lookup+0xa6/frame 0xfffffe010882fac0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x5f/frame 0xfffffe010882faf0 vfs_lookup() at vfs_lookup+0x487/frame 0xfffffe010882fb70 namei() at namei+0x2d1/frame 0xfffffe010882fbd0 kern_statat() at kern_statat+0x115/frame 0xfffffe010882fd00 sys_fstatat() at sys_fstatat+0x27/frame 0xfffffe010882fe00 amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe010882ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010882ff30 --- syscall (552, FreeBSD ELF64, fstatat), rip = 0x3adf6e2e403a, rsp = 0x3adf6c84bda8, rbp = 0x3adf6c84bec0 --- db> reset Uptime: 23h5m13s (kgdb) proc 55088 (kgdb) bt #0 sched_switch (td=td@entry=0xfffff8034abd1000, flags=flags@entry=0x103) at ../../../kern/sched_ule.c:2290 #1 0xffffffff80b54df0 in mi_switch (flags=flags@entry=0x103) at ../../../kern/kern_synch.c:546 #2 0xffffffff80ba8aa9 in sleepq_switch (wchan=wchan@entry=0xfffffe018e383770, pri=pri@entry=0x0) at ../../../kern/subr_sleepqueue.c:607 #3 0xffffffff80ba8ea2 in sleepq_catch_signals (wchan=0xfffffe018e383770, pri=0x0) at ../../../kern/subr_sleepqueue.c:523 #4 0xffffffff80ba8be9 in sleepq_wait_sig (wchan=, pri=) at ../../../kern/subr_sleepqueue.c:670 #5 0xffffffff80b5432e in _sleep (ident=ident@entry=0xfffffe018e383770, lock=lock@entry=0xfffffe018e3838a0, priority=priority@entry=0x100, wmesg=0xffffffff82856cca "unioncp", sbt=sbt@entry=0x0, pr=pr@entry=0x0, flags=0x100) at ../../../kern/kern_synch.c:219 #6 0xffffffff8284d434 in unionfs_set_in_progress_flag (vp=vp@entry=0xfffffe018e383770, flag=flag@entry=0x8) at /usr/src/sys/fs/unionfs/union_subr.c:791 #7 0xffffffff8284fef5 in unionfs_lookup (ap=0xfffffe010882fa80) at /usr/src/sys/fs/unionfs/union_vnops.c:140 #8 0xffffffff811329bf in VOP_CACHEDLOOKUP_APV (vop=0xffffffff828574f8 , a=a@entry=0xfffffe010882fa80) at vnode_if.c:178 #9 0xffffffff80c20756 in VOP_CACHEDLOOKUP (dvp=0xfffffe018e383770, vpp=0xfffffe010882fc38, cnp=0xfffffe010882fc60) at ./vnode_if.h:101 #10 vfs_cache_lookup (ap=) at ../../../kern/vfs_cache.c:3167 #11 0xffffffff811327df in VOP_LOOKUP_APV (vop=0xffffffff828574f8 , a=a@entry=0xfffffe010882fb20) at vnode_if.c:115 #12 0xffffffff80c31ea7 in VOP_LOOKUP (dvp=0xfffffe018e383770, vpp=0xfffffe010882fc38, cnp=0xfffffe010882fc60) at ./vnode_if.h:67 #13 vfs_lookup (ndp=ndp@entry=0xfffffe010882fbe0) at ../../../kern/vfs_lookup.c:1283 #14 0xffffffff80c30f41 in namei (ndp=ndp@entry=0xfffffe010882fbe0) at ../../../kern/vfs_lookup.c:705 #15 0xffffffff80c52db5 in kern_statat (td=0xfffff8034abd1000, flag=, fd=0xffffff9c, path=0x4467a9aad178 , pathseg=pathseg@entry=UIO_USERSPACE, sbp=sbp@entry=0xfffffe010882fd18) at ../../../kern/vfs_syscalls.c:2471 #16 0xffffffff80c53497 in sys_fstatat (td=, uap=0xfffff8034abd1400) at ../../../kern/vfs_syscalls.c:2449 #17 0xffffffff8106c6a8 in syscallenter (td=0xfffff8034abd1000) at ../../../amd64/amd64/../../kern/subr_syscall.c:189 #18 amd64_syscall (td=0xfffff8034abd1000, traced=0x0) at ../../../amd64/amd64/trap.c:1192 #19 #20 0x00003adf6e2e403a in ?? () Backtrace stopped: Cannot access memory at address 0x3adf6c84bda8 (kgdb) f 6 #6 0xffffffff8284d434 in unionfs_set_in_progress_flag (vp=vp@entry=0xfffffe018e383770, flag=flag@entry=0x8) at /usr/src/sys/fs/unionfs/union_subr.c:791 791 error = msleep(vp, VI_MTX(vp), PCATCH, "unioncp", 0); (kgdb) l 786 ASSERT_VOP_ELOCKED(vp, __func__); 787 VI_LOCK(vp); 788 unp = VTOUNIONFS(vp); 789 while (error == 0 && (unp->un_flag & flag) != 0) { 790 VOP_UNLOCK(vp); 791 error = msleep(vp, VI_MTX(vp), PCATCH, "unioncp", 0); 792 VI_UNLOCK(vp); 793 vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); 794 VI_LOCK(vp); 795 if (error == 0) { (kgdb) info loc error = 0x0 unp = (kgdb) p *vp $1 = {v_type = VBAD, v_state = VSTATE_DEAD, v_irflag = 0x1, v_seqc = 0x3, v_nchash = 0xc058d13d, v_hash = 0x18e3837, v_op = 0xffffffff818ac6c0 , v_data = 0x0, v_mount = 0x0, v_nmntvnodes = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, {v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0}, v_hashlist = { le_next = 0xffffffffffffffff, le_prev = 0xffffffffffffffff}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffffe018e3837c8}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff82856c7c "unionfs", lo_flags = 0x6fb0008, lo_data = 0x0, lo_witness = 0xfffff8087fd85400}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_pri = 0x40, lk_timo = 0x33, lk_stack = { depth = 0xa, pcs = {0xffffffff80b140c8, 0xffffffff8113629a, 0xffffffff80c5aa83, 0xffffffff80c5ad90, 0xffffffff80c5930d, 0xffffffff80ae151b, 0xffffffff80ae4d73, 0xffffffff80ae8c21, 0xffffffff8106c6a8, 0xffffffff8103debb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = {lo_name = 0xffffffff811e77d2 "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd75500}, mtx_lock = 0x0}, v_vnlock = 0xfffffe018e3837e0, v_vnodelist = {tqe_next = 0xfffffe017eb58708, tqe_prev = 0xfffffe0170b551c0}, v_lazylist = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, v_bufobj = {bo_lock = { lock_object = {lo_name = 0xffffffff81233330 "bufobj interlock", lo_flags = 0x5230000, lo_data = 0x0, lo_witness = 0xfffff8087fd80c00}, rw_lock = 0x1}, bo_ops = 0xffffffff818ed240 , bo_object = 0x0, bo_synclist = {le_next = 0xffffffffffffffff, le_prev = 0xffffffffffffffff}, bo_private = 0xfffffe018e383770, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffffe018e383930}, bv_root = {pt_root = 0x1}, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffffe018e383950}, bv_root = { pt_root = 0x1}, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x4, bo_domain = 0x3, bo_bsize = 0x0}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffffe018e383998}, rl_currdep = 0x0}, v_holdcnt = 0x1, v_usecount = 0x3, v_iflag = 0x0, v_vflag = 0x1, v_mflag = 0x0, v_dbatchcpu = 0xffff, v_writecount = 0x0, v_seqc_users = 0x1} (kgdb) diff --git a/sys/fs/unionfs/union.h b/sys/fs/unionfs/union.h index 467db3b29ff8..c535a3b288b8 100644 --- a/sys/fs/unionfs/union.h +++ b/sys/fs/unionfs/union.h @@ -97,15 +97,17 @@ struct unionfs_node { char *un_path; /* path */ int un_pathlen; /* strlen of path */ - int un_flag; /* unionfs node flag */ -}; -/* - * unionfs node flags - * It needs the vnode with exclusive lock, when changing the un_flag variable. - */ -#define UNIONFS_OPENEXTL 0x01 /* openextattr (lower) */ -#define UNIONFS_OPENEXTU 0x02 /* openextattr (upper) */ + /* + * unionfs node flags + * Changing these flags requires the vnode to be locked exclusive. + */ + #define UNIONFS_OPENEXTL 0x01 /* openextattr (lower) */ + #define UNIONFS_OPENEXTU 0x02 /* openextattr (upper) */ + #define UNIONFS_COPY_IN_PROGRESS 0x04 /* copy/dir shadow in progres */ + #define UNIONFS_LOOKUP_IN_PROGRESS 0x08 + unsigned int un_flag; /* unionfs node flag */ +}; extern struct vop_vector unionfs_vnodeops; @@ -136,29 +138,25 @@ void unionfs_get_node_status(struct unionfs_node *, struct thread *, void unionfs_tryrem_node_status(struct unionfs_node *, struct unionfs_node_status *); int unionfs_check_rmdir(struct vnode *, struct ucred *, struct thread *td); -int unionfs_copyfile(struct unionfs_node *, int, struct ucred *, +int unionfs_copyfile(struct vnode *, int, struct ucred *, struct thread *); void unionfs_create_uppervattr_core(struct unionfs_mount *, struct vattr *, struct vattr *, struct thread *); int unionfs_create_uppervattr(struct unionfs_mount *, struct vnode *, struct vattr *, struct ucred *, struct thread *); -int unionfs_mkshadowdir(struct unionfs_mount *, struct vnode *, - struct unionfs_node *, struct componentname *, struct thread *); +int unionfs_mkshadowdir(struct vnode *, struct vnode *, + struct componentname *, struct thread *); int unionfs_mkwhiteout(struct vnode *, struct vnode *, struct componentname *, struct thread *, char *, int); int unionfs_relookup(struct vnode *, struct vnode **, struct componentname *, struct componentname *, struct thread *, char *, int, u_long); -int unionfs_relookup_for_create(struct vnode *, struct componentname *, - struct thread *); -int unionfs_relookup_for_delete(struct vnode *, struct componentname *, - struct thread *); -int unionfs_relookup_for_rename(struct vnode *, struct componentname *, - struct thread *); void unionfs_forward_vop_start_pair(struct vnode *, int *, struct vnode *, int *); bool unionfs_forward_vop_finish_pair(struct vnode *, struct vnode *, int, struct vnode *, struct vnode *, int); +int unionfs_set_in_progress_flag(struct vnode *, unsigned int); +void unionfs_clear_in_progress_flag(struct vnode *, unsigned int); static inline void unionfs_forward_vop_start(struct vnode *basevp, int *lkflags) diff --git a/sys/fs/unionfs/union_subr.c b/sys/fs/unionfs/union_subr.c index bb57f3d56ade..c46696760e85 100644 --- a/sys/fs/unionfs/union_subr.c +++ b/sys/fs/unionfs/union_subr.c @@ -203,19 +203,19 @@ unionfs_ins_cached_vnode(struct unionfs_node *uncp, struct unionfs_node_hashhead *hd; struct vnode *vp; - ASSERT_VOP_ELOCKED(uncp->un_uppervp, __func__); - ASSERT_VOP_ELOCKED(uncp->un_lowervp, __func__); - KASSERT(uncp->un_uppervp == NULLVP || uncp->un_uppervp->v_type == VDIR, - ("%s: v_type != VDIR", __func__)); - KASSERT(uncp->un_lowervp == NULLVP || uncp->un_lowervp->v_type == VDIR, - ("%s: v_type != VDIR", __func__)); - vp = NULLVP; VI_LOCK(dvp); - if (uncp->un_uppervp != NULL) + if (uncp->un_uppervp != NULLVP) { + ASSERT_VOP_ELOCKED(uncp->un_uppervp, __func__); + KASSERT(uncp->un_uppervp->v_type == VDIR, + ("%s: v_type != VDIR", __func__)); vp = unionfs_get_cached_vnode_locked(uncp->un_uppervp, dvp); - else if (uncp->un_lowervp != NULL) + } else if (uncp->un_lowervp != NULLVP) { + ASSERT_VOP_ELOCKED(uncp->un_lowervp, __func__); + KASSERT(uncp->un_lowervp->v_type == VDIR, + ("%s: v_type != VDIR", __func__)); vp = unionfs_get_cached_vnode_locked(uncp->un_lowervp, dvp); + } if (vp == NULLVP) { hd = unionfs_get_hashhead(dvp, (uncp->un_uppervp != NULLVP ? uncp->un_uppervp : uncp->un_lowervp)); @@ -278,8 +278,10 @@ unionfs_nodeget_cleanup(struct vnode *vp, struct unionfs_node *unp) vrele(unp->un_dvp); if (unp->un_uppervp != NULLVP) vput(unp->un_uppervp); + else if (unp->un_lowervp != NULLVP) + VOP_UNLOCK(unp->un_lowervp); if (unp->un_lowervp != NULLVP) - vput(unp->un_lowervp); + vrele(unp->un_lowervp); if (unp->un_hashtbl != NULL) hashdestroy(unp->un_hashtbl, M_UNIONFSHASH, UNIONFSHASHMASK); free(unp->un_path, M_UNIONFSPATH); @@ -314,7 +316,7 @@ unionfs_nodeget(struct mount *mp, struct vnode *uppervp, *vpp = NULLVP; if (uppervp == NULLVP && lowervp == NULLVP) - panic("%s: upper and lower is null", __func__); + panic("%s: upper and lower are both null", __func__); vt = (uppervp != NULLVP ? uppervp->v_type : lowervp->v_type); @@ -327,7 +329,9 @@ unionfs_nodeget(struct mount *mp, struct vnode *uppervp, vp = unionfs_get_cached_vnode(uppervp, lowervp, dvp); if (vp != NULLVP) { *vpp = vp; - goto unionfs_nodeget_out; + if (lkflags != 0) + vn_lock(*vpp, lkflags | LK_RETRY); + return (0); } } @@ -385,24 +389,36 @@ unionfs_nodeget(struct mount *mp, struct vnode *uppervp, KASSERT(dvp != NULL || (vp->v_vflag & VV_ROOT) != 0, ("%s: NULL dvp for non-root vp %p", __func__, vp)); - vn_lock_pair(lowervp, false, LK_EXCLUSIVE, uppervp, false, - LK_EXCLUSIVE); + + /* + * NOTE: There is still a possibility for cross-filesystem locking here. + * If dvp has an upper FS component and is locked, while the new vnode + * created here only has a lower-layer FS component, then we will end + * up taking a lower-FS lock while holding an upper-FS lock. + * That situation could be dealt with here, with a fair amount of added + * complexity, using either vn_lock_pair() or by attempting to + * lock lowervp with LK_NOWAIT and returning ERELOOKUP upon lock failure. + * However, that would only address one instance out of many on in which + * a child vnode lock is taken while holding a lock on its parent + * directory. This is done in many places in common VFS code, as well as + * a few places within unionfs (which could lead to the same cross-FS + * locking issue if, for example, the upper FS is another nested unionfs + * instance). + */ + if (uppervp != NULLVP) + vn_lock(uppervp, LK_EXCLUSIVE | LK_RETRY); + else + vn_lock(lowervp, LK_EXCLUSIVE | LK_RETRY); error = insmntque1(vp, mp); if (error != 0) { unionfs_nodeget_cleanup(vp, unp); return (error); } - if (lowervp != NULL && VN_IS_DOOMED(lowervp)) { - vput(lowervp); - unp->un_lowervp = lowervp = NULL; - } - if (uppervp != NULL && VN_IS_DOOMED(uppervp)) { - vput(uppervp); - unp->un_uppervp = uppervp = NULL; - if (lowervp != NULLVP) - vp->v_vnlock = lowervp->v_vnlock; + if (uppervp == NULLVP && VN_IS_DOOMED(lowervp)) { + unionfs_nodeget_cleanup(vp, unp); + return (ENOENT); } - if (lowervp == NULL && uppervp == NULL) { + if (uppervp != NULLVP && VN_IS_DOOMED(uppervp)) { unionfs_nodeget_cleanup(vp, unp); return (ENOENT); } @@ -413,18 +429,16 @@ unionfs_nodeget(struct mount *mp, struct vnode *uppervp, *vpp = unionfs_ins_cached_vnode(unp, dvp); if (*vpp != NULLVP) { unionfs_nodeget_cleanup(vp, unp); - vp = *vpp; - } else { - if (uppervp != NULL) - VOP_UNLOCK(uppervp); - if (lowervp != NULL) - VOP_UNLOCK(lowervp); + if (lkflags != 0) + vn_lock(*vpp, lkflags | LK_RETRY); + return (0); + } else *vpp = vp; - } -unionfs_nodeget_out: - if (lkflags & LK_TYPE_MASK) - vn_lock(vp, lkflags | LK_RETRY); + if ((lkflags & LK_SHARED) != 0) + vn_lock(vp, LK_DOWNGRADE); + else if ((lkflags & LK_EXCLUSIVE) == 0) + VOP_UNLOCK(vp); return (0); } @@ -443,6 +457,7 @@ unionfs_noderem(struct vnode *vp) struct vnode *dvp; int count; int writerefs; + bool unlock_lvp; /* * The root vnode lock may be recursed during unmount, because @@ -455,18 +470,25 @@ unionfs_noderem(struct vnode *vp) */ KASSERT(vp->v_vnlock->lk_recurse == 0 || (vp->v_vflag & VV_ROOT) != 0, ("%s: vnode %p locked recursively", __func__, vp)); + + unp = VTOUNIONFS(vp); + VNASSERT(unp != NULL, vp, ("%s: already reclaimed", __func__)); + lvp = unp->un_lowervp; + uvp = unp->un_uppervp; + dvp = unp->un_dvp; + unlock_lvp = (uvp == NULLVP); + if (uvp != NULLVP && lvp != NULLVP && (vp->v_vflag & VV_ROOT) == 0) { + vn_lock_pair(uvp, true, LK_EXCLUSIVE, lvp, false, LK_EXCLUSIVE); + unlock_lvp = true; + } + if (lockmgr(&vp->v_lock, LK_EXCLUSIVE | LK_NOWAIT, NULL) != 0) panic("%s: failed to acquire lock for vnode lock", __func__); - /* * Use the interlock to protect the clearing of v_data to * prevent faults in unionfs_lock(). */ VI_LOCK(vp); - unp = VTOUNIONFS(vp); - lvp = unp->un_lowervp; - uvp = unp->un_uppervp; - dvp = unp->un_dvp; unp->un_lowervp = unp->un_uppervp = NULLVP; vp->v_vnlock = &(vp->v_lock); vp->v_data = NULL; @@ -502,18 +524,16 @@ unionfs_noderem(struct vnode *vp) ("%s: write reference without upper vnode", __func__)); VOP_ADD_WRITECOUNT(uvp, -writerefs); } - if (lvp != NULLVP) - VOP_UNLOCK(lvp); if (uvp != NULLVP) - VOP_UNLOCK(uvp); + vput(uvp); + if (unlock_lvp) + vput(lvp); + else if (lvp != NULLVP) + vrele(lvp); if (dvp != NULLVP) unionfs_rem_cached_vnode(unp, dvp); - if (lvp != NULLVP) - vrele(lvp); - if (uvp != NULLVP) - vrele(uvp); if (unp->un_path != NULL) { free(unp->un_path, M_UNIONFSPATH); unp->un_path = NULL; @@ -696,110 +716,6 @@ unionfs_relookup(struct vnode *dvp, struct vnode **vpp, return (error); } -/* - * relookup for CREATE namei operation. - * - * dvp is unionfs vnode. dvp should be locked. - * - * If it called 'unionfs_copyfile' function by unionfs_link etc, - * VOP_LOOKUP information is broken. - * So it need relookup in order to create link etc. - */ -int -unionfs_relookup_for_create(struct vnode *dvp, struct componentname *cnp, - struct thread *td) -{ - struct vnode *udvp; - struct vnode *vp; - struct componentname cn; - int error; - - udvp = UNIONFSVPTOUPPERVP(dvp); - vp = NULLVP; - - error = unionfs_relookup(udvp, &vp, cnp, &cn, td, cnp->cn_nameptr, - cnp->cn_namelen, CREATE); - if (error) - return (error); - - if (vp != NULLVP) { - if (udvp == vp) - vrele(vp); - else - vput(vp); - - error = EEXIST; - } - - return (error); -} - -/* - * relookup for DELETE namei operation. - * - * dvp is unionfs vnode. dvp should be locked. - */ -int -unionfs_relookup_for_delete(struct vnode *dvp, struct componentname *cnp, - struct thread *td) -{ - struct vnode *udvp; - struct vnode *vp; - struct componentname cn; - int error; - - udvp = UNIONFSVPTOUPPERVP(dvp); - vp = NULLVP; - - error = unionfs_relookup(udvp, &vp, cnp, &cn, td, cnp->cn_nameptr, - cnp->cn_namelen, DELETE); - if (error) - return (error); - - if (vp == NULLVP) - error = ENOENT; - else { - if (udvp == vp) - vrele(vp); - else - vput(vp); - } - - return (error); -} - -/* - * relookup for RENAME namei operation. - * - * dvp is unionfs vnode. dvp should be locked. - */ -int -unionfs_relookup_for_rename(struct vnode *dvp, struct componentname *cnp, - struct thread *td) -{ - struct vnode *udvp; - struct vnode *vp; - struct componentname cn; - int error; - - udvp = UNIONFSVPTOUPPERVP(dvp); - vp = NULLVP; - - error = unionfs_relookup(udvp, &vp, cnp, &cn, td, cnp->cn_nameptr, - cnp->cn_namelen, RENAME); - if (error) - return (error); - - if (vp != NULLVP) { - if (udvp == vp) - vrele(vp); - else - vput(vp); - } - - return (error); -} - /* * Update the unionfs_node. * @@ -836,6 +752,8 @@ unionfs_node_update(struct unionfs_node *unp, struct vnode *uvp, vp->v_vnlock = uvp->v_vnlock; VI_UNLOCK(vp); + for (count = 0; count < lockrec + 1; count++) + VOP_UNLOCK(lvp); /* * Re-cache the unionfs vnode against the upper vnode */ @@ -850,19 +768,89 @@ unionfs_node_update(struct unionfs_node *unp, struct vnode *uvp, } } +/* + * Mark a unionfs operation as being in progress, sleeping if the + * same operation is already in progress. + * This is useful, for example, during copy-up operations in which + * we may drop the target vnode lock, but we want to avoid the + * possibility of a concurrent copy-up on the same vnode triggering + * a spurious failure. + */ +int +unionfs_set_in_progress_flag(struct vnode *vp, unsigned int flag) +{ + struct unionfs_node *unp; + int error; + + error = 0; + ASSERT_VOP_ELOCKED(vp, __func__); + VI_LOCK(vp); + unp = VTOUNIONFS(vp); + while (error == 0 && (unp->un_flag & flag) != 0) { + VOP_UNLOCK(vp); + error = msleep(vp, VI_MTX(vp), PCATCH, "unioncp", 0); + VI_UNLOCK(vp); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); + VI_LOCK(vp); + if (error == 0) { + /* + * If we waited on a concurrent copy-up and that + * copy-up was successful, return a non-fatal + * indication that the desired operation is already + * complete. If we waited on a concurrent lookup, + * return ERELOOKUP to indicate the VFS cache should + * be re-queried to avoid creating a duplicate unionfs + * vnode. + */ + unp = VTOUNIONFS(vp); + if (unp == NULL) + error = ENOENT; + else if (flag == UNIONFS_COPY_IN_PROGRESS && + unp->un_uppervp != NULLVP) + error = EJUSTRETURN; + else if (flag == UNIONFS_LOOKUP_IN_PROGRESS) + error = ERELOOKUP; + } + } + if (error == 0) + unp->un_flag |= flag; + VI_UNLOCK(vp); + + return (error); +} + +void +unionfs_clear_in_progress_flag(struct vnode *vp, unsigned int flag) +{ + struct unionfs_node *unp; + + ASSERT_VOP_ELOCKED(vp, __func__); + unp = VTOUNIONFS(vp); + if (unp != NULL) { + VNASSERT((unp->un_flag & flag) != 0, vp, + ("%s: copy not in progress", __func__)); + VI_LOCK(vp); + unp->un_flag &= ~flag; + wakeup(vp); + VI_UNLOCK(vp); + } +} + /* * Create a new shadow dir. * - * udvp should be locked on entry and will be locked on return. + * dvp and vp are unionfs vnodes representing a parent directory and + * child file, should be locked on entry, and will be locked on return. * * If no error returned, unp will be updated. */ int -unionfs_mkshadowdir(struct unionfs_mount *ump, struct vnode *udvp, - struct unionfs_node *unp, struct componentname *cnp, struct thread *td) +unionfs_mkshadowdir(struct vnode *dvp, struct vnode *vp, + struct componentname *cnp, struct thread *td) { struct vnode *lvp; struct vnode *uvp; + struct vnode *udvp; struct vattr va; struct vattr lva; struct nameidata nd; @@ -870,10 +858,25 @@ unionfs_mkshadowdir(struct unionfs_mount *ump, struct vnode *udvp, struct ucred *cred; struct ucred *credbk; struct uidinfo *rootinfo; + struct unionfs_mount *ump; + struct unionfs_node *dunp; + struct unionfs_node *unp; int error; + ASSERT_VOP_ELOCKED(dvp, __func__); + ASSERT_VOP_ELOCKED(vp, __func__); + ump = MOUNTTOUNIONFSMOUNT(vp->v_mount); + unp = VTOUNIONFS(vp); if (unp->un_uppervp != NULLVP) return (EEXIST); + dunp = VTOUNIONFS(dvp); + udvp = dunp->un_uppervp; + + error = unionfs_set_in_progress_flag(vp, UNIONFS_COPY_IN_PROGRESS); + if (error == EJUSTRETURN) + return (0); + else if (error != 0) + return (error); lvp = unp->un_lowervp; uvp = NULLVP; @@ -897,11 +900,15 @@ unionfs_mkshadowdir(struct unionfs_mount *ump, struct vnode *udvp, NDPREINIT(&nd); if ((error = VOP_GETATTR(lvp, &lva, cnp->cn_cred))) - goto unionfs_mkshadowdir_abort; + goto unionfs_mkshadowdir_exit1; + vref(udvp); + VOP_UNLOCK(vp); if ((error = unionfs_relookup(udvp, &uvp, cnp, &nd.ni_cnd, td, - cnp->cn_nameptr, cnp->cn_namelen, CREATE))) - goto unionfs_mkshadowdir_abort; + cnp->cn_nameptr, cnp->cn_namelen, CREATE))) { + vput(udvp); + goto unionfs_mkshadowdir_exit2; + } if (uvp != NULLVP) { if (udvp == uvp) vrele(uvp); @@ -909,11 +916,14 @@ unionfs_mkshadowdir(struct unionfs_mount *ump, struct vnode *udvp, vput(uvp); error = EEXIST; - goto unionfs_mkshadowdir_abort; + vput(udvp); + goto unionfs_mkshadowdir_exit2; } - if ((error = vn_start_write(udvp, &mp, V_WAIT | V_PCATCH))) - goto unionfs_mkshadowdir_abort; + if ((error = vn_start_write(udvp, &mp, V_WAIT | V_PCATCH))) { + vput(udvp); + goto unionfs_mkshadowdir_exit2; + } unionfs_create_uppervattr_core(ump, &lva, &va, td); /* @@ -934,27 +944,47 @@ unionfs_mkshadowdir(struct unionfs_mount *ump, struct vnode *udvp, *pathend = '\0'; error = VOP_MKDIR(udvp, &uvp, &nd.ni_cnd, &va); *pathend = pathterm; + if (error != 0) { + vput(udvp); + goto unionfs_mkshadowdir_exit3; + } - if (!error) { - /* - * XXX The bug which cannot set uid/gid was corrected. - * Ignore errors. - */ - va.va_type = VNON; - VOP_SETATTR(uvp, &va, nd.ni_cnd.cn_cred); + /* + * XXX The bug which cannot set uid/gid was corrected. + * Ignore errors. + */ + va.va_type = VNON; + /* + * VOP_SETATTR() may transiently drop uvp's lock, so it's + * important to call it before unionfs_node_update() transfers + * the unionfs vnode's lock from lvp to uvp; otherwise the + * unionfs vnode itself would be transiently unlocked and + * potentially doomed. + */ + VOP_SETATTR(uvp, &va, nd.ni_cnd.cn_cred); - /* - * VOP_SETATTR() may transiently drop uvp's lock, so it's - * important to call it before unionfs_node_update() transfers - * the unionfs vnode's lock from lvp to uvp; otherwise the - * unionfs vnode itself would be transiently unlocked and - * potentially doomed. - */ + VOP_UNLOCK(uvp); + vput(udvp); + vn_lock_pair(vp, false, LK_EXCLUSIVE, uvp, false, LK_EXCLUSIVE); + unp = VTOUNIONFS(vp); + if (unp == NULL) { + vput(uvp); + error = ENOENT; + } else unionfs_node_update(unp, uvp, td); - } + VOP_UNLOCK(vp); + +unionfs_mkshadowdir_exit3: vn_finished_write(mp); -unionfs_mkshadowdir_abort: +unionfs_mkshadowdir_exit2: + vn_lock(dvp, LK_EXCLUSIVE | LK_RETRY); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); + if (error == 0 && (VN_IS_DOOMED(dvp) || VN_IS_DOOMED(vp))) + error = ENOENT; + +unionfs_mkshadowdir_exit1: + unionfs_clear_in_progress_flag(vp, UNIONFS_COPY_IN_PROGRESS); cnp->cn_cred = credbk; chgproccnt(cred->cr_ruidinfo, -1, 0); crfree(cred); @@ -1116,23 +1146,31 @@ unionfs_forward_vop_finish_pair( /* * Create a new whiteout. * - * udvp and dvp should be locked on entry and will be locked on return. + * dvp and vp are unionfs vnodes representing a parent directory and + * child file, should be locked on entry, and will be locked on return. */ int -unionfs_mkwhiteout(struct vnode *dvp, struct vnode *udvp, +unionfs_mkwhiteout(struct vnode *dvp, struct vnode *vp, struct componentname *cnp, struct thread *td, char *path, int pathlen) { + struct vnode *udvp; struct vnode *wvp; struct nameidata nd; struct mount *mp; int error; - int lkflags; + bool dvp_locked; + + ASSERT_VOP_ELOCKED(dvp, __func__); + ASSERT_VOP_ELOCKED(vp, __func__); + udvp = VTOUNIONFS(dvp)->un_uppervp; wvp = NULLVP; NDPREINIT(&nd); + vref(udvp); + VOP_UNLOCK(vp); if ((error = unionfs_relookup(udvp, &wvp, cnp, &nd.ni_cnd, td, path, pathlen, CREATE))) { - return (error); + goto unionfs_mkwhiteout_cleanup; } if (wvp != NULLVP) { if (udvp == wvp) @@ -1140,18 +1178,27 @@ unionfs_mkwhiteout(struct vnode *dvp, struct vnode *udvp, else vput(wvp); - return (EEXIST); + if (nd.ni_cnd.cn_flags & ISWHITEOUT) + error = 0; + else + error = EEXIST; + goto unionfs_mkwhiteout_cleanup; } if ((error = vn_start_write(udvp, &mp, V_WAIT | V_PCATCH))) - goto unionfs_mkwhiteout_free_out; - unionfs_forward_vop_start(udvp, &lkflags); + goto unionfs_mkwhiteout_cleanup; error = VOP_WHITEOUT(udvp, &nd.ni_cnd, CREATE); - unionfs_forward_vop_finish(dvp, udvp, lkflags); - vn_finished_write(mp); -unionfs_mkwhiteout_free_out: +unionfs_mkwhiteout_cleanup: + if (VTOUNIONFS(dvp) == NULL) { + vput(udvp); + dvp_locked = false; + } else { + vrele(udvp); + dvp_locked = true; + } + vn_lock_pair(dvp, dvp_locked, LK_EXCLUSIVE, vp, false, LK_EXCLUSIVE); return (error); } @@ -1165,10 +1212,11 @@ unionfs_mkwhiteout(struct vnode *dvp, struct vnode *udvp, */ static int unionfs_vn_create_on_upper(struct vnode **vpp, struct vnode *udvp, - struct unionfs_node *unp, struct vattr *uvap, struct thread *td) + struct vnode *vp, struct vattr *uvap, struct thread *td) { struct unionfs_mount *ump; - struct vnode *vp; + struct unionfs_node *unp; + struct vnode *uvp; struct vnode *lvp; struct ucred *cred; struct vattr lva; @@ -1176,8 +1224,10 @@ unionfs_vn_create_on_upper(struct vnode **vpp, struct vnode *udvp, int fmode; int error; + ASSERT_VOP_ELOCKED(vp, __func__); + unp = VTOUNIONFS(vp); ump = MOUNTTOUNIONFSMOUNT(UNIONFSTOV(unp)->v_mount); - vp = NULLVP; + uvp = NULLVP; lvp = unp->un_lowervp; cred = td->td_ucred; fmode = FFLAGS(O_WRONLY | O_CREAT | O_TRUNC | O_EXCL); @@ -1200,42 +1250,39 @@ unionfs_vn_create_on_upper(struct vnode **vpp, struct vnode *udvp, NDPREINIT(&nd); vref(udvp); - if ((error = vfs_relookup(udvp, &vp, &nd.ni_cnd, false)) != 0) - goto unionfs_vn_create_on_upper_free_out2; - vrele(udvp); + VOP_UNLOCK(vp); + if ((error = vfs_relookup(udvp, &uvp, &nd.ni_cnd, false)) != 0) { + vrele(udvp); + return (error); + } - if (vp != NULLVP) { - if (vp == udvp) - vrele(vp); + if (uvp != NULLVP) { + if (uvp == udvp) + vrele(uvp); else - vput(vp); + vput(uvp); error = EEXIST; - goto unionfs_vn_create_on_upper_free_out1; + goto unionfs_vn_create_on_upper_cleanup; } - if ((error = VOP_CREATE(udvp, &vp, &nd.ni_cnd, uvap)) != 0) - goto unionfs_vn_create_on_upper_free_out1; + if ((error = VOP_CREATE(udvp, &uvp, &nd.ni_cnd, uvap)) != 0) + goto unionfs_vn_create_on_upper_cleanup; - if ((error = VOP_OPEN(vp, fmode, cred, td, NULL)) != 0) { - vput(vp); - goto unionfs_vn_create_on_upper_free_out1; + if ((error = VOP_OPEN(uvp, fmode, cred, td, NULL)) != 0) { + vput(uvp); + goto unionfs_vn_create_on_upper_cleanup; } - error = VOP_ADD_WRITECOUNT(vp, 1); - CTR3(KTR_VFS, "%s: vp %p v_writecount increased to %d", - __func__, vp, vp->v_writecount); + error = VOP_ADD_WRITECOUNT(uvp, 1); + CTR3(KTR_VFS, "%s: newvp %p v_writecount increased to %d", + __func__, newvp, newvp->v_writecount); if (error == 0) { - *vpp = vp; + *vpp = uvp; } else { - VOP_CLOSE(vp, fmode, cred, td); + VOP_CLOSE(uvp, fmode, cred, td); } -unionfs_vn_create_on_upper_free_out1: - VOP_UNLOCK(udvp); - -unionfs_vn_create_on_upper_free_out2: - KASSERT(nd.ni_cnd.cn_pnbuf == unp->un_path, - ("%s: cn_pnbuf changed", __func__)); - +unionfs_vn_create_on_upper_cleanup: + vput(udvp); return (error); } @@ -1310,13 +1357,18 @@ unionfs_copyfile_core(struct vnode *lvp, struct vnode *uvp, * * If you need copy of the contents, set 1 to docopy. Otherwise, set 0 to * docopy. + * + * vp is a unionfs vnode that should be locked on entry and will be + * locked on return. * * If no error returned, unp will be updated. */ int -unionfs_copyfile(struct unionfs_node *unp, int docopy, struct ucred *cred, +unionfs_copyfile(struct vnode *vp, int docopy, struct ucred *cred, struct thread *td) { + struct unionfs_node *unp; + struct unionfs_node *dunp; struct mount *mp; struct vnode *udvp; struct vnode *lvp; @@ -1324,6 +1376,8 @@ unionfs_copyfile(struct unionfs_node *unp, int docopy, struct ucred *cred, struct vattr uva; int error; + ASSERT_VOP_ELOCKED(vp, __func__); + unp = VTOUNIONFS(vp); lvp = unp->un_lowervp; uvp = NULLVP; @@ -1333,22 +1387,51 @@ unionfs_copyfile(struct unionfs_node *unp, int docopy, struct ucred *cred, return (EINVAL); if (unp->un_uppervp != NULLVP) return (EEXIST); - udvp = VTOUNIONFS(unp->un_dvp)->un_uppervp; + + udvp = NULLVP; + VI_LOCK(unp->un_dvp); + dunp = VTOUNIONFS(unp->un_dvp); + if (dunp != NULL) + udvp = dunp->un_uppervp; + VI_UNLOCK(unp->un_dvp); + if (udvp == NULLVP) return (EROFS); if ((udvp->v_mount->mnt_flag & MNT_RDONLY)) return (EROFS); + ASSERT_VOP_UNLOCKED(udvp, __func__); + + error = unionfs_set_in_progress_flag(vp, UNIONFS_COPY_IN_PROGRESS); + if (error == EJUSTRETURN) + return (0); + else if (error != 0) + return (error); error = VOP_ACCESS(lvp, VREAD, cred, td); if (error != 0) - return (error); + goto unionfs_copyfile_cleanup; if ((error = vn_start_write(udvp, &mp, V_WAIT | V_PCATCH)) != 0) - return (error); - error = unionfs_vn_create_on_upper(&uvp, udvp, unp, &uva, td); + goto unionfs_copyfile_cleanup; + error = unionfs_vn_create_on_upper(&uvp, udvp, vp, &uva, td); if (error != 0) { vn_finished_write(mp); - return (error); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); + goto unionfs_copyfile_cleanup; + } + + /* + * Note that it's still possible for e.g. VOP_WRITE to relock + * uvp below while holding vp[=lvp] locked. Replacing + * unionfs_copyfile_core with vn_generic_copy_file_range() will + * allow us to avoid the problem by moving this vn_lock_pair() + * call much later. + */ + vn_lock_pair(vp, false, LK_EXCLUSIVE, uvp, true, LK_EXCLUSIVE); + unp = VTOUNIONFS(vp); + if (unp == NULL) { + error = ENOENT; + goto unionfs_copyfile_cleanup; } if (docopy != 0) { @@ -1369,18 +1452,30 @@ unionfs_copyfile(struct unionfs_node *unp, int docopy, struct ucred *cred, /* Reset the attributes. Ignore errors. */ uva.va_type = VNON; VOP_SETATTR(uvp, &uva, cred); + unionfs_node_update(unp, uvp, td); } - unionfs_node_update(unp, uvp, td); - +unionfs_copyfile_cleanup: + unionfs_clear_in_progress_flag(vp, UNIONFS_COPY_IN_PROGRESS); return (error); } /* - * It checks whether vp can rmdir. (check empty) + * Determine if the unionfs view of a directory is empty such that + * an rmdir operation can be permitted. + * + * We assume the VOP_RMDIR() against the upper layer vnode will take + * care of this check for us where the upper FS is concerned, so here + * we concentrate on the lower FS. We need to check for the presence + * of files other than "." and ".." in the lower FS directory and + * then cross-check any files we find against the upper FS to see if + * a whiteout is present (in which case we treat the lower file as + * non-present). + * + * The logic here is based heavily on vn_dir_check_empty(). * - * vp is unionfs vnode. - * vp should be locked. + * vp should be a locked unionfs node, and vp's lowervp should also be + * locked. */ int unionfs_check_rmdir(struct vnode *vp, struct ucred *cred, struct thread *td) @@ -1388,115 +1483,130 @@ unionfs_check_rmdir(struct vnode *vp, struct ucred *cred, struct thread *td) struct vnode *uvp; struct vnode *lvp; struct vnode *tvp; + char *dirbuf; + size_t dirbuflen, len; + off_t off; struct dirent *dp; - struct dirent *edp; struct componentname cn; - struct iovec iov; - struct uio uio; struct vattr va; int error; int eofflag; - int lookuperr; - - /* - * The size of buf needs to be larger than DIRBLKSIZ. - */ - char buf[256 * 6]; - - ASSERT_VOP_ELOCKED(vp, __func__); eofflag = 0; - uvp = UNIONFSVPTOUPPERVP(vp); lvp = UNIONFSVPTOLOWERVP(vp); + uvp = UNIONFSVPTOUPPERVP(vp); + + /* + * Note that the locking here still isn't ideal: We expect the caller + * to hold both the upper and lower layer locks as well as the upper + * parent directory lock, which it can do in a manner that avoids + * deadlock. However, if the cross-check logic below needs to call + * VOP_LOOKUP(), that may relock the upper vnode and lock any found + * child vnode in a way that doesn't protect against deadlock given + * the other held locks. Beyond that, the various other VOPs we issue + * below, such as VOP_OPEN() and VOP_READDIR(), may also re-lock the + * lower vnode. + * We might instead just handoff between the upper vnode lock + * (and its parent directory lock) and the lower vnode lock as needed, + * so that the lower lock is never held at the same time as the upper + * locks, but that opens up a wider window in which the upper + * directory (and also the lower directory if it isn't truly + * read-only) may change while the relevant lock is dropped. But + * since re-locking may happen here and open up such a window anyway, + * perhaps that is a worthwile tradeoff? Or perhaps we can ultimately + * do sufficient tracking of empty state within the unionfs vnode + * (in conjunction with upcalls from the lower FSes to notify us + * of out-of-band state changes) that we can avoid these costly checks + * altogether. + */ + ASSERT_VOP_LOCKED(lvp, __func__); + ASSERT_VOP_ELOCKED(uvp, __func__); - /* check opaque */ if ((error = VOP_GETATTR(uvp, &va, cred)) != 0) return (error); if (va.va_flags & OPAQUE) return (0); - /* open vnode */ #ifdef MAC - if ((error = mac_vnode_check_open(cred, vp, VEXEC|VREAD)) != 0) + if ((error = mac_vnode_check_open(cred, lvp, VEXEC | VREAD)) != 0) return (error); #endif - if ((error = VOP_ACCESS(vp, VEXEC|VREAD, cred, td)) != 0) + if ((error = VOP_ACCESS(lvp, VEXEC | VREAD, cred, td)) != 0) + return (error); + /* + * VOP_OPEN() shouldn't be truly needed here, except that + * unionfs currently requires OPEN() before READDIR(), + * and we might be dealing with a nested unionfs. + */ + if ((error = VOP_OPEN(lvp, FREAD, cred, td, NULL)) != 0) return (error); - if ((error = VOP_OPEN(vp, FREAD, cred, td, NULL)) != 0) + if ((error = VOP_GETATTR(lvp, &va, cred)) != 0) return (error); - uio.uio_rw = UIO_READ; - uio.uio_segflg = UIO_SYSSPACE; - uio.uio_td = td; - uio.uio_offset = 0; + dirbuflen = max(DEV_BSIZE, GENERIC_MAXDIRSIZ); + if (dirbuflen < va.va_blocksize) + dirbuflen = va.va_blocksize; + dirbuf = malloc(dirbuflen, M_TEMP, M_WAITOK); -#ifdef MAC - error = mac_vnode_check_readdir(td->td_ucred, lvp); -#endif - while (!error && !eofflag) { - iov.iov_base = buf; - iov.iov_len = sizeof(buf); - uio.uio_iov = &iov; - uio.uio_iovcnt = 1; - uio.uio_resid = iov.iov_len; + len = 0; + off = 0; + eofflag = 0; - error = VOP_READDIR(lvp, &uio, cred, &eofflag, NULL, NULL); + for (;;) { + error = vn_dir_next_dirent(lvp, td, dirbuf, dirbuflen, + &dp, &len, &off, &eofflag); if (error != 0) break; - KASSERT(eofflag != 0 || uio.uio_resid < sizeof(buf), - ("%s: empty read from lower FS", __func__)); - - edp = (struct dirent*)&buf[sizeof(buf) - uio.uio_resid]; - for (dp = (struct dirent*)buf; !error && dp < edp; - dp = (struct dirent*)((caddr_t)dp + dp->d_reclen)) { - if (dp->d_type == DT_WHT || dp->d_fileno == 0 || - (dp->d_namlen == 1 && dp->d_name[0] == '.') || - (dp->d_namlen == 2 && !bcmp(dp->d_name, "..", 2))) - continue; - - cn.cn_namelen = dp->d_namlen; - cn.cn_pnbuf = NULL; - cn.cn_nameptr = dp->d_name; - cn.cn_nameiop = LOOKUP; - cn.cn_flags = LOCKPARENT | LOCKLEAF | RDONLY | ISLASTCN; - cn.cn_lkflags = LK_EXCLUSIVE; - cn.cn_cred = cred; - - /* - * check entry in lower. - * Sometimes, readdir function returns - * wrong entry. - */ - lookuperr = VOP_LOOKUP(lvp, &tvp, &cn); - if (!lookuperr) - vput(tvp); - else - continue; /* skip entry */ - - /* - * check entry - * If it has no exist/whiteout entry in upper, - * directory is not empty. - */ - cn.cn_flags = LOCKPARENT | LOCKLEAF | RDONLY | ISLASTCN; - lookuperr = VOP_LOOKUP(uvp, &tvp, &cn); - - if (!lookuperr) - vput(tvp); + if (len == 0) { + /* EOF */ + error = 0; + break; + } - /* ignore exist or whiteout entry */ - if (!lookuperr || - (lookuperr == ENOENT && (cn.cn_flags & ISWHITEOUT))) - continue; + if (dp->d_type == DT_WHT) + continue; - error = ENOTEMPTY; + /* + * Any file in the directory which is not '.' or '..' indicates + * the directory is not empty. + */ + switch (dp->d_namlen) { + case 2: + if (dp->d_name[1] != '.') { + /* Can't be '..' (nor '.') */ + break; + } + /* FALLTHROUGH */ + case 1: + if (dp->d_name[0] != '.') { + /* Can't be '..' nor '.' */ + break; + } + continue; + default: + break; } - } - /* close vnode */ - VOP_CLOSE(vp, FREAD, cred, td); + cn.cn_namelen = dp->d_namlen; + cn.cn_pnbuf = NULL; + cn.cn_nameptr = dp->d_name; + cn.cn_nameiop = LOOKUP; + cn.cn_flags = LOCKPARENT | LOCKLEAF | RDONLY | ISLASTCN; + cn.cn_lkflags = LK_EXCLUSIVE; + cn.cn_cred = cred; + + if (VOP_LOOKUP(uvp, &tvp, &cn) == 0) + vput(tvp); + + if (cn.cn_flags & ISWHITEOUT) + continue; + + error = ENOTEMPTY; + break; + } + VOP_CLOSE(lvp, FREAD, cred, td); + free(dirbuf, M_TEMP); return (error); } - diff --git a/sys/fs/unionfs/union_vfsops.c b/sys/fs/unionfs/union_vfsops.c index cb55c2dd6474..9342317ad08e 100644 --- a/sys/fs/unionfs/union_vfsops.c +++ b/sys/fs/unionfs/union_vfsops.c @@ -327,18 +327,15 @@ unionfs_domount(struct mount *mp) * unionfs_lock()) and the mountpoint's busy count. Without this, * unmount will lock the covered vnode lock (directly through the * covered vnode) and wait for the busy count to drain, while a - * concurrent lookup will increment the busy count and then lock + * concurrent lookup will increment the busy count and then may lock * the covered vnode lock (indirectly through unionfs_lock()). * - * Note that we can't yet use this facility for the 'below' case - * in which the upper vnode is the covered vnode, because that would - * introduce a different LOR in which the cross-mount lookup would - * effectively hold the upper vnode lock before acquiring the lower - * vnode lock, while an unrelated lock operation would still acquire - * the lower vnode lock before the upper vnode lock, which is the - * order unionfs currently requires. + * Note that this is only needed for the 'below' case in which the + * upper vnode is also the covered vnode, because unionfs_lock() + * only locks the upper vnode as long as both lower and upper vnodes + * are present (which they will always be for the unionfs mount root). */ - if (!below) { + if (below) { vn_lock(mp->mnt_vnodecovered, LK_EXCLUSIVE | LK_RETRY | LK_CANRECURSE); mp->mnt_vnodecovered->v_vflag |= VV_CROSSLOCK; VOP_UNLOCK(mp->mnt_vnodecovered); diff --git a/sys/fs/unionfs/union_vnops.c b/sys/fs/unionfs/union_vnops.c index aa2a7273825a..5259b0270c69 100644 --- a/sys/fs/unionfs/union_vnops.c +++ b/sys/fs/unionfs/union_vnops.c @@ -85,12 +85,11 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) struct componentname *cnp; struct thread *td; u_long nameiop; - u_long cnflags, cnflagsbk; - int iswhiteout; + u_long cnflags; int lockflag; - int error , uerror, lerror; + int lkflags; + int error, uerror, lerror; - iswhiteout = 0; lockflag = 0; error = uerror = lerror = ENOENT; cnp = ap->a_cnp; @@ -119,30 +118,49 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) LOOKUP != nameiop) return (EROFS); + /* + * Note that a lookup is in-flight, and block if another lookup + * is already in-flight against dvp. This is done because we may + * end up dropping dvp's lock to look up a lower vnode or to create + * a shadow directory, opening up the possibility of parallel lookups + * against the same directory creating duplicate unionfs vnodes for + * the same file(s). Note that if this function encounters an + * in-progress lookup for the directory, it will block until the + * lookup is complete and then return ERELOOKUP to allow any + * existing unionfs vnode to be loaded from the VFS cache. + * This is really a hack; filesystems that support MNTK_LOOKUP_SHARED + * (which unionfs currently doesn't) seem to deal with this by using + * the vfs_hash_* functions to manage a per-mount vnode cache keyed + * by the inode number (or some roughly equivalent unique ID + * usually assocated with the storage medium). Ideally unionfs + * should adopt something similar as a replacement for its current + * half-baked directory-only cache implementation, particularly if + * we ever want to support MNTK_LOOKUP_SHARED here. + */ + error = unionfs_set_in_progress_flag(dvp, UNIONFS_LOOKUP_IN_PROGRESS); + if (error != 0) + return (error); /* * lookup dotdot */ if (cnflags & ISDOTDOT) { - if (LOOKUP != nameiop && udvp == NULLVP) - return (EROFS); + if (LOOKUP != nameiop && udvp == NULLVP) { + error = EROFS; + goto unionfs_lookup_return; + } - if (udvp != NULLVP) { + if (udvp != NULLVP) dtmpvp = udvp; - if (ldvp != NULLVP) - VOP_UNLOCK(ldvp); - } else dtmpvp = ldvp; + unionfs_forward_vop_start(dtmpvp, &lkflags); error = VOP_LOOKUP(dtmpvp, &vp, cnp); + unionfs_forward_vop_finish(dvp, dtmpvp, lkflags); - if (dtmpvp == udvp && ldvp != NULLVP) { - VOP_UNLOCK(udvp); - vn_lock(dvp, LK_EXCLUSIVE | LK_RETRY); - dunp = VTOUNIONFS(dvp); - if (error == 0 && dunp == NULL) - error = ENOENT; - } + dunp = VTOUNIONFS(dvp); + if (error == 0 && dunp == NULL) + error = ENOENT; if (error == 0) { /* @@ -173,14 +191,81 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) goto unionfs_lookup_return; } + /* + * Lookup lower layer. We do this before looking up the the upper + * layer, as we may drop the upper parent directory's lock, and we + * want to ensure the upper parent remains locked from the point of + * lookup through any ensuing VOP that may require it to be locked. + * The cost of this is that we may end up performing an unnecessary + * lower layer lookup if a whiteout is present in the upper layer. + */ + if (ldvp != NULLVP && !(cnflags & DOWHITEOUT)) { + struct componentname lcn; + bool is_dot; + + if (udvp != NULLVP) { + vref(ldvp); + VOP_UNLOCK(dvp); + vn_lock(ldvp, LK_EXCLUSIVE | LK_RETRY); + } + + lcn = *cnp; + /* always op is LOOKUP */ + lcn.cn_nameiop = LOOKUP; + lcn.cn_flags = cnflags; + is_dot = false; + + if (udvp == NULLVP) + unionfs_forward_vop_start(ldvp, &lkflags); + lerror = VOP_LOOKUP(ldvp, &lvp, &lcn); + if (udvp == NULLVP && + unionfs_forward_vop_finish(dvp, ldvp, lkflags)) { + error = ENOENT; + goto unionfs_lookup_cleanup; + } + + if (udvp == NULLVP) + cnp->cn_flags = lcn.cn_flags; + + if (lerror == 0) { + if (ldvp == lvp) { /* is dot */ + vrele(lvp); + *(ap->a_vpp) = dvp; + vref(dvp); + is_dot = true; + error = lerror; + } else if (cnp->cn_lkflags & LK_TYPE_MASK) + VOP_UNLOCK(lvp); + } + + if (udvp != NULLVP) { + vput(ldvp); + vn_lock(dvp, LK_EXCLUSIVE | LK_RETRY); + if (VN_IS_DOOMED(dvp)) + error = ENOENT; + } + if (is_dot) + goto unionfs_lookup_return; + else if (error != 0) + goto unionfs_lookup_cleanup; + } /* * lookup upper layer */ if (udvp != NULLVP) { + bool iswhiteout = false; + + unionfs_forward_vop_start(udvp, &lkflags); uerror = VOP_LOOKUP(udvp, &uvp, cnp); + if (unionfs_forward_vop_finish(dvp, udvp, lkflags)) { + error = ENOENT; + goto unionfs_lookup_return; + } if (uerror == 0) { if (udvp == uvp) { /* is dot */ + if (lvp != NULLVP) + vrele(lvp); vrele(uvp); *(ap->a_vpp) = dvp; vref(dvp); @@ -194,13 +279,18 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) } /* check whiteout */ - if (uerror == ENOENT || uerror == EJUSTRETURN) - if (cnp->cn_flags & ISWHITEOUT) - iswhiteout = 1; /* don't lookup lower */ - if (iswhiteout == 0 && ldvp != NULLVP) - if (!VOP_GETATTR(udvp, &va, cnp->cn_cred) && - (va.va_flags & OPAQUE)) - iswhiteout = 1; /* don't lookup lower */ + if ((uerror == ENOENT || uerror == EJUSTRETURN) && + (cnp->cn_flags & ISWHITEOUT)) + iswhiteout = true; + else if (VOP_GETATTR(udvp, &va, cnp->cn_cred) == 0 && + (va.va_flags & OPAQUE)) + iswhiteout = true; + + if (iswhiteout && lvp != NULLVP) { + vrele(lvp); + lvp = NULLVP; + } + #if 0 UNIONFS_INTERNAL_DEBUG( "unionfs_lookup: debug: whiteout=%d, path=%s\n", @@ -208,39 +298,6 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) #endif } - /* - * lookup lower layer - */ - if (ldvp != NULLVP && !(cnflags & DOWHITEOUT) && iswhiteout == 0) { - /* always op is LOOKUP */ - cnp->cn_nameiop = LOOKUP; - cnflagsbk = cnp->cn_flags; - cnp->cn_flags = cnflags; - - lerror = VOP_LOOKUP(ldvp, &lvp, cnp); - - cnp->cn_nameiop = nameiop; - if (udvp != NULLVP && (uerror == 0 || uerror == EJUSTRETURN)) - cnp->cn_flags = cnflagsbk; - - if (lerror == 0) { - if (ldvp == lvp) { /* is dot */ - if (uvp != NULLVP) - vrele(uvp); /* no need? */ - vrele(lvp); - *(ap->a_vpp) = dvp; - vref(dvp); - - UNIONFS_INTERNAL_DEBUG( - "unionfs_lookup: leave (%d)\n", lerror); - - return (lerror); - } - if (cnp->cn_lkflags & LK_TYPE_MASK) - VOP_UNLOCK(lvp); - } - } - /* * check lookup result */ @@ -280,8 +337,7 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) if (unp == NULL) error = ENOENT; else - error = unionfs_mkshadowdir(MOUNTTOUNIONFSMOUNT(dvp->v_mount), - udvp, unp, cnp, td); + error = unionfs_mkshadowdir(dvp, vp, cnp, td); if (lockflag != 0) VOP_UNLOCK(vp); if (error != 0) { @@ -293,6 +349,10 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) vrele(vp); goto unionfs_lookup_cleanup; } + /* + * TODO: Since unionfs_mkshadowdir() relocks udvp after + * creating the new directory, return ERELOOKUP here? + */ if ((cnp->cn_lkflags & LK_TYPE_MASK) == LK_SHARED) vn_lock(vp, LK_SHARED | LK_RETRY); } @@ -318,6 +378,12 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); } + if (VN_IS_DOOMED(dvp) || VN_IS_DOOMED(vp)) { + error = ENOENT; + vput(vp); + goto unionfs_lookup_cleanup; + } + *(ap->a_vpp) = vp; if (cnflags & MAKEENTRY) @@ -329,10 +395,12 @@ unionfs_lookup(struct vop_cachedlookup_args *ap) if (lvp != NULLVP) vrele(lvp); - if (error == ENOENT && (cnflags & MAKEENTRY) != 0) + if (error == ENOENT && (cnflags & MAKEENTRY) != 0 && + !VN_IS_DOOMED(dvp)) cache_enter(dvp, NULLVP, cnp); unionfs_lookup_return: + unionfs_clear_in_progress_flag(dvp, UNIONFS_LOOKUP_IN_PROGRESS); UNIONFS_INTERNAL_DEBUG("unionfs_lookup: leave (%d)\n", error); @@ -492,6 +560,61 @@ unionfs_downgrade_lock(struct vnode *vp, enum unionfs_lkupgrade status) vn_lock(vp, LK_DOWNGRADE | LK_RETRY); } +/* + * Exchange the default (upper vnode) lock on a unionfs vnode for the lower + * vnode lock, in support of operations that require access to the lower vnode + * even when an upper vnode is present. We don't use vn_lock_pair() to hold + * both vnodes at the same time, primarily because the caller may proceed + * to issue VOPs to the lower layer which re-lock or perform other operations + * which may not be safe in the presence of a locked vnode from another FS. + * Moreover, vn_lock_pair()'s deadlock resolution approach can introduce + * additional overhead that isn't necessary on these paths. + * + * vp must be a locked unionfs vnode; the lock state of this vnode is + * returned through *lkflags for later use in unionfs_unlock_lvp(). + * + * Returns the locked lower vnode, or NULL if the lower vnode (and therefore + * also the unionfs vnode above it) has been doomed. + */ +static struct vnode * +unionfs_lock_lvp(struct vnode *vp, int *lkflags) +{ + struct unionfs_node *unp; + struct vnode *lvp; + + unp = VTOUNIONFS(vp); + lvp = unp->un_lowervp; + ASSERT_VOP_LOCKED(vp, __func__); + ASSERT_VOP_UNLOCKED(lvp, __func__); + *lkflags = VOP_ISLOCKED(vp); + vref(lvp); + VOP_UNLOCK(vp); + vn_lock(lvp, *lkflags | LK_RETRY); + if (VN_IS_DOOMED(lvp)) { + vput(lvp); + lvp = NULLVP; + vn_lock(vp, *lkflags | LK_RETRY); + } + return (lvp); +} + +/* + * Undo a previous call to unionfs_lock_lvp(), restoring the default lock + * on the unionfs vnode. This function reloads and returns the vnode + * private data for the unionfs vnode, which will be NULL if the unionfs + * vnode became doomed while its lock was dropped. The caller must check + * for this case. + */ +static struct unionfs_node * +unionfs_unlock_lvp(struct vnode *vp, struct vnode *lvp, int lkflags) +{ + ASSERT_VOP_LOCKED(lvp, __func__); + ASSERT_VOP_UNLOCKED(vp, __func__); + vput(lvp); + vn_lock(vp, lkflags | LK_RETRY); + return (VTOUNIONFS(vp)); +} + static int unionfs_open(struct vop_open_args *ap) { @@ -504,7 +627,9 @@ unionfs_open(struct vop_open_args *ap) struct ucred *cred; struct thread *td; int error; + int lkflags; enum unionfs_lkupgrade lkstatus; + bool lock_lvp, open_lvp; UNIONFS_INTERNAL_DEBUG("unionfs_open: enter\n"); @@ -515,6 +640,7 @@ unionfs_open(struct vop_open_args *ap) targetvp = NULLVP; cred = ap->a_cred; td = ap->a_td; + open_lvp = lock_lvp = false; /* * The executable loader path may call this function with vp locked @@ -546,7 +672,7 @@ unionfs_open(struct vop_open_args *ap) if (targetvp == NULLVP) { if (uvp == NULLVP) { if ((ap->a_mode & FWRITE) && lvp->v_type == VREG) { - error = unionfs_copyfile(unp, + error = unionfs_copyfile(vp, !(ap->a_mode & O_TRUNC), cred, td); if (error != 0) goto unionfs_open_abort; @@ -557,30 +683,69 @@ unionfs_open(struct vop_open_args *ap) targetvp = uvp; } + if (targetvp == uvp && uvp->v_type == VDIR && lvp != NULLVP && + unsp->uns_lower_opencnt <= 0) + open_lvp = true; + else if (targetvp == lvp && uvp != NULLVP) + lock_lvp = true; + + if (lock_lvp) { + unp = NULL; + lvp = unionfs_lock_lvp(vp, &lkflags); + if (lvp == NULLVP) { + error = ENOENT; + goto unionfs_open_abort; + } + } else + unionfs_forward_vop_start(targetvp, &lkflags); + error = VOP_OPEN(targetvp, ap->a_mode, cred, td, ap->a_fp); - if (error == 0) { - if (targetvp == uvp) { - if (uvp->v_type == VDIR && lvp != NULLVP && - unsp->uns_lower_opencnt <= 0) { - /* open lower for readdir */ - error = VOP_OPEN(lvp, FREAD, cred, td, NULL); - if (error != 0) { - VOP_CLOSE(uvp, ap->a_mode, cred, td); - goto unionfs_open_abort; - } - unsp->uns_node_flag |= UNS_OPENL_4_READDIR; - unsp->uns_lower_opencnt++; + + if (lock_lvp) { + unp = unionfs_unlock_lvp(vp, lvp, lkflags); + if (unp == NULL && error == 0) + error = ENOENT; + } else if (unionfs_forward_vop_finish(vp, targetvp, lkflags)) + error = error ? error : ENOENT; + + if (error != 0) + goto unionfs_open_abort; + + if (targetvp == uvp) { + if (open_lvp) { + unp = NULL; + lvp = unionfs_lock_lvp(vp, &lkflags); + if (lvp == NULLVP) { + error = ENOENT; + goto unionfs_open_abort; } - unsp->uns_upper_opencnt++; - } else { + /* open lower for readdir */ + error = VOP_OPEN(lvp, FREAD, cred, td, NULL); + unp = unionfs_unlock_lvp(vp, lvp, lkflags); + if (unp == NULL) { + error = error ? error : ENOENT; + goto unionfs_open_abort; + } + if (error != 0) { + unionfs_forward_vop_start(uvp, &lkflags); + VOP_CLOSE(uvp, ap->a_mode, cred, td); + if (unionfs_forward_vop_finish(vp, uvp, lkflags)) + unp = NULL; + goto unionfs_open_abort; + } + unsp->uns_node_flag |= UNS_OPENL_4_READDIR; unsp->uns_lower_opencnt++; - unsp->uns_lower_openmode = ap->a_mode; } - vp->v_object = targetvp->v_object; + unsp->uns_upper_opencnt++; + } else { + unsp->uns_lower_opencnt++; + unsp->uns_lower_openmode = ap->a_mode; } + vp->v_object = targetvp->v_object; unionfs_open_abort: - if (error != 0) + + if (error != 0 && unp != NULL) unionfs_tryrem_node_status(unp, unsp); unionfs_open_cleanup: @@ -599,9 +764,13 @@ unionfs_close(struct vop_close_args *ap) struct ucred *cred; struct thread *td; struct vnode *vp; + struct vnode *uvp; + struct vnode *lvp; struct vnode *ovp; int error; + int lkflags; enum unionfs_lkupgrade lkstatus; + bool lock_lvp; UNIONFS_INTERNAL_DEBUG("unionfs_close: enter\n"); @@ -611,6 +780,7 @@ unionfs_close(struct vop_close_args *ap) cred = ap->a_cred; td = ap->a_td; error = 0; + lock_lvp = false; /* * If the vnode is reclaimed while upgrading, we can't safely use unp @@ -621,44 +791,73 @@ unionfs_close(struct vop_close_args *ap) goto unionfs_close_cleanup; unp = VTOUNIONFS(vp); + lvp = unp->un_lowervp; + uvp = unp->un_uppervp; unionfs_get_node_status(unp, td, &unsp); if (unsp->uns_lower_opencnt <= 0 && unsp->uns_upper_opencnt <= 0) { -#ifdef DIAGNOSTIC - printf("unionfs_close: warning: open count is 0\n"); -#endif - if (unp->un_uppervp != NULLVP) - ovp = unp->un_uppervp; + if (uvp != NULLVP) + ovp = uvp; else - ovp = unp->un_lowervp; + ovp = lvp; } else if (unsp->uns_upper_opencnt > 0) - ovp = unp->un_uppervp; + ovp = uvp; else - ovp = unp->un_lowervp; + ovp = lvp; + + if (ovp == lvp && uvp != NULLVP) { + lock_lvp = true; + unp = NULL; + lvp = unionfs_lock_lvp(vp, &lkflags); + if (lvp == NULLVP) { + error = ENOENT; + goto unionfs_close_abort; + } + } else + unionfs_forward_vop_start(ovp, &lkflags); error = VOP_CLOSE(ovp, ap->a_fflag, cred, td); + if (lock_lvp) { + unp = unionfs_unlock_lvp(vp, lvp, lkflags); + if (unp == NULL && error == 0) + error = ENOENT; + } else if (unionfs_forward_vop_finish(vp, ovp, lkflags)) + error = error ? error : ENOENT; + if (error != 0) goto unionfs_close_abort; vp->v_object = ovp->v_object; - if (ovp == unp->un_uppervp) { + if (ovp == uvp) { unsp->uns_upper_opencnt--; if (unsp->uns_upper_opencnt == 0) { if (unsp->uns_node_flag & UNS_OPENL_4_READDIR) { - VOP_CLOSE(unp->un_lowervp, FREAD, cred, td); + unp = NULL; + lvp = unionfs_lock_lvp(vp, &lkflags); + if (lvp == NULLVP) { + error = ENOENT; + goto unionfs_close_abort; + } + VOP_CLOSE(lvp, FREAD, cred, td); + unp = unionfs_unlock_lvp(vp, lvp, lkflags); + if (unp == NULL) { + error = ENOENT; + goto unionfs_close_abort; + } unsp->uns_node_flag &= ~UNS_OPENL_4_READDIR; unsp->uns_lower_opencnt--; } if (unsp->uns_lower_opencnt > 0) - vp->v_object = unp->un_lowervp->v_object; + vp->v_object = lvp->v_object; } } else unsp->uns_lower_opencnt--; unionfs_close_abort: - unionfs_tryrem_node_status(unp, unsp); + if (unp != NULL) + unionfs_tryrem_node_status(unp, unsp); unionfs_close_cleanup: unionfs_downgrade_lock(vp, lkstatus); @@ -883,7 +1082,7 @@ unionfs_setattr(struct vop_setattr_args *ap) return (EROFS); if (uvp == NULLVP && lvp->v_type == VREG) { - error = unionfs_copyfile(unp, (vap->va_size != 0), + error = unionfs_copyfile(ap->a_vp, (vap->va_size != 0), ap->a_cred, td); if (error != 0) return (error); @@ -1078,8 +1277,10 @@ unionfs_remove(struct vop_remove_args *ap) error = VOP_REMOVE(udvp, uvp, cnp); unionfs_forward_vop_finish_pair(ap->a_dvp, udvp, udvp_lkflags, ap->a_vp, uvp, uvp_lkflags); - } else if (lvp != NULLVP) - error = unionfs_mkwhiteout(ap->a_dvp, udvp, cnp, td, path, pathlen); + } else if (lvp != NULLVP) { + error = unionfs_mkwhiteout(ap->a_dvp, ap->a_vp, cnp, td, + path, pathlen); + } UNIONFS_INTERNAL_DEBUG("unionfs_remove: leave (%d)\n", error); @@ -1096,7 +1297,6 @@ unionfs_link(struct vop_link_args *ap) struct componentname *cnp; struct thread *td; int error; - int needrelookup; UNIONFS_INTERNAL_DEBUG("unionfs_link: enter\n"); @@ -1104,7 +1304,6 @@ unionfs_link(struct vop_link_args *ap) KASSERT_UNIONFS_VNODE(ap->a_vp); error = 0; - needrelookup = 0; dunp = VTOUNIONFS(ap->a_tdvp); unp = NULL; udvp = dunp->un_uppervp; @@ -1121,16 +1320,15 @@ unionfs_link(struct vop_link_args *ap) if (ap->a_vp->v_type != VREG) return (EOPNOTSUPP); - error = unionfs_copyfile(unp, 1, cnp->cn_cred, td); - if (error != 0) - return (error); - needrelookup = 1; + VOP_UNLOCK(ap->a_tdvp); + error = unionfs_copyfile(ap->a_vp, 1, cnp->cn_cred, td); + vn_lock(ap->a_tdvp, LK_EXCLUSIVE | LK_RETRY); + if (error == 0) + error = ERELOOKUP; + return (error); } uvp = unp->un_uppervp; - if (needrelookup != 0) - error = unionfs_relookup_for_create(ap->a_tdvp, cnp, td); - if (error == 0) { int udvp_lkflags, uvp_lkflags; unionfs_forward_vop_start_pair(udvp, &udvp_lkflags, @@ -1154,8 +1352,6 @@ unionfs_rename(struct vop_rename_args *ap) struct vnode *tdvp; struct vnode *tvp; struct componentname *tcnp; - struct vnode *ltdvp; - struct vnode *ltvp; struct thread *td; /* rename target vnodes */ @@ -1164,7 +1360,6 @@ unionfs_rename(struct vop_rename_args *ap) struct vnode *rtdvp; struct vnode *rtvp; - struct unionfs_mount *ump; struct unionfs_node *unp; int error; @@ -1177,8 +1372,6 @@ unionfs_rename(struct vop_rename_args *ap) tdvp = ap->a_tdvp; tvp = ap->a_tvp; tcnp = ap->a_tcnp; - ltdvp = NULLVP; - ltvp = NULLVP; td = curthread; rfdvp = fdvp; rfvp = fvp; @@ -1238,7 +1431,6 @@ unionfs_rename(struct vop_rename_args *ap) UNIONFS_INTERNAL_DEBUG("fvp=%p, ufvp=%p, lfvp=%p\n", fvp, unp->un_uppervp, unp->un_lowervp); #endif - ump = MOUNTTOUNIONFSMOUNT(fvp->v_mount); /* * If we only have a lower vnode, copy the source file to the upper * FS so that the rename operation can be issued against the upper FS. @@ -1282,10 +1474,10 @@ unionfs_rename(struct vop_rename_args *ap) else if (unp->un_uppervp == NULLVP) { switch (fvp->v_type) { case VREG: - error = unionfs_copyfile(unp, 1, fcnp->cn_cred, td); + error = unionfs_copyfile(fvp, 1, fcnp->cn_cred, td); break; case VDIR: - error = unionfs_mkshadowdir(ump, rfdvp, unp, fcnp, td); + error = unionfs_mkshadowdir(fdvp, fvp, fcnp, td); break; default: error = ENODEV; @@ -1327,7 +1519,6 @@ unionfs_rename(struct vop_rename_args *ap) goto unionfs_rename_abort; } rtdvp = unp->un_uppervp; - ltdvp = unp->un_lowervp; vref(rtdvp); if (tvp != NULLVP) { @@ -1348,7 +1539,6 @@ unionfs_rename(struct vop_rename_args *ap) goto unionfs_rename_abort; } rtvp = unp->un_uppervp; - ltvp = unp->un_lowervp; vref(rtvp); } } @@ -1365,12 +1555,8 @@ unionfs_rename(struct vop_rename_args *ap) cache_purge(fdvp); } - if (ltdvp != NULLVP) - VOP_UNLOCK(ltdvp); if (tdvp != rtdvp) vrele(tdvp); - if (ltvp != NULLVP) - VOP_UNLOCK(ltvp); if (tvp != rtvp && tvp != NULLVP) { if (rtvp == NULLVP) vput(tvp); @@ -1504,43 +1690,55 @@ unionfs_rmdir(struct vop_rmdir_args *ap) if (uvp != NULLVP) { if (lvp != NULLVP) { + /* + * We need to keep dvp and vp's upper vnodes locked + * going into the VOP_RMDIR() call, but the empty + * directory check also requires the lower vnode lock. + * For this third, cross-filesystem lock we use a + * similar approach taken by various FS' VOP_RENAME + * implementations (which require 2-4 vnode locks). + * First we attempt a NOWAIT acquisition, then if + * that fails we drops the other two vnode locks, + * acquire lvp's lock in the normal fashion to reduce + * the likelihood of spinning on it in the future, + * then drop, reacquire the other locks, and return + * ERELOOKUP to re-drive the lookup in case the dvp-> + * vp relationship has changed. + */ + if (vn_lock(lvp, LK_SHARED | LK_NOWAIT) != 0) { + VOP_UNLOCK(ap->a_vp); + VOP_UNLOCK(ap->a_dvp); + vn_lock(lvp, LK_SHARED | LK_RETRY); + VOP_UNLOCK(lvp); + vn_lock(ap->a_dvp, LK_EXCLUSIVE | LK_RETRY); + vn_lock(ap->a_vp, LK_EXCLUSIVE | LK_RETRY); + return (ERELOOKUP); + } error = unionfs_check_rmdir(ap->a_vp, cnp->cn_cred, td); + /* + * It's possible for a direct operation on the lower FS + * to make the lower directory non-empty after we drop + * the lock, but it's also possible for the upper-layer + * VOP_RENAME to relock udvp/uvp which would lead to + * LOR if we kept lvp locked across that call. + */ + VOP_UNLOCK(lvp); if (error != 0) return (error); } ump = MOUNTTOUNIONFSMOUNT(ap->a_vp->v_mount); if (ump->um_whitemode == UNIONFS_WHITE_ALWAYS || lvp != NULLVP) cnp->cn_flags |= DOWHITEOUT; - /* - * The relookup path will need to relock the parent dvp and - * possibly the vp as well. Locking is expected to be done - * in parent->child order; drop the lock on vp to avoid LOR - * and potential recursion on vp's lock. - * vp is expected to remain referenced during VOP_RMDIR(), - * so vref/vrele should not be necessary here. - */ - VOP_UNLOCK(ap->a_vp); - VNPASS(vrefcnt(ap->a_vp) > 0, ap->a_vp); - error = unionfs_relookup_for_delete(ap->a_dvp, cnp, td); - vn_lock(ap->a_vp, LK_EXCLUSIVE | LK_RETRY); - /* - * VOP_RMDIR is dispatched against udvp, so if uvp became - * doomed while the lock was dropped above the target - * filesystem may not be able to cope. - */ - if (error == 0 && VN_IS_DOOMED(uvp)) - error = ENOENT; - if (error == 0) { - int udvp_lkflags, uvp_lkflags; - unionfs_forward_vop_start_pair(udvp, &udvp_lkflags, - uvp, &uvp_lkflags); - error = VOP_RMDIR(udvp, uvp, cnp); - unionfs_forward_vop_finish_pair(ap->a_dvp, udvp, udvp_lkflags, - ap->a_vp, uvp, uvp_lkflags); - } - } else if (lvp != NULLVP) - error = unionfs_mkwhiteout(ap->a_dvp, udvp, cnp, td, + int udvp_lkflags, uvp_lkflags; + unionfs_forward_vop_start_pair(udvp, &udvp_lkflags, + uvp, &uvp_lkflags); + error = VOP_RMDIR(udvp, uvp, cnp); + unionfs_forward_vop_finish_pair(ap->a_dvp, udvp, udvp_lkflags, + ap->a_vp, uvp, uvp_lkflags); + } else if (lvp != NULLVP) { + error = unionfs_mkwhiteout(ap->a_dvp, ap->a_vp, cnp, td, unp->un_path, unp->un_pathlen); + } if (error == 0) { cache_purge(ap->a_dvp); @@ -1613,6 +1811,7 @@ unionfs_readdir(struct vop_readdir_args *ap) uint64_t *cookies_bk; int error; int eofflag; + int lkflags; int ncookies_bk; int uio_offset_bk; enum unionfs_lkupgrade lkstatus; @@ -1668,18 +1867,26 @@ unionfs_readdir(struct vop_readdir_args *ap) /* upper only */ if (uvp != NULLVP && lvp == NULLVP) { + unionfs_forward_vop_start(uvp, &lkflags); error = VOP_READDIR(uvp, uio, ap->a_cred, ap->a_eofflag, ap->a_ncookies, ap->a_cookies); - unsp->uns_readdir_status = 0; + if (unionfs_forward_vop_finish(vp, uvp, lkflags)) + error = error ? error : ENOENT; + else + unsp->uns_readdir_status = 0; goto unionfs_readdir_exit; } /* lower only */ if (uvp == NULLVP && lvp != NULLVP) { + unionfs_forward_vop_start(lvp, &lkflags); error = VOP_READDIR(lvp, uio, ap->a_cred, ap->a_eofflag, ap->a_ncookies, ap->a_cookies); - unsp->uns_readdir_status = 2; + if (unionfs_forward_vop_finish(vp, lvp, lkflags)) + error = error ? error : ENOENT; + else + unsp->uns_readdir_status = 2; goto unionfs_readdir_exit; } @@ -1689,14 +1896,17 @@ unionfs_readdir(struct vop_readdir_args *ap) */ KASSERT(uvp != NULLVP, ("unionfs_readdir: null upper vp")); KASSERT(lvp != NULLVP, ("unionfs_readdir: null lower vp")); + if (uio->uio_offset == 0) unsp->uns_readdir_status = 0; if (unsp->uns_readdir_status == 0) { /* read upper */ + unionfs_forward_vop_start(uvp, &lkflags); error = VOP_READDIR(uvp, uio, ap->a_cred, &eofflag, ap->a_ncookies, ap->a_cookies); - + if (unionfs_forward_vop_finish(vp, uvp, lkflags) && error == 0) + error = ENOENT; if (error != 0 || eofflag == 0) goto unionfs_readdir_exit; unsp->uns_readdir_status = 1; @@ -1735,14 +1945,22 @@ unionfs_readdir(struct vop_readdir_args *ap) uio->uio_offset = 0; } - if (lvp == NULLVP) { - error = EBADF; + lvp = unionfs_lock_lvp(vp, &lkflags); + if (lvp == NULL) { + error = ENOENT; goto unionfs_readdir_exit; } + /* read lower */ error = VOP_READDIR(lvp, uio, ap->a_cred, ap->a_eofflag, ap->a_ncookies, ap->a_cookies); + + unp = unionfs_unlock_lvp(vp, lvp, lkflags); + if (unp == NULL && error == 0) + error = ENOENT; + + /* * We can't return an uio_offset of 0: this would trigger an * infinite loop, because the next call to unionfs_readdir would @@ -1906,97 +2124,51 @@ unionfs_print(struct vop_print_args *ap) return (0); } -static int -unionfs_get_llt_revlock(struct vnode *vp, int flags) -{ - int revlock; - - revlock = 0; - - switch (flags & LK_TYPE_MASK) { - case LK_SHARED: - if (VOP_ISLOCKED(vp) == LK_EXCLUSIVE) - revlock = LK_UPGRADE; - else - revlock = LK_RELEASE; - break; - case LK_EXCLUSIVE: - case LK_UPGRADE: - revlock = LK_RELEASE; - break; - case LK_DOWNGRADE: - revlock = LK_UPGRADE; - break; - default: - break; - } - - return (revlock); -} - -/* - * The state of an acquired lock is adjusted similarly to - * the time of error generating. - * flags: LK_RELEASE or LK_UPGRADE - */ -static void -unionfs_revlock(struct vnode *vp, int flags) -{ - if (flags & LK_RELEASE) - VOP_UNLOCK_FLAGS(vp, flags); - else { - /* UPGRADE */ - if (vn_lock(vp, flags) != 0) - vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - } -} - static int unionfs_lock(struct vop_lock1_args *ap) { struct unionfs_node *unp; struct vnode *vp; - struct vnode *uvp; - struct vnode *lvp; + struct vnode *tvp; int error; int flags; - int revlock; - int interlock; - int uhold; + bool lvp_locked; - /* - * TODO: rework the unionfs locking scheme. - * It's not guaranteed to be safe to blindly lock two vnodes on - * different mounts as is done here. Further, the entanglement - * of locking both vnodes with the various options that can be - * passed to VOP_LOCK() makes this code hard to reason about. - * Instead, consider locking only the upper vnode, or the lower - * vnode is the upper is not present, and taking separate measures - * to lock both vnodes in the few cases when that is needed. - */ error = 0; - interlock = 1; - uhold = 0; flags = ap->a_flags; vp = ap->a_vp; if (LK_RELEASE == (flags & LK_TYPE_MASK) || !(flags & LK_TYPE_MASK)) return (VOP_UNLOCK_FLAGS(vp, flags | LK_RELEASE)); - if ((flags & LK_INTERLOCK) == 0) +unionfs_lock_restart: + /* + * We currently need the interlock here to ensure we can safely + * access the unionfs vnode's private data. We may be able to + * eliminate this extra locking by instead using vfs_smr_enter() + * and vn_load_v_data_smr() here in conjunction with an SMR UMA + * zone for unionfs nodes. + */ + if ((ap->a_flags & LK_INTERLOCK) == 0) VI_LOCK(vp); + else + ap->a_flags &= ~LK_INTERLOCK; unp = VTOUNIONFS(vp); - if (unp == NULL) - goto unionfs_lock_null_vnode; + if (unp == NULL) { + VI_UNLOCK(vp); + return (vop_stdlock(ap)); + } KASSERT_UNIONFS_VNODE(ap->a_vp); - lvp = unp->un_lowervp; - uvp = unp->un_uppervp; - - if ((revlock = unionfs_get_llt_revlock(vp, flags)) == 0) - panic("unknown lock type: 0x%x", flags & LK_TYPE_MASK); + if (unp->un_uppervp != NULL) { + tvp = unp->un_uppervp; + lvp_locked = false; + } else { + tvp = unp->un_lowervp; + lvp_locked = true; + } /* * During unmount, the root vnode lock may be taken recursively, @@ -2009,150 +2181,59 @@ unionfs_lock(struct vop_lock1_args *ap) (vp->v_vflag & VV_ROOT) != 0) flags |= LK_CANRECURSE; - if (lvp != NULLVP) { - if (uvp != NULLVP && flags & LK_UPGRADE) { - /* - * Share Lock is once released and a deadlock is - * avoided. - */ - vholdnz(uvp); - uhold = 1; - VOP_UNLOCK(uvp); - } - VI_LOCK_FLAGS(lvp, MTX_DUPOK); - flags |= LK_INTERLOCK; - vholdl(lvp); - - VI_UNLOCK(vp); - ap->a_flags &= ~LK_INTERLOCK; - - error = VOP_LOCK(lvp, flags); - - VI_LOCK(vp); - unp = VTOUNIONFS(vp); - if (unp == NULL) { - /* vnode is released. */ - VI_UNLOCK(vp); - if (error == 0) - VOP_UNLOCK(lvp); - vdrop(lvp); - if (uhold != 0) - vdrop(uvp); - goto unionfs_lock_fallback; - } - } - - if (error == 0 && uvp != NULLVP) { - if (uhold && flags & LK_UPGRADE) { - flags &= ~LK_TYPE_MASK; - flags |= LK_EXCLUSIVE; - } - VI_LOCK_FLAGS(uvp, MTX_DUPOK); - flags |= LK_INTERLOCK; - if (uhold == 0) { - vholdl(uvp); - uhold = 1; - } - - VI_UNLOCK(vp); - ap->a_flags &= ~LK_INTERLOCK; - - error = VOP_LOCK(uvp, flags); - - VI_LOCK(vp); + vholdnz(tvp); + VI_UNLOCK(vp); + error = VOP_LOCK(tvp, ap->a_flags); + vdrop(tvp); + if (error == 0 && (lvp_locked || VTOUNIONFS(vp) == NULL)) { + /* + * If unp is not already NULL, we assume that we can safely + * access it because we currently hold lvp's lock. + * unionfs_noderem() acquires lvp's lock before freeing + * the vnode private data, ensuring it can't be concurrently + * freed while we are using it here. Without that guarantee, + * we would need to reacquire the vnode interlock here. + */ unp = VTOUNIONFS(vp); - if (unp == NULL) { - /* vnode is released. */ - VI_UNLOCK(vp); - if (error == 0) - VOP_UNLOCK(uvp); - vdrop(uvp); - if (lvp != NULLVP) { - VOP_UNLOCK(lvp); - vdrop(lvp); + if (unp == NULL || unp->un_uppervp != NULLVP) { + VOP_UNLOCK(tvp); + if (flags & LK_UPGRADE) { + ap->a_flags &= ~LK_TYPE_MASK; + ap->a_flags |= LK_EXCLUSIVE; + } else if (flags & LK_DOWNGRADE) { + ap->a_flags &= ~LK_TYPE_MASK; + ap->a_flags |= LK_SHARED; } - goto unionfs_lock_fallback; - } - if (error != 0 && lvp != NULLVP) { - /* rollback */ - VI_UNLOCK(vp); - unionfs_revlock(lvp, revlock); - interlock = 0; + goto unionfs_lock_restart; } } - if (interlock) - VI_UNLOCK(vp); - if (lvp != NULLVP) - vdrop(lvp); - if (uhold != 0) - vdrop(uvp); - return (error); - -unionfs_lock_null_vnode: - ap->a_flags |= LK_INTERLOCK; - return (vop_stdlock(ap)); - -unionfs_lock_fallback: - /* - * If we reach this point, we've discovered the unionfs vnode - * has been reclaimed while the upper/lower vnode locks were - * temporarily dropped. Such temporary droppage may happen - * during the course of an LK_UPGRADE operation itself, and in - * that case LK_UPGRADE must be cleared as the unionfs vnode's - * lock has been reset to point to the standard v_lock field, - * which has not previously been held. - */ - if (flags & LK_UPGRADE) { - ap->a_flags &= ~LK_TYPE_MASK; - ap->a_flags |= LK_EXCLUSIVE; - } - return (vop_stdlock(ap)); } static int unionfs_unlock(struct vop_unlock_args *ap) { struct vnode *vp; - struct vnode *lvp; - struct vnode *uvp; + struct vnode *tvp; struct unionfs_node *unp; int error; - int uhold; KASSERT_UNIONFS_VNODE(ap->a_vp); - error = 0; - uhold = 0; vp = ap->a_vp; unp = VTOUNIONFS(vp); if (unp == NULL) - goto unionfs_unlock_null_vnode; - lvp = unp->un_lowervp; - uvp = unp->un_uppervp; + return (vop_stdunlock(ap)); - if (lvp != NULLVP) { - vholdnz(lvp); - error = VOP_UNLOCK(lvp); - } + tvp = (unp->un_uppervp != NULL ? unp->un_uppervp : unp->un_lowervp); - if (error == 0 && uvp != NULLVP) { - vholdnz(uvp); - uhold = 1; - error = VOP_UNLOCK(uvp); - } - - if (lvp != NULLVP) - vdrop(lvp); - if (uhold != 0) - vdrop(uvp); + vholdnz(tvp); + error = VOP_UNLOCK(tvp); + vdrop(tvp); - return error; - -unionfs_unlock_null_vnode: - return (vop_stdunlock(ap)); + return (error); } static int @@ -2192,7 +2273,7 @@ unionfs_advlock(struct vop_advlock_args *ap) uvp = unp->un_uppervp; if (uvp == NULLVP) { - error = unionfs_copyfile(unp, 1, td->td_ucred, td); + error = unionfs_copyfile(ap->a_vp, 1, td->td_ucred, td); if (error != 0) goto unionfs_advlock_abort; uvp = unp->un_uppervp; @@ -2294,7 +2375,7 @@ unionfs_setacl(struct vop_setacl_args *ap) return (EROFS); if (uvp == NULLVP && lvp->v_type == VREG) { - if ((error = unionfs_copyfile(unp, 1, ap->a_cred, td)) != 0) + if ((error = unionfs_copyfile(ap->a_vp, 1, ap->a_cred, td)) != 0) return (error); uvp = unp->un_uppervp; } @@ -2467,9 +2548,10 @@ unionfs_setextattr(struct vop_setextattr_args *ap) if (ovp == lvp && lvp->v_type == VREG) { VOP_CLOSEEXTATTR(lvp, 0, cred, td); if (uvp == NULLVP && - (error = unionfs_copyfile(unp, 1, cred, td)) != 0) { + (error = unionfs_copyfile(ap->a_vp, 1, cred, td)) != 0) { unionfs_setextattr_reopen: - if ((unp->un_flag & UNIONFS_OPENEXTL) && + unp = VTOUNIONFS(ap->a_vp); + if (unp != NULL && (unp->un_flag & UNIONFS_OPENEXTL) && VOP_OPENEXTATTR(lvp, cred, td)) { #ifdef DIAGNOSTIC panic("unionfs: VOP_OPENEXTATTR failed"); @@ -2561,7 +2643,7 @@ unionfs_deleteextattr(struct vop_deleteextattr_args *ap) if (ovp == lvp && lvp->v_type == VREG) { VOP_CLOSEEXTATTR(lvp, 0, cred, td); if (uvp == NULLVP && - (error = unionfs_copyfile(unp, 1, cred, td)) != 0) { + (error = unionfs_copyfile(ap->a_vp, 1, cred, td)) != 0) { unionfs_deleteextattr_reopen: if ((unp->un_flag & UNIONFS_OPENEXTL) && VOP_OPENEXTATTR(lvp, cred, td)) { @@ -2613,7 +2695,7 @@ unionfs_setlabel(struct vop_setlabel_args *ap) return (EROFS); if (uvp == NULLVP && lvp->v_type == VREG) { - if ((error = unionfs_copyfile(unp, 1, ap->a_cred, td)) != 0) + if ((error = unionfs_copyfile(ap->a_vp, 1, ap->a_cred, td)) != 0) return (error); uvp = unp->un_uppervp; } @@ -2665,7 +2747,7 @@ static int unionfs_vput_pair(struct vop_vput_pair_args *ap) { struct mount *mp; - struct vnode *dvp, *vp, **vpp, *lvp, *ldvp, *uvp, *udvp, *tempvp; + struct vnode *dvp, *vp, **vpp, *lvp, *uvp, *tvp, *tdvp, *tempvp; struct unionfs_node *dunp, *unp; int error, res; @@ -2674,11 +2756,14 @@ unionfs_vput_pair(struct vop_vput_pair_args *ap) vp = NULLVP; lvp = NULLVP; uvp = NULLVP; + tvp = NULLVP; unp = NULL; dunp = VTOUNIONFS(dvp); - udvp = dunp->un_uppervp; - ldvp = dunp->un_lowervp; + if (dunp->un_uppervp != NULL) + tdvp = dunp->un_uppervp; + else + tdvp = dunp->un_lowervp; /* * Underlying vnodes should be locked because the encompassing unionfs @@ -2686,10 +2771,7 @@ unionfs_vput_pair(struct vop_vput_pair_args *ap) * only be on the unionfs node. Reference them now so that the vput()s * performed by VOP_VPUT_PAIR() will have a reference to drop. */ - if (udvp != NULLVP) - vref(udvp); - if (ldvp != NULLVP) - vref(ldvp); + vref(tdvp); if (vpp != NULL) vp = *vpp; @@ -2699,9 +2781,10 @@ unionfs_vput_pair(struct vop_vput_pair_args *ap) uvp = unp->un_uppervp; lvp = unp->un_lowervp; if (uvp != NULLVP) - vref(uvp); - if (lvp != NULLVP) - vref(lvp); + tvp = uvp; + else + tvp = lvp; + vref(tvp); /* * If we're being asked to return a locked child vnode, then @@ -2721,31 +2804,19 @@ unionfs_vput_pair(struct vop_vput_pair_args *ap) } } - /* - * TODO: Because unionfs_lock() locks both the lower and upper vnodes - * (if available), we must also call VOP_VPUT_PAIR() on both the lower - * and upper parent/child pairs. If unionfs_lock() is reworked to lock - * only a single vnode, this code will need to change to also only - * operate on one vnode pair. - */ - ASSERT_VOP_LOCKED(ldvp, __func__); - ASSERT_VOP_LOCKED(udvp, __func__); - ASSERT_VOP_LOCKED(lvp, __func__); - ASSERT_VOP_LOCKED(uvp, __func__); - - KASSERT(lvp == NULLVP || ldvp != NULLVP, - ("%s: NULL ldvp with non-NULL lvp", __func__)); - if (ldvp != NULLVP) - res = VOP_VPUT_PAIR(ldvp, lvp != NULLVP ? &lvp : NULL, true); - KASSERT(uvp == NULLVP || udvp != NULLVP, - ("%s: NULL udvp with non-NULL uvp", __func__)); - if (udvp != NULLVP) - res = VOP_VPUT_PAIR(udvp, uvp != NULLVP ? &uvp : NULL, true); - - ASSERT_VOP_UNLOCKED(ldvp, __func__); - ASSERT_VOP_UNLOCKED(udvp, __func__); - ASSERT_VOP_UNLOCKED(lvp, __func__); - ASSERT_VOP_UNLOCKED(uvp, __func__); + ASSERT_VOP_LOCKED(tdvp, __func__); + ASSERT_VOP_LOCKED(tvp, __func__); + + if (tdvp == dunp->un_uppervp && tvp != NULLVP && tvp == lvp) { + vput(tvp); + vput(tdvp); + res = 0; + } else { + res = VOP_VPUT_PAIR(tdvp, tvp != NULLVP ? &tvp : NULL, true); + } + + ASSERT_VOP_UNLOCKED(tdvp, __func__); + ASSERT_VOP_UNLOCKED(tvp, __func__); /* * VOP_VPUT_PAIR() dropped the references we added to the underlying -- Test scenario: src/tools/test/stress2/misc/unionfs9.sh