No panic, No vmcore KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2022 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #1 ufs-n252659-7f3a2938bfe-dirty: Wed Jan 26 11:07:45 CET 2022 pho@mercat1.netperf.freebsd.org:/var/tmp/deviant3/sys/amd64/compile/PHO amd64 FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.17-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33176588288 (31639 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 11 7 5 4 2 8 6 9 10 3 random: entropy device external interface kbd1 at kbdmux0 vtvga0: smbios0: at iomem 0xf0560-0xf057e smbios0: Version: 2.8, BCD Revision: 2.7 aesni0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 apei0: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: EEPROM V1.63-0 eTrack 0x800009fa igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: EEPROM V1.63-0 eTrack 0x800009fa igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb433000-0xfb4333ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb432000-0xfb4327ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounter "TSC-low" frequency 1749998475 Hz quality 1000 Timecounters tick every 1.000 msec Expensive timeout(9) function: 0xffffffff8050a720(0xfffffe0109fa4000) 0.004797126 s ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> ses0: ada0 in 'Slot 01', SATA Slot: scbus1 target 0 Expensive timeout(9) function: 0xffffffff80a57690(0xffffffff81ae9ad8) 0.007401421 s mlx5_core0: mem 0xf8000000-0xf9ffffff irq 32 at device 0.0 on pci3 mlx5: Mellanox Core driver 3.7.0 (July 2021)ugen0.1: <0x8086 XHCI root HUB> at usbus0 ugen1.1: at usbus1 ugen2.1: at usbus2 uhub0 on usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1 on usbus1 uhub1: on usbus1 uhub2 on usbus2 uhub2: on usbus2 mlx5_core0: WARN: mlx5_vsc_set_space:124:(pid 0): Space 0x7 is not supported. mlx5_core0: WARN: mlx5_fwdump_prep:99:(pid 0): VSC scan space is not supported mlx5_core0: INFO: init_one:1614:(pid 0): r 2 mlx5_core1: mem 0xf6000000-0xf7ffffff irq 32 at device 0.1 on pci3 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 813 files, 3059 used, 1262348 free (84 frags, 157783 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc10 /usr/local/lib/gcc8 /usr/local/lib/gcc9 /usr/local/lib/perl5/5.32/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: mercat1.netperf.freebsd.org. Setting up harvesting: PURE_RDRAND,[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . Autoloading module: acpi_wmi.ko Autoloading module: ichsmb.ko Autoloading module: ioat.ko ioat0: mem 0xfb42c000-0xfb42ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0xfb428000-0xfb42bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0xfb424000-0xfb427fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0xfb420000-0xfb423fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0xfb41c000-0xfb41ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0xfb418000-0xfb41bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0xfb414000-0xfb417fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0xfb410000-0xfb413fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 ichsmb0: port 0x580-0x59f mem 0xfb431000-0xfb4310ff irq 18 at device 31.3 on pci1 smbus0: on ichsmb0 acpi_wmi0: on acpi0 acpi_wmi0: cannot find EC device lo0: link state changed to UP Starting Network: lo0 igb0 igb1 mce0 mce1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8863 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect status: no carrier nd6 options=21 igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Autoloading module: acpi_wmi.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: ichsmb.ko Starting Network: mce0. mce0: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:38 media: Ethernet autoselect status: no carrier nd6 options=29 Starting Network: mce1. mce1: flags=8822 metric 0 mtu 1500 options=7eed07bb ether 7c:fe:90:30:ee:39 media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid.ko Autoloading module: ums.ko Autoloading module: usbhid.ko ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192if_delmulti_locked: detaching ifnet instance 0xfffff800045d3800 .168.5.1 bound to 192.168.5.171 -- renewal in 300 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Recovering vi editor sessions:. Updating motd:. Clearing /tmp (X related). Starting syslogd. Starting watchdogd. No core dumps found. Setting date via ntp. 26 Jan 11:10:20 ntpdate[2599]: step time server 192.168.5.1 offset -0.874360 sec Starting rpcbind. NFS access cache time=60 Starting statd. Starting lockd. Starting powerd. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Mounting late filesystems:. Starting inetd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. sysctl: unknown oid 'debug.fail_point.pmap_allocpte_page_alloc' at line 12 Wed Jan 26 11:10 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Jan 26 11:10:25 mercat1 sshd[2741]: error: PAM: Authentication error for pho from 192.168.5.1 Jan 26 11:10:29 mercat1 su[2752]: pho to root on /dev/pts/0 20220126 11:10:59 all (1/1): snap11.sh lock order reversal: 1st 0xfffffe010e60e080 pbufwait (pbufwait, lockmgr) @ vm/vm_pager.c:470 2nd 0xfffff804a3ef7730 snaplk (snaplk, lockmgr) @ dev/md/md.c:977 lock order pbufwait -> snaplk attempted at: #0 0xffffffff80c7cc3d at witness_checkorder+0xbdd #1 0xffffffff80bd9edc at lockmgr_lock_flags+0x16c #2 0xffffffff80f6cd4d at ffs_lock+0x9d #3 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #4 0xffffffff80d149e4 at _vn_lock+0x54 #5 0xffffffff80708629 at mdstart_vnode+0x499 #6 0xffffffff807060ec at md_kthread+0x20c #7 0xffffffff80bc3180 at fork_exit+0x80 #8 0xffffffff810d59ce at fork_trampoline+0xe lock order reversal: 1st 0xfffffe0038287b90 bufwait (bufwait, lockmgr) @ ufs/ffs/ffs_vnops.c:318 2nd 0xfffff804a3ef7730 snaplk (snaplk, lockmgr) @ ufs/ffs/ffs_snapshot.c:2508 lock order snaplk -> bufwait established at: #0 0xffffffff80c7c37e at witness_checkorder+0x31e #1 0xffffffff80bda5fc at lockmgr_xlock_hard+0x6c #2 0xffffffff80bdaf09 at __lockmgr_args+0x1f9 #3 0xffffffff80cfaac0 at bnoreuselist+0xc0 #4 0xffffffff80ce709e at vop_stdadvise+0x14e #5 0xffffffff811d3149 at VOP_ADVISE_APV+0x59 #6 0xffffffff807086f4 at mdstart_vnode+0x564 #7 0xffffffff807060ec at md_kthread+0x20c #8 0xffffffff80bc3180 at fork_exit+0x80 #9 0xffffffff810d59ce at fork_trampoline+0xe lock order bufwait -> snaplk attempted at: #0 0xffffffff80c7cc3d at witness_checkorder+0xbdd #1 0xffffffff80bda5fc at lockmgr_xlock_hard+0x6c #2 0xffffffff80bdaf09 at __lockmgr_args+0x1f9 #3 0xffffffff80f344a5 at _lockmgr_args+0x85 #4 0xffffffff80f37f36 at ffs_copyonwrite+0x2e6 #5 0xffffffff80f6bb4f at ffs_geom_strategy+0xcf #6 0xffffffff80f80cf3 at ufs_strategy+0x83 #7 0xffffffff811d0bd1 at VOP_STRATEGY_APV+0x61 #8 0xffffffff80ccd45e at bufstrategy+0x3e #9 0xffffffff80ccd2fc at bufwrite+0x24c #10 0xffffffff80f6e1d1 at ffs_syncvnode+0x3c1 #11 0xffffffff80f6cbee at ffs_fsync+0x1e #12 0xffffffff811cf34c at VOP_FSYNC_APV+0x3c #13 0xffffffff80f7e954 at ufs_remove+0x114 #14 0xffffffff811cf4f5 at VOP_REMOVE_APV+0x85 #15 0xffffffff80d0c48a at kern_funlinkat+0x35a #16 0xffffffff80d0c128 at sys_unlink+0x28 #17 0xffffffff81104455 at amd64_syscall+0x145 lock order reversal: 1st 0xfffff804a3ef7730 snaplk (snaplk, lockmgr) @ ufs/ufs/ufs_vnops.c:1037 2nd 0xfffffe014ecb6070 ufs (ufs, lockmgr) @ ufs/ffs/ffs_snapshot.c:1786 lock order ufs -> snaplk established at: #0 0xffffffff80c7c37e at witness_checkorder+0x31e #1 0xffffffff80bd9dec at lockmgr_lock_flags+0x7c #2 0xffffffff80f6cd4d at ffs_lock+0x9d #3 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #4 0xffffffff80d149e4 at _vn_lock+0x54 #5 0xffffffff80cfc1ba at vget_finish+0x7a #6 0xffffffff80cea27d at vfs_hash_get+0xcd #7 0xffffffff80f682c8 at ffs_vgetf+0x58 #8 0xffffffff80f77047 at ufs_lookup_ino+0xbe7 #9 0xffffffff811cd14a at VOP_CACHEDLOOKUP_APV+0x5a #10 0xffffffff80cdb946 at vfs_cache_lookup+0xa6 #11 0xffffffff811ccf8a at VOP_LOOKUP_APV+0x5a #12 0xffffffff80cecd81 at lookup+0x4c1 #13 0xffffffff80cebc18 at namei+0x3f8 #14 0xffffffff80d14140 at vn_open_cred+0x4b0 #15 0xffffffff80d0ab06 at kern_openat+0x286 #16 0xffffffff81104455 at amd64_syscall+0x145 #17 0xffffffff810d526b at fast_syscall_common+0xf8 lock order snaplk -> ufs attempted at: #0 0xffffffff80c7cc3d at witness_checkorder+0xbdd #1 0xffffffff80bda5fc at lockmgr_xlock_hard+0x6c #2 0xffffffff80bdaf09 at __lockmgr_args+0x1f9 #3 0xffffffff80f344a5 at _lockmgr_args+0x85 #4 0xffffffff80f35988 at ffs_snapremove+0x7e8 #5 0xffffffff80f2de85 at ffs_truncate+0x255 #6 0xffffffff80f7618a at ufs_inactive+0x1aa #7 0xffffffff811d02e9 at VOP_INACTIVE_APV+0x59 #8 0xffffffff80cfd42a at vinactivef+0xda #9 0xffffffff80cfca7a at vput_final+0x2aa #10 0xffffffff80d0c3a5 at kern_funlinkat+0x275 #11 0xffffffff80d0c128 at sys_unlink+0x28 #12 0xffffffff81104455 at amd64_syscall+0x145 #13 0xffffffff810d526b at fast_syscall_common+0xf8 20220126 11:13:03 all (1/1): snap11.sh 20220126 11:15:06 all (1/1): snap11.sh 20220126 11:17:12 all (1/1): snap11.sh 20220126 11:19:17 all (1/1): snap11.sh 20220126 11:21:24 all (1/1): snap11.sh FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Jan 26 11:22:01 mercat1 su[14115]: pho to root on /dev/pts/1 stress2: pts leak: 1/2 20220126 11:23:27 all (1/1): snap11.sh Jan 26 11:24:30 mercat1 kernel: pid 30732 (swap), jid 0, uid 0, was killed: a thread waited too long to allocate a page Jan 26 11:24:47 mercat1 kernel: pid 30747 (rw), uid 0 inumber 4 on /mnt: filesystem full Jan 26 11:25:09 mercat1 kernel: pid 30740 (rw), uid 0 inumber 4325 on /mnt: filesystem full 20220126 11:25:17 all (1/1): snap11.sh 20220126 11:27:23 all (1/1): snap11.sh 20220126 11:29:27 all (1/1): snap11.sh 20220126 11:31:33 all (1/1): snap11.sh 20220126 11:33:37 all (1/1): snap11.sh 20220126 11:35:39 all (1/1): snap11.sh 20220126 11:37:42 all (1/1): snap11.sh 20220126 11:39:44 all (1/1): snap11.sh 20220126 11:41:48 all (1/1): snap11.sh 20220126 11:43:54 all (1/1): snap11.sh 20220126 11:45:57 all (1/1): snap11.sh 20220126 11:48:01 all (1/1): snap11.sh 20220126 11:50:04 all (1/1): snap11.sh 20220126 11:52:09 all (1/1): snap11.sh 20220126 11:54:11 all (1/1): snap11.sh 20220126 11:56:15 all (1/1): snap11.sh 20220126 11:58:20 all (1/1): snap11.sh 20220126 12:00:25 all (1/1): snap11.sh 20220126 12:02:27 all (1/1): snap11.sh 20220126 12:04:30 all (1/1): snap11.sh 20220126 12:06:33 all (1/1): snap11.sh 20220126 12:08:36 all (1/1): snap11.sh 20220126 12:10:38 all (1/1): snap11.sh Jan 26 12:11:30 mercat1 kernel: pid 66129 (swap), jid 0, uid 0, was killed: a thread waited too long to allocate a page 20220126 12:12:47 all (1/1): snap11.sh 20220126 12:14:51 all (1/1): snap11.sh 20220126 12:16:54 all (1/1): snap11.sh 20220126 12:18:58 all (1/1): snap11.sh 20220126 12:21:00 all (1/1): snap11.sh 20220126 12:23:04 all (1/1): snap11.sh 20220126 12:25:09 all (1/1): snap11.sh 20220126 12:27:16 all (1/1): snap11.sh 20220126 12:29:19 all (1/1): snap11.sh 20220126 12:31:23 all (1/1): snap11.sh 20220126 12:33:26 all (1/1): snap11.sh 20220126 12:35:29 all (1/1): snap11.sh 20220126 12:37:32 all (1/1): snap11.sh 20220126 12:39:34 all (1/1): snap11.sh 20220126 12:41:38 all (1/1): snap11.sh 20220126 12:43:45 all (1/1): snap11.sh 20220126 12:45:51 all (1/1): snap11.sh 20220126 12:47:53 all (1/1): snap11.sh 20220126 12:49:57 all (1/1): snap11.sh 20220126 12:52:01 all (1/1): snap11.sh 20220126 12:54:03 all (1/1): snap11.sh swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio Jan 26 12:54:59 mercat1 kernel: pid 80596 (mkdir), uid 0 inumber 3486 on /mnt: filesystem full IN_ENDOFF: failed to truncate, error 28 0xfffffe015235b518: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8050820e630 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe014180e740 (pid 80587, creat, tid 100216) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #1 0xffffffff80f6cd4d at ffs_lock+0x9d #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d149e4 at _vn_lock+0x54 #4 0xffffffff80cec985 at lookup+0xc5 #5 0xffffffff80cebc18 at namei+0x3f8 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #7 0xffffffff80d0ab06 at kern_openat+0x286 #8 0xffffffff81104455 at amd64_syscall+0x145 #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 generation=4d07ffd2, uid=0, gid=0, flags=0x0 ino 196614, on dev md10a IN_ENDOFF: failed to truncate, error 28 IN_ENDOFF: failed to truncate, error 28 0xfffffe015235b518: 0xfffffe0151ed59c8: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8071e04a840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 v_object 0xfffff8050820e630 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe014180e740 (pid 80587, creat, tid 100216) lock type ufs: EXCL by thread 0xfffffe014180ae40 (pid 80580, creat, tid 100214) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #1 0xffffffff80f6cd4d at ffs_lock+0x9d #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #1 0xffffffff80f6cd4d at ffs_lock+0x9d #3 0xffffffff80d149e4 at _vn_lock+0x54 #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #4 0xffffffff80cec985 at lookup+0xc5 #3 0xffffffff80d149e4 at _vn_lock+0x54 #4 0xffffffff80cec985 at lookup+0xc5 #5 0xffffffff80cebc18 at namei+0x3f8 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #5 0xffffffff80cebc18 at namei+0x3f8 #7 0xffffffff80d0ab06 at kern_openat+0x286 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #8 0xffffffff81104455 at amd64_syscall+0x145 #7 0xffffffff80d0ab06 at kern_openat+0x286 #8 0xffffffff81104455 at amd64_syscall+0x145 #9 0xffffffff810d526b at fast_syscall_common+0xf8 #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 nlink=2, effnlink=2, size=24576 generation=cfb2a289, uid=0, gid=0, flags=0x0 , extsize 0 ino 196610, on dev md10a generation=4d07ffd2, uid=0, gid=0, flags=0x0 ino 196614, on dev md10a IN_ENDOFF: failed to truncate, error 28 0xfffffe015235b518: type VDIR IN_ENDOFF: failed to truncate, error 28 0xfffffe0151ed59c8: usecount 2, writecount 0, refcount 2 seqc users 0type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8050820e630 ref 0 pages 0 cleanbuf 0 dirtybuf 1 flags (VMP_LAZYLIST) v_object 0xfffff8071e04a840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe014180ae40 (pid 80580, creat, tid 100214) lock type ufs: EXCL by thread 0xfffffe014180e740 (pid 80587, creat, tid 100216) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #1 0xffffffff80f6cd4d at ffs_lock+0x9d #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #1 0xffffffff80f6cd4d at ffs_lock+0x9d #3 0xffffffff80d149e4 at _vn_lock+0x54 #4 0xffffffff80cec985 at lookup+0xc5 #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #5 0xffffffff80cebc18 at namei+0x3f8 #3 0xffffffff80d149e4 at _vn_lock+0x54 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #4 0xffffffff80cec985 at lookup+0xc5 #7 0xffffffff80d0ab06 at kern_openat+0x286 #5 0xffffffff80cebc18 at namei+0x3f8 #8 0xffffffff81104455 at amd64_syscall+0x145 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 #7 0xffffffff80d0ab06 at kern_openat+0x286 generation=cfb2a289, uid=0, gid=0, flags=0x0 ino 196610, on dev md10a #8 0xffffffff81104455 at amd64_syscall+0x145 #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 generation=4d07ffd2, uid=0, gid=0, flags=0x0 ino 196614, on dev md10a Jan 26 12:55:00 mercat1 kernel: pid 80576 (creat), uid 0 inumber 3486 on /mnt: filesystem full Jan 26 12:55:02 mercat1 kernel: pid 80593 (creat), uid 0 inumber 3486 on /mnt: filesystem full IN_ENDOFF: failed to truncate, error 28 0xfffffe0151ed59c8: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8071e04a840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe014180ae40 (pid 80580, creat, tid 100214) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #1 0xffffffff80f6cd4d at ffs_lock+0x9d #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d149e4 at _vn_lock+0x54 #4 0xffffffff80cec985 at lookup+0xc5 #5 0xffffffff80cebc18 at namei+0x3f8 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #7 0xffffffff80d0ab06 at kern_openat+0x286 #8 0xffffffff81104455 at amd64_syscall+0x145 #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 generation=cfb2a289, uid=0, gid=0, flags=0x0 ino 196610, on dev md10a IN_ENDOFF: failed to truncate, error 28 0xfffffe0151ed59c8: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8071e04a840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe014180ae40 (pid 80580, creat, tid 100214) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #1 0xffffffff80f6cd4d at ffs_lock+0x9d #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #3 0xffffffff80d149e4 at _vn_lock+0x54 #4 0xffffffff80cec985 at lookup+0xc5 #5 0xffffffff80cebc18 at namei+0x3f8 IN_ENDOFF: failed to truncate, error 28 #6 0xffffffff80d13dbc at vn_open_cred+0x12c 0xfffffe0153da9068: type VDIR #7 0xffffffff80d0ab06 at kern_openat+0x286 usecount 2, writecount 0, refcount 2 seqc users 0 mountedhere 0 #8 0xffffffff81104455 at amd64_syscall+0x145 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8054f93c318 ref 0 pages 0 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xfffffe01508f73a0 (pid 80583, creat, tid 100662) #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf IN_ENDOFF: failed to truncate, error 28 0xfffffe015235b518: type VDIR usecount 2, writecount 0, refcount 2 seqc users 0#1 0xffffffff80f6cd4d at ffs_lock+0x9d mountedhere 0 hold count flags () #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 flags (VMP_LAZYLIST) #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c v_object 0xfffff8050820e630 ref 0 pages 0 cleanbuf 0 dirtybuf 1 generation=cfb2a289, uid=0, gid=0, flags=0x0 lock type ufs: EXCL by thread 0xfffffe014180e740 (pid 80587, creat, tid 100216) #3 0xffffffff80d149e4 at _vn_lock+0x54 #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #4 0xffffffff80cec985 at lookup+0xc5 ino 196610, on dev md10a#5 0xffffffff80cebc18 at namei+0x3f8 #1 0xffffffff80f6cd4d at ffs_lock+0x9d IN_ENDOFF: failed to truncate, error 28 #6 0xffffffff80d13dbc at vn_open_cred+0x12c 0xfffffe0151ed59c8: #7 0xffffffff80d0ab06 at kern_openat+0x286 #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #8 0xffffffff81104455 at amd64_syscall+0x145 #3 0xffffffff80d149e4 at _vn_lock+0x54 type VDIR #9 0xffffffff810d526b at fast_syscall_common+0xf8 usecount 2, writecount 0, refcount 2 seqc users 0#4 0xffffffff80cec985 at lookup+0xc5 mountedhere 0 hold count flags () flags (VMP_LAZYLIST) v_object 0xfffff8071e04a840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 nlink=2, effnlink=2, size=24576, extsize 0 #5 0xffffffff80cebc18 at namei+0x3f8 IN_ENDOFF: failed to truncate, error 28 generation=dd2a96dc, uid=0, gid=0, flags=0x0 lock type ufs: EXCL by thread 0xfffffe014180ae40 (pid 80580, creat, tid 100214) ino 196612, on dev md10a #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #6 0xffffffff80d13dbc at vn_open_cred+0x12c #1 0xffffffff80f6cd4d at ffs_lock+0x9d 0xfffffe0156e86e10: #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c #7 0xffffffff80d0ab06 at kern_openat+0x286 #3 0xffffffff80d149e4 at _vn_lock+0x54 #8 0xffffffff81104455 at amd64_syscall+0x145 type VDIR #9 0xffffffff810d526b at fast_syscall_common+0xf8 usecount 2, writecount 0, refcount 2 seqc users 0 nlink=2, effnlink=2, size=24576, extsize 0 generation=4d07ffd2, uid=0, gid=0, flags=0x0 ino 196614, on dev md10a mountedhere 0 hold count flags () #4 0xffffffff80cec985 at lookup+0xc5 flags (VMP_LAZYLIST)#5 0xffffffff80cebc18 at namei+0x3f8 v_object 0xfffff8013c806840 ref 0 pages 0 cleanbuf 0 dirtybuf 1 #6 0xffffffff80d13dbc at vn_open_cred+0x12c lock type ufs: EXCL by thread 0xfffffe0141831e40 (pid 80593, creat, tid 100558) #7 0xffffffff80d0ab06 at kern_openat+0x286 #0 0xffffffff80bd9f3f at lockmgr_lock_flags+0x1cf #8 0xffffffff81104455 at amd64_syscall+0x145 #1 0xffffffff80f6cd4d at ffs_lock+0x9d #9 0xffffffff810d526b at fast_syscall_common+0xf8 #2 0xffffffff811d075c at VOP_LOCK1_APV+0x3c nlink=2, effnlink=2, size=24576, extsize 0 generation=cfb2a289, uid=0, gid=0, flags=0x0 #3 0xffffffff80d149e4 at _vn_lock+0x54 ino 196610, on dev md10a#4 0xffffffff80cec985 at lookup+0xc5 #5 0xffffffff80cebc18 at namei+0x3f8 #6 0xffffffff80d13dbc at vn_open_cred+0x12c #7 0xffffffff80d0ab06 at kern_openat+0x286 #8 0xffffffff81104455 at amd64_syscall+0x145 #9 0xffffffff810d526b at fast_syscall_common+0xf8 nlink=2, effnlink=2, size=24576, extsize 0 generation=41053a0f, uid=0, gid=0, flags=0x0 ino 196616, on dev md10a 20220126 12:56:13 all (1/1): snap11.sh 20220126 12:58:15 all (1/1): snap11.sh 20220126 13:00:19 all (1/1): snap11.sh 20220126 13:02:22 all (1/1): snap11.sh 20220126 13:04:26 all (1/1): snap11.sh 20220126 13:06:30 all (1/1): snap11.sh 20220126 13:08:33 all (1/1): snap11.sh 20220126 13:10:37 all (1/1): snap11.sh 20220126 13:12:40 all (1/1): snap11.sh 20220126 13:14:44 all (1/1): snap11.sh 20220126 13:16:49 all (1/1): snap11.sh 20220126 13:18:52 all (1/1): snap11.sh 20220126 13:21:12 all (1/1): snap11.sh 20220126 13:23:18 all (1/1): snap11.sh 20220126 13:25:21 all (1/1): snap11.sh 20220126 13:27:24 all (1/1): snap11.sh 20220126 13:29:27 all (1/1): snap11.sh 20220126 13:31:31 all (1/1): snap11.sh 20220126 13:33:33 all (1/1): snap11.sh 20220126 13:35:37 all (1/1): snap11.sh 20220126 13:37:46 all (1/1): snap11.sh 20220126 13:39:50 all (1/1): snap11.sh 20220126 13:41:53 all (1/1): snap11.sh 20220126 13:43:56 all (1/1): snap11.sh 20220126 13:46:02 all (1/1): snap11.sh 20220126 13:48:04 all (1/1): snap11.sh 20220126 13:50:08 all (1/1): snap11.sh 20220126 13:50:48 all.sh done, elapsed 0 day(s), 02:39.50 -- Test scenario: misc/snap11.sh